BDWatchdog: Real-time monitoring and profiling of Big Data applications and frameworks

• Published in: Future generation computer systems Vol. 87; pp. 420 - 437
• Main Authors: Enes, Jonatan, Expósito, Roberto R., Touriño, Juan
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01-10-2018
• Subjects: Big data; Flame graphs; Monitoring; Process-based analysis; Profiling; Time series; Process-based analysis; Flame graphs; Big data; Profiling; Monitoring; Time series
• ISSN: 0167-739X; 1872-7115
• DOI: 10.1016/j.future.2017.12.068

Current Big Data applications are characterized by a heavy use of system resources (e.g., CPU, disk) generally distributed across a cluster. To effectively improve their performance there is a critical need for an accurate analysis of both Big Data workloads and frameworks. This means to fully understand how the system resources are being used in order to identify potential bottlenecks, from resource to code bottlenecks. This paper presents BDWatchdog, a novel framework that allows real-time and scalable analysis of Big Data applications by combining time series for resource monitorization and flame graphs for code profiling, focusing on the processes that make up the workload rather than the underlying instances on which they are executed. This shift from the traditional system-based monitorization to a process-based analysis is interesting for new paradigms such as software containers or serverless computing, where the focus is put on applications and not on instances. BDWatchdog has been evaluated on a Big Data cloud-based service deployed at the CESGA supercomputing center. The experimental results show that a process-based analysis allows for a more effective visualization and overall improves the understanding of Big Data workloads. BDWatchdog is publicly available at http://bdwatchdog.dec.udc.es.
•New framework for real-time monitoring and profiling of Big Data applications.•Monitoring with resource time series and profiling with system and JVM flame graphs.•Process-based analysis allows higher-level operations like filtering and aggregation.•Architecture allows scalability to analyze Big Data-scale applications across clusters.•Framework allows to spot resource and code bottlenecks and characterize applications.