-
Chimbuko: A Workflow-Level Scalable Performance Trace Analysis Tool
Authors:
Sungsoo Ha,
Wonyong Jeong,
Gyorgy Matyasfalvi,
Cong Xie,
Kevin Huck,
Jong Youl Choi,
Abid Malik,
Li Tang,
Hubertus Van Dam,
Line Pouchard,
Wei Xu,
Shinjae Yoo,
Nicholas D'Imperio,
Kerstin Kleese Van Dam
Abstract:
Because of the limits input/output systems currently impose on high-performance computing systems, a new generation of workflows that include online data reduction and analysis is emerging. Diagnosing their performance requires sophisticated performance analysis capabilities due to the complexity of execution patterns and underlying hardware, and no tool could handle the voluminous performance tra…
▽ More
Because of the limits input/output systems currently impose on high-performance computing systems, a new generation of workflows that include online data reduction and analysis is emerging. Diagnosing their performance requires sophisticated performance analysis capabilities due to the complexity of execution patterns and underlying hardware, and no tool could handle the voluminous performance trace data needed to detect potential problems. This work introduces Chimbuko, a performance analysis framework that provides real-time, distributed, in situ anomaly detection. Data volumes are reduced for human-level processing without losing necessary details. Chimbuko supports online performance monitoring via a visualization module that presents the overall workflow anomaly distribution, call stacks, and timelines. Chimbuko also supports the capture and reduction of performance provenance. To the best of our knowledge, Chimbuko is the first online, distributed, and scalable workflow-level performance trace analysis framework, and we demonstrate the tool's usefulness on Oak Ridge National Laboratory's Summit system.
△ Less
Submitted 31 August, 2020;
originally announced August 2020.
-
Building Near-Real-Time Processing Pipelines with the Spark-MPI Platform
Authors:
Nikolay Malitsky,
Aashish Chaudhary,
Sebastien Jourdain,
Matt Cowan,
Patrick O'Leary,
Marcus Hanwell,
Kerstin Kleese Van Dam
Abstract:
Advances in detectors and computational technologies provide new opportunities for applied research and the fundamental sciences. Concurrently, dramatic increases in the three Vs (Volume, Velocity, and Variety) of experimental data and the scale of computational tasks produced the demand for new real-time processing systems at experimental facilities. Recently, this demand was addressed by the Spa…
▽ More
Advances in detectors and computational technologies provide new opportunities for applied research and the fundamental sciences. Concurrently, dramatic increases in the three Vs (Volume, Velocity, and Variety) of experimental data and the scale of computational tasks produced the demand for new real-time processing systems at experimental facilities. Recently, this demand was addressed by the Spark-MPI approach connecting the Spark data-intensive platform with the MPI high-performance framework. In contrast with existing data management and analytics systems, Spark introduced a new middleware based on resilient distributed datasets (RDDs), which decoupled various data sources from high-level processing algorithms. The RDD middleware significantly advanced the scope of data-intensive applications, spreading from SQL queries to machine learning to graph processing. Spark-MPI further extended the Spark ecosystem with the MPI applications using the Process Management Interface. The paper explores this integrated platform within the context of online ptychographic and tomographic reconstruction pipelines.
△ Less
Submitted 13 May, 2018;
originally announced May 2018.
-
Use Cases of Computational Reproducibility for Scientific Workflows at Exascale
Authors:
Line Pouchard,
Sterling Baldwin,
Todd Elsethagen,
Carlos Gamboa,
Shantenu Jha,
Bibi Raju,
Eric Stephan,
Li Tang,
Kerstin Kleese Van Dam
Abstract:
We propose an approach for improved reproducibility that includes capturing and relating provenance characteristics and performance metrics, in a hybrid queriable system, the ProvEn server. The system capabilities are illustrated on two use cases: scientific reproducibility of results in the ACME climate simulations and performance reproducibility in molecular dynamics workflows on HPC computing p…
▽ More
We propose an approach for improved reproducibility that includes capturing and relating provenance characteristics and performance metrics, in a hybrid queriable system, the ProvEn server. The system capabilities are illustrated on two use cases: scientific reproducibility of results in the ACME climate simulations and performance reproducibility in molecular dynamics workflows on HPC computing platforms.
△ Less
Submitted 20 April, 2018;
originally announced May 2018.
-
Status Report of the DPHEP Study Group: Towards a Global Effort for Sustainable Data Preservation in High Energy Physics
Authors:
Z. Akopov,
Silvia Amerio,
David Asner,
Eduard Avetisyan,
Olof Barring,
James Beacham,
Matthew Bellis,
Gregorio Bernardi,
Siegfried Bethke,
Amber Boehnlein,
Travis Brooks,
Thomas Browder,
Rene Brun,
Concetta Cartaro,
Marco Cattaneo,
Gang Chen,
David Corney,
Kyle Cranmer,
Ray Culbertson,
Sunje Dallmeier-Tiessen,
Dmitri Denisov,
Cristinel Diaconu,
Vitaliy Dodonov,
Tony Doyle,
Gregory Dubois-Felsmann
, et al. (65 additional authors not shown)
Abstract:
Data from high-energy physics (HEP) experiments are collected with significant financial and human effort and are mostly unique. An inter-experimental study group on HEP data preservation and long-term analysis was convened as a panel of the International Committee for Future Accelerators (ICFA). The group was formed by large collider-based experiments and investigated the technical and organisati…
▽ More
Data from high-energy physics (HEP) experiments are collected with significant financial and human effort and are mostly unique. An inter-experimental study group on HEP data preservation and long-term analysis was convened as a panel of the International Committee for Future Accelerators (ICFA). The group was formed by large collider-based experiments and investigated the technical and organisational aspects of HEP data preservation. An intermediate report was released in November 2009 addressing the general issues of data preservation in HEP. This paper includes and extends the intermediate report. It provides an analysis of the research case for data preservation and a detailed description of the various projects at experiment, laboratory and international levels. In addition, the paper provides a concrete proposal for an international organisation in charge of the data management and policies in high-energy physics.
△ Less
Submitted 21 May, 2012;
originally announced May 2012.
