-
A Bespoke Workflow Management System for Data-Driven Urgent HPC
Authors:
Gordon P. S. Gibb,
Nick Brown,
Rupert W,
Nash,
Miguel Mendes,
Santiago Monedero,
Humberto Díaz Fidalgo,
Joaquín Ramírez Cisneros,
Adrián Cardil,
Max Kontak
Abstract:
In this paper we present a workflow management system which permits the kinds of data-driven workflows required by urgent computing, namely where new data is integrated into the workflow as a disaster progresses in order refine the predictions as time goes on. This allows the workflow to adapt to new data at runtime, a capability that most workflow management systems do not possess. The workflow m…
▽ More
In this paper we present a workflow management system which permits the kinds of data-driven workflows required by urgent computing, namely where new data is integrated into the workflow as a disaster progresses in order refine the predictions as time goes on. This allows the workflow to adapt to new data at runtime, a capability that most workflow management systems do not possess. The workflow management system was developed for the EU-funded VESTEC project, which aims to fuse HPC with real-time data for supporting urgent decision making. We first describe an example workflow from the VESTEC project, and show why existing workflow technologies do not meet the needs of the project. We then go on to present the design of our Workflow Management System, describe how it is implemented into the VESTEC system, and provide an example of the workflow system in use for a test case.
△ Less
Submitted 23 October, 2020;
originally announced October 2020.
-
The role of interactive super-computing in using HPC for urgent decision making
Authors:
Nick Brown,
Rupert Nash,
Gordon Gibb,
Bianca Prodan,
Max Kontak,
Vyacheslav Olshevsky,
Wei Der Chien
Abstract:
Technological advances are creating exciting new opportunities that have the potential to move HPC well beyond traditional computational workloads. In this paper we focus on the potential for HPC to be instrumental in responding to disasters such as wildfires, hurricanes, extreme flooding, earthquakes, tsunamis, winter weather conditions, and accidents. Driven by the VESTEC EU funded H2020 project…
▽ More
Technological advances are creating exciting new opportunities that have the potential to move HPC well beyond traditional computational workloads. In this paper we focus on the potential for HPC to be instrumental in responding to disasters such as wildfires, hurricanes, extreme flooding, earthquakes, tsunamis, winter weather conditions, and accidents. Driven by the VESTEC EU funded H2020 project, our research looks to prove HPC as a tool not only capable of simulating disasters once they have happened, but also one which is able to operate in a responsive mode, supporting disaster response teams making urgent decisions in real-time. Whilst this has the potential to revolutionise disaster response, it requires the ability to drive HPC interactively, both from the user's perspective and also based upon the arrival of data. As such interactivity is a critical component in enabling HPC to be exploited in the role of supporting disaster response teams so that urgent decision makers can make the correct decision first time, every time.
△ Less
Submitted 17 October, 2020;
originally announced October 2020.
-
Supercomputing with MPI meets the Common Workflow Language standards: an experience report
Authors:
Rupert W. Nash,
Nick Brown,
Michael R. Crusoe,
Max Kontak
Abstract:
Use of standards-based workflows is still somewhat unusual by high-performance computing users. In this paper we describe the experience of using the Common Workflow Language (CWL) standards to describe the execution, in parallel, of MPI-parallelised applications. In particular, we motivate and describe the simple extension to the specification which was required, as well as our implementation of…
▽ More
Use of standards-based workflows is still somewhat unusual by high-performance computing users. In this paper we describe the experience of using the Common Workflow Language (CWL) standards to describe the execution, in parallel, of MPI-parallelised applications. In particular, we motivate and describe the simple extension to the specification which was required, as well as our implementation of this within the CWL reference runner. We discuss some of the unexpected benefits, such as simple use of HPC-oriented performance measurement tools, and CWL software requirements interfacing with HPC module systems. We close with a request for comment from the community on how these features could be adopted within versions of the CWL standards.
△ Less
Submitted 1 October, 2020;
originally announced October 2020.
-
Statistical Parameter Selection for Clustering Persistence Diagrams
Authors:
Max Kontak,
Jules Vidal,
Julien Tierny
Abstract:
In urgent decision making applications, ensemble simulations are an important way to determine different outcome scenarios based on currently available data. In this paper, we will analyze the output of ensemble simulations by considering so-called persistence diagrams, which are reduced representations of the original data, motivated by the extraction of topological features. Based on a recently…
▽ More
In urgent decision making applications, ensemble simulations are an important way to determine different outcome scenarios based on currently available data. In this paper, we will analyze the output of ensemble simulations by considering so-called persistence diagrams, which are reduced representations of the original data, motivated by the extraction of topological features. Based on a recently published progressive algorithm for the clustering of persistence diagrams, we determine the optimal number of clusters, and therefore the number of significantly different outcome scenarios, by the minimization of established statistical score functions. Furthermore, we present a proof-of-concept prototype implementation of the statistical selection of the number of clusters and provide the results of an experimental study, where this implementation has been applied to real-world ensemble data sets.
△ Less
Submitted 17 October, 2019;
originally announced October 2019.
-
Three-dimensional simulation of nonwoven fabrics using a greedy approximation of the distribution of fiber directions
Authors:
Simone Gramsch,
Max Kontak,
Volker Michel
Abstract:
An elementary algorithm is used to simulate the industrial production of a fiber of a 3-dimensional nonwoven fabric. The algorithm simulates the fiber as a polyline where the direction of each segment is stochastically drawn based on a given probability density function (PDF) on the unit sphere. This PDF is obtained from data of directions of fiber fragments which originate from computer tomograph…
▽ More
An elementary algorithm is used to simulate the industrial production of a fiber of a 3-dimensional nonwoven fabric. The algorithm simulates the fiber as a polyline where the direction of each segment is stochastically drawn based on a given probability density function (PDF) on the unit sphere. This PDF is obtained from data of directions of fiber fragments which originate from computer tomography scans of a real non-woven fabric. However, the simulation algorithm requires numerous evaluations of the PDF. Since the established technique of a kernel density estimator leads to very high computational costs, a novel greedy algorithm for estimating a sparse representation of the PDF is introduced. Numerical tests for a synthetic and a real example are presented. In a realistic scenario, the introduced sparsity ansatz leads to a reduction of the computation time for 100 fibers from nearly 40 days to 41 minutes.
△ Less
Submitted 4 August, 2016;
originally announced August 2016.
