-
Workflows Community Summit 2022: A Roadmap Revolution
Authors:
Rafael Ferreira da Silva,
Rosa M. Badia,
Venkat Bala,
Debbie Bard,
Peer-Timo Bremer,
Ian Buckley,
Silvina Caino-Lores,
Kyle Chard,
Carole Goble,
Shantenu Jha,
Daniel S. Katz,
Daniel Laney,
Manish Parashar,
Frederic Suter,
Nick Tyler,
Thomas Uram,
Ilkay Altintas,
Stefan Andersson,
William Arndt,
Juan Aznar,
Jonathan Bader,
Bartosz Balis,
Chris Blanton,
Kelly Rosa Braghetto,
Aharon Brodutch
, et al. (80 additional authors not shown)
Abstract:
Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing and t…
▽ More
Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing and the evolving needs of emerging scientific applications, it is paramount that the development of novel scientific workflows and system functionalities seek to increase the efficiency, resilience, and pervasiveness of existing systems and applications. Specifically, the proliferation of machine learning/artificial intelligence (ML/AI) workflows, need for processing large scale datasets produced by instruments at the edge, intensification of near real-time data processing, support for long-term experiment campaigns, and emergence of quantum computing as an adjunct to HPC, have significantly changed the functional and operational requirements of workflow systems. Workflow systems now need to, for example, support data streams from the edge-to-cloud-to-HPC enable the management of many small-sized files, allow data reduction while ensuring high accuracy, orchestrate distributed services (workflows, instruments, data movement, provenance, publication, etc.) across computing and user facilities, among others. Further, to accelerate science, it is also necessary that these systems implement specifications/standards and APIs for seamless (horizontal and vertical) integration between systems and applications, as well as enabling the publication of workflows and their associated products according to the FAIR principles. This document reports on discussions and findings from the 2022 international edition of the Workflows Community Summit that took place on November 29 and 30, 2022.
△ Less
Submitted 31 March, 2023;
originally announced April 2023.
-
Information Entropy-based Camera Path Estimation for In-Situ Visualization
Authors:
Ken Iwata,
Naohisa Sakamoto,
Jorji Nonaka,
Chongke Bi
Abstract:
In-situ processing has widely been recognized as an effective approach for the visualization and analysis of large-scale simulation outputs from modern HPC systems. One of the most common approaches for batch-based in-situ visualization is the image- or video-based approach. In this kind of approach, a large number of rendered images are generated from different viewpoints at each time step and ha…
▽ More
In-situ processing has widely been recognized as an effective approach for the visualization and analysis of large-scale simulation outputs from modern HPC systems. One of the most common approaches for batch-based in-situ visualization is the image- or video-based approach. In this kind of approach, a large number of rendered images are generated from different viewpoints at each time step and has proven useful for detailed analysis of the main simulation results. However, during test runs and model calibration runs before the main simulation run, a quick overview might be sufficient and useful. In this work, we focused on selecting the viewpoints which provide as much information as possible by using information entropy to maximize the subsequent visual analysis task. However, by simply following the selected viewpoints at each of the visualization time steps will probably lead to a rapidly changing video, which can impact the understanding. Therefore, we have also worked on an efficient camera path estimation approach for connecting selected viewpoints, at regular intervals, to generate a smooth video. This resulting video is expected to assist in rapid understanding of the underlying simulation phenomena and can be helpful to narrow down the temporal region of interest to minimize the turnaround time during detailed visual exploration via image- or video-based visual analysis of the main simulation run. We implemented and evaluated the proposed approach using the OpenFOAM CFD application, on an x86-based Server and an ARM A64FX-based supercomputer (Fugaku), and we obtained positive evaluations from domain scientists.
△ Less
Submitted 30 January, 2023; v1 submitted 27 January, 2023;
originally announced January 2023.
-
Angular-based Edge Bundled Parallel Coordinates Plot for the Visual Analysis of Large Ensemble Simulation Data
Authors:
Keita Watanabe,
Naohisa Sakamoto,
Jorji Nonaka,
Yasumitsu Maejima
Abstract:
With the continuous increase in the computational power and resources of modern high-performance computing (HPC) systems, large-scale ensemble simulations have become widely used in various fields of science and engineering, and especially in meteorological and climate science. It is widely known that the simulation outputs are large time-varying, multivariate, and multivalued datasets which pose…
▽ More
With the continuous increase in the computational power and resources of modern high-performance computing (HPC) systems, large-scale ensemble simulations have become widely used in various fields of science and engineering, and especially in meteorological and climate science. It is widely known that the simulation outputs are large time-varying, multivariate, and multivalued datasets which pose a particular challenge to the visualization and analysis tasks. In this work, we focused on the widely used Parallel Coordinates Plot (PCP) to analyze the interrelations between different parameters, such as variables, among the members. However, PCP may suffer from visual cluttering and drawing performance with the increase on the data size to be analyzed, that is, the number of polylines. To overcome this problem, we present an extension to the PCP by adding Bézier curves connecting the angular distribution plots representing the mean and variance of the inclination of the line segments between parallel axes. The proposed Angular-based Parallel Coordinates Plot (APCP) is capable of presenting a simplified overview of the entire ensemble data set while maintaining the correlation information between the adjacent variables. To verify its effectiveness, we developed a visual analytics prototype system and evaluated by using a meteorological ensemble simulation output from the supercomputer Fugaku.
△ Less
Submitted 22 September, 2022;
originally announced September 2022.
-
A Visual Analytics Approach for Hardware System Monitoring with Streaming Functional Data Analysis
Authors:
Fnu Shilpika,
Takanori Fujiwara,
Naohisa Sakamoto,
Jorji Nonaka,
Kwan-Liu Ma
Abstract:
Many real-world applications involve analyzing time-dependent phenomena, which are intrinsically functional, consisting of curves varying over a continuum (e.g., time). When analyzing continuous data, functional data analysis (FDA) provides substantial benefits, such as the ability to study the derivatives and to restrict the ordering of data. However, continuous data inherently has infinite dimen…
▽ More
Many real-world applications involve analyzing time-dependent phenomena, which are intrinsically functional, consisting of curves varying over a continuum (e.g., time). When analyzing continuous data, functional data analysis (FDA) provides substantial benefits, such as the ability to study the derivatives and to restrict the ordering of data. However, continuous data inherently has infinite dimensions, and for a long time series, FDA methods often suffer from high computational costs. The analysis problem becomes even more challenging when updating the FDA results for continuously arriving data. In this paper, we present a visual analytics approach for monitoring and reviewing time series data streamed from a hardware system with a focus on identifying outliers by using FDA. To perform FDA while addressing the computational problem, we introduce new incremental and progressive algorithms that promptly generate the magnitude-shape (MS) plot, which conveys both the functional magnitude and shape outlyingness of time series data. In addition, by using an MS plot in conjunction with an FDA version of principal component analysis, we enhance the analyst's ability to investigate the visually-identified outliers. We illustrate the effectiveness of our approach with two use scenarios using real-world datasets. The resulting tool is evaluated by industry experts using real-world streaming datasets.
△ Less
Submitted 21 February, 2022; v1 submitted 25 November, 2020;
originally announced November 2020.
-
A Visual Analytics Framework for Reviewing Multivariate Time-Series Data with Dimensionality Reduction
Authors:
Takanori Fujiwara,
Shilpika,
Naohisa Sakamoto,
Jorji Nonaka,
Keiji Yamamoto,
Kwan-Liu Ma
Abstract:
Data-driven problem solving in many real-world applications involves analysis of time-dependent multivariate data, for which dimensionality reduction (DR) methods are often used to uncover the intrinsic structure and features of the data. However, DR is usually applied to a subset of data that is either single-time-point multivariate or univariate time-series, resulting in the need to manually exa…
▽ More
Data-driven problem solving in many real-world applications involves analysis of time-dependent multivariate data, for which dimensionality reduction (DR) methods are often used to uncover the intrinsic structure and features of the data. However, DR is usually applied to a subset of data that is either single-time-point multivariate or univariate time-series, resulting in the need to manually examine and correlate the DR results out of different data subsets. When the number of dimensions is large either in terms of the number of time points or attributes, this manual task becomes too tedious and infeasible. In this paper, we present MulTiDR, a new DR framework that enables processing of time-dependent multivariate data as a whole to provide a comprehensive overview of the data. With the framework, we employ DR in two steps. When treating the instances, time points, and attributes of the data as a 3D array, the first DR step reduces the three axes of the array to two, and the second DR step visualizes the data in a lower-dimensional space. In addition, by coupling with a contrastive learning method and interactive visualizations, our framework enhances analysts' ability to interpret DR results. We demonstrate the effectiveness of our framework with four case studies using real-world datasets.
△ Less
Submitted 27 October, 2021; v1 submitted 2 August, 2020;
originally announced August 2020.
