-
Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities
Authors:
William F Godoy,
Andrei T Savici,
Steven E Hahn,
Peter F Peterson
Abstract:
We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This mak…
▽ More
We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This makes it a scalability challenge, as the number of experiments stored increases in a single ensemble file. The present work follows up on our previous efforts on data management algorithms, to address identified input output (I/O) bottlenecks in Mantid, an open-source data analysis framework used across several neutron science facilities around the world. We reuse an in-memory binary-tree metadata index that resembles data access patterns, to provide a scalable search and extraction mechanism. In addition, several memory operations are refactored and optimized for the current common use cases, ranging most frequently from 10 to 180, and up to 360 separate measurement configurations. Results from this work show consistent speed ups in wall-clock time on the Mantid LoadMD routine, ranging from 19\% to 23\% on average, on ORNL production computing systems. The latter depends on the complexity of the targeted instrument-specific data and the system I/O and compute variability for the shared computational resources available to users of ORNL's Spallation Neutron Source (SNS) and the High Flux Isotope Reactor (HFIR) instruments. Nevertheless, we continue to highlight the need for more research to address reduction challenges as experimental data volumes, user time and processing costs increase.
△ Less
Submitted 30 November, 2021;
originally announced December 2021.
-
Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL
Authors:
William F Godoy,
Peter F Peterson,
Steven E Hahn,
Jay J Billings
Abstract:
Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2\,TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis…
▽ More
Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2\,TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11\% to 30\% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.
△ Less
Submitted 5 January, 2021;
originally announced January 2021.
-
User software for the next generation
Authors:
T. G. Worlton,
A. Chatterjee,
J. P. Hammonds,
P. F. Peterson,
D. J. Mikkelson,
R. L. Mikkelson
Abstract:
New generations of neutron scattering sources and instrumentation are providing challenges in data handling for user software. Time-of-Flight instruments used at pulsed sources typically produce hundreds or thousands of channels of data for each detector segment. New instruments are being designed with thousands to hundreds of thousands of detector segments. High intensity neutron sources make p…
▽ More
New generations of neutron scattering sources and instrumentation are providing challenges in data handling for user software. Time-of-Flight instruments used at pulsed sources typically produce hundreds or thousands of channels of data for each detector segment. New instruments are being designed with thousands to hundreds of thousands of detector segments. High intensity neutron sources make possible parametric studies and texture studies which further increase data handling requirements. The Integrated Spectral Analysis Workbench (ISAW) software developed at Argonne handles large numbers of spectra simultaneously while providing operations to reduce, sort, combine and export the data. It includes viewers to inspect the data in detail in real time. ISAW uses existing software components and packages where feasible and takes advantage of the excellent support for user interface design and network communication in Java. The included scripting language simplifies repetitive operations for analyzing many files related to a given experiment. Recent additions to ISAW include a contour view, a time-slice table view, routines for finding and fitting peaks in data, and support for data from other facilities using the NeXus format. In this paper, I give an overview of features and planned improvements of ISAW. Details of some of the improvements are covered in other presentations at this conference.
△ Less
Submitted 18 October, 2002;
originally announced October 2002.
