-
Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities
Authors:
William F Godoy,
Andrei T Savici,
Steven E Hahn,
Peter F Peterson
Abstract:
We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This mak…
▽ More
We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This makes it a scalability challenge, as the number of experiments stored increases in a single ensemble file. The present work follows up on our previous efforts on data management algorithms, to address identified input output (I/O) bottlenecks in Mantid, an open-source data analysis framework used across several neutron science facilities around the world. We reuse an in-memory binary-tree metadata index that resembles data access patterns, to provide a scalable search and extraction mechanism. In addition, several memory operations are refactored and optimized for the current common use cases, ranging most frequently from 10 to 180, and up to 360 separate measurement configurations. Results from this work show consistent speed ups in wall-clock time on the Mantid LoadMD routine, ranging from 19\% to 23\% on average, on ORNL production computing systems. The latter depends on the complexity of the targeted instrument-specific data and the system I/O and compute variability for the shared computational resources available to users of ORNL's Spallation Neutron Source (SNS) and the High Flux Isotope Reactor (HFIR) instruments. Nevertheless, we continue to highlight the need for more research to address reduction challenges as experimental data volumes, user time and processing costs increase.
△ Less
Submitted 30 November, 2021;
originally announced December 2021.
-
Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL
Authors:
William F Godoy,
Peter F Peterson,
Steven E Hahn,
Jay J Billings
Abstract:
Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2\,TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis…
▽ More
Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2\,TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11\% to 30\% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.
△ Less
Submitted 5 January, 2021;
originally announced January 2021.
-
Mantid - Data Analysis and Visualization Package for Neutron Scattering and $μSR$ Experiments
Authors:
O. Arnold,
J. C. Bilheux,
J. M. Borreguero,
A. Buts,
S. I. Campbell,
L. Chapon,
M. Doucet,
N. Draper,
R. Ferraz Leal,
M. A. Gigg,
V. E. Lynch,
A. Markvardsen,
D. J. Mikkelson,
R. L. Mikkelson,
R. Miller,
K. Palmen,
P. Parker,
G. Passos,
T. G. Perring,
P. F. Peterson,
S. Ren,
M. A. Reuter,
A. T. Savici,
J. W. Taylor,
R. J. Taylor
, et al. (3 additional authors not shown)
Abstract:
The Mantid framework is a software solution developed for the analysis and visualization of neutron scattering and muon spin measurements. The framework is jointly developed by software engineers and scientists at the ISIS Neutron and Muon Facility and the Oak Ridge National Laboratory. The objectives, functionality and novel design aspects of Mantid are described.
The Mantid framework is a software solution developed for the analysis and visualization of neutron scattering and muon spin measurements. The framework is jointly developed by software engineers and scientists at the ISIS Neutron and Muon Facility and the Oak Ridge National Laboratory. The objectives, functionality and novel design aspects of Mantid are described.
△ Less
Submitted 22 July, 2014;
originally announced July 2014.
-
Coordinated, Interactive Data Visualization for Neutron Scattering Data
Authors:
D. J. Mikkelson,
R. L. Mikkelson,
T. G. Worlton,
A. Chatterjee,
J. P. Hammonds,
P. F. Peterson,
A. J. Schultz
Abstract:
The overall design of the Integrated Spectral Analysis Workbench (ISAW), being developed at Argonne, provides for an extensible, highly interactive, collaborating set of viewers for neutron scattering data. Large arbitrary collections of spectra from multiple detectors can be viewed as an image, a scrolled list of individual graphs, or using a 3D representation of the instrument showing the dete…
▽ More
The overall design of the Integrated Spectral Analysis Workbench (ISAW), being developed at Argonne, provides for an extensible, highly interactive, collaborating set of viewers for neutron scattering data. Large arbitrary collections of spectra from multiple detectors can be viewed as an image, a scrolled list of individual graphs, or using a 3D representation of the instrument showing the detector positions. Data from an area detector can be displayed using a contour or intensity map as well as an interactive table. Selected spectra can be displayed in tables or on a conventional graph. A unique characteristic of these viewers is their interactivity and coordination. The position "pointed at" by the user in one viewer is sent to other viewers of the same DataSet so they can track the position and display relevant information. Specialized viewers for single crystal neutron diffractometers are being developed. A "proof-of-concept" viewer that directly displays the 3D reciprocal lattice from a complete series of runs on a single crystal diffractometer has been implemented.
△ Less
Submitted 20 October, 2002;
originally announced October 2002.
-
Developments for the IPNS Data Acquisition System Upgrade
Authors:
J. P. Hammonds,
R. R. Porter,
A. Chatterjee,
T. G. Worlton,
P. F. Peterson,
J. W. Weizeorick,
P. M. Delurgio,
I. Naday,
D. J. Mikkelson,
R. L. Mikkelson
Abstract:
The Intense Pulsed Neutron Source has been an operating user facility for more than 20 years. Development of an upgrade for the data acquisition system has been in progress for some time now. Now that the initial installation on the test bed instrument (HRMECS) is completed new features and possibilities are starting to come forward. Run setup is moving from large monolithic programs to more fle…
▽ More
The Intense Pulsed Neutron Source has been an operating user facility for more than 20 years. Development of an upgrade for the data acquisition system has been in progress for some time now. Now that the initial installation on the test bed instrument (HRMECS) is completed new features and possibilities are starting to come forward. Run setup is moving from large monolithic programs to more flexible scripting. EPICS based controls allow flexible and extendable instrument configurations. Hardware is also being made more flexible. This flexibility allows implementation of new methods of binning data, that better match instrument resolution. This allows better use of memory and disk space. Increased flexibiliy has also allowed use of the new time-of-flight modules in applications past their original intent.
△ Less
Submitted 18 October, 2002; v1 submitted 18 October, 2002;
originally announced October 2002.
-
User software for the next generation
Authors:
T. G. Worlton,
A. Chatterjee,
J. P. Hammonds,
P. F. Peterson,
D. J. Mikkelson,
R. L. Mikkelson
Abstract:
New generations of neutron scattering sources and instrumentation are providing challenges in data handling for user software. Time-of-Flight instruments used at pulsed sources typically produce hundreds or thousands of channels of data for each detector segment. New instruments are being designed with thousands to hundreds of thousands of detector segments. High intensity neutron sources make p…
▽ More
New generations of neutron scattering sources and instrumentation are providing challenges in data handling for user software. Time-of-Flight instruments used at pulsed sources typically produce hundreds or thousands of channels of data for each detector segment. New instruments are being designed with thousands to hundreds of thousands of detector segments. High intensity neutron sources make possible parametric studies and texture studies which further increase data handling requirements. The Integrated Spectral Analysis Workbench (ISAW) software developed at Argonne handles large numbers of spectra simultaneously while providing operations to reduce, sort, combine and export the data. It includes viewers to inspect the data in detail in real time. ISAW uses existing software components and packages where feasible and takes advantage of the excellent support for user interface design and network communication in Java. The included scripting language simplifies repetitive operations for analyzing many files related to a given experiment. Recent additions to ISAW include a contour view, a time-slice table view, routines for finding and fitting peaks in data, and support for data from other facilities using the NeXus format. In this paper, I give an overview of features and planned improvements of ISAW. Details of some of the improvements are covered in other presentations at this conference.
△ Less
Submitted 18 October, 2002;
originally announced October 2002.
-
Little helpers for your experiment - NOBUGS2002/013
Authors:
Th. Proffen,
P. F. Peterson
Abstract:
The World Wide Web (WWW) is a wonderful tool to provide users with documentation and tools that make the preparation and running of an experiment a little easier. We present a set of tools that allow one to estimate the activation of a sample in the neutron beam, or to calculate the absorption of a sample. Ever tried to figure out the time for the next sample change - here our scheduler can help…
▽ More
The World Wide Web (WWW) is a wonderful tool to provide users with documentation and tools that make the preparation and running of an experiment a little easier. We present a set of tools that allow one to estimate the activation of a sample in the neutron beam, or to calculate the absorption of a sample. Ever tried to figure out the time for the next sample change - here our scheduler can help. In addition there is a simple database that can be used to keep track of all the data files collected at various facilities. We present these little tools, but also demonstrate how easy it is to create new WWW based tools that make the life of users easier.
△ Less
Submitted 21 October, 2002;
originally announced October 2002.
-
Analyzing Single Crystal Time-of-Flight Neutron Data
Authors:
P. F. Peterson,
D. J. Mikkelson,
R. L. Mikkelson,
A. J. Schultz,
J. P. Hammonds,
A. Chatterjee,
T. G. Worlton
Abstract:
Software for time-of-flight single crystal diffractometer (SCD) data visualization and analysis was originally written to be run in batch mode on VMS-VAX systems. Modern computers and software tools allow a new level of visualization and user interaction. The Integrated Spectral Analysis Workbench (ISAW) is being extended and customized for SCD measurements. To this end, new viewers and operator…
▽ More
Software for time-of-flight single crystal diffractometer (SCD) data visualization and analysis was originally written to be run in batch mode on VMS-VAX systems. Modern computers and software tools allow a new level of visualization and user interaction. The Integrated Spectral Analysis Workbench (ISAW) is being extended and customized for SCD measurements. To this end, new viewers and operators have been added to ISAW. One of the new operators allows running a program written in another language, such as C or FORTRAN, to integrate tested software into the object-oriented package. This provides a method for preserving earlier software development while making use of modern tools.
△ Less
Submitted 19 October, 2002;
originally announced October 2002.
-
Sharing data between facilities: using the NeXus time-of-flight powder diffractometer file format
Authors:
P. F. Peterson,
Th. Proffen,
R. L. Mikkelson,
T. Kozlowski,
D. J. Mikkelson,
G. Cooper,
T. G. Worlton
Abstract:
NeXus is an international standard data format intended to reduce the need for redundant software development efforts in the neutron and x-ray scattering communities. As the NeXus standard matures it is starting to be used at laboratories for storing raw data. The Manuel Lujan Jr. Neutron Scattering Center (MLNSC) at Los Alamos National Laboratory and the Intense Pulsed Neutron Source (IPNS) at…
▽ More
NeXus is an international standard data format intended to reduce the need for redundant software development efforts in the neutron and x-ray scattering communities. As the NeXus standard matures it is starting to be used at laboratories for storing raw data. The Manuel Lujan Jr. Neutron Scattering Center (MLNSC) at Los Alamos National Laboratory and the Intense Pulsed Neutron Source (IPNS) at Argonne National Laboratory have been working with NeXus in an effort to share data and software. MLNSC is now writing files compliant with NeXus and the Integrated Spectral Analysis Workbench (ISAW) software from IPNS is being used with this data. Problems can arise if the standard is interpreted in different ways and information that belongs in the file is not accounted for in the standard. This paper will discuss an inter-laboratory collaboration in relation to a maturing data standard.
△ Less
Submitted 18 October, 2002;
originally announced October 2002.
-
A New face on old code
Authors:
P. F. Peterson,
Th. Proffen
Abstract:
In science we often use established code that was developed many years ago. Frequently the documentation is lost and many of us know the feeling of putting a number in the wrong column and trying to understand why the program is not working. However, these old codes are usually very useful, well tested and established. Rather than writing new code from scratch, which must first be tested against…
▽ More
In science we often use established code that was developed many years ago. Frequently the documentation is lost and many of us know the feeling of putting a number in the wrong column and trying to understand why the program is not working. However, these old codes are usually very useful, well tested and established. Rather than writing new code from scratch, which must first be tested against the established standard, it is desirable to put a new front end on the standard, commonly referred to as 'wrap**'. This paper will discuss wrap** of FORTRAN code using a selection of other languages to provide simpler user interface.
△ Less
Submitted 16 October, 2002;
originally announced October 2002.
-
Local Atomic Strain in ZnSe(1-x)Te(x) from High Real Space Resolution Neutron Pair Distribution Function Measurements
Authors:
P. F. Peterson,
Th. Proffen,
I. -K. Jeong,
S. J. L. Billinge,
K. -S. Choi,
M. G. Kanatzidis,
P. G. Radaelli
Abstract:
High real-space resolution atomic pair distribution functions (PDFs) have been obtained from ZnSe(1-x)Te(x) using neutron powder diffraction. Distinct Zn-Se and Zn-Te nearest neighbor (nn) bonds, differing in length by delta_r= 0.14Angstroms, are resolved in the measured PDF allowing the evolution with composition of the individual bond-lengths to be studied. The local bond-lengths change much m…
▽ More
High real-space resolution atomic pair distribution functions (PDFs) have been obtained from ZnSe(1-x)Te(x) using neutron powder diffraction. Distinct Zn-Se and Zn-Te nearest neighbor (nn) bonds, differing in length by delta_r= 0.14Angstroms, are resolved in the measured PDF allowing the evolution with composition of the individual bond-lengths to be studied. The local bond-lengths change much more slowly with do** than the average bond-length obtained crystallographically. The nn bond-length distributions are constant with do** but higher-neighbor pair distributions broaden significantly indicating that most of the strain from the alloying is accommodated by bond-bending forces in the alloy. The PDFs of alloys across the whole do** range are well fit using a model based on the Kirkwood potential. The resulting PDFs give excellent agreement with the measured PDFs over the entire alloy range with no adjustable parameters.
△ Less
Submitted 29 March, 2001; v1 submitted 22 September, 2000;
originally announced September 2000.
