-
Data management and execution systems for the Rubin Observatory Science Pipelines
Authors:
Nate B. Lust,
Tim Jenness,
James F. Bosch,
Andrei Salnikov,
Nathan M. Pease,
Michelle Gower,
Mikolaj Kowalik,
Gregory P. Dubois-Felsmann,
Fritz Mueller,
Pim Schellart
Abstract:
We present the Rubin Observatory system for data storage/retrieval and pipelined code execution. The layer for data storage and retrieval is named the Butler. It consists of a relational database, known as the registry, to keep track of metadata and relations, and a system to manage where the data is located, named the datastore. Together these systems create an abstraction layer that science algo…
▽ More
We present the Rubin Observatory system for data storage/retrieval and pipelined code execution. The layer for data storage and retrieval is named the Butler. It consists of a relational database, known as the registry, to keep track of metadata and relations, and a system to manage where the data is located, named the datastore. Together these systems create an abstraction layer that science algorithms can be written against. This abstraction layer manages the complexities of the large data volumes expected and allows algorithms to be written independently, yet be tied together automatically into a coherent processing pipeline. This system consists of tools which execute these pipelines by transforming them into execution graphs which contain concrete data stored in the Butler. The pipeline infrastructure is designed to be scalable in nature, allowing execution on environments ranging from a laptop all the way up to multi-facility data centers. This presentation will focus on the data management aspects as well as an overview on the creation of pipelines and the corresponding execution graphs.
△ Less
Submitted 6 March, 2023;
originally announced March 2023.
-
A Gateway to Astronomical Image Processing: Vera C. RubinObservatory LSST Science Pipelines on AWS
Authors:
Dino Bektesevic,
Hsin-Fang Chiang,
Kian-Tat Lim,
Todd L. Miller,
Greg Thain,
Tim Jenness,
James Bosch,
Andrei Salnikov,
Andrew Connolly
Abstract:
The Legacy Survey of Space and Time, operated by the Vera C. Rubin Observatory, is a 10-year astronomical survey due to start operations in 2022 that will image half the sky every three nights. LSST will produce ~20TB of raw data per night which will be calibrated and analyzed in almost real time. Given the volume of LSST data, the traditional subset-download-process paradigm of data reprocessing…
▽ More
The Legacy Survey of Space and Time, operated by the Vera C. Rubin Observatory, is a 10-year astronomical survey due to start operations in 2022 that will image half the sky every three nights. LSST will produce ~20TB of raw data per night which will be calibrated and analyzed in almost real time. Given the volume of LSST data, the traditional subset-download-process paradigm of data reprocessing faces significant challenges. We describe here, the first steps towards a gateway for astronomical science that would enable astronomers to analyze images and catalogs at scale. In this first step we focus on executing the Rubin LSST Science Pipelines, a collection of image and catalog processing algorithms, on Amazon Web Services (AWS). We describe our initial impressions on the performance, scalability and cost of deploying such a system in the cloud.
△ Less
Submitted 11 November, 2020;
originally announced November 2020.
-
Abstracting the storage and retrieval of image data at the LSST
Authors:
Tim Jenness,
James F. Bosch,
Pim Schellart,
Kian-Ta Lim,
Andrei Salnikov,
Michelle Gower
Abstract:
Writing generic data processing pipelines requires that the algorithmic code does not ever have to know about data formats of files, or the locations of those files. At LSST we have a software system known as "the Data Butler," that abstracts these details from the software developer. Scientists can specify the dataset they want in terms they understand, such as filter, observation identifier, dat…
▽ More
Writing generic data processing pipelines requires that the algorithmic code does not ever have to know about data formats of files, or the locations of those files. At LSST we have a software system known as "the Data Butler," that abstracts these details from the software developer. Scientists can specify the dataset they want in terms they understand, such as filter, observation identifier, date of observation, and instrument name, and the Butler translates that to one or more files which are read and returned to them as a single Python object. Conversely, once they have created a new dataset they can give it back to the Butler, with a label describing its new status, and the Butler can write it in whatever format it has been configured to use. All configuration is in YAML and supports standard defaults whilst allowing overrides.
△ Less
Submitted 19 December, 2018;
originally announced December 2018.
