Data management and execution systems for the Rubin Observatory Science Pipelines
Authors:
Nate B. Lust,
Tim Jenness,
James F. Bosch,
Andrei Salnikov,
Nathan M. Pease,
Michelle Gower,
Mikolaj Kowalik,
Gregory P. Dubois-Felsmann,
Fritz Mueller,
Pim Schellart
Abstract:
We present the Rubin Observatory system for data storage/retrieval and pipelined code execution. The layer for data storage and retrieval is named the Butler. It consists of a relational database, known as the registry, to keep track of metadata and relations, and a system to manage where the data is located, named the datastore. Together these systems create an abstraction layer that science algo…
▽ More
We present the Rubin Observatory system for data storage/retrieval and pipelined code execution. The layer for data storage and retrieval is named the Butler. It consists of a relational database, known as the registry, to keep track of metadata and relations, and a system to manage where the data is located, named the datastore. Together these systems create an abstraction layer that science algorithms can be written against. This abstraction layer manages the complexities of the large data volumes expected and allows algorithms to be written independently, yet be tied together automatically into a coherent processing pipeline. This system consists of tools which execute these pipelines by transforming them into execution graphs which contain concrete data stored in the Butler. The pipeline infrastructure is designed to be scalable in nature, allowing execution on environments ranging from a laptop all the way up to multi-facility data centers. This presentation will focus on the data management aspects as well as an overview on the creation of pipelines and the corresponding execution graphs.
△ Less
Submitted 6 March, 2023;
originally announced March 2023.
The Vera C. Rubin Observatory Data Butler and Pipeline Execution System
Authors:
Tim Jenness,
James F. Bosch,
Nate B. Lust,
Nathan M. Pease,
Michelle Gower,
Mikolaj Kowalik,
Gregory P. Dubois-Felsmann,
Fritz Mueller,
Pim Schellart
Abstract:
The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a lapto…
▽ More
The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a laptop using a local file system. The Butler and pipeline system are now in daily use during Rubin construction and early operations.
△ Less
Submitted 29 June, 2022;
originally announced June 2022.
LSST: from Science Drivers to Reference Design and Anticipated Data Products
Authors:
Željko Ivezić,
Steven M. Kahn,
J. Anthony Tyson,
Bob Abel,
Emily Acosta,
Robyn Allsman,
David Alonso,
Yusra AlSayyad,
Scott F. Anderson,
John Andrew,
James Roger P. Angel,
George Z. Angeli,
Reza Ansari,
Pierre Antilogus,
Constanza Araujo,
Robert Armstrong,
Kirk T. Arndt,
Pierre Astier,
Éric Aubourg,
Nicole Auza,
Tim S. Axelrod,
Deborah J. Bard,
Jeff D. Barr,
Aurelian Barrau,
James G. Bartlett
, et al. (288 additional authors not shown)
Abstract:
(Abridged) We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). A vast array of science will be enabled by a single wide-deep-fast sky survey, and LSST will have unique survey capability in the faint time domain. The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the…
▽ More
(Abridged) We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). A vast array of science will be enabled by a single wide-deep-fast sky survey, and LSST will have unique survey capability in the faint time domain. The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the Solar System, exploring the transient optical sky, and map** the Milky Way. LSST will be a wide-field ground-based system sited at Cerro Pachón in northern Chile. The telescope will have an 8.4 m (6.5 m effective) primary mirror, a 9.6 deg$^2$ field of view, and a 3.2 Gigapixel camera. The standard observing sequence will consist of pairs of 15-second exposures in a given field, with two such visits in each pointing in a given night. With these repeats, the LSST system is capable of imaging about 10,000 square degrees of sky in a single filter in three nights. The typical 5$σ$ point-source depth in a single visit in $r$ will be $\sim 24.5$ (AB). The project is in the construction phase and will begin regular survey operations by 2022. The survey area will be contained within 30,000 deg$^2$ with $δ<+34.5^\circ$, and will be imaged multiple times in six bands, $ugrizy$, covering the wavelength range 320--1050 nm. About 90\% of the observing time will be devoted to a deep-wide-fast survey mode which will uniformly observe a 18,000 deg$^2$ region about 800 times (summed over all six bands) during the anticipated 10 years of operations, and yield a coadded map to $r\sim27.5$. The remaining 10\% of the observing time will be allocated to projects such as a Very Deep and Fast time domain survey. The goal is to make LSST data products, including a relational database of about 32 trillion observations of 40 billion objects, available to the public and scientists around the world.
△ Less
Submitted 23 May, 2018; v1 submitted 15 May, 2008;
originally announced May 2008.
