-
The LSST Data Management System
Authors:
Mario Jurić,
Jeffrey Kantor,
K-T Lim,
Robert H. Lupton,
Gregory Dubois-Felsmann,
Tim Jenness,
Tim S. Axelrod,
Jovan Aleksić,
Roberta A. Allsman,
Yusra AlSayyad,
Jason Alt,
Robert Armstrong,
Jim Basney,
Andrew C. Becker,
Jacek Becla,
Steven J. Bickerton,
Rahul Biswas,
James Bosch,
Dominique Boutigny,
Matias Carrasco Kind,
David R. Ciardi,
Andrew J. Connolly,
Scott F. Daniel,
Gregory E. Daues,
Frossie Economou
, et al. (40 additional authors not shown)
Abstract:
The Large Synoptic Survey Telescope (LSST) is a large-aperture, wide-field, ground-based survey system that will image the sky in six optical bands from 320 to 1050 nm, uniformly covering approximately $18,000$deg$^2$ of the sky over 800 times. The LSST is currently under construction on Cerro Pachón in Chile, and expected to enter operations in 2022. Once operational, the LSST will explore a wide…
▽ More
The Large Synoptic Survey Telescope (LSST) is a large-aperture, wide-field, ground-based survey system that will image the sky in six optical bands from 320 to 1050 nm, uniformly covering approximately $18,000$deg$^2$ of the sky over 800 times. The LSST is currently under construction on Cerro Pachón in Chile, and expected to enter operations in 2022. Once operational, the LSST will explore a wide range of astrophysical questions, from discovering "killer" asteroids to examining the nature of Dark Energy.
The LSST will generate on average 15 TB of data per night, and will require a comprehensive Data Management system to reduce the raw data to scientifically useful catalogs and images with minimum human intervention. These reductions will result in a real-time alert stream, and eleven data releases over the 10-year duration of LSST operations. To enable this processing, the LSST project is develo** a new, general-purpose, high-performance, scalable, well documented, open source data processing software stack for O/IR surveys. Prototypes of this stack are already capable of processing data from existing cameras (e.g., SDSS, DECam, MegaCam), and form the basis of the Hyper-Suprime Cam (HSC) Survey data reduction pipeline.
△ Less
Submitted 24 December, 2015;
originally announced December 2015.
-
The Dark Energy Survey Data Processing and Calibration System
Authors:
Joseph J. Mohr,
Robert Armstrong,
Emmanuel Bertin,
Gregory E. Daues,
Shantanu Desai,
Michelle Gower,
Robert Gruendl,
William Hanlon,
Nikolay Kuropatkin,
Huan Lin,
John Marriner,
Don Petravick,
Ignacio Sevilla,
Molly Swanson,
Todd Tomashek,
Douglas Tucker,
Brian Yanny,
the Dark Energy Survey Collaboration
Abstract:
The Dark Energy Survey (DES) is a 5000 deg2 grizY survey reaching characteristic photometric depths of 24th magnitude (10 sigma) and enabling accurate photometry and morphology of objects ten times fainter than in SDSS. Preparations for DES have included building a dedicated 3 deg2 CCD camera (DECam), upgrading the existing CTIO Blanco 4m telescope and develo** a new high performance computing (…
▽ More
The Dark Energy Survey (DES) is a 5000 deg2 grizY survey reaching characteristic photometric depths of 24th magnitude (10 sigma) and enabling accurate photometry and morphology of objects ten times fainter than in SDSS. Preparations for DES have included building a dedicated 3 deg2 CCD camera (DECam), upgrading the existing CTIO Blanco 4m telescope and develo** a new high performance computing (HPC) enabled data management system (DESDM).
The DESDM system will be used for processing, calibrating and serving the DES data. The total data volumes are high (~2PB), and so considerable effort has gone into designing an automated processing and quality control system. Special purpose image detrending and photometric calibration codes have been developed to meet the data quality requirements, while survey astrometric calibration, coaddition and cataloging rely on new extensions of the AstrOmatic codes which now include tools for PSF modeling, PSF homogenization, PSF corrected model fitting cataloging and joint model fitting across multiple input images.
The DESDM system has been deployed on dedicated development clusters and HPC systems in the US and Germany. An extensive program of testing with small rapid turn-around and larger campaign simulated datasets has been carried out. The system has also been tested on large real datasets, including Blanco Cosmology Survey data from the Mosaic2 camera. In Fall 2012 the DESDM system will be used for DECam commissioning, and, thereafter, the system will go into full science operations.
△ Less
Submitted 13 July, 2012;
originally announced July 2012.
-
The Dark Energy Survey Data Management System: The Processing Framework
Authors:
Michelle Gower,
Joseph J. Mohr,
Darren Adams,
Y. Dora Cai,
Gregory E. Daues,
Tony Darnell,
Chow-Choong Ngeow,
Shantanu Desai,
Cristina Beldica,
Mike Freemon,
Huan Lin,
Eric H. Neilsen,
Douglas Tucker,
Emmanuel Bertin,
Luiz A. Nicolaci da Costa,
Leandro Martelli,
Ricardo L. C. Ogando,
Michael Jarvis,
Erin Sheldon
Abstract:
The Dark Energy Survey Data Management (DESDM) system will process and archive the data from the Dark Energy Survey (DES) over the five year period of operation. This paper focuses on a new adaptable processing framework developed to perform highly automated, high performance data parallel processing. The new processing framework has been used to process 45 nights of simulated DECam supernova im…
▽ More
The Dark Energy Survey Data Management (DESDM) system will process and archive the data from the Dark Energy Survey (DES) over the five year period of operation. This paper focuses on a new adaptable processing framework developed to perform highly automated, high performance data parallel processing. The new processing framework has been used to process 45 nights of simulated DECam supernova imaging data, and was extensively used in the DES Data Challenge 4, where it was used to process thousands of square degrees of simulated DES data.
△ Less
Submitted 5 May, 2009;
originally announced May 2009.
-
The Dark Energy Survey Data Management System
Authors:
Joseph J. Mohr,
Wayne Barkhouse,
Cristina Beldica,
Emmanuel Bertin,
Y. Dora Cai,
Luiz da Costa,
J. Anthony Darnell,
Gregory E. Daues,
Michael Jarvis,
Michelle Gower,
Huan Lin,
leandro Martelli,
Eric Neilsen,
Chow-Choong Ngeow,
Ricardo Ogando,
Alex Parga,
Erin Sheldon,
Douglas Tucker,
Nikolay Kuropatkin,
Chris Stoughton
Abstract:
The Dark Energy Survey collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a da…
▽ More
The Dark Energy Survey collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a data access framework. We are develo** the DESDM system for operation in the high performance computing (HPC) environments at NCSA and Fermilab. Operating the DESDM system in an HPC environment offers both speed and flexibility. We will employ it for our regular nightly processing needs, and for more compute-intensive tasks such as large scale image coaddition campaigns, extraction of weak lensing shear from the full survey dataset, and massive seasonal reprocessing of the DES data. Data products will be available to the Collaboration and later to the public through a virtual-observatory compatible web portal. Our approach leverages investments in publicly available HPC systems, greatly reducing hardware and maintenance costs to the project, which must deploy and maintain only the storage, database platforms and orchestration and web portal nodes that are specific to DESDM. In Fall 2007, we tested the current DESDM system on both simulated and real survey data. We used Teragrid to process 10 simulated DES nights (3TB of raw data), ingesting and calibrating approximately 250 million objects into the DES Archive database. We also used DESDM to process and calibrate over 50 nights of survey data acquired with the Mosaic2 camera. Comparison to truth tables in the case of the simulated data and internal crosschecks in the case of the real data indicate that astrometric and photometric data quality is excellent.
△ Less
Submitted 16 July, 2008;
originally announced July 2008.
