-
The Dark Energy Survey Data Management System: The Processing Framework
Authors:
Michelle Gower,
Joseph J. Mohr,
Darren Adams,
Y. Dora Cai,
Gregory E. Daues,
Tony Darnell,
Chow-Choong Ngeow,
Shantanu Desai,
Cristina Beldica,
Mike Freemon,
Huan Lin,
Eric H. Neilsen,
Douglas Tucker,
Emmanuel Bertin,
Luiz A. Nicolaci da Costa,
Leandro Martelli,
Ricardo L. C. Ogando,
Michael Jarvis,
Erin Sheldon
Abstract:
The Dark Energy Survey Data Management (DESDM) system will process and archive the data from the Dark Energy Survey (DES) over the five year period of operation. This paper focuses on a new adaptable processing framework developed to perform highly automated, high performance data parallel processing. The new processing framework has been used to process 45 nights of simulated DECam supernova im…
▽ More
The Dark Energy Survey Data Management (DESDM) system will process and archive the data from the Dark Energy Survey (DES) over the five year period of operation. This paper focuses on a new adaptable processing framework developed to perform highly automated, high performance data parallel processing. The new processing framework has been used to process 45 nights of simulated DECam supernova imaging data, and was extensively used in the DES Data Challenge 4, where it was used to process thousands of square degrees of simulated DES data.
△ Less
Submitted 5 May, 2009;
originally announced May 2009.
-
The Dark Energy Survey Data Management System
Authors:
Joseph J. Mohr,
Wayne Barkhouse,
Cristina Beldica,
Emmanuel Bertin,
Y. Dora Cai,
Luiz da Costa,
J. Anthony Darnell,
Gregory E. Daues,
Michael Jarvis,
Michelle Gower,
Huan Lin,
leandro Martelli,
Eric Neilsen,
Chow-Choong Ngeow,
Ricardo Ogando,
Alex Parga,
Erin Sheldon,
Douglas Tucker,
Nikolay Kuropatkin,
Chris Stoughton
Abstract:
The Dark Energy Survey collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a da…
▽ More
The Dark Energy Survey collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a data access framework. We are develo** the DESDM system for operation in the high performance computing (HPC) environments at NCSA and Fermilab. Operating the DESDM system in an HPC environment offers both speed and flexibility. We will employ it for our regular nightly processing needs, and for more compute-intensive tasks such as large scale image coaddition campaigns, extraction of weak lensing shear from the full survey dataset, and massive seasonal reprocessing of the DES data. Data products will be available to the Collaboration and later to the public through a virtual-observatory compatible web portal. Our approach leverages investments in publicly available HPC systems, greatly reducing hardware and maintenance costs to the project, which must deploy and maintain only the storage, database platforms and orchestration and web portal nodes that are specific to DESDM. In Fall 2007, we tested the current DESDM system on both simulated and real survey data. We used Teragrid to process 10 simulated DES nights (3TB of raw data), ingesting and calibrating approximately 250 million objects into the DES Archive database. We also used DESDM to process and calibrate over 50 nights of survey data acquired with the Mosaic2 camera. Comparison to truth tables in the case of the simulated data and internal crosschecks in the case of the real data indicate that astrometric and photometric data quality is excellent.
△ Less
Submitted 16 July, 2008;
originally announced July 2008.
-
LSST: from Science Drivers to Reference Design and Anticipated Data Products
Authors:
Željko Ivezić,
Steven M. Kahn,
J. Anthony Tyson,
Bob Abel,
Emily Acosta,
Robyn Allsman,
David Alonso,
Yusra AlSayyad,
Scott F. Anderson,
John Andrew,
James Roger P. Angel,
George Z. Angeli,
Reza Ansari,
Pierre Antilogus,
Constanza Araujo,
Robert Armstrong,
Kirk T. Arndt,
Pierre Astier,
Éric Aubourg,
Nicole Auza,
Tim S. Axelrod,
Deborah J. Bard,
Jeff D. Barr,
Aurelian Barrau,
James G. Bartlett
, et al. (288 additional authors not shown)
Abstract:
(Abridged) We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). A vast array of science will be enabled by a single wide-deep-fast sky survey, and LSST will have unique survey capability in the faint time domain. The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the…
▽ More
(Abridged) We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). A vast array of science will be enabled by a single wide-deep-fast sky survey, and LSST will have unique survey capability in the faint time domain. The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the Solar System, exploring the transient optical sky, and map** the Milky Way. LSST will be a wide-field ground-based system sited at Cerro Pachón in northern Chile. The telescope will have an 8.4 m (6.5 m effective) primary mirror, a 9.6 deg$^2$ field of view, and a 3.2 Gigapixel camera. The standard observing sequence will consist of pairs of 15-second exposures in a given field, with two such visits in each pointing in a given night. With these repeats, the LSST system is capable of imaging about 10,000 square degrees of sky in a single filter in three nights. The typical 5$σ$ point-source depth in a single visit in $r$ will be $\sim 24.5$ (AB). The project is in the construction phase and will begin regular survey operations by 2022. The survey area will be contained within 30,000 deg$^2$ with $δ<+34.5^\circ$, and will be imaged multiple times in six bands, $ugrizy$, covering the wavelength range 320--1050 nm. About 90\% of the observing time will be devoted to a deep-wide-fast survey mode which will uniformly observe a 18,000 deg$^2$ region about 800 times (summed over all six bands) during the anticipated 10 years of operations, and yield a coadded map to $r\sim27.5$. The remaining 10\% of the observing time will be allocated to projects such as a Very Deep and Fast time domain survey. The goal is to make LSST data products, including a relational database of about 32 trillion observations of 40 billion objects, available to the public and scientists around the world.
△ Less
Submitted 23 May, 2018; v1 submitted 15 May, 2008;
originally announced May 2008.
-
Photometric Calibration of the DES
Authors:
Douglas L. Tucker,
James T. Annis,
Huan Lin,
Stephen Kent,
Chris Stoughton,
John Peoples,
Sahar Allam,
Joseph J. Mohr,
Wayne A. Barkhouse,
Choong Ngeow,
Tanweer Alam,
Cristina Beldica,
Dora Cai,
Greg Daues,
Ray Plante,
Chris Miller,
Chris Smith,
Nicholas B. Suntzeff
Abstract:
The Dark Energy Survey (DES) is a 5000 sq deg griz imaging survey to be conducted using a proposed 3 sq deg (2.2deg-diameter) wide-field mosaic camera on the CTIO Blanco 4m telescope. The primary scientific goal of the DES is to constrain dark energy cosmological parameters via four complementary methods: galaxy cluster counting, weak lensing, galaxy angular correlations, and Type Ia supernovae,…
▽ More
The Dark Energy Survey (DES) is a 5000 sq deg griz imaging survey to be conducted using a proposed 3 sq deg (2.2deg-diameter) wide-field mosaic camera on the CTIO Blanco 4m telescope. The primary scientific goal of the DES is to constrain dark energy cosmological parameters via four complementary methods: galaxy cluster counting, weak lensing, galaxy angular correlations, and Type Ia supernovae, supported by precision photometric redshifts. Here we present the photometric calibration plans for the DES, including a discussion of standard stars and field-to-field calibrations.
△ Less
Submitted 3 November, 2006;
originally announced November 2006.
-
Cyber-infrastructure to Support Science and Data Management for the Dark Energy Survey
Authors:
C. Ngeow,
J. J. Mohr,
T. Alam,
W. A. Barkhouse,
C. Beldica,
D. Cai,
G. Daues,
R. Plante,
J. Annis,
H. Lin,
D. Tucker,
R. C. Smith
Abstract:
The Dark Energy Survey (DES; operations 2009-2015) will address the nature of dark energy using four independent and complementary techniques: (1) a galaxy cluster survey over 4000 deg2 in collaboration with the South Pole Telescope Sunyaev-Zel'dovich effect map** experiment, (2) a cosmic shear measurement over 5000 deg2, (3) a galaxy angular clustering measurement within redshift shells to re…
▽ More
The Dark Energy Survey (DES; operations 2009-2015) will address the nature of dark energy using four independent and complementary techniques: (1) a galaxy cluster survey over 4000 deg2 in collaboration with the South Pole Telescope Sunyaev-Zel'dovich effect map** experiment, (2) a cosmic shear measurement over 5000 deg2, (3) a galaxy angular clustering measurement within redshift shells to redshift=1.35, and (4) distance measurements to 1900 supernovae Ia. The DES will produce 200 TB of raw data in four bands, These data will be processed into science ready images and catalogs and co-added into deeper, higher quality images and catalogs. In total, the DES dataset will exceed 1 PB, including a 100 TB catalog database that will serve as a key science analysis tool for the astronomy/cosmology community. The data rate, volume, and duration of the survey require a new type of data management (DM) system that (1) offers a high degree of automation and robustness and (2) leverages the existing high performance computing infrastructure to meet the project's DM targets. The DES DM system consists of (1) a grid-enabled, flexible and scalable middleware developed at NCSA for the broader scientific community, (2) astronomy modules that build upon community software, and (3) a DES archive to support automated processing and to serve DES catalogs and images to the collaboration and the public. In the recent DES Data Challenge 1 we deployed and tested the first version of the DES DM system, successfully reducing 700 GB of raw simulated images into 5 TB of reduced data products and cataloguing 50 million objects with calibrated astrometry and photometry.
△ Less
Submitted 11 August, 2006;
originally announced August 2006.
