-
The Future of Astronomical Data Infrastructure: Meeting Report
Authors:
Michael R. Blanton,
Janet D. Evans,
Dara Norman,
William O'Mullane,
Adrian Price-Whelan,
Luca Rizzi,
Alberto Accomazzi,
Megan Ansdell,
Stephen Bailey,
Paul Barrett,
Steven Berukoff,
Adam Bolton,
Julian Borrill,
Kelle Cruz,
Julianne Dalcanton,
Vandana Desai,
Gregory P. Dubois-Felsmann,
Frossie Economou,
Henry Ferguson,
Bryan Field,
Dan Foreman-Mackey,
Jaime Forero-Romero,
Niall Gaffney,
Kim Gillies,
Matthew J. Graham
, et al. (47 additional authors not shown)
Abstract:
The astronomical community is grappling with the increasing volume and complexity of data produced by modern telescopes, due to difficulties in reducing, accessing, analyzing, and combining archives of data. To address this challenge, we propose the establishment of a coordinating body, an "entity," with the specific mission of enhancing the interoperability, archiving, distribution, and productio…
▽ More
The astronomical community is grappling with the increasing volume and complexity of data produced by modern telescopes, due to difficulties in reducing, accessing, analyzing, and combining archives of data. To address this challenge, we propose the establishment of a coordinating body, an "entity," with the specific mission of enhancing the interoperability, archiving, distribution, and production of both astronomical data and software. This report is the culmination of a workshop held in February 2023 on the Future of Astronomical Data Infrastructure. Attended by 70 scientists and software professionals from ground-based and space-based missions and archives spanning the entire spectrum of astronomical research, the group deliberated on the prevailing state of software and data infrastructure in astronomy, identified pressing issues, and explored potential solutions. In this report, we describe the ecosystem of astronomical data, its existing flaws, and the many gaps, duplication, inconsistencies, barriers to access, drags on productivity, missed opportunities, and risks to the long-term integrity of essential data sets. We also highlight the successes and failures in a set of deep dives into several different illustrative components of the ecosystem, included as an appendix.
△ Less
Submitted 7 November, 2023;
originally announced November 2023.
-
Target of Opportunity Observations of Gravitational Wave Events with LSST
Authors:
R. Margutti,
P. Cowperthwaite,
Z. Doctor,
K. Mortensen,
C. P. Pankow,
O. Salafia,
V. A. Villar,
K. Alexander,
J. Annis,
I. Andreoni,
A. Baldeschi,
B. Balmaverde,
E. Berger,
M. G. Bernardini,
C. P. L. Berry,
F. Bianco,
P. K. Blanchard,
E. Brocato,
M. I. Carnerero,
R. Cartier,
S. B. Cenko,
R. Chornock,
L. Chomiuk,
C. M. Copperwheat,
M. W. Coughlin
, et al. (57 additional authors not shown)
Abstract:
The discovery of the electromagnetic counterparts to the binary neutron star merger GW170817 has opened the era of GW+EM multi-messenger astronomy. Exploiting this breakthrough requires increasing samples to explore the diversity of kilonova behaviour and provide more stringent constraints on the Hubble constant, and tests of fundamental physics. LSST can play a key role in this field in the 2020s…
▽ More
The discovery of the electromagnetic counterparts to the binary neutron star merger GW170817 has opened the era of GW+EM multi-messenger astronomy. Exploiting this breakthrough requires increasing samples to explore the diversity of kilonova behaviour and provide more stringent constraints on the Hubble constant, and tests of fundamental physics. LSST can play a key role in this field in the 2020s, when the gravitational wave detector network is expected to detect higher rates of merger events involving neutron stars ($\sim$10s per year) out to distances of several hundred Mpc. Here we propose comprehensive target-of-opportunity (ToOs) strategies for follow-up of gravitational-wave sources that will make LSST the premiere machine for discovery and early characterization for neutron star mergers and other gravitational-wave sources.
△ Less
Submitted 10 December, 2018;
originally announced December 2018.
-
A near-Sun Solar System Twilight Survey with LSST
Authors:
Rob Seaman,
Paul Abell,
Eric Christensen,
Michael S. P. Kelley,
Megan E. Schwamb,
Renu Malhotra,
Mario Juric,
Quanzhi Ye,
Michael Mommert,
Matthew M. Knight,
Colin Snodgrass,
Andrew S. Rivkin
Abstract:
We propose a LSST Solar System near-Sun Survey, to be implemented during twilight hours, that extends the seasonal reach of LSST to its maximum as fresh sky is uncovered at about 50 square degrees per night (1500 sq. deg. per lunation) in the morning eastern sky, and surveyable sky is lost at the same rate to the western evening sky due to the Earth's synodic motion. By establishing near-horizon f…
▽ More
We propose a LSST Solar System near-Sun Survey, to be implemented during twilight hours, that extends the seasonal reach of LSST to its maximum as fresh sky is uncovered at about 50 square degrees per night (1500 sq. deg. per lunation) in the morning eastern sky, and surveyable sky is lost at the same rate to the western evening sky due to the Earth's synodic motion. By establishing near-horizon fence post picket lines to the far west and far east we address Solar System science use cases (including Near Earth Objects, Interior Earth Objects, Potentially Hazardous Asteroids, Earth Trojans, near-Sun asteroids, sun-grazing comets, and dormant comets) as well as provide the first look and last look that LSST will have at the transient and variable objects within each survey field. This proposed near-Sun Survey will also maximize the overlap with the field of regard of the proposed NEOCam spacecraft that will be stationed at the Earth's L1 Lagrange point and survey near quadrature with the Sun. This will allow LSST to incidently follow-up NEOCam targets and vice-versa (as well as targets from missions such as Euclid), and will roughly correspond to the Earth's L4 and L5 regions.
△ Less
Submitted 2 December, 2018;
originally announced December 2018.
-
Timekee** infrastructure for the Catalina Sky Survey
Authors:
Robert L. Seaman,
Alex R. Gibbs
Abstract:
Time domain science forms an increasing fraction of astronomical programs at many facilities. Synoptic and targeted observing modes of transient, varying, and moving sources rely on precise clocks to provide the underlying time tags. Often precision is mistaken for accuracy, or the precise time signals never reach the instrumentation in the first place. We will discuss issues of deploying a stable…
▽ More
Time domain science forms an increasing fraction of astronomical programs at many facilities. Synoptic and targeted observing modes of transient, varying, and moving sources rely on precise clocks to provide the underlying time tags. Often precision is mistaken for accuracy, or the precise time signals never reach the instrumentation in the first place. We will discuss issues of deploying a stable high-precision GNSS clock on a remote mountaintop, and of conveying the resulting time signals to a computer in a way that permits hardware timestam** of the camera shutter (or equivalent) rather than the arbitrary delays encountered with non-real-time data acquisition software. Strengths and limitations of the Network Time Protocol will be reviewed. Timekee** infrastructure deployed for the Catalina Sky Survey will serve as an example.
△ Less
Submitted 3 July, 2018;
originally announced July 2018.
-
Large Synoptic Survey Telescope Solar System Science Roadmap
Authors:
Megan E. Schwamb,
R. Lynne Jones,
Steven R. Chesley,
Alan Fitzsimmons,
Wesley C. Fraser,
Matthew J. Holman,
Henry Hsieh,
Darin Ragozzine,
Cristina A. Thomas,
David E. Trilling,
Michael E. Brown,
Michele T. Bannister,
Dennis Bodewits,
Miguel de Val-Borro,
David Gerdes,
Mikael Granvik,
Michael S. P. Kelley,
Matthew M. Knight,
Robert L. Seaman,
Quan-Zhi Ye,
Leslie A. Young
Abstract:
The Large Synoptic Survey Telescope (LSST) is uniquely equipped to search for Solar System bodies due to its unprecedented combination of depth and wide field coverage. Over a ten-year period starting in 2022, LSST will generate the largest catalog of Solar System objects to date. The main goal of the LSST Solar System Science Collaboration (SSSC) is to facilitate the efforts of the planetary comm…
▽ More
The Large Synoptic Survey Telescope (LSST) is uniquely equipped to search for Solar System bodies due to its unprecedented combination of depth and wide field coverage. Over a ten-year period starting in 2022, LSST will generate the largest catalog of Solar System objects to date. The main goal of the LSST Solar System Science Collaboration (SSSC) is to facilitate the efforts of the planetary community to study the planets and small body populations residing within our Solar System using LSST data. To prepare for future survey cadence decisions and ensure that interesting and novel Solar System science is achievable with LSST, the SSSC has identified and prioritized key Solar System research areas for investigation with LSST in this roadmap. The ranked science priorities highlighted in this living document will inform LSST survey cadence decisions and aid in identifying software tools and pipelines needed to be developed by the planetary community as added value products and resources before the planned start of LSST science operations.
△ Less
Submitted 5 February, 2018;
originally announced February 2018.
-
Machine Learning-based Brokers for Real-time Classification of the LSST Alert Stream
Authors:
Gautham Narayan,
Tayeb Zaidi,
Monika D. Soraisam,
Zhe Wang,
Michelle Lochner,
Thomas Matheson,
Abhijit Saha,
Shuo Yang,
Zhenge Zhao,
John Kececioglu,
Carlos Scheidegger,
Richard T. Snodgrass,
Tim Axelrod,
Tim Jenness,
Robert S. Maier,
Stephen T. Ridgway,
Robert L. Seaman,
Eric Michael Evans,
Navdeep Singh,
Clark Taylor,
Jackson Toeniskoetter,
Eric Welch,
Songzhe Zhu
Abstract:
The unprecedented volume and rate of transient events that will be discovered by the Large Synoptic Survey Telescope (LSST) demands that the astronomical community update its followup paradigm. Alert-brokers -- automated software system to sift through, characterize, annotate and prioritize events for followup -- will be critical tools for managing alert streams in the LSST era. The Arizona-NOAO T…
▽ More
The unprecedented volume and rate of transient events that will be discovered by the Large Synoptic Survey Telescope (LSST) demands that the astronomical community update its followup paradigm. Alert-brokers -- automated software system to sift through, characterize, annotate and prioritize events for followup -- will be critical tools for managing alert streams in the LSST era. The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES) is one such broker. In this work, we develop a machine learning pipeline to characterize and classify variable and transient sources only using the available multiband optical photometry. We describe three illustrative stages of the pipeline, serving the three goals of early, intermediate and retrospective classification of alerts. The first takes the form of variable vs transient categorization, the second, a multi-class ty** of the combined variable and transient dataset, and the third, a purity-driven subty** of a transient class. While several similar algorithms have proven themselves in simulations, we validate their performance on real observations for the first time. We quantitatively evaluate our pipeline on sparse, unevenly sampled, heteroskedastic data from various existing observational campaigns, and demonstrate very competitive classification performance. We describe our progress towards adapting the pipeline developed in this work into a real-time broker working on live alert streams from time-domain surveys.
△ Less
Submitted 22 January, 2018;
originally announced January 2018.
-
ANTARES: Progress towards building a `Broker' of time-domain alerts
Authors:
Abhijit Saha,
Zhe Wang,
Thomas Matheson,
Gautham Narayan,
Richard Snodgrass,
John Kececioglu,
Carlos Scheidegger,
Tim Axelrod,
Tim Jenness,
Stephen Ridgway,
Robert Seaman,
Clark Taylor,
Jackson Toeniskoetter,
Eric Welch,
Shuo Yang,
Tayeb Zaidi
Abstract:
The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES) is a joint effort of NOAO and the Department of Computer Science at the University of Arizona to build prototype software to process alerts from time-domain surveys, especially LSST, to identify those alerts that must be followed up immediately. Value is added by annotating incoming alerts with existing information from pre…
▽ More
The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES) is a joint effort of NOAO and the Department of Computer Science at the University of Arizona to build prototype software to process alerts from time-domain surveys, especially LSST, to identify those alerts that must be followed up immediately. Value is added by annotating incoming alerts with existing information from previous surveys and compilations across the electromagnetic spectrum and from the history of past alerts. Comparison against a knowledge repository of properties and features of known or predicted kinds of variable phenomena is used for categorization. The architecture and algorithms being employed are described.
△ Less
Submitted 17 November, 2016;
originally announced November 2016.
-
The NOAO KOSMOS Data Handling System
Authors:
Rob Seaman
Abstract:
KOSMOS and COSMOS are twin high-efficiency imaging spectrographs that have been deployed as NOAO facility instruments for the Mayall 4-meter telescope on Kitt Peak in Arizona and for the Blanco telescope on Cerro Tololo in Chile, respectively. The NOAO Data Handling System (DHS) has seen aggressive use over several years at both the Blanco and Mayall telescopes with NEWFIRM (the NOAO Extremely Wid…
▽ More
KOSMOS and COSMOS are twin high-efficiency imaging spectrographs that have been deployed as NOAO facility instruments for the Mayall 4-meter telescope on Kitt Peak in Arizona and for the Blanco telescope on Cerro Tololo in Chile, respectively. The NOAO Data Handling System (DHS) has seen aggressive use over several years at both the Blanco and Mayall telescopes with NEWFIRM (the NOAO Extremely Wide-Field Infrared Imager) and the Mosaic-1.1 wide-field optical imager. Both of these instruments also rely on the Monsoon array controller and related software, and on instrument-specific versions of the NOAO Observation Control System (NOCS). NOCS, Monsoon and DHS are thus a well-tested software suite that was adopted by the KOSMOS project. This document describes the specifics of the KOSMOS implementation of DHS, in particular in support of the original two-amplifier e2v 2Kx4K CCD detectors with which the instruments were commissioned. The emphasis will be on the general layout of the DHS software components and the flow of data and metadata through the system as received from Monsoon and the NOCS. Instructions will be provided for retrieving and building the software, and for taking simulated and actual exposures.
△ Less
Submitted 27 January, 2015;
originally announced January 2015.
-
The Past, Present and Future of Astronomical Data Formats
Authors:
Jessica Mink,
Robert G. Mann,
Robert Hanisch,
Arnold Rots,
Rob Seaman,
Tim Jenness,
Brian Thomas,
William O'Mullane
Abstract:
The future of astronomy is inextricably entwined with the care and feeding of astronomical data products. Community standards such as FITS and NDF have been instrumental in the success of numerous astronomy projects. Their very success challenges us to entertain pragmatic strategies to adapt and evolve the standards to meet the aggressive data-handling requirements of facilities now being designed…
▽ More
The future of astronomy is inextricably entwined with the care and feeding of astronomical data products. Community standards such as FITS and NDF have been instrumental in the success of numerous astronomy projects. Their very success challenges us to entertain pragmatic strategies to adapt and evolve the standards to meet the aggressive data-handling requirements of facilities now being designed and built. We discuss characteristics that have made standards successful in the past, as well as desirable features for the future, and an open discussion follows.
△ Less
Submitted 4 November, 2014;
originally announced November 2014.
-
Data engineering for archive evolution
Authors:
Rob Seaman
Abstract:
From the moment astronomical observations are made the resulting data products begin to grow stale. Even if perfect binary copies are preserved through repeated timely migration to more robust storage media, data standards evolve and new tools are created that require different kinds of data or metadata. The expectations of the astronomical community change even if the data do not. We discuss data…
▽ More
From the moment astronomical observations are made the resulting data products begin to grow stale. Even if perfect binary copies are preserved through repeated timely migration to more robust storage media, data standards evolve and new tools are created that require different kinds of data or metadata. The expectations of the astronomical community change even if the data do not. We discuss data engineering to mitigate the ensuing risks with examples from a recent project to refactor seven million archival images to new standards of nomenclature, metadata, format, and compression.
△ Less
Submitted 13 October, 2014;
originally announced October 2014.
-
ANTARES: A Prototype Transient Broker System
Authors:
Abhijit Saha,
Thomas Matheson,
Richard Snodgrass,
John Kececioglu,
Gautham Narayan,
Robert Seaman,
Tim Jenness,
Tim Axelrod
Abstract:
The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES) is a joint project of the National Optical Astronomy Observatory and the Department of Computer Science at the University of Arizona. The goal is to build the software infrastructure necessary to process and filter alerts produced by time-domain surveys, with the ultimate source of such alerts being the Large Synoptic Surve…
▽ More
The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES) is a joint project of the National Optical Astronomy Observatory and the Department of Computer Science at the University of Arizona. The goal is to build the software infrastructure necessary to process and filter alerts produced by time-domain surveys, with the ultimate source of such alerts being the Large Synoptic Survey Telescope (LSST). The ANTARES broker will add value to alerts by annotating them with information from external sources such as previous surveys from across the electromagnetic spectrum. In addition, the temporal history of annotated alerts will provide further annotation for analysis. These alerts will go through a cascade of filters to select interesting candidates. For the prototype, `interesting' is defined as the rarest or most unusual alert, but future systems will accommodate multiple filtering goals. The system is designed to be flexible, allowing users to access the stream at multiple points throughout the process, and to insert custom filters where necessary. We describe the basic architecture of ANTARES and the principles that will guide development and implementation.
△ Less
Submitted 29 August, 2014;
originally announced September 2014.
-
Reengineering observatory operations for the time domain
Authors:
Robert L. Seaman,
W. Thomas Vestrand,
Frederic V. Hessman
Abstract:
Observatories are complex scientific and technical institutions serving diverse users and purposes. Their telescopes, instruments, software, and human resources engage in interwoven workflows over a broad range of timescales. These workflows have been tuned to be responsive to concepts of observatory operations that were applicable when various assets were commissioned, years or decades in the pas…
▽ More
Observatories are complex scientific and technical institutions serving diverse users and purposes. Their telescopes, instruments, software, and human resources engage in interwoven workflows over a broad range of timescales. These workflows have been tuned to be responsive to concepts of observatory operations that were applicable when various assets were commissioned, years or decades in the past. The astronomical community is entering an era of rapid change increasingly characterized by large time domain surveys, robotic telescopes and automated infrastructures, and - most significantly - of operating modes and scientific consortia that span our individual facilities, joining them into complex network entities.
Observatories must adapt and numerous initiatives are in progress that focus on redesigning individual components out of the astronomical toolkit. New instrumentation is both more capable and more complex than ever, and even simple instruments may have powerful observation scripting capabilities. Remote and queue observing modes are now widespread. Data archives are becoming ubiquitous. Virtual observatory standards and protocols and astroinformatics data-mining techniques layered on these are areas of active development. Indeed, new large-aperture ground-based telescopes may be as expensive as space missions and have similarly formal project management processes and large data management requirements.
This piecewise approach is not enough. Whatever challenges of funding or politics facing the national and international astronomical communities it will be more efficient - scientifically as well as in the usual figures of merit of cost, schedule, performance, and risks - to explicitly address the systems engineering of the astronomical community as a whole.
△ Less
Submitted 28 July, 2014;
originally announced July 2014.
-
KOSMOS and COSMOS: New facility instruments for the NOAO 4-meter telescopes
Authors:
Paul Martini,
J. Elias,
S. Points,
D. Sprayberry,
M. A. Derwent,
R. Gonzalez,
J. A. Mason,
T. P. O'Brien,
D. P. Pappalardo,
R. W. Pogge,
R. Stoll,
R. Zhelem,
P. Daly,
M. Fitzpatrick,
J. R. George,
M. Hunten,
R. Marshall,
G. Poczulp,
S. Rath,
R. Seaman,
M. Trueblood,
K. Zelaya
Abstract:
We describe the design, construction and measured performance of the Kitt Peak Ohio State Multi-Object Spectrograph (KOSMOS) for the 4-m Mayall telescope and the Cerro Tololo Ohio State Multi-Object Spectrograph (COSMOS) for the 4-m Blanco telescope. These nearly identical imaging spectrographs are modified versions of the OSMOS instrument; they provide a pair of new, high-efficiency instruments t…
▽ More
We describe the design, construction and measured performance of the Kitt Peak Ohio State Multi-Object Spectrograph (KOSMOS) for the 4-m Mayall telescope and the Cerro Tololo Ohio State Multi-Object Spectrograph (COSMOS) for the 4-m Blanco telescope. These nearly identical imaging spectrographs are modified versions of the OSMOS instrument; they provide a pair of new, high-efficiency instruments to the NOAO user community. KOSMOS and COSMOS may be used for imaging, long-slit, and multi-slit spectroscopy over a 100 square arcminute field of view with a pixel scale of 0.29 arcseconds. Each contains two VPH grisms that provide R~2500 with a one arcsecond slit and their wavelengths of peak diffraction efficiency are approximately 510nm and 750nm. Both may also be used with either a thin, blue-optimized CCD from e2v or a thick, fully depleted, red-optimized CCD from LBNL. These instruments were developed in response to the ReSTAR process. KOSMOS was commissioned in 2013B and COSMOS was commissioned in 2014A.
△ Less
Submitted 16 July, 2014;
originally announced July 2014.
-
FITS Foreign File Encapsulation Convention
Authors:
Nelson Zarate,
Rob Seaman,
Doug Tody
Abstract:
This document describes a FITS convention developed by the IRAF Group (D. Tody, R. Seaman, and N. Zarate) at the National Optical Astronomical Observatory (NOAO). This convention is implemented by the fgread/fgwrite tasks in the IRAF fitsutil package. It was first used in May 1999 to encapsulate preview PNG-format graphics files into FITS files in the NOAO High Performance Pipeline System. A FITS…
▽ More
This document describes a FITS convention developed by the IRAF Group (D. Tody, R. Seaman, and N. Zarate) at the National Optical Astronomical Observatory (NOAO). This convention is implemented by the fgread/fgwrite tasks in the IRAF fitsutil package. It was first used in May 1999 to encapsulate preview PNG-format graphics files into FITS files in the NOAO High Performance Pipeline System. A FITS extension of type 'FOREIGN' provides a mechanism for storing an arbitrary file or tree of files in FITS, allowing it to be restored to disk at a later time.
△ Less
Submitted 5 January, 2012;
originally announced January 2012.
-
FITS Checksum Proposal
Authors:
Rob Seaman,
William Pence,
Arnold Rots
Abstract:
The checksum keywords described here provide an integrity check on the information contained in FITS HDUs. (Header and Data Units are the basic components of FITS files, consisting of header keyword records followed by optional associated data records). The CHECKSUM keyword is defined to have a value that forces the 32-bit 1's complement checksum accumulated over all the 2880-byte FITS logical rec…
▽ More
The checksum keywords described here provide an integrity check on the information contained in FITS HDUs. (Header and Data Units are the basic components of FITS files, consisting of header keyword records followed by optional associated data records). The CHECKSUM keyword is defined to have a value that forces the 32-bit 1's complement checksum accumulated over all the 2880-byte FITS logical records in the HDU to equal negative 0. (Note that 1's complement arithmetic has both positive and negative zero elements). Verifying that the accumulated checksum is still equal to -0 provides a fast and fairly reliable way to determine that the HDU has not been modified by subsequent data processing operations or corrupted while copying or storing the file on physical media.
△ Less
Submitted 5 January, 2012;
originally announced January 2012.
-
A Tiled-Table Convention for Compressing FITS Binary Tables
Authors:
William Pence,
Rob Seaman,
Richard L. White
Abstract:
This document describes a convention for compressing FITS binary tables that is modeled after the FITS tiled-image compression method (White et al. 2009) that has been in use for about a decade. The input table is first optionally subdivided into tiles, each containing an equal number of rows, then every column of data within each tile is compressed and stored as a variable-length array of bytes i…
▽ More
This document describes a convention for compressing FITS binary tables that is modeled after the FITS tiled-image compression method (White et al. 2009) that has been in use for about a decade. The input table is first optionally subdivided into tiles, each containing an equal number of rows, then every column of data within each tile is compressed and stored as a variable-length array of bytes in the output FITS binary table. All the header keywords from the input table are copied to the header of the output table and remain uncompressed for efficient access. The output compressed table contains the same number and order of columns as in the input uncompressed binary table. There is one row in the output table corresponding to each tile of rows in the input table. In principle, each column of data can be compressed using a different algorithm that is optimized for the type of data within that column, however in the prototype implementation described here, the gzip algorithm is used to compress every column.
△ Less
Submitted 5 January, 2012;
originally announced January 2012.
-
Tiled Image Convention for Storing Compressed Images in FITS Binary Tables
Authors:
Richard L. White,
Perry Greenfield,
William Pence,
Doug Tody,
Rob Seaman
Abstract:
This document describes a convention for compressing n-dimensional images and storing the resulting byte stream in a variable-length column in a FITS binary table. The FITS file structure outlined here is independent of the specific data compression algorithm that is used. The implementation details for 4 widely used compression algorithms are described here, but any other compression technique co…
▽ More
This document describes a convention for compressing n-dimensional images and storing the resulting byte stream in a variable-length column in a FITS binary table. The FITS file structure outlined here is independent of the specific data compression algorithm that is used. The implementation details for 4 widely used compression algorithms are described here, but any other compression technique could also be supported by this convention. The general principle used in this convention is to first divide the n-dimensional image into a rectangular grid of subimages or 'tiles'. Each tile is then compressed as a block of data, and the resulting compressed byte stream is stored in a row of a variable length column in a FITS binary table. By dividing the image into tiles it is generally possible to extract and uncompress subsections of the image without having to uncompress the whole image.
△ Less
Submitted 5 January, 2012;
originally announced January 2012.
-
Using the VO to Study the Time Domain
Authors:
Rob Seaman,
Roy Williams,
Matthew Graham,
Tara Murphy
Abstract:
Just as the astronomical "Time Domain" is a catch-phrase for a diverse group of different science objectives involving time-varying phenomena in all astrophysical regimes from the solar system to cosmological scales, so the "Virtual Observatory" is a complex set of community-wide activities from archives to astroinformatics. This workshop touched on some aspects of adapting and develo** those se…
▽ More
Just as the astronomical "Time Domain" is a catch-phrase for a diverse group of different science objectives involving time-varying phenomena in all astrophysical regimes from the solar system to cosmological scales, so the "Virtual Observatory" is a complex set of community-wide activities from archives to astroinformatics. This workshop touched on some aspects of adapting and develo** those semantic and network technologies in order to address transient and time-domain research challenges. It discussed the VOEvent format for representing alerts and reports on celestial transient events, the SkyAlert and ATELstream facilities for distributing these alerts, and the IVOA time-series protocol and time-series tools provided by the VAO. Those tools and infrastructure are available today to address the real-world needs of astronomers.
△ Less
Submitted 3 January, 2012;
originally announced January 2012.
-
Time in the 10,000-Year Clock
Authors:
Danny Hillis,
Rob Seaman,
Steve Allen,
Jon Giorgini
Abstract:
The Long Now Foundation is building a mechanical clock that is designed to keep time for the next 10,000 years. The clock maintains its long-term accuracy by synchronizing to the Sun. The 10,000-Year Clock keeps track of five different types of time: Pendulum Time, Uncorrected Solar Time, Corrected Solar Time, Displayed Solar Time and Orrery Time. Pendulum Time is generated from the mechanical pen…
▽ More
The Long Now Foundation is building a mechanical clock that is designed to keep time for the next 10,000 years. The clock maintains its long-term accuracy by synchronizing to the Sun. The 10,000-Year Clock keeps track of five different types of time: Pendulum Time, Uncorrected Solar Time, Corrected Solar Time, Displayed Solar Time and Orrery Time. Pendulum Time is generated from the mechanical pendulum and adjusted according to the equation of time to produce Uncorrected Solar Time, which is in turn mechanically corrected by the Sun to create Corrected Solar Time. Displayed Solar Time advances each time the clock is wound, at which point it catches up with Corrected Solar Time. The clock uses Displayed Solar Time to compute various time indicators to be displayed, including the positions of the Sun, and Gregorian calendar date. Orrery Time is a better approximation of Dynamical Time, used to compute positions of the Moon, planets and stars and the phase of the Moon. This paper describes how the clock reckons time over the 10,000-year design lifetime, in particular how it reconciles the approximate Dynamical Time generated by its mechanical pendulum with the unpredictable rotation of the Earth.
△ Less
Submitted 13 December, 2011;
originally announced December 2011.
-
Fpack and Funpack User's Guide: FITS Image Compression Utilities
Authors:
William Pence,
Rob Seaman,
Rick White
Abstract:
Fpack is a utility program for optimally compressing images in the FITS (Flexible Image Transport System) data format (see http://fits.gsfc.nasa.gov). The associated funpack program restores the compressed image file back to its original state (if a lossless compression algorithm is used). (An experimental method for compressing FITS binary tables is also available; see section 7). These programs…
▽ More
Fpack is a utility program for optimally compressing images in the FITS (Flexible Image Transport System) data format (see http://fits.gsfc.nasa.gov). The associated funpack program restores the compressed image file back to its original state (if a lossless compression algorithm is used). (An experimental method for compressing FITS binary tables is also available; see section 7). These programs may be run from the host operating system command line and are analogous to the gzip and gunzip utility programs except that they are optimized for FITS format images and offer a wider choice of compression options.
△ Less
Submitted 12 December, 2011;
originally announced December 2011.
-
Systems Engineering for Civil Timekee**
Authors:
Rob Seaman
Abstract:
The future of Coordinated Universal Time has been a topic of energetic discussions for more than a dozen years. Different communities view the issue in different ways. Diametrically opposed visions exist for the range of appropriate solutions that should be entertained. Rather than an insoluble quandary, we suggest that well-known systems engineering best practices would provide a framework for re…
▽ More
The future of Coordinated Universal Time has been a topic of energetic discussions for more than a dozen years. Different communities view the issue in different ways. Diametrically opposed visions exist for the range of appropriate solutions that should be entertained. Rather than an insoluble quandary, we suggest that well-known systems engineering best practices would provide a framework for reaching consensus. This starts with the coherent collection of project requirements.
△ Less
Submitted 4 December, 2011;
originally announced December 2011.
-
An Inventory of UTC Dependencies for IRAF
Authors:
Rob Seaman
Abstract:
The Image Reduction and Analysis Facility is a scientific image processing package widely used throughout the astronomical community. IRAF has been developed and distributed by the National Optical Astronomy Observatory in Tucson, Arizona since the early 1980's. Other observatories and projects have written many dozens of layered external application packages. More than ten thousand journal articl…
▽ More
The Image Reduction and Analysis Facility is a scientific image processing package widely used throughout the astronomical community. IRAF has been developed and distributed by the National Optical Astronomy Observatory in Tucson, Arizona since the early 1980's. Other observatories and projects have written many dozens of layered external application packages. More than ten thousand journal articles acknowledge the use of IRAF and thousands of professional astronomers rely on it. As with many other classes of astronomical software, IRAF depends on Universal Time (UT) in many modules throughout its codebase. The author was the Y2K lead for IRAF in the late 1990's. A conservative underestimate of the initial inventory of UTC "hits" in IRAF (e.g., from search terms like "UT", "GMT" and "MJD") contains several times as many files as the corresponding Y2K ("millennium bug") inventory did in the 1990's. We will discuss dependencies of IRAF upon Coordinated Universal Time, and implications of these for the broader astronomical community.
△ Less
Submitted 30 November, 2011;
originally announced December 2011.
-
The Colloquium on Decoupling Civil Timekee** from Earth Rotation
Authors:
John H. Seago,
Robert L. Seaman,
Steven L. Allen
Abstract:
On October 5 and October 6, 2011, the Colloquium on the Decoupling Civil Timekee** from Earth Rotation was hosted in Exton, Pennsylvania by Analytical Graphics, Inc. (AGI). This paper highlights various technical perspectives offered through these proceedings, including expressions of concern and various recommendations offered by colloquium participants.
On October 5 and October 6, 2011, the Colloquium on the Decoupling Civil Timekee** from Earth Rotation was hosted in Exton, Pennsylvania by Analytical Graphics, Inc. (AGI). This paper highlights various technical perspectives offered through these proceedings, including expressions of concern and various recommendations offered by colloquium participants.
△ Less
Submitted 29 November, 2011;
originally announced November 2011.
-
The VAO Transient Facility
Authors:
Matthew J. Graham,
S. G. Djorgovski,
Andrew Drake,
Ashish Mahabal,
Roy Williams,
Rob Seaman
Abstract:
The time domain community wants robust and reliable tools to enable production of and subscription to community-endorsed event notification packets (VOEvent). The VAO Transient Facility (VTF) is being designed to be the premier brokering service for the community, both collecting and disseminating observations about time-critical astronomical transients but also supporting annotations and the appl…
▽ More
The time domain community wants robust and reliable tools to enable production of and subscription to community-endorsed event notification packets (VOEvent). The VAO Transient Facility (VTF) is being designed to be the premier brokering service for the community, both collecting and disseminating observations about time-critical astronomical transients but also supporting annotations and the application of intelligent machine-learning to those observations. This distinguishes two types of activity associated with the facility: core infrastructure and user services. In this paper, we will review the prior art in both areas and describe the planned capabilities of the VTF. In particular, we will focus on scalability and quality-of-service issues required by the next generation of sky surveys, such as LSST and SKA.
△ Less
Submitted 9 November, 2011;
originally announced November 2011.
-
IVOA Recommendation: Sky Event Reporting Metadata Version 2.0
Authors:
Rob Seaman,
Roy Williams,
Alasdair Allan,
Scott Barthelmy,
Joshua Bloom,
John Brewer,
Robert Denny,
Mike Fitzpatrick,
Matthew Graham,
Norman Gray,
Frederic Hessman,
Szabolcs Marka,
Arnold Rots,
Tom Vestrand,
Przemyslaw Wozniak
Abstract:
VOEvent defines the content and meaning of a standard information packet for representing, transmitting, publishing and archiving information about a transient celestial event, with the implication that timely follow-up is of interest. The objective is to motivate the observation of targets-of-opportunity, to drive robotic telescopes, to trigger archive searches, and to alert the community. VOEven…
▽ More
VOEvent defines the content and meaning of a standard information packet for representing, transmitting, publishing and archiving information about a transient celestial event, with the implication that timely follow-up is of interest. The objective is to motivate the observation of targets-of-opportunity, to drive robotic telescopes, to trigger archive searches, and to alert the community. VOEvent is focused on the reporting of photon events, but events mediated by disparate phenomena such as neutrinos, gravitational waves, and solar or atmospheric particle bursts may also be reported.
Structured data is used, rather than natural language, so that automated systems can effectively interpret VOEvent packets. Each packet may contain zero or more of the "who, what, where, when & how" of a detected event, but in addition, may contain a hypothesis (a "why") regarding the nature of the underlying physical cause of the event. Citations to previous VOEvents may be used to place each event in its correct context. Proper curation is encouraged throughout each event's life cycle from discovery through successive follow-ups. VOEvent packets gain persistent identifiers and are typically stored in databases reached via registries. VOEvent packets may therefore reference other packets in various ways. Packets are encouraged to be small and to be processed quickly. This standard does not define a transport layer or the design of clients, repositories, publishers or brokers; it does not cover policy issues such as who can publish, who can build a registry of events, who can subscribe to a particular registry, nor the intellectual property issues.
△ Less
Submitted 3 October, 2011;
originally announced October 2011.
-
IVOA Recommendation: Vocabularies in the Virtual Observatory Version 1.19
Authors:
Sebastien Derriere,
Alasdair J G Gray,
Norman Gray,
Frederic V Hessman,
Tony Linde,
Andrea Preite Martinez,
Rob Seaman,
Brian Thomas
Abstract:
This document specifies a standard format for vocabularies based on the W3C's Resource Description Framework (RDF) and Simple Knowledge Organization System (SKOS). By adopting a standard and simple format, the IVOA will permit different groups to create and maintain their own specialised vocabularies while letting the rest of the astronomical community access, use, and combine them. The use of cur…
▽ More
This document specifies a standard format for vocabularies based on the W3C's Resource Description Framework (RDF) and Simple Knowledge Organization System (SKOS). By adopting a standard and simple format, the IVOA will permit different groups to create and maintain their own specialised vocabularies while letting the rest of the astronomical community access, use, and combine them. The use of current, open standards ensures that VO applications will be able to tap into resources of the growing semantic web. The document provides several examples of useful astronomical vocabularies.
△ Less
Submitted 3 October, 2011;
originally announced October 2011.
-
The Future of Time: UTC and the Leap Second
Authors:
David Finkleman,
Steve Allen,
John Seago,
Rob Seaman,
P. Kenneth Seidelmann
Abstract:
Before atomic timekee**, clocks were set to the skies. But starting in 1972, radio signals began broadcasting atomic seconds and leap seconds have occasionally been added to that stream of atomic seconds to keep the signals synchronized with the actual rotation of Earth. Such adjustments were considered necessary because Earth's rotation is less regular than atomic timekee**. In January 2012,…
▽ More
Before atomic timekee**, clocks were set to the skies. But starting in 1972, radio signals began broadcasting atomic seconds and leap seconds have occasionally been added to that stream of atomic seconds to keep the signals synchronized with the actual rotation of Earth. Such adjustments were considered necessary because Earth's rotation is less regular than atomic timekee**. In January 2012, a United Nations-affiliated organization could permanently break this link by redefining Coordinated Universal Time. To understand the importance of this potential change, it's important to understand the history of human timekee**.
△ Less
Submitted 16 June, 2011;
originally announced June 2011.
-
Optimal Compression of Floating-point Astronomical Images Without Significant Loss of Information
Authors:
W. D. Pence,
R. L. White,
R. Seaman
Abstract:
We describe a compression method for floating-point astronomical images that gives compression ratios of 6 -- 10 while still preserving the scientifically important information in the image. The pixel values are first preprocessed by quantizing them into scaled integer intensity levels, which removes some of the uncompressible noise in the image. The integers are then losslessly compressed using t…
▽ More
We describe a compression method for floating-point astronomical images that gives compression ratios of 6 -- 10 while still preserving the scientifically important information in the image. The pixel values are first preprocessed by quantizing them into scaled integer intensity levels, which removes some of the uncompressible noise in the image. The integers are then losslessly compressed using the fast and efficient Rice algorithm and stored in a portable FITS format file. Quantizing an image more coarsely gives greater image compression, but it also increases the noise and degrades the precision of the photometric and astrometric measurements in the quantized image. Dithering the pixel values during the quantization process can greatly improve the precision of measurements in the images. This is especially important if the analysis algorithm relies on the mode or the median which would be similarly quantized if the pixel values are not dithered. We perform a series of experiments on both synthetic and real astronomical CCD images to quantitatively demonstrate that the magnitudes and positions of stars in the quantized images can be measured with the predicted amount of precision. In order to encourage wider use of these image compression methods, we have made available a pair of general-purpose image compression programs, called fpack and funpack, which can be used to compress any FITS format image.
△ Less
Submitted 7 July, 2010;
originally announced July 2010.
-
Optimal DN encoding for CCD detectors
Authors:
Robert L. Seaman,
Richard L. White,
William D. Pence
Abstract:
Image compression has been a frequent topic of presentations at ADASS. Compression is often viewed as just a technique to fit more data into a smaller space. Rather, the packing of data - its "density" - affects every facet of local data handling, long distance data transport, and the end-to-end throughput of workflows. In short, compression is one aspect of proper data structuring. For example,…
▽ More
Image compression has been a frequent topic of presentations at ADASS. Compression is often viewed as just a technique to fit more data into a smaller space. Rather, the packing of data - its "density" - affects every facet of local data handling, long distance data transport, and the end-to-end throughput of workflows. In short, compression is one aspect of proper data structuring. For example, with FITS tile compression the efficient representation of data is combined with an expressive logistical paradigm for its manipulation.
A deeper question remains. Not just how best to represent the data, but which data to represent. CCDs are linear devices. What does this mean? One thing it does not mean is that the analog-to-digital conversion of pixels must be stored using linear data numbers (DN). An alternative strategy of using non- linear representations is presented, with one motivation being to magnify the efficiency of numerical compression algorithms such as Rice.
△ Less
Submitted 19 October, 2009;
originally announced October 2009.
-
Fully Automated Approaches to Analyze Large-Scale Astronomy Survey Data
Authors:
A. Prsa,
E. F. Guinan,
E. J. Devinney,
S. G. Engle,
M. DeGeorge,
G. P. McCook,
P. A. Maurone,
J. Pepper,
D. J. James,
D. H. Bradstreet,
C. R. Alcock,
J. Devor,
R. Seaman,
T. Zwitter,
K. Long,
R. E. Wilson,
I. Ribas,
A. Gimenez
Abstract:
Observational astronomy has changed drastically in the last decade: manually driven target-by-target instruments have been replaced by fully automated robotic telescopes. Data acquisition methods have advanced to the point that terabytes of data are flowing in and being stored on a daily basis. At the same time, the vast majority of analysis tools in stellar astrophysics still rely on manual exp…
▽ More
Observational astronomy has changed drastically in the last decade: manually driven target-by-target instruments have been replaced by fully automated robotic telescopes. Data acquisition methods have advanced to the point that terabytes of data are flowing in and being stored on a daily basis. At the same time, the vast majority of analysis tools in stellar astrophysics still rely on manual expert interaction. To bridge this gap, we foresee that the next decade will witness a fundamental shift in the approaches to data analysis: case-by-case methods will be replaced by fully automated pipelines that will process the data from their reduction stage, through analysis, to storage. While major effort has been invested in data reduction automation, automated data analysis has mostly been neglected despite the urgent need. Scientific data mining will face serious challenges to identify, understand and eliminate the sources of systematic errors that will arise from this automation. As a special case, we present an artificial intelligence (AI) driven pipeline that is prototyped in the domain of stellar astrophysics (eclipsing binaries in particular), current results and the challenges still ahead.
△ Less
Submitted 4 April, 2009;
originally announced April 2009.
-
Lossless Astronomical Image Compression and the Effects of Noise
Authors:
W. D. Pence,
R. Seaman,
R. L. White
Abstract:
We compare a variety of lossless image compression methods on a large sample of astronomical images and show how the compression ratios and speeds of the algorithms are affected by the amount of noise in the images. In the ideal case where the image pixel values have a random Gaussian distribution, the equivalent number of uncompressible noise bits per pixel is given by Nbits =log2(sigma * sqrt(…
▽ More
We compare a variety of lossless image compression methods on a large sample of astronomical images and show how the compression ratios and speeds of the algorithms are affected by the amount of noise in the images. In the ideal case where the image pixel values have a random Gaussian distribution, the equivalent number of uncompressible noise bits per pixel is given by Nbits =log2(sigma * sqrt(12)) and the lossless compression ratio is given by R = BITPIX / Nbits + K where BITPIX is the bit length of the pixel values and K is a measure of the efficiency of the compression algorithm.
We perform image compression tests on a large sample of integer astronomical CCD images using the GZIP compression program and using a newer FITS tiled-image compression method that currently supports 4 compression algorithms: Rice, Hcompress, PLIO, and GZIP. Overall, the Rice compression algorithm strikes the best balance of compression and computational efficiency; it is 2--3 times faster and produces about 1.4 times greater compression than GZIP. The Rice algorithm produces 75%--90% (depending on the amount of noise in the image) as much compression as an ideal algorithm with K = 0.
The image compression and uncompression utility programs used in this study (called fpack and funpack) are publicly available from the HEASARC web site. A simple command-line interface may be used to compress or uncompress any FITS image file.
△ Less
Submitted 12 March, 2009;
originally announced March 2009.
-
Thread Safe Astronomy
Authors:
Robert Seaman
Abstract:
Observational astronomy is the beneficiary of an ancient chain of apprenticeship. Kepler's laws required Tycho's data. As the pace of discoveries has increased over the centuries, so has the cadence of tutelage (literally, "watching over"). Naked eye astronomy is thousands of years old, the telescope hundreds, digital imaging a few decades, but today's undergraduates will use instrumentation yet…
▽ More
Observational astronomy is the beneficiary of an ancient chain of apprenticeship. Kepler's laws required Tycho's data. As the pace of discoveries has increased over the centuries, so has the cadence of tutelage (literally, "watching over"). Naked eye astronomy is thousands of years old, the telescope hundreds, digital imaging a few decades, but today's undergraduates will use instrumentation yet unbuilt - and thus, unfamiliar to their professors - to complete their doctoral dissertations. Not only has the quickening cadence of astronomical data-taking overrun the apprehension of the science within, but the contingent pace of experimental design threatens our capacity to learn new techniques and apply them productively. Virtual technologies are necessary to accelerate our human processes of perception and comprehension to keep up with astronomical instrumentation and pipelined dataflows. Necessary, but not sufficient. Computers can confuse us as efficiently as they illuminate. Rather, as with neural pathways evolved to meet competitive ecological challenges, astronomical software and data must become organized into ever more coherent "threads" of execution. These are the same threaded constructs as understood by computer science. No datum is an island.
△ Less
Submitted 2 February, 2008;
originally announced February 2008.
