-
Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams
Authors:
P. D. Aleo,
A. W. Engel,
G. Narayan,
C. R. Angus,
K. Malanchev,
K. Auchettl,
V. F. Baldassare,
A. Berres,
T. J. L. de Boer,
B. M. Boyd,
K. C. Chambers,
K. W. Davis,
N. Esquivel,
D. Farias,
R. J. Foley,
A. Gagliano,
C. Gall,
H. Gao,
S. Gomez,
M. Grayling,
C. -C. Lin,
E. A. Magnier,
K. S. Mandel,
T. Matheson,
S. I. Raimundo
, et al. (5 additional authors not shown)
Abstract:
We present LAISS (Lightcurve Anomaly Identification and Similarity Search), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly ZTF Alert Stream via the ANTARES broker, identifying a manageable $\sim$1-5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages…
▽ More
We present LAISS (Lightcurve Anomaly Identification and Similarity Search), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly ZTF Alert Stream via the ANTARES broker, identifying a manageable $\sim$1-5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages statistical light-curve and contextual host-galaxy features within a random forest classifier, tagging transients of rare classes (spectroscopic anomalies), of uncommon host-galaxy environments (contextual anomalies), and of peculiar or interaction-powered phenomena (behavioral anomalies). Moreover, we demonstrate the power of a low-latency ($\sim$ms) approximate similarity search method to find transient analogs with similar light-curve evolution and host-galaxy environments. We use analogs for data-driven discovery, characterization, (re-)classification, and imputation in retrospective and real-time searches. To date we have identified $\sim$50 previously known and previously missed rare transients from real-time and retrospective searches, including but not limited to: SLSNe, TDEs, SNe IIn, SNe IIb, SNe Ia-CSM, SNe Ia-91bg-like, SNe Ib, SNe Ic, SNe Ic-BL, and M31 novae. Lastly, we report the discovery of 325 total transients, all observed between 2018-2021 and absent from public catalogs ($\sim$1% of all ZTF Astronomical Transient reports to the Transient Name Server through 2021). These methods enable a systematic approach to finding the "needle in the haystack" in large-volume data streams. Because of its integration with the ANTARES broker, LAISS is built to detect exciting transients in Rubin data.
△ Less
Submitted 1 April, 2024;
originally announced April 2024.
-
Preliminary Report on Mantis Shrimp: a Multi-Survey Computer Vision Photometric Redshift Model
Authors:
Andrew Engel,
Gautham Narayan,
Nell Byler
Abstract:
The availability of large, public, multi-modal astronomical datasets presents an opportunity to execute novel research that straddles the line between science of AI and science of astronomy. Photometric redshift estimation is a well-established subfield of astronomy. Prior works show that computer vision models typically outperform catalog-based models, but these models face additional complexitie…
▽ More
The availability of large, public, multi-modal astronomical datasets presents an opportunity to execute novel research that straddles the line between science of AI and science of astronomy. Photometric redshift estimation is a well-established subfield of astronomy. Prior works show that computer vision models typically outperform catalog-based models, but these models face additional complexities when incorporating images from more than one instrument or sensor. In this report, we detail our progress creating Mantis Shrimp, a multi-survey computer vision model for photometric redshift estimation that fuses ultra-violet (GALEX), optical (PanSTARRS), and infrared (UnWISE) imagery. We use deep learning interpretability diagnostics to measure how the model leverages information from the different inputs. We reason about the behavior of the CNNs from the interpretability metrics, specifically framing the result in terms of physically-grounded knowledge of galaxy properties.
△ Less
Submitted 5 February, 2024;
originally announced February 2024.
-
Evaluating Physically Motivated Loss Functions for Photometric Redshift Estimation
Authors:
Andrew Engel,
Jan Strube
Abstract:
Physical constraints have been suggested to make neural network models more generalizable, act scientifically plausible, and be more data-efficient over unconstrained baselines. In this report, we present preliminary work on evaluating the effects of adding soft physical constraints to computer vision neural networks trained to estimate the conditional density of redshift on input galaxy images fo…
▽ More
Physical constraints have been suggested to make neural network models more generalizable, act scientifically plausible, and be more data-efficient over unconstrained baselines. In this report, we present preliminary work on evaluating the effects of adding soft physical constraints to computer vision neural networks trained to estimate the conditional density of redshift on input galaxy images for the Sloan Digital Sky Survey. We introduce physically motivated soft constraint terms that are not implemented with differential or integral operators. We frame this work as a simple ablation study where the effect of including soft physical constraints is compared to an unconstrained baseline. We compare networks using standard point estimate metrics for photometric redshift estimation, as well as metrics to evaluate how faithful our conditional density estimate represents the probability over the ensemble of our test dataset. We find no evidence that the implemented soft physical constraints are more effective regularizers than augmentation.
△ Less
Submitted 20 October, 2023;
originally announced October 2023.
-
YSE-PZ: A Transient Survey Management Platform that Empowers the Human-in-the-Loop
Authors:
D. A. Coulter,
D. O. Jones,
P. McGill,
R. J. Foley,
P. D. Aleo,
M. J. Bustamante-Rosell,
D. Chatterjee,
K. W. Davis,
C. Dickinson,
A. Engel,
A. Gagliano,
W. V. Jacobson-Galán,
C. D. Kilpatrick,
J. Kutcka,
X. K. Le Saux,
Y. -C. Pan,
P. J. Quiñonez,
C. Rojas-Bravo,
M. R. Siebert,
K. Taggart,
S. Tinyanont,
Q. Wang
Abstract:
The modern study of astrophysical transients has been transformed by an exponentially growing volume of data. Within the last decade, the transient discovery rate has increased by a factor of ~20, with associated survey data, archival data, and metadata also increasing with the number of discoveries. To manage the data at this increased rate, we require new tools. Here we present YSE-PZ, a transie…
▽ More
The modern study of astrophysical transients has been transformed by an exponentially growing volume of data. Within the last decade, the transient discovery rate has increased by a factor of ~20, with associated survey data, archival data, and metadata also increasing with the number of discoveries. To manage the data at this increased rate, we require new tools. Here we present YSE-PZ, a transient survey management platform that ingests multiple live streams of transient discovery alerts, identifies the host galaxies of those transients, downloads coincident archival data, and retrieves photometry and spectra from ongoing surveys. YSE-PZ also presents a user with a range of tools to make and support timely and informed transient follow-up decisions. Those subsequent observations enhance transient science and can reveal physics only accessible with rapid follow-up observations. Rather than automating out human interaction, YSE-PZ focuses on accelerating and enhancing human decision making, a role we describe as empowering the human-in-the-loop. Finally, YSE-PZ is built to be flexibly used and deployed; YSE-PZ can support multiple, simultaneous, and independent transient collaborations through group-level data permissions, allowing a user to view the data associated with the union of all groups in which they are a member. YSE-PZ can be used as a local instance installed via Docker or deployed as a service hosted in the cloud. We provide YSE-PZ as an open-source tool for the community.
△ Less
Submitted 3 March, 2023;
originally announced March 2023.
-
The Young Supernova Experiment Data Release 1 (YSE DR1): Light Curves and Photometric Classification of 1975 Supernovae
Authors:
P. D. Aleo,
K. Malanchev,
S. Sharief,
D. O. Jones,
G. Narayan,
R. J. Foley,
V. A. Villar,
C. R. Angus,
V. F. Baldassare,
M. J. Bustamante-Rosell,
D. Chatterjee,
C. Cold,
D. A. Coulter,
K. W. Davis,
S. Dhawan,
M. R. Drout,
A. Engel,
K. D. French,
A. Gagliano,
C. Gall,
J. Hjorth,
M. E. Huber,
W. V. Jacobson-Galán,
C. D. Kilpatrick,
D. Langeroodi
, et al. (58 additional authors not shown)
Abstract:
We present the Young Supernova Experiment Data Release 1 (YSE DR1), comprised of processed multi-color Pan-STARRS1 (PS1) griz and Zwicky Transient Facility (ZTF) gr photometry of 1975 transients with host-galaxy associations, redshifts, spectroscopic/photometric classifications, and additional data products from 2019 November 24 to 2021 December 20. YSE DR1 spans discoveries and observations from…
▽ More
We present the Young Supernova Experiment Data Release 1 (YSE DR1), comprised of processed multi-color Pan-STARRS1 (PS1) griz and Zwicky Transient Facility (ZTF) gr photometry of 1975 transients with host-galaxy associations, redshifts, spectroscopic/photometric classifications, and additional data products from 2019 November 24 to 2021 December 20. YSE DR1 spans discoveries and observations from young and fast-rising supernovae (SNe) to transients that persist for over a year, with a redshift distribution reaching z~0.5. We present relative SN rates from YSE's magnitude- and volume-limited surveys, which are consistent with previously published values within estimated uncertainties for untargeted surveys. We combine YSE and ZTF data, and create multi-survey SN simulations to train the ParSNIP and SuperRAENN photometric classification algorithms; when validating our ParSNIP classifier on 472 spectroscopically classified YSE DR1 SNe, we achieve 82% accuracy across three SN classes (SNe Ia, II, Ib/Ic) and 90% accuracy across two SN classes (SNe Ia, core-collapse SNe). Our classifier performs particularly well on SNe Ia, with high (>90%) individual completeness and purity, which will help build an anchor photometric SNe Ia sample for cosmology. We then use our photometric classifier to characterize our photometric sample of 1483 SNe, labeling 1048 (~71%) SNe Ia, 339 (~23%) SNe II, and 96 (~6%) SNe Ib/Ic. YSE DR1 provides a training ground for building discovery, anomaly detection, and classification algorithms, performing cosmological analyses, understanding the nature of red and rare transients, exploring tidal disruption events and nuclear variability, and preparing for the forthcoming Vera C. Rubin Observatory Legacy Survey of Space and Time.
△ Less
Submitted 21 February, 2023; v1 submitted 14 November, 2022;
originally announced November 2022.
-
The Young Supernova Experiment: Survey Goals, Overview, and Operations
Authors:
D. O. Jones,
R. J. Foley,
G. Narayan,
J. Hjorth,
M. E. Huber,
P. D. Aleo,
K. D. Alexander,
C. R. Angus,
K. Auchettl,
V. F. Baldassare,
S. H. Bruun,
K. C. Chambers,
D. Chatterjee,
D. L. Coppejans,
D. A. Coulter,
L. DeMarchi,
G. Dimitriadis,
M. R. Drout,
A. Engel,
K. D. French,
A. Gagliano,
C. Gall,
T. Hung,
L. Izzo,
W. V. Jacobson-Galán
, et al. (46 additional authors not shown)
Abstract:
Time domain science has undergone a revolution over the past decade, with tens of thousands of new supernovae (SNe) discovered each year. However, several observational domains, including SNe within days or hours of explosion and faint, red transients, are just beginning to be explored. Here, we present the Young Supernova Experiment (YSE), a novel optical time-domain survey on the Pan-STARRS tele…
▽ More
Time domain science has undergone a revolution over the past decade, with tens of thousands of new supernovae (SNe) discovered each year. However, several observational domains, including SNe within days or hours of explosion and faint, red transients, are just beginning to be explored. Here, we present the Young Supernova Experiment (YSE), a novel optical time-domain survey on the Pan-STARRS telescopes. Our survey is designed to obtain well-sampled $griz$ light curves for thousands of transient events up to $z \approx 0.2$. This large sample of transients with 4-band light curves will lay the foundation for the Vera C. Rubin Observatory and the Nancy Grace Roman Space Telescope, providing a critical training set in similar filters and a well-calibrated low-redshift anchor of cosmologically useful SNe Ia to benefit dark energy science. As the name suggests, YSE complements and extends other ongoing time-domain surveys by discovering fast-rising SNe within a few hours to days of explosion. YSE is the only current four-band time-domain survey and is able to discover transients as faint $\sim$21.5 mag in $gri$ and $\sim$20.5 mag in $z$, depths that allow us to probe the earliest epochs of stellar explosions. YSE is currently observing approximately 750 square degrees of sky every three days and we plan to increase the area to 1500 square degrees in the near future. When operating at full capacity, survey simulations show that YSE will find $\sim$5000 new SNe per year and at least two SNe within three days of explosion per month. To date, YSE has discovered or observed 8.3% of the transient candidates reported to the International Astronomical Union in 2020. We present an overview of YSE, including science goals, survey characteristics and a summary of our transient discoveries to date.
△ Less
Submitted 5 January, 2021; v1 submitted 19 October, 2020;
originally announced October 2020.
-
GHOST: Using Only Host Galaxy Information to Accurately Associate and Distinguish Supernovae
Authors:
Alex Gagliano,
Gautham Narayan,
Andrew Engel,
Matias Carrasco Kind
Abstract:
We present GHOST, a database of 16,175 spectroscopically classified supernovae and the properties of their host galaxies. We have developed a host galaxy association method using image gradients that achieves fewer misassociations for low-z hosts and higher completeness for high-z hosts than previous methods. We use dimensionality reduction to identify the host galaxy properties that distinguish s…
▽ More
We present GHOST, a database of 16,175 spectroscopically classified supernovae and the properties of their host galaxies. We have developed a host galaxy association method using image gradients that achieves fewer misassociations for low-z hosts and higher completeness for high-z hosts than previous methods. We use dimensionality reduction to identify the host galaxy properties that distinguish supernova classes. Our results suggest that the hosts of SLSNe, SNe Ia, and core collapse supernovae can be separated using host brightness information and extendedness measures derived from the host's light profile. Next, we train a random forest model with data from GHOST to predict supernova class using exclusively host galaxy information and the radial offset of the supernova. We can distinguish SNe Ia and core collapse supernovae with ~70% accuracy without any photometric data from the event itself. Vera C. Rubin Observatory will usher in a new era of transient population studies, demanding improved photometric tools for rapid identification and classification of transient events. By identifying the host features with high discriminatory power, we will maintain SN sample purities and continue to identify scientifically relevant events as data volumes increase. The GHOST database and our corresponding software for associating transients with host galaxies are both publicly available.
△ Less
Submitted 13 January, 2021; v1 submitted 21 August, 2020;
originally announced August 2020.
-
Calculated spectra for HeH+ and its effect on the opacity of cool metal poor stars
Authors:
Elodie A. Engel,
Natasha Doss,
Gregory J. Harris,
Jonathan Tennyson
Abstract:
The wavelength and Einstein A coefficient are calculated for all rotation-vibration transitions of $^4$He$^1$H$^+$, $^3$He$^1$H$^+$, $^4$He$^2$H$^+$ and $^3$He$^2$H$^+$, giving a complete line list and the partition function for $^4$HeH$^+$ and its isotopologues. This opacity is included in the calculation of the total opacity of low-metallicity stars and its effect is analysed for different con…
▽ More
The wavelength and Einstein A coefficient are calculated for all rotation-vibration transitions of $^4$He$^1$H$^+$, $^3$He$^1$H$^+$, $^4$He$^2$H$^+$ and $^3$He$^2$H$^+$, giving a complete line list and the partition function for $^4$HeH$^+$ and its isotopologues. This opacity is included in the calculation of the total opacity of low-metallicity stars and its effect is analysed for different conditions of temperature, density and hydrogen number fraction. For a low helium number fraction (as in the Sun), it is found that HeH$^+$ has a visible but small effect for very low densities ($ρ\leq 10^{-10}$g cm$^{-3}$), at temperatures around 3500 K. However, for high helium number fraction, the effect of HeH$^+$ becomes important for higher densities ($ρ\leq 10^{-6}$g cm$^{-3}$), its effect being most important for a temperature around 3500 K. Synthetic spectra for a variety of different conditions are presented.
△ Less
Submitted 10 November, 2004;
originally announced November 2004.
-
Simulating Electron Transport and Synchrotron Emission in Radio Galaxies: Shock Acceleration and Synchrotron Aging in Axis-Symmetric Flows
Authors:
T. W. Jones,
Dongsu Ryu,
Andrew Engel
Abstract:
We introduce a simple and economical but effective method for including relativistic electron transport in multi-dimensional simulations of radio galaxies. The method is designed to follow explicitly diffusive acceleration at shocks, and, in smooth flows 2nd order Fermi acceleration plus adiabatic and synchrotron cooling. We are able to follow both the spatial and energy distributions of the ele…
▽ More
We introduce a simple and economical but effective method for including relativistic electron transport in multi-dimensional simulations of radio galaxies. The method is designed to follow explicitly diffusive acceleration at shocks, and, in smooth flows 2nd order Fermi acceleration plus adiabatic and synchrotron cooling. We are able to follow both the spatial and energy distributions of the electrons, so that direct synchrotron emission properties can be modeled in time-dependent flows for the first time.
Here we present first results in the form of some axis-symmetric MHD simulations of Mach 20 light jet flows. These show clearly the importance of nonsteady terminal shocks that develop in such flows even when the jet inflow is steady. As a result of this and other consequences of the fundamentally driven character of jets, we find complex patterns of emissivities and synchrotron spectra, including steep spectral gradients in hot spots, islands of distinct spectra electrons within the lobes and spectral gradients coming from the dynamical histories of a given flow element rather than from synchrotron aging of the embedded electrons. In addition, spectral aging in the lobes tends to proceed more slowly than one would estimate from regions of high emissivity.
△ Less
Submitted 7 September, 1998;
originally announced September 1998.
