Showing 1–40 of 40 results for author: Cranmer, M

Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Task

Authors: Siavash Golkar, Alberto Bietti, Mariel Pettee, Michael Eickenberg, Miles Cranmer, Keiya Hirashima, Geraud Krawezik, Nicholas Lourie, Michael McCabe, Rudy Morel, Ruben Ohana, Liam Holden Parker, Bruno Régaldo-Saint Blancard, Kyunghyun Cho, Shirley Ho

Abstract: Transformers have revolutionized machine learning across diverse domains, yet understanding their behavior remains crucial, particularly in high-stakes applications. This paper introduces the contextual counting task, a novel toy problem aimed at enhancing our understanding of Transformers in quantitative and scientific contexts. This task requires precise localization and computation within datas… ▽ More Transformers have revolutionized machine learning across diverse domains, yet understanding their behavior remains crucial, particularly in high-stakes applications. This paper introduces the contextual counting task, a novel toy problem aimed at enhancing our understanding of Transformers in quantitative and scientific contexts. This task requires precise localization and computation within datasets, akin to object detection or region-based scientific analysis. We present theoretical and empirical analysis using both causal and non-causal Transformer architectures, investigating the influence of various positional encodings on performance and interpretability. In particular, we find that causal attention is much better suited for the task, and that no positional embeddings lead to the best accuracy, though rotary embeddings are competitive and easier to train. We also show that out of distribution performance is tightly linked to which tokens it uses as a bias term. △ Less

Submitted 30 May, 2024; originally announced June 2024.

Five parameters are all you need (in $Λ$CDM)

Authors: Paulo Montero-Camacho, Yin Li, Miles Cranmer

Abstract: The standard cosmological model, with its six independent parameters, has proven remarkably successful in describing the evolution of the Universe. One of these parameters, the optical depth to reionization $τ_{\mathrm{reio}}$, represents the scatterings that Cosmic Microwave Background (CMB) photons will experience after decoupling from the primordial plasma as the intergalactic medium transition… ▽ More The standard cosmological model, with its six independent parameters, has proven remarkably successful in describing the evolution of the Universe. One of these parameters, the optical depth to reionization $τ_{\mathrm{reio}}$, represents the scatterings that Cosmic Microwave Background (CMB) photons will experience after decoupling from the primordial plasma as the intergalactic medium transitions from neutral to ionized. $τ_{\mathrm{reio}}$ depends on the neutral hydrogen fraction $x_{\mathrm{HI}}$, which, in turn, should theoretically depend on cosmology. We present a novel method to establish the missing link between cosmology and reionization timeline. We discover the timeline has a universal shape well described by the Gompertz mortality law, applicable to any cosmology within our simulated data. This enables us to map cosmology to reionization using symbolic regression and to treat $τ_{\mathrm{reio}}$ as a derived parameter. Reanalyzing CMB data with our universal $x_{\mathrm{HI}}$ tightens the constraint on $τ_{\mathrm {reio}}$ by more than one order of magnitude to $\approx 1\%$ and reduced the error on the amplitude of the primordial fluctuations by a factor of 2.5 compared to Planck's PR3 constraint. While our results rely on the astrophysical assumptions in our simulations, the methodology itself is independent of these assumptions; after all, $τ_{\mathrm{reio}}$ is fundamentally a function of cosmology. △ Less

Submitted 22 May, 2024; originally announced May 2024.

Comments: 9 figures. Comments welcomed

AstroCLIP: A Cross-Modal Foundation Model for Galaxies

Authors: Liam Parker, Francois Lanusse, Siavash Golkar, Leopoldo Sarra, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Geraud Krawezik, Michael McCabe, Ruben Ohana, Mariel Pettee, Bruno Regaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho, Shirley Ho

Abstract: We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation fro… ▽ More We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19\% in terms of $R^2$. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of $R^2$, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra. △ Less

Submitted 14 June, 2024; v1 submitted 4 October, 2023; originally announced October 2023.

Comments: 18 pages, accepted in Monthly Notices of the Royal Astronomical Society, Presented at the NeurIPS 2023 AI4Science Workshop

Multiple Physics Pretraining for Physical Surrogate Models

Authors: Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, Mariel Pettee, Tiberiu Tesileanu, Kyunghyun Cho, Shirley Ho

Abstract: We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling. MPP involves training large surrogate models to predict the dynamics of multiple heterogeneous physical systems simultaneously by learning features that are broadly useful across diverse physical tasks. In order to learn effectively in this setting, we introduce a… ▽ More We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling. MPP involves training large surrogate models to predict the dynamics of multiple heterogeneous physical systems simultaneously by learning features that are broadly useful across diverse physical tasks. In order to learn effectively in this setting, we introduce a shared embedding and normalization strategy that projects the fields of multiple systems into a single shared embedding space. We validate the efficacy of our approach on both pretraining and downstream tasks over a broad fluid mechanics-oriented benchmark. We show that a single MPP-pretrained transformer is able to match or outperform task-specific baselines on all pretraining sub-tasks without the need for finetuning. For downstream tasks, we demonstrate that finetuning MPP-trained models results in more accurate predictions across multiple time-steps on new physics compared to training from scratch or finetuning pretrained video foundation models. We open-source our code and model weights trained at multiple scales for reproducibility and community experimentation. △ Less

Submitted 4 October, 2023; originally announced October 2023.

xVal: A Continuous Number Encoding for Large Language Models

Authors: Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, Bruno Régaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho, Shirley Ho

Abstract: Large Language Models have not yet been broadly adapted for the analysis of scientific datasets due in part to the unique difficulties of tokenizing numbers. We propose xVal, a numerical encoding scheme that represents any real number using just a single token. xVal represents a given real number by scaling a dedicated embedding vector by the number value. Combined with a modified number-inference… ▽ More Large Language Models have not yet been broadly adapted for the analysis of scientific datasets due in part to the unique difficulties of tokenizing numbers. We propose xVal, a numerical encoding scheme that represents any real number using just a single token. xVal represents a given real number by scaling a dedicated embedding vector by the number value. Combined with a modified number-inference approach, this strategy renders the model end-to-end continuous when considered as a map from the numbers of the input string to those of the output string. This leads to an inductive bias that is generally more suitable for applications in scientific domains. We empirically evaluate our proposal on a number of synthetic and real-world datasets. Compared with existing number encoding schemes, we find that xVal is more token-efficient and demonstrates improved generalization. △ Less

Submitted 4 October, 2023; originally announced October 2023.

Comments: 10 pages 7 figures. Supplementary: 5 pages 2 figures

Reusability report: Prostate cancer stratification with diverse biologically-informed neural architectures

Authors: Christian Pedersen, Tiberiu Tesileanu, Tinghui Wu, Siavash Golkar, Miles Cranmer, Zijun Zhang, Shirley Ho

Abstract: In Elmarakeby et al., "Biologically informed deep neural network for prostate cancer discovery", a feedforward neural network with biologically informed, sparse connections (P-NET) was presented to model the state of prostate cancer. We verified the reproducibility of the study conducted by Elmarakeby et al., using both their original codebase, and our own re-implementation using more up-to-date l… ▽ More In Elmarakeby et al., "Biologically informed deep neural network for prostate cancer discovery", a feedforward neural network with biologically informed, sparse connections (P-NET) was presented to model the state of prostate cancer. We verified the reproducibility of the study conducted by Elmarakeby et al., using both their original codebase, and our own re-implementation using more up-to-date libraries. We quantified the contribution of network sparsification by Reactome biological pathways, and confirmed its importance to P-NET's superior performance. Furthermore, we explored alternative neural architectures and approaches to incorporating biological information into the networks. We experimented with three types of graph neural networks on the same training data, and investigated the clinical prediction agreement between different models. Our analyses demonstrated that deep neural networks with distinct architectures make incorrect predictions for individual patient that are persistent across different initializations of a specific neural architecture. This suggests that different neural architectures are sensitive to different aspects of the data, an important yet under-explored challenge for clinical prediction tasks. △ Less

Submitted 30 October, 2023; v1 submitted 28 September, 2023; originally announced September 2023.

Comments: 9 pages, 3 figures. Submitted to Nature Machine Intelligence

Symbolic Regression on FPGAs for Fast Machine Learning Inference

Authors: Ho Fung Tsoi, Adrian Alan Pol, Vladimir Loncar, Ekaterina Govorkova, Miles Cranmer, Sridhara Dasu, Peter Elmer, Philip Harris, Isobel Ojalvo, Maurizio Pierini

Abstract: The high-energy physics community is investigating the potential of deploying machine-learning-based solutions on Field-Programmable Gate Arrays (FPGAs) to enhance physics sensitivity while still meeting data processing time constraints. In this contribution, we introduce a novel end-to-end procedure that utilizes a machine learning technique called symbolic regression (SR). It searches the equati… ▽ More The high-energy physics community is investigating the potential of deploying machine-learning-based solutions on Field-Programmable Gate Arrays (FPGAs) to enhance physics sensitivity while still meeting data processing time constraints. In this contribution, we introduce a novel end-to-end procedure that utilizes a machine learning technique called symbolic regression (SR). It searches the equation space to discover algebraic relations approximating a dataset. We use PySR (a software to uncover these expressions based on an evolutionary algorithm) and extend the functionality of hls4ml (a package for machine learning inference in FPGAs) to support PySR-generated expressions for resource-constrained production environments. Deep learning models often optimize the top metric by pinning the network size because the vast hyperparameter space prevents an extensive search for neural architecture. Conversely, SR selects a set of models on the Pareto front, which allows for optimizing the performance-resource trade-off directly. By embedding symbolic forms, our implementation can dramatically reduce the computational resources needed to perform critical tasks. We validate our method on a physics benchmark: the multiclass classification of jets produced in simulated proton-proton collisions at the CERN Large Hadron Collider. We show that our approach can approximate a 3-layer neural network using an inference model that achieves up to a 13-fold decrease in execution time, down to 5 ns, while still preserving more than 90% approximation accuracy. △ Less

Submitted 17 January, 2024; v1 submitted 6 May, 2023; originally announced May 2023.

Comments: 9 pages. Accepted to 26th International Conference on Computing in High Energy & Nuclear Physics (CHEP 2023)

Journal ref: EPJ Web of Conferences 295, 09036 (2024)

Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl

Authors: Miles Cranmer

Abstract: PySR is an open-source library for practical symbolic regression, a type of machine learning which aims to discover human-interpretable symbolic models. PySR was developed to democratize and popularize symbolic regression for the sciences, and is built on a high-performance distributed back-end, a flexible search algorithm, and interfaces with several deep learning packages. PySR's internal search… ▽ More PySR is an open-source library for practical symbolic regression, a type of machine learning which aims to discover human-interpretable symbolic models. PySR was developed to democratize and popularize symbolic regression for the sciences, and is built on a high-performance distributed back-end, a flexible search algorithm, and interfaces with several deep learning packages. PySR's internal search algorithm is a multi-population evolutionary algorithm, which consists of a unique evolve-simplify-optimize loop, designed for optimization of unknown scalar constants in newly-discovered empirical expressions. PySR's backend is the extremely optimized Julia library SymbolicRegression.jl, which can be used directly from Julia. It is capable of fusing user-defined operators into SIMD kernels at runtime, performing automatic differentiation, and distributing populations of expressions to thousands of cores across a cluster. In describing this software, we also introduce a new benchmark, "EmpiricalBench," to quantify the applicability of symbolic regression algorithms in science. This benchmark measures recovery of historical empirical equations from original and synthetic datasets. △ Less

Submitted 5 May, 2023; v1 submitted 2 May, 2023; originally announced May 2023.

Comments: 24 pages, 5 figures, 3 tables. Feedback welcome. Paper source found at https://github.com/MilesCranmer/pysr_paper ; PySR at https://github.com/MilesCranmer/PySR ; SymbolicRegression.jl at https://github.com/MilesCranmer/SymbolicRegression.jl

Interpretable Symbolic Regression for Data Science: Analysis of the 2022 Competition

Authors: F. O. de Franca, M. Virgolin, M. Kommenda, M. S. Majumder, M. Cranmer, G. Espada, L. Ingelse, A. Fonseca, M. Landajuela, B. Petersen, R. Glatt, N. Mundhenk, C. S. Lee, J. D. Hochhalter, D. L. Randall, P. Kamienny, H. Zhang, G. Dick, A. Simon, B. Burlacu, Jaan Kasak, Meera Machado, Casper Wilstrup, W. G. La Cava

Abstract: Symbolic regression searches for analytic expressions that accurately describe studied phenomena. The main attraction of this approach is that it returns an interpretable model that can be insightful to users. Historically, the majority of algorithms for symbolic regression have been based on evolutionary algorithms. However, there has been a recent surge of new proposals that instead utilize appr… ▽ More Symbolic regression searches for analytic expressions that accurately describe studied phenomena. The main attraction of this approach is that it returns an interpretable model that can be insightful to users. Historically, the majority of algorithms for symbolic regression have been based on evolutionary algorithms. However, there has been a recent surge of new proposals that instead utilize approaches such as enumeration algorithms, mixed linear integer programming, neural networks, and Bayesian optimization. In order to assess how well these new approaches behave on a set of common challenges often faced in real-world data, we hosted a competition at the 2022 Genetic and Evolutionary Computation Conference consisting of different synthetic and real-world datasets which were blind to entrants. For the real-world track, we assessed interpretability in a realistic way by using a domain expert to judge the trustworthiness of candidate models.We present an in-depth analysis of the results obtained in this competition, discuss current challenges of symbolic regression algorithms and highlight possible improvements for future competitions. △ Less

Submitted 3 July, 2023; v1 submitted 3 April, 2023; originally announced April 2023.

Comments: 13 pages, 13 figures, submitted to IEEE Transactions on Evolutionary Computation

Learning Integrable Dynamics with Action-Angle Networks

Authors: Ameya Daigavane, Arthur Kosmala, Miles Cranmer, Tess Smidt, Shirley Ho

Abstract: Machine learning has become increasingly popular for efficiently modelling the dynamics of complex physical systems, demonstrating a capability to learn effective models for dynamics which ignore redundant degrees of freedom. Learned simulators typically predict the evolution of the system in a step-by-step manner with numerical integration techniques. However, such models often suffer from instab… ▽ More Machine learning has become increasingly popular for efficiently modelling the dynamics of complex physical systems, demonstrating a capability to learn effective models for dynamics which ignore redundant degrees of freedom. Learned simulators typically predict the evolution of the system in a step-by-step manner with numerical integration techniques. However, such models often suffer from instability over long roll-outs due to the accumulation of both estimation and integration error at each prediction step. Here, we propose an alternative construction for learned physical simulators that are inspired by the concept of action-angle coordinates from classical mechanics for describing integrable systems. We propose Action-Angle Networks, which learn a nonlinear transformation from input coordinates to the action-angle space, where evolution of the system is linear. Unlike traditional learned simulators, Action-Angle Networks do not employ any higher-order numerical integration methods, making them extremely efficient at modelling the dynamics of integrable physical systems. △ Less

Submitted 24 November, 2022; originally announced November 2022.

Comments: Accepted at Machine Learning and the Physical Sciences workshop at NeurIPS 2022

Normalizing Flows for Hierarchical Bayesian Analysis: A Gravitational Wave Population Study

Authors: David Ruhe, Kaze Wong, Miles Cranmer, Patrick Forré

Abstract: We propose parameterizing the population distribution of the gravitational wave population modeling framework (Hierarchical Bayesian Analysis) with a normalizing flow. We first demonstrate the merit of this method on illustrative experiments and then analyze four parameters of the latest LIGO/Virgo data release: primary mass, secondary mass, redshift, and effective spin. Our results show that desp… ▽ More We propose parameterizing the population distribution of the gravitational wave population modeling framework (Hierarchical Bayesian Analysis) with a normalizing flow. We first demonstrate the merit of this method on illustrative experiments and then analyze four parameters of the latest LIGO/Virgo data release: primary mass, secondary mass, redshift, and effective spin. Our results show that despite the small and notoriously noisy dataset, the posterior predictive distributions (assuming a prior over the parameters of the flow) of the observed gravitational wave population recover structure that agrees with robust previous phenomenological modeling results while being less susceptible to biases introduced by less flexible models. Therefore, the method forms a promising flexible, reliable replacement for population inference distributions, even when data is highly noisy. △ Less

Submitted 29 December, 2022; v1 submitted 15 November, 2022; originally announced November 2022.

Hierarchical Inference of the Lensing Convergence from Photometric Catalogs with Bayesian Graph Neural Networks

Authors: Ji Won Park, Simon Birrer, Madison Ueland, Miles Cranmer, Adriano Agnello, Sebastian Wagner-Carena, Philip J. Marshall, Aaron Roodman, the LSST Dark Energy Science Collaboration

Abstract: We present a Bayesian graph neural network (BGNN) that can estimate the weak lensing convergence ($κ$) from photometric measurements of galaxies along a given line of sight. The method is of particular interest in strong gravitational time delay cosmography (TDC), where characterizing the "external convergence" ($κ_{\rm ext}$) from the lens environment and line of sight is necessary for precise in… ▽ More We present a Bayesian graph neural network (BGNN) that can estimate the weak lensing convergence ($κ$) from photometric measurements of galaxies along a given line of sight. The method is of particular interest in strong gravitational time delay cosmography (TDC), where characterizing the "external convergence" ($κ_{\rm ext}$) from the lens environment and line of sight is necessary for precise inference of the Hubble constant ($H_0$). Starting from a large-scale simulation with a $κ$ resolution of $\sim$1$'$, we introduce fluctuations on galaxy-galaxy lensing scales of $\sim$1$''$ and extract random sightlines to train our BGNN. We then evaluate the model on test sets with varying degrees of overlap with the training distribution. For each test set of 1,000 sightlines, the BGNN infers the individual $κ$ posteriors, which we combine in a hierarchical Bayesian model to yield constraints on the hyperparameters governing the population. For a test field well sampled by the training set, the BGNN recovers the population mean of $κ$ precisely and without bias, resulting in a contribution to the $H_0$ error budget well under 1\%. In the tails of the training set with sparse samples, the BGNN, which can ingest all available information about each sightline, extracts more $κ$ signal compared to a simplified version of the traditional method based on matching galaxy number counts, which is limited by sample variance. Our hierarchical inference pipeline using BGNNs promises to improve the $κ_{\rm ext}$ characterization for precision TDC. The implementation of our pipeline is available as a public Python package, Node to Joy. △ Less

Submitted 14 November, 2022; originally announced November 2022.

Comments: 15 pages, 8 figures (+ 6 pages, 2 figures in Appendix). Submitted to ApJ. Code at https://github.com/jiwoncpark/node-to-joy

A Neural Network Subgrid Model of the Early Stages of Planet Formation

Authors: Thomas Pfeil, Miles Cranmer, Shirley Ho, Philip J. Armitage, Tilman Birnstiel, Hubert Klahr

Abstract: Planet formation is a multi-scale process in which the coagulation of $\mathrm{μm}$-sized dust grains in protoplanetary disks is strongly influenced by the hydrodynamic processes on scales of astronomical units ($\approx 1.5\times 10^8 \,\mathrm{km}$). Studies are therefore dependent on subgrid models to emulate the micro physics of dust coagulation on top of a large scale hydrodynamic simulation.… ▽ More Planet formation is a multi-scale process in which the coagulation of $\mathrm{μm}$-sized dust grains in protoplanetary disks is strongly influenced by the hydrodynamic processes on scales of astronomical units ($\approx 1.5\times 10^8 \,\mathrm{km}$). Studies are therefore dependent on subgrid models to emulate the micro physics of dust coagulation on top of a large scale hydrodynamic simulation. Numerical simulations which include the relevant physical effects are complex and computationally expensive. Here, we present a fast and accurate learned effective model for dust coagulation, trained on data from high resolution numerical coagulation simulations. Our model captures details of the dust coagulation process that were so far not tractable with other dust coagulation prescriptions with similar computational efficiency. △ Less

Submitted 8 November, 2022; originally announced November 2022.

Comments: 6 pages, 4 figures, accepted at the Machine Learning and the Physical Sciences workshop, NeurIPS 2022

$\texttt{Mangrove}$: Learning Galaxy Properties from Merger Trees

Authors: Christian Kragh Jespersen, Miles Cranmer, Peter Melchior, Shirley Ho, Rachel S. Somerville, Austen Gabrielpillai

Abstract: Efficiently map** baryonic properties onto dark matter is a major challenge in astrophysics. Although semi-analytic models (SAMs) and hydrodynamical simulations have made impressive advances in reproducing galaxy observables across cosmologically significant volumes, these methods still require significant computation times, representing a barrier to many applications. Graph Neural Networks (GNN… ▽ More Efficiently map** baryonic properties onto dark matter is a major challenge in astrophysics. Although semi-analytic models (SAMs) and hydrodynamical simulations have made impressive advances in reproducing galaxy observables across cosmologically significant volumes, these methods still require significant computation times, representing a barrier to many applications. Graph Neural Networks (GNNs) have recently proven to be the natural choice for learning physical relations. Among the most inherently graph-like structures found in astrophysics are the dark matter merger trees that encode the evolution of dark matter halos. In this paper we introduce a new, graph-based emulator framework, $\texttt{Mangrove}$, and show that it emulates the galactic stellar mass, cold gas mass and metallicity, instantaneous and time-averaged star formation rate, and black hole mass -- as predicted by a SAM -- with root mean squared error up to two times lower than other methods across a $(75 Mpc/h)^3$ simulation box in 40 seconds, 4 orders of magnitude faster than the SAM. We show that $\texttt{Mangrove}$ allows for quantification of the dependence of galaxy properties on merger history. We compare our results to the current state of the art in the field and show significant improvements for all target properties. $\texttt{Mangrove}$ is publicly available. △ Less

Submitted 24 October, 2022; originally announced October 2022.

Comments: 15 pages, 9 figures, 3 tables, 10 pages of Appendices. Accepted for publication in ApJ

The SZ flux-mass ($Y$-$M$) relation at low halo masses: improvements with symbolic regression and strong constraints on baryonic feedback

Authors: Digvijay Wadekar, Leander Thiele, J. Colin Hill, Shivam Pandey, Francisco Villaescusa-Navarro, David N. Spergel, Miles Cranmer, Daisuke Nagai, Daniel Anglés-Alcázar, Shirley Ho, Lars Hernquist

Abstract: Feedback from active galactic nuclei (AGN) and supernovae can affect measurements of integrated SZ flux of halos ($Y_\mathrm{SZ}$) from CMB surveys, and cause its relation with the halo mass ($Y_\mathrm{SZ}-M$) to deviate from the self-similar power-law prediction of the virial theorem. We perform a comprehensive study of such deviations using CAMELS, a suite of hydrodynamic simulations with exten… ▽ More Feedback from active galactic nuclei (AGN) and supernovae can affect measurements of integrated SZ flux of halos ($Y_\mathrm{SZ}$) from CMB surveys, and cause its relation with the halo mass ($Y_\mathrm{SZ}-M$) to deviate from the self-similar power-law prediction of the virial theorem. We perform a comprehensive study of such deviations using CAMELS, a suite of hydrodynamic simulations with extensive variations in feedback prescriptions. We use a combination of two machine learning tools (random forest and symbolic regression) to search for analogues of the $Y-M$ relation which are more robust to feedback processes for low masses ($M\lesssim 10^{14}\, h^{-1} \, M_\odot$); we find that simply replacing $Y\rightarrow Y(1+M_*/M_\mathrm{gas})$ in the relation makes it remarkably self-similar. This could serve as a robust multiwavelength mass proxy for low-mass clusters and galaxy groups. Our methodology can also be generally useful to improve the domain of validity of other astrophysical scaling relations. We also forecast that measurements of the $Y-M$ relation could provide percent-level constraints on certain combinations of feedback parameters and/or rule out a major part of the parameter space of supernova and AGN feedback models used in current state-of-the-art hydrodynamic simulations. Our results can be useful for using upcoming SZ surveys (e.g., SO, CMB-S4) and galaxy surveys (e.g., DESI and Rubin) to constrain the nature of baryonic feedback. Finally, we find that the an alternative relation, $Y-M_*$, provides complementary information on feedback than $Y-M$ △ Less

Submitted 28 April, 2023; v1 submitted 5 September, 2022; originally announced September 2022.

Comments: Version appearing in MNRAS. Minor change to Fig.6 and added Fig. A5 compared to the previous version. 7+5 pages. The code and data associated with this paper are available at https://github.com/JayWadekar/ScalingRelations_ML

Journal ref: MNRAS, 522, 2628 (2023)

Automated discovery of interpretable gravitational-wave population models

Authors: Kaze W. K Wong, Miles Cranmer

Abstract: We present an automatic approach to discover analytic population models for gravitational-wave (GW) events from data. As more gravitational-wave (GW) events are detected, flexible models such as Gaussian Mixture Models have become more important in fitting the distribution of GW properties due to their expressivity. However, flexible models come with many parameters that lack physical motivation,… ▽ More We present an automatic approach to discover analytic population models for gravitational-wave (GW) events from data. As more gravitational-wave (GW) events are detected, flexible models such as Gaussian Mixture Models have become more important in fitting the distribution of GW properties due to their expressivity. However, flexible models come with many parameters that lack physical motivation, making interpreting the implication of these models challenging. In this work, we demonstrate symbolic regression can complement flexible models by distilling the posterior predictive distribution of such flexible models into interpretable analytic expressions. We recover common GW population models such as a power-law-plus-Gaussian, and find a new empirical population model which combines accuracy and simplicity. This demonstrates a strategy to automatically discover interpretable population models in the ever-growing GW catalog, which can potentially be applied to other astrophysical phenomena. △ Less

Submitted 25 July, 2022; originally announced July 2022.

Comments: Published in ML4Astro Workshop at ICML 2022. 8 pages, 1 figure. Code at https://github.com/kazewong/SymbolicGWPopulation_paper

Robust Simulation-Based Inference in Cosmology with Bayesian Neural Networks

Authors: Pablo Lemos, Miles Cranmer, Muntazir Abidi, ChangHoon Hahn, Michael Eickenberg, Elena Massara, David Yallup, Shirley Ho

Abstract: Simulation-based inference (SBI) is rapidly establishing itself as a standard machine learning technique for analyzing data in cosmological surveys. Despite continual improvements to the quality of density estimation by learned models, applications of such techniques to real data are entirely reliant on the generalization power of neural networks far outside the training distribution, which is mos… ▽ More Simulation-based inference (SBI) is rapidly establishing itself as a standard machine learning technique for analyzing data in cosmological surveys. Despite continual improvements to the quality of density estimation by learned models, applications of such techniques to real data are entirely reliant on the generalization power of neural networks far outside the training distribution, which is mostly unconstrained. Due to the imperfections in scientist-created simulations, and the large computational expense of generating all possible parameter combinations, SBI methods in cosmology are vulnerable to such generalization issues. Here, we discuss the effects of both issues, and show how using a Bayesian neural network framework for training SBI can mitigate biases, and result in more reliable inference outside the training set. We introduce cosmoSWAG, the first application of Stochastic Weight Averaging to cosmology, and apply it to SBI trained for inference on the cosmic microwave background. △ Less

Submitted 2 March, 2023; v1 submitted 18 July, 2022; originally announced July 2022.

Comments: 5 pages, 3 figures. Preliminary version accepted at the ML4Astro Machine Learning for Astrophysics Workshop at the Thirty-ninth International Conference on Machine Learning (ICML 2022). Final version published at Machine Learning: Science and Technology

Journal ref: Mach. Learn.: Sci. Technol. 4 01LT01 (2023)

GaMPEN: A Machine Learning Framework for Estimating Bayesian Posteriors of Galaxy Morphological Parameters

Authors: Aritra Ghosh, C. Megan Urry, Amrit Rau, Laurence Perreault-Levasseur, Miles Cranmer, Kevin Schawinski, Dominic Stark, Chuan Tian, Ryan Ofman, Tonima Tasnim Ananna, Connor Auge, Nico Cappelluti, David B. Sanders, Ezequiel Treister

Abstract: We introduce a novel machine learning framework for estimating the Bayesian posteriors of morphological parameters for arbitrarily large numbers of galaxies. The Galaxy Morphology Posterior Estimation Network (GaMPEN) estimates values and uncertainties for a galaxy's bulge-to-total light ratio ($L_B/L_T$), effective radius ($R_e$), and flux ($F$). To estimate posteriors, GaMPEN uses the Monte Carl… ▽ More We introduce a novel machine learning framework for estimating the Bayesian posteriors of morphological parameters for arbitrarily large numbers of galaxies. The Galaxy Morphology Posterior Estimation Network (GaMPEN) estimates values and uncertainties for a galaxy's bulge-to-total light ratio ($L_B/L_T$), effective radius ($R_e$), and flux ($F$). To estimate posteriors, GaMPEN uses the Monte Carlo Dropout technique and incorporates the full covariance matrix between the output parameters in its loss function. GaMPEN also uses a Spatial Transformer Network (STN) to automatically crop input galaxy frames to an optimal size before determining their morphology. This will allow it to be applied to new data without prior knowledge of galaxy size. Training and testing GaMPEN on galaxies simulated to match $z < 0.25$ galaxies in Hyper Suprime-Cam Wide $g$-band images, we demonstrate that GaMPEN achieves typical errors of $0.1$ in $L_B/L_T$, $0.17$ arcsec ($\sim 7\%$) in $R_e$, and $6.3\times10^4$ nJy ($\sim 1\%$) in $F$. GaMPEN's predicted uncertainties are well-calibrated and accurate ($<5\%$ deviation) -- for regions of the parameter space with high residuals, GaMPEN correctly predicts correspondingly large uncertainties. We also demonstrate that we can apply categorical labels (i.e., classifications such as "highly bulge-dominated") to predictions in regions with high residuals and verify that those labels are $\gtrsim 97\%$ accurate. To the best of our knowledge, GaMPEN is the first machine learning framework for determining joint posterior distributions of multiple morphological parameters and is also the first application of an STN to optical imaging in astronomy. △ Less

Submitted 11 July, 2022; originally announced July 2022.

Comments: Accepted for publication in The Astrophysical Journal. We welcome comments and constructive criticism. Digital assets will be available at http://gampen.ghosharitra.com

Journal ref: The Astrophysical Journal 935.2 (2022): 138

TNT: Vision Transformer for Turbulence Simulations

Authors: Yuchen Dang, Zheyuan Hu, Miles Cranmer, Michael Eickenberg, Shirley Ho

Abstract: Turbulence is notoriously difficult to model due to its multi-scale nature and sensitivity to small perturbations. Classical solvers of turbulence simulation generally operate on finer grids and are computationally inefficient. In this paper, we propose the Turbulence Neural Transformer (TNT), which is a learned simulator based on the transformer architecture, to predict turbulent dynamics on coar… ▽ More Turbulence is notoriously difficult to model due to its multi-scale nature and sensitivity to small perturbations. Classical solvers of turbulence simulation generally operate on finer grids and are computationally inefficient. In this paper, we propose the Turbulence Neural Transformer (TNT), which is a learned simulator based on the transformer architecture, to predict turbulent dynamics on coarsened grids. TNT extends the positional embeddings of vanilla transformers to a spatiotemporal setting to learn the representation in the 3D time-series domain, and applies Temporal Mutual Self-Attention (TMSA), which captures adjacent dependencies, to extract deep and dynamic features. TNT is capable of generating comparatively long-range predictions stably and accurately, and we show that TNT outperforms the state-of-the-art U-net simulator on several metrics. We also test the model performance with different components removed and evaluate robustness to different initial conditions. Although more experiments are needed, we conclude that TNT has great potential to outperform existing solvers and generalize to additional simulation datasets. △ Less

Submitted 11 July, 2022; originally announced July 2022.

Charting Galactic Accelerations with Stellar Streams and Machine Learning

Authors: Jacob Nibauer, Vasily Belokurov, Miles Cranmer, Jeremy Goodman, Shirley Ho

Abstract: We present a data-driven method for reconstructing the galactic acceleration field from phase-space measurements of stellar streams. Our approach is based on a flexible and differentiable fit to the stream in phase-space, enabling a direct estimate of the acceleration vector along the stream. Reconstruction of the local acceleration field can be applied independently to each of several streams, al… ▽ More We present a data-driven method for reconstructing the galactic acceleration field from phase-space measurements of stellar streams. Our approach is based on a flexible and differentiable fit to the stream in phase-space, enabling a direct estimate of the acceleration vector along the stream. Reconstruction of the local acceleration field can be applied independently to each of several streams, allowing us to sample the acceleration field due to the underlying galactic potential across a range of scales. Our approach is methodologically different from previous works, since a model for the gravitational potential does not need to be adopted beforehand. Instead, our flexible neural-network-based model treats the stream as a collection of orbits with a locally similar mixture of energies, rather than assuming that the stream delineates a single stellar orbit. Accordingly, our approach allows for distinct regions of the stream to have different mean energies, as is the case for real stellar streams. Once the acceleration vector is sampled along the stream, standard analytic models for the galactic potential can then be rapidly constrained. We find our method recovers the correct parameters for a ground-truth triaxial logarithmic halo potential when applied to simulated stellar streams. Alternatively, we demonstrate that a flexible potential can be constrained with a neural network, though standard multipole expansions can also be constrained. Our approach is applicable to simple and complicated gravitational potentials alike, and enables potential reconstruction from a fully data-driven standpoint using measurements of slowly phase-mixing tidal debris. △ Less

Submitted 30 March, 2023; v1 submitted 24 May, 2022; originally announced May 2022.

Comments: 32 pages, 10 figures, Accepted for publication in ApJ. Code will be made available. Comments welcome

Journal ref: Astrophysical Journal, Volume 940, Issue 22, 24 pp. (2022)

Stability Constrained Characterization of the 23 Myr-old V1298 Tau System: Do Young Planets Form in Mean Motion Resonance Chains?

Authors: Roberto Tejada Arevalo, Daniel Tamayo, Miles Cranmer

Abstract: A leading theoretical expectation for the final stages of planet formation is that disk migration should naturally drive orbits into chains of mean motion resonances (MMRs). In order to explain the dearth of MMR chains observed at Gyr ages ($<1\%$), this picture requires such configurations to destabilize and scramble period ratios following disk dispersal. Strikingly, the only two known stars wit… ▽ More A leading theoretical expectation for the final stages of planet formation is that disk migration should naturally drive orbits into chains of mean motion resonances (MMRs). In order to explain the dearth of MMR chains observed at Gyr ages ($<1\%$), this picture requires such configurations to destabilize and scramble period ratios following disk dispersal. Strikingly, the only two known stars with three or more planets younger than $\lesssim 100$ Myrs, HR 8799 and V1298 Tau, have been suggested to be in such MMR chains, given the orbits' near-integer period ratios. We incorporate recent transit and radial velocity observations of the V1298 Tau system, and investigate constraints on the system's orbital architecture imposed by requiring dynamical stability on timescales much shorter than the system's age. We show that the recent radial-velocity mass measurement of V1298 Tau $b$ places it within a factor of two of the instability limit, and that this allows us to set significantly lower limits on the eccentricity ($e_b \leq 0.17$ at $99.7\%$ confidence). Additionally, we rule out a resonant chain configuration for V1298 Tau at $\gtrsim 99\%$ confidence. Thus, if the $\sim 23$ Myr-old V1298 Tau system did form as a resonant chain, it must have undergone instability and rearrangement shortly after disk dispersal. We expect that similar stability constrained characterization of future young multi-planet systems will be valuable in informing planet formation models. △ Less

Submitted 5 March, 2022; originally announced March 2022.

Comments: Submitted, 11 pages, appendix, 3 figures

Predicting the Thermal Sunyaev-Zel'dovich Field using Modular and Equivariant Set-Based Neural Networks

Authors: Leander Thiele, Miles Cranmer, William Coulton, Shirley Ho, David N. Spergel

Abstract: Theoretical uncertainty limits our ability to extract cosmological information from baryonic fields such as the thermal Sunyaev-Zel'dovich (tSZ) effect. Being sourced by the electron pressure field, the tSZ effect depends on baryonic physics that is usually modeled by expensive hydrodynamic simulations. We train neural networks on the IllustrisTNG-300 cosmological simulation to predict the continu… ▽ More Theoretical uncertainty limits our ability to extract cosmological information from baryonic fields such as the thermal Sunyaev-Zel'dovich (tSZ) effect. Being sourced by the electron pressure field, the tSZ effect depends on baryonic physics that is usually modeled by expensive hydrodynamic simulations. We train neural networks on the IllustrisTNG-300 cosmological simulation to predict the continuous electron pressure field in galaxy clusters from gravity-only simulations. Modeling clusters is challenging for neural networks as most of the gas pressure is concentrated in a handful of voxels and even the largest hydrodynamical simulations contain only a few hundred clusters that can be used for training. Instead of conventional convolutional neural net (CNN) architectures, we choose to employ a rotationally equivariant DeepSets architecture to operate directly on the set of dark matter particles. We argue that set-based architectures provide distinct advantages over CNNs. For example, we can enforce exact rotational and permutation equivariance, incorporate existing knowledge on the tSZ field, and work with sparse fields as are standard in cosmology. We compose our architecture with separate, physically meaningful modules, making it amenable to interpretation. For example, we can separately study the influence of local and cluster-scale environment, determine that cluster triaxiality has negligible impact, and train a module that corrects for mis-centering. Our model improves by 70 % on analytic profiles fit to the same simulation data. We argue that the electron pressure field, viewed as a function of a gravity-only simulation, has inherent stochasticity, and model this property through a conditional-VAE extension to the network. This modification yields further improvement by 7 %, it is limited by our small training set however. (abridged) △ Less

Submitted 10 June, 2022; v1 submitted 28 February, 2022; originally announced March 2022.

Comments: 11 pages, 5 figures; condensed version accepted at the Fourth Workshop on Machine Learning and the Physical Sciences (NeurIPS 2021) as "Equivariant and Modular DeepSets with Applications in Cluster Cosmology"; revised version accepted by MLST includes numerous clarifications and a new appendix comparing to CNN

Rediscovering orbital mechanics with machine learning

Authors: Pablo Lemos, Niall Jeffrey, Miles Cranmer, Shirley Ho, Peter Battaglia

Abstract: We present an approach for using machine learning to automatically discover the governing equations and hidden properties of real physical systems from observations. We train a "graph neural network" to simulate the dynamics of our solar system's Sun, planets, and large moons from 30 years of trajectory data. We then use symbolic regression to discover an analytical expression for the force law im… ▽ More We present an approach for using machine learning to automatically discover the governing equations and hidden properties of real physical systems from observations. We train a "graph neural network" to simulate the dynamics of our solar system's Sun, planets, and large moons from 30 years of trajectory data. We then use symbolic regression to discover an analytical expression for the force law implicitly learned by the neural network, which our results showed is equivalent to Newton's law of gravitation. The key assumptions that were required were translational and rotational equivariance, and Newton's second and third laws of motion. Our approach correctly discovered the form of the symbolic force law. Furthermore, our approach did not require any assumptions about the masses of planets and moons or physical constants. They, too, were accurately inferred through our methods. Though, of course, the classical law of gravitation has been known since Isaac Newton, our result serves as a validation that our method can discover unknown laws and hidden properties from observed data. More broadly this work represents a key step toward realizing the potential of machine learning for accelerating scientific discovery. △ Less

Submitted 4 February, 2022; originally announced February 2022.

Comments: 12 pages, 6 figures, under review

Augmenting astrophysical scaling relations with machine learning: application to reducing the Sunyaev-Zeldovich flux-mass scatter

Authors: Digvijay Wadekar, Leander Thiele, Francisco Villaescusa-Navarro, J. Colin Hill, Miles Cranmer, David N. Spergel, Nicholas Battaglia, Daniel Anglés-Alcázar, Lars Hernquist, Shirley Ho

Abstract: Complex astrophysical systems often exhibit low-scatter relations between observable properties (e.g., luminosity, velocity dispersion, oscillation period). These scaling relations illuminate the underlying physics, and can provide observational tools for estimating masses and distances. Machine learning can provide a fast and systematic way to search for new scaling relations (or for simple exten… ▽ More Complex astrophysical systems often exhibit low-scatter relations between observable properties (e.g., luminosity, velocity dispersion, oscillation period). These scaling relations illuminate the underlying physics, and can provide observational tools for estimating masses and distances. Machine learning can provide a fast and systematic way to search for new scaling relations (or for simple extensions to existing relations) in abstract high-dimensional parameter spaces. We use a machine learning tool called symbolic regression (SR), which models patterns in a dataset in the form of analytic equations. We focus on the Sunyaev-Zeldovich flux$-$cluster mass relation ($Y_\mathrm{SZ}-M$), the scatter in which affects inference of cosmological parameters from cluster abundance data. Using SR on the data from the IllustrisTNG hydrodynamical simulation, we find a new proxy for cluster mass which combines $Y_\mathrm{SZ}$ and concentration of ionized gas ($c_\mathrm{gas}$): $M \propto Y_\mathrm{conc}^{3/5} \equiv Y_\mathrm{SZ}^{3/5} (1-A\, c_\mathrm{gas})$. $Y_\mathrm{conc}$ reduces the scatter in the predicted $M$ by $\sim 20-30$\% for large clusters ($M\gtrsim 10^{14}\, h^{-1} \, M_\odot$), as compared to using just $Y_\mathrm{SZ}$. We show that the dependence on $c_\mathrm{gas}$ is linked to cores of clusters exhibiting larger scatter than their outskirts. Finally, we test $Y_\mathrm{conc}$ on clusters from CAMELS simulations and show that $Y_\mathrm{conc}$ is robust against variations in cosmology, subgrid physics, and cosmic variance. Our results and methodology can be useful for accurate multiwavelength cluster mass estimation from upcoming CMB and X-ray surveys like ACT, SO, eROSITA and CMB-S4. △ Less

Submitted 17 March, 2023; v1 submitted 4 January, 2022; originally announced January 2022.

Comments: Version appearing in PNAS. Added additional tests but results unchanged compared to previous version. The code and data associated with this paper are available at https://github.com/JayWadekar/ScalingRelations_ML

Journal ref: PNAS March 17, 2023, 120 (12) e2202074120

Learned Coarse Models for Efficient Turbulence Simulation

Authors: Kimberly Stachenfeld, Drummond B. Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, Alvaro Sanchez-Gonzalez

Abstract: Turbulence simulation with classical numerical solvers requires high-resolution grids to accurately resolve dynamics. Here we train learned simulators at low spatial and temporal resolutions to capture turbulent dynamics generated at high resolution. We show that our proposed model can simulate turbulent dynamics more accurately than classical numerical solvers at the comparably low resolutions ac… ▽ More Turbulence simulation with classical numerical solvers requires high-resolution grids to accurately resolve dynamics. Here we train learned simulators at low spatial and temporal resolutions to capture turbulent dynamics generated at high resolution. We show that our proposed model can simulate turbulent dynamics more accurately than classical numerical solvers at the comparably low resolutions across various scientifically relevant metrics. Our model is trained end-to-end from data and is capable of learning a range of challenging chaotic and turbulent dynamics at low resolution, including trajectories generated by the state-of-the-art Athena++ engine. We show that our simpler, general-purpose architecture outperforms various more specialized, turbulence-specific architectures from the learned turbulence simulation literature. In general, we see that learned simulators yield unstable trajectories; however, we show that tuning training noise and temporal downsampling solves this problem. We also find that while generalization beyond the training distribution is a challenge for learned models, training noise, added loss constraints, and dataset augmentation can help. Broadly, we conclude that our learned simulator outperforms traditional solvers run on coarser grids, and emphasize that simple design choices can offer stability and robust generalization. △ Less

Submitted 22 April, 2022; v1 submitted 30 December, 2021; originally announced December 2021.

Journal ref: (2022) International Conference on Learning Representations

HIFlow: Generating Diverse HI Maps and Inferring Cosmology while Marginalizing over Astrophysics using Normalizing Flows

Authors: Sultan Hassan, Francisco Villaescusa-Navarro, Benjamin Wandelt, David N. Spergel, Daniel Anglés-Alcázar, Shy Genel, Miles Cranmer, Greg L. Bryan, Romeel Davé, Rachel S. Somerville, Michael Eickenberg, Desika Narayanan, Shirley Ho, Sambatra Andrianomena

Abstract: A wealth of cosmological and astrophysical information is expected from many ongoing and upcoming large-scale surveys. It is crucial to prepare for these surveys now and develop tools that can efficiently extract most information. We present HIFlow: a fast generative model of the neutral hydrogen (HI) maps that is conditioned only on cosmology ($Ω_{m}$ and $σ_{8}$) and designed using a class of no… ▽ More A wealth of cosmological and astrophysical information is expected from many ongoing and upcoming large-scale surveys. It is crucial to prepare for these surveys now and develop tools that can efficiently extract most information. We present HIFlow: a fast generative model of the neutral hydrogen (HI) maps that is conditioned only on cosmology ($Ω_{m}$ and $σ_{8}$) and designed using a class of normalizing flow models, the Masked Autoregressive Flow (MAF). HIFlow is trained on the state-of-the-art simulations from the Cosmology and Astrophysics with MachinE Learning Simulations (CAMELS) project. HIFlow has the ability to generate realistic diverse maps without explicitly incorporating the expected 2D maps structure into the flow as an inductive bias. We find that HIFlow is able to reproduce the CAMELS average and standard deviation HI power spectrum (Pk) within a factor of $\lesssim$ 2, scoring a very high $R^{2} > 90\%$. By inverting the flow, HIFlow provides a tractable high-dimensional likelihood for efficient parameter inference. We show that the conditional HIFlow on cosmology is successfully able to marginalize over astrophysics at the field level, regardless of the stellar and AGN feedback strengths. This new tool represents a first step toward a more powerful parameter inference, maximizing the scientific return of future HI surveys, and opening a new avenue to minimize the loss of complex information due to data compression down to summary statistics. △ Less

Submitted 18 August, 2022; v1 submitted 6 October, 2021; originally announced October 2021.

Comments: updated: 14 pages, 10 figures, a new section on inference has been added during revision. Accepted for publication in ApJ

Unsupervised Resource Allocation with Graph Neural Networks

Authors: Miles Cranmer, Peter Melchior, Brian Nord

Abstract: We present an approach for maximizing a global utility function by learning how to allocate resources in an unsupervised way. We expect interactions between allocation targets to be important and therefore propose to learn the reward structure for near-optimal allocation policies with a GNN. By relaxing the resource constraint, we can employ gradient-based optimization in contrast to more standard… ▽ More We present an approach for maximizing a global utility function by learning how to allocate resources in an unsupervised way. We expect interactions between allocation targets to be important and therefore propose to learn the reward structure for near-optimal allocation policies with a GNN. By relaxing the resource constraint, we can employ gradient-based optimization in contrast to more standard evolutionary algorithms. Our algorithm is motivated by a problem in modern astronomy, where one needs to select-based on limited initial information-among $10^9$ galaxies those whose detailed measurement will lead to optimal inference of the composition of the universe. Our technique presents a way of flexibly learning an allocation strategy by only requiring forward simulators for the physics of interest and the measurement process. We anticipate that our technique will also find applications in a range of resource allocation problems. △ Less

Submitted 17 June, 2021; originally announced June 2021.

Comments: Accepted to PMLR/contributed oral at NeurIPS 2020 Pre-registration Workshop. Code at https://github.com/MilesCranmer/gnn_resource_allocation

A Deep Learning Approach for Active Anomaly Detection of Extragalactic Transients

Authors: V. Ashley Villar, Miles Cranmer, Edo Berger, Gabriella Contardo, Shirley Ho, Griffin Hosseinzadeh, Joshua Yao-Yu Lin

Abstract: There is a shortage of multi-wavelength and spectroscopic followup capabilities given the number of transient and variable astrophysical events discovered through wide-field, optical surveys such as the upcoming Vera C. Rubin Observatory. From the haystack of potential science targets, astronomers must allocate scarce resources to study a selection of needles in real time. Here we present a variat… ▽ More There is a shortage of multi-wavelength and spectroscopic followup capabilities given the number of transient and variable astrophysical events discovered through wide-field, optical surveys such as the upcoming Vera C. Rubin Observatory. From the haystack of potential science targets, astronomers must allocate scarce resources to study a selection of needles in real time. Here we present a variational recurrent autoencoder neural network to encode simulated Rubin Observatory extragalactic transient events using 1% of the PLAsTiCC dataset to train the autoencoder. Our unsupervised method uniquely works with unlabeled, real time, multivariate and aperiodic data. We rank 1,129,184 events based on an anomaly score estimated using an isolation forest. We find that our pipeline successfully ranks rarer classes of transients as more anomalous. Using simple cuts in anomaly score and uncertainty, we identify a pure (~95% pure) sample of rare transients (i.e., transients other than Type Ia, Type II and Type Ibc supernovae) including superluminous and pair-instability supernovae. Finally, our algorithm is able to identify these transients as anomalous well before peak, enabling real-time follow up studies in the era of the Rubin Observatory. △ Less

Submitted 22 March, 2021; originally announced March 2021.

Comments: 13 pages,9 figures, submitted to AAS Journals. Comments welcome!

A Bayesian neural network predicts the dissolution of compact planetary systems

Authors: Miles Cranmer, Daniel Tamayo, Hanno Rein, Peter Battaglia, Samuel Hadden, Philip J. Armitage, Shirley Ho, David N. Spergel

Abstract: Despite over three hundred years of effort, no solutions exist for predicting when a general planetary configuration will become unstable. We introduce a deep learning architecture to push forward this problem for compact systems. While current machine learning algorithms in this area rely on scientist-derived instability metrics, our new technique learns its own metrics from scratch, enabled by a… ▽ More Despite over three hundred years of effort, no solutions exist for predicting when a general planetary configuration will become unstable. We introduce a deep learning architecture to push forward this problem for compact systems. While current machine learning algorithms in this area rely on scientist-derived instability metrics, our new technique learns its own metrics from scratch, enabled by a novel internal structure inspired from dynamics theory. Our Bayesian neural network model can accurately predict not only if, but also when a compact planetary system with three or more planets will go unstable. Our model, trained directly from short N-body time series of raw orbital elements, is more than two orders of magnitude more accurate at predicting instability times than analytical estimators, while also reducing the bias of existing machine learning algorithms by nearly a factor of three. Despite being trained on compact resonant and near-resonant three-planet configurations, the model demonstrates robust generalization to both non-resonant and higher multiplicity configurations, in the latter case outperforming models fit to that specific set of integrations. The model computes instability estimates up to five orders of magnitude faster than a numerical integrator, and unlike previous efforts provides confidence intervals on its predictions. Our inference model is publicly available in the SPOCK package, with training code open-sourced. △ Less

Submitted 11 January, 2021; originally announced January 2021.

Comments: 8 content pages, 7 appendix and references. 8 figures. Source code at: https://github.com/MilesCranmer/bnn_chaos_model inference code at https://github.com/dtamayo/spock

Anomaly Detection for Multivariate Time Series of Exotic Supernovae

Authors: V. Ashley Villar, Miles Cranmer, Gabriella Contardo, Shirley Ho, Joshua Yao-Yu Lin

Abstract: Supernovae mark the explosive deaths of stars and enrich the cosmos with heavy elements. Future telescopes will discover thousands of new supernovae nightly, creating a need to flag astrophysically interesting events rapidly for followup study. Ideally, such an anomaly detection pipeline would be independent of our current knowledge and be sensitive to unexpected phenomena. Here we present an unsu… ▽ More Supernovae mark the explosive deaths of stars and enrich the cosmos with heavy elements. Future telescopes will discover thousands of new supernovae nightly, creating a need to flag astrophysically interesting events rapidly for followup study. Ideally, such an anomaly detection pipeline would be independent of our current knowledge and be sensitive to unexpected phenomena. Here we present an unsupervised method to search for anomalous time series in real time for transient, multivariate, and aperiodic signals. We use a RNN-based variational autoencoder to encode supernova time series and an isolation forest to search for anomalous events in the learned encoded space. We apply this method to a simulated dataset of 12,159 supernovae, successfully discovering anomalous supernovae and objects with catastrophically incorrect redshift measurements. This work is the first anomaly detection pipeline for supernovae which works with online datastreams. △ Less

Submitted 21 October, 2020; originally announced October 2020.

Comments: 6 pages, 2 figures, written for non-astronomers, submitted to the NeurIPS workshop Machine Learning and the Physical Sciences. Comments welcome!!

Predicting the long-term stability of compact multiplanet systems

Authors: Daniel Tamayo, Miles Cranmer, Samuel Hadden, Hanno Rein, Peter Battaglia, Alysa Obertas, Philip J. Armitage, Shirley Ho, David Spergel, Christian Gilbertson, Naireen Hussain, Ari Silburt, Daniel Jontof-Hutter, Kristen Menou

Abstract: We combine analytical understanding of resonant dynamics in two-planet systems with machine learning techniques to train a model capable of robustly classifying stability in compact multi-planet systems over long timescales of $10^9$ orbits. Our Stability of Planetary Orbital Configurations Klassifier (SPOCK) predicts stability using physically motivated summary statistics measured in integrations… ▽ More We combine analytical understanding of resonant dynamics in two-planet systems with machine learning techniques to train a model capable of robustly classifying stability in compact multi-planet systems over long timescales of $10^9$ orbits. Our Stability of Planetary Orbital Configurations Klassifier (SPOCK) predicts stability using physically motivated summary statistics measured in integrations of the first $10^4$ orbits, thus achieving speed-ups of up to $10^5$ over full simulations. This computationally opens up the stability constrained characterization of multi-planet systems. Our model, trained on $\approx 100,000$ three-planet systems sampled at discrete resonances, generalizes both to a sample spanning a continuous period-ratio range, as well as to a large five-planet sample with qualitatively different configurations to our training dataset. Our approach significantly outperforms previous methods based on systems' angular momentum deficit, chaos indicators, and parametrized fits to numerical integrations. We use SPOCK to constrain the free eccentricities between the inner and outer pairs of planets in the Kepler-431 system of three approximately Earth-sized planets to both be below 0.05. Our stability analysis provides significantly stronger eccentricity constraints than currently achievable through either radial velocity or transit duration measurements for small planets, and within a factor of a few of systems that exhibit transit timing variations (TTVs). Given that current exoplanet detection strategies now rarely allow for strong TTV constraints (Hadden et al., 2019), SPOCK enables a powerful complementary method for precisely characterizing compact multi-planet systems. We publicly release SPOCK for community use. △ Less

Submitted 14 July, 2020; v1 submitted 13 July, 2020; originally announced July 2020.

Comments: Published week of July 13th in Proceedings of the National Academy of Sciences: https://www.pnas.org/cgi/doi/10.1073/pnas.2001258117. Check out simple usage of package (and regenerate paper figures) at: https://github.com/dtamayo/spock

Meta-Learning for One-Class Classification with Few Examples using Order-Equivariant Network

Authors: Ademola Oladosu, Tony Xu, Philip Ekfeldt, Brian A. Kelly, Miles Cranmer, Shirley Ho, Adrian M. Price-Whelan, Gabriella Contardo

Abstract: This paper presents a meta-learning framework for few-shots One-Class Classification (OCC) at test-time, a setting where labeled examples are only available for the positive class, and no supervision is given for the negative example. We consider that we have a set of `one-class classification' objective-tasks with only a small set of positive examples available for each task, and a set of trainin… ▽ More This paper presents a meta-learning framework for few-shots One-Class Classification (OCC) at test-time, a setting where labeled examples are only available for the positive class, and no supervision is given for the negative example. We consider that we have a set of `one-class classification' objective-tasks with only a small set of positive examples available for each task, and a set of training tasks with full supervision (i.e. highly imbalanced classification). We propose an approach using order-equivariant networks to learn a 'meta' binary-classifier. The model will take as input an example to classify from a given task, as well as the corresponding supervised set of positive examples for this OCC task. Thus, the output of the model will be 'conditioned' on the available positive example of a given task, allowing to predict on new tasks and new examples without labeled negative examples. In this paper, we are motivated by an astronomy application. Our goal is to identify if stars belong to a specific stellar group (the 'one-class' for a given task), called \textit{stellar streams}, where each stellar stream is a different OCC-task. We show that our method transfers well on unseen (test) synthetic streams, and outperforms the baselines even though it is not retrained and accesses a much smaller part of the data per task to predict (only positive supervision). We see however that it doesn't transfer as well on the real stream GD-1. This could come from intrinsic differences from the synthetic and real stream, highlighting the need for consistency in the 'nature' of the task for this method. However, light fine-tuning improve performances and outperform our baselines. Our experiments show encouraging results to further explore meta-learning methods for OCC tasks. △ Less

Submitted 21 May, 2021; v1 submitted 8 July, 2020; originally announced July 2020.

Discovering Symbolic Models from Deep Learning with Inductive Biases

Authors: Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, Shirley Ho

Abstract: We develop a general approach to distill symbolic representations of a learned deep model by introducing strong inductive biases. We focus on Graph Neural Networks (GNNs). The technique works as follows: we first encourage sparse latent representations when we train a GNN in a supervised setting, then we apply symbolic regression to components of the learned model to extract explicit physical rela… ▽ More We develop a general approach to distill symbolic representations of a learned deep model by introducing strong inductive biases. We focus on Graph Neural Networks (GNNs). The technique works as follows: we first encourage sparse latent representations when we train a GNN in a supervised setting, then we apply symbolic regression to components of the learned model to extract explicit physical relations. We find the correct known equations, including force laws and Hamiltonians, can be extracted from the neural network. We then apply our method to a non-trivial cosmology example-a detailed dark matter simulation-and discover a new analytic formula which can predict the concentration of dark matter from the mass distribution of nearby cosmic structures. The symbolic expressions extracted from the GNN using our technique also generalized to out-of-distribution data better than the GNN itself. Our approach offers alternative directions for interpreting neural networks and discovering novel physical principles from the representations they learn. △ Less

Submitted 17 November, 2020; v1 submitted 19 June, 2020; originally announced June 2020.

Comments: Accepted to NeurIPS 2020. 9 pages content + 16 pages appendix/references. Supporting code found at https://github.com/MilesCranmer/symbolic_deep_learning

Lagrangian Neural Networks

Authors: Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, Shirley Ho

Abstract: Accurate models of the world are built upon notions of its underlying symmetries. In physics, these symmetries correspond to conservation laws, such as for energy and momentum. Yet even though neural network models see increasing use in the physical sciences, they struggle to learn these symmetries. In this paper, we propose Lagrangian Neural Networks (LNNs), which can parameterize arbitrary Lagra… ▽ More Accurate models of the world are built upon notions of its underlying symmetries. In physics, these symmetries correspond to conservation laws, such as for energy and momentum. Yet even though neural network models see increasing use in the physical sciences, they struggle to learn these symmetries. In this paper, we propose Lagrangian Neural Networks (LNNs), which can parameterize arbitrary Lagrangians using neural networks. In contrast to models that learn Hamiltonians, LNNs do not require canonical coordinates, and thus perform well in situations where canonical momenta are unknown or difficult to compute. Unlike previous approaches, our method does not restrict the functional form of learned energies and will produce energy-conserving models for a variety of tasks. We test our approach on a double pendulum and a relativistic particle, demonstrating energy conservation where a baseline approach incurs dissipation and modeling relativity without canonical coordinates where a Hamiltonian approach fails. Finally, we show how this model can be applied to graphs and continuous systems using a Lagrangian Graph Network, and demonstrate it on the 1D wave equation. △ Less

Submitted 30 July, 2020; v1 submitted 10 March, 2020; originally announced March 2020.

Comments: 7 pages (+2 appendix). Published in ICLR 2020 Deep Differential Equations Workshop. Code at github.com/MilesCranmer/lagrangian_nns

Learning Symbolic Physics with Graph Networks

Authors: Miles D. Cranmer, Rui Xu, Peter Battaglia, Shirley Ho

Abstract: We introduce an approach for imposing physically motivated inductive biases on graph networks to learn interpretable representations and improved zero-shot generalization. Our experiments show that our graph network models, which implement this inductive bias, can learn message representations equivalent to the true force vector when trained on n-body gravitational and spring-like simulations. We… ▽ More We introduce an approach for imposing physically motivated inductive biases on graph networks to learn interpretable representations and improved zero-shot generalization. Our experiments show that our graph network models, which implement this inductive bias, can learn message representations equivalent to the true force vector when trained on n-body gravitational and spring-like simulations. We use symbolic regression to fit explicit algebraic equations to our trained model's message function and recover the symbolic form of Newton's law of gravitation without prior knowledge. We also show that our model generalizes better at inference time to systems with more bodies than had been experienced during training. Our approach is extensible, in principle, to any unknown interaction law learned by a graph network, and offers a valuable technique for interpreting and inferring explicit causal theories about the world from implicit knowledge captured by deep learning. △ Less

Submitted 1 November, 2019; v1 submitted 12 September, 2019; originally announced September 2019.

Comments: 6 pages; references added + improvements to writing and clarity; accepted for an oral presentation at Machine Learning and the Physical Sciences Workshop @ NeurIPS 2019

Modeling the Gaia Color-Magnitude Diagram with Bayesian Neural Flows to Constrain Distance Estimates

Authors: Miles D. Cranmer, Richard Galvez, Lauren Anderson, David N. Spergel, Shirley Ho

Abstract: We demonstrate an algorithm for learning a flexible color-magnitude diagram from noisy parallax and photometry measurements using a normalizing flow, a deep neural network capable of learning an arbitrary multi-dimensional probability distribution. We present a catalog of 640M photometric distance posteriors to nearby stars derived from this data-driven model using Gaia DR2 photometry and parallax… ▽ More We demonstrate an algorithm for learning a flexible color-magnitude diagram from noisy parallax and photometry measurements using a normalizing flow, a deep neural network capable of learning an arbitrary multi-dimensional probability distribution. We present a catalog of 640M photometric distance posteriors to nearby stars derived from this data-driven model using Gaia DR2 photometry and parallaxes. Dust estimation and dereddening is done iteratively inside the model and without prior distance information, using the Bayestar map. The signal-to-noise (precision) of distance measurements improves on average by more than 48% over the raw Gaia data, and we also demonstrate how the accuracy of distances have improved over other models, especially in the noisy-parallax regime. Applications are discussed, including significantly improved Milky Way disk separation and substructure detection. We conclude with a discussion of future work, which exploits the normalizing flow architecture to allow us to exactly marginalize over missing photometry, enabling the inclusion of many surveys without losing coverage. △ Less

Submitted 21 August, 2019; originally announced August 2019.

Comments: 15 pages, 8 figures

The CHIME Fast Radio Burst Project: System Overview

Authors: The CHIME/FRB Collaboration, :, M. Amiri, K. Bandura, P. Berger, M. Bhardwaj, M. M. Boyce, P. J. Boyle, C. Brar, M. Burhanpurkar, P. Chawla, J. Chowdhury, J. F. Cliche, M. D. Cranmer, D. Cubranic, M. Deng, N. Denman, M. Dobbs, M. Fandino, E. Fonseca, B. M. Gaensler, U. Giri, A. J. Gilbert, D. C. Good, S. Guliani , et al. (28 additional authors not shown)

Abstract: The Canadian Hydrogen Intensity Map** Experiment (CHIME) is a novel transit radio telescope operating across the 400-800-MHz band. CHIME is comprised of four 20-m x 100-m semi-cylindrical paraboloid reflectors, each of which has 256 dual-polarization feeds suspended along its axis, giving it a >200 square degree field-of-view. This, combined with wide bandwidth, high sensitivity, and a powerful… ▽ More The Canadian Hydrogen Intensity Map** Experiment (CHIME) is a novel transit radio telescope operating across the 400-800-MHz band. CHIME is comprised of four 20-m x 100-m semi-cylindrical paraboloid reflectors, each of which has 256 dual-polarization feeds suspended along its axis, giving it a >200 square degree field-of-view. This, combined with wide bandwidth, high sensitivity, and a powerful correlator makes CHIME an excellent instrument for the detection of Fast Radio Bursts (FRBs). The CHIME Fast Radio Burst Project (CHIME/FRB) will search beam-formed, high time-and frequency-resolution data in real time for FRBs in the CHIME field-of-view. Here we describe the CHIME/FRB backend, including the real-time FRB search and detection software pipeline as well as the planned offline analyses. We estimate a CHIME/FRB detection rate of 2-42 FRBs/sky/day normalizing to the rate estimated at 1.4-GHz by Vander Wiel et al. (2016). Likely science outcomes of CHIME/FRB are also discussed. CHIME/FRB is currently operational in a commissioning phase, with science operations expected to commence in the latter half of 2018. △ Less

Submitted 29 March, 2018; originally announced March 2018.

Comments: 27 pages, submitted to ApJ

Bifrost: a Python/C++ Framework for High-Throughput Stream Processing in Astronomy

Authors: Miles D. Cranmer, Benjamin R. Barsdell, Danny C. Price, Jayce Dowell, Hugh Garsden, Veronica Dike, Tarraneh Eftekhari, Alexander M. Hegedus, Joseph Malins, Kenneth S. Obenberger, Frank Schinzel, Kevin Stovall, Gregory B. Taylor, Lincoln J. Greenhill

Abstract: Radio astronomy observatories with high throughput back end instruments require real-time data processing. While computing hardware continues to advance rapidly, development of real-time processing pipelines remains difficult and time-consuming, which can limit scientific productivity. Motivated by this, we have developed Bifrost: an open-source software framework for rapid pipeline development. B… ▽ More Radio astronomy observatories with high throughput back end instruments require real-time data processing. While computing hardware continues to advance rapidly, development of real-time processing pipelines remains difficult and time-consuming, which can limit scientific productivity. Motivated by this, we have developed Bifrost: an open-source software framework for rapid pipeline development. Bifrost combines a high-level Python interface with highly efficient reconfigurable data transport and a library of computing blocks for CPU and GPU processing. The framework is generalizable, but initially it emphasizes the needs of high-throughput radio astronomy pipelines, such as the ability to process data buffers as if they were continuous streams, the capacity to partition processing into distinct data sequences (e.g., separate observations), and the ability to extract specific intervals from buffered data. Computing blocks in the library are designed for applications such as interferometry, pulsar dedispersion and timing, and transient search pipelines. We describe the design and implementation of the Bifrost framework and demonstrate its use as the backbone in the correlation and beamforming back end of the Long Wavelength Array station in the Sevilleta National Wildlife Refuge, NM. △ Less

Submitted 2 August, 2017; originally announced August 2017.

Comments: 25 pages, 13 figures, submitted to JAI. For the code, see https://github.com/ledatelescope/bifrost

Mitigating radiation damage of single photon detectors for space applications

Authors: Elena Anisimova, Brendon L. Higgins, Jean-Philippe Bourgoin, Miles Cranmer, Eric Choi, Danya Hudson, Louis P. Piche, Alan Scott, Vadim Makarov, Thomas Jennewein

Abstract: Single-photon detectors in space must retain useful performance characteristics despite being bombarded with sub-atomic particles. Mitigating the effects of this space radiation is vital to enabling new space applications which require high-fidelity single-photon detection. To this end, we conducted proton radiation tests of various models of avalanche photodiodes (APDs) and one model of photomult… ▽ More Single-photon detectors in space must retain useful performance characteristics despite being bombarded with sub-atomic particles. Mitigating the effects of this space radiation is vital to enabling new space applications which require high-fidelity single-photon detection. To this end, we conducted proton radiation tests of various models of avalanche photodiodes (APDs) and one model of photomultiplier tube potentially suitable for satellite-based quantum communications. The samples were irradiated with 106 MeV protons at doses approximately equivalent to lifetimes of 0.6 , 6, 12 and 24 months in a low-Earth polar orbit. Although most detection properties were preserved, including efficiency, timing jitter and afterpulsing probability, all APD samples demonstrated significant increases in dark count rate (DCR) due to radiation-induced damage, many orders of magnitude higher than the 200 counts per second (cps) required for ground-to-satellite quantum communications. We then successfully demonstrated the mitigation of this DCR degradation through the use of deep cooling, to as low as -86 degrees C. This achieved DCR below the required 200 cps over the 24 months orbit duration. DCR was further reduced by thermal annealing at temperatures of +50 to +100 degrees C. △ Less

Submitted 12 October, 2017; v1 submitted 3 February, 2017; originally announced February 2017.

Comments: The license has been corrected. Note that the license of v2 was incorrect and not valid. No other changes since v2

MSC Class: 81-05

Journal ref: EPJ Quantum Technol. 4, 10 (2017)

Free-space quantum key distribution to a moving receiver

Authors: Jean-Philippe Bourgoin, Brendon L. Higgins, Nick Gigov, Catherine Holloway, Christopher J. Pugh, Sarah Kaiser, Miles Cranmer, Thomas Jennewein

Abstract: Technological realities limit terrestrial quantum key distribution (QKD) to single-link distances of a few hundred kilometers. One promising avenue for global-scale quantum communication networks is to use low-Earth-orbit satellites. Here we report the first demonstration of QKD from a stationary transmitter to a receiver platform traveling at an angular speed equivalent to a 600 km altitude satel… ▽ More Technological realities limit terrestrial quantum key distribution (QKD) to single-link distances of a few hundred kilometers. One promising avenue for global-scale quantum communication networks is to use low-Earth-orbit satellites. Here we report the first demonstration of QKD from a stationary transmitter to a receiver platform traveling at an angular speed equivalent to a 600 km altitude satellite, located on a moving truck. We overcome the challenges of actively correcting beam pointing, photon polarization and time-of-flight. Our system generates an asymptotic secure key at 40 bits/s. △ Less

Submitted 17 December, 2015; v1 submitted 1 May, 2015; originally announced May 2015.

Comments: 7 pages, 7 figures, 3 tables

Journal ref: Opt. Express 23, 33437 (2015)