-
Classification under Nuisance Parameters and Generalized Label Shift in Likelihood-Free Inference
Authors:
Luca Masserano,
Alex Shen,
Michele Doro,
Tommaso Dorigo,
Rafael Izbicki,
Ann B. Lee
Abstract:
An open scientific challenge is how to classify events with reliable measures of uncertainty, when we have a mechanistic model of the data-generating process but the distribution over both labels and latent nuisance parameters is different between train and target data. We refer to this type of distributional shift as generalized label shift (GLS). Direct classification using observed data…
▽ More
An open scientific challenge is how to classify events with reliable measures of uncertainty, when we have a mechanistic model of the data-generating process but the distribution over both labels and latent nuisance parameters is different between train and target data. We refer to this type of distributional shift as generalized label shift (GLS). Direct classification using observed data $\mathbf{X}$ as covariates leads to biased predictions and invalid uncertainty estimates of labels $Y$. We overcome these biases by proposing a new method for robust uncertainty quantification that casts classification as a hypothesis testing problem under nuisance parameters. The key idea is to estimate the classifier's receiver operating characteristic (ROC) across the entire nuisance parameter space, which allows us to devise cutoffs that are invariant under GLS. Our method effectively endows a pre-trained classifier with domain adaptation capabilities and returns valid prediction sets while maintaining high power. We demonstrate its performance on two challenging scientific problems in biology and astroparticle physics with data from realistic mechanistic models.
△ Less
Submitted 1 July, 2024; v1 submitted 7 February, 2024;
originally announced February 2024.
-
End-To-End Optimization of the Layout of a Gamma Ray Observatory
Authors:
Tommaso Dorigo,
Max Aehle,
Julien Donini,
Michele Doro,
Nicolas R. Gauger,
Rafael Izbicki,
Ann Lee,
Luca Masserano,
Federico Nardi,
Sidharth S S,
Alexander Shen
Abstract:
In this document we describe a model of an array of water Cherenkov detectors proposed to study ultra-high-energy gamma rays in the southern hemisphere, and a continuous model of secondary particles produced on the ground from gamma and proton showers. We use the model of the detector and the parametrization of showers for the identification of the most promising configuration of detector elements…
▽ More
In this document we describe a model of an array of water Cherenkov detectors proposed to study ultra-high-energy gamma rays in the southern hemisphere, and a continuous model of secondary particles produced on the ground from gamma and proton showers. We use the model of the detector and the parametrization of showers for the identification of the most promising configuration of detector elements, using a likelihood ratio test statistic to classify showers and a stochastic gradient descent technique to maximize a utility function describing the measurement precision on the gamma-ray flux.
△ Less
Submitted 3 October, 2023;
originally announced October 2023.
-
Adaptive Sampling for Probabilistic Forecasting under Distribution Shift
Authors:
Luca Masserano,
Syama Sundar Rangapuram,
Shubham Kapoor,
Rajbir Singh Nirwan,
Youngsuk Park,
Michael Bohlke-Schneider
Abstract:
The world is not static: This causes real-world time series to change over time through external, and potentially disruptive, events such as macroeconomic cycles or the COVID-19 pandemic. We present an adaptive sampling strategy that selects the part of the time series history that is relevant for forecasting. We achieve this by learning a discrete distribution over relevant time steps by Bayesian…
▽ More
The world is not static: This causes real-world time series to change over time through external, and potentially disruptive, events such as macroeconomic cycles or the COVID-19 pandemic. We present an adaptive sampling strategy that selects the part of the time series history that is relevant for forecasting. We achieve this by learning a discrete distribution over relevant time steps by Bayesian optimization. We instantiate this idea with a two-step method that is pre-trained with uniform sampling and then training a lightweight adaptive architecture with adaptive sampling. We show with synthetic and real-world experiments that this method adapts to distribution shift and significantly reduces the forecasting error of the base model for three out of five datasets.
△ Less
Submitted 23 February, 2023;
originally announced February 2023.
-
Simulator-Based Inference with Waldo: Confidence Regions by Leveraging Prediction Algorithms and Posterior Estimators for Inverse Problems
Authors:
Luca Masserano,
Tommaso Dorigo,
Rafael Izbicki,
Mikael Kuusela,
Ann B. Lee
Abstract:
Prediction algorithms, such as deep neural networks (DNNs), are used in many domain sciences to directly estimate internal parameters of interest in simulator-based models, especially in settings where the observations include images or complex high-dimensional data. In parallel, modern neural density estimators, such as normalizing flows, are becoming increasingly popular for uncertainty quantifi…
▽ More
Prediction algorithms, such as deep neural networks (DNNs), are used in many domain sciences to directly estimate internal parameters of interest in simulator-based models, especially in settings where the observations include images or complex high-dimensional data. In parallel, modern neural density estimators, such as normalizing flows, are becoming increasingly popular for uncertainty quantification, especially when both parameters and observations are high-dimensional. However, parameter inference is an inverse problem and not a prediction task; thus, an open challenge is to construct conditionally valid and precise confidence regions, with a guaranteed probability of covering the true parameters of the data-generating process, no matter what the (unknown) parameter values are, and without relying on large-sample theory. Many simulator-based inference (SBI) methods are indeed known to produce biased or overly confident parameter regions, yielding misleading uncertainty estimates. This paper presents WALDO, a novel method to construct confidence regions with finite-sample conditional validity by leveraging prediction algorithms or posterior estimators that are currently widely adopted in SBI. WALDO reframes the well-known Wald test statistic, and uses a computationally efficient regression-based machinery for classical Neyman inversion of hypothesis tests. We apply our method to a recent high-energy physics problem, where prediction with DNNs has previously led to estimates with prediction bias. We also illustrate how our approach can correct overly confident posterior regions computed with normalizing flows.
△ Less
Submitted 13 November, 2023; v1 submitted 31 May, 2022;
originally announced May 2022.
-
Likelihood-Free Frequentist Inference: Bridging Classical Statistics and Machine Learning for Reliable Simulator-Based Inference
Authors:
Niccolò Dalmasso,
Luca Masserano,
David Zhao,
Rafael Izbicki,
Ann B. Lee
Abstract:
Many areas of science make extensive use of computer simulators that implicitly encode intractable likelihood functions of complex systems. Classical statistical methods are poorly suited for these so-called likelihood-free inference (LFI) settings, especially outside asymptotic and low-dimensional regimes. At the same time, traditional LFI methods - such as Approximate Bayesian Computation or mor…
▽ More
Many areas of science make extensive use of computer simulators that implicitly encode intractable likelihood functions of complex systems. Classical statistical methods are poorly suited for these so-called likelihood-free inference (LFI) settings, especially outside asymptotic and low-dimensional regimes. At the same time, traditional LFI methods - such as Approximate Bayesian Computation or more recent machine learning techniques - do not guarantee confidence sets with nominal coverage in general settings (i.e., with high-dimensional data, finite sample sizes, and for any parameter value). In addition, there are no diagnostic tools to check the empirical coverage of confidence sets provided by such methods across the entire parameter space. In this work, we propose a unified and modular inference framework that bridges classical statistics and modern machine learning providing (i) a practical approach to the Neyman construction of confidence sets with frequentist finite-sample coverage for any value of the unknown parameters; and (ii) interpretable diagnostics that estimate the empirical coverage across the entire parameter space. We refer to the general framework as likelihood-free frequentist inference (LF2I). Any method that defines a test statistic can leverage LF2I to create valid confidence sets and diagnostics without costly Monte Carlo samples at fixed parameter settings. We study the power of two likelihood-based test statistics (ACORE and BFF) and demonstrate their empirical performance on high-dimensional, complex data. Code is available at https://github.com/lee-group-cmu/lf2i.
△ Less
Submitted 19 November, 2023; v1 submitted 8 July, 2021;
originally announced July 2021.
