-
Computationally Efficient and Error Aware Surrogate Construction for Numerical Solutions of Subsurface Flow Through Porous Media
Authors:
Aleksei G. Sorokin,
Aleksandra Pachalieva,
Daniel O'Malley,
James M. Hyman,
Fred J. Hickernell,
Nicolas W. Hengartner
Abstract:
Limiting the injection rate to restrict the pressure below a threshold at a critical location can be an important goal of simulations that model the subsurface pressure between injection and extraction wells. The pressure is approximated by the solution of Darcy's partial differential equation (PDE) for a given permeability field. The subsurface permeability is modeled as a random field since it i…
▽ More
Limiting the injection rate to restrict the pressure below a threshold at a critical location can be an important goal of simulations that model the subsurface pressure between injection and extraction wells. The pressure is approximated by the solution of Darcy's partial differential equation (PDE) for a given permeability field. The subsurface permeability is modeled as a random field since it is known only up to statistical properties. This induces uncertainty in the computed pressure. Solving the PDE for an ensemble of random permeability simulations enables estimating a probability distribution for the pressure at the critical location. These simulations are computationally expensive, and practitioners often need rapid online guidance for real-time pressure management. An ensemble of numerical PDE solutions is used to construct a Gaussian process regression model that can quickly predict the pressure at the critical location as a function of the extraction rate and permeability realization.
Our first novel contribution is to identify a sampling methodology for the random environment and matching kernel technology for which fitting the Gaussian process regression model scales as O(n log n) instead of the typical O(n^3) rate in the number of samples n used to fit the surrogate. The surrogate model allows almost instantaneous predictions for the pressure at the critical location as a function of the extraction rate and permeability realization. Our second contribution is a novel algorithm to calibrate the uncertainty in the surrogate model to the discrepancy between the true pressure solution of Darcy's equation and the numerical solution. Although our method is derived for building a surrogate for the solution of Darcy's equation with a random permeability field, the framework broadly applies to solutions of other PDE with random coefficients.
△ Less
Submitted 20 October, 2023;
originally announced October 2023.
-
A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks
Authors:
Xiaoyu Ma,
Sylvain Sardy,
Nick Hengartner,
Nikolai Bobenko,
Yen Ting Lin
Abstract:
To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample size is smaller than the dimension of the input vector (haystack). More recently learners known as artificial neural networks (ANN) have shown great successes in many machine learnin…
▽ More
To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample size is smaller than the dimension of the input vector (haystack). More recently learners known as artificial neural networks (ANN) have shown great successes in many machine learning tasks, in particular fitting nonlinear associations. Small learning rate, stochastic gradient descent algorithm and large training set help to cope with the explosion in the number of parameters present in deep neural networks. Yet few ANN learners have been developed and studied to find needles in nonlinear haystacks. Driven by a single hyperparameter, our ANN learner, like for sparse linear associations, exhibits a phase transition in the probability of retrieving the needles, which we do not observe with other ANN learners. To select our penalty parameter, we generalize the universal threshold of Donoho and Johnstone (1994) which is a better rule than the conservative (too many false detections) and expensive cross-validation. In the spirit of simulated annealing, we propose a warm-start sparsity inducing algorithm to solve the high-dimensional, non-convex and non-differentiable optimization problem. We perform precise Monte Carlo simulations to show the effectiveness of our approach.
△ Less
Submitted 21 January, 2022;
originally announced January 2022.
-
A modified Susceptible-Infected-Recovered model for observed under-reported incidence data
Authors:
Imelda Trejo,
Nicolas Hengartner
Abstract:
Fitting Susceptible-Infected-Recovered (SIR) models to incidence data is problematic when not all infected individuals are reported. Assuming an underlying SIR model with general but known distribution for the time to recovery, this paper derives the implied differential-integral equations for observed incidence data when a fixed fraction of newly infected individuals are not observed. The paramet…
▽ More
Fitting Susceptible-Infected-Recovered (SIR) models to incidence data is problematic when not all infected individuals are reported. Assuming an underlying SIR model with general but known distribution for the time to recovery, this paper derives the implied differential-integral equations for observed incidence data when a fixed fraction of newly infected individuals are not observed. The parameters of the resulting system of differential equations are identifiable. Using these differential equations, we develop a stochastic model for the conditional distribution of current disease incidence given the entire past history of reported cases. We estimate the model parameters using Bayesian Markov Chain Monte-Carlo sampling of the posterior distribution. We use our model to estimate the transmission rate and fraction of asymptomatic individuals for the current Coronavirus 2019 outbreak in eight American Countries: the United States of America, Brazil, Mexico, Argentina, Chile, Colombia, Peru, and Panama, from January 2020 to May 2021. Our analysis reveals that consistently, about 40-60% of the infections were not observed in the American outbreaks. The two exception are Mexico and Peru, with acute under-reporting in Mexico.
△ Less
Submitted 12 August, 2021; v1 submitted 9 December, 2020;
originally announced December 2020.
-
A Note on Using Discretized Simulated Data to Estimate Implicit Likelihoods in Bayesian Analyses
Authors:
M. S. Hamada,
T. L. Graves,
N. W. Hengartner,
D. M. Higdon,
A. V. Huzurbazar,
E. C. Lawrence,
C. D. Linkletter,
C. S. Reese,
D. W. Scott,
R. R. Sitter,
R. L. Warr,
B. J. Williams
Abstract:
This article presents a Bayesian inferential method where the likelihood for a model is unknown but where data can easily be simulated from the model. We discretize simulated (continuous) data to estimate the implicit likelihood in a Bayesian analysis employing a Markov chain Monte Carlo algorithm. Three examples are presented as well as a small study on some of the method's properties.
This article presents a Bayesian inferential method where the likelihood for a model is unknown but where data can easily be simulated from the model. We discretize simulated (continuous) data to estimate the implicit likelihood in a Bayesian analysis employing a Markov chain Monte Carlo algorithm. Three examples are presented as well as a small study on some of the method's properties.
△ Less
Submitted 6 August, 2020;
originally announced August 2020.
-
What needles do sparse neural networks find in nonlinear haystacks
Authors:
Sylvain Sardy,
Nicolas W Hengartner,
Nikolai Bonenko,
Yen Ting Lin
Abstract:
Using a sparsity inducing penalty in artificial neural networks (ANNs) avoids over-fitting, especially in situations where noise is high and the training set is small in comparison to the number of features. For linear models, such an approach provably also recovers the important features with high probability in regimes for a well-chosen penalty parameter. The typical way of setting the penalty p…
▽ More
Using a sparsity inducing penalty in artificial neural networks (ANNs) avoids over-fitting, especially in situations where noise is high and the training set is small in comparison to the number of features. For linear models, such an approach provably also recovers the important features with high probability in regimes for a well-chosen penalty parameter. The typical way of setting the penalty parameter is by splitting the data set and performing the cross-validation, which is (1) computationally expensive and (2) not desirable when the data set is already small to be further split (for example, whole-genome sequence data). In this study, we establish the theoretical foundation to select the penalty parameter without cross-validation based on bounding with a high probability the infinite norm of the gradient of the loss function at zero under the zero-feature assumption. Our approach is a generalization of the universal threshold of Donoho and Johnstone (1994) to nonlinear ANN learning. We perform a set of comprehensive Monte Carlo simulations on a simple model, and the numerical results show the effectiveness of the proposed approach.
△ Less
Submitted 7 June, 2020;
originally announced June 2020.
-
Quantile universal threshold for model selection
Authors:
Caroline Giacobino,
Sylvain Sardy,
Jairo Diaz-Rodriguez,
Nick Hengartner
Abstract:
Efficient recovery of a low-dimensional structure from high-dimensional data has been pursued in various settings including wavelet denoising, generalized linear models and low-rank matrix estimation. By thresholding some parameters to zero, estimators such as lasso, elastic net and subset selection allow to perform not only parameter estimation but also variable selection, leading to sparsity. Ye…
▽ More
Efficient recovery of a low-dimensional structure from high-dimensional data has been pursued in various settings including wavelet denoising, generalized linear models and low-rank matrix estimation. By thresholding some parameters to zero, estimators such as lasso, elastic net and subset selection allow to perform not only parameter estimation but also variable selection, leading to sparsity. Yet one crucial step challenges all these estimators: the choice of the threshold parameter~$λ$. If too large, important features are missing; if too small, incorrect features are included.
Within a unified framework, we propose a new selection of $λ$ at the detection edge under the null model. To that aim, we introduce the concept of a zero-thresholding function and a null-thresholding statistic, that we explicitly derive for a large class of estimators. The new approach has the great advantage of transforming the selection of $λ$ from an unknown scale to a probabilistic scale with the simple selection of a probability level. Numerical results show the effectiveness of our approach in terms of model selection and prediction.
△ Less
Submitted 20 March, 2017; v1 submitted 17 November, 2015;
originally announced November 2015.
-
Iterative bias reduction multivariate smoothing in R: The ibr package
Authors:
P. A. Cornillon,
N. Hengartner,
E. Matzner-Løber
Abstract:
In multivariate nonparametric analysis, sparseness of the covariates also called curse of dimensionality, forces one to use large smoothing parameters. This leads to a biased smoother. Instead of focusing on optimally selecting the smoothing parameter, we fix it to some reasonably large value to ensure an over-smoothing of the data. The resulting base smoother has a small variance but a substantia…
▽ More
In multivariate nonparametric analysis, sparseness of the covariates also called curse of dimensionality, forces one to use large smoothing parameters. This leads to a biased smoother. Instead of focusing on optimally selecting the smoothing parameter, we fix it to some reasonably large value to ensure an over-smoothing of the data. The resulting base smoother has a small variance but a substantial bias. In this paper, we propose an R package named ibr to iteratively correct the initial bias of the (base) estimator by an estimate of the bias obtained by smoothing the residuals. After a brief description of Iterated Bias Reduction smoothers, we examine the base smoothers implemented in the packages: Nadaraya-Watson kernel smoothers and thin plate splines smoothers. Then, we explain the stop** rules available in the package and their implementation. Finally we illustrate the package on two examples: a toy example in RxR and the original Los Angeles ozone dataset.
△ Less
Submitted 18 May, 2011;
originally announced May 2011.
-
Recursive bias estimation for multivariate regression smoothers
Authors:
P. A. Cornillon,
N. Hengartner,
E. Matzner-Løber
Abstract:
This paper presents a practical and simple fully nonparametric multivariate smoothing procedure that adapts to the underlying smoothness of the true regression function. Our estimator is easily computed by successive application of existing base smoothers (without the need of selecting an optimal smoothing parameter), such as thin-plate spline or kernel smoothers. The resulting smoother has better…
▽ More
This paper presents a practical and simple fully nonparametric multivariate smoothing procedure that adapts to the underlying smoothness of the true regression function. Our estimator is easily computed by successive application of existing base smoothers (without the need of selecting an optimal smoothing parameter), such as thin-plate spline or kernel smoothers. The resulting smoother has better out of sample predictive capabilities than the underlying base smoother, or competing structurally constrained models (GAM) for small dimension (3 < d < 8) and moderate sample size (n < 800). Moreover our estimator is still useful when (d> 10) and to our knowledge, no other adaptive fully nonparametric regression estimator is available without constrained assumption such as additivity for example. On a real example, the Boston Housing Data, our method reduces the out of sample prediction error by 20 %. An R package ibr, available at CRAN, implements the proposed multivariate nonparametric method in R.
△ Less
Submitted 7 June, 2011; v1 submitted 17 May, 2011;
originally announced May 2011.
-
Recursive Bias Estimation and $L_2$ Boosting
Authors:
Pierre Andre Cornillon,
Nicolas Hengartner,
Eric Matzner-Lober
Abstract:
This paper presents a general iterative bias correction procedure for regression smoothers. This bias reduction schema is shown to correspond operationally to the $L_2$ Boosting algorithm and provides a new statistical interpretation for $L_2$ Boosting. We analyze the behavior of the Boosting algorithm applied to common smoothers $S$ which we show depend on the spectrum of $I-S$. We present exam…
▽ More
This paper presents a general iterative bias correction procedure for regression smoothers. This bias reduction schema is shown to correspond operationally to the $L_2$ Boosting algorithm and provides a new statistical interpretation for $L_2$ Boosting. We analyze the behavior of the Boosting algorithm applied to common smoothers $S$ which we show depend on the spectrum of $I-S$. We present examples of common smoother for which Boosting generates a divergent sequence. The statistical interpretation suggest combining algorithm with an appropriate stop** rule for the iterative procedure. Finally we illustrate the practical finite sample performances of the iterative smoother via a simulation study. simulations.
△ Less
Submitted 30 January, 2008;
originally announced January 2008.
