Showing 1–10 of 10 results for author: Tuo, R

Asymptotic Theory for Linear Functionals of Kernel Ridge Regression

Authors: Rui Tuo, Lu Zou

Abstract: An asymptotic theory is established for linear functionals of the predictive function given by kernel ridge regression, when the reproducing kernel Hilbert space is equivalent to a Sobolev space. The theory covers a wide variety of linear functionals, including point evaluations, evaluation of derivatives, $L_2$ inner products, etc. We establish the upper and lower bounds of the estimates and thei… ▽ More An asymptotic theory is established for linear functionals of the predictive function given by kernel ridge regression, when the reproducing kernel Hilbert space is equivalent to a Sobolev space. The theory covers a wide variety of linear functionals, including point evaluations, evaluation of derivatives, $L_2$ inner products, etc. We establish the upper and lower bounds of the estimates and their asymptotic normality. It is shown that $λ\sim n^{-1}$ is the universal optimal order of magnitude for the smoothing parameter to balance the variance and the worst-case bias. The theory also implies that the optimal $L_\infty$ error of kernel ridge regression can be attained under the optimal smoothing parameter $λ\sim n^{-1}\log n$. These optimal rates for the smoothing parameter differ from the known optimal rate $λ\sim n^{-\frac{2m}{2m+d}}$ that minimizes the $L_2$ error of the kernel ridge regression. △ Less

Submitted 7 March, 2024; originally announced March 2024.

High-Dimensional Simulation Optimization via Brownian Fields and Sparse Grids

Authors: Liang Ding, Rui Tuo, Xiaowei Zhang

Abstract: High-dimensional simulation optimization is notoriously challenging. We propose a new sampling algorithm that converges to a global optimal solution and suffers minimally from the curse of dimensionality. The algorithm consists of two stages. First, we take samples following a sparse grid experimental design and approximate the response surface via kernel ridge regression with a Brownian field ker… ▽ More High-dimensional simulation optimization is notoriously challenging. We propose a new sampling algorithm that converges to a global optimal solution and suffers minimally from the curse of dimensionality. The algorithm consists of two stages. First, we take samples following a sparse grid experimental design and approximate the response surface via kernel ridge regression with a Brownian field kernel. Second, we follow the expected improvement strategy -- with critical modifications that boost the algorithm's sample efficiency -- to iteratively sample from the next level of the sparse grid. Under mild conditions on the smoothness of the response surface and the simulation noise, we establish upper bounds on the convergence rate for both noise-free and noisy simulation samples. These upper bounds deteriorate only slightly in the dimension of the feasible set, and they can be improved if the objective function is known to be of a higher-order smoothness. Extensive numerical experiments demonstrate that the proposed algorithm dramatically outperforms typical alternatives in practice. △ Less

Submitted 19 July, 2021; v1 submitted 18 July, 2021; originally announced July 2021.

Comments: Main body: 36 pages, 7 figures, 2 tables. Supplemental material: 32 pages, 1 figure

Uncertainty Quantification for Bayesian Optimization

Authors: Rui Tuo, Wenjia Wang

Abstract: Bayesian optimization is a class of global optimization techniques. In Bayesian optimization, the underlying objective function is modeled as a realization of a Gaussian process. Although the Gaussian process assumption implies a random distribution of the Bayesian optimization outputs, quantification of this uncertainty is rarely studied in the literature. In this work, we propose a novel approac… ▽ More Bayesian optimization is a class of global optimization techniques. In Bayesian optimization, the underlying objective function is modeled as a realization of a Gaussian process. Although the Gaussian process assumption implies a random distribution of the Bayesian optimization outputs, quantification of this uncertainty is rarely studied in the literature. In this work, we propose a novel approach to assess the output uncertainty of Bayesian optimization algorithms, which proceeds by constructing confidence regions of the maximum point (or value) of the objective function. These regions can be computed efficiently, and their confidence levels are guaranteed by the uniform error bounds for sequential Gaussian process regression newly developed in the present work. Our theory provides a unified uncertainty quantification framework for all existing sequential sampling policies and stop** criteria. △ Less

Submitted 5 May, 2023; v1 submitted 4 February, 2020; originally announced February 2020.

On the Improved Rates of Convergence for Matérn-type Kernel Ridge Regression, with Application to Calibration of Computer Models

Authors: Rui Tuo, Yan Wang, C. F. Jeff Wu

Abstract: Kernel ridge regression is an important nonparametric method for estimating smooth functions. We introduce a new set of conditions, under which the actual rates of convergence of the kernel ridge regression estimator under both the L_2 norm and the norm of the reproducing kernel Hilbert space exceed the standard minimax rates. An application of this theory leads to a new understanding of the Kenne… ▽ More Kernel ridge regression is an important nonparametric method for estimating smooth functions. We introduce a new set of conditions, under which the actual rates of convergence of the kernel ridge regression estimator under both the L_2 norm and the norm of the reproducing kernel Hilbert space exceed the standard minimax rates. An application of this theory leads to a new understanding of the Kennedy-O'Hagan approach for calibrating model parameters of computer simulation. We prove that, under certain conditions, the Kennedy-O'Hagan calibration estimator with a known covariance function converges to the minimizer of the norm of the residual function in the reproducing kernel Hilbert space. △ Less

Submitted 1 January, 2020; originally announced January 2020.

Comments: 24pages, 1 figure

Kriging prediction with isotropic Matérn correlations: Robustness and experimental design

Authors: Rui Tuo, Wenjia Wang

Abstract: This work investigates the prediction performance of the kriging predictors. We derive some error bounds for the prediction error in terms of non-asymptotic probability under the uniform metric and $L_p$ metrics when the spectral densities of both the true and the imposed correlation functions decay algebraically. The Matérn family is a prominent class of correlation functions of this kind. Our an… ▽ More This work investigates the prediction performance of the kriging predictors. We derive some error bounds for the prediction error in terms of non-asymptotic probability under the uniform metric and $L_p$ metrics when the spectral densities of both the true and the imposed correlation functions decay algebraically. The Matérn family is a prominent class of correlation functions of this kind. Our analysis shows that, when the smoothness of the imposed correlation function exceeds that of the true correlation function, the prediction error becomes more sensitive to the space-filling property of the design points. In particular, the kriging predictor can still reach the optimal rate of convergence, if the experimental design scheme is quasi-uniform. Lower bounds of the kriging prediction error are also derived under the uniform metric and $L_p$ metrics. An accurate characterization of this error is obtained, when an oversmoothed correlation function and a space-filling design is used. △ Less

Submitted 7 September, 2020; v1 submitted 13 November, 2019; originally announced November 2019.

Differentially Private Change-Point Detection

Authors: Rachel Cummings, Sara Krehbiel, Yajun Mei, Rui Tuo, Wanrong Zhang

Abstract: The change-point detection problem seeks to identify distributional changes at an unknown change-point k* in a stream of data. This problem appears in many important practical settings involving personal data, including biosurveillance, fault detection, finance, signal detection, and security systems. The field of differential privacy offers data analysis tools that provide powerful worst-case pri… ▽ More The change-point detection problem seeks to identify distributional changes at an unknown change-point k* in a stream of data. This problem appears in many important practical settings involving personal data, including biosurveillance, fault detection, finance, signal detection, and security systems. The field of differential privacy offers data analysis tools that provide powerful worst-case privacy guarantees. We study the statistical problem of change-point detection through the lens of differential privacy. We give private algorithms for both online and offline change-point detection, analyze these algorithms theoretically, and provide empirical validation of our results. △ Less

Submitted 29 August, 2018; originally announced August 2018.

Journal ref: Proceedings of the 32nd International Conference on Neural Information Processing Systems (2018) Pages 10848-10857

On Prediction Properties of Kriging: Uniform Error Bounds and Robustness

Authors: Wenjia Wang, Rui Tuo, C. F. Jeff Wu

Abstract: Kriging based on Gaussian random fields is widely used in reconstructing unknown functions. The kriging method has pointwise predictive distributions which are computationally simple. However, in many applications one would like to predict for a range of untried points simultaneously. In this work we obtain some error bounds for the (simple) kriging predictor under the uniform metric. It works for… ▽ More Kriging based on Gaussian random fields is widely used in reconstructing unknown functions. The kriging method has pointwise predictive distributions which are computationally simple. However, in many applications one would like to predict for a range of untried points simultaneously. In this work we obtain some error bounds for the (simple) kriging predictor under the uniform metric. It works for a scattered set of input points in an arbitrary dimension, and also covers the case where the covariance function of the Gaussian process is misspecified. These results lead to a better understanding of the rate of convergence of kriging under the Gaussian or the Matérn correlation functions, the relationship between space-filling designs and kriging models, and the robustness of the Matérn correlation functions. △ Less

Submitted 18 March, 2019; v1 submitted 18 October, 2017; originally announced October 2017.

Prediction based on the Kennedy-O'Hagan calibration model: asymptotic consistency and other properties

Authors: Rui Tuo, C. F. Jeff Wu

Abstract: Kennedy and O'Hagan (2001) propose a model for calibrating some unknown parameters in a computer model and estimating the discrepancy between the computer output and physical response. This model is known to have certain identifiability issues. Tuo and Wu (2016) show that there are examples for which the Kennedy-O'Hagan method renders unreasonable results in calibration. In spite of its unstable p… ▽ More Kennedy and O'Hagan (2001) propose a model for calibrating some unknown parameters in a computer model and estimating the discrepancy between the computer output and physical response. This model is known to have certain identifiability issues. Tuo and Wu (2016) show that there are examples for which the Kennedy-O'Hagan method renders unreasonable results in calibration. In spite of its unstable performance in calibration, the Kennedy-O'Hagan approach has a more robust behavior in predicting the physical response. In this work, we present some theoretical analysis to show the consistency of predictor based on their calibration model in the context of radial basis functions. △ Less

Submitted 3 March, 2017; originally announced March 2017.

A finite element method for elliptic problems with observational boundary data

Authors: Zhiming Chen, Rui Tuo, Wenlong Zhang

Abstract: In this paper we propose a finite element method for solving elliptic equations with the observational Dirichlet boundary data which may subject to random noises. The method is based on the weak formulation of Lagrangian multiplier. We show the convergence of the random finite element error in expectation and, when the noise is sub-Gaussian, in the Orlicz 2- norm which implies the probability that… ▽ More In this paper we propose a finite element method for solving elliptic equations with the observational Dirichlet boundary data which may subject to random noises. The method is based on the weak formulation of Lagrangian multiplier. We show the convergence of the random finite element error in expectation and, when the noise is sub-Gaussian, in the Orlicz 2- norm which implies the probability that the finite element error estimates are violated decays exponentially. Numerical examples are included. △ Less

Submitted 16 February, 2017; originally announced February 2017.

Stochastic Convergence of A Nonconforming Finite Element Method for the Thin Plate Spline Smoother for Observational Data

Authors: Zhiming Chen, Rui Tuo, Wenlong Zhang

Abstract: The thin plate spline smoother is a classical model for fnding a smooth function from the knowledge of its observation at scattered locations which may have random noises. We consider a nonconforming Morley finite element method to approximate the model. We prove the stochastic convergence of the finite element method which characterizes the tail property of the probability distribution function o… ▽ More The thin plate spline smoother is a classical model for fnding a smooth function from the knowledge of its observation at scattered locations which may have random noises. We consider a nonconforming Morley finite element method to approximate the model. We prove the stochastic convergence of the finite element method which characterizes the tail property of the probability distribution function of the finite element error. We also propose a self-consistent iterative algorithm to determine the smoothing parameter based on our theoretical analysis. Numerical examples are included to confirm the theoretical analysis and to show the competitive performance of the self- consistent algorithm for finding the smoothing parameter. △ Less

Submitted 30 January, 2017; originally announced January 2017.