Showing 1–34 of 34 results for author: Kuchibhotla, A K

Generalized van Trees inequality: Local minimax bounds for non-smooth functionals and irregular statistical models

Authors: Kenta Takatsu, Arun Kumar Kuchibhotla

Abstract: In a decision-theoretic framework, minimax lower bound provides the worst-case performance of estimators relative to a given class of statistical models. For parametric and semiparametric models, the Hájek--Le Cam local asymptotic minimax (LAM) theorem provides the optimal and sharp asymptotic lower bound. Despite its relative generality, this result comes with limitations as it only applies to th… ▽ More In a decision-theoretic framework, minimax lower bound provides the worst-case performance of estimators relative to a given class of statistical models. For parametric and semiparametric models, the Hájek--Le Cam local asymptotic minimax (LAM) theorem provides the optimal and sharp asymptotic lower bound. Despite its relative generality, this result comes with limitations as it only applies to the estimation of differentiable functionals under regular statistical models. On the other hand, non-asymptotic minimax lower bounds, such as those based on the reduction to hypothesis testing, do not often yield sharp asymptotic constants. Inspired by the recent improvement of the van Trees inequality and related methods in the literature, we provide new non-asymptotic minimax lower bounds under minimal regularity assumptions, which imply sharp asymptotic constants. The proposed lower bounds do not require the differentiability of functionals or regularity of statistical models, extending the efficiency theory to broader situations where standard approaches fail. Additionally, new lower bounds provide non-asymptotic constants, which can shed light on more refined fundamental limits of estimation in finite samples. We demonstrate that new lower bounds recover many classical results, including the LAM theorem and semiparametric efficiency bounds. We also illustrate the use of the new lower bound by deriving the local minimax lower bound for estimating the density at a point and directionally differentiable parameters. △ Less

Submitted 10 May, 2024; originally announced May 2024.

Inference for Median and a Generalization of HulC

Authors: Manit Paul, Arun Kumar Kuchibhotla

Abstract: Constructing distribution-free confidence intervals for the median, a classic problem in statistics, has seen numerous solutions in the literature. While coverage validity has received ample attention, less has been explored about interval width. Our study breaks new ground by investigating the width of these intervals under non-standard assumptions. Surprisingly, we find that properly scaled, the… ▽ More Constructing distribution-free confidence intervals for the median, a classic problem in statistics, has seen numerous solutions in the literature. While coverage validity has received ample attention, less has been explored about interval width. Our study breaks new ground by investigating the width of these intervals under non-standard assumptions. Surprisingly, we find that properly scaled, the interval width converges to a non-degenerate random variable, unlike traditional intervals. We also adapt our findings for constructing improved confidence intervals for general parameters, enhancing the existing HulC procedure. These advances provide practitioners with more robust tools for data analysis, reducing the need for strict distributional assumptions. △ Less

Submitted 10 March, 2024; originally announced March 2024.

Comments: 75 pages, 10 figures

Rectangular Hull Confidence Regions for Multivariate Parameters

Authors: Aniket Jain, Arun K Kuchibhotla

Abstract: We introduce three notions of multivariate median bias, namely, rectilinear, Tukey, and orthant median bias. Each of these median biases is zero under a suitable notion of multivariate symmetry. We study the coverage probabilities of rectangular hull of $B$ independent multivariate estimators, with special attention to the number of estimators $B$ needed to ensure a miscoverage of at most $α$. It… ▽ More We introduce three notions of multivariate median bias, namely, rectilinear, Tukey, and orthant median bias. Each of these median biases is zero under a suitable notion of multivariate symmetry. We study the coverage probabilities of rectangular hull of $B$ independent multivariate estimators, with special attention to the number of estimators $B$ needed to ensure a miscoverage of at most $α$. It is proved that for estimators with zero orthant median bias, we need $B\geq c\log_2(d/α)$ for some constant $c > 0$. Finally, we show that there exists an asymptotically valid (non-trivial) confidence region for a multivariate parameter $θ_0$ if and only if there exists a (non-trivial) estimator with an asymptotic orthant median bias of zero. △ Less

Submitted 5 December, 2023; v1 submitted 28 November, 2023; originally announced November 2023.

Comments: Added the proof of Proposition 3.9

New Asymptotic Limit Theory and Inference for Monotone Regression

Authors: Soham Mallick, Siddhaarth Sarkar, Arun Kumar Kuchibhotla

Abstract: Nonparametric regression problems with qualitative constraints such as monotonicity or convexity are ubiquitous in applications. For example, in predicting the yield of a factory in terms of the number of labor hours, the monotonicity of the conditional mean function is a natural constraint. One can estimate a monotone conditional mean function using nonparametric least squares estimation, which i… ▽ More Nonparametric regression problems with qualitative constraints such as monotonicity or convexity are ubiquitous in applications. For example, in predicting the yield of a factory in terms of the number of labor hours, the monotonicity of the conditional mean function is a natural constraint. One can estimate a monotone conditional mean function using nonparametric least squares estimation, which involves no tuning parameters. Several interesting properties of the isotonic LSE are known including its rate of convergence, adaptivity properties, and pointwise asymptotic distribution. However, we believe that the full richness of the asymptotic limit theory has not been explored in the literature which we do in this paper. Moreover, the inference problem is not fully settled. In this paper, we present some new results for monotone regression including an extension of existing results to triangular arrays, and provide asymptotically valid confidence intervals that are uniformly valid over a large class of distributions. △ Less

Submitted 17 November, 2023; v1 submitted 30 October, 2023; originally announced October 2023.

Comments: Additional simulations added along with link to R code files

Forster-Warmuth Counterfactual Regression: A Unified Learning Approach

Authors: Yachong Yang, Arun Kumar Kuchibhotla, Eric Tchetgen Tchetgen

Abstract: Series or orthogonal basis regression is one of the most popular non-parametric regression techniques in practice, obtained by regressing the response on features generated by evaluating the basis functions at observed covariate values. The most routinely used series estimator is based on ordinary least squares fitting, which is known to be minimax rate optimal in various settings, albeit under st… ▽ More Series or orthogonal basis regression is one of the most popular non-parametric regression techniques in practice, obtained by regressing the response on features generated by evaluating the basis functions at observed covariate values. The most routinely used series estimator is based on ordinary least squares fitting, which is known to be minimax rate optimal in various settings, albeit under stringent restrictions on the basis functions and the distribution of covariates. In this work, inspired by the recently developed Forster-Warmuth (FW) learner, we propose an alternative series regression estimator that can attain the minimax estimation rate under strictly weaker conditions imposed on the basis functions and the joint law of covariates, than existing series estimators in the literature. Moreover, a key contribution of this work generalizes the FW-learner to a so-called counterfactual regression problem, in which the response variable of interest may not be directly observed (hence, the name ``counterfactual'') on all sampled units, and therefore needs to be inferred in order to identify and estimate the regression in view from the observed data. Although counterfactual regression is not entirely a new area of inquiry, we propose the first-ever systematic study of this challenging problem from a unified pseudo-outcome perspective. In fact, we provide what appears to be the first generic and constructive approach for generating the pseudo-outcome (to substitute for the unobserved response) which leads to the estimation of the counterfactual regression curve of interest with small bias, namely bias of second order. Several applications are used to illustrate the resulting FW-learner including many nonparametric regression problems in missing data and causal inference literature, for which we establish high-level conditions for minimax rate optimality of the proposed FW-learner. △ Less

Submitted 20 March, 2024; v1 submitted 31 July, 2023; originally announced July 2023.

Comments: Add grant acknowledgement

From isotonic to Lipschitz regression: a new interpolative perspective on shape-restricted estimation

Authors: Kenta Takatsu, Tianyu Zhang, Arun Kumar Kuchibhotla

Abstract: This manuscript seeks to bridge two seemingly disjoint paradigms of nonparametric regression estimation based on smoothness assumptions and shape constraints. The proposed approach is motivated by a conceptually simple observation: Every Lipschitz function is a sum of monotonic and linear functions. This principle is further generalized to the higher-order monotonicity and multivariate covariates.… ▽ More This manuscript seeks to bridge two seemingly disjoint paradigms of nonparametric regression estimation based on smoothness assumptions and shape constraints. The proposed approach is motivated by a conceptually simple observation: Every Lipschitz function is a sum of monotonic and linear functions. This principle is further generalized to the higher-order monotonicity and multivariate covariates. A family of estimators is proposed based on a sample-splitting procedure, which inherits desirable methodological, theoretical, and computational properties of shape-restricted estimators. Our theoretical analysis provides convergence guarantees of the estimator under heteroscedastic and heavy-tailed errors, as well as adaptive properties to the complexity of the true regression function. The generality of the proposed decomposition framework is demonstrated through new approximation results, and extensive numerical studies validate the theoretical properties and empirical evidence for the practicalities of the proposed estimation framework. △ Less

Submitted 20 June, 2024; v1 submitted 11 July, 2023; originally announced July 2023.

Inference for Projection Parameters in Linear Regression: beyond $d = o(n^{1/2})$

Authors: Woonyoung Chang, Arun Kumar Kuchibhotla, Alessandro Rinaldo

Abstract: We consider the problem of inference for projection parameters in linear regression with increasing dimensions. This problem has been studied under a variety of assumptions in the literature. The classical asymptotic normality result for the least squares estimator of the projection parameter only holds when the dimension $d$ of the covariates is of a smaller order than $n^{1/2}$, where $n$ is the… ▽ More We consider the problem of inference for projection parameters in linear regression with increasing dimensions. This problem has been studied under a variety of assumptions in the literature. The classical asymptotic normality result for the least squares estimator of the projection parameter only holds when the dimension $d$ of the covariates is of a smaller order than $n^{1/2}$, where $n$ is the sample size. Traditional sandwich estimator-based Wald intervals are asymptotically valid in this regime. In this work, we propose a bias correction for the least squares estimator and prove the asymptotic normality of the resulting debiased estimator. Precisely, we provide an explicit finite sample Berry Esseen bound on the Normal approximation to the law of the linear contrasts of the proposed estimator normalized by the sandwich standard error estimate. Our bound, under only finite moment conditions on covariates and errors, tends to 0 as long as $d = o(n^{2/3})$ up to the polylogarithmic factors. Furthermore, we leverage recent methods of statistical inference that do not require an estimator of the variance to perform asymptotically valid statistical inference and that leads to a sharper miscoverage control compared to Wald's. We provide a discussion of how our techniques can be generalized to increase the allowable range of $d$ even further. △ Less

Submitted 11 January, 2024; v1 submitted 3 July, 2023; originally announced July 2023.

Comments: Updated Jan 11, 2024

Central Limit Theorems and Approximation Theory: Part II

Authors: Arun Kumar Kuchibhotla

Abstract: In Part I of this article (Banerjee and Kuchibhotla (2023)), we have introduced a new method to bound the difference in expectations of an average of independent random vector and the limiting Gaussian random vector using level sets. In the current article, we further explore this idea using finite sample Edgeworth expansions and also established integral representation theorems. In Part I of this article (Banerjee and Kuchibhotla (2023)), we have introduced a new method to bound the difference in expectations of an average of independent random vector and the limiting Gaussian random vector using level sets. In the current article, we further explore this idea using finite sample Edgeworth expansions and also established integral representation theorems. △ Less

Submitted 25 June, 2023; originally announced June 2023.

Dual Induction CLT for High-dimensional m-dependent Data

Authors: Heejong Bong, Arun Kumar Kuchibhotla, Alessandro Rinaldo

Abstract: We derive novel and sharp high-dimensional Berry--Esseen bounds for the sum of $m$-dependent random vectors over the class of hyper-rectangles exhibiting only a poly-logarithmic dependence in the dimension. Our results hold under minimal assumptions, such as non-degenerate covariances and finite third moments, and yield a sample complexity of order $\sqrt{m/n}$, aside from logarithmic terms, match… ▽ More We derive novel and sharp high-dimensional Berry--Esseen bounds for the sum of $m$-dependent random vectors over the class of hyper-rectangles exhibiting only a poly-logarithmic dependence in the dimension. Our results hold under minimal assumptions, such as non-degenerate covariances and finite third moments, and yield a sample complexity of order $\sqrt{m/n}$, aside from logarithmic terms, matching the optimal rates established in the univariate case. When specialized to the sums of independent non-degenerate random vectors, we obtain sharp rates under the weakest possible conditions. On the technical side, we develop an inductive relationship between anti-concentration inequalities and Berry--Esseen bounds, inspired by the classical Lindeberg swap** method and the concentration inequality approach for dependent data, that may be of independent interest. △ Less

Submitted 16 November, 2023; v1 submitted 25 June, 2023; originally announced June 2023.

Comments: 25 pages

MSC Class: 60B12; 60F05

Central Limit Theorems and Approximation Theory: Part I

Authors: Arisina Banerjee, Arun K Kuchibhotla

Abstract: Central limit theorems (CLTs) have a long history in probability and statistics. They play a fundamental role in constructing valid statistical inference procedures. Over the last century, various techniques have been developed in probability and statistics to prove CLTs under a variety of assumptions on random variables. Quantitative versions of CLTs (e.g., Berry--Esseen bounds) have also been pa… ▽ More Central limit theorems (CLTs) have a long history in probability and statistics. They play a fundamental role in constructing valid statistical inference procedures. Over the last century, various techniques have been developed in probability and statistics to prove CLTs under a variety of assumptions on random variables. Quantitative versions of CLTs (e.g., Berry--Esseen bounds) have also been parallelly developed. In this article, we propose to use approximation theory from functional analysis to derive explicit bounds on the difference between expectations of functions. △ Less

Submitted 25 June, 2023; v1 submitted 9 June, 2023; originally announced June 2023.

Comments: 25 pages

Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validation

Authors: **-Hong Du, Pratik Patil, Arun Kumar Kuchibhotla

Abstract: We study subsampling-based ridge ensembles in the proportional asymptotics regime, where the feature size grows proportionally with the sample size such that their ratio converges to a constant. By analyzing the squared prediction risk of ridge ensembles as a function of the explicit penalty $λ$ and the limiting subsample aspect ratio $φ_s$ (the ratio of the feature size to the subsample size), we… ▽ More We study subsampling-based ridge ensembles in the proportional asymptotics regime, where the feature size grows proportionally with the sample size such that their ratio converges to a constant. By analyzing the squared prediction risk of ridge ensembles as a function of the explicit penalty $λ$ and the limiting subsample aspect ratio $φ_s$ (the ratio of the feature size to the subsample size), we characterize contours in the $(λ, φ_s)$-plane at any achievable risk. As a consequence, we prove that the risk of the optimal full ridgeless ensemble (fitted on all possible subsamples) matches that of the optimal ridge predictor. In addition, we prove strong uniform consistency of generalized cross-validation (GCV) over the subsample sizes for estimating the prediction risk of ridge ensembles. This allows for GCV-based tuning of full ridgeless ensembles without sample splitting and yields a predictor whose risk matches optimal ridge risk. △ Less

Submitted 16 July, 2023; v1 submitted 25 April, 2023; originally announced April 2023.

Comments: 47 pages, 11 figures; this version fixes minor typos. arXiv admin note: text overlap with arXiv:2210.11445

Tight Concentration Inequality for Sub-Weibull Random Variables with Generalized Bernstien Orlicz norm

Authors: Heejong Bong, Arun Kumar Kuchibhotla

Abstract: Recent development in high-dimensional statistical inference has necessitated concentration inequalities for a broader range of random variables. We focus on sub-Weibull random variables, which extend sub-Gaussian or sub-exponential random variables to allow heavy-tailed distributions. This paper presents concentration inequalities for independent sub-Weibull random variables with finite Generaliz… ▽ More Recent development in high-dimensional statistical inference has necessitated concentration inequalities for a broader range of random variables. We focus on sub-Weibull random variables, which extend sub-Gaussian or sub-exponential random variables to allow heavy-tailed distributions. This paper presents concentration inequalities for independent sub-Weibull random variables with finite Generalized Bernstein-Orlicz norms, providing generalized Bernstein's inequalities and Rosenthal-type moment bounds. The tightness of the proposed bounds is shown through lower bounds of the concentration inequalities obtained via the Paley-Zygmund inequality. The results are applied to a graphical model inference problem, improving previous sample complexity bounds. △ Less

Submitted 25 February, 2023; v1 submitted 7 February, 2023; originally announced February 2023.

MSC Class: 60G50; 60E15 (Primary) 60B20; 62E22 (Secondary)

High-dimensional Berry-Esseen Bound for $m$-Dependent Random Samples

Authors: Heejong Bong, Arun Kumar Kuchibhotla, Alessandro Rinaldo

Abstract: In this work, we provide a $(n/m)^{-1/2}$-rate finite sample Berry-Esseen bound for $m$-dependent high-dimensional random vectors over the class of hyper-rectangles. This bound imposes minimal assumptions on the random vectors such as nondegenerate covariances and finite third moments. The proof uses inductive relationships between anti-concentration inequalities and Berry--Esseen bounds, which ar… ▽ More In this work, we provide a $(n/m)^{-1/2}$-rate finite sample Berry-Esseen bound for $m$-dependent high-dimensional random vectors over the class of hyper-rectangles. This bound imposes minimal assumptions on the random vectors such as nondegenerate covariances and finite third moments. The proof uses inductive relationships between anti-concentration inequalities and Berry--Esseen bounds, which are inspired by the telesco** method of Chen and Shao (2004) and the recursion method of Kuchibhotla and Rinaldo (2020). Performing a dual induction based on the relationships, we obtain tight Berry-Esseen bounds for dependent samples. △ Less

Submitted 10 December, 2022; originally announced December 2022.

Bagging in overparameterized learning: Risk characterization and risk monotonization

Authors: Pratik Patil, **-Hong Du, Arun Kumar Kuchibhotla

Abstract: Bagging is a commonly used ensemble technique in statistics and machine learning to improve the performance of prediction procedures. In this paper, we study the prediction risk of variants of bagged predictors under the proportional asymptotics regime, in which the ratio of the number of features to the number of observations converges to a constant. Specifically, we propose a general strategy to… ▽ More Bagging is a commonly used ensemble technique in statistics and machine learning to improve the performance of prediction procedures. In this paper, we study the prediction risk of variants of bagged predictors under the proportional asymptotics regime, in which the ratio of the number of features to the number of observations converges to a constant. Specifically, we propose a general strategy to analyze the prediction risk under squared error loss of bagged predictors using classical results on simple random sampling. Specializing the strategy, we derive the exact asymptotic risk of the bagged ridge and ridgeless predictors with an arbitrary number of bags under a well-specified linear model with arbitrary feature covariance matrices and signal vectors. Furthermore, we prescribe a generic cross-validation procedure to select the optimal subsample size for bagging and discuss its utility to eliminate the non-monotonic behavior of the limiting risk in the sample size (i.e., double or multiple descents). In demonstrating the proposed procedure for bagged ridge and ridgeless predictors, we thoroughly investigate the oracle properties of the optimal subsample size and provide an in-depth comparison between different bagging variants. △ Less

Submitted 24 October, 2023; v1 submitted 20 October, 2022; originally announced October 2022.

Comments: 102 pages, 34 figures; this version add minor clarifications at few places

Median Regularity and Honest Inference

Authors: Arun Kumar Kuchibhotla, Sivaraman Balakrishnan, Larry Wasserman

Abstract: We introduce a new notion of regularity of an estimator called median regularity. We prove that uniformly valid (honest) inference for a functional is possible if and only if there exists a median regular estimator of that functional. To our knowledge, such a notion of regularity that is necessary for uniformly valid inference is unavailable in the literature. We introduce a new notion of regularity of an estimator called median regularity. We prove that uniformly valid (honest) inference for a functional is possible if and only if there exists a median regular estimator of that functional. To our knowledge, such a notion of regularity that is necessary for uniformly valid inference is unavailable in the literature. △ Less

Submitted 6 June, 2022; originally announced June 2022.

Comments: 10 pages

Mitigating multiple descents: A model-agnostic framework for risk monotonization

Authors: Pratik Patil, Arun Kumar Kuchibhotla, Yuting Wei, Alessandro Rinaldo

Abstract: Recent empirical and theoretical analyses of several commonly used prediction procedures reveal a peculiar risk behavior in high dimensions, referred to as double/multiple descent, in which the asymptotic risk is a non-monotonic function of the limiting aspect ratio of the number of features or parameters to the sample size. To mitigate this undesirable behavior, we develop a general framework for… ▽ More Recent empirical and theoretical analyses of several commonly used prediction procedures reveal a peculiar risk behavior in high dimensions, referred to as double/multiple descent, in which the asymptotic risk is a non-monotonic function of the limiting aspect ratio of the number of features or parameters to the sample size. To mitigate this undesirable behavior, we develop a general framework for risk monotonization based on cross-validation that takes as input a generic prediction procedure and returns a modified procedure whose out-of-sample prediction risk is, asymptotically, monotonic in the limiting aspect ratio. As part of our framework, we propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting, respectively, and show that, under very mild assumptions, they provably achieve monotonic asymptotic risk behavior. Our results are applicable to a broad variety of prediction procedures and loss functions, and do not require a well-specified (parametric) model. We exemplify our framework with concrete analyses of the minimum $\ell_2$, $\ell_1$-norm least squares prediction procedures. As one of the ingredients in our analysis, we also derive novel additive and multiplicative forms of oracle risk inequalities for split cross-validation that are of independent interest. △ Less

Submitted 25 May, 2022; originally announced May 2022.

Comments: 110 pages, 15 figures

Doubly Robust Calibration of Prediction Sets under Covariate Shift

Authors: Yachong Yang, Arun Kumar Kuchibhotla, Eric Tchetgen Tchetgen

Abstract: Conformal prediction has received tremendous attention in recent years and has offered new solutions to problems in missing data and causal inference; yet these advances have not leveraged modern semiparametric efficiency theory for more robust and efficient uncertainty quantification. In this paper, we consider the problem of obtaining distribution-free prediction regions accounting for a shift i… ▽ More Conformal prediction has received tremendous attention in recent years and has offered new solutions to problems in missing data and causal inference; yet these advances have not leveraged modern semiparametric efficiency theory for more robust and efficient uncertainty quantification. In this paper, we consider the problem of obtaining distribution-free prediction regions accounting for a shift in the distribution of the covariates between the training and test data. Under an explainable covariate shift assumption analogous to the standard missing at random assumption, we propose three variants of a general framework to construct well-calibrated prediction regions for the unobserved outcome in the test sample. Our approach is based on the efficient influence function for the quantile of the unobserved outcome in the test population combined with an arbitrary machine learning prediction algorithm, without compromising asymptotic coverage. Next, we extend our approach to account for departure from the explainable covariate shift assumption in a semiparametric sensitivity analysis for potential latent covariate shift. In all cases, we establish that the resulting prediction sets eventually attain nominal average coverage in large samples. This guarantee is a consequence of the product bias form of our proposal which implies correct coverage if either the propensity score or the conditional distribution of the response is estimated sufficiently well. Our results also provide a framework for construction of doubly robust prediction sets of individual treatment effects, under both unconfoundedness and allowing for some degree of unmeasured confounding. Finally, we discuss aggregation of prediction sets from different machine learning algorithms for optimal prediction and illustrate the performance of our methods in both synthetic and real data. △ Less

Submitted 13 December, 2022; v1 submitted 3 March, 2022; originally announced March 2022.

Comments: New contribution about impossibility of finite sample results and asymptotic conditional coverage through CQR score

Median bias of M-estimators

Authors: Arun Kumar Kuchibhotla

Abstract: In this note, we derive bounds on the median bias of univariate M-estimators under mild regularity conditions. These requirements are not sufficient to imply convergence in distribution of the M-estimators. We also discuss median bias of some multivariate M-estimators. In this note, we derive bounds on the median bias of univariate M-estimators under mild regularity conditions. These requirements are not sufficient to imply convergence in distribution of the M-estimators. We also discuss median bias of some multivariate M-estimators. △ Less

Submitted 31 May, 2021; originally announced June 2021.

The HulC: Confidence Regions from Convex Hulls

Authors: Arun Kumar Kuchibhotla, Sivaraman Balakrishnan, Larry Wasserman

Abstract: We develop and analyze the HulC, an intuitive and general method for constructing confidence sets using the convex hull of estimates constructed from subsets of the data. Unlike classical methods which are based on estimating the (limiting) distribution of an estimator, the HulC is often simpler to use and effectively bypasses this step. In comparison to the bootstrap, the HulC requires fewer regu… ▽ More We develop and analyze the HulC, an intuitive and general method for constructing confidence sets using the convex hull of estimates constructed from subsets of the data. Unlike classical methods which are based on estimating the (limiting) distribution of an estimator, the HulC is often simpler to use and effectively bypasses this step. In comparison to the bootstrap, the HulC requires fewer regularity conditions and succeeds in many examples where the bootstrap provably fails. Unlike subsampling, the HulC does not require knowledge of the rate of convergence of the estimators on which it is based. The validity of the HulC requires knowledge of the (asymptotic) median-bias of the estimators. We further analyze a variant of our basic method, called the Adaptive HulC, which is fully data-driven and estimates the median-bias using subsampling. We show that the Adaptive HulC retains the aforementioned strengths of the HulC. In certain cases where the underlying estimators are pathologically asymmetric the HulC and Adaptive HulC can fail to provide useful confidence sets. We propose a final variant, the Unimodal HulC, which can salvage the situation in cases where the distribution of the underlying estimator is (asymptotically) unimodal. We discuss these methods in the context of several challenging inferential problems which arise in parametric, semi-parametric, and non-parametric inference. Although our focus is on validity under weak regularity conditions, we also provide some general results on the width of the HulC confidence sets, showing that in many cases the HulC confidence sets have near-optimal width. △ Less

Submitted 8 September, 2023; v1 submitted 30 May, 2021; originally announced May 2021.

Comments: Latest version. Fixed a gap in Proposition and Theorem 1 pointed out by Prof. Hannes Leeb. Now all the simulations include a comparison with subsampling. Also, added several new simulation settings including quantile regression, isotonic regression both under non-standard assumptions

High-dimensional CLT for Sums of Non-degenerate Random Vectors: $n^{-1/2}$-rate

Authors: Arun Kumar Kuchibhotla, Alessandro Rinaldo

Abstract: In this note, we provide a Berry--Esseen bounds for rectangles in high-dimensions when the random vectors have non-singular covariance matrices. Under this assumption of non-singularity, we prove an $n^{-1/2}$ scaling for the Berry--Esseen bound for sums of mean independent random vectors with a finite third moment. The proof is essentially the method of compositions proof of multivariate Berry--E… ▽ More In this note, we provide a Berry--Esseen bounds for rectangles in high-dimensions when the random vectors have non-singular covariance matrices. Under this assumption of non-singularity, we prove an $n^{-1/2}$ scaling for the Berry--Esseen bound for sums of mean independent random vectors with a finite third moment. The proof is essentially the method of compositions proof of multivariate Berry--Esseen bound from Senatov (2011). Similar to other existing works (Kuchibhotla et al. 2018, Fang and Koike 2020a), this note considers the applicability and effectiveness of classical CLT proof techniques for the high-dimensional case. △ Less

Submitted 28 September, 2020; originally announced September 2020.

Comments: 21 pages

Berry-Esseen Bounds for Projection Parameters and Partial Correlations with Increasing Dimension

Authors: Arun Kumar Kuchibhotla, Alessandro Rinaldo, Larry Wasserman

Abstract: We provide finite sample bounds on the Normal approximation to the law of the least squares estimator of the projection parameters normalized by the sandwich-based standard errors. Our results hold in the increasing dimension setting and under minimal assumptions on the data generating distribution. In particular, we do not assume a linear regression function and only require the existence of fini… ▽ More We provide finite sample bounds on the Normal approximation to the law of the least squares estimator of the projection parameters normalized by the sandwich-based standard errors. Our results hold in the increasing dimension setting and under minimal assumptions on the data generating distribution. In particular, we do not assume a linear regression function and only require the existence of finitely many moments for the response and the covariates. Furthermore, we construct confidence sets for the projection parameters in the form of hyper-rectangles and establish finite sample bounds on their coverage and accuracy. We derive analogous results for partial correlations among the entries of sub-Gaussian vectors. \end{abstract} △ Less

Submitted 22 October, 2021; v1 submitted 19 July, 2020; originally announced July 2020.

Comments: 58 pages, 0 figures

Near-Optimal Confidence Sequences for Bounded Random Variables

Authors: Arun Kumar Kuchibhotla, Qinqing Zheng

Abstract: Many inference problems, such as sequential decision problems like A/B testing, adaptive sampling schemes like bandit selection, are often online in nature. The fundamental problem for online inference is to provide a sequence of confidence intervals that are valid uniformly over the growing-into-infinity sample sizes. To address this question, we provide a near-optimal confidence sequence for bou… ▽ More Many inference problems, such as sequential decision problems like A/B testing, adaptive sampling schemes like bandit selection, are often online in nature. The fundamental problem for online inference is to provide a sequence of confidence intervals that are valid uniformly over the growing-into-infinity sample sizes. To address this question, we provide a near-optimal confidence sequence for bounded random variables by utilizing Bentkus' concentration results. We show that it improves on the existing approaches that use the Cram{é}r-Chernoff technique such as the Hoeffding, Bernstein, and Bennett inequalities. The resulting confidence sequence is confirmed to be favorable in both synthetic coverage problems and an application to adaptive stop** algorithms. △ Less

Submitted 3 June, 2021; v1 submitted 8 June, 2020; originally announced June 2020.

Comments: Accepted to ICML 2021

Nested conformal prediction and quantile out-of-bag ensemble methods

Authors: Chirag Gupta, Arun K. Kuchibhotla, Aaditya K. Ramdas

Abstract: Conformal prediction is a popular tool for providing valid prediction sets for classification and regression problems, without relying on any distributional assumptions on the data. While the traditional description of conformal prediction starts with a nonconformity score, we provide an alternate (but equivalent) view that starts with a sequence of nested sets and calibrates them to find a valid… ▽ More Conformal prediction is a popular tool for providing valid prediction sets for classification and regression problems, without relying on any distributional assumptions on the data. While the traditional description of conformal prediction starts with a nonconformity score, we provide an alternate (but equivalent) view that starts with a sequence of nested sets and calibrates them to find a valid prediction set. The nested framework subsumes all nonconformity scores, including recent proposals based on quantile regression and density estimation. While these ideas were originally derived based on sample splitting, our framework seamlessly extends them to other aggregation schemes like cross-conformal, jackknife+ and out-of-bag methods. We use the framework to derive a new algorithm (QOOB, pronounced cube) that combines four ideas: quantile regression, cross-conformalization, ensemble methods and out-of-bag predictions. We develop a computationally efficient implementation of cross-conformal, that is also used by QOOB. In a detailed numerical investigation, QOOB performs either the best or close to the best on all simulated and real datasets. Code for QOOB is available at https://github.com/aigen/QOOB. △ Less

Submitted 9 May, 2022; v1 submitted 23 October, 2019; originally announced October 2019.

Comments: 38 pages, 4 figures, 8 tables. This version fixes a bug in the proof of Proposition 3. Published paper available at https://www.sciencedirect.com/science/article/abs/pii/S0031320321006725

Journal ref: Pattern Recognition 127 (2022): 108496

All of Linear Regression

Authors: Arun K. Kuchibhotla, Lawrence D. Brown, Andreas Buja, Junhui Cai

Abstract: Least squares linear regression is one of the oldest and widely used data analysis tools. Although the theoretical analysis of the ordinary least squares (OLS) estimator is as old, several fundamental questions are yet to be answered. Suppose regression observations $(X_1,Y_1),\ldots,(X_n,Y_n)\in\mathbb{R}^d\times\mathbb{R}$ (not necessarily independent) are available. Some of the questions we dea… ▽ More Least squares linear regression is one of the oldest and widely used data analysis tools. Although the theoretical analysis of the ordinary least squares (OLS) estimator is as old, several fundamental questions are yet to be answered. Suppose regression observations $(X_1,Y_1),\ldots,(X_n,Y_n)\in\mathbb{R}^d\times\mathbb{R}$ (not necessarily independent) are available. Some of the questions we deal with are as follows: under what conditions, does the OLS estimator converge and what is the limit? What happens if the dimension is allowed to grow with $n$? What happens if the observations are dependent with dependence possibly strengthening with $n$? How to do statistical inference under these kinds of misspecification? What happens to the OLS estimator under variable selection? How to do inference under misspecification and variable selection? We answer all the questions raised above with one simple deterministic inequality which holds for any set of observations and any sample size. This implies that all our results are a finite sample (non-asymptotic) in nature. In the end, one only needs to bound certain random quantities under specific settings of interest to get concrete rates and we derive these bounds for the case of independent observations. In particular, the problem of inference after variable selection is studied, for the first time, when $d$, the number of covariates increases (almost exponentially) with sample size $n$. We provide comments on the ``right'' statistic to consider for inference under variable selection and efficient computation of quantiles. △ Less

Submitted 14 October, 2019; originally announced October 2019.

First order expansion of convex regularized estimators

Authors: Pierre C Bellec, Arun K Kuchibhotla

Abstract: We consider first order expansions of convex penalized estimators in high-dimensional regression problems with random designs. Our setting includes linear regression and logistic regression as special cases. For a given penalty function $h$ and the corresponding penalized estimator $\hatβ$, we construct a quantity $η$, the first order expansion of $\hatβ$, such that the distance between $\hatβ$ an… ▽ More We consider first order expansions of convex penalized estimators in high-dimensional regression problems with random designs. Our setting includes linear regression and logistic regression as special cases. For a given penalty function $h$ and the corresponding penalized estimator $\hatβ$, we construct a quantity $η$, the first order expansion of $\hatβ$, such that the distance between $\hatβ$ and $η$ is an order of magnitude smaller than the estimation error $\|\hatβ - β^*\|$. In this sense, the first order expansion $η$ can be thought of as a generalization of influence functions from the mathematical statistics literature to regularized estimators in high-dimensions. Such first order expansion implies that the risk of $\hatβ$ is asymptotically the same as the risk of $η$ which leads to a precise characterization of the MSE of $\hatβ$; this characterization takes a particularly simple form for isotropic design. Such first order expansion also leads to inference results based on $\hatβ$. We provide sufficient conditions for the existence of such first order expansion for three regularizers: the Lasso in its constrained form, the lasso in its penalized form, and the Group-Lasso. The results apply to general loss functions under some conditions and those conditions are satisfied for the squared loss in linear regression and for the logistic loss in the logistic model. △ Less

Submitted 8 March, 2020; v1 submitted 11 October, 2019; originally announced October 2019.

Comments: Accepted at NeurIPS 2019 and published at https://papers.nips.cc/paper/8606-first-order-expansion-of-convex-regularized-estimators . The version here includes the supplementary material

On Least Squares Estimation under Heteroscedastic and Heavy-Tailed Errors

Authors: Arun K. Kuchibhotla, Rohit K. Patra

Abstract: We consider least squares estimation in a general nonparametric regression model. The rate of convergence of the least squares estimator (LSE) for the unknown regression function is well studied when the errors are sub-Gaussian. We find upper bounds on the rates of convergence of the LSE when the errors have uniformly bounded conditional variance and have only finitely many moments. We show that t… ▽ More We consider least squares estimation in a general nonparametric regression model. The rate of convergence of the least squares estimator (LSE) for the unknown regression function is well studied when the errors are sub-Gaussian. We find upper bounds on the rates of convergence of the LSE when the errors have uniformly bounded conditional variance and have only finitely many moments. We show that the interplay between the moment assumptions on the error, the metric entropy of the class of functions involved, and the "local" structure of the function class around the truth drives the rate of convergence of the LSE. We find sufficient conditions on the errors under which the rate of the LSE matches the rate of the LSE under sub-Gaussian error. Our results are finite sample and allow for heteroscedastic and heavy-tailed errors. △ Less

Submitted 8 April, 2021; v1 submitted 4 September, 2019; originally announced September 2019.

Comments: 49 pages, 2 figures, and 3 tables

Model-free Study of Ordinary Least Squares Linear Regression

Authors: Arun K. Kuchibhotla, Lawrence D. Brown, Andreas Buja

Abstract: Ordinary least squares (OLS) linear regression is one of the most basic statistical techniques for data analysis. In the main stream literature and the statistical education, the study of linear regression is typically restricted to the case where the covariates are fixed, errors are mean zero Gaussians with variance independent of the (fixed) covariates. Even though OLS has been studied under mis… ▽ More Ordinary least squares (OLS) linear regression is one of the most basic statistical techniques for data analysis. In the main stream literature and the statistical education, the study of linear regression is typically restricted to the case where the covariates are fixed, errors are mean zero Gaussians with variance independent of the (fixed) covariates. Even though OLS has been studied under misspecification from as early as the 1960's, the implications have not yet caught up with the main stream literature and applied sciences. The present article is an attempt at a unified viewpoint that makes the various implications of misspecification stand out. △ Less

Submitted 27 September, 2018; originally announced September 2018.

Comments: 33 pages

Deterministic Inequalities for Smooth M-estimators

Authors: Arun Kumar Kuchibhotla

Abstract: Ever since the proof of asymptotic normality of maximum likelihood estimator by Cramer (1946), it has been understood that a basic technique of the Taylor series expansion suffices for asymptotics of $M$-estimators with smooth/differentiable loss function. Although the Taylor series expansion is a purely deterministic tool, the realization that the asymptotic normality results can also be made det… ▽ More Ever since the proof of asymptotic normality of maximum likelihood estimator by Cramer (1946), it has been understood that a basic technique of the Taylor series expansion suffices for asymptotics of $M$-estimators with smooth/differentiable loss function. Although the Taylor series expansion is a purely deterministic tool, the realization that the asymptotic normality results can also be made deterministic (and so finite sample) received far less attention. With the advent of big data and high-dimensional statistics, the need for finite sample results has increased. In this paper, we use the (well-known) Banach fixed point theorem to derive various deterministic inequalities that lead to the classical results when studied under randomness. In addition, we provide applications of these deterministic inequalities for crossvalidation/subsampling, marginal screening and uniform-in-submodel results that are very useful for post-selection inference and in the study of post-regularization estimators. Our results apply to many classical estimators, in particular, generalized linear models, non-linear regression and cox proportional hazards model. Extensions to non-smooth and constrained problems are also discussed. △ Less

Submitted 13 September, 2018; originally announced September 2018.

Comments: 49 pages

High-dimensional CLT: Improvements, Non-uniform Extensions and Large Deviations

Authors: Arun Kumar Kuchibhotla, Somabha Mukherjee, Debapratim Banerjee

Abstract: Central limit theorems (CLTs) for high-dimensional random vectors with dimension possibly growing with the sample size have received a lot of attention in the recent times. Chernozhukov et al. (2017) proved a Berry--Esseen type result for high-dimensional averages for the class of hyperrectangles and they proved that the rate of convergence can be upper bounded by $n^{-1/6}$ upto a polynomial fact… ▽ More Central limit theorems (CLTs) for high-dimensional random vectors with dimension possibly growing with the sample size have received a lot of attention in the recent times. Chernozhukov et al. (2017) proved a Berry--Esseen type result for high-dimensional averages for the class of hyperrectangles and they proved that the rate of convergence can be upper bounded by $n^{-1/6}$ upto a polynomial factor of $\log p$ (where $n$ represents the sample size and $p$ denotes the dimension). Convergence to zero of the bound requires $\log^7p = o(n)$. We improve upon their result which only requires $\log^4p = o(n)$ (in the best case). This improvement is made possible by a sharper dimension-free anti-concentration inequality for Gaussian process on a compact metric space. In addition, we prove two non-uniform variants of the high-dimensional CLT based on the large deviation and non-uniform CLT results for random variables in a Banach space by Bentkus, Ra{\v c}kauskas, and Paulauskas. We apply our results in the context of post-selection inference in linear regression and of empirical processes. △ Less

Submitted 24 June, 2019; v1 submitted 15 June, 2018; originally announced June 2018.

Comments: 76 pages

Valid Post-selection Inference in Assumption-lean Linear Regression

Authors: Arun Kumar Kuchibhotla, Lawrence D. Brown, Andreas Buja, Edward I. George, Linda Zhao

Abstract: Construction of valid statistical inference for estimators based on data-driven selection has received a lot of attention in the recent times. Berk et al. (2013) is possibly the first work to provide valid inference for Gaussian homoscedastic linear regression with fixed covariates under arbitrary covariate/variable selection. The setting is unrealistic and is extended by Bachoc et al. (2016) by r… ▽ More Construction of valid statistical inference for estimators based on data-driven selection has received a lot of attention in the recent times. Berk et al. (2013) is possibly the first work to provide valid inference for Gaussian homoscedastic linear regression with fixed covariates under arbitrary covariate/variable selection. The setting is unrealistic and is extended by Bachoc et al. (2016) by relaxing the distributional assumptions. A major drawback of the aforementioned works is that the construction of valid confidence regions is computationally intensive. In this paper, we first prove that post-selection inference is equivalent to simultaneous inference and then construct valid post-selection confidence regions which are computationally simple. Our construction is based on deterministic inequalities and apply to independent as well as dependent random variables without the requirement of correct distributional assumptions. Finally, we compare the volume of our confidence regions with the existing ones and show that under non-stochastic covariates, our regions are much smaller. △ Less

Submitted 11 June, 2018; originally announced June 2018.

Comments: 49 pages

Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression

Authors: Arun Kumar Kuchibhotla, Abhishek Chakrabortty

Abstract: Concentration inequalities form an essential toolkit in the study of high dimensional (HD) statistical methods. Most of the relevant statistics literature in this regard is based on sub-Gaussian or sub-exponential tail assumptions. In this paper, we first bring together various probabilistic inequalities for sums of independent random variables under much more general exponential type (namely sub-… ▽ More Concentration inequalities form an essential toolkit in the study of high dimensional (HD) statistical methods. Most of the relevant statistics literature in this regard is based on sub-Gaussian or sub-exponential tail assumptions. In this paper, we first bring together various probabilistic inequalities for sums of independent random variables under much more general exponential type (namely sub-Weibull) tail assumptions. These results extract a part sub-Gaussian tail behavior in finite samples, matching the asymptotics governed by the central limit theorem, and are compactly represented in terms of a new Orlicz quasi-norm - the Generalized Bernstein-Orlicz norm - that typifies such tail behaviors. We illustrate the usefulness of these inequalities through the analysis of four fundamental problems in HD statistics. In the first two problems, we study the rate of convergence of the sample covariance matrix in terms of the maximum elementwise norm and the maximum k-sub-matrix operator norm which are key quantities of interest in bootstrap, HD covariance matrix estimation and HD inference. The third example concerns the restricted eigenvalue condition, required in HD linear regression, which we verify for all sub-Weibull random vectors through a unified analysis, and also prove a more general result related to restricted strong convexity in the process. In the final example, we consider the Lasso estimator for linear regression and establish its rate of convergence under much weaker than usual tail assumptions (on the errors as well as the covariates), while also allowing for misspecified models and both fixed and random design. To our knowledge, these are the first such results for Lasso obtained in this generality. The common feature in all our results over all the examples is that the convergence rates under most exponential tails match the usual ones under sub-Gaussian assumptions. △ Less

Submitted 9 May, 2022; v1 submitted 7 April, 2018; originally announced April 2018.

Comments: 68 pages; Revised version; To appear in Information and Inference: A Journal of the IMA

MSC Class: 60G50; 62J05; 60B20; 62J07; 62E17; 60F05; 60E15

Journal ref: Information and Inference: A Journal of the IMA (2022), Vol. 11, No. 4, 1389-1456

Uniform-in-Submodel Bounds for Linear Regression in a Model Free Framework

Authors: Arun Kumar Kuchibhotla, Lawrence D. Brown, Andreas Buja, Edward I. George, Linda Zhao

Abstract: For the last two decades, high-dimensional data and methods have proliferated throughout the literature. Yet, the classical technique of linear regression has not lost its usefulness in applications. In fact, many high-dimensional estimation techniques can be seen as variable selection that leads to a smaller set of variables (a ``sub-model'') where classical linear regression applies. We analyze… ▽ More For the last two decades, high-dimensional data and methods have proliferated throughout the literature. Yet, the classical technique of linear regression has not lost its usefulness in applications. In fact, many high-dimensional estimation techniques can be seen as variable selection that leads to a smaller set of variables (a ``sub-model'') where classical linear regression applies. We analyze linear regression estimators resulting from model-selection by proving estimation error and linear representation bounds uniformly over sets of submodels. Based on deterministic inequalities, our results provide ``good'' rates when applied to both independent and dependent data. These results are useful in meaningfully interpreting the linear regression estimator obtained after exploring and reducing the variables and also in justifying post model-selection inference. All results are derived under no model assumptions and are non-asymptotic in nature. △ Less

Submitted 17 May, 2021; v1 submitted 15 February, 2018; originally announced February 2018.

Comments: Forthcoming at Econometric Theory

Semiparametric Efficiency in Convexity Constrained Single Index Model

Authors: Arun K. Kuchibhotla, Rohit K. Patra, Bodhisattva Sen

Abstract: We consider estimation and inference in a single index regression model with an unknown convex link function. We introduce a convex and Lipschitz constrained least squares estimator (CLSE) for both the parametric and the nonparametric components given independent and identically distributed observations. We prove the consistency and find the rates of convergence of the CLSE when the errors are ass… ▽ More We consider estimation and inference in a single index regression model with an unknown convex link function. We introduce a convex and Lipschitz constrained least squares estimator (CLSE) for both the parametric and the nonparametric components given independent and identically distributed observations. We prove the consistency and find the rates of convergence of the CLSE when the errors are assumed to have only $q \ge 2$ moments and are allowed to depend on the covariates. When $q\ge 5$, we establish $n^{-1/2}$-rate of convergence and asymptotic normality of the estimator of the parametric component. Moreover, the CLSE is proved to be semiparametrically efficient if the errors happen to be homoscedastic. {We develop and implement a numerically stable and computationally fast algorithm to compute our proposed estimator in the R package~\texttt{simest}}. We illustrate our methodology through extensive simulations and data analysis. Finally, our proof of efficiency is geometric and provides a general framework that can be used to prove efficiency of estimators in a wide variety of semiparametric models even when they do not satisfy the efficient score equation directly. △ Less

Submitted 13 January, 2021; v1 submitted 31 July, 2017; originally announced August 2017.

Comments: Removed the density bounded away from zero assumption in assumption (A5). Weakened assumption (B2)

Models as Approximations II: A Model-Free Theory of Parametric Regression

Authors: Andreas Buja, Lawrence Brown, Arun Kumar Kuchibhotla, Richard Berk, Ed George, Linda Zhao

Abstract: We develop a model-free theory of general types of parametric regression for iid observations. The theory replaces the parameters of parametric models with statistical functionals, to be called "regression functionals'', defined on large non-parametric classes of joint $\xy$ distributions, without assuming a correct model. Parametric models are reduced to heuristics to suggest plausible objective… ▽ More We develop a model-free theory of general types of parametric regression for iid observations. The theory replaces the parameters of parametric models with statistical functionals, to be called "regression functionals'', defined on large non-parametric classes of joint $\xy$ distributions, without assuming a correct model. Parametric models are reduced to heuristics to suggest plausible objective functions. An example of a regression functional is the vector of slopes of linear equations fitted by OLS to largely arbitrary $\xy$ distributions, without assuming a linear model (see Part~I). More generally, regression functionals can be defined by minimizing objective functions or solving estimating equations at joint $\xy$ distributions. In this framework it is possible to achieve the following: (1)~define a notion of well-specification for regression functionals that replaces the notion of correct specification of models, (2)~propose a well-specification diagnostic for regression functionals based on reweighting distributions and data, (3)~decompose sampling variability of regression functionals into two sources, one due to the conditional response distribution and another due to the regressor distribution interacting with misspecification, both of order $N^{-1/2}$, (4)~exhibit plug-in/sandwich estimators of standard error as limit cases of $\xy$ bootstrap estimators, and (5)~provide theoretical heuristics to indicate that $\xy$ bootstrap standard errors may generally be more stable than sandwich estimators. △ Less

Submitted 6 July, 2019; v1 submitted 10 December, 2016; originally announced December 2016.

Comments: Submitted

MSC Class: 62A01