-
FDR control and FDP bounds for conformal link prediction
Authors:
Gilles Blanchard,
Guillermo Durand,
Ariane Marandon-Carlhian,
Romain Périer
Abstract:
In Marandon (2023), the author introduces a procedure to detect true edges from a partially observed graph using a conformal prediction fashion: first computing scores from a trained function, deriving conformal p-values from them and finally applying a multiple testing procedure. In this paper, we prove that the resulting procedure indeed controls the FDR, and we also derive uniform FDP bounds, t…
▽ More
In Marandon (2023), the author introduces a procedure to detect true edges from a partially observed graph using a conformal prediction fashion: first computing scores from a trained function, deriving conformal p-values from them and finally applying a multiple testing procedure. In this paper, we prove that the resulting procedure indeed controls the FDR, and we also derive uniform FDP bounds, thanks to an exchangeability argument and the previous work of Marandon et al. (2022).
△ Less
Submitted 3 April, 2024;
originally announced April 2024.
-
Moment inequalities for sums of weakly dependent random fields
Authors:
Gilles Blanchard,
Alexandra Carpentier,
Oleksandr Zadorozhnyi
Abstract:
We derive both Azuma-Hoeffding and Burkholder-type inequalities for partial sums over a rectangular grid of dimension $d$ of a random field satisfying a weak dependency assumption of projective type: the difference between the expectation of an element of the random field and its conditional expectation given the rest of the field at a distance more than $δ$ is bounded, in $L^p$ distance, by a kno…
▽ More
We derive both Azuma-Hoeffding and Burkholder-type inequalities for partial sums over a rectangular grid of dimension $d$ of a random field satisfying a weak dependency assumption of projective type: the difference between the expectation of an element of the random field and its conditional expectation given the rest of the field at a distance more than $δ$ is bounded, in $L^p$ distance, by a known decreasing function of $δ$. The analysis is based on the combination of a multi-scale approximation of random sums by martingale difference sequences, and of a careful decomposition of the domain. The obtained results extend previously known bounds under comparable hypotheses, and do not use the assumption of commuting filtrations.
△ Less
Submitted 28 June, 2023;
originally announced June 2023.
-
False discovery proportion envelopes with consistency
Authors:
Iqraa Meah,
Gilles Blanchard,
Etienne Roquain
Abstract:
We provide new false discovery proportion (FDP) confidence envelopes in several multiple testing settings relevant to modern high dimensional-data methods. We revisit the scenarios considered in the recent work of \cite{katsevich2020simultaneous}(top-$k$, preordered -- including knockoffs -- , online) with a particular emphasis on obtaining FDP bounds that have both non-asymptotical coverage and a…
▽ More
We provide new false discovery proportion (FDP) confidence envelopes in several multiple testing settings relevant to modern high dimensional-data methods. We revisit the scenarios considered in the recent work of \cite{katsevich2020simultaneous}(top-$k$, preordered -- including knockoffs -- , online) with a particular emphasis on obtaining FDP bounds that have both non-asymptotical coverage and asymptotical consistency, i.e. converge below the desired level $α$ when applied to a classical $α$-level false discovery rate (FDR) controlling procedure. This way, we derive new bounds that provide improvements over existing ones, both theoretically and practically, and are suitable for situations where at least a moderate number of rejections is expected. These improvements are illustrated with numerical experiments and real data examples. In particular, the improvement is significant in the knockoff setting, which shows the impact of the method for practical use. As side results, we introduce a new confidence envelope for the empirical cumulative distribution function of i.i.d. uniform variables and we provide new power results in sparse cases, both being of independent interest.
△ Less
Submitted 13 June, 2023;
originally announced June 2023.
-
Statistical learning on measures: an application to persistence diagrams
Authors:
Olympio Hacquard,
Gilles Blanchard,
Clément Levrard
Abstract:
We consider a binary supervised learning classification problem where instead of having data in a finite-dimensional Euclidean space, we observe measures on a compact space $\mathcal{X}$. Formally, we observe data $D_N = (μ_1, Y_1), \ldots, (μ_N, Y_N)$ where $μ_i$ is a measure on $\mathcal{X}$ and $Y_i$ is a label in $\{0, 1\}$. Given a set $\mathcal{F}$ of base-classifiers on $\mathcal{X}$, we bu…
▽ More
We consider a binary supervised learning classification problem where instead of having data in a finite-dimensional Euclidean space, we observe measures on a compact space $\mathcal{X}$. Formally, we observe data $D_N = (μ_1, Y_1), \ldots, (μ_N, Y_N)$ where $μ_i$ is a measure on $\mathcal{X}$ and $Y_i$ is a label in $\{0, 1\}$. Given a set $\mathcal{F}$ of base-classifiers on $\mathcal{X}$, we build corresponding classifiers in the space of measures. We provide upper and lower bounds on the Rademacher complexity of this new class of classifiers that can be expressed simply in terms of corresponding quantities for the class $\mathcal{F}$. If the measures $μ_i$ are uniform over a finite set, this classification task boils down to a multi-instance learning problem. However, our approach allows more flexibility and diversity in the input data we can deal with. While such a framework has many possible applications, this work strongly emphasizes on classifying data via topological descriptors called persistence diagrams. These objects are discrete measures on $\mathbb{R}^2$, where the coordinates of each point correspond to the range of scales at which a topological feature exists. We will present several classifiers on measures and show how they can heuristically and theoretically enable a good classification performance in various settings in the case of persistence diagrams.
△ Less
Submitted 31 May, 2023; v1 submitted 15 March, 2023;
originally announced March 2023.
-
Constant regret for sequence prediction with limited advice
Authors:
El Mehdi Saad,
G. Blanchard
Abstract:
We investigate the problem of cumulative regret minimization for individual sequence prediction with respect to the best expert in a finite family of size K under limited access to information. We assume that in each round, the learner can predict using a convex combination of at most p experts for prediction, then they can observe a posteriori the losses of at most m experts. We assume that the l…
▽ More
We investigate the problem of cumulative regret minimization for individual sequence prediction with respect to the best expert in a finite family of size K under limited access to information. We assume that in each round, the learner can predict using a convex combination of at most p experts for prediction, then they can observe a posteriori the losses of at most m experts. We assume that the loss function is range-bounded and exp-concave. In the standard multi-armed bandits setting, when the learner is allowed to play only one expert per round and observe only its feedback, known optimal regret bounds are of the order O($\sqrt$ KT). We show that allowing the learner to play one additional expert per round and observe one additional feedback improves substantially the guarantees on regret. We provide a strategy combining only p = 2 experts per round for prediction and observing m $\ge$ 2 experts' losses. Its randomized regret (wrt. internal randomization of the learners' strategy) is of order O (K/m) log(K$δ$ --1) with probability 1 -- $δ$, i.e., is independent of the horizon T ("constant" or "fast rate" regret) if (p $\ge$ 2 and m $\ge$ 3). We prove that this rate is optimal up to a logarithmic factor in K. In the case p = m = 2, we provide an upper bound of order O(K 2 log(K$δ$ --1)), with probability 1 -- $δ$. Our strategies do not require any prior knowledge of the horizon T nor of the confidence parameter $δ$. Finally, we show that if the learner is constrained to observe only one expert feedback per round, the worst-case regret is the "slow rate" $Ω$($\sqrt$ KT), suggesting that synchronous observation of at least two experts per round is necessary to have a constant regret.
△ Less
Submitted 5 October, 2022;
originally announced October 2022.
-
Fast rates for prediction with limited expert advice
Authors:
El Mehdi Saad,
Gilles Blanchard
Abstract:
We investigate the problem of minimizing the excess generalization error with respect to the best expert prediction in a finite family in the stochastic setting, under limited access to information. We assume that the learner only has access to a limited number of expert advices per training round, as well as for prediction. Assuming that the loss function is Lipschitz and strongly convex, we show…
▽ More
We investigate the problem of minimizing the excess generalization error with respect to the best expert prediction in a finite family in the stochastic setting, under limited access to information. We assume that the learner only has access to a limited number of expert advices per training round, as well as for prediction. Assuming that the loss function is Lipschitz and strongly convex, we show that if we are allowed to see the advice of only one expert per round for T rounds in the training phase, or to use the advice of only one expert for prediction in the test phase, the worst-case excess risk is $Ω$(1/ $\sqrt$ T) with probability lower bounded by a constant. However, if we are allowed to see at least two actively chosen expert advices per training round and use at least two experts for prediction, the fast rate O(1/T) can be achieved. We design novel algorithms achieving this rate in this setting, and in the setting where the learner has a budget constraint on the total number of observed expert advices, and give precise instance-dependent bounds on the number of training rounds and queries needed to achieve a given generalization error precision.
△ Less
Submitted 27 October, 2021;
originally announced October 2021.
-
Topologically penalized regression on manifolds
Authors:
Olympio Hacquard,
Krishnakumar Balasubramanian,
Gilles Blanchard,
Clément Levrard,
Wolfgang Polonik
Abstract:
We study a regression problem on a compact manifold M. In order to take advantage of the underlying geometry and topology of the data, the regression task is performed on the basis of the first several eigenfunctions of the Laplace-Beltrami operator of the manifold, that are regularized with topological penalties. The proposed penalties are based on the topology of the sub-level sets of either the…
▽ More
We study a regression problem on a compact manifold M. In order to take advantage of the underlying geometry and topology of the data, the regression task is performed on the basis of the first several eigenfunctions of the Laplace-Beltrami operator of the manifold, that are regularized with topological penalties. The proposed penalties are based on the topology of the sub-level sets of either the eigenfunctions or the estimated function. The overall approach is shown to yield promising and competitive performance on various applications to both synthetic and real data sets. We also provide theoretical guarantees on the regression function estimates, on both its prediction error and its smoothness (in a topological sense). Taken together, these results support the relevance of our approach in the case where the targeted function is ''topologically smooth''.
△ Less
Submitted 10 June, 2022; v1 submitted 26 October, 2021;
originally announced October 2021.
-
Error rate control for classification rules in multiclass mixture models
Authors:
Tristan Mary-Huard,
Vittorio Perduca,
Gilles Blanchard,
Martin-Magniette Marie-Laure
Abstract:
In the context of finite mixture models one considers the problem of classifying as many observations as possible in the classes of interest while controlling the classification error rate in these same classes. Similar to what is done in the framework of statistical test theory, different type I and type II-like classification error rates can be defined, along with their associated optimal rules,…
▽ More
In the context of finite mixture models one considers the problem of classifying as many observations as possible in the classes of interest while controlling the classification error rate in these same classes. Similar to what is done in the framework of statistical test theory, different type I and type II-like classification error rates can be defined, along with their associated optimal rules, where optimality is defined as minimizing type II error rate while controlling type I error rate at some nominal level. It is first shown that finding an optimal classification rule boils down to searching an optimal region in the observation space where to apply the classical Maximum A Posteriori (MAP) rule. Depending on the misclassification rate to be controlled, the shape of the optimal region is provided, along with a heuristic to compute the optimal classification rule in practice. In particular, a multiclass FDR-like optimal rule is defined and compared to the thresholded MAP rules that is used in most applications. It is shown on both simulated and real datasets that the FDR-like optimal rule may be significantly less conservative than the thresholded MAP rule.
△ Less
Submitted 29 September, 2021;
originally announced September 2021.
-
Nonasymptotic one-and two-sample tests in high dimension with unknown covariance structure
Authors:
Gilles Blanchard,
Jean-Baptiste Fermanian
Abstract:
Let $\mathbf{X} = (X_i)_{1\leq i \leq n}$ be an i.i.d. sample of square-integrable variables in $\mathbb{R}^d$, \GB{with common expectation $μ$ and covariance matrix $Σ$, both unknown.} We consider the problem of testing if $μ$ is $η$-close to zero, i.e. $\|μ\| \leq η$ against $\|μ\| \geq (η+ δ)$; we also tackle the more general two-sample mean closeness (also known as {\em relevant difference}) t…
▽ More
Let $\mathbf{X} = (X_i)_{1\leq i \leq n}$ be an i.i.d. sample of square-integrable variables in $\mathbb{R}^d$, \GB{with common expectation $μ$ and covariance matrix $Σ$, both unknown.} We consider the problem of testing if $μ$ is $η$-close to zero, i.e. $\|μ\| \leq η$ against $\|μ\| \geq (η+ δ)$; we also tackle the more general two-sample mean closeness (also known as {\em relevant difference}) testing problem. The aim of this paper is to obtain nonasymptotic upper and lower bounds on the minimal separation distance $δ$ such that we can control both the Type I and Type II errors at a given level. The main technical tools are concentration inequalities, first for a suitable estimator of $\|μ\|^2$ used a test statistic, and secondly for estimating the operator and Frobenius norms of $Σ$ coming into the quantiles of said test statistic. These properties are obtained for Gaussian and bounded distributions. A particular attention is given to the dependence in the pseudo-dimension $d_*$ of the distribution, defined as $d_* := \|Σ\|_2^2/\|Σ\|_\infty^2$. In particular, for $η=0$, the minimum separation distance is $Θ( d_*^{\frac{1}{4}}\sqrt{\|Σ\|_\infty/n})$, in contrast with the minimax estimation distance for $μ$, which is $Θ(d_e^{\frac{1}{2}}\sqrt{\|Σ\|_\infty/n})$ (where $d_e:=\|Σ\|_1/\|Σ\|_\infty$). This generalizes a phenomenon spelled out in particular by Baraud (2002).
△ Less
Submitted 8 October, 2021; v1 submitted 1 September, 2021;
originally announced September 2021.
-
High-Dimensional Multi-Task Averaging and Application to Kernel Mean Embedding
Authors:
Hannah Marienwald,
Jean-Baptiste Fermanian,
Gilles Blanchard
Abstract:
We propose an improved estimator for the multi-task averaging problem, whose goal is the joint estimation of the means of multiple distributions using separate, independent data sets. The naive approach is to take the empirical mean of each data set individually, whereas the proposed method exploits similarities between tasks, without any related information being known in advance. First, for each…
▽ More
We propose an improved estimator for the multi-task averaging problem, whose goal is the joint estimation of the means of multiple distributions using separate, independent data sets. The naive approach is to take the empirical mean of each data set individually, whereas the proposed method exploits similarities between tasks, without any related information being known in advance. First, for each data set, similar or neighboring means are determined from the data by multiple testing. Then each naive estimator is shrunk towards the local average of its neighbors. We prove theoretically that this approach provides a reduction in mean squared error. This improvement can be significant when the dimension of the input space is large, demonstrating a "blessing of dimensionality" phenomenon. An application of this approach is the estimation of multiple kernel mean embeddings, which plays an important role in many modern applications. The theoretical results are verified on artificial and real world data.
△ Less
Submitted 13 November, 2020;
originally announced November 2020.
-
Statistical Learning Guarantees for Compressive Clustering and Compressive Mixture Modeling
Authors:
Rémi Gribonval,
Gilles Blanchard,
Nicolas Keriven,
Yann Traonmilin
Abstract:
We provide statistical learning guarantees for two unsupervised learning tasks in the context of compressive statistical learning, a general framework for resource-efficient large-scale learning that we introduced in a companion paper.The principle of compressive statistical learning is to compress a training collection, in one pass, into a low-dimensional sketch (a vector of random empirical gen…
▽ More
We provide statistical learning guarantees for two unsupervised learning tasks in the context of compressive statistical learning, a general framework for resource-efficient large-scale learning that we introduced in a companion paper.The principle of compressive statistical learning is to compress a training collection, in one pass, into a low-dimensional sketch (a vector of random empirical generalized moments) that captures the information relevant to the considered learning task. We explicitly describe and analyze random feature functions which empirical averages preserve the needed information for compressive clustering and compressive Gaussian mixture modeling with fixed known variance, and establish sufficient sketch sizes given the problem dimensions.
△ Less
Submitted 17 August, 2021; v1 submitted 17 April, 2020;
originally announced April 2020.
-
On agnostic post hoc approaches to false positive control
Authors:
Gilles Blanchard,
Pierre Neuvial,
Etienne Roquain
Abstract:
This document is a book chapter which gives a partial survey on post hoc approaches to false positive control.
This document is a book chapter which gives a partial survey on post hoc approaches to false positive control.
△ Less
Submitted 25 October, 2019;
originally announced October 2019.
-
Volume Doubling Condition and a Local Poincaré Inequality on Unweighted Random Geometric Graphs
Authors:
Franziska Göbel,
Gilles Blanchard
Abstract:
The aim of this paper is to establish two fundamental measure-metric properties of particular random geometric graphs. We consider $\varepsilon$-neighborhood graphs whose vertices are drawn independently and identically distributed from a common distribution defined on a regular submanifold of $\mathbb{R}^K$. We show that a volume doubling condition (VD) and local Poincaré inequality (LPI) hold fo…
▽ More
The aim of this paper is to establish two fundamental measure-metric properties of particular random geometric graphs. We consider $\varepsilon$-neighborhood graphs whose vertices are drawn independently and identically distributed from a common distribution defined on a regular submanifold of $\mathbb{R}^K$. We show that a volume doubling condition (VD) and local Poincaré inequality (LPI) hold for the random geometric graph (with high probability, and uniformly over all shortest path distance balls in a certain radius range) under suitable regularity conditions of the underlying submanifold and the sampling distribution.
△ Less
Submitted 23 March, 2020; v1 submitted 6 July, 2019;
originally announced July 2019.
-
Lepskii Principle in Supervised Learning
Authors:
Gilles Blanchard,
Peter Mathé,
Nicole Mücke
Abstract:
In the setting of supervised learning using reproducing kernel methods, we propose a data-dependent regularization parameter selection rule that is adaptive to the unknown regularity of the target function and is optimal both for the least-square (prediction) error and for the reproducing kernel Hilbert space (reconstruction) norm error. It is based on a modified Lepskii balancing principle using…
▽ More
In the setting of supervised learning using reproducing kernel methods, we propose a data-dependent regularization parameter selection rule that is adaptive to the unknown regularity of the target function and is optimal both for the least-square (prediction) error and for the reproducing kernel Hilbert space (reconstruction) norm error. It is based on a modified Lepskii balancing principle using a varying family of norms.
△ Less
Submitted 26 May, 2019;
originally announced May 2019.
-
Convergence analysis of Tikhonov regularization for non-linear statistical inverse learning problems
Authors:
Abhishake Rastogi,
Gilles Blanchard,
Peter Mathé
Abstract:
We study a non-linear statistical inverse learning problem, where we observe the noisy image of a quantity through a non-linear operator at some random design points. We consider the widely used Tikhonov regularization (or method of regularization, MOR) approach to reconstruct the estimator of the quantity for the non-linear ill-posed inverse problem. The estimator is defined as the minimizer of a…
▽ More
We study a non-linear statistical inverse learning problem, where we observe the noisy image of a quantity through a non-linear operator at some random design points. We consider the widely used Tikhonov regularization (or method of regularization, MOR) approach to reconstruct the estimator of the quantity for the non-linear ill-posed inverse problem. The estimator is defined as the minimizer of a Tikhonov functional, which is the sum of a data misfit term and a quadratic penalty term. We develop a theoretical analysis for the minimizer of the Tikhonov regularization scheme using the ansatz of reproducing kernel Hilbert spaces. We discuss optimal rates of convergence for the proposed scheme, uniformly over classes of admissible solutions, defined through appropriate source conditions.
△ Less
Submitted 1 March, 2019; v1 submitted 14 February, 2019;
originally announced February 2019.
-
Post hoc false positive control for spatially structured hypotheses
Authors:
Guillermo Durand,
Gilles Blanchard,
Pierre Neuvial,
Etienne Roquain
Abstract:
In a high dimensional multiple testing framework, we present new confidence bounds on the false positives contained in subsets S of selected null hypotheses. The coverage probability holds simultaneously over all subsets S, which means that the obtained confidence bounds are post hoc. Therefore, S can be chosen arbitrarily, possibly by using the data set several times. We focus in this paper speci…
▽ More
In a high dimensional multiple testing framework, we present new confidence bounds on the false positives contained in subsets S of selected null hypotheses. The coverage probability holds simultaneously over all subsets S, which means that the obtained confidence bounds are post hoc. Therefore, S can be chosen arbitrarily, possibly by using the data set several times. We focus in this paper specifically on the case where the null hypotheses are spatially structured. Our method is based on recent advances in post hoc inference and particularly on the general methodology of Blanchard et al. (2017); we build confidence bounds for some pre-specified forest-structured subsets {R k , k $\in$ K}, called the reference family, and then we deduce a bound for any subset S by interpolation. The proposed bounds are shown to improve substantially previous ones when the signal is locally structured. Our findings are supported both by theoretical results and numerical experiments. Moreover, we show that our bound can be obtained by a low-complexity algorithm, which makes our approach completely operational for a practical use. The proposed bounds are implemented in the open-source R package sansSouci.
△ Less
Submitted 19 September, 2018; v1 submitted 4 July, 2018;
originally announced July 2018.
-
On the Post Selection Inference constant under Restricted Isometry Properties
Authors:
François Bachoc,
Gilles Blanchard,
Pierre Neuvial
Abstract:
Uniformly valid confidence intervals post model selection in regression can be constructed based on Post-Selection Inference (PoSI) constants. PoSI constants are minimal for orthogonal design matrices, and can be upper bounded in function of the sparsity of the set of models under consideration, for generic design matrices. In order to improve on these generic sparse upper bounds, we consider desi…
▽ More
Uniformly valid confidence intervals post model selection in regression can be constructed based on Post-Selection Inference (PoSI) constants. PoSI constants are minimal for orthogonal design matrices, and can be upper bounded in function of the sparsity of the set of models under consideration, for generic design matrices. In order to improve on these generic sparse upper bounds, we consider design matrices satisfying a Restricted Isometry Property (RIP) condition. We provide a new upper bound on the PoSI constant in this setting. This upper bound is an explicit function of the RIP constant of the design matrix, thereby giving an interpolation between the orthogonal setting and the generic sparse setting. We show that this upper bound is asymptotically optimal in many settings by constructing a matching lower bound.
△ Less
Submitted 22 November, 2018; v1 submitted 20 April, 2018;
originally announced April 2018.
-
Concentration of weakly dependent Banach-valued sums and applications to statistical learning methods
Authors:
Gilles Blanchard,
Oleksandr Zadorozhnyi
Abstract:
We obtain a Bernstein-type inequality for sums of Banach-valued random variables satisfying a weak dependence assumption of general type and under certain smoothness assumptions of the underlying Banach norm. We use this inequality in order to investigate in the asymptotical regime the error upper bounds for the broad family of spectral regularization methods for reproducing kernel decision rules,…
▽ More
We obtain a Bernstein-type inequality for sums of Banach-valued random variables satisfying a weak dependence assumption of general type and under certain smoothness assumptions of the underlying Banach norm. We use this inequality in order to investigate in the asymptotical regime the error upper bounds for the broad family of spectral regularization methods for reproducing kernel decision rules, when trained on a sample coming from a $τ-$mixing process.
△ Less
Submitted 9 December, 2018; v1 submitted 5 December, 2017;
originally announced December 2017.
-
Early stop** for statistical inverse problems via truncated SVD estimation
Authors:
Gilles Blanchard,
Marc Hoffmann,
Markus Reiß
Abstract:
We consider truncated SVD (or spectral cut-off, projection) estimators for a prototypical statistical inverse problem in dimension $D$. Since calculating the singular value decomposition (SVD) only for the largest singular values is much less costly than the full SVD, our aim is to select a data-driven truncation level $\widehat m\in\{1,\ldots,D\}$ only based on the knowledge of the first…
▽ More
We consider truncated SVD (or spectral cut-off, projection) estimators for a prototypical statistical inverse problem in dimension $D$. Since calculating the singular value decomposition (SVD) only for the largest singular values is much less costly than the full SVD, our aim is to select a data-driven truncation level $\widehat m\in\{1,\ldots,D\}$ only based on the knowledge of the first $\widehat m$ singular values and vectors. We analyse in detail whether sequential {\it early stop**} rules of this type can preserve statistical optimality. Information-constrained lower bounds and matching upper bounds for a residual based stop** rule are provided, which give a clear picture in which situation optimal sequential adaptation is feasible. Finally, a hybrid two-step approach is proposed which allows for classical oracle inequalities while considerably reducing numerical complexity.
△ Less
Submitted 7 September, 2018; v1 submitted 19 October, 2017;
originally announced October 2017.
-
Compressive Statistical Learning with Random Feature Moments
Authors:
Rémi Gribonval,
Gilles Blanchard,
Nicolas Keriven,
Yann Traonmilin
Abstract:
We describe a general framework -- compressive statistical learning -- for resource-efficient large-scale learning: the training collection is compressed in one pass into a low-dimensional sketch (a vector of random empirical generalized moments) that captures the information relevant to the considered learning task. A near-minimizer of the risk is computed from the sketch through the solution of…
▽ More
We describe a general framework -- compressive statistical learning -- for resource-efficient large-scale learning: the training collection is compressed in one pass into a low-dimensional sketch (a vector of random empirical generalized moments) that captures the information relevant to the considered learning task. A near-minimizer of the risk is computed from the sketch through the solution of a nonlinear least squares problem. We investigate sufficient sketch sizes to control the generalization error of this procedure. The framework is illustrated on compressive PCA, compressive clustering, and compressive Gaussian mixture Modeling with fixed known variance. The latter two are further developed in a companion paper.
△ Less
Submitted 22 June, 2021; v1 submitted 22 June, 2017;
originally announced June 2017.
-
Post hoc inference via joint family-wise error rate control
Authors:
Gilles Blanchard,
Pierre Neuvial,
Etienne Roquain
Abstract:
We introduce a general methodology for post hoc inference in a large-scale multiple testing framework. The approach is called "user-agnostic" in the sense that the statistical guarantee on the number of correct rejections holds for any set of candidate items selected by the user (after having seen the data). This task is investigated by defining a suitable criterion, named the joint-family-wise-er…
▽ More
We introduce a general methodology for post hoc inference in a large-scale multiple testing framework. The approach is called "user-agnostic" in the sense that the statistical guarantee on the number of correct rejections holds for any set of candidate items selected by the user (after having seen the data). This task is investigated by defining a suitable criterion, named the joint-family-wise-error rate (JER for short). We propose several procedures for controlling the JER, with a special focus on incorporating dependencies while adapting to the unknown quantity of signal (via a step-down approach). We show that our proposed setting incorporates as particular cases a version of the higher criticism as well as the closed testing based approach of Goeman and Solari (2011). Our theoretical statements are supported by numerical experiments.
△ Less
Submitted 8 January, 2018; v1 submitted 7 March, 2017;
originally announced March 2017.
-
Minimax Euclidean Separation Rates for Testing Convex Hypotheses in $\mathbb{R}^d$
Authors:
Gilles Blanchard,
Alexandra Carpentier,
Maurilio Gutzeit
Abstract:
We consider composite-composite testing problems for the expectation in the Gaussian sequence model where the null hypothesis corresponds to a convex subset $\mathcal{C}$ of $\mathbb{R}^d$. We adopt a minimax point of view and our primary objective is to describe the smallest Euclidean distance between the null and alternative hypotheses such that there is a test with small total error probability…
▽ More
We consider composite-composite testing problems for the expectation in the Gaussian sequence model where the null hypothesis corresponds to a convex subset $\mathcal{C}$ of $\mathbb{R}^d$. We adopt a minimax point of view and our primary objective is to describe the smallest Euclidean distance between the null and alternative hypotheses such that there is a test with small total error probability. In particular, we focus on the dependence of this distance on the dimension $d$ and the sample size/variance parameter $n$ giving rise to the minimax separation rate. In this paper we discuss lower and upper bounds on this rate for different smooth and non- smooth choices for $\mathcal{C}$.
△ Less
Submitted 23 August, 2018; v1 submitted 13 February, 2017;
originally announced February 2017.
-
Parallelizing Spectral Algorithms for Kernel Learning
Authors:
Gilles Blanchard,
Nicole Mücke
Abstract:
We consider a distributed learning approach in supervised learning for a large class of spectral regularization methods in an RKHS framework. The data set of size n is partitioned into $m=O(n^α)$ disjoint subsets. On each subset, some spectral regularization method (belonging to a large class, including in particular Kernel Ridge Regression, $L^2$-boosting and spectral cut-off) is applied. The reg…
▽ More
We consider a distributed learning approach in supervised learning for a large class of spectral regularization methods in an RKHS framework. The data set of size n is partitioned into $m=O(n^α)$ disjoint subsets. On each subset, some spectral regularization method (belonging to a large class, including in particular Kernel Ridge Regression, $L^2$-boosting and spectral cut-off) is applied. The regression function $f$ is then estimated via simple averaging, leading to a substantial reduction in computation time. We show that minimax optimal rates of convergence are preserved if m grows sufficiently slowly (corresponding to an upper bound for $α$) as $n \to \infty$, depending on the smoothness assumptions on $f$ and the intrinsic dimensionality. In spirit, our approach is classical.
△ Less
Submitted 9 August, 2017; v1 submitted 24 October, 2016;
originally announced October 2016.
-
Convergence rates of Kernel Conjugate Gradient for random design regression
Authors:
Gilles Blanchard,
Nicole Krämer
Abstract:
We prove statistical rates of convergence for kernel-based least squares regression from i.i.d. data using a conjugate gradient algorithm, where regularization against overfitting is obtained by early stop**. This method is related to Kernel Partial Least Squares, a regression method that combines supervised dimensionality reduction with least squares projection. Following the setting introduced…
▽ More
We prove statistical rates of convergence for kernel-based least squares regression from i.i.d. data using a conjugate gradient algorithm, where regularization against overfitting is obtained by early stop**. This method is related to Kernel Partial Least Squares, a regression method that combines supervised dimensionality reduction with least squares projection. Following the setting introduced in earlier related literature, we study so-called "fast convergence rates" depending on the regularity of the target regression function (measured by a source condition in terms of the kernel integral operator) and on the effective dimensionality of the data mapped into the kernel space. We obtain upper bounds, essentially matching known minimax lower bounds, for the $\mathcal{L}^2$ (prediction) norm as well as for the stronger Hilbert norm, if the true regression function belongs to the reproducing kernel Hilbert space. If the latter assumption is not fulfilled, we obtain similar convergence rates for appropriate norms, provided additional unlabeled data are available.
△ Less
Submitted 8 July, 2016;
originally announced July 2016.
-
Optimal adaptation for early stop** in statistical inverse problems
Authors:
Gilles Blanchard,
Marc Hoffmann,
Markus Reiß
Abstract:
For linear inverse problems $Y=\mathsf{A}μ+ξ$, it is classical to recover the unknown signal $μ$ by iterative regularisation methods $(\widehat μ^{(m)}, m=0,1,\ldots)$ and halt at a data-dependent iteration $τ$ using some stop** rule, typically based on a discrepancy principle, so that the weak (or prediction) squared-error $\|\mathsf{A}(\widehat μ^{(τ)}-μ)\|^2$ is controlled. In the context of…
▽ More
For linear inverse problems $Y=\mathsf{A}μ+ξ$, it is classical to recover the unknown signal $μ$ by iterative regularisation methods $(\widehat μ^{(m)}, m=0,1,\ldots)$ and halt at a data-dependent iteration $τ$ using some stop** rule, typically based on a discrepancy principle, so that the weak (or prediction) squared-error $\|\mathsf{A}(\widehat μ^{(τ)}-μ)\|^2$ is controlled. In the context of statistical estimation with stochastic noise $ξ$, we study oracle adaptation (that is, compared to the best possible stop** iteration) in strong squared-error $E[\|\hat μ^{(τ)}-μ\|^2]$.
For a residual-based stop** rule oracle adaptation bounds are established for general spectral regularisation methods. The proofs use bias and variance transfer techniques from weak prediction error to strong $L^2$-error, as well as convexity arguments and concentration bounds for the stochastic part. Adaptive early stop** for the Landweber method is studied in further detail and illustrated numerically.
△ Less
Submitted 26 October, 2017; v1 submitted 24 June, 2016;
originally announced June 2016.
-
Optimal learning rates for Kernel Conjugate Gradient regression
Authors:
Gilles Blanchard,
Nicole Kraemer
Abstract:
We prove rates of convergence in the statistical sense for kernel-based least squares regression using a conjugate gradient algorithm, where regularization against overfitting is obtained by early stop**. This method is directly related to Kernel Partial Least Squares, a regression method that combines supervised dimensionality reduction with least squares projection. The rates depend on two key…
▽ More
We prove rates of convergence in the statistical sense for kernel-based least squares regression using a conjugate gradient algorithm, where regularization against overfitting is obtained by early stop**. This method is directly related to Kernel Partial Least Squares, a regression method that combines supervised dimensionality reduction with least squares projection. The rates depend on two key quantities: first, on the regularity of the target regression function and second, on the intrinsic dimensionality of the data mapped into the kernel space. Lower bounds on attainable rates depending on these two quantities were established in earlier literature, and we obtain upper bounds for the considered method that match these lower bounds (up to a log factor) if the true regression function belongs to the reproducing kernel Hilbert space. If this assumption is not fulfilled, we obtain similar convergence rates provided additional unlabeled data are available. The order of the learning rates match state-of-the-art results that were recently obtained for least squares support vector machines and for linear regularization operators.
△ Less
Submitted 29 September, 2010;
originally announced September 2010.
-
Kernel Partial Least Squares is Universally Consistent
Authors:
Gilles Blanchard,
Nicole Kraemer
Abstract:
We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as the solution of a global cost minimization procedure over a fixed model nor i…
▽ More
We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as the solution of a global cost minimization procedure over a fixed model nor is it a linear estimator. Instead, approximate solutions are constructed by projections onto a nested set of data-dependent subspaces. To prove consistency, we exploit the known fact that Partial Least Squares is equivalent to the conjugate gradient algorithm in combination with early stop**. The choice of the stop** rule (number of iterations) is a crucial point. We study two empirical stop** rules. The first one monitors the estimation error in each iteration step of Partial Least Squares, and the second one estimates the empirical complexity in terms of a condition number. Both stop** rules lead to universally consistent estimators provided the kernel is universal.
△ Less
Submitted 14 January, 2010; v1 submitted 25 February, 2009;
originally announced February 2009.
-
Statistical performance of support vector machines
Authors:
Gilles Blanchard,
Olivier Bousquet,
Pascal Massart
Abstract:
The support vector machine (SVM) algorithm is well known to the computer learning community for its very good practical results. The goal of the present paper is to study this algorithm from a statistical perspective, using tools of concentration theory and empirical processes. Our main result builds on the observation made by other authors that the SVM can be viewed as a statistical regularizat…
▽ More
The support vector machine (SVM) algorithm is well known to the computer learning community for its very good practical results. The goal of the present paper is to study this algorithm from a statistical perspective, using tools of concentration theory and empirical processes. Our main result builds on the observation made by other authors that the SVM can be viewed as a statistical regularization procedure. From this point of view, it can also be interpreted as a model selection principle using a penalized criterion. It is then possible to adapt general methods related to model selection in this framework to study two important points: (1) what is the minimum penalty and how does it compare to the penalty actually used in the SVM algorithm; (2) is it possible to obtain ``oracle inequalities'' in that setting, for the specific loss function used in the SVM algorithm? We show that the answer to the latter question is positive and provides relevant insight to the former. Our result shows that it is possible to obtain fast rates of convergence for SVMs.
△ Less
Submitted 3 April, 2008;
originally announced April 2008.
-
Two simple sufficient conditions for FDR control
Authors:
Gilles Blanchard,
Etienne Roquain
Abstract:
We show that the control of the false discovery rate (FDR) for a multiple testing procedure is implied by two coupled simple sufficient conditions. The first one, which we call ``self-consistency condition'', concerns the algorithm itself, and the second, called ``dependency control condition'' is related to the dependency assumptions on the $p$-value family. Many standard multiple testing proce…
▽ More
We show that the control of the false discovery rate (FDR) for a multiple testing procedure is implied by two coupled simple sufficient conditions. The first one, which we call ``self-consistency condition'', concerns the algorithm itself, and the second, called ``dependency control condition'' is related to the dependency assumptions on the $p$-value family. Many standard multiple testing procedures are self-consistent (e.g. step-up, step-down or step-up-down procedures), and we prove that the dependency control condition can be fulfilled when choosing correspondingly appropriate rejection functions, in three classical types of dependency: independence, positive dependency (PRDS) and unspecified dependency. As a consequence, we recover earlier results through simple and unifying proofs while extending their scope to several regards: weighted FDR, $p$-value reweighting, new family of step-up procedures under unspecified $p$-value dependency and adaptive step-up procedures. We give additional examples of other possible applications. This framework also allows for defining and studying FDR control for multiple testing procedures over a continuous, uncountable space of hypotheses.
△ Less
Submitted 21 October, 2008; v1 submitted 11 February, 2008;
originally announced February 2008.
-
Some nonasymptotic results on resampling in high dimension, I: Confidence regions, II: Multiple tests
Authors:
Sylvain Arlot,
Gilles Blanchard,
Etienne Roquain
Abstract:
We study generalized bootstrap confidence regions for the mean of a random vector whose coordinates have an unknown dependency structure. The random vector is supposed to be either Gaussian or to have a symmetric and bounded distribution. The dimensionality of the vector can possibly be much larger than the number of observations and we focus on a nonasymptotic control of the confidence level, f…
▽ More
We study generalized bootstrap confidence regions for the mean of a random vector whose coordinates have an unknown dependency structure. The random vector is supposed to be either Gaussian or to have a symmetric and bounded distribution. The dimensionality of the vector can possibly be much larger than the number of observations and we focus on a nonasymptotic control of the confidence level, following ideas inspired by recent results in learning theory. We consider two approaches, the first based on a concentration principle (valid for a large class of resampling weights) and the second on a resampled quantile, specifically using Rademacher weights. Several intermediate results established in the approach based on concentration principles are of interest in their own right. We also discuss the question of accuracy when using Monte Carlo approximations of the resampled quantities.
△ Less
Submitted 11 January, 2010; v1 submitted 5 December, 2007;
originally announced December 2007.
-
Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii
Authors:
Gilles Blanchard,
Pascal Massart
Abstract:
Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii [arXiv:0708.0083]
Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii [arXiv:0708.0083]
△ Less
Submitted 1 August, 2007;
originally announced August 2007.
-
Adaptive FDR control under independence and dependence
Authors:
Gilles Blanchard,
Etienne Roquain
Abstract:
In the context of multiple hypotheses testing, the proportion $π_0$ of true null hypotheses in the pool of hypotheses to test often plays a crucial role, although it is generally unknown a priori. A testing procedure using an implicit or explicit estimate of this quantity in order to improve its efficency is called adaptive. In this paper, we focus on the issue of False Discovery Rate (FDR) cont…
▽ More
In the context of multiple hypotheses testing, the proportion $π_0$ of true null hypotheses in the pool of hypotheses to test often plays a crucial role, although it is generally unknown a priori. A testing procedure using an implicit or explicit estimate of this quantity in order to improve its efficency is called adaptive. In this paper, we focus on the issue of False Discovery Rate (FDR) control and we present new adaptive multiple testing procedures with control of the FDR. First, in the context of assuming independent $p$-values, we present two new procedures and give a unified review of other existing adaptive procedures that have provably controlled FDR. We report extensive simulation results comparing these procedures and testing their robustness when the independence assumption is violated. The new proposed procedures appear competitive with existing ones. The overall best, though, is reported to be Storey's estimator, but for a parameter setting that does not appear to have been considered before. Second, we propose adaptive versions of step-up procedures that have provably controlled FDR under positive dependences and unspecified dependences of the $p$-values, respectively. While simulations only show an improvement over non-adaptive procedures in limited situations, these are to our knowledge among the first theoretically founded adaptive multiple testing procedures that control the FDR when the $p$-values are not independent.
△ Less
Submitted 17 February, 2009; v1 submitted 4 July, 2007;
originally announced July 2007.
-
Resampling-based confidence regions and multiple tests for a correlated random vector
Authors:
Sylvain Arlot,
Gilles Blanchard,
Etienne Roquain
Abstract:
We derive non-asymptotic confidence regions for the mean of a random vector whose coordinates have an unknown dependence structure. The random vector is supposed to be either Gaussian or to have a symmetric bounded distribution, and we observe $n$ i.i.d copies of it. The confidence regions are built using a data-dependent threshold based on a weighted bootstrap procedure. We consider two approac…
▽ More
We derive non-asymptotic confidence regions for the mean of a random vector whose coordinates have an unknown dependence structure. The random vector is supposed to be either Gaussian or to have a symmetric bounded distribution, and we observe $n$ i.i.d copies of it. The confidence regions are built using a data-dependent threshold based on a weighted bootstrap procedure. We consider two approaches, the first based on a concentration approach and the second on a direct boostrapped quantile approach. The first one allows to deal with a very large class of resampling weights while our results for the second are restricted to Rademacher weights. However, the second method seems more accurate in practice. Our results are motivated by multiple testing problems, and we show on simulations that our procedures are better than the Bonferroni procedure (union bound) as soon as the observed vector has sufficiently correlated coordinates.
△ Less
Submitted 22 January, 2007;
originally announced January 2007.
-
Occam's hammer: a link between randomized learning and multiple testing FDR control
Authors:
Gilles Blanchard,
François Fleuret
Abstract:
We establish a generic theoretical tool to construct probabilistic bounds for algorithms where the output is a subset of objects from an initial pool of candidates (or more generally, a probability distribution on said pool). This general device, dubbed "Occam's hammer'', acts as a meta layer when a probabilistic bound is already known on the objects of the pool taken individually, and aims at c…
▽ More
We establish a generic theoretical tool to construct probabilistic bounds for algorithms where the output is a subset of objects from an initial pool of candidates (or more generally, a probability distribution on said pool). This general device, dubbed "Occam's hammer'', acts as a meta layer when a probabilistic bound is already known on the objects of the pool taken individually, and aims at controlling the proportion of the objects in the set output not satisfying their individual bound. In this regard, it can be seen as a non-trivial generalization of the "union bound with a prior'' ("Occam's razor''), a familiar tool in learning theory. We give applications of this principle to randomized classifiers (providing an interesting alternative approach to PAC-Bayes bounds) and multiple testing (where it allows to retrieve exactly and extend the so-called Benjamini-Yekutieli testing procedure).
△ Less
Submitted 29 August, 2006;
originally announced August 2006.
-
Hierarchical testing designs for pattern recognition
Authors:
Gilles Blanchard,
Donald Geman
Abstract:
We explore the theoretical foundations of a ``twenty questions'' approach to pattern recognition. The object of the analysis is the computational process itself rather than probability distributions (Bayesian inference) or decision boundaries (statistical learning). Our formulation is motivated by applications to scene interpretation in which there are a great many possible explanations for the…
▽ More
We explore the theoretical foundations of a ``twenty questions'' approach to pattern recognition. The object of the analysis is the computational process itself rather than probability distributions (Bayesian inference) or decision boundaries (statistical learning). Our formulation is motivated by applications to scene interpretation in which there are a great many possible explanations for the data, one (``background'') is statistically dominant, and it is imperative to restrict intensive computation to genuinely ambiguous regions. The focus here is then on pattern filtering: Given a large set Y of possible patterns or explanations, narrow down the true one Y to a small (random) subset \hat Y\subsetY of ``detected'' patterns to be subjected to further, more intense, processing. To this end, we consider a family of hypothesis tests for Y\in A versus the nonspecific alternatives Y\in A^c. Each test has null type I error and the candidate sets A\subsetY are arranged in a hierarchy of nested partitions. These tests are then characterized by scope (|A|), power (or type II error) and algorithmic cost. We consider sequential testing strategies in which decisions are made iteratively, based on past outcomes, about which test to perform next and when to stop testing. The set \hat Y is then taken to be the set of patterns that have not been ruled out by the tests performed. The total cost of a strategy is the sum of the ``testing cost'' and the ``postprocessing cost'' (proportional to |\hat Y|) and the corresponding optimization problem is analyzed.
△ Less
Submitted 21 July, 2005;
originally announced July 2005.