Showing 1–9 of 9 results for author: de Heide, R

E-Statistics, Group Invariance and Anytime Valid Testing

Authors: Muriel Felipe Pérez-Ortiz, Tyron Lardy, Rianne de Heide, Peter Grünwald

Abstract: We study worst-case-growth-rate-optimal (GROW) e-statistics for hypothesis testing between two group models. It is known that under a mild condition on the action of the underlying group G on the data, there exists a maximally invariant statistic. We show that among all e-statistics, invariant or not, the likelihood ratio of the maximally invariant statistic is GROW, both in the absolute and in th… ▽ More We study worst-case-growth-rate-optimal (GROW) e-statistics for hypothesis testing between two group models. It is known that under a mild condition on the action of the underlying group G on the data, there exists a maximally invariant statistic. We show that among all e-statistics, invariant or not, the likelihood ratio of the maximally invariant statistic is GROW, both in the absolute and in the relative sense, and that an anytime-valid test can be based on it. The GROW e-statistic is equal to a Bayes factor with a right Haar prior on G. Our treatment avoids nonuniqueness issues that sometimes arise for such priors in Bayesian contexts. A crucial assumption on the group G is its amenability, a well-known group-theoretical condition, which holds, for instance, in scale-location families. Our results also apply to finite-dimensional linear regression. △ Less

Submitted 17 October, 2023; v1 submitted 16 August, 2022; originally announced August 2022.

Comments: 30 pages. Major rewrite of previous version. Submitted to the Annals of Statistics

Top Two Algorithms Revisited

Authors: Marc Jourdan, Rémy Degenne, Dorian Baudry, Rianne de Heide, Emilie Kaufmann

Abstract: Top Two algorithms arose as an adaptation of Thompson sampling to best arm identification in multi-armed bandit models (Russo, 2016), for parametric families of arms. They select the next arm to sample from by randomizing among two candidate arms, a leader and a challenger. Despite their good empirical performance, theoretical guarantees for fixed-confidence best arm identification have only been… ▽ More Top Two algorithms arose as an adaptation of Thompson sampling to best arm identification in multi-armed bandit models (Russo, 2016), for parametric families of arms. They select the next arm to sample from by randomizing among two candidate arms, a leader and a challenger. Despite their good empirical performance, theoretical guarantees for fixed-confidence best arm identification have only been obtained when the arms are Gaussian with known variances. In this paper, we provide a general analysis of Top Two methods, which identifies desirable properties of the leader, the challenger, and the (possibly non-parametric) distributions of the arms. As a result, we obtain theoretically supported Top Two algorithms for best arm identification with bounded distributions. Our proof method demonstrates in particular that the sampling step used to select the leader inherited from Thompson sampling can be replaced by other choices, like selecting the empirical best arm. △ Less

Submitted 4 October, 2022; v1 submitted 13 June, 2022; originally announced June 2022.

Comments: 75 pages, 8 figures, 3 tables

Attribution-based Explanations that Provide Recourse Cannot be Robust

Authors: Hidde Fokkema, Rianne de Heide, Tim van Erven

Abstract: Different users of machine learning methods require different explanations, depending on their goals. To make machine learning accountable to society, one important goal is to get actionable options for recourse, which allow an affected user to change the decision $f(x)$ of a machine learning system by making limited changes to its input $x$. We formalize this by providing a general definition of… ▽ More Different users of machine learning methods require different explanations, depending on their goals. To make machine learning accountable to society, one important goal is to get actionable options for recourse, which allow an affected user to change the decision $f(x)$ of a machine learning system by making limited changes to its input $x$. We formalize this by providing a general definition of recourse sensitivity, which needs to be instantiated with a utility function that describes which changes to the decisions are relevant to the user. This definition applies to local attribution methods, which attribute an importance weight to each input feature. It is often argued that such local attributions should be robust, in the sense that a small change in the input $x$ that is being explained, should not cause a large change in the feature weights. However, we prove formally that it is in general impossible for any single attribution method to be both recourse sensitive and robust at the same time. It follows that there must always exist counterexamples to at least one of these properties. We provide such counterexamples for several popular attribution methods, including LIME, SHAP, Integrated Gradients and SmoothGrad. Our results also cover counterfactual explanations, which may be viewed as attributions that describe a perturbation of $x$. We further discuss possible ways to work around our impossibility result, for instance by allowing the output to consist of sets with multiple attributions, and we provide sufficient conditions for specific classes of continuous functions to be recourse sensitive. Finally, we strengthen our impossibility result for the restricted case where users are only able to change a single attribute of $x$, by providing an exact characterization of the functions $f$ to which impossibility applies. △ Less

Submitted 20 December, 2023; v1 submitted 31 May, 2022; originally announced May 2022.

Comments: 32 pages, 6 figures

Bandits with many optimal arms

Authors: Rianne de Heide, James Cheshire, Pierre Ménard, Alexandra Carpentier

Abstract: We consider a stochastic bandit problem with a possibly infinite number of arms. We write $p^*$ for the proportion of optimal arms and $Δ$ for the minimal mean-gap between optimal and sub-optimal arms. We characterize the optimal learning rates both in the cumulative regret setting, and in the best-arm identification setting in terms of the problem parameters $T$ (the budget), $p^*$ and $Δ$. For t… ▽ More We consider a stochastic bandit problem with a possibly infinite number of arms. We write $p^*$ for the proportion of optimal arms and $Δ$ for the minimal mean-gap between optimal and sub-optimal arms. We characterize the optimal learning rates both in the cumulative regret setting, and in the best-arm identification setting in terms of the problem parameters $T$ (the budget), $p^*$ and $Δ$. For the objective of minimizing the cumulative regret, we provide a lower bound of order $Ω(\log(T)/(p^*Δ))$ and a UCB-style algorithm with matching upper bound up to a factor of $\log(1/Δ)$. Our algorithm needs $p^*$ to calibrate its parameters, and we prove that this knowledge is necessary, since adapting to $p^*$ in this setting is impossible. For best-arm identification we also provide a lower bound of order $Ω(\exp(-cTΔ^2 p^*))$ on the probability of outputting a sub-optimal arm where $c>0$ is an absolute constant. We also provide an elimination algorithm with an upper bound matching the lower bound up to a factor of order $\log(T)$ in the exponential, and that does not need $p^*$ or $Δ$ as parameter. Our results apply directly to the three related problems of competing against the $j$-th best arm, identifying an $ε$ good arm, and finding an arm with mean larger than a quantile of a known order. △ Less

Submitted 5 November, 2021; v1 submitted 23 March, 2021; originally announced March 2021.

Comments: Substantial rewrite and added experiments. Accepted for NeurIPS 2021

Fixed-Confidence Guarantees for Bayesian Best-Arm Identification

Authors: Xuedong Shang, Rianne de Heide, Emilie Kaufmann, Pierre Ménard, Michal Valko

Abstract: We investigate and provide new insights on the sampling rule called Top-Two Thompson Sampling (TTTS). In particular, we justify its use for fixed-confidence best-arm identification. We further propose a variant of TTTS called Top-Two Transportation Cost (T3C), which disposes of the computational burden of TTTS. As our main contribution, we provide the first sample complexity analysis of TTTS and T… ▽ More We investigate and provide new insights on the sampling rule called Top-Two Thompson Sampling (TTTS). In particular, we justify its use for fixed-confidence best-arm identification. We further propose a variant of TTTS called Top-Two Transportation Cost (T3C), which disposes of the computational burden of TTTS. As our main contribution, we provide the first sample complexity analysis of TTTS and T3C when coupled with a very natural Bayesian stop** rule, for bandits with Gaussian rewards, solving one of the open questions raised by Russo (2016). We also provide new posterior convergence results for TTTS under two models that are commonly used in practice: bandits with Gaussian and Bernoulli rewards and conjugate priors. △ Less

Submitted 28 October, 2019; v1 submitted 24 October, 2019; originally announced October 2019.

Safe-Bayesian Generalized Linear Regression

Authors: Rianne de Heide, Alisa Kirichenko, Nishant Mehta, Peter Grünwald

Abstract: We study generalized Bayesian inference under misspecification, i.e. when the model is 'wrong but useful'. Generalized Bayes equips the likelihood with a learning rate $η$. We show that for generalized linear models (GLMs), $η$-generalized Bayes concentrates around the best approximation of the truth within the model for specific $η\neq 1$, even under severely misspecified noise, as long as the ta… ▽ More We study generalized Bayesian inference under misspecification, i.e. when the model is 'wrong but useful'. Generalized Bayes equips the likelihood with a learning rate $η$. We show that for generalized linear models (GLMs), $η$-generalized Bayes concentrates around the best approximation of the truth within the model for specific $η\neq 1$, even under severely misspecified noise, as long as the tails of the true distribution are exponential. We derive MCMC samplers for generalized Bayesian lasso and logistic regression and give examples of both simulated and real-world data in which generalized Bayes substantially outperforms standard Bayes. △ Less

Submitted 29 May, 2021; v1 submitted 21 October, 2019; originally announced October 2019.

Comments: Final version. Accepted to AISTATS 2020

Safe Testing

Authors: Peter Grünwald, Rianne de Heide, Wouter Koolen

Abstract: We develop the theory of hypothesis testing based on the e-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e-values are safe, i.e. they preserve Type-I error guarantees, under such optional continuation. We define grow… ▽ More We develop the theory of hypothesis testing based on the e-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e-values are safe, i.e. they preserve Type-I error guarantees, under such optional continuation. We define growth-rate optimality (GRO) as an analogue of power in an optional continuation context, and we show how to construct GRO e-variables for general testing problems with composite null and alternative, emphasizing models with nuisance parameters. GRO e-values take the form of Bayes factors with special priors. We illustrate the theory using several classic examples including a one-sample safe t-test and the 2 x 2 contingency table. Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, e-values may provide a methodology acceptable to adherents of all three schools. △ Less

Submitted 10 March, 2023; v1 submitted 18 June, 2019; originally announced June 2019.

Comments: Accepted as discussion paper to the Journal of the Royal Statistical Society series B

Optional Stop** with Bayes Factors: a categorization and extension of folklore results, with an application to invariant situations

Authors: Allard Hendriksen, Rianne de Heide, Peter Grünwald

Abstract: It is often claimed that Bayesian methods, in particular Bayes factor methods for hypothesis testing, can deal with optional stop**. We first give an overview, using elementary probability theory, of three different mathematical meanings that various authors give to this claim: (1) stop** rule independence, (2) posterior calibration and (3) (semi-) frequentist robustness to optional stop**.… ▽ More It is often claimed that Bayesian methods, in particular Bayes factor methods for hypothesis testing, can deal with optional stop**. We first give an overview, using elementary probability theory, of three different mathematical meanings that various authors give to this claim: (1) stop** rule independence, (2) posterior calibration and (3) (semi-) frequentist robustness to optional stop**. We then prove theorems to the effect that these claims do indeed hold in a general measure-theoretic setting. For claims of type (2) and (3), such results are new. By allowing for non-integrable measures based on improper priors, we obtain particularly strong results for the practically important case of models with nuisance parameters satisfying a group invariance (such as location or scale). We also discuss the practical relevance of (1)--(3), and conclude that whether Bayes factor methods actually perform well under optional stop** crucially depends on details of models, priors and the goal of the analysis. △ Less

Submitted 29 April, 2020; v1 submitted 24 July, 2018; originally announced July 2018.

Comments: 29 pages

Why optional stop** can be a problem for Bayesians

Authors: Rianne de Heide, Peter D. Grünwald

Abstract: Recently, optional stop** has been a subject of debate in the Bayesian psychology community. Rouder (2014) argues that optional stop** is no problem for Bayesians, and even recommends the use of optional stop** in practice, as do Wagenmakers et al. (2012). This article addresses the question whether optional stop** is problematic for Bayesian methods, and specifies under which circumstance… ▽ More Recently, optional stop** has been a subject of debate in the Bayesian psychology community. Rouder (2014) argues that optional stop** is no problem for Bayesians, and even recommends the use of optional stop** in practice, as do Wagenmakers et al. (2012). This article addresses the question whether optional stop** is problematic for Bayesian methods, and specifies under which circumstances and in which sense it is and is not. By slightly varying and extending Rouder's (2014) experiments, we illustrate that, as soon as the parameters of interest are equipped with default or pragmatic priors - which means, in most practical applications of Bayes factor hypothesis testing - resilience to optional stop** can break down. We distinguish between three types of default priors, each having their own specific issues with optional stop**, ranging from no-problem-at-all (Type 0 priors) to quite severe (Type II priors). △ Less

Submitted 25 March, 2021; v1 submitted 28 August, 2017; originally announced August 2017.

Comments: Replacement of Figures 7a-7d in the appendix. There was a mistake in the sampling plan. Thanks to Jorge Tendeiro for pointing this out. Replaced the main text with the final (published) version. Psychonomic Bulletin & Review 2020 Advance Publication