Showing 1–17 of 17 results for author: Solari, A

Rank tests for outlier detection

Authors: Chiara G. Magnani, Aldo Solari

Abstract: In novelty detection, the objective is to determine whether the test sample contains any outliers, using a sample of controls (inliers). This involves many-to-one comparisons of individual test points against the control sample. A recent approach applies the Benjamini-Hochberg procedure to the conformal $p$-values resulting from these comparisons, ensuring false discovery rate control. In this p… ▽ More In novelty detection, the objective is to determine whether the test sample contains any outliers, using a sample of controls (inliers). This involves many-to-one comparisons of individual test points against the control sample. A recent approach applies the Benjamini-Hochberg procedure to the conformal $p$-values resulting from these comparisons, ensuring false discovery rate control. In this paper, we suggest using Wilcoxon-Mann-Whitney tests for the comparisons and subsequently applying the closed testing principle to derive post-hoc confidence bounds for the number of outliers in any subset of the test sample. We revisit an elegant result that under a nonparametric alternative known as Lehmann's alternative, Wilcoxon-Mann-Whitney is locally most powerful among rank tests. By combining this result with a simple observation, we demonstrate that the proposed procedure is more powerful for the null hypothesis of no outliers than the Benjamini-Hochberg procedure applied to conformal $p$-values. △ Less

Submitted 10 August, 2023; originally announced August 2023.

Simultaneous directional inference

Authors: Ruth Heller, Aldo Solari

Abstract: We consider the problem of inference on the signs of $n>1$ parameters. We aim to provide $1-α$ post-hoc confidence bounds on the number of positive and negative (or non-positive) parameters. The guarantee is simultaneous, for all subsets of parameters. Our suggestion is as follows: start by using the data to select the direction of the hypothesis test for each parameter; then, adjust the $p$-value… ▽ More We consider the problem of inference on the signs of $n>1$ parameters. We aim to provide $1-α$ post-hoc confidence bounds on the number of positive and negative (or non-positive) parameters. The guarantee is simultaneous, for all subsets of parameters. Our suggestion is as follows: start by using the data to select the direction of the hypothesis test for each parameter; then, adjust the $p$-values of the one-sided hypotheses for the selection, and use the adjusted $p$-values for simultaneous inference on the selected $n$ one-sided hypotheses. The adjustment is straightforward assuming that the $p$-values of one-sided hypotheses have densities with monotone likelihood ratio, and are mutually independent. We show that the bounds we provide are tighter (often by a great margin) than existing alternatives, and that they can be obtained by at most a polynomial time. We demonstrate the usefulness of our simultaneous post-hoc bounds in the evaluation of treatment effects across studies or subgroups. Specifically, we provide a tight lower bound on the number of studies which are beneficial, as well as on the number of studies which are harmful (or non-beneficial), and in addition conclude on the effect direction of individual studies, while guaranteeing that the probability of at least one wrong inference is at most 0.05. △ Less

Submitted 6 August, 2023; v1 submitted 4 January, 2023; originally announced January 2023.

Comments: 59 pages, 11 figures, 7 tables

Journal ref: Ruth Heller, Aldo Solari, Simultaneous directional inference, Journal of the Royal Statistical Society Series B: Statistical Methodology, 2023

Flexible control of the median of the false discovery proportion

Authors: Jesse Hemerik, Aldo Solari, Jelle J Goeman

Abstract: We introduce a multiple testing procedure that controls the median of the proportion of false discoveries (FDP) in a flexible way. The procedure only requires a vector of p-values as input and is comparable to the Benjamini-Hochberg method, which controls the mean of the FDP. Our method allows freely choosing one or several values of alpha after seeing the data -- unlike Benjamini-Hochberg, which… ▽ More We introduce a multiple testing procedure that controls the median of the proportion of false discoveries (FDP) in a flexible way. The procedure only requires a vector of p-values as input and is comparable to the Benjamini-Hochberg method, which controls the mean of the FDP. Our method allows freely choosing one or several values of alpha after seeing the data -- unlike Benjamini-Hochberg, which can be very liberal when alpha is chosen post hoc. We prove these claims and illustrate them with simulations. Our procedure is inspired by a popular estimator of the total number of true hypotheses. We adapt this estimator to provide simultaneously median unbiased estimators of the FDP, valid for finite samples. This simultaneity allows for the claimed flexibility. Our approach does not assume independence. The time complexity of our method is linear in the number of hypotheses, after sorting the p-values. △ Less

Submitted 13 March, 2024; v1 submitted 24 August, 2022; originally announced August 2022.

MSC Class: 62F03

On Selecting and Conditioning in Multiple Testing and Selective Inference

Authors: Jelle Goeman, Aldo Solari

Abstract: We investigate a class of methods for selective inference that condition on a selection event. Such methods follow a two-stage process. First, a data-driven (sub)collection of hypotheses is chosen from some large universe of hypotheses. Subsequently, inference takes place within this data-driven collection, conditioned on the information that was used for the selection. Examples of such methods in… ▽ More We investigate a class of methods for selective inference that condition on a selection event. Such methods follow a two-stage process. First, a data-driven (sub)collection of hypotheses is chosen from some large universe of hypotheses. Subsequently, inference takes place within this data-driven collection, conditioned on the information that was used for the selection. Examples of such methods include basic data splitting, as well as modern data carving methods and post-selection inference methods for lasso coefficients based on the polyhedral lemma. In this paper, we adopt a holistic view on such methods, considering the selection, conditioning, and final error control steps together as a single method. From this perspective, we demonstrate that multiple testing methods defined directly on the full universe of hypotheses are always at least as powerful as selective inference methods based on selection and conditioning. This result holds true even when the universe is potentially infinite and only implicitly defined, such as in the case of data splitting. We provide a comprehensive theoretical framework, along with insights, and delve into several case studies to illustrate instances where a shift to a non-selective or unconditional perspective can yield a power gain. △ Less

Submitted 5 December, 2023; v1 submitted 27 July, 2022; originally announced July 2022.

conformalInference.multi and conformalInference.fd: Twin Packages for Conformal Prediction

Authors: Paolo Vergottini, Matteo Fontana, Jacopo Diquigiovanni, Aldo Solari, Simone Vantini

Abstract: Building on top of a regression model, Conformal Prediction methods produce distribution free prediction sets, requiring only i.i.d. data. While R packages implementing such methods for the univariate response framework have been developed, this is not the case with multivariate and functional responses. conformalInference.multi and conformalInference.fd address this void, by extending classical a… ▽ More Building on top of a regression model, Conformal Prediction methods produce distribution free prediction sets, requiring only i.i.d. data. While R packages implementing such methods for the univariate response framework have been developed, this is not the case with multivariate and functional responses. conformalInference.multi and conformalInference.fd address this void, by extending classical and more advanced conformal prediction methods like full conformal, split conformal, jackknife+ and multi split conformal to deal with the multivariate and functional case. The extreme flexibility of conformal prediction, fully embraced by the structure of the package, which does not require any specific regression model, enables users to pass in any regression function as input while using basic regression models as reference. Finally, the issue of visualisation is addressed by providing embedded plotting functions to visualize prediction regions. △ Less

Submitted 29 June, 2022; originally announced June 2022.

Comments: 15 Pages, 2 Figures

Multi Split Conformal Prediction

Authors: Aldo Solari, Vera Djordjilović

Abstract: Split conformal prediction is a computationally efficient method for performing distribution-free predictive inference in regression. It involves, however, a one-time random split of the data, and the result depends on the particular split. To address this problem, we propose multi split conformal prediction, a simple method based on Markov's inequality to aggregate single split conformal predicti… ▽ More Split conformal prediction is a computationally efficient method for performing distribution-free predictive inference in regression. It involves, however, a one-time random split of the data, and the result depends on the particular split. To address this problem, we propose multi split conformal prediction, a simple method based on Markov's inequality to aggregate single split conformal prediction intervals across multiple splits. △ Less

Submitted 21 July, 2021; v1 submitted 28 February, 2021; originally announced March 2021.

Comments: 12 pages, 1 figure, 2 table

Comparing three groups

Authors: Jelle Goeman, Aldo Solari

Abstract: We revisit simple and powerful methods for multiple pairwise comparisons that can be used in designs with three groups. We argue that the proper choice of method should be determined by the assessment which of the comparisons are considered primary and which are secondary, as determined by subject-matter considerations. We review four different methods that are simple to use with any standard soft… ▽ More We revisit simple and powerful methods for multiple pairwise comparisons that can be used in designs with three groups. We argue that the proper choice of method should be determined by the assessment which of the comparisons are considered primary and which are secondary, as determined by subject-matter considerations. We review four different methods that are simple to use with any standard software, but are substantially more powerful than frequently-used methods such as an ANOVA test followed by Tukey's method. △ Less

Submitted 11 May, 2020; originally announced May 2020.

Pathway Testing in Metabolomics with Globaltest, Allowing Post Hoc Choice of Pathways

Authors: Ningning Xu, Aldo Solari, Jelle Goeman

Abstract: The Globaltest is a powerful test for the global null hypothesis that there is no association between a group of features and a response of interest, which is popular in pathway testing in metabolomics. Evaluating multiple pathways, however, requires multiple testing correction. In this paper, we propose a multiple testing method, based on closed testing, specifically designed for the Globaltest.… ▽ More The Globaltest is a powerful test for the global null hypothesis that there is no association between a group of features and a response of interest, which is popular in pathway testing in metabolomics. Evaluating multiple pathways, however, requires multiple testing correction. In this paper, we propose a multiple testing method, based on closed testing, specifically designed for the Globaltest. The proposed method controls the family-wise error rate simultaneously over all possible feature sets, and therefore allows post hoc inference, i.e. the researcher may choose the pathway database after seeing the data without jeopardizing error control. To circumvent the exponential computation time of closed testing, we derive a novel shortcut that allows exact closed testing to be performed on the scale of metabolomics data. An R package ctgt is available on CRAN. We illustrate the shortcut on several metabolomics data examples. △ Less

Submitted 4 February, 2021; v1 submitted 6 January, 2020; originally announced January 2020.

Comments: 33 pages, 5 figures, 4 tables

Genetic assignment of illegally trafficked neotropical primates and implications for reintroduction programs

Authors: L. I. Oklander, M. Caputo, A. Solari, D. Corach

Abstract: The black and gold howler monkey (Alouatta caraya) is a neotropical primate that faces the highest capture pressure for illegal trade in Argentina. We evaluate the applicability of genetic assignment tests based on microsatellite genotypic data to accurately assign individuals to their site of origin. The search was conducted on a genetic database to determine the nearest sampled population or to… ▽ More The black and gold howler monkey (Alouatta caraya) is a neotropical primate that faces the highest capture pressure for illegal trade in Argentina. We evaluate the applicability of genetic assignment tests based on microsatellite genotypic data to accurately assign individuals to their site of origin. The search was conducted on a genetic database to determine the nearest sampled population or to associate them to three clusters described here for the Argentinean populations of A. caraya. We correctly assign 73% of the individuals in the database to nearest population of origin, and 93.3% to their cluster of origin. With this database, we were able to determine the probable origin of 17 confiscated individuals, 12 of which were reintroduced in the province of Misiones and 5 confiscated individuals reintroduced in the province of Santa Fe. Moreover, we also determined the probable origin of 3 individuals found dead in cities in northern Argentina. This approach highlights the relevance of generating genotype indexing databases of species to assist with in-situ and ex-situ conservation and management programs. Our results underscore the importance of knowing the origin of individuals for reintroduction and/or species recovery programs and to pinpoint the hotspots of illegal capture of various species. △ Less

Submitted 12 November, 2019; originally announced November 2019.

Comments: 16 pages, 2 figures

Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions

Authors: Jelle Goeman, Jesse Hemerik, Aldo Solari

Abstract: We consider the class of all multiple testing methods controlling tail probabilities of the false discovery proportion, either for one random set or simultaneously for many such sets. This class encompasses methods controlling familywise error rate, generalized familywise error rate, false discovery exceedance, joint error rate, simultaneous control of all false discovery proportions, and others,… ▽ More We consider the class of all multiple testing methods controlling tail probabilities of the false discovery proportion, either for one random set or simultaneously for many such sets. This class encompasses methods controlling familywise error rate, generalized familywise error rate, false discovery exceedance, joint error rate, simultaneous control of all false discovery proportions, and others, as well as seemingly unrelated methods such as gene set testing in genomics and cluster inference methods in neuroimaging. We show that all such methods are either equivalent to a closed testing method, or are uniformly improved by one. Moreover, we show that a closed testing method is admissible as a method controlling tail probabilities of false discovery proportions if and only if all its local tests are admissible. This implies that, when designing such methods, it is sufficient to restrict attention to closed testing methods only. We demonstrate the practical usefulness of this design principle by constructing a uniform improvement of a recently proposed method. △ Less

Submitted 29 April, 2022; v1 submitted 15 January, 2019; originally announced January 2019.

MSC Class: 62F03

Permutation-based simultaneous confidence bounds for the false discovery proportion

Authors: Jesse Hemerik, Aldo Solari, Jelle J. Goeman

Abstract: When multiple hypotheses are tested, interest is often in ensuring that the proportion of false discoveries (FDP) is small with high confidence. In this paper, confidence upper bounds for the FDP are constructed, which are simultaneous over all rejection cut-offs. In particular this allows the user to select a set of hypotheses post hoc such that the FDP lies below some constant with high confiden… ▽ More When multiple hypotheses are tested, interest is often in ensuring that the proportion of false discoveries (FDP) is small with high confidence. In this paper, confidence upper bounds for the FDP are constructed, which are simultaneous over all rejection cut-offs. In particular this allows the user to select a set of hypotheses post hoc such that the FDP lies below some constant with high confidence. Our method uses permutations to account for the dependence structure in the data. So far only Meinshausen provided an exact, permutation-based and computationally feasible method for simultaneous FDP bounds. We provide an exact method, which uniformly improves this procedure. Further, we provide a generalization of this method. It lets the user select the shape of the simultaneous confidence bounds. This gives the user more freedom in determining the power properties of the method. Interestingly, several existing permutation methods, such as Significance Analysis of Microarrays (SAM) and Westfall and Young's maxT method, are obtained as special cases. △ Less

Submitted 16 August, 2018; originally announced August 2018.

MSC Class: 62G09; 62H15

Journal ref: Biometrika, 106(3):635-649, 2019

A shortcut for Hommel's procedure in linearithmic time

Authors: Rosa Meijer, Thijmen Krebs, Aldo Solari, Jelle Goeman

Abstract: Hommel's and Hochberg's procedures for familywise error control are both derived as shortcuts in a closed testing procedure with the Simes local test. Hommel's shortcut is exact but takes quadratic time in the number of hypotheses. Hochberg's shortcut takes only linearithmic time, but is conservative. In this paper we present an exact shortcut in linearithmic time, combining the strengths of both… ▽ More Hommel's and Hochberg's procedures for familywise error control are both derived as shortcuts in a closed testing procedure with the Simes local test. Hommel's shortcut is exact but takes quadratic time in the number of hypotheses. Hochberg's shortcut takes only linearithmic time, but is conservative. In this paper we present an exact shortcut in linearithmic time, combining the strengths of both procedures. The novel shortcut also applies to a robust variant of Hommel's procedure that does not require the assumption of the Simes inequality. △ Less

Submitted 23 October, 2017; originally announced October 2017.

Comments: arXiv admin note: text overlap with arXiv:1611.06739

Simultaneous confidence sets for ranks using the partitioning principle - Technical report

Authors: Diaa Al Mohamad, Erik W. van Zwet, Jelle J. Goeman, Aldo Solari

Abstract: Ranking institutions such as medical centers or universities is based on an indicator accompanied with an uncertainty measure such as a standard deviation, and confidence intervals should be calculated to assess the quality of these ranks. We consider the problem of constructing simultaneous confidence intervals for the ranks of centers based on an observed sample. We present in this paper a novel… ▽ More Ranking institutions such as medical centers or universities is based on an indicator accompanied with an uncertainty measure such as a standard deviation, and confidence intervals should be calculated to assess the quality of these ranks. We consider the problem of constructing simultaneous confidence intervals for the ranks of centers based on an observed sample. We present in this paper a novel method based on multiple testing which uses the partitioning principle and employs the likelihood ratio (LR) test on the partitions. The complexity of the algorithm is super exponential. We present several ways and shortcuts to reduce this complexity. We provide also a polynomial algorithm which produces a very good bracketing for the multiple testing by linearizing the critical value of the LR test. We show that Tukey's Honest Significant Difference (HSD) test can be written as a partitioning procedure. The new methodology has promising properties in the sens that it opens the door in a simple and easy way to construct new methods which may trade the exponential complexity with power of the test or vice versa. In comparison to Tukey's HSD test, the LR test seems to give better results when the centers are close to each others or the uncertainty in the data is high which is confirmed during a simulation study. △ Less

Submitted 9 August, 2017; originally announced August 2017.

Comments: Technical report. A reduced version will be submitted soon to JRSSB

Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing

Authors: Jelle Goeman, Rosa Meijer, Thijmen Krebs, Aldo Solari

Abstract: Closed testing procedures are classically used for familywise error rate (FWER) control, but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion (FDP) in all subsets of the hypotheses. In this paper we investigate the special case of closed testing with Simes local tests. We construct a novel fast and exact shortcut which we use to investigate the powe… ▽ More Closed testing procedures are classically used for familywise error rate (FWER) control, but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion (FDP) in all subsets of the hypotheses. In this paper we investigate the special case of closed testing with Simes local tests. We construct a novel fast and exact shortcut which we use to investigate the power of this method when the number of hypotheses goes to infinity. We show that, if a minimal amount of signal is present, the average power to detect false hypotheses at any desired FDP level does not vanish. Additionally, we show that the confidence bounds for FDP are consistent estimators for the true FDP for every non-vanishing subset. For the case of a finite number of hypotheses, we show connections between Simes-based closed testing and the procedure of Benjamini and Hochberg. △ Less

Submitted 23 October, 2017; v1 submitted 21 November, 2016; originally announced November 2016.

The sequential rejection principle of familywise error control

Authors: Jelle J. Goeman, Aldo Solari

Abstract: Closed testing and partitioning are recognized as fundamental principles of familywise error control. In this paper, we argue that sequential rejection can be considered equally fundamental as a general principle of multiple testing. We present a general sequentially rejective multiple testing procedure and show that many well-known familywise error controlling methods can be constructed as specia… ▽ More Closed testing and partitioning are recognized as fundamental principles of familywise error control. In this paper, we argue that sequential rejection can be considered equally fundamental as a general principle of multiple testing. We present a general sequentially rejective multiple testing procedure and show that many well-known familywise error controlling methods can be constructed as special cases of this procedure, among which are the procedures of Holm, Shaffer and Hochberg, parallel and serial gatekee** procedures, modern procedures for multiple testing in graphs, resampling-based multiple testing procedures and even the closed testing and partitioning procedures themselves. We also give a general proof that sequentially rejective multiple testing procedures strongly control the familywise error if they fulfill simple criteria of monotonicity of the critical values and a limited form of weak familywise error control in each single step. The sequential rejection principle gives a novel theoretical perspective on many well-known multiple testing procedures, emphasizing the sequential aspect. Its main practical usefulness is for the development of multiple testing procedures for null hypotheses, possibly logically related, that are structured in a graph. We illustrate this by presenting a uniform improvement of a recently published procedure. △ Less

Submitted 14 November, 2012; originally announced November 2012.

Comments: Published in at http://dx.doi.org/10.1214/10-AOS829 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS829

Journal ref: Annals of Statistics 2010, Vol. 38, No. 6, 3782-3810

Rejoinder to "Multiple Testing for Exploratory Research"

Authors: Jelle J. Goeman, Aldo Solari

Abstract: Rejoinder to "Multiple Testing for Exploratory Research" by J. J. Goeman, A. Solari [arXiv:1208.2841]. Rejoinder to "Multiple Testing for Exploratory Research" by J. J. Goeman, A. Solari [arXiv:1208.2841]. △ Less

Submitted 16 August, 2012; originally announced August 2012.

Comments: Published in at http://dx.doi.org/10.1214/11-STS356REJ the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-STS-STS356REJ

Journal ref: Statistical Science 2011, Vol. 26, No. 4, 608-612

Multiple Testing for Exploratory Research

Authors: Jelle J. Goeman, Aldo Solari

Abstract: Motivated by the practice of exploratory research, we formulate an approach to multiple testing that reverses the conventional roles of the user and the multiple testing procedure. Traditionally, the user chooses the error criterion, and the procedure the resulting rejected set. Instead, we propose to let the user choose the rejected set freely, and to let the multiple testing procedure return a c… ▽ More Motivated by the practice of exploratory research, we formulate an approach to multiple testing that reverses the conventional roles of the user and the multiple testing procedure. Traditionally, the user chooses the error criterion, and the procedure the resulting rejected set. Instead, we propose to let the user choose the rejected set freely, and to let the multiple testing procedure return a confidence statement on the number of false rejections incurred. In our approach, such confidence statements are simultaneous for all choices of the rejected set, so that post hoc selection of the rejected set does not compromise their validity. The proposed reversal of roles requires nothing more than a review of the familiar closed testing procedure, but with a focus on the non-consonant rejections that this procedure makes. We suggest several shortcuts to avoid the computational problems associated with closed testing. △ Less

Submitted 2 October, 2013; v1 submitted 14 August, 2012; originally announced August 2012.

Comments: Published in at http://dx.doi.org/10.1214/11-STS356 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-STS-STS356

Journal ref: Statistical Science 2011, Vol. 26, No. 4, 584-597