-
Permutation-based multiple testing when fitting many generalized linear models
Authors:
Riccardo De Santis,
Jelle J. Goeman,
Samuel Davenport,
Jesse Hemerik,
Livio Finos
Abstract:
The multiple testing problem appears when fitting multivariate generalized linear models for high dimensional data. We show that the sign-flip test can be combined with permutation-based procedures for assessing the multiple testing problem
The multiple testing problem appears when fitting multivariate generalized linear models for high dimensional data. We show that the sign-flip test can be combined with permutation-based procedures for assessing the multiple testing problem
△ Less
Submitted 4 March, 2024;
originally announced March 2024.
-
On the term "randomization test"
Authors:
Jesse Hemerik
Abstract:
There exists no consensus on the meaning of the term "randomization test". Contradicting uses of the term are leading to confusion, misunderstandings and indeed invalid data analyses. As we point out, a main source of the confusion is that the term was not explicitly defined when it was first used in the 1930's. Later authors made clear proposals to reach a consensus regarding the term. This resul…
▽ More
There exists no consensus on the meaning of the term "randomization test". Contradicting uses of the term are leading to confusion, misunderstandings and indeed invalid data analyses. As we point out, a main source of the confusion is that the term was not explicitly defined when it was first used in the 1930's. Later authors made clear proposals to reach a consensus regarding the term. This resulted in some level of agreement around the 1970's. However, in the last few decades, the term has often been used in ways that contradict these proposals. This paper provides an overview of the history of the term per se, for the first time tracing it back to 1937. This will hopefully lead to more agreement on terminology and less confusion on the related fundamental concepts.
△ Less
Submitted 13 June, 2023;
originally announced June 2023.
-
Inference in generalized linear models with robustness to misspecified variances
Authors:
Riccardo De Santis,
Jelle J. Goeman,
Jesse Hemerik,
Livio Finos
Abstract:
Generalized linear models usually assume a common dispersion parameter. This assumption is seldom true in practice, and may cause appreciable loss of type I error control if standard parametric methods are used. We present an alternative semi-parametric group invariance method based on sign flip** of score contributions. Our method requires only the correct specification of the mean model, but i…
▽ More
Generalized linear models usually assume a common dispersion parameter. This assumption is seldom true in practice, and may cause appreciable loss of type I error control if standard parametric methods are used. We present an alternative semi-parametric group invariance method based on sign flip** of score contributions. Our method requires only the correct specification of the mean model, but is robust against any misspecification of the variance. The method is available in the R library flipscores.
△ Less
Submitted 26 October, 2022; v1 submitted 28 September, 2022;
originally announced September 2022.
-
More Efficient Exact Group-Invariance Testing: using a Representative Subgroup
Authors:
Nick W. Koning,
Jesse Hemerik
Abstract:
Non-parametric tests based on permutation, rotation or sign-flip** are examples of group-invariance tests. These tests test invariance of the null distribution under a set of transformations that has a group structure, in the algebraic sense. Such groups are often huge, which makes it computationally infeasible to test using the entire group. Hence, it is standard practice to test using a random…
▽ More
Non-parametric tests based on permutation, rotation or sign-flip** are examples of group-invariance tests. These tests test invariance of the null distribution under a set of transformations that has a group structure, in the algebraic sense. Such groups are often huge, which makes it computationally infeasible to test using the entire group. Hence, it is standard practice to test using a randomly sampled set of transformations from the group. This random sample still needs to be substantial to obtain good power and replicability. We improve upon this standard practice by using a well-designed subgroup of transformations instead of a random sample. The resulting subgroup-invariance test is still exact, as invariance under a group implies invariance under its subgroups.
We illustrate this in a generalized location model and obtain more powerful tests based on the same number of transformations. In particular, we show that a subgroup-invariance test is consistent for lower signal-to-noise ratios than a test based on a random sample. For the special case of a normal location model and a particular design of the subgroup, we show that the power improvement is equivalent to the power difference between a Monte Carlo $Z$-test and a Monte Carlo $t$-test.
△ Less
Submitted 22 November, 2022; v1 submitted 2 February, 2022;
originally announced February 2022.
-
Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
Authors:
Jelle Goeman,
Jesse Hemerik,
Aldo Solari
Abstract:
We consider the class of all multiple testing methods controlling tail probabilities of the false discovery proportion, either for one random set or simultaneously for many such sets. This class encompasses methods controlling familywise error rate, generalized familywise error rate, false discovery exceedance, joint error rate, simultaneous control of all false discovery proportions, and others,…
▽ More
We consider the class of all multiple testing methods controlling tail probabilities of the false discovery proportion, either for one random set or simultaneously for many such sets. This class encompasses methods controlling familywise error rate, generalized familywise error rate, false discovery exceedance, joint error rate, simultaneous control of all false discovery proportions, and others, as well as seemingly unrelated methods such as gene set testing in genomics and cluster inference methods in neuroimaging. We show that all such methods are either equivalent to a closed testing method, or are uniformly improved by one. Moreover, we show that a closed testing method is admissible as a method controlling tail probabilities of false discovery proportions if and only if all its local tests are admissible. This implies that, when designing such methods, it is sufficient to restrict attention to closed testing methods only. We demonstrate the practical usefulness of this design principle by constructing a uniform improvement of a recently proposed method.
△ Less
Submitted 29 April, 2022; v1 submitted 15 January, 2019;
originally announced January 2019.
-
Exact testing with random permutations
Authors:
Jesse Hemerik,
Jelle Goeman
Abstract:
When permutation methods are used in practice, often a limited number of random permutations are used to decrease the computational burden. However, most theoretical literature assumes that the whole permutation group is used, and methods based on random permutations tend to be seen as approximate. There exists a very limited amount of literature on exact testing with random permutations and only…
▽ More
When permutation methods are used in practice, often a limited number of random permutations are used to decrease the computational burden. However, most theoretical literature assumes that the whole permutation group is used, and methods based on random permutations tend to be seen as approximate. There exists a very limited amount of literature on exact testing with random permutations and only recently a thorough proof of exactness was given. In this paper we provide an alternative proof, viewing the test as a "conditional Monte Carlo test" as it has been called in the literature. We also provide extensions of the result. Importantly, our results can be used to prove properties of various multiple testing procedures based on random permutations.
△ Less
Submitted 17 August, 2018; v1 submitted 27 November, 2014;
originally announced November 2014.
