-
ALL-IN meta-analysis: breathing life into living systematic reviews
Authors:
Judith ter Schure,
Peter Grünwald
Abstract:
Science is idolized as a cumulative process ("standing on the shoulders of giants"), yet scientific knowledge is typically built on a patchwork of research contributions without much coordination. This lack of efficiency has specifically been addressed in clinical research by recommendations for living systematic reviews and against research waste. We propose to further those recommendations with…
▽ More
Science is idolized as a cumulative process ("standing on the shoulders of giants"), yet scientific knowledge is typically built on a patchwork of research contributions without much coordination. This lack of efficiency has specifically been addressed in clinical research by recommendations for living systematic reviews and against research waste. We propose to further those recommendations with ALL-IN meta-analysis: Anytime Live and Leading INterim meta-analysis. ALL-IN provides statistical methodology for a meta-analysis that can be updated at any time -- reanalyzing after each new observation while retaining type-I error guarantees, live -- no need to prespecify the looks, and leading -- in the decisions on whether individual studies should be initiated, stopped or expanded, the meta-analysis can be the leading source of information. We illustrate the method for time-to-event data, showing how synthesizing data at interim stages of studies can increase efficiency when studies are slow in themselves to provide the necessary number of events for completion. The meta-analysis can be performed on interim data, but does not have to. The analysis design requires no information about the number of patients in trials or the number of trials eventually included. So it can breathe life into living systematic reviews, through better and simpler statistics, efficiency, collaboration and communication.
△ Less
Submitted 24 September, 2021;
originally announced September 2021.
-
The Anytime-Valid Logrank Test: Error Control Under Continuous Monitoring with Unlimited Horizon
Authors:
J. ter Schure,
M. F. Perez-Ortiz,
A. Ly,
P. Grunwald
Abstract:
We introduce the anytime-valid (AV) logrank test, a version of the logrank test that provides type-I error guarantees under optional stop** and optional continuation. The test is sequential without the need to specify a maximum sample size or stop** rule, and allows for cumulative meta-analysis with type-I error control. The method can be extended to define anytime-valid confidence intervals.…
▽ More
We introduce the anytime-valid (AV) logrank test, a version of the logrank test that provides type-I error guarantees under optional stop** and optional continuation. The test is sequential without the need to specify a maximum sample size or stop** rule, and allows for cumulative meta-analysis with type-I error control. The method can be extended to define anytime-valid confidence intervals. The logrank test is an instance of the martingale tests based on E-variables that have been recently developed. We demonstrate type-I error guarantees for the test in a semiparametric setting of proportional hazards and show how to extend it to ties, Cox' regression and confidence sequences. Using a Gaussian approximation on the logrank statistic, we show that the AV logrank test (which itself is always exact) has a similar rejection region to O'Brien-Fleming alpha-spending but with the potential to achieve 100% power by optional continuation. Although our approach to study design requires a larger sample size, the *expected* sample size is competitive by optional stop**.
△ Less
Submitted 1 May, 2023; v1 submitted 13 November, 2020;
originally announced November 2020.
-
Accumulation Bias in Meta-Analysis: The Need to Consider Time in Error Control
Authors:
Judith ter Schure,
Peter D. Grünwald
Abstract:
Studies accumulate over time and meta-analyses are mainly retrospective. These two characteristics introduce dependencies between the analysis time, at which a series of studies is up for meta-analysis, and results within the series. Dependencies introduce bias --- Accumulation Bias --- and invalidate the sampling distribution assumed for p-value tests, thus inflating type-I errors. But dependenci…
▽ More
Studies accumulate over time and meta-analyses are mainly retrospective. These two characteristics introduce dependencies between the analysis time, at which a series of studies is up for meta-analysis, and results within the series. Dependencies introduce bias --- Accumulation Bias --- and invalidate the sampling distribution assumed for p-value tests, thus inflating type-I errors. But dependencies are also inevitable, since for science to accumulate efficiently, new research needs to be informed by past results. Here, we investigate various ways in which time influences error control in meta-analysis testing. We introduce an Accumulation Bias Framework that allows us to model a wide variety of practically occurring dependencies, including study series accumulation, meta-analysis timing, and approaches to multiple testing in living systematic reviews. The strength of this framework is that it shows how all dependencies affect p-value-based tests in a similar manner. This leads to two main conclusions. First, Accumulation Bias is inevitable, and even if it can be approximated and accounted for, no valid p-value tests can be constructed. Second, tests based on likelihood ratios withstand Accumulation Bias: they provide bounds on error probabilities that remain valid despite the bias. We leave the reader with a choice between two proposals to consider time in error control: either treat individual (primary) studies and meta-analyses as two separate worlds --- each with their own timing --- or integrate individual studies in the meta-analysis world. Taking up likelihood ratios in either approach allows for valid tests that relate well to the accumulating nature of scientific knowledge. Likelihood ratios can be interpreted as betting profits, earned in previous studies and invested in new ones, while the meta-analyst is allowed to cash out at any time and advise against future studies.
△ Less
Submitted 31 May, 2019;
originally announced May 2019.
