-
Why multiple hypothesis test corrections provide poor control of false positives in the real world
Authors:
Stanley E. Lazic
Abstract:
Most scientific disciplines use significance testing to draw conclusions about experimental or observational data. This classical approach provides a theoretical guarantee for controlling the number of false positives across a set of hypothesis tests, making it an appealing framework for scientists seeking to limit the number of false effects or associations that they claim to observe. Unfortunate…
▽ More
Most scientific disciplines use significance testing to draw conclusions about experimental or observational data. This classical approach provides a theoretical guarantee for controlling the number of false positives across a set of hypothesis tests, making it an appealing framework for scientists seeking to limit the number of false effects or associations that they claim to observe. Unfortunately, this theoretical guarantee applies to few experiments, and the true false positive rate (FPR) is much higher. Scientists have plenty of freedom to choose the error rate to control, the tests to include in the adjustment, and the method of correction, making strong error control difficult to attain. In addition, hypotheses are often tested after finding unexpected relationships or patterns, the data are analysed in several ways, and analyses may be run repeatedly as data accumulate. As a result, adjusted p-values are too small, incorrect conclusions are often reached, and results are harder to reproduce. In the following, I argue why the FPR is rarely controlled meaningfully and why shrinking parameter estimates is preferable to p-value adjustments.
△ Less
Submitted 3 March, 2023; v1 submitted 10 August, 2021;
originally announced August 2021.
-
Quantifying sources of uncertainty in drug discovery predictions with probabilistic models
Authors:
Stanley E. Lazic,
Dominic P. Williams
Abstract:
Knowing the uncertainty in a prediction is critical when making expensive investment decisions and when patient safety is paramount, but machine learning (ML) models in drug discovery typically provide only a single best estimate and ignore all sources of uncertainty. Predictions from these models may therefore be over-confident, which can put patients at risk and waste resources when compounds th…
▽ More
Knowing the uncertainty in a prediction is critical when making expensive investment decisions and when patient safety is paramount, but machine learning (ML) models in drug discovery typically provide only a single best estimate and ignore all sources of uncertainty. Predictions from these models may therefore be over-confident, which can put patients at risk and waste resources when compounds that are destined to fail are further developed. Probabilistic predictive models (PPMs) can incorporate uncertainty in both the data and model, and return a distribution of predicted values that represents the uncertainty in the prediction. PPMs not only let users know when predictions are uncertain, but the intuitive output from these models makes communicating risk easier and decision making better. Many popular machine learning methods have a PPM or Bayesian analogue, making PPMs easy to fit into current workflows. We use toxicity prediction as a running example, but the same principles apply for all prediction models used in drug discovery. The consequences of ignoring uncertainty and how PPMs account for uncertainty are also described. We aim to make the discussion accessible to a broad non-mathematical audience. Equations are provided to make ideas concrete for mathematical readers (but can be skipped without loss of understanding) and code is available for computational researchers (https://github.com/stanlazic/ML_uncertainty_quantification).
△ Less
Submitted 18 May, 2021;
originally announced May 2021.
-
Quantifying the behavioural relevance of hippocampal neurogenesis
Authors:
Stanley E. Lazic,
Johannes Fuss,
Peter Gass
Abstract:
Few studies that examine the neurogenesis--behaviour relationship formally establish covariation between neurogenesis and behaviour or rule out competing explanations. The behavioural relevance of neurogenesis might therefore be overestimated if other mechanisms account for some, or even all, of the experimental effects. A systematic review of the literature was conducted and the data reanalysed u…
▽ More
Few studies that examine the neurogenesis--behaviour relationship formally establish covariation between neurogenesis and behaviour or rule out competing explanations. The behavioural relevance of neurogenesis might therefore be overestimated if other mechanisms account for some, or even all, of the experimental effects. A systematic review of the literature was conducted and the data reanalysed using causal mediation analysis, which can estimate the behavioural contribution of new hippocampal neurons separately from other mechanisms that might be operating. Results from eleven eligible individual studies were then combined in a meta-analysis to increase precision (representing data from 215 animals) and showed that neurogenesis made a negligible contribution to behaviour (standarised effect = 0.15; 95% CI = -0.04 to 0.34; p = 0.128); other mechanisms accounted for the majority of experimental effects (standardised effect = 1.06; 95% CI = 0.74 to 1.38; p = 1.7 $\times 10^{-11}$).
△ Less
Submitted 9 November, 2014; v1 submitted 21 May, 2014;
originally announced May 2014.
-
Improving basic and translational science by accounting for litter-to-litter variation in animal models
Authors:
Stanley E. Lazic,
Laurent Essioux
Abstract:
Background: Animals from the same litter are often more alike compared with animals from different litters. This litter-to-litter variation, or "litter effects", can influence the results in addition to the experimental factors of interest. Furthermore, an experimental treatment can be applied to whole litters rather than to individual offspring. For example, in the valproic acid (VPA) model of au…
▽ More
Background: Animals from the same litter are often more alike compared with animals from different litters. This litter-to-litter variation, or "litter effects", can influence the results in addition to the experimental factors of interest. Furthermore, an experimental treatment can be applied to whole litters rather than to individual offspring. For example, in the valproic acid (VPA) model of autism, VPA is administered to pregnant females thereby inducing the disease phenotype in the offspring. With this type of experiment the sample size is the number of litters and not the total number of offspring. If such experiments are not appropriately designed and analysed, the results can be severely biased as well as extremely underpowered.
Results: A review of the VPA literature showed that only 9% (3/34) of studies correctly determined that the experimental unit (n) was the litter and therefore made valid statistical inferences. In addition, litter effects accounted for up to 61% (p <0.001) of the variation in behavioural outcomes, which was larger than the treatment effects. In addition, few studies reported using randomisation (12%) or blinding (18%), and none indicated that a sample size calculation or power analysis had been conducted.
Conclusions: Litter effects are common, large, and ignoring them can make replication of findings difficult and can contribute to the low rate of translating preclinical in vivo studies into successful therapies. Only a minority of studies reported using rigorous experimental methods, which is consistent with much of the preclinical in vivo literature.
△ Less
Submitted 22 March, 2013; v1 submitted 30 November, 2012;
originally announced November 2012.
-
Modelling hippocampal neurogenesis across the lifespan in seven species
Authors:
Stanley E. Lazic
Abstract:
The aim of this study was to estimate the number of new cells and neurons added to the dentate gyrus across the lifespan, and to compare the rate of age-associated decline in neurogenesis across species. Data from mice (Mus musculus), rats (Rattus norvegicus), lesser hedgehog tenrecs (Echinops telfairi), macaques (Macaca mulatta), marmosets (Callithrix jacchus), tree shrews (Tupaia belangeri), and…
▽ More
The aim of this study was to estimate the number of new cells and neurons added to the dentate gyrus across the lifespan, and to compare the rate of age-associated decline in neurogenesis across species. Data from mice (Mus musculus), rats (Rattus norvegicus), lesser hedgehog tenrecs (Echinops telfairi), macaques (Macaca mulatta), marmosets (Callithrix jacchus), tree shrews (Tupaia belangeri), and humans (Homo sapiens) were extracted from twenty one data sets published in fourteen different papers. ANOVA, exponential, Weibull, and power models were fit to the data to determine which best described the relationship between age and neurogenesis. Exponential models provided a suitable fit and were used to estimate the relevant parameters. The rate of decrease of neurogenesis correlated with species longevity r = 0.769, p = 0.043), but not body mass or basal metabolic rate. Of all the cells added postnatally to the mouse dentate gyrus, only 8.5% (95% CI = 1.0% to 14.7%) of these will be added after middle age. In addition, only 5.7% (95% CI = 0.7% to 9.9%) of the existing cell population turns over from middle age onwards. Thus, relatively few new cells are added for much of an animal's life, and only a proportion of these will mature into functional neurons.
△ Less
Submitted 3 May, 2011;
originally announced May 2011.
-
Using causal models to distinguish between neurogenesis-dependent and -independent effects on behaviour
Authors:
Stanley E. Lazic
Abstract:
There has been a substantial amount of research on the relationship between hippocampal neurogenesis and behaviour over the past fifteen years, but the causal role that new neurons have on cognitive and affective behavioural tasks is still far from clear. This is partly due to the difficulty of manipulating levels of neurogenesis without inducing off-target effects, which might also influence beha…
▽ More
There has been a substantial amount of research on the relationship between hippocampal neurogenesis and behaviour over the past fifteen years, but the causal role that new neurons have on cognitive and affective behavioural tasks is still far from clear. This is partly due to the difficulty of manipulating levels of neurogenesis without inducing off-target effects, which might also influence behaviour. In addition, the analytical methods typically used do not directly test whether neurogenesis mediates the effect of an intervention on behaviour. Previous studies may have incorrectly attributed changes in behavioural performance to neurogenesis because the role of known (or unknown) neurogenesis-independent mechanisms were not formally taken into consideration during the analysis. Causal models can tease apart complex causal relationships and were used to demonstrate that the effect of exercise on pattern separation is via neurogenesis-independent mechanisms. Many studies in the neurogenesis literature would benefit from the use of statistical methods that can separate neurogenesis-dependent from neurogenesis-independent effects on behaviour.
△ Less
Submitted 7 September, 2011; v1 submitted 29 April, 2011;
originally announced April 2011.
