-
Replicability of Simulation Studies for the Investigation of Statistical Methods: The RepliSims Project
Authors:
K. Luijken,
A. Lohmann,
U. Alter,
J. Claramunt Gonzalez,
F. J. Clouth,
J. L. Fossum,
L. Hesen,
A. H. J. Huizing,
J. Ketelaar,
A. K. Montoya,
L. Nab,
R. C. C. Nijman,
B. B. L. Penning de Vries,
T. D. Tibbe,
Y. A. Wang,
R. H. H. Groenwold
Abstract:
Results of simulation studies evaluating the performance of statistical methods are often considered actionable and thus can have a major impact on the way empirical research is implemented. However, so far there is limited evidence about the reproducibility and replicability of statistical simulation studies. Therefore, eight highly cited statistical simulation studies were selected, and their re…
▽ More
Results of simulation studies evaluating the performance of statistical methods are often considered actionable and thus can have a major impact on the way empirical research is implemented. However, so far there is limited evidence about the reproducibility and replicability of statistical simulation studies. Therefore, eight highly cited statistical simulation studies were selected, and their replicability was assessed by teams of replicators with formal training in quantitative methodology. The teams found relevant information in the original publications and used it to write simulation code with the aim of replicating the results. The primary outcome was the feasibility of replicability based on reported information in the original publications. Replicability varied greatly: Some original studies provided detailed information leading to almost perfect replication of results, whereas other studies did not provide enough information to implement any of the reported simulations. Replicators had to make choices regarding missing or ambiguous information in the original studies, error handling, and software environment. Factors facilitating replication included public availability of code, and descriptions of the data-generating procedure and methods in graphs, formulas, structured text, and publicly accessible additional resources such as technical reports. Replicability of statistical simulation studies was mainly impeded by lack of information and sustainability of information sources. Reproducibility could be achieved for simulation studies by providing open code and data as a supplement to the publication. Additionally, simulation studies should be transparently reported with all relevant information either in the research paper itself or in easily accessible supplementary material to allow for replicability.
△ Less
Submitted 5 July, 2023;
originally announced July 2023.
-
Sensitivity analysis for random measurement error using regression calibration and simulation-extrapolation
Authors:
Linda Nab,
Rolf H. H. Groenwold
Abstract:
Sensitivity analysis for measurement error can be applied in the absence of validation data by means of regression calibration and simulation-extrapolation. These have not been compared for this purpose. A simulation study was conducted comparing the performance of regression calibration and simulation-extrapolation in a multivariable model. The performance of the two methods was evaluated in term…
▽ More
Sensitivity analysis for measurement error can be applied in the absence of validation data by means of regression calibration and simulation-extrapolation. These have not been compared for this purpose. A simulation study was conducted comparing the performance of regression calibration and simulation-extrapolation in a multivariable model. The performance of the two methods was evaluated in terms of bias, mean squared error (MSE) and confidence interval coverage, for ranging reliability of the error-prone measurement (0.2-0.9), sample size (125-1,000), number of replicates (2-10), and R-squared (0.03-0.75). It was assumed that no validation data were available about the error-free measures, while measurement error variance was correctly estimated. In various scenarios, regression calibration was unbiased while simulation-extrapolation was biased: median bias was 1.4% (interquartile range (IQR): 0.8;2%), and -12.8% (IQR: -13.2;-11.0%), respectively. A small gain in efficiency was observed for simulation-extrapolation (median MSE: 0.005, IQR: 0.004;0.006) versus regression calibration (median MSE: 0.006, IQR: 0.004;0.007). Confidence interval coverage was at the nominal level of 95% for regression calibration, and smaller than 95% for simulation-extrapolation (median coverage: 92%, IQR: 85;94%). In the absence of validation data, the use of regression calibration is recommended for sensitivity analysis for measurement error.
△ Less
Submitted 8 June, 2021;
originally announced June 2021.
-
mecor: An R package for measurement error correction in linear regression models with a continuous outcome
Authors:
Linda Nab,
Maarten van Smeden,
Ruth H. Keogh,
Rolf H. H. Groenwold
Abstract:
Measurement error in a covariate or the outcome of regression models is common, but is often ignored, even though measurement error can lead to substantial bias in the estimated covariate-outcome association. While several texts on measurement error correction methods are available, these methods remain seldomly applied. To improve the use of measurement error correction methodology, we developed…
▽ More
Measurement error in a covariate or the outcome of regression models is common, but is often ignored, even though measurement error can lead to substantial bias in the estimated covariate-outcome association. While several texts on measurement error correction methods are available, these methods remain seldomly applied. To improve the use of measurement error correction methodology, we developed mecor, an R package that implements measurement error correction methods for regression models with continuous outcomes. Measurement error correction requires information about the measurement error model and its parameters. This information can be obtained from four types of studies, used to estimate the parameters of the measurement error model: an internal validation study, a replicates study, a calibration study and an external validation study. In the package mecor, regression calibration methods and a maximum likelihood method are implemented to correct for measurement error in a continuous covariate in regression analyses. Additionally, methods of moments methods are implemented to correct for measurement error in the continuous outcome in regression analyses. Variance estimation of the corrected estimators is provided in closed form and using the bootstrap.
△ Less
Submitted 9 February, 2021;
originally announced February 2021.
-
Sensitivity analysis for bias due to a misclassfied confounding variable in marginal structural models
Authors:
Linda Nab,
Rolf H. H. Groenwold,
Maarten van Smeden,
Ruth H. Keogh
Abstract:
In observational research treatment effects, the average treatment effect (ATE) estimator may be biased if a confounding variable is misclassified. We discuss the impact of classification error in a dichotomous confounding variable in analyses using marginal structural models estimated using inverse probability weighting (MSMs-IPW) and compare this with its impact in conditional regression models,…
▽ More
In observational research treatment effects, the average treatment effect (ATE) estimator may be biased if a confounding variable is misclassified. We discuss the impact of classification error in a dichotomous confounding variable in analyses using marginal structural models estimated using inverse probability weighting (MSMs-IPW) and compare this with its impact in conditional regression models, focusing on a point-treatment study with a continuous outcome. Expressions were derived for the bias in the ATE estimator from a MSM-IPW and conditional model by using the potential outcome framework. Based on these expressions, we propose a sensitivity analysis to investigate and quantify the bias due to classification error in a confounding variable in MSMs-IPW. Compared to bias in the ATE estimator from a conditional model, the bias in MSM-IPW can be dissimilar in magnitude but the bias will always be equal in sign. A simulation study was conducted to study the finite sample performance of MSMs-IPW and conditional models if a confounding variable is misclassified. Simulation results showed that confidence intervals of the treatment effect obtained from MSM-IPW are generally wider and coverage of the true treatment effect is higher compared to a conditional model, ranging from over coverage if there is no classification error to smaller under coverage when there is classification error. The use of the bias expressions to inform a sensitivity analysis was demonstrated in a study of blood pressure lowering therapy. It is important to consider the potential impact of classification error in a confounding variable in studies of treatment effects and a sensitivity analysis provides an opportunity to quantify the impact of such errors on causal conclusions. An online tool for sensitivity analyses was developed: https://lindanab.shinyapps.io/SensitivityAnalysis.
△ Less
Submitted 12 December, 2019;
originally announced December 2019.
-
Measurement error in continuous endpoints in randomised trials: problems and solutions
Authors:
Linda Nab,
Rolf H. H. Groenwold,
Paco M. J. Welsing,
Maarten van Smeden
Abstract:
In randomised trials, continuous endpoints are often measured with some degree of error. This study explores the impact of ignoring measurement error, and proposes methods to improve statistical inference in the presence of measurement error. Three main types of measurement error in continuous endpoints are considered: classical, systematic and differential. For each measurement error type, a corr…
▽ More
In randomised trials, continuous endpoints are often measured with some degree of error. This study explores the impact of ignoring measurement error, and proposes methods to improve statistical inference in the presence of measurement error. Three main types of measurement error in continuous endpoints are considered: classical, systematic and differential. For each measurement error type, a corrected effect estimator is proposed. The corrected estimators and several methods for confidence interval estimation are tested in a simulation study. These methods combine information about error-prone and error-free measurements of the endpoint in individuals not included in the trial (external calibration sample). We show that if measurement error in continuous endpoints is ignored, the treatment effect estimator is unbiased when measurement error is classical, while Type-II error is increased at a given sample size. Conversely, the estimator can be substantially biased when measurement error is systematic or differential. In those cases, bias can largely be prevented and inferences improved upon using information from an external calibration sample, of which the required sample size increases as the strength of the association between the error-prone and error-free endpoint decreases. Measurement error correction using already a small (external) calibration sample is shown to improve inferences and should be considered in trials with error-prone endpoints. Implementation of the proposed correction methods is accommodated by a new software package for R.
△ Less
Submitted 29 August, 2019; v1 submitted 19 September, 2018;
originally announced September 2018.
