-
Covariate-adjusted marginal cumulative incidence curves for competing risk analysis
Authors:
Patrick van Hage,
Saskia le Cessie,
Marissa C. van Maaren,
Hein Putter,
Nan van Geloven
Abstract:
Covariate imbalance between treatment groups makes it difficult to compare cumulative incidence curves in competing risk analyses. In this paper we discuss different methods to estimate adjusted cumulative incidence curves including inverse probability of treatment weighting and outcome regression modeling. For these methods to work, correct specification of the propensity score model or outcome r…
▽ More
Covariate imbalance between treatment groups makes it difficult to compare cumulative incidence curves in competing risk analyses. In this paper we discuss different methods to estimate adjusted cumulative incidence curves including inverse probability of treatment weighting and outcome regression modeling. For these methods to work, correct specification of the propensity score model or outcome regression model, respectively, is needed. We introduce a new doubly robust estimator, which requires correct specification of only one of the two models. We conduct a simulation study to assess the performance of these three methods, including scenarios with model misspecification of the relationship between covariates and treatment and/or outcome. We illustrate their usage in a cohort study of breast cancer patients estimating covariate-adjusted marginal cumulative incidence curves for recurrence, second primary tumour development and death after undergoing mastectomy treatment or breast-conserving therapy. Our study points out the advantages and disadvantages of each covariate adjustment method when applied in competing risk analysis.
△ Less
Submitted 24 March, 2024;
originally announced March 2024.
-
The risks of risk assessment: causal blind spots when using prediction models for treatment decisions
Authors:
Nan van Geloven,
Ruth H Keogh,
Wouter van Amsterdam,
Giovanni Cinà,
Jesse H. Krijthe,
Niels Peek,
Kim Luijken,
Sara Magliacane,
Paweł Morzywołek,
Thijs van Ommen,
Hein Putter,
Matthew Sperrin,
Junfeng Wang,
Daniala L. Weir,
Vanessa Didelez
Abstract:
Prediction models are increasingly proposed for guiding treatment decisions, but most fail to address the special role of treatments, leading to inappropriate use. This paper highlights the limitations of using standard prediction models for treatment decision support. We identify `causal blind spots' in three common approaches to handling treatments in prediction modelling: including treatment as…
▽ More
Prediction models are increasingly proposed for guiding treatment decisions, but most fail to address the special role of treatments, leading to inappropriate use. This paper highlights the limitations of using standard prediction models for treatment decision support. We identify `causal blind spots' in three common approaches to handling treatments in prediction modelling: including treatment as a predictor, restricting data based on treatment status and ignoring treatments. When predictions are used to inform treatment decisions, confounders, colliders and mediators, as well as changes in treatment protocols over time may lead to misinformed decision-making. We illustrate potential harmful consequences in several medical applications. We advocate for an extension of guidelines for development, reporting and evaluation of prediction models to ensure that the intended use of the model is matched to an appropriate risk estimand. When prediction models are intended to inform treatment decisions, prediction models should specify upfront the treatment decisions they aim to support and target a prediction estimand in line with that goal. This requires a shift towards develo** predictions under the specific treatment options under consideration (`predictions under interventions'). Predictions under interventions need causal reasoning and inference techniques during development and validation. We argue that this will improve the efficacy of prediction models in guiding treatment decisions and prevent potential negative effects on patient outcomes.
△ Less
Submitted 6 May, 2024; v1 submitted 27 February, 2024;
originally announced February 2024.
-
When accurate prediction models yield harmful self-fulfilling prophecies
Authors:
Wouter A. C. van Amsterdam,
Nan van Geloven,
Jesse H. Krijthe,
Rajesh Ranganath,
Giovanni Ciná
Abstract:
Objective: Prediction models are popular in medical research and practice. By predicting an outcome of interest for specific patients, these models may help inform difficult treatment decisions, and are often hailed as the poster children for personalized, data-driven healthcare. Many prediction models are deployed for decision support based on their prediction accuracy in validation studies. We i…
▽ More
Objective: Prediction models are popular in medical research and practice. By predicting an outcome of interest for specific patients, these models may help inform difficult treatment decisions, and are often hailed as the poster children for personalized, data-driven healthcare. Many prediction models are deployed for decision support based on their prediction accuracy in validation studies. We investigate whether this is a safe and valid approach.
Materials and Methods: We show that using prediction models for decision making can lead to harmful decisions, even when the predictions exhibit good discrimination after deployment. These models are harmful self-fulfilling prophecies: their deployment harms a group of patients but the worse outcome of these patients does not invalidate the predictive power of the model.
Results: Our main result is a formal characterization of a set of such prediction models. Next we show that models that are well calibrated before and after deployment are useless for decision making as they made no change in the data distribution.
Discussion: Our results point to the need to revise standard practices for validation, deployment and evaluation of prediction models that are used in medical decisions.
Conclusion: Outcome prediction models can yield harmful self-fulfilling prophecies when used for decision making, a new perspective on prediction model development, deployment and monitoring is needed.
△ Less
Submitted 8 February, 2024; v1 submitted 2 December, 2023;
originally announced December 2023.
-
Risk-based decision making: estimands for sequential prediction under interventions
Authors:
Kim Luijken,
Paweł Morzywołek,
Wouter van Amsterdam,
Giovanni Cinà,
Jeroen Hoogland,
Ruth Keogh,
Jesse Krijthe,
Sara Magliacane,
Thijs van Ommen,
Niels Peek,
Hein Putter,
Maarten van Smeden,
Matthew Sperrin,
Junfeng Wang,
Daniala Weir,
Vanessa Didelez,
Nan van Geloven
Abstract:
Prediction models are used amongst others to inform medical decisions on interventions. Typically, individuals with high risks of adverse outcomes are advised to undergo an intervention while those at low risk are advised to refrain from it. Standard prediction models do not always provide risks that are relevant to inform such decisions: e.g., an individual may be estimated to be at low risk beca…
▽ More
Prediction models are used amongst others to inform medical decisions on interventions. Typically, individuals with high risks of adverse outcomes are advised to undergo an intervention while those at low risk are advised to refrain from it. Standard prediction models do not always provide risks that are relevant to inform such decisions: e.g., an individual may be estimated to be at low risk because similar individuals in the past received an intervention which lowered their risk. Therefore, prediction models supporting decisions should target risks belonging to defined intervention strategies. Previous works on prediction under interventions assumed that the prediction model was used only at one time point to make an intervention decision. In clinical practice, intervention decisions are rarely made only once: they might be repeated, deferred and re-evaluated. This requires estimated risks under interventions that can be reconsidered at several potential decision moments. In the current work, we highlight key considerations for formulating estimands in sequential prediction under interventions that can inform such intervention decisions. We illustrate these considerations by giving examples of estimands for a case study about choosing between vaginal delivery and cesarean section for women giving birth. Our formalization of prediction tasks in a sequential, causal, and estimand context provides guidance for future studies to ensure that the right question is answered and appropriate causal estimation approaches are chosen to develop sequential prediction models that can inform intervention decisions.
△ Less
Submitted 29 November, 2023;
originally announced November 2023.
-
Prediction under interventions: evaluation of counterfactual performance using longitudinal observational data
Authors:
Ruth H. Keogh,
Nan van Geloven
Abstract:
Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision making. However, evaluating predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance d…
▽ More
Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision making. However, evaluating predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different to those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting which involves creating a validation dataset that mimics the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.
△ Less
Submitted 10 January, 2024; v1 submitted 19 April, 2023;
originally announced April 2023.
-
Time-lag bias induced by unobserved heterogeneity: comparing treated patients to controls with a different start of follow-up
Authors:
Rik van Eekelen,
Patrick M. M. Bossuyt,
Nan van Geloven
Abstract:
In comparative effectiveness research, treated and control patients might have a different start of follow-up as treatment is often started later in the disease trajectory. This typically occurs when data from treated and controls are not collected within the same source. Only patients who did not yet experience the event of interest whilst in the control condition end up in the treatment data sou…
▽ More
In comparative effectiveness research, treated and control patients might have a different start of follow-up as treatment is often started later in the disease trajectory. This typically occurs when data from treated and controls are not collected within the same source. Only patients who did not yet experience the event of interest whilst in the control condition end up in the treatment data source. In case of unobserved heterogeneity, these treated patients will have a lower average risk than the controls. We illustrate how failing to account for this time-lag between treated and controls leads to bias in the estimated treatment effect. We define estimands and time axes, then explore five methods to adjust for this time-lag bias by utilising the time between diagnosis and treatment initiation in different ways. We conducted a simulation study to evaluate whether these methods reduce the bias and then applied the methods to a comparison between fertility patients treated with insemination and similar but untreated patients. We conclude that time-lag bias can be vast and that the time between diagnosis and treatment initiation should be taken into account in the analysis to respect the chronology of the disease and treatment trajectory.
△ Less
Submitted 7 June, 2024; v1 submitted 17 May, 2021;
originally announced May 2021.
-
Prediction meets causal inference: the role of treatment in clinical prediction models
Authors:
Nan van Geloven,
Sonja Swanson,
Chava Ramspek,
Kim Luijken,
Merel van Diepen,
Tim Morris,
Rolf Groenwold,
Hans van Houwelingen,
Hein Putter,
Saskia le Cessie
Abstract:
In this paper we study approaches for dealing with treatment when develo** a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a `predictimand' framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition…
▽ More
In this paper we study approaches for dealing with treatment when develo** a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a `predictimand' framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition of the estimands matching these questions, give examples of settings in which each is useful and discuss appropriate estimators including their assumptions. We illustrate the impact of the predictimand choice in a dataset of patients with end-stage kidney disease. We argue that clearly defining the estimand is equally important in prediction research as in causal inference.
△ Less
Submitted 15 April, 2020;
originally announced April 2020.
