-
A Tutorial on Time-Dependent Cohort State-Transition Models in R using a Cost-Effectiveness Analysis Example
Authors:
Fernando Alarid-Escudero,
Eline M. Krijkamp,
Eva A. Enns,
Alan Yang,
M. G. Myriam Hunink,
Petros Pechlivanoglou,
Hawre Jalal
Abstract:
In an introductory tutorial, we illustrated building cohort state-transition models (cSTMs) in R, where the state transitions probabilities were constant over time. However, in practice, many cSTMs require transitions, rewards, or both to vary over time (time-dependent). This tutorial illustrates adding two types of time-dependency using a previously published cost-effectiveness analysis of multip…
▽ More
In an introductory tutorial, we illustrated building cohort state-transition models (cSTMs) in R, where the state transitions probabilities were constant over time. However, in practice, many cSTMs require transitions, rewards, or both to vary over time (time-dependent). This tutorial illustrates adding two types of time-dependency using a previously published cost-effectiveness analysis of multiple strategies as an example. The first is simulation-time dependence, which allows for the transition probabilities to vary as a function of time as measured since the start of the simulation (e.g., varying probability of death as the cohort ages). The second is state-residence time dependence, allowing for history by tracking the time spent in any particular health state using tunnel states. We use these time-dependent cSTMs to conduct cost-effectiveness and probabilistic sensitivity analyses. We also obtain various epidemiological outcomes of interest from the outputs generated from the cSTM, such as survival probability and disease prevalence, often used for model calibration and validation. We present the mathematical notation first, followed by the R code to execute the calculations. The full R code is provided in a public code repository for broader implementation.
△ Less
Submitted 7 June, 2022; v1 submitted 30 August, 2021;
originally announced August 2021.
-
BayCANN: Streamlining Bayesian Calibration with Artificial Neural Network Metamodeling
Authors:
Hawre Jalal,
Fernando Alarid-Escudero
Abstract:
Purpose: Bayesian calibration is theoretically superior to standard direct-search algorithm because it can reveal the full joint posterior distribution of the calibrated parameters. However, to date, Bayesian calibration has not been used often in health decision sciences due to practical and computational burdens. In this paper we propose to use artificial neural networks (ANN) as one solution to…
▽ More
Purpose: Bayesian calibration is theoretically superior to standard direct-search algorithm because it can reveal the full joint posterior distribution of the calibrated parameters. However, to date, Bayesian calibration has not been used often in health decision sciences due to practical and computational burdens. In this paper we propose to use artificial neural networks (ANN) as one solution to these limitations.
Methods: Bayesian Calibration using Artificial Neural Networks (BayCANN) involves (1) training an ANN metamodel on a sample of model inputs and outputs, and (2) then calibrating the trained ANN metamodel instead of the full model in a probabilistic programming language to obtain the posterior joint distribution of the calibrated parameters. We demonstrate BayCANN by calibrating a natural history model of colorectal cancer to adenoma prevalence and cancer incidence data. In addition, we compare the efficiency and accuracy of BayCANN against performing a Bayesian calibration directly on the simulation model using an incremental mixture importance sampling (IMIS) algorithm.
Results: BayCANN was generally more accurate than IMIS in recovering the "true" parameter values. The ratio of the absolute ANN deviation from the truth compared to IMIS for eight out of the nine calibrated parameters were less than one indicating that BayCANN was more accurate than IMIS. In addition, BayCANN took about 15 minutes total compared to the IMIS method which took 80 minutes.
Conclusions: In our case study, BayCANN was more accurate than IMIS and was five-folds faster. Because BayCANN does not depend on the structure of the simulation model, it can be adapted to models of various levels of complexity with minor changes to its structure. We provide BayCANN's open-source implementation in R.
△ Less
Submitted 26 October, 2020;
originally announced October 2020.
-
An Introductory Tutorial on Cohort State-Transition Models in R Using a Cost-Effectiveness Analysis Example
Authors:
Fernando Alarid-Escudero,
Eline M. Krijkamp,
Eva A. Enns,
Alan Yang,
M. G. Myriam Hunink,
Petros Pechlivanoglou,
Hawre Jalal
Abstract:
Decision models can combine information from different sources to simulate the long-term consequences of alternative strategies in the presence of uncertainty. A cohort state-transition model (cSTM) is a decision model commonly used in medical decision-making to simulate the transitions of a hypothetical cohort among various health states over time. This tutorial focuses on time-independent cSTM,…
▽ More
Decision models can combine information from different sources to simulate the long-term consequences of alternative strategies in the presence of uncertainty. A cohort state-transition model (cSTM) is a decision model commonly used in medical decision-making to simulate the transitions of a hypothetical cohort among various health states over time. This tutorial focuses on time-independent cSTM, where transition probabilities among health states remain constant over time. We implement time-independent cSTM in R, an open-source mathematical and statistical programming language. We illustrate time-independent cSTMs using a previously published decision model, calculate costs and effectiveness outcomes, conduct a cost-effectiveness analysis of multiple strategies, including a probabilistic sensitivity analysis. We provide open-source code in R to facilitate wider adoption. In a second, more advanced tutorial, we illustrate time-dependent cSTMs.
△ Less
Submitted 30 March, 2022; v1 submitted 21 January, 2020;
originally announced January 2020.
-
Computing the Expected Value of Sample Information Efficiently: Expertise and Skills Required for Four Model-Based Methods
Authors:
Natalia R. Kunst,
Edward Wilson,
Fernando Alarid-Escudero,
Gianluca Baio,
Alan Brennan,
Michael Fairley,
David Glynn,
Jeremy D. Goldhaber-Fiebert,
Chris Jackson,
Hawre Jalal,
Nicolas A. Menzies,
Mark Strong,
Howard Thom,
Anna Heath
Abstract:
Objectives: Value of information (VOI) analyses can help policy-makers make informed decisions about whether to conduct and how to design future studies. Historically, a computationally expensive method to compute the Expected Value of Sample Information (EVSI) restricted the use of VOI to simple decision models and study designs. Recently, four EVSI approximation methods have made such analyses m…
▽ More
Objectives: Value of information (VOI) analyses can help policy-makers make informed decisions about whether to conduct and how to design future studies. Historically, a computationally expensive method to compute the Expected Value of Sample Information (EVSI) restricted the use of VOI to simple decision models and study designs. Recently, four EVSI approximation methods have made such analyses more feasible and accessible. We provide practical recommendations for analysts computing EVSI by evaluating these novel methods. Methods: Members of the Collaborative Network for Value of Information (ConVOI) compared the inputs, analyst's expertise and skills, and software required for four recently developed approximation methods. Information was also collected on the strengths and limitations of each approximation method. Results: All four EVSI methods require a decision-analytic model's probabilistic sensitivity analysis (PSA) output. One of the methods also requires the model to be re-run to obtain new PSA outputs for each EVSI estimation. To compute EVSI, analysts must be familiar with at least one of the following skills: advanced regression modeling, likelihood specification, and Bayesian modeling. All methods have different strengths and limitations, e.g., some methods handle evaluation of study designs with more outcomes more efficiently while others quantify uncertainty in EVSI estimates. All methods are programmed in the statistical language R and two of the methods provide online applications. Conclusion: Our paper helps to inform the choice between four efficient EVSI estimation methods, enabling analysts to assess the methods' strengths and limitations and select the most appropriate EVSI method given their situation and skills.
△ Less
Submitted 8 October, 2019;
originally announced October 2019.
-
Characterization and valuation of uncertainty of calibrated parameters in stochastic decision models
Authors:
Fernando Alarid-Escudero,
Amy B. Knudsen,
Jonathan Ozik,
Nicholson Collier,
Karen M. Kuntz
Abstract:
We evaluated the implications of different approaches to characterize uncertainty of calibrated parameters of stochastic decision models (DMs) in the quantified value of such uncertainty in decision making. We used a microsimulation DM of colorectal cancer (CRC) screening to conduct a cost-effectiveness analysis (CEA) of a 10-year colonoscopy screening. We calibrated the natural history model of C…
▽ More
We evaluated the implications of different approaches to characterize uncertainty of calibrated parameters of stochastic decision models (DMs) in the quantified value of such uncertainty in decision making. We used a microsimulation DM of colorectal cancer (CRC) screening to conduct a cost-effectiveness analysis (CEA) of a 10-year colonoscopy screening. We calibrated the natural history model of CRC to epidemiological data with different degrees of uncertainty and obtained the joint posterior distribution of the parameters using a Bayesian approach. We conducted a probabilistic sensitivity analysis (PSA) on all the model parameters with different characterizations of uncertainty of the calibrated parameters and estimated the value of uncertainty of the different characterizations with a value of information analysis. All analyses were conducted using high performance computing resources running the Extreme-scale Model Exploration with Swift (EMEWS) framework. The posterior distribution had high correlation among some parameters. The parameters of the Weibull hazard function for the age of onset of adenomas had the highest posterior correlation of -0.958. Considering full posterior distributions and the maximum-a-posteriori estimate of the calibrated parameters, there is little difference on the spread of the distribution of the CEA outcomes with a similar expected value of perfect information (EVPI) of \$653 and \$685, respectively, at a WTP of \$66,000/QALY. Ignoring correlation on the posterior distribution of the calibrated parameters, produced the widest distribution of CEA outcomes and the highest EVPI of \$809 at the same WTP. Different characterizations of uncertainty of calibrated parameters have implications on the expect value of reducing uncertainty on the CEA. Ignoring inherent correlation among calibrated parameters on a PSA overestimates the value of uncertainty.
△ Less
Submitted 11 June, 2019;
originally announced June 2019.
-
Calculating the Expected Value of Sample Information in Practice: Considerations from Three Case Studies
Authors:
Anna Heath,
Natalia R. Kunst,
Christopher Jackson,
Mark Strong,
Fernando Alarid-Escudero,
Jeremy D. Goldhaber-Fiebert,
Gianluca Baio,
Nicolas A. Menzies,
Hawre Jalal
Abstract:
Investing efficiently in future research to improve policy decisions is an important goal. Expected Value of Sample Information (EVSI) can be used to select the specific design and sample size of a proposed study by assessing the benefit of a range of different studies. Estimating EVSI with the standard nested Monte Carlo algorithm has a notoriously high computational burden, especially when using…
▽ More
Investing efficiently in future research to improve policy decisions is an important goal. Expected Value of Sample Information (EVSI) can be used to select the specific design and sample size of a proposed study by assessing the benefit of a range of different studies. Estimating EVSI with the standard nested Monte Carlo algorithm has a notoriously high computational burden, especially when using a complex decision model or when optimizing over study sample sizes and designs. Therefore, a number of more efficient EVSI approximation methods have been developed. However, these approximation methods have not been compared and therefore their relative advantages and disadvantages are not clear. A consortium of EVSI researchers, including the developers of several approximation methods, compared four EVSI methods using three previously published health economic models. The examples were chosen to represent a range of real-world contexts, including situations with multiple study outcomes, missing data, and data from an observational rather than a randomized study. The computational speed and accuracy of each method were compared, and the relative advantages and implementation challenges of the methods were highlighted. In each example, the approximation methods took minutes or hours to achieve reasonably accurate EVSI estimates, whereas the traditional Monte Carlo method took weeks. Specific methods are particularly suited to problems where we wish to compare multiple proposed sample sizes, when the proposed sample size is large, or when the health economic model is computationally expensive. All the evaluated methods gave estimates similar to those given by traditional Monte Carlo, suggesting that EVSI can now be efficiently computed with confidence in realistic examples.
△ Less
Submitted 28 May, 2019;
originally announced May 2019.
