-
Multi-Source Conformal Inference Under Distribution Shift
Authors:
Yi Liu,
Alexander W. Levis,
Sharon-Lise Normand,
Larry Han
Abstract:
Recent years have experienced increasing utilization of complex machine learning models across multiple sources of data to inform more generalizable decision-making. However, distribution shifts across data sources and privacy concerns related to sharing individual-level data, coupled with a lack of uncertainty quantification from machine learning predictions, make it challenging to achieve valid…
▽ More
Recent years have experienced increasing utilization of complex machine learning models across multiple sources of data to inform more generalizable decision-making. However, distribution shifts across data sources and privacy concerns related to sharing individual-level data, coupled with a lack of uncertainty quantification from machine learning predictions, make it challenging to achieve valid inferences in multi-source environments. In this paper, we consider the problem of obtaining distribution-free prediction intervals for a target population, leveraging multiple potentially biased data sources. We derive the efficient influence functions for the quantiles of unobserved outcomes in the target and source populations, and show that one can incorporate machine learning prediction algorithms in the estimation of nuisance functions while still achieving parametric rates of convergence to nominal coverage probabilities. Moreover, when conditional outcome invariance is violated, we propose a data-adaptive strategy to upweight informative data sources for efficiency gain and downweight non-informative data sources for bias reduction. We highlight the robustness and efficiency of our proposals for a variety of conformal scores and data-generating mechanisms via extensive synthetic experiments. Hospital length of stay prediction intervals for pediatric patients undergoing a high-risk cardiac surgical procedure between 2016-2022 in the U.S. illustrate the utility of our methodology.
△ Less
Submitted 15 May, 2024;
originally announced May 2024.
-
A Transfer Learning Causal Approach to Evaluate Racial/Ethnic and Geographic Variation in Outcomes Following Congenital Heart Surgery
Authors:
Larry Han,
Yi Zhang,
Meena Nathan,
John E. Mayer, Jr.,
Sara K. Pasquali,
Katya Zelevinsky,
Rui Duan,
Sharon-Lise T. Normand
Abstract:
Congenital heart defects (CHD) are the most prevalent birth defects in the United States and surgical outcomes vary considerably across the country. The outcomes of treatment for CHD differ for specific patient subgroups, with non-Hispanic Black and Hispanic populations experiencing higher rates of mortality and morbidity. A valid comparison of outcomes within racial/ethnic subgroups is difficult…
▽ More
Congenital heart defects (CHD) are the most prevalent birth defects in the United States and surgical outcomes vary considerably across the country. The outcomes of treatment for CHD differ for specific patient subgroups, with non-Hispanic Black and Hispanic populations experiencing higher rates of mortality and morbidity. A valid comparison of outcomes within racial/ethnic subgroups is difficult given large differences in case-mix and small subgroup sizes. We propose a causal inference framework for outcome assessment and leverage advances in transfer learning to incorporate data from both target and source populations to help estimate causal effects while accounting for different sources of risk factor and outcome differences across populations. Using the Society of Thoracic Surgeons' Congenital Heart Surgery Database (STS-CHSD), we focus on a national cohort of patients undergoing the Norwood operation from 2016-2022 to assess operative mortality and morbidity outcomes across U.S. geographic regions by race/ethnicity. We find racial and ethnic outcome differences after controlling for potential confounding factors. While geography does not have a causal effect on outcomes for non-Hispanic Caucasian patients, non-Hispanic Black patients experience wide variability in outcomes with estimated 30-day mortality ranging from 5.9% (standard error 2.2%) to 21.6% (4.4%) across U.S. regions.
△ Less
Submitted 21 March, 2024;
originally announced March 2024.
-
Multi-Dimensional Item Response Theory Models for Estimating Racial/Ethnic Health Care Quality Disparities
Authors:
Sharon-Lise Normand,
Katya Zelevinsky,
Marcela Horvitz-Lennon
Abstract:
Quality metrics in health care refer to a variety of measures used mainly to characterize what should have been done for a patient or the health consequences of what was done. When estimating quality of health care, often many metrics are measured and then combined to provide an overall estimate either at the patient level or at higher levels of accountability, such as the provider organization, i…
▽ More
Quality metrics in health care refer to a variety of measures used mainly to characterize what should have been done for a patient or the health consequences of what was done. When estimating quality of health care, often many metrics are measured and then combined to provide an overall estimate either at the patient level or at higher levels of accountability, such as the provider organization, insurer, or even geographic area. Racial/ethnic disparities are defined as the mean difference in overall quality between minorities and Whites not justified by underlying health conditions or patient preferences. However, several statistical features of health care quality data have frequently been ignored: quality is a theoretical construct that is not directly observed; the quality metrics are measured on different scales or, if measured on the same scale, have different baseline rates; the structure of the construct is likely multidimensional; and metrics are correlated within-patients. We address these features and utilize multi-dimensional item response theory models to estimate racial/ethnic quality disparities. Quality metrics measured on 93,000 adults with schizophrenia residing in 5 U.S. states illustrate approaches.
△ Less
Submitted 4 May, 2023;
originally announced May 2023.
-
Targeted learning in observational studies with multi-valued treatments: An evaluation of antipsychotic drug treatment safety
Authors:
Jason Poulos,
Marcela Horvitz-Lennon,
Katya Zelevinsky,
Tudor Cristea-Platon,
Thomas Huijskens,
Pooja Tyagi,
Jiaju Yan,
Jordi Diaz,
Sharon-Lise Normand
Abstract:
We investigate estimation of causal effects of multiple competing (multi-valued) treatments in the absence of randomization. Our work is motivated by an intention-to-treat study of the relative cardiometabolic risk of assignment to one of six commonly prescribed antipsychotic drugs in a cohort of nearly 39,000 adults with serious mental illnesses. Doubly-robust estimators, such as targeted minimum…
▽ More
We investigate estimation of causal effects of multiple competing (multi-valued) treatments in the absence of randomization. Our work is motivated by an intention-to-treat study of the relative cardiometabolic risk of assignment to one of six commonly prescribed antipsychotic drugs in a cohort of nearly 39,000 adults with serious mental illnesses. Doubly-robust estimators, such as targeted minimum loss-based estimation (TMLE), require correct specification of either the treatment model or outcome model to ensure consistent estimation; however, common TMLE implementations estimate treatment probabilities using multiple binomial regressions rather than multinomial regression. We implement a TMLE estimator that uses multinomial treatment assignment and ensemble machine learning to estimate average treatment effects. Our multinomial implementation improves coverage, but does not necessarily reduce bias, relative to the binomial implementation in simulation experiments with varying treatment propensity overlap and event rates. Evaluating the causal effects of the antipsychotics on three-year diabetes risk or death, we find a safety benefit of moving from a second-generation drug considered among the safest of the second-generation drugs to an infrequently prescribed first-generation drug known for having low cardiometabolic risk.
△ Less
Submitted 28 November, 2023; v1 submitted 30 June, 2022;
originally announced June 2022.
-
Measuring performance for end-of-life care
Authors:
Sebastien Haneuse,
Deborah Schrag,
Francesca Dominici,
Sharon-Lise Normand,
Kyu Ha Lee
Abstract:
Although not without controversy, readmission is entrenched as a hospital quality metric, with statistical analyses generally based on fitting a logistic-Normal generalized linear mixed model. Such analyses, however, ignore death as a competing risk, although doing so for clinical conditions with high mortality can have profound effects; a hospitals seemingly good performance for readmission may b…
▽ More
Although not without controversy, readmission is entrenched as a hospital quality metric, with statistical analyses generally based on fitting a logistic-Normal generalized linear mixed model. Such analyses, however, ignore death as a competing risk, although doing so for clinical conditions with high mortality can have profound effects; a hospitals seemingly good performance for readmission may be an artifact of it having poor performance for mortality. In this paper we propose novel multivariate hospital-level performance measures for readmission and mortality, that derive from framing the analysis as one of cluster-correlated semi-competing risks data. We also consider a number of profiling-related goals, including the identification of extreme performers and a bivariate classification of whether the hospital has higher-/lower-than-expected readmission and mortality rates, via a Bayesian decision-theoretic approach that characterizes hospitals on the basis of minimizing the posterior expected loss for an appropriate loss function. In some settings, particularly if the number of hospitals is large, the computational burden may be prohibitive. To resolve this, we propose a series of analysis strategies that will be useful in practice. Throughout the methods are illustrated with data from CMS on N=17,685 patients diagnosed with pancreatic cancer between 2000-2012 at one of J=264 hospitals in California.
△ Less
Submitted 18 May, 2021;
originally announced May 2021.
-
Nonparametric Bayesian Instrumental Variable Analysis: Evaluating Heterogeneous Effects of Coronary Arterial Access Site Strategies
Authors:
Samrachana Adhikari,
Sherri Rose,
Sharon-Lise Normand
Abstract:
Percutaneous coronary interventions (PCIs) are nonsurgical procedures to open blocked blood vessels to the heart, frequently using a catheter to place a stent. The catheter can be inserted into the blood vessels using an artery in the groin or an artery in the wrist. Because clinical trials have indicated that access via the wrist may result in fewer post procedure complications, shortening the le…
▽ More
Percutaneous coronary interventions (PCIs) are nonsurgical procedures to open blocked blood vessels to the heart, frequently using a catheter to place a stent. The catheter can be inserted into the blood vessels using an artery in the groin or an artery in the wrist. Because clinical trials have indicated that access via the wrist may result in fewer post procedure complications, shortening the length of stay, and ultimately cost less than groin access, adoption of access via the wrist has been encouraged. However, patients treated in usual care are likely to differ from those participating in clinical trials, and there is reason to believe that the effectiveness of wrist access may differ between males and females. Moreover, the choice of artery access strategy is likely to be influenced by patient or physician unmeasured factors. To study the effectiveness of the two artery access site strategies on hospitalization charges, we use data from a state-mandated clinical registry including 7,963 patients undergoing PCI. A hierarchical Bayesian likelihood-based instrumental variable analysis under a latent index modeling framework is introduced to jointly model outcomes and treatment status. Our approach accounts for unobserved heterogeneity via a latent factor structure, and permits nonparametric error distributions with Dirichlet process mixture models. Our results demonstrate that artery access in the wrist reduces hospitalization charges compared to access in the groin, with higher mean reduction for male patients.
△ Less
Submitted 3 November, 2019; v1 submitted 21 April, 2018;
originally announced April 2018.
-
Bayesian Meta-Analysis of Multiple Continuous Treatments: An Application to Antipsychotic Drugs
Authors:
Jacob Spertus,
Marcela Horvitz-Lennon,
Sharon-Lise Normand
Abstract:
Modeling dose-response relationships of drugs is essential to understanding their effect on patient outcomes under realistic circumstances. While intention-to-treat analyses of clinical trials provide the effect of assignment to a particular drug and dose, they do not capture observed exposure after factoring in non-adherence and dropout. We develop Bayesian methods to flexibly model dose-response…
▽ More
Modeling dose-response relationships of drugs is essential to understanding their effect on patient outcomes under realistic circumstances. While intention-to-treat analyses of clinical trials provide the effect of assignment to a particular drug and dose, they do not capture observed exposure after factoring in non-adherence and dropout. We develop Bayesian methods to flexibly model dose-response relationships of binary outcomes with continuous treatment, allowing for treatment effect heterogeneity and a non-linear response surface. We use a hierarchical framework for meta-analysis with the explicit goal of combining information from multiple trials while accounting for heterogeneity. In an application, we examine the risk of excessive weight gain for patients with schizophrenia treated with the second generation antipsychotics paliperidone, risperidone, or olanzapine in 14 clinical trials. Averaging over the sample population, we found that olanzapine contributed to a 15.6% (95% CrI: 6.7, 27.1) excess risk of weight gain at a 500mg cumulative dose. Paliperidone conferred a 3.2% (95% CrI: 1.5, 5.2) and risperidone a 14.9% (95% CrI: 0.0, 38.7) excess risk at 500mg olanzapine equivalent cumulative doses. Blacks had an additional 6.8% (95% CrI: 1.0, 12.4) risk of weight gain over non-blacks at 1000mg olanzapine equivalent cumulative doses of paliperidone.
△ Less
Submitted 14 February, 2018;
originally announced February 2018.
-
Regularization and Hierarchical Prior Distributions for Adjustment with Health Care Claims Data: Rethinking Comorbidity Scores
Authors:
Jacob Spertus,
Samrachana Adhikari,
Sharon-Lise Normand
Abstract:
Health care claims data refer to information generated from interactions within health systems. They have been used in health services research for decades to assess effectiveness of interventions, determine the quality of medical care, predict disease prognosis, and monitor population health. While claims data are relatively cheap and ubiquitous, they are high-dimensional, sparse, and noisy, typi…
▽ More
Health care claims data refer to information generated from interactions within health systems. They have been used in health services research for decades to assess effectiveness of interventions, determine the quality of medical care, predict disease prognosis, and monitor population health. While claims data are relatively cheap and ubiquitous, they are high-dimensional, sparse, and noisy, typically requiring dimension reduction. In health services research, the most common data reduction strategy involves use of a comorbidity index -- a single number summary reflecting overall patient health. We discuss Bayesian regularization strategies and a novel hierarchical prior distribution as better options for dimension reduction in claims data. The specifications are designed to work with a large number of codes while controlling variance by shrinking coefficients towards zero or towards a group-level mean. A comparison of drug-eluting to bare-metal coronary stents illustrates approaches. In our application, regularization and a hierarchical prior improved over comorbidity scores in terms of prediction and causal inference, as evidenced by out-of-sample fit and the ability to meet falsifiability endpoints.
△ Less
Submitted 14 November, 2017;
originally announced November 2017.
-
Bayesian Propensity Scores for High-Dimensional Causal Inference: A Comparison of Drug-Eluting to Bare-Metal Coronary Stents
Authors:
Jacob Spertus,
Sharon-Lise Normand
Abstract:
High-dimensional data can be useful for causal inference by providing many confounders that may bolster the plausibility of the ignorability assumption. Propensity score methods are powerful tools for causal inference, are popular in health care research, and are particularly useful for high-dimensional data. Recent interest has surrounded a Bayesian formulation of these methods in order to flexib…
▽ More
High-dimensional data can be useful for causal inference by providing many confounders that may bolster the plausibility of the ignorability assumption. Propensity score methods are powerful tools for causal inference, are popular in health care research, and are particularly useful for high-dimensional data. Recent interest has surrounded a Bayesian formulation of these methods in order to flexibly estimate propensity scores and summarize posterior quantities while incorporating variance from the (potentially high-dimensional) treatment model. We discuss methods for Bayesian propensity score analysis of binary treatments, focusing on modern methods for high-dimensional Bayesian regression and the propagation of uncertainty from the treatment regression. We introduce a novel and simple estimator for the average treatment effect that capitalizes on conjugancy of the beta and binomial distributions. Through simulations, we show the utility of horseshoe priors and Bayesian additive regression trees paired with our new estimator, while demonstrating the importance of including variance from the treatment and outcome models. Cardiac stent data with almost 500 confounders and 9000 patients illustrate approaches and compare among existing frequentist alternatives.
△ Less
Submitted 9 October, 2017;
originally announced October 2017.
-
Statistical and Clinical Aspects of Hospital Outcomes Profiling
Authors:
Sharon-Lise T. Normand,
David M. Shahian
Abstract:
Hospital profiling involves a comparison of a health care provider's structure, processes of care, or outcomes to a standard, often in the form of a report card. Given the ubiquity of report cards and similar consumer ratings in contemporary American culture, it is notable that these are a relatively recent phenomenon in health care. Prior to the 1986 release of Medicare hospital outcome data, l…
▽ More
Hospital profiling involves a comparison of a health care provider's structure, processes of care, or outcomes to a standard, often in the form of a report card. Given the ubiquity of report cards and similar consumer ratings in contemporary American culture, it is notable that these are a relatively recent phenomenon in health care. Prior to the 1986 release of Medicare hospital outcome data, little such information was publicly available. We review the historical evolution of hospital profiling with special emphasis on outcomes; present a detailed history of cardiac surgery report cards, the paradigm for modern provider profiling; discuss the potential unintended negative consequences of public report cards; and describe various statistical methodologies for quantifying the relative performance of cardiac surgery programs. Outstanding statistical issues are also described.
△ Less
Submitted 25 October, 2007;
originally announced October 2007.
-
Effect of physical inactivity on the oxidation of saturated and monounsaturated dietary Fatty acids: results of a randomized trial
Authors:
Audrey Bergouignan,
Dale A Schoeller,
Sylvie Normand,
Guillemette Gauquelin-Koch,
Martine Laville,
Timothy Shriver,
Michel Desage,
Yvon Le Maho,
Hiroshi Ohshima,
Claude Gharib,
Stéphane Blanc
Abstract:
OBJECTIVES: Changes in the way dietary fat is metabolized can be considered causative in obesity. The role of sedentary behavior in this defect has not been determined. We hypothesized that physical inactivity partitions dietary fats toward storage and that a resistance exercise training program mitigates storage.
OBJECTIVES: Changes in the way dietary fat is metabolized can be considered causative in obesity. The role of sedentary behavior in this defect has not been determined. We hypothesized that physical inactivity partitions dietary fats toward storage and that a resistance exercise training program mitigates storage.
△ Less
Submitted 28 June, 2007;
originally announced June 2007.
