Showing 1–8 of 8 results for author: Benz, L

Comparing Causal Inference Methods for Point Exposures with Missing Confounders: A Simulation Study

Authors: Luke Benz, Alexander Levis, Sebastien Haneuse

Abstract: Causal inference methods based on electronic health record (EHR) databases must simultaneously handle confounding and missing data. Vast scholarship exists aimed at addressing these two issues separately, but surprisingly few papers attempt to address them simultaneously. In practice, when faced with simultaneous missing data and confounding, analysts may proceed by first imputing missing data and… ▽ More Causal inference methods based on electronic health record (EHR) databases must simultaneously handle confounding and missing data. Vast scholarship exists aimed at addressing these two issues separately, but surprisingly few papers attempt to address them simultaneously. In practice, when faced with simultaneous missing data and confounding, analysts may proceed by first imputing missing data and subsequently using outcome regression or inverse-probability weighting (IPW) to address confounding. However, little is known about the theoretical performance of such $\textit{ad hoc}$ methods. In a recent paper Levis $\textit{et al.}$ outline a robust framework for tackling these problems together under certain identifying conditions, and introduce a pair of estimators for the average treatment effect (ATE), one of which is non-parametric efficient. In this work we present a series of simulations, motivated by a published EHR based study of the long-term effects of bariatric surgery on weight outcomes, to investigate these new estimators and compare them to existing $\textit{ad hoc}$ methods. While the latter perform well in certain scenarios, no single estimator is uniformly best. As such, the work of Levis $\textit{et al.}$ may serve as a reasonable default for causal inference when handling confounding and missing data together. △ Less

Submitted 8 July, 2024; originally announced July 2024.

Adjusting for Selection Bias Due to Missing Eligibility Criteria in Emulated Target Trials

Authors: Luke Benz, Rajarshi Mukherjee, Issa Dahabreh, Rui Wang, David Arterburn, Catherine Lee, Heidi Fischer, Susan Shortreed, Sebastien Haneuse

Abstract: Target trial emulation (TTE) is a popular framework for observational studies based on electronic health records (EHR). A key component of this framework is determining the patient population eligible for inclusion in both a target trial of interest and its observational emulation. Missingness in variables that define eligibility criteria, however, presents a major challenge towards determining th… ▽ More Target trial emulation (TTE) is a popular framework for observational studies based on electronic health records (EHR). A key component of this framework is determining the patient population eligible for inclusion in both a target trial of interest and its observational emulation. Missingness in variables that define eligibility criteria, however, presents a major challenge towards determining the eligible population when emulating a target trial with an observational study. In practice, patients with incomplete data are almost always excluded from analysis despite the possibility of selection bias, which can arise when subjects with observed eligibility data are fundamentally different than excluded subjects. Despite this, to the best of our knowledge, very little work has been done to mitigate this concern. In this paper, we propose a novel conceptual framework to address selection bias in TTE studies, tailored towards time-to-event endpoints, and describe estimation and inferential procedures via inverse probability weighting (IPW). Under an EHR-based simulation infrastructure, developed to reflect the complexity of EHR data, we characterize common settings under which missing eligibility data poses the threat of selection bias and investigate the ability of the proposed methods to address it. Finally, using EHR databases from Kaiser Permanente, we demonstrate the use of our method to evaluate the effect of bariatric surgery on microvascular outcomes among a cohort of severely obese patients with Type II diabetes mellitus (T2DM). △ Less

Submitted 24 June, 2024; originally announced June 2024.

A comprehensive survey of the home advantage in American football

Authors: Luke S. Benz, Thompson J. Bliss, Michael J. Lopez

Abstract: The existence and justification to the home advantage -- the benefit a sports team receives when playing at home -- has been studied across sport. The majority of research on this topic is limited to individual leagues in short time frames, which hinders extrapolation and a deeper understanding of possible causes. Using nearly two decades of data from the National Football League (NFL), the Nation… ▽ More The existence and justification to the home advantage -- the benefit a sports team receives when playing at home -- has been studied across sport. The majority of research on this topic is limited to individual leagues in short time frames, which hinders extrapolation and a deeper understanding of possible causes. Using nearly two decades of data from the National Football League (NFL), the National Collegiate Athletic Association (NCAA), and high schools from across the United States, we provide a uniform approach to understanding the home advantage in American football. Our findings suggest home advantage is declining in the NFL and the highest levels of collegiate football, but not in amateur football. This increases the possibility that characteristics of the NCAA and NFL, such as travel improvements and instant replay, have helped level the playing field. △ Less

Submitted 27 June, 2024; v1 submitted 29 January, 2024; originally announced January 2024.

Human-Aligned Calibration for AI-Assisted Decision Making

Authors: Nina L. Corvelo Benz, Manuel Gomez Rodriguez

Abstract: Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicte… ▽ More Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of empirical evidence suggest that decision makers have difficulties at develo** a good sense on when to trust a prediction using these confidence values. In this paper, our goal is first to understand why and then investigate how to construct more useful confidence values. We first argue that, for a broad class of utility functions, there exist data distributions for which a rational decision maker is, in general, unlikely to discover the optimal decision policy using the above confidence values -- an optimal decision maker would need to sometimes place more (less) trust on predictions with lower (higher) confidence values. However, we then show that, if the confidence values satisfy a natural alignment property with respect to the decision maker's confidence on her own predictions, there always exists an optimal decision policy under which the level of trust the decision maker would need to place on predictions is monotone on the confidence values, facilitating its discoverability. Further, we show that multicalibration with respect to the decision maker's confidence on her own predictions is a sufficient condition for alignment. Experiments on four different AI-assisted decision making tasks where a classifier provides decision support to real human experts validate our theoretical results and suggest that alignment may lead to better decisions. △ Less

Submitted 23 February, 2024; v1 submitted 31 May, 2023; originally announced June 2023.

Counterfactual Inference of Second Opinions

Authors: Nina L. Corvelo Benz, Manuel Gomez Rodriguez

Abstract: Automated decision support systems that are able to infer second opinions from experts can potentially facilitate a more efficient allocation of resources; they can help decide when and from whom to seek a second opinion. In this paper, we look at the design of this type of support systems from the perspective of counterfactual inference. We focus on a multiclass classification setting and first s… ▽ More Automated decision support systems that are able to infer second opinions from experts can potentially facilitate a more efficient allocation of resources; they can help decide when and from whom to seek a second opinion. In this paper, we look at the design of this type of support systems from the perspective of counterfactual inference. We focus on a multiclass classification setting and first show that, if experts make predictions on their own, the underlying causal mechanism generating their predictions needs to satisfy a desirable set invariant property. Further, we show that, for any causal mechanism satisfying this property, there exists an equivalent mechanism where the predictions by each expert are generated by independent sub-mechanisms governed by a common noise. This motivates the design of a set invariant Gumbel-Max structural causal model where the structure of the noise governing the sub-mechanisms underpinning the model depends on an intuitive notion of similarity between experts which can be estimated from data. Experiments on both synthetic and real data show that our model can be used to infer second opinions more accurately than its non-causal counterpart. △ Less

Submitted 30 June, 2022; v1 submitted 16 March, 2022; originally announced March 2022.

Generalized Turán densities in the hypercube

Authors: Maria Axenovich, Laurin Benz, David Offner, Casey Tompkins

Abstract: A classical extremal, or Turán-type problem asks to determine ${\rm ex}(G, H)$, the largest number of edges in a subgraph of a graph $G$ which does not contain a subgraph isomorphic to $H$. Alon and Shikhelman introduced the so-called generalized extremal number ${\rm ex}(G,T,H)$, defined to be the maximum number of subgraphs isomorphic to $T$ in a subgraph of $G$ that contains no subgraphs isomor… ▽ More A classical extremal, or Turán-type problem asks to determine ${\rm ex}(G, H)$, the largest number of edges in a subgraph of a graph $G$ which does not contain a subgraph isomorphic to $H$. Alon and Shikhelman introduced the so-called generalized extremal number ${\rm ex}(G,T,H)$, defined to be the maximum number of subgraphs isomorphic to $T$ in a subgraph of $G$ that contains no subgraphs isomorphic to $H$. In this paper we investigate the case when $G = Q_n$, the hypercube of dimension $n$, and $T$ and $H$ are smaller hypercubes or cycles. △ Less

Submitted 22 January, 2022; v1 submitted 12 January, 2022; originally announced January 2022.

A Machine Learning Approach for Recruitment Prediction in Clinical Trial Design

Authors: **gshu Liu, Patricia J Allen, Luke Benz, Daniel Blickstein, Evon Okidi, Xiao Shi

Abstract: Significant advancements have been made in recent years to optimize patient recruitment for clinical trials, however, improved methods for patient recruitment prediction are needed to support trial site selection and to estimate appropriate enrollment timelines in the trial design stage. In this paper, using data from thousands of historical clinical trials, we explore machine learning methods to… ▽ More Significant advancements have been made in recent years to optimize patient recruitment for clinical trials, however, improved methods for patient recruitment prediction are needed to support trial site selection and to estimate appropriate enrollment timelines in the trial design stage. In this paper, using data from thousands of historical clinical trials, we explore machine learning methods to predict the number of patients enrolled per month at a clinical trial site over the course of a trial's enrollment duration. We show that these methods can reduce the error that is observed with current industry standards and propose opportunities for further improvement. △ Less

Submitted 14 November, 2021; originally announced November 2021.

Comments: Machine Learning for Health (ML4H) - Extended Abstract

Estimating the change in soccer's home advantage during the Covid-19 pandemic using bivariate Poisson regression

Authors: Luke S. Benz, Michael J. Lopez

Abstract: In wake of the Covid-19 pandemic, 2019-2020 soccer seasons across the world were postponed and eventually made up during the summer months of 2020. Researchers from a variety of disciplines jumped at the opportunity to compare the rescheduled games, played in front of empty stadia, to previous games, played in front of fans. To date, most of this post-Covid soccer research has used linear regressi… ▽ More In wake of the Covid-19 pandemic, 2019-2020 soccer seasons across the world were postponed and eventually made up during the summer months of 2020. Researchers from a variety of disciplines jumped at the opportunity to compare the rescheduled games, played in front of empty stadia, to previous games, played in front of fans. To date, most of this post-Covid soccer research has used linear regression models, or versions thereof, to estimate potential changes to the home advantage. But because soccer outcomes are non-linear, we argue that leveraging the Poisson distribution would be more appropriate. We begin by using simulations to show that bivariate Poisson regression reduces absolute bias when estimating the home advantage benefit in a single season of soccer games, relative to linear regression, by almost 85 percent. Next, with data from 17 professional soccer leagues, we extend bivariate Poisson models estimate the change in home advantage due to games being played without fans. In contrast to current research that overwhelmingly suggests a drop in the home advantage, our findings are mixed; in some leagues, evidence points to a decrease, while in others, the home advantage may have risen. Altogether, this suggests a more complex causal mechanism for the impact of fans on sporting events. △ Less

Submitted 28 May, 2021; v1 submitted 29 December, 2020; originally announced December 2020.