Showing 1–7 of 7 results for author: Vallejos, C A

Ethical considerations of use of hold-out sets in clinical prediction model management

Authors: Louis Chislett, Louis JM Aslett, Alisha R Davies, Catalina A Vallejos, James Liley

Abstract: Clinical prediction models are statistical or machine learning models used to quantify the risk of a certain health outcome using patient data. These can then inform potential interventions on patients, causing an effect called performative prediction: predictions inform interventions which influence the outcome they were trying to predict, leading to a potential underestimation of risk in some pa… ▽ More Clinical prediction models are statistical or machine learning models used to quantify the risk of a certain health outcome using patient data. These can then inform potential interventions on patients, causing an effect called performative prediction: predictions inform interventions which influence the outcome they were trying to predict, leading to a potential underestimation of risk in some patients if a model is updated on this data. One suggested resolution to this is the use of hold-out sets, in which a set of patients do not receive model derived risk scores, such that a model can be safely retrained. We present an overview of clinical and research ethics regarding potential implementation of hold-out sets for clinical prediction models in health settings. We focus on the ethical principles of beneficence, non-maleficence, autonomy and justice. We also discuss informed consent, clinical equipoise, and truth-telling. We present illustrative cases of potential hold-out set implementations and discuss statistical issues arising from different hold-out set sampling methods. We also discuss differences between hold-out sets and randomised control trials, in terms of ethics and statistical issues. Finally, we give practical recommendations for researchers interested in the use hold-out sets for clinical prediction models. △ Less

Submitted 5 June, 2024; originally announced June 2024.

A review on competing risks methods for survival analysis

Authors: Karla Monterrubio-Gómez, Nathan Constantine-Cooke, Catalina A. Vallejos

Abstract: When modelling competing risks survival data, several techniques have been proposed in both the statistical and machine learning literature. State-of-the-art methods have extended classical approaches with more flexible assumptions that can improve predictive performance, allow high dimensional data and missing values, among others. Despite this, modern approaches have not been widely employed in… ▽ More When modelling competing risks survival data, several techniques have been proposed in both the statistical and machine learning literature. State-of-the-art methods have extended classical approaches with more flexible assumptions that can improve predictive performance, allow high dimensional data and missing values, among others. Despite this, modern approaches have not been widely employed in applied settings. This article aims to aid the uptake of such methods by providing a condensed compendium of competing risks survival methods with a unified notation and interpretation across approaches. We highlight available software and, when possible, demonstrate their usage via reproducible R vignettes. Moreover, we discuss two major concerns that can affect benchmark studies in this context: the choice of performance metrics and reproducibility. △ Less

Submitted 9 December, 2022; originally announced December 2022.

Comments: 22 pages, 2 tables

MSC Class: 62N02

Model updating after interventions paradoxically introduces bias

Authors: James Liley, Samuel R Emerson, Bilal A Mateen, Catalina A Vallejos, Louis J M Aslett, Sebastian J Vollmer

Abstract: Machine learning is increasingly being used to generate prediction models for use in a number of real-world settings, from credit risk assessment to clinical decision support. Recent discussions have highlighted potential problems in the updating of a predictive score for a binary outcome when an existing predictive score forms part of the standard workflow, driving interventions. In this setting,… ▽ More Machine learning is increasingly being used to generate prediction models for use in a number of real-world settings, from credit risk assessment to clinical decision support. Recent discussions have highlighted potential problems in the updating of a predictive score for a binary outcome when an existing predictive score forms part of the standard workflow, driving interventions. In this setting, the existing score induces an additional causative pathway which leads to miscalibration when the original score is replaced. We propose a general causal framework to describe and address this problem, and demonstrate an equivalent formulation as a partially observed Markov decision process. We use this model to demonstrate the impact of such `naive updating' when performed repeatedly. Namely, we show that successive predictive scores may converge to a point where they predict their own effect, or may eventually tend toward a stable oscillation between two values, and we argue that neither outcome is desirable. Furthermore, we demonstrate that even if model-fitting procedures improve, actual performance may worsen. We complement these findings with a discussion of several potential routes to overcome these issues. △ Less

Submitted 22 February, 2021; v1 submitted 22 October, 2020; originally announced October 2020.

Comments: Sections of this preprint on 'Successive adjuvancy' (section 4, theorem 2, figures 4,5, and associated discussions) were not included in the originally submitted version of this paper due to length. This material does not appear in the published version of this manuscript, and the reader should be aware that these sections did not undergo peer review

Design choices for productive, secure, data-intensive research at scale in the cloud

Authors: Diego Arenas, Jon Atkins, Claire Austin, David Beavan, Alvaro Cabrejas Egea, Steven Carlysle-Davies, Ian Carter, Rob Clarke, James Cunningham, Tom Doel, Oliver Forrest, Evelina Gabasova, James Geddes, James Hetherington, Radka Jersakova, Franz Kiraly, Catherine Lawrence, Jules Manser, Martin T. O'Reilly, James Robinson, Helen Sherwood-Taylor, Serena Tierney, Catalina A. Vallejos, Sebastian Vollmer, Kirstie Whitaker

Abstract: We present a policy and process framework for secure environments for productive data science research projects at scale, by combining prevailing data security threat and risk profiles into five sensitivity tiers, and, at each tier, specifying recommended policies for data classification, data ingress, software ingress, data egress, user access, user device control, and analysis environments. By p… ▽ More We present a policy and process framework for secure environments for productive data science research projects at scale, by combining prevailing data security threat and risk profiles into five sensitivity tiers, and, at each tier, specifying recommended policies for data classification, data ingress, software ingress, data egress, user access, user device control, and analysis environments. By presenting design patterns for security choices for each tier, and using software defined infrastructure so that a different, independent, secure research environment can be instantiated for each project appropriate to its classification, we hope to maximise researcher productivity and minimise risk, allowing research organisations to operate with confidence. △ Less

Submitted 15 September, 2019; v1 submitted 23 August, 2019; originally announced August 2019.

Shrinkage estimation of large covariance matrices using multiple shrinkage targets

Authors: Harry Gray, Gwenaël G. R. Leday, Catalina A. Vallejos, Sylvia Richardson

Abstract: Linear shrinkage estimators of a covariance matrix --- defined by a weighted average of the sample covariance matrix and a pre-specified shrinkage target matrix --- are popular when analysing high-throughput molecular data. However, their performance strongly relies on an appropriate choice of target matrix. This paper introduces a more flexible class of linear shrinkage estimators that can accomm… ▽ More Linear shrinkage estimators of a covariance matrix --- defined by a weighted average of the sample covariance matrix and a pre-specified shrinkage target matrix --- are popular when analysing high-throughput molecular data. However, their performance strongly relies on an appropriate choice of target matrix. This paper introduces a more flexible class of linear shrinkage estimators that can accommodate multiple shrinkage target matrices, directly accounting for the uncertainty regarding the target choice. This is done within a conjugate Bayesian framework, which is computationally efficient. Using both simulated and real data, we show that the proposed estimator is less sensitive to target misspecification and can outperform state-of-the-art (nonparametric) single-target linear shrinkage estimators. Using protein expression data from The Cancer Proteome Atlas we illustrate how multiple sources of prior information (obtained from more than 30 different cancer types) can be incorporated into the proposed multi-target linear shrinkage estimator. In particular, it is shown that the target-specific weights can provide insights into the differences and similarities between cancer types. Software for the method is freely available as an R-package at http://github.com/HGray384/TAS. △ Less

Submitted 21 September, 2018; originally announced September 2018.

Bayesian Survival Modelling of University Outcomes

Authors: Catalina A. Vallejos, Mark F. J. Steel

Abstract: The aim of this paper is to model the length of registration at university and its associated academic outcome for undergraduate students at the Pontificia Universidad Católica de Chile. Survival time is defined as the time until the end of the enrollment period, which can relate to different reasons - graduation or two types of dropout - that are driven by different processes. Hence, a competing… ▽ More The aim of this paper is to model the length of registration at university and its associated academic outcome for undergraduate students at the Pontificia Universidad Católica de Chile. Survival time is defined as the time until the end of the enrollment period, which can relate to different reasons - graduation or two types of dropout - that are driven by different processes. Hence, a competing risks model is employed for the analysis. The issue of separation of the outcomes (which precludes maximum likelihood estimation) is handled through the use of Bayesian inference with an appropriately chosen prior. We are interested in identifying important determinants of university outcomes and the associ- ated model uncertainty is formally addressed through Bayesian model averaging. The methodology introduced for modelling university outcomes is applied to three selected degree programmes, which are particularly affected by dropout and late graduation. △ Less

Submitted 25 June, 2014; originally announced June 2014.

On posterior propriety for the Student-$t$ linear regression model under Jeffreys priors

Authors: Catalina A. Vallejos, Mark F. J. Steel

Abstract: Regression models with fat-tailed error terms are an increasingly popular choice to obtain more robust inference to the presence of outlying observations. This article focuses on Bayesian inference for the Student-$t$ linear regression model under objective priors that are based on the Jeffreys rule. Posterior propriety results presented in Fonseca et al. (2008) are revisited and corrected. In par… ▽ More Regression models with fat-tailed error terms are an increasingly popular choice to obtain more robust inference to the presence of outlying observations. This article focuses on Bayesian inference for the Student-$t$ linear regression model under objective priors that are based on the Jeffreys rule. Posterior propriety results presented in Fonseca et al. (2008) are revisited and corrected. In particular, it is shown that the standard Jeffreys-rule prior precludes the existence of a proper posterior distribution. △ Less

Submitted 8 November, 2013; v1 submitted 6 November, 2013; originally announced November 2013.

Comments: minor editorial changes in this version