-
Deployment of a Robust and Explainable Mortality Prediction Model: The COVID-19 Pandemic and Beyond
Authors:
Jacob R. Epifano,
Stephen Glass,
Ravi P. Ramachandran,
Sharad Patel,
Aaron J. Masino,
Ghulam Rasool
Abstract:
This study investigated the performance, explainability, and robustness of deployed artificial intelligence (AI) models in predicting mortality during the COVID-19 pandemic and beyond. The first study of its kind, we found that Bayesian Neural Networks (BNNs) and intelligent training techniques allowed our models to maintain performance amidst significant data shifts. Our results emphasize the imp…
▽ More
This study investigated the performance, explainability, and robustness of deployed artificial intelligence (AI) models in predicting mortality during the COVID-19 pandemic and beyond. The first study of its kind, we found that Bayesian Neural Networks (BNNs) and intelligent training techniques allowed our models to maintain performance amidst significant data shifts. Our results emphasize the importance of develo** robust AI models capable of matching or surpassing clinician predictions, even under challenging conditions. Our exploration of model explainability revealed that stochastic models generate more diverse and personalized explanations thereby highlighting the need for AI models that provide detailed and individualized insights in real-world clinical settings. Furthermore, we underscored the importance of quantifying uncertainty in AI models which enables clinicians to make better-informed decisions based on reliable predictions. Our study advocates for prioritizing implementation science in AI research for healthcare and ensuring that AI solutions are practical, beneficial, and sustainable in real-world clinical environments. By addressing unique challenges and complexities in healthcare settings, researchers can develop AI models that effectively improve clinical practice and patient outcomes.
△ Less
Submitted 28 November, 2023;
originally announced November 2023.
-
Revisiting the Fragility of Influence Functions
Authors:
Jacob R. Epifano,
Ravi P. Ramachandran,
Aaron J. Masino,
Ghulam Rasool
Abstract:
In the last few years, many works have tried to explain the predictions of deep learning models. Few methods, however, have been proposed to verify the accuracy or faithfulness of these explanations. Recently, influence functions, which is a method that approximates the effect that leave-one-out training has on the loss function, has been shown to be fragile. The proposed reason for their fragilit…
▽ More
In the last few years, many works have tried to explain the predictions of deep learning models. Few methods, however, have been proposed to verify the accuracy or faithfulness of these explanations. Recently, influence functions, which is a method that approximates the effect that leave-one-out training has on the loss function, has been shown to be fragile. The proposed reason for their fragility remains unclear. Although previous work suggests the use of regularization to increase robustness, this does not hold in all cases. In this work, we seek to investigate the experiments performed in the prior work in an effort to understand the underlying mechanisms of influence function fragility. First, we verify influence functions using procedures from the literature under conditions where the convexity assumptions of influence functions are met. Then, we relax these assumptions and study the effects of non-convexity by using deeper models and more complex datasets. Here, we analyze the key metrics and procedures that are used to validate influence functions. Our results indicate that the validation procedures may cause the observed fragility.
△ Less
Submitted 7 April, 2023; v1 submitted 22 March, 2023;
originally announced March 2023.
-
Opening Access to Visual Exploration of Audiovisual Digital Biomarkers: an OpenDBM Analytics Tool
Authors:
Carla Floricel,
Jacob Epifano,
Stephanie Caamano,
Sarah Kark,
Rich Christie,
Aaron Masino,
Andre D Paredes
Abstract:
Digital biomarkers (DBMs) are a growing field and increasingly tested in the therapeutic areas of psychiatric and neurodegenerative disorders. Meanwhile, isolated silos of knowledge of audiovisual DBMs use in industry, academia, and clinics hinder their widespread adoption in clinical research. How can we help these non-technical domain experts to explore audiovisual digital biomarkers? The use of…
▽ More
Digital biomarkers (DBMs) are a growing field and increasingly tested in the therapeutic areas of psychiatric and neurodegenerative disorders. Meanwhile, isolated silos of knowledge of audiovisual DBMs use in industry, academia, and clinics hinder their widespread adoption in clinical research. How can we help these non-technical domain experts to explore audiovisual digital biomarkers? The use of open source software in biomedical research to extract patient behavior changes is growing and inspiring a shift toward accessibility to address this problem. OpenDBM integrates several popular audio and visual open source behavior extraction toolkits. We present a visual analysis tool as an extension of the growing open source software, OpenDBM, to promote the adoption of audiovisual DBMs in basic and applied research. Our tool illustrates patterns in behavioral data while supporting interactive visual analysis of any subset of derived or raw DBM variables extracted through OpenDBM.
△ Less
Submitted 4 October, 2022;
originally announced October 2022.
-
Unsupervised learning with GLRM feature selection reveals novel traumatic brain injury phenotypes
Authors:
Aaron J. Masino,
Kaitlin A. Folweiler
Abstract:
Baseline injury categorization is important to traumatic brain injury (TBI) research and treatment. Current categorization is dominated by symptom-based scores that insufficiently capture injury heterogeneity. In this work, we apply unsupervised clustering to identify novel TBI phenotypes. Our approach uses a generalized low-rank model (GLRM) model for feature selection in a procedure analogous to…
▽ More
Baseline injury categorization is important to traumatic brain injury (TBI) research and treatment. Current categorization is dominated by symptom-based scores that insufficiently capture injury heterogeneity. In this work, we apply unsupervised clustering to identify novel TBI phenotypes. Our approach uses a generalized low-rank model (GLRM) model for feature selection in a procedure analogous to wrapper methods. The resulting clusters reveal four novel TBI phenotypes with distinct feature profiles and that correlate to 90-day functional and cognitive status.
△ Less
Submitted 30 November, 2018;
originally announced December 2018.
