-
Generative Large Language Models are autonomous practitioners of evidence-based medicine
Authors:
Akhil Vaid,
Joshua Lampert,
Juhee Lee,
Ashwin Sawant,
Donald Apakama,
Ankit Sakhuja,
Ali Soroush,
Denise Lee,
Isotta Landi,
Nicole Bussola,
Ismail Nabeel,
Robbie Freeman,
Patricia Kovatch,
Brendan Carr,
Benjamin Glicksberg,
Edgar Argulian,
Stamatios Lerakis,
Monica Kraft,
Alexander Charney,
Girish Nadkarni
Abstract:
Background: Evidence-based medicine (EBM) is fundamental to modern clinical practice, requiring clinicians to continually update their knowledge and apply the best clinical evidence in patient care. The practice of EBM faces challenges due to rapid advancements in medical research, leading to information overload for clinicians. The integration of artificial intelligence (AI), specifically Generat…
▽ More
Background: Evidence-based medicine (EBM) is fundamental to modern clinical practice, requiring clinicians to continually update their knowledge and apply the best clinical evidence in patient care. The practice of EBM faces challenges due to rapid advancements in medical research, leading to information overload for clinicians. The integration of artificial intelligence (AI), specifically Generative Large Language Models (LLMs), offers a promising solution towards managing this complexity.
Methods: This study involved the curation of real-world clinical cases across various specialties, converting them into .json files for analysis. LLMs, including proprietary models like ChatGPT 3.5 and 4, Gemini Pro, and open-source models like LLaMA v2 and Mixtral-8x7B, were employed. These models were equipped with tools to retrieve information from case files and make clinical decisions similar to how clinicians must operate in the real world. Model performance was evaluated based on correctness of final answer, judicious use of tools, conformity to guidelines, and resistance to hallucinations.
Results: GPT-4 was most capable of autonomous operation in a clinical setting, being generally more effective in ordering relevant investigations and conforming to clinical guidelines. Limitations were observed in terms of model ability to handle complex guidelines and diagnostic nuances. Retrieval Augmented Generation made recommendations more tailored to patients and healthcare systems.
Conclusions: LLMs can be made to function as autonomous practitioners of evidence-based medicine. Their ability to utilize tooling can be harnessed to interact with the infrastructure of a real-world healthcare system and perform the tasks of patient management in a guideline directed manner. Prompt engineering may help to further enhance this potential and transform healthcare for the clinician and the patient.
△ Less
Submitted 5 January, 2024;
originally announced January 2024.
-
HeartBEiT: Vision Transformer for Electrocardiogram Data Improves Diagnostic Performance at Low Sample Sizes
Authors:
Akhil Vaid,
Joy Jiang,
Ashwin Sawant,
Stamatios Lerakis,
Edgar Argulian,
Yuri Ahuja,
Joshua Lampert,
Alexander Charney,
Hayit Greenspan,
Benjamin Glicksberg,
Jagat Narula,
Girish Nadkarni
Abstract:
The electrocardiogram (ECG) is a ubiquitous diagnostic modality. Convolutional neural networks (CNNs) applied towards ECG analysis require large sample sizes, and transfer learning approaches result in suboptimal performance when pre-training is done on natural images. We leveraged masked image modeling to create the first vision-based transformer model, HeartBEiT, for electrocardiogram waveform a…
▽ More
The electrocardiogram (ECG) is a ubiquitous diagnostic modality. Convolutional neural networks (CNNs) applied towards ECG analysis require large sample sizes, and transfer learning approaches result in suboptimal performance when pre-training is done on natural images. We leveraged masked image modeling to create the first vision-based transformer model, HeartBEiT, for electrocardiogram waveform analysis. We pre-trained this model on 8.5 million ECGs and then compared performance vs. standard CNN architectures for diagnosis of hypertrophic cardiomyopathy, low left ventricular ejection fraction and ST elevation myocardial infarction using differing training sample sizes and independent validation datasets. We show that HeartBEiT has significantly higher performance at lower sample sizes compared to other models. Finally, we also show that HeartBEiT improves explainability of diagnosis by highlighting biologically relevant regions of the EKG vs. standard CNNs. Thus, we present the first vision-based waveform transformer that can be used to develop specialized models for ECG analysis especially at low sample sizes.
△ Less
Submitted 13 December, 2022;
originally announced December 2022.
-
DoSA : A System to Accelerate Annotations on Business Documents with Human-in-the-Loop
Authors:
Neelesh K Shukla,
Msp Raja,
Raghu Katikeri,
Amit Vaid
Abstract:
Business documents come in a variety of structures, formats and information needs which makes information extraction a challenging task. Due to these variations, having a document generic model which can work well across all types of documents and for all the use cases seems far-fetched. For document-specific models, we would need customized document-specific labels. We introduce DoSA (Document Sp…
▽ More
Business documents come in a variety of structures, formats and information needs which makes information extraction a challenging task. Due to these variations, having a document generic model which can work well across all types of documents and for all the use cases seems far-fetched. For document-specific models, we would need customized document-specific labels. We introduce DoSA (Document Specific Automated Annotations), which helps annotators in generating initial annotations automatically using our novel bootstrap approach by leveraging document generic datasets and models. These initial annotations can further be reviewed by a human for correctness. An initial document-specific model can be trained and its inference can be used as feedback for generating more automated annotations. These automated annotations can be reviewed by human-in-the-loop for the correctness and a new improved model can be trained using the current model as pre-trained model before going for the next iteration. In this paper, our scope is limited to Form like documents due to limited availability of generic annotated datasets, but this idea can be extended to a variety of other documents as more datasets are built. An open-source ready-to-use implementation is made available on GitHub https://github.com/neeleshkshukla/DoSA.
△ Less
Submitted 9 November, 2022;
originally announced November 2022.
-
Pinning of extended dislocations in atomically disordered crystals
Authors:
Aviral Vaid,
De'an Wei,
Erik Bitzek,
Samaneh Nasiri,
Michael Zaiser
Abstract:
In recent years there has been renewed interest in the behavior of dislocations in crystals that exhibit strong atomic scale disorder, as typical of compositionally complex single phase alloys. The behavior of dislocations in such crystals has been often studied in the framework of elastic manifold pinning in disordered systems. Here we discuss modifications of this framework that may need to be a…
▽ More
In recent years there has been renewed interest in the behavior of dislocations in crystals that exhibit strong atomic scale disorder, as typical of compositionally complex single phase alloys. The behavior of dislocations in such crystals has been often studied in the framework of elastic manifold pinning in disordered systems. Here we discuss modifications of this framework that may need to be adapted when dealing with extended dislocations that split into widely separated partials. We demonstrate that the presence of a stacking fault gives rise to an additional stress scale that needs to be compared with the pinning stress of elastic manifold theory to decide whether the partials are pinned individually or the dislocation is pinned as a whole. For the case of weakly interacting partial dislocations, we demonstrate the existence of multiple metastable states at stresses below the depinning threshold and analyze the stress evolution of the stacking fault width during loading. In addition we investigate how geometrical constraints can modulate the dislocation-solute interaction and enhance the pinning stress. We compare our theoretical arguments with results of atomistic and discrete (partial) dislocation dynamics (D(P)DD) simulations.
△ Less
Submitted 24 October, 2021;
originally announced October 2021.
-
Contrastive Learning Improves Critical Event Prediction in COVID-19 Patients
Authors:
Tingyi Wanyan,
Hossein Honarvar,
Suraj K. Jaladanki,
Chengxi Zang,
Nidhi Naik,
Sulaiman Somani,
Jessica K. De Freitas,
Ishan Paranjpe,
Akhil Vaid,
Riccardo Miotto,
Girish N. Nadkarni,
Marinka Zitnik,
ArifulAzad,
Fei Wang,
Ying Ding,
Benjamin S. Glicksberg
Abstract:
Machine Learning (ML) models typically require large-scale, balanced training data to be robust, generalizable, and effective in the context of healthcare. This has been a major issue for develo** ML models for the coronavirus-disease 2019 (COVID-19) pandemic where data is highly imbalanced, particularly within electronic health records (EHR) research. Conventional approaches in ML use cross-ent…
▽ More
Machine Learning (ML) models typically require large-scale, balanced training data to be robust, generalizable, and effective in the context of healthcare. This has been a major issue for develo** ML models for the coronavirus-disease 2019 (COVID-19) pandemic where data is highly imbalanced, particularly within electronic health records (EHR) research. Conventional approaches in ML use cross-entropy loss (CEL) that often suffers from poor margin classification. For the first time, we show that contrastive loss (CL) improves the performance of CEL especially for imbalanced EHR data and the related COVID-19 analyses. This study has been approved by the Institutional Review Board at the Icahn School of Medicine at Mount Sinai. We use EHR data from five hospitals within the Mount Sinai Health System (MSHS) to predict mortality, intubation, and intensive care unit (ICU) transfer in hospitalized COVID-19 patients over 24 and 48 hour time windows. We train two sequential architectures (RNN and RETAIN) using two loss functions (CEL and CL). Models are tested on full sample data set which contain all available data and restricted data set to emulate higher class imbalance.CL models consistently outperform CEL models with the restricted data set on these tasks with differences ranging from 0.04 to 0.15 for AUPRC and 0.05 to 0.1 for AUROC. For the restricted sample, only the CL model maintains proper clustering and is able to identify important features, such as pulse oximetry. CL outperforms CEL in instances of severe class imbalance, on three EHR outcomes with respect to three performance metrics: predictive power, clustering, and feature importance. We believe that the developed CL framework can be expanded and used for EHR ML work in general.
△ Less
Submitted 11 January, 2021;
originally announced January 2021.
-
Assessment and optimization of the fast inertial relaxation engine (FIRE) for energy minimization in atomistic simulations and its implementation in LAMMPS
Authors:
Julien Guénolé,
Wolfram G. Nöhring,
Aviral Vaid,
Frédéric Houllé,
Zhuocheng Xie,
Aruna Prakash,
Erik Bitzek
Abstract:
In atomistic simulations, pseudo-dynamics relaxation schemes often exhibit better performance and accuracy in finding local minima than line-search-based descent algorithms like steepest descent or conjugate gradient. Here, an improved version of the fast inertial relaxation engine (FIRE) and its implementation within the open-source code LAMMPS is presented. It is shown that the correct choice of…
▽ More
In atomistic simulations, pseudo-dynamics relaxation schemes often exhibit better performance and accuracy in finding local minima than line-search-based descent algorithms like steepest descent or conjugate gradient. Here, an improved version of the fast inertial relaxation engine (FIRE) and its implementation within the open-source code LAMMPS is presented. It is shown that the correct choice of time integration scheme and minimization parameters is crucial for performance.
△ Less
Submitted 30 January, 2020; v1 submitted 6 August, 2019;
originally announced August 2019.
-
Atomistic Simulations of Basal Dislocations Interacting with Mg$_{17}$Al$_{12}$ Precipitates in Mg
Authors:
Aviral Vaid,
Julien Guénolé,
Aruna Prakash,
Sandra Korte-Kerzel,
Erik Bitzek
Abstract:
The mechanical properties of Mg-Al alloys are greatly influenced by the complex intermetallic phase Mg$_{17}$Al$_{12}$, which is the most dominant precipitate found in this alloy system. The interaction of basal edge and 30$^\text{o}$ dislocations with Mg$_{17}$Al$_{12}$ precipitates is studied by molecular dynamics and statics simulations, varying the inter-precipitate spacing ($L$), and size (…
▽ More
The mechanical properties of Mg-Al alloys are greatly influenced by the complex intermetallic phase Mg$_{17}$Al$_{12}$, which is the most dominant precipitate found in this alloy system. The interaction of basal edge and 30$^\text{o}$ dislocations with Mg$_{17}$Al$_{12}$ precipitates is studied by molecular dynamics and statics simulations, varying the inter-precipitate spacing ($L$), and size ($D$), shape and orientation of the precipitates. The critical resolved shear stress $τ_c$ to pass an array of precipitates follows the usual $\ln((1/D + 1/L)^{-1})$ proportionality. In all cases but the smallest precipitate, the dislocations pass the obstacles by depositing dislocation segments in the disordered interphase boundary rather than shearing the precipitate or leaving Orowan loops in the matrix around the precipitate. An absorbed dislocation increases the stress necessary for a second dislocation to pass the precipitate also by absorbing dislocation segments into the boundary. Replacing the precipitate with a void of identical size and shape decreases the critical passing stress and work hardening contribution while an artificially impenetrable Mg$_{17}$Al$_{12}$ precipitate increases both. These insights will help improve mesoscale models of hardening by incoherent particles.
△ Less
Submitted 20 May, 2019; v1 submitted 25 February, 2019;
originally announced February 2019.
-
Graphene quantum dots probed by scanning tunneling spectroscopy and transport spectroscopy after local anodic oxidation
Authors:
Markus Morgenstern,
Nils Freitag,
Aviral Vaid,
Marco Pratzer,
Marcus Liebmann
Abstract:
Graphene quantum dots are considered as promising alternatives to quantum dots in III-V semiconductors, e.g., for the use as spin qubits due to their consistency made of light atoms including spin-free nuclei which both imply relatively long spin decoherene times. However, this potential has not been realized in experiments so far, most likely, due to a missing control of the edge configurations o…
▽ More
Graphene quantum dots are considered as promising alternatives to quantum dots in III-V semiconductors, e.g., for the use as spin qubits due to their consistency made of light atoms including spin-free nuclei which both imply relatively long spin decoherene times. However, this potential has not been realized in experiments so far, most likely, due to a missing control of the edge configurations of the quantum dots. Thus, a more fundamental investigation of Graphene quantum dots appears to be necessary including a full control of the wave function properties most favorably during transport spectroscopy measurements. Here, we review the recent success in map** wave functions of graphene quantum dots supported by metals, in particular Ir(111), and show how the goal of probing such wave functions on insulating supports during transport spectroscopy might be achieved.
△ Less
Submitted 15 May, 2015;
originally announced May 2015.
