-
PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery
Authors:
Runlong He,
Mengya Xu,
Adrito Das,
Danyal Z. Khan,
Sophia Bano,
Hani J. Marcus,
Danail Stoyanov,
Matthew J. Clarkson,
Mobarakol Islam
Abstract:
Visual Question Answering (VQA) within the surgical domain, utilizing Large Language Models (LLMs), offers a distinct opportunity to improve intra-operative decision-making and facilitate intuitive surgeon-AI interaction. However, the development of LLMs for surgical VQA is hindered by the scarcity of diverse and extensive datasets with complex reasoning tasks. Moreover, contextual fusion of the i…
▽ More
Visual Question Answering (VQA) within the surgical domain, utilizing Large Language Models (LLMs), offers a distinct opportunity to improve intra-operative decision-making and facilitate intuitive surgeon-AI interaction. However, the development of LLMs for surgical VQA is hindered by the scarcity of diverse and extensive datasets with complex reasoning tasks. Moreover, contextual fusion of the image and text modalities remains an open research challenge due to the inherent differences between these two types of information and the complexity involved in aligning them. This paper introduces PitVQA, a novel dataset specifically designed for VQA in endonasal pituitary surgery and PitVQA-Net, an adaptation of the GPT2 with a novel image-grounded text embedding for surgical VQA. PitVQA comprises 25 procedural videos and a rich collection of question-answer pairs spanning crucial surgical aspects such as phase and step recognition, context understanding, tool detection and localization, and tool-tissue interactions. PitVQA-Net consists of a novel image-grounded text embedding that projects image and text features into a shared embedding space and GPT2 Backbone with an excitation block classification head to generate contextually relevant answers within the complex domain of endonasal pituitary surgery. Our image-grounded text embedding leverages joint embedding, cross-attention and contextual representation to understand the contextual relationship between questions and surgical images. We demonstrate the effectiveness of PitVQA-Net on both the PitVQA and the publicly available EndoVis18-VQA dataset, achieving improvements in balanced accuracy of 8% and 9% over the most recent baselines, respectively. Our code and dataset is available at https://github.com/mobarakol/PitVQA.
△ Less
Submitted 22 May, 2024;
originally announced May 2024.
-
Shifted-Windows Transformers for the Detection of Cerebral Aneurysms in Microsurgery
Authors:
**fan Zhou,
William Muirhead,
Simon C. Williams,
Danail Stoyanov,
Hani J. Marcus,
Evangelos B. Mazomenos
Abstract:
Purpose: Microsurgical Aneurysm Clip** Surgery (MACS) carries a high risk for intraoperative aneurysm rupture. Automated recognition of instances when the aneurysm is exposed in the surgical video would be a valuable reference point for neuronavigation, indicating phase transitioning and more importantly designating moments of high risk for rupture. This article introduces the MACS dataset conta…
▽ More
Purpose: Microsurgical Aneurysm Clip** Surgery (MACS) carries a high risk for intraoperative aneurysm rupture. Automated recognition of instances when the aneurysm is exposed in the surgical video would be a valuable reference point for neuronavigation, indicating phase transitioning and more importantly designating moments of high risk for rupture. This article introduces the MACS dataset containing 16 surgical videos with frame-level expert annotations and proposes a learning methodology for surgical scene understanding identifying video frames with the aneurysm present in the operating microscope's field-of-view. Methods: Despite the dataset imbalance (80% no presence, 20% presence) and developed without explicit annotations, we demonstrate the applicability of Transformer-based deep learning architectures (MACSSwin-T, vidMACSSwin-T) to detect the aneurysm and classify MACS frames accordingly. We evaluate the proposed models in multiple-fold cross-validation experiments with independent sets and in an unseen set of 15 images against 10 human experts (neurosurgeons). Results: Average (across folds) accuracy of 80.8% (range 78.5%-82.4%) and 87.1% (range 85.1%-91.3%) is obtained for the image- and video-level approach respectively, demonstrating that the models effectively learn the classification task. Qualitative evaluation of the models' class activation maps show these to be localized on the aneurysm's actual location. Depending on the decision threshold, MACSWin-T achieves 66.7% to 86.7% accuracy in the unseen images, compared to 82% of human raters, with moderate to strong correlation.
△ Less
Submitted 16 March, 2023;
originally announced March 2023.
-
Coexistence and Spectrum Sharing Above 100 GHz
Authors:
Michele Polese,
Xavier Cantos-Roman,
Arjun Singh,
Michael J. Marcus,
Thomas J. Maccarone,
Tommaso Melodia,
Josep M. Jornet
Abstract:
[...] This paper explores how spectrum policy and spectrum technologies can evolve to enable sharing among different stakeholders in the above 100 GHz spectrum, without introducing harmful interference or disrupting either security applications or fundamental science exploration. This portion of the spectrum presents new opportunities to design spectrum sharing schemes, based on novel antenna desi…
▽ More
[...] This paper explores how spectrum policy and spectrum technologies can evolve to enable sharing among different stakeholders in the above 100 GHz spectrum, without introducing harmful interference or disrupting either security applications or fundamental science exploration. This portion of the spectrum presents new opportunities to design spectrum sharing schemes, based on novel antenna designs, directional ultra-high-rate communications, and active/passive user coordination. The paper provides a tutorial on current regulations above 100 GHz, and highlights how sharing is central to allowing each stakeholder to make the most out of this spectrum. It then defines - through detailed simulations based on standard International Telecommunications Union (ITU) channel and antenna models - scenarios in which active users may introduce harmful interference to passive sensing. Based on this evaluation, it reviews a number of promising techniques that can enable active/passive sharing above 100 GHz. The critical review and tutorial on policy and technologies of this paper have the potential to kickstart future research and regulations that promote safe coexistence between active and passive users above 100 GHz, further benefiting the development of digital technologies and scientific exploration.
△ Less
Submitted 12 June, 2023; v1 submitted 28 October, 2021;
originally announced October 2021.
-
Do Transformer Modifications Transfer Across Implementations and Applications?
Authors:
Sharan Narang,
Hyung Won Chung,
Yi Tay,
William Fedus,
Thibault Fevry,
Michael Matena,
Karishma Malkan,
Noah Fiedel,
Noam Shazeer,
Zhenzhong Lan,
Yanqi Zhou,
Wei Li,
Nan Ding,
Jake Marcus,
Adam Roberts,
Colin Raffel
Abstract:
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we f…
▽ More
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.
△ Less
Submitted 10 September, 2021; v1 submitted 23 February, 2021;
originally announced February 2021.
-
Surgical Data Science -- from Concepts toward Clinical Translation
Authors:
Lena Maier-Hein,
Matthias Eisenmann,
Duygu Sarikaya,
Keno März,
Toby Collins,
Anand Malpani,
Johannes Fallert,
Hubertus Feussner,
Stamatia Giannarou,
Pietro Mascagni,
Hirenkumar Nakawala,
Adrian Park,
Carla Pugh,
Danail Stoyanov,
Swaroop S. Vedula,
Kevin Cleary,
Gabor Fichtinger,
Germain Forestier,
Bernard Gibaud,
Teodor Grantcharov,
Makoto Hashizume,
Doreen Heckmann-Nötzel,
Hannes G. Kenngott,
Ron Kikinis,
Lars Mündermann
, et al. (25 additional authors not shown)
Abstract:
Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applica…
▽ More
Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.
△ Less
Submitted 30 July, 2021; v1 submitted 30 October, 2020;
originally announced November 2020.
-
Surgical Video Motion Magnification with Suppression of Instrument Artefacts
Authors:
Mirek Janatka,
Hani J. Marcus,
Neil L. Dorward,
Danail Stoyanov
Abstract:
Video motion magnification could directly highlight subsurface blood vessels in endoscopic video in order to prevent inadvertent damage and bleeding. Applying motion filters to the full surgical image is however sensitive to residual motion from the surgical instruments and can impede practical application due to aberration motion artefacts. By storing the temporal filter response from local spati…
▽ More
Video motion magnification could directly highlight subsurface blood vessels in endoscopic video in order to prevent inadvertent damage and bleeding. Applying motion filters to the full surgical image is however sensitive to residual motion from the surgical instruments and can impede practical application due to aberration motion artefacts. By storing the temporal filter response from local spatial frequency information for a single cardiovascular cycle prior to tool introduction to the scene, a filter can be used to determine if motion magnification should be active for a spatial region of the surgical image. In this paper, we propose a strategy to reduce aberration due to non-physiological motion for surgical video motion magnification. We present promising results on endoscopic transnasal transsphenoidal pituitary surgery with a quantitative comparison to recent methods using Structural Similarity (SSIM), as well as qualitative analysis by comparing spatio-temporal cross sections of the videos and individual frames.
△ Less
Submitted 15 September, 2020;
originally announced September 2020.
-
Scalable and accurate deep learning for electronic health records
Authors:
Alvin Rajkomar,
Eyal Oren,
Kai Chen,
Andrew M. Dai,
Nissan Hajaj,
Peter J. Liu,
Xiaobing Liu,
Mimi Sun,
Patrik Sundberg,
Hector Yee,
Kun Zhang,
Gavin E. Duggan,
Gerardo Flores,
Michaela Hardt,
Jamie Irvine,
Quoc Le,
Kurt Litsch,
Jake Marcus,
Alexander Mossin,
Justin Tansuwan,
De Wang,
James Wexler,
Jimbo Wilson,
Dana Ludwig,
Samuel L. Volchenboum
, et al. (9 additional authors not shown)
Abstract:
Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of p…
▽ More
Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of patients' entire, raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two U.S. academic medical centers with 216,221 adult patients hospitalized for at least 24 hours. In the sequential format we propose, this volume of EHR data unrolled into a total of 46,864,534,945 data points, including clinical notes. Deep learning models achieved high accuracy for tasks such as predicting in-hospital mortality (AUROC across sites 0.93-0.94), 30-day unplanned readmission (AUROC 0.75-0.76), prolonged length of stay (AUROC 0.85-0.86), and all of a patient's final discharge diagnoses (frequency-weighted AUROC 0.90). These models outperformed state-of-the-art traditional predictive models in all cases. We also present a case-study of a neural-network attribution system, which illustrates how clinicians can gain some transparency into the predictions. We believe that this approach can be used to create accurate and scalable predictions for a variety of clinical scenarios, complete with explanations that directly highlight evidence in the patient's chart.
△ Less
Submitted 11 May, 2018; v1 submitted 24 January, 2018;
originally announced January 2018.
-
Multi-task Prediction of Disease Onsets from Longitudinal Lab Tests
Authors:
Narges Razavian,
Jake Marcus,
David Sontag
Abstract:
Disparate areas of machine learning have benefited from models that can take raw data with little preprocessing as input and learn rich representations of that raw data in order to perform well on a given prediction task. We evaluate this approach in healthcare by using longitudinal measurements of lab tests, one of the more raw signals of a patient's health state widely available in clinical data…
▽ More
Disparate areas of machine learning have benefited from models that can take raw data with little preprocessing as input and learn rich representations of that raw data in order to perform well on a given prediction task. We evaluate this approach in healthcare by using longitudinal measurements of lab tests, one of the more raw signals of a patient's health state widely available in clinical data, to predict disease onsets. In particular, we train a Long Short-Term Memory (LSTM) recurrent neural network and two novel convolutional neural networks for multi-task prediction of disease onset for 133 conditions based on 18 common lab tests measured over time in a cohort of 298K patients derived from 8 years of administrative claims data. We compare the neural networks to a logistic regression with several hand-engineered, clinically relevant features. We find that the representation-based learning approaches significantly outperform this baseline. We believe that our work suggests a new avenue for patient risk stratification based solely on lab results.
△ Less
Submitted 20 September, 2016; v1 submitted 1 August, 2016;
originally announced August 2016.
