-
MAIRA-2: Grounded Radiology Report Generation
Authors:
Shruthi Bannur,
Kenza Bouzid,
Daniel C. Castro,
Anton Schwaighofer,
Sam Bond-Taylor,
Maximilian Ilse,
Fernando Pérez-García,
Valentina Salvatelli,
Harshita Sharma,
Felix Meissen,
Mercy Ranjit,
Shaury Srivastav,
Julia Gong,
Fabian Falck,
Ozan Oktay,
Anja Thieme,
Matthew P. Lungren,
Maria Teodora Wetscherek,
Javier Alvarez-Valle,
Stephanie L. Hyland
Abstract:
Radiology reporting is a complex task that requires detailed image understanding, integration of multiple inputs, including comparison with prior imaging, and precise language generation. This makes it ideal for the development and use of generative multimodal models. Here, we extend report generation to include the localisation of individual findings on the image - a task we call grounded report…
▽ More
Radiology reporting is a complex task that requires detailed image understanding, integration of multiple inputs, including comparison with prior imaging, and precise language generation. This makes it ideal for the development and use of generative multimodal models. Here, we extend report generation to include the localisation of individual findings on the image - a task we call grounded report generation. Prior work indicates that grounding is important for clarifying image understanding and interpreting AI-generated text. Therefore, grounded reporting stands to improve the utility and transparency of automated report drafting. To enable evaluation of grounded reporting, we propose a novel evaluation framework - RadFact - leveraging the reasoning capabilities of large language models (LLMs). RadFact assesses the factuality of individual generated sentences, as well as correctness of generated spatial localisations when present. We introduce MAIRA-2, a large multimodal model combining a radiology-specific image encoder with a LLM, and trained for the new task of grounded report generation on chest X-rays. MAIRA-2 uses more comprehensive inputs than explored previously: the current frontal image, the current lateral image, the prior frontal image and prior report, as well as the Indication, Technique and Comparison sections of the current report. We demonstrate that these additions significantly improve report quality and reduce hallucinations, establishing a new state of the art on findings generation (without grounding) on MIMIC-CXR while demonstrating the feasibility of grounded reporting as a novel and richer task.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Challenges for Responsible AI Design and Workflow Integration in Healthcare: A Case Study of Automatic Feeding Tube Qualification in Radiology
Authors:
Anja Thieme,
Abhijith Rajamohan,
Benjamin Cooper,
Heather Groombridge,
Robert Simister,
Barney Wong,
Nicholas Woznitza,
Mark Ames Pinnock,
Maria Teodora Wetscherek,
Cecily Morrison,
Hannah Richardson,
Fernando Pérez-García,
Stephanie L. Hyland,
Shruthi Bannur,
Daniel C. Castro,
Kenza Bouzid,
Anton Schwaighofer,
Mercy Ranjit,
Harshita Sharma,
Matthew P. Lungren,
Ozan Oktay,
Javier Alvarez-Valle,
Aditya Nori,
Stephen Harris,
Joseph Jacob
Abstract:
Nasogastric tubes (NGTs) are feeding tubes that are inserted through the nose into the stomach to deliver nutrition or medication. If not placed correctly, they can cause serious harm, even death to patients. Recent AI developments demonstrate the feasibility of robustly detecting NGT placement from Chest X-ray images to reduce risks of sub-optimally or critically placed NGTs being missed or delay…
▽ More
Nasogastric tubes (NGTs) are feeding tubes that are inserted through the nose into the stomach to deliver nutrition or medication. If not placed correctly, they can cause serious harm, even death to patients. Recent AI developments demonstrate the feasibility of robustly detecting NGT placement from Chest X-ray images to reduce risks of sub-optimally or critically placed NGTs being missed or delayed in their detection, but gaps remain in clinical practice integration. In this study, we present a human-centered approach to the problem and describe insights derived following contextual inquiry and in-depth interviews with 15 clinical stakeholders. The interviews helped understand challenges in existing workflows, and how best to align technical capabilities with user needs and expectations. We discovered the trade-offs and complexities that need consideration when choosing suitable workflow stages, target users, and design configurations for different AI proposals. We explored how to balance AI benefits and risks for healthcare staff and patients within broader organizational and medical-legal constraints. We also identified data issues related to edge cases and data biases that affect model training and evaluation; how data documentation practices influence data preparation and labelling; and how to measure relevant AI outcomes reliably in future evaluations. We discuss how our work informs design and development of AI applications that are clinically useful, ethical, and acceptable in real-world healthcare services.
△ Less
Submitted 8 May, 2024;
originally announced May 2024.
-
Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
Authors:
Nur Yildirim,
Hannah Richardson,
Maria T. Wetscherek,
Junaid Bajwa,
Joseph Jacob,
Mark A. Pinnock,
Stephen Harris,
Daniel Coelho de Castro,
Shruthi Bannur,
Stephanie L. Hyland,
Pratik Ghosh,
Mercy Ranjit,
Kenza Bouzid,
Anton Schwaighofer,
Fernando Pérez-García,
Harshita Sharma,
Ozan Oktay,
Matthew Lungren,
Javier Alvarez-Valle,
Aditya Nori,
Anja Thieme
Abstract:
Recent advances in AI combine large language models (LLMs) with vision encoders that bring forward unprecedented technical capabilities to leverage for a wide range of healthcare applications. Focusing on the domain of radiology, vision-language models (VLMs) achieve good performance results for tasks such as generating radiology findings based on a patient's medical image, or answering visual que…
▽ More
Recent advances in AI combine large language models (LLMs) with vision encoders that bring forward unprecedented technical capabilities to leverage for a wide range of healthcare applications. Focusing on the domain of radiology, vision-language models (VLMs) achieve good performance results for tasks such as generating radiology findings based on a patient's medical image, or answering visual questions (e.g., 'Where are the nodules in this chest X-ray?'). However, the clinical utility of potential applications of these capabilities is currently underexplored. We engaged in an iterative, multidisciplinary design process to envision clinically relevant VLM interactions, and co-designed four VLM use concepts: Draft Report Generation, Augmented Report Review, Visual Search and Querying, and Patient Imaging History Highlights. We studied these concepts with 13 radiologists and clinicians who assessed the VLM concepts as valuable, yet articulated many design considerations. Reflecting on our findings, we discuss implications for integrating VLM capabilities in radiology, and for healthcare AI more generally.
△ Less
Submitted 21 February, 2024;
originally announced February 2024.
-
RadEdit: stress-testing biomedical vision models via diffusion image editing
Authors:
Fernando Pérez-García,
Sam Bond-Taylor,
Pedro P. Sanchez,
Boris van Breugel,
Daniel C. Castro,
Harshita Sharma,
Valentina Salvatelli,
Maria T. A. Wetscherek,
Hannah Richardson,
Matthew P. Lungren,
Aditya Nori,
Javier Alvarez-Valle,
Ozan Oktay,
Maximilian Ilse
Abstract:
Biomedical imaging datasets are often small and biased, meaning that real-world performance of predictive models can be substantially lower than expected from internal testing. This work proposes using generative image editing to simulate dataset shifts and diagnose failure modes of biomedical vision models; this can be used in advance of deployment to assess readiness, potentially reducing cost a…
▽ More
Biomedical imaging datasets are often small and biased, meaning that real-world performance of predictive models can be substantially lower than expected from internal testing. This work proposes using generative image editing to simulate dataset shifts and diagnose failure modes of biomedical vision models; this can be used in advance of deployment to assess readiness, potentially reducing cost and patient harm. Existing editing methods can produce undesirable changes, with spurious correlations learned due to the co-occurrence of disease and treatment interventions, limiting practical applicability. To address this, we train a text-to-image diffusion model on multiple chest X-ray datasets and introduce a new editing method RadEdit that uses multiple masks, if present, to constrain changes and ensure consistency in the edited images. We consider three types of dataset shifts: acquisition shift, manifestation shift, and population shift, and demonstrate that our approach can diagnose failures and quantify model robustness without additional data collection, complementing more qualitative tools for explainable AI.
△ Less
Submitted 3 April, 2024; v1 submitted 20 December, 2023;
originally announced December 2023.
-
MAIRA-1: A specialised large multimodal model for radiology report generation
Authors:
Stephanie L. Hyland,
Shruthi Bannur,
Kenza Bouzid,
Daniel C. Castro,
Mercy Ranjit,
Anton Schwaighofer,
Fernando Pérez-García,
Valentina Salvatelli,
Shaury Srivastav,
Anja Thieme,
Noel Codella,
Matthew P. Lungren,
Maria Teodora Wetscherek,
Ozan Oktay,
Javier Alvarez-Valle
Abstract:
We present a radiology-specific multimodal model for the task for generating radiological reports from chest X-rays (CXRs). Our work builds on the idea that large language model(s) can be equipped with multimodal capabilities through alignment with pre-trained vision encoders. On natural images, this has been shown to allow multimodal models to gain image understanding and description capabilities…
▽ More
We present a radiology-specific multimodal model for the task for generating radiological reports from chest X-rays (CXRs). Our work builds on the idea that large language model(s) can be equipped with multimodal capabilities through alignment with pre-trained vision encoders. On natural images, this has been shown to allow multimodal models to gain image understanding and description capabilities. Our proposed model (MAIRA-1) leverages a CXR-specific image encoder in conjunction with a fine-tuned large language model based on Vicuna-7B, and text-based data augmentation, to produce reports with state-of-the-art quality. In particular, MAIRA-1 significantly improves on the radiologist-aligned RadCliQ metric and across all lexical metrics considered. Manual review of model outputs demonstrates promising fluency and accuracy of generated reports while uncovering failure modes not captured by existing evaluation practices. More information and resources can be found on the project website: https://aka.ms/maira.
△ Less
Submitted 26 April, 2024; v1 submitted 22 November, 2023;
originally announced November 2023.
-
Exploring the Boundaries of GPT-4 in Radiology
Authors:
Qianchu Liu,
Stephanie Hyland,
Shruthi Bannur,
Kenza Bouzid,
Daniel C. Castro,
Maria Teodora Wetscherek,
Robert Tinn,
Harshita Sharma,
Fernando Pérez-García,
Anton Schwaighofer,
Pranav Rajpurkar,
Sameer Tajdin Khanna,
Hoifung Poon,
Naoto Usuyama,
Anja Thieme,
Aditya V. Nori,
Matthew P. Lungren,
Ozan Oktay,
Javier Alvarez-Valle
Abstract:
The recent success of general-domain large language models (LLMs) has significantly changed the natural language processing paradigm towards a unified foundation model across domains and applications. In this paper, we focus on assessing the performance of GPT-4, the most capable LLM so far, on the text-based applications for radiology reports, comparing against state-of-the-art (SOTA) radiology-s…
▽ More
The recent success of general-domain large language models (LLMs) has significantly changed the natural language processing paradigm towards a unified foundation model across domains and applications. In this paper, we focus on assessing the performance of GPT-4, the most capable LLM so far, on the text-based applications for radiology reports, comparing against state-of-the-art (SOTA) radiology-specific models. Exploring various prompting strategies, we evaluated GPT-4 on a diverse range of common radiology tasks and we found GPT-4 either outperforms or is on par with current SOTA radiology models. With zero-shot prompting, GPT-4 already obtains substantial gains ($\approx$ 10% absolute improvement) over radiology models in temporal sentence similarity classification (accuracy) and natural language inference ($F_1$). For tasks that require learning dataset-specific style or schema (e.g. findings summarisation), GPT-4 improves with example-based prompting and matches supervised SOTA. Our extensive error analysis with a board-certified radiologist shows GPT-4 has a sufficient level of radiology knowledge with only occasional errors in complex context that require nuanced domain knowledge. For findings summarisation, GPT-4 outputs are found to be overall comparable with existing manually-written impressions.
△ Less
Submitted 23 October, 2023;
originally announced October 2023.
