Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation
Authors:
Ryutaro Tanno,
David G. T. Barrett,
Andrew Sellergren,
Sumedh Ghaisas,
Sumanth Dathathri,
Abigail See,
Johannes Welbl,
Karan Singhal,
Shekoofeh Azizi,
Tao Tu,
Mike Schaekermann,
Rhys May,
Roy Lee,
SiWai Man,
Zahra Ahmed,
Sara Mahdavi,
Yossi Matias,
Joelle Barral,
Ali Eslami,
Danielle Belgrave,
Vivek Natarajan,
Shravya Shetty,
Pushmeet Kohli,
Po-Sen Huang,
Alan Karthikesalingam
, et al. (1 additional authors not shown)
Abstract:
Radiology reports are an instrumental part of modern medicine, informing key clinical decisions such as diagnosis and treatment. The worldwide shortage of radiologists, however, restricts access to expert care and imposes heavy workloads, contributing to avoidable errors and delays in report delivery. While recent progress in automated report generation with vision-language models offer clear pote…
▽ More
Radiology reports are an instrumental part of modern medicine, informing key clinical decisions such as diagnosis and treatment. The worldwide shortage of radiologists, however, restricts access to expert care and imposes heavy workloads, contributing to avoidable errors and delays in report delivery. While recent progress in automated report generation with vision-language models offer clear potential in ameliorating the situation, the path to real-world adoption has been stymied by the challenge of evaluating the clinical quality of AI-generated reports. In this study, we build a state-of-the-art report generation system for chest radiographs, $\textit{Flamingo-CXR}$, by fine-tuning a well-known vision-language foundation model on radiology data. To evaluate the quality of the AI-generated reports, a group of 16 certified radiologists provide detailed evaluations of AI-generated and human written reports for chest X-rays from an intensive care setting in the United States and an inpatient setting in India. At least one radiologist (out of two per case) preferred the AI report to the ground truth report in over 60$\%$ of cases for both datasets. Amongst the subset of AI-generated reports that contain errors, the most frequently cited reasons were related to the location and finding, whereas for human written reports, most mistakes were related to severity and finding. This disparity suggested potential complementarity between our AI system and human experts, prompting us to develop an assistive scenario in which Flamingo-CXR generates a first-draft report, which is subsequently revised by a clinician. This is the first demonstration of clinician-AI collaboration for report writing, and the resultant reports are assessed to be equivalent or preferred by at least one radiologist to reports written by experts alone in 80$\%$ of in-patient cases and 60$\%$ of intensive care cases.
△ Less
Submitted 20 December, 2023; v1 submitted 30 November, 2023;
originally announced November 2023.
Inferring a Continuous Distribution of Atom Coordinates from Cryo-EM Images using VAEs
Authors:
Dan Rosenbaum,
Marta Garnelo,
Michal Zielinski,
Charlie Beattie,
Ellen Clancy,
Andrea Huber,
Pushmeet Kohli,
Andrew W. Senior,
John Jumper,
Carl Doersch,
S. M. Ali Eslami,
Olaf Ronneberger,
Jonas Adler
Abstract:
Cryo-electron microscopy (cryo-EM) has revolutionized experimental protein structure determination. Despite advances in high resolution reconstruction, a majority of cryo-EM experiments provide either a single state of the studied macromolecule, or a relatively small number of its conformations. This reduces the effectiveness of the technique for proteins with flexible regions, which are known to…
▽ More
Cryo-electron microscopy (cryo-EM) has revolutionized experimental protein structure determination. Despite advances in high resolution reconstruction, a majority of cryo-EM experiments provide either a single state of the studied macromolecule, or a relatively small number of its conformations. This reduces the effectiveness of the technique for proteins with flexible regions, which are known to play a key role in protein function. Recent methods for capturing conformational heterogeneity in cryo-EM data model it in volume space, making recovery of continuous atomic structures challenging. Here we present a fully deep-learning-based approach using variational auto-encoders (VAEs) to recover a continuous distribution of atomic protein structures and poses directly from picked particle images and demonstrate its efficacy on realistic simulated data. We hope that methods built on this work will allow incorporation of stronger prior information about protein structure and enable better understanding of non-rigid protein structures.
△ Less
Submitted 26 June, 2021;
originally announced June 2021.
Multi-scale Microaneurysms Segmentation Using Embedding Triplet Loss
Authors:
Mhd Hasan Sarhan,
Shadi Albarqouni,
Mehmet Yigitsoy,
Nassir Navab,
Abouzar Eslami
Abstract:
Deep learning techniques are recently being used in fundus image analysis and diabetic retinopathy detection. Microaneurysms are an important indicator of diabetic retinopathy progression. We introduce a two-stage deep learning approach for microaneurysms segmentation using multiple scales of the input with selective sampling and embedding triplet loss. The model first segments on two scales and t…
▽ More
Deep learning techniques are recently being used in fundus image analysis and diabetic retinopathy detection. Microaneurysms are an important indicator of diabetic retinopathy progression. We introduce a two-stage deep learning approach for microaneurysms segmentation using multiple scales of the input with selective sampling and embedding triplet loss. The model first segments on two scales and then the segmentations are refined with a classification model. To enhance the discriminative power of the classification model, we incorporate triplet embedding loss with a selective sampling routine. The model is evaluated quantitatively to assess the segmentation performance and qualitatively to analyze the model predictions. This approach introduces a 30.29% relative improvement over the fully convolutional neural network.
△ Less
Submitted 14 August, 2019; v1 submitted 18 April, 2019;
originally announced April 2019.
