-
Fair Evaluation of Federated Learning Algorithms for Automated Breast Density Classification: The Results of the 2022 ACR-NCI-NVIDIA Federated Learning Challenge
Authors:
Kendall Schmidt,
Benjamin Bearce,
Ken Chang,
Laura Coombs,
Keyvan Farahani,
Marawan Elbatele,
Kaouther Mouhebe,
Robert Marti,
Ruipeng Zhang,
Yao Zhang,
Yanfeng Wang,
Yaojun Hu,
Haochao Ying,
Yuyang Xu,
Conrad Testagrose,
Mutlu Demirer,
Vikash Gupta,
Ünal Akünal,
Markus Bujotzek,
Klaus H. Maier-Hein,
Yi Qin,
Xiaomeng Li,
Jayashree Kalpathy-Cramer,
Holger R. Roth
Abstract:
The correct interpretation of breast density is important in the assessment of breast cancer risk. AI has been shown capable of accurately predicting breast density, however, due to the differences in imaging characteristics across mammography systems, models built using data from one system do not generalize well to other systems. Though federated learning (FL) has emerged as a way to improve the…
▽ More
The correct interpretation of breast density is important in the assessment of breast cancer risk. AI has been shown capable of accurately predicting breast density, however, due to the differences in imaging characteristics across mammography systems, models built using data from one system do not generalize well to other systems. Though federated learning (FL) has emerged as a way to improve the generalizability of AI without the need to share data, the best way to preserve features from all training data during FL is an active area of research. To explore FL methodology, the breast density classification FL challenge was hosted in partnership with the American College of Radiology, Harvard Medical School's Mass General Brigham, University of Colorado, NVIDIA, and the National Institutes of Health National Cancer Institute. Challenge participants were able to submit docker containers capable of implementing FL on three simulated medical facilities, each containing a unique large mammography dataset. The breast density FL challenge ran from June 15 to September 5, 2022, attracting seven finalists from around the world. The winning FL submission reached a linear kappa score of 0.653 on the challenge test data and 0.413 on an external testing dataset, scoring comparably to a model trained on the same data in a central location.
△ Less
Submitted 22 May, 2024;
originally announced May 2024.
-
Real-World Federated Learning in Radiology: Hurdles to overcome and Benefits to gain
Authors:
Markus R. Bujotzek,
Ünal Akünal,
Stefan Denner,
Peter Neher,
Maximilian Zenk,
Eric Frodl,
Astha Jaiswal,
Moon Kim,
Nicolai R. Krekiehn,
Manuel Nickel,
Richard Ruppel,
Marcus Both,
Felix Döllinger,
Marcel Opitz,
Thorsten Persigehl,
Jens Kleesiek,
Tobias Penzkofer,
Klaus Maier-Hein,
Rickmer Braren,
Andreas Bucher
Abstract:
Objective: Federated Learning (FL) enables collaborative model training while kee** data locally. Currently, most FL studies in radiology are conducted in simulated environments due to numerous hurdles impeding its translation into practice. The few existing real-world FL initiatives rarely communicate specific measures taken to overcome these hurdles, leaving behind a significant knowledge gap.…
▽ More
Objective: Federated Learning (FL) enables collaborative model training while kee** data locally. Currently, most FL studies in radiology are conducted in simulated environments due to numerous hurdles impeding its translation into practice. The few existing real-world FL initiatives rarely communicate specific measures taken to overcome these hurdles, leaving behind a significant knowledge gap. Minding efforts to implement real-world FL, there is a notable lack of comprehensive assessment comparing FL to less complex alternatives. Materials & Methods: We extensively reviewed FL literature, categorizing insights along with our findings according to their nature and phase while establishing a FL initiative, summarized to a comprehensive guide. We developed our own FL infrastructure within the German Radiological Cooperative Network (RACOON) and demonstrated its functionality by training FL models on lung pathology segmentation tasks across six university hospitals. We extensively evaluated FL against less complex alternatives in three distinct evaluation scenarios. Results: The proposed guide outlines essential steps, identified hurdles, and proposed solutions for establishing successful FL initiatives conducting real-world experiments. Our experimental results show that FL outperforms less complex alternatives in all evaluation scenarios, justifying the effort required to translate FL into real-world applications. Discussion & Conclusion: Our proposed guide aims to aid future FL researchers in circumventing pitfalls and accelerating translation of FL into radiological applications. Our results underscore the value of efforts needed to translate FL into real-world applications by demonstrating advantageous performance over alternatives, and emphasize the importance of strategic organization, robust management of distributed data and infrastructure in real-world settings.
△ Less
Submitted 15 May, 2024;
originally announced May 2024.
-
Leveraging Foundation Models for Content-Based Medical Image Retrieval in Radiology
Authors:
Stefan Denner,
David Zimmerer,
Dimitrios Bounias,
Markus Bujotzek,
Shuhan Xiao,
Lisa Kausch,
Philipp Schader,
Tobias Penzkofer,
Paul F. Jäger,
Klaus Maier-Hein
Abstract:
Content-based image retrieval (CBIR) has the potential to significantly improve diagnostic aid and medical research in radiology. Current CBIR systems face limitations due to their specialization to certain pathologies, limiting their utility. In response, we propose using vision foundation models as powerful and versatile off-the-shelf feature extractors for content-based medical image retrieval.…
▽ More
Content-based image retrieval (CBIR) has the potential to significantly improve diagnostic aid and medical research in radiology. Current CBIR systems face limitations due to their specialization to certain pathologies, limiting their utility. In response, we propose using vision foundation models as powerful and versatile off-the-shelf feature extractors for content-based medical image retrieval. By benchmarking these models on a comprehensive dataset of 1.6 million 2D radiological images spanning four modalities and 161 pathologies, we identify weakly-supervised models as superior, achieving a P@1 of up to 0.594. This performance not only competes with a specialized model but does so without the need for fine-tuning. Our analysis further explores the challenges in retrieving pathological versus anatomical structures, indicating that accurate retrieval of pathological features presents greater difficulty. Despite these challenges, our research underscores the vast potential of foundation models for CBIR in radiology, proposing a shift towards versatile, general-purpose medical image retrieval systems that do not require specific tuning.
△ Less
Submitted 17 April, 2024; v1 submitted 11 March, 2024;
originally announced March 2024.
-
Efficient Large Scale Medical Image Dataset Preparation for Machine Learning Applications
Authors:
Stefan Denner,
Jonas Scherer,
Klaus Kades,
Dimitrios Bounias,
Philipp Schader,
Lisa Kausch,
Markus Bujotzek,
Andreas Michael Bucher,
Tobias Penzkofer,
Klaus Maier-Hein
Abstract:
In the rapidly evolving field of medical imaging, machine learning algorithms have become indispensable for enhancing diagnostic accuracy. However, the effectiveness of these algorithms is contingent upon the availability and organization of high-quality medical imaging datasets. Traditional Digital Imaging and Communications in Medicine (DICOM) data management systems are inadequate for handling…
▽ More
In the rapidly evolving field of medical imaging, machine learning algorithms have become indispensable for enhancing diagnostic accuracy. However, the effectiveness of these algorithms is contingent upon the availability and organization of high-quality medical imaging datasets. Traditional Digital Imaging and Communications in Medicine (DICOM) data management systems are inadequate for handling the scale and complexity of data required to be facilitated in machine learning algorithms. This paper introduces an innovative data curation tool, developed as part of the Kaapana open-source toolkit, aimed at streamlining the organization, management, and processing of large-scale medical imaging datasets. The tool is specifically tailored to meet the needs of radiologists and machine learning researchers. It incorporates advanced search, auto-annotation and efficient tagging functionalities for improved data curation. Additionally, the tool facilitates quality control and review, enabling researchers to validate image and segmentation quality in large datasets. It also plays a critical role in uncovering potential biases in datasets by aggregating and visualizing metadata, which is essential for develo** robust machine learning models. Furthermore, Kaapana is integrated within the Radiological Cooperative Network (RACOON), a pioneering initiative aimed at creating a comprehensive national infrastructure for the aggregation, transmission, and consolidation of radiological data across all university clinics throughout Germany. A supplementary video showcasing the tool's functionalities can be accessed at https://bit.ly/MICCAI-DEMI2023.
△ Less
Submitted 29 September, 2023;
originally announced September 2023.
