-
MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation
Authors:
Alexandros Karargyris,
Renato Umeton,
Micah J. Sheller,
Alejandro Aristizabal,
Johnu George,
Srini Bala,
Daniel J. Beutel,
Victor Bittorf,
Akshay Chaudhari,
Alexander Chowdhury,
Cody Coleman,
Bala Desinghu,
Gregory Diamos,
Debo Dutta,
Diane Feddema,
Grigori Fursin,
Junyi Guo,
Xinyuan Huang,
David Kanter,
Satyananda Kashyap,
Nicholas Lane,
Indranil Mallick,
Pietro Mascagni,
Virendra Mehta,
Vivek Natarajan
, et al. (17 additional authors not shown)
Abstract:
Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf,…
▽ More
Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap. We call for researchers and organizations to join us in creating the MedPerf open benchmarking platform.
△ Less
Submitted 28 December, 2021; v1 submitted 29 September, 2021;
originally announced October 2021.
-
OpenFL: An open-source framework for Federated Learning
Authors:
G Anthony Reina,
Alexey Gruzdev,
Patrick Foley,
Olga Perepelkina,
Mansi Sharma,
Igor Davidyuk,
Ilya Trushkin,
Maksim Radionov,
Aleksandr Mokrov,
Dmitry Agapov,
Jason Martin,
Brandon Edwards,
Micah J. Sheller,
Sarthak Pati,
Prakash Narayana Moorthy,
Shih-han Wang,
Prashant Shah,
Spyridon Bakas
Abstract:
Federated learning (FL) is a computational paradigm that enables organizations to collaborate on machine learning (ML) projects without sharing sensitive data, such as, patient records, financial data, or classified secrets. Open Federated Learning (OpenFL https://github.com/intel/openfl) is an open-source framework for training ML algorithms using the data-private collaborative learning paradigm…
▽ More
Federated learning (FL) is a computational paradigm that enables organizations to collaborate on machine learning (ML) projects without sharing sensitive data, such as, patient records, financial data, or classified secrets. Open Federated Learning (OpenFL https://github.com/intel/openfl) is an open-source framework for training ML algorithms using the data-private collaborative learning paradigm of FL. OpenFL works with training pipelines built with both TensorFlow and PyTorch, and can be easily extended to other ML and deep learning frameworks. Here, we summarize the motivation and development characteristics of OpenFL, with the intention of facilitating its application to existing ML model training in a production environment. Finally, we describe the first use of the OpenFL framework to train consensus ML models in a consortium of international healthcare organizations, as well as how it facilitates the first computational competition on FL.
△ Less
Submitted 13 May, 2021;
originally announced May 2021.
-
Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation
Authors:
Micah J Sheller,
G Anthony Reina,
Brandon Edwards,
Jason Martin,
Spyridon Bakas
Abstract:
Deep learning models for semantic segmentation of images require large amounts of data. In the medical imaging domain, acquiring sufficient data is a significant challenge. Labeling medical image data requires expert knowledge. Collaboration between institutions could address this challenge, but sharing medical data to a centralized location faces various legal, privacy, technical, and data-owners…
▽ More
Deep learning models for semantic segmentation of images require large amounts of data. In the medical imaging domain, acquiring sufficient data is a significant challenge. Labeling medical image data requires expert knowledge. Collaboration between institutions could address this challenge, but sharing medical data to a centralized location faces various legal, privacy, technical, and data-ownership challenges, especially among international institutions. In this study, we introduce the first use of federated learning for multi-institutional collaboration, enabling deep learning modeling without sharing patient data. Our quantitative results demonstrate that the performance of federated semantic segmentation models (Dice=0.852) on multimodal brain scans is similar to that of models trained by sharing data (Dice=0.862). We compare federated learning with two alternative collaborative learning methods and find that they fail to match the performance of federated learning.
△ Less
Submitted 22 October, 2018; v1 submitted 9 October, 2018;
originally announced October 2018.
