Evaluation of software impact designed for biomedical research: Are we measuring what's meaningful?
Authors:
Awan Afiaz,
Andrey Ivanov,
John Chamberlin,
David Hanauer,
Candace Savonen,
Mary J Goldman,
Martin Morgan,
Michael Reich,
Alexander Getka,
Aaron Holmes,
Sarthak Pati,
Dan Knight,
Paul C. Boutros,
Spyridon Bakas,
J. Gregory Caporaso,
Guilherme Del Fiol,
Harry Hochheiser,
Brian Haas,
Patrick D. Schloss,
James A. Eddy,
Jake Albrecht,
Andrey Fedorov,
Levi Waldron,
Ava M. Hoffman,
Richard L. Bradshaw
, et al. (2 additional authors not shown)
Abstract:
Software is vital for the advancement of biology and medicine. Analysis of usage and impact metrics can help developers determine user and community engagement, justify additional funding, encourage additional use, identify unanticipated use cases, and help define improvement areas. However, there are challenges associated with these analyses including distorted or misleading metrics, as well as e…
▽ More
Software is vital for the advancement of biology and medicine. Analysis of usage and impact metrics can help developers determine user and community engagement, justify additional funding, encourage additional use, identify unanticipated use cases, and help define improvement areas. However, there are challenges associated with these analyses including distorted or misleading metrics, as well as ethical and security concerns. More attention to the nuances involved in capturing impact across the spectrum of biological software is needed. Furthermore, some tools may be especially beneficial to a small audience, yet may not have compelling typical usage metrics. We propose more general guidelines, as well as strategies for more specific types of software. We highlight outstanding issues regarding how communities measure or evaluate software impact. To get a deeper understanding of current practices for software evaluations, we performed a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We also investigated software among this community and others to assess how often infrastructure that supports such evaluations is implemented and how this impacts rates of papers describing usage of the software. We find that developers recognize the utility of analyzing software usage, but struggle to find the time or funding for such analyses. We also find that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seem to be associated with increased usage rates. Our findings can help scientific software developers make the most out of evaluations of their software.
△ Less
Submitted 5 June, 2023;
originally announced June 2023.
GaNDLF: A Generally Nuanced Deep Learning Framework for Scalable End-to-End Clinical Workflows in Medical Imaging
Authors:
Sarthak Pati,
Siddhesh P. Thakur,
İbrahim Ethem Hamamcı,
Ujjwal Baid,
Bhakti Baheti,
Megh Bhalerao,
Orhun Güley,
Sofia Mouchtaris,
David Lang,
Spyridon Thermos,
Karol Gotkowski,
Camila González,
Caleb Grenko,
Alexander Getka,
Brandon Edwards,
Micah Sheller,
Junwen Wu,
Deepthi Karkada,
Ravi Panchumarthy,
Vinayak Ahluwalia,
Chunrui Zou,
Vishnu Bashyam,
Yuemeng Li,
Babak Haghighi,
Rhea Chitalia
, et al. (17 additional authors not shown)
Abstract:
Deep Learning (DL) has the potential to optimize machine learning in both the scientific and clinical communities. However, greater expertise is required to develop DL algorithms, and the variability of implementations hinders their reproducibility, translation, and deployment. Here we present the community-driven Generally Nuanced Deep Learning Framework (GaNDLF), with the goal of lowering these…
▽ More
Deep Learning (DL) has the potential to optimize machine learning in both the scientific and clinical communities. However, greater expertise is required to develop DL algorithms, and the variability of implementations hinders their reproducibility, translation, and deployment. Here we present the community-driven Generally Nuanced Deep Learning Framework (GaNDLF), with the goal of lowering these barriers. GaNDLF makes the mechanism of DL development, training, and inference more stable, reproducible, interpretable, and scalable, without requiring an extensive technical background. GaNDLF aims to provide an end-to-end solution for all DL-related tasks in computational precision medicine. We demonstrate the ability of GaNDLF to analyze both radiology and histology images, with built-in support for k-fold cross-validation, data augmentation, multiple modalities and output classes. Our quantitative performance evaluation on numerous use cases, anatomies, and computational tasks supports GaNDLF as a robust application framework for deployment in clinical workflows.
△ Less
Submitted 16 May, 2023; v1 submitted 25 February, 2021;
originally announced March 2021.
