-
VLST: Virtual Lung Screening Trial for Lung Cancer Detection Using Virtual Imaging Trial
Authors:
Fakrul Islam Tushar,
Liesbeth Vancoillie,
Cindy McCabe,
Amareswararao Kavuri,
Lavsen Dahal,
Brian Harrawood,
Milo Fryling,
Mojtaba Zarei,
Saman Sotoudeh-Paima,
Fong Chi Ho,
Dhrubajyoti Ghosh,
Sheng Luo,
W. Paul Segars,
Ehsan Abadi,
Kyle J. Lafata,
Ehsan Samei,
Joseph Y. Lo
Abstract:
Importance: The efficacy of lung cancer screening can be significantly impacted by the imaging modality used. This Virtual Lung Screening Trial (VLST) addresses the critical need for precision in lung cancer diagnostics and the potential for reducing unnecessary radiation exposure in clinical settings.
Objectives: To establish a virtual imaging trial (VIT) platform that accurately simulates real…
▽ More
Importance: The efficacy of lung cancer screening can be significantly impacted by the imaging modality used. This Virtual Lung Screening Trial (VLST) addresses the critical need for precision in lung cancer diagnostics and the potential for reducing unnecessary radiation exposure in clinical settings.
Objectives: To establish a virtual imaging trial (VIT) platform that accurately simulates real-world lung screening trials (LSTs) to assess the diagnostic accuracy of CT and CXR modalities.
Design, Setting, and Participants: Utilizing computational models and machine learning algorithms, we created a diverse virtual patient population. The cohort, designed to mirror real-world demographics, was assessed using virtual imaging techniques that reflect historical imaging technologies.
Main Outcomes and Measures: The primary outcome was the difference in the Area Under the Curve (AUC) for CT and CXR modalities across lesion types and sizes.
Results: The study analyzed 298 CT and 313 CXR simulated images from 313 virtual patients, with a lesion-level AUC of 0.81 (95% CI: 0.78-0.84) for CT and 0.55 (95% CI: 0.53-0.56) for CXR. At the patient level, CT demonstrated an AUC of 0.85 (95% CI: 0.80-0.89), compared to 0.53 (95% CI: 0.47-0.60) for CXR. Subgroup analyses indicated CT's superior performance in detecting homogeneous lesions (AUC of 0.97 for lesion-level) and heterogeneous lesions (AUC of 0.71 for lesion-level) as well as in identifying larger nodules (AUC of 0.98 for nodules > 8 mm).
Conclusion and Relevance: The VIT platform validated the superior diagnostic accuracy of CT over CXR, especially for smaller nodules, underscoring its potential to replicate real clinical imaging trials. These findings advocate for the integration of virtual trials in the evaluation and improvement of imaging-based diagnostic tools.
△ Less
Submitted 17 April, 2024;
originally announced April 2024.
-
Virtual imaging trials improved the transparency and reliability of AI systems in COVID-19 imaging
Authors:
Fakrul Islam Tushar,
Lavsen Dahal,
Saman Sotoudeh-Paima,
Ehsan Abadi,
W. Paul Segars,
Ehsan Samei,
Joseph Y. Lo
Abstract:
The credibility of AI models in medical imaging is often challenged by reproducibility issues and obscured clinical insights, a reality highlighted during the COVID-19 pandemic by many reports of near-perfect artificial intelligence (AI) models that all failed to generalize. To address these concerns, we propose a virtual imaging trial framework, employing a diverse collection of medical images th…
▽ More
The credibility of AI models in medical imaging is often challenged by reproducibility issues and obscured clinical insights, a reality highlighted during the COVID-19 pandemic by many reports of near-perfect artificial intelligence (AI) models that all failed to generalize. To address these concerns, we propose a virtual imaging trial framework, employing a diverse collection of medical images that are both clinical and simulated. In this study, COVID-19 serves as a case example to unveil the intrinsic and extrinsic factors influencing AI performance. Our findings underscore a significant impact of dataset characteristics on AI efficacy. Even when trained on large, diverse clinical datasets with thousands of patients, AI performance plummeted by up to 20% in generalization. However, virtual imaging trials offer a robust platform for objective assessment, unveiling nuanced insights into the relationships between patient- and physics-based factors and AI performance. For instance, disease extent markedly influenced AI efficacy, computed tomography (CT) out-performed chest radiography (CXR), while imaging dose exhibited minimal impact. Using COVID-19 as a case study, this virtual imaging trial study verified that radiology AI models often suffer from a reproducibility crisis. Virtual imaging trials not only offered a solution for objective performance assessment but also extracted several clinical insights. This study illuminates the path for leveraging virtual imaging to augment the reliability, transparency, and clinical relevance of AI in medical imaging.
△ Less
Submitted 31 March, 2024; v1 submitted 17 August, 2023;
originally announced August 2023.
-
Virtual vs. Reality: External Validation of COVID-19 Classifiers using XCAT Phantoms for Chest Computed Tomography
Authors:
Fakrul Islam Tushar,
Ehsan Abadi,
Saman Sotoudeh-Paima,
Rafael B. Fricks,
Maciej A. Mazurowski,
W. Paul Segars,
Ehsan Samei,
Joseph Y. Lo
Abstract:
Research studies of artificial intelligence models in medical imaging have been hampered by poor generalization. This problem has been especially concerning over the last year with numerous applications of deep learning for COVID-19 diagnosis. Virtual imaging trials (VITs) could provide a solution for objective evaluation of these models. In this work utilizing the VITs, we created the CVIT-COVID…
▽ More
Research studies of artificial intelligence models in medical imaging have been hampered by poor generalization. This problem has been especially concerning over the last year with numerous applications of deep learning for COVID-19 diagnosis. Virtual imaging trials (VITs) could provide a solution for objective evaluation of these models. In this work utilizing the VITs, we created the CVIT-COVID dataset including 180 virtually imaged computed tomography (CT) images from simulated COVID-19 and normal phantom models under different COVID-19 morphology and imaging properties. We evaluated the performance of an open-source, deep-learning model from the University of Waterloo trained with multi-institutional data and an in-house model trained with the open clinical dataset called MosMed. We further validated the model's performance against open clinical data of 305 CT images to understand virtual vs. real clinical data performance. The open-source model was published with nearly perfect performance on the original Waterloo dataset but showed a consistent performance drop in external testing on another clinical dataset (AUC=0.77) and our simulated CVIT-COVID dataset (AUC=0.55). The in-house model achieved an AUC of 0.87 while testing on the internal test set (MosMed test set). However, performance dropped to an AUC of 0.65 and 0.69 when evaluated on clinical and our simulated CVIT-COVID dataset. The VIT framework offered control over imaging conditions, allowing us to show there was no change in performance as CT exposure was changed from 28.5 to 57 mAs. The VIT framework also provided voxel-level ground truth, revealing that performance of in-house model was much higher at AUC=0.87 for diffuse COVID-19 infection size >2.65% lung volume versus AUC=0.52 for focal disease with <2.65% volume. The virtual imaging framework enabled these uniquely rigorous analyses of model performance.
△ Less
Submitted 6 March, 2022;
originally announced March 2022.
-
Multi-Scale Convolutional Neural Network for Automated AMD Classification using Retinal OCT Images
Authors:
Saman Sotoudeh-Paima,
Ata Jodeiri,
Fedra Hajizadeh,
Hamid Soltanian-Zadeh
Abstract:
Age-related macular degeneration (AMD) is the most common cause of blindness in developed countries, especially in people over 60 years of age. The workload of specialists and the healthcare system in this field has increased in recent years mainly due to the prevalence of population aging worldwide and the chronic nature of AMD. Recent developments in deep learning have provided a unique opportun…
▽ More
Age-related macular degeneration (AMD) is the most common cause of blindness in developed countries, especially in people over 60 years of age. The workload of specialists and the healthcare system in this field has increased in recent years mainly due to the prevalence of population aging worldwide and the chronic nature of AMD. Recent developments in deep learning have provided a unique opportunity to develop fully automated diagnosis frameworks. Considering the presence of AMD-related retinal pathologies in varying sizes in OCT images, our objective was to propose a multi-scale convolutional neural network (CNN) capable of distinguishing pathologies using receptive fields with various sizes. The multi-scale CNN was designed based on the feature pyramid network (FPN) structure and was used to diagnose normal and two common clinical characteristics of dry and wet AMD, namely drusen and choroidal neovascularization (CNV). The proposed method was evaluated on a national dataset gathered at Noor Eye Hospital (NEH) and the UCSD public dataset. Experimental results show the superior performance of our proposed multi-scale structure over several well-known OCT classification frameworks. This feature combination strategy has proved to be effective on all tested backbone models, with improvements ranging from 0.4% to 3.3%. In addition, gradual learning has proven to improve performance in two consecutive stages. In the first stage, the performance was boosted from 87.2%+-2.5% to 92.0%+-1.6% using pre-trained ImageNet weights. In the second stage, another performance boost from 92.0%+-1.6% to 93.4%+-1.4% was observed due to fine-tuning the previous model on the UCSD dataset. Lastly, generating heatmaps provided additional proof for the effectiveness of our multi-scale structure, enabling the detection of retinal pathologies appearing in different sizes.
△ Less
Submitted 4 February, 2022; v1 submitted 6 October, 2021;
originally announced October 2021.
