-
Digital staining in optical microscopy using deep learning -- a review
Authors:
Lucas Kreiss,
Shaowei Jiang,
Xiang Li,
Shiqi Xu,
Kevin C. Zhou,
Alexander Mühlberg,
Kyung Chul Lee,
Kanghyun Kim,
Amey Chaware,
Michael Ando,
Laura Barisoni,
Seung Ah Lee,
Guoan Zheng,
Kyle Lafata,
Oliver Friedrich,
Roarke Horstmeyer
Abstract:
Until recently, conventional biochemical staining had the undisputed status as well-established benchmark for most biomedical problems related to clinical diagnostics, fundamental research and biotechnology. Despite this role as gold-standard, staining protocols face several challenges, such as a need for extensive, manual processing of samples, substantial time delays, altered tissue homeostasis,…
▽ More
Until recently, conventional biochemical staining had the undisputed status as well-established benchmark for most biomedical problems related to clinical diagnostics, fundamental research and biotechnology. Despite this role as gold-standard, staining protocols face several challenges, such as a need for extensive, manual processing of samples, substantial time delays, altered tissue homeostasis, limited choice of contrast agents for a given sample, 2D imaging instead of 3D tomography and many more. Label-free optical technologies, on the other hand, do not rely on exogenous and artificial markers, by exploiting intrinsic optical contrast mechanisms, where the specificity is typically less obvious to the human observer. Over the past few years, digital staining has emerged as a promising concept to use modern deep learning for the translation from optical contrast to established biochemical contrast of actual stainings. In this review article, we provide an in-depth analysis of the current state-of-the-art in this field, suggest methods of good practice, identify pitfalls and challenges and postulate promising advances towards potential future implementations and applications.
△ Less
Submitted 14 March, 2023;
originally announced March 2023.
-
A Deep Learning Model with Radiomics Analysis Integration for Glioblastoma Post-Resection Survival Prediction
Authors:
Zongsheng Hu,
Zhenyu Yang,
Haozhao Zhang,
Eugene Vaios,
Kyle Lafata,
Fang-Fang Yin,
Chunhao Wang
Abstract:
Purpose: To develop a novel deep-learning model that integrates radiomics analysis in a multi-dimensional feature fusion workflow for glioblastoma (GBM) post-resection survival prediction. Methods: A cohort of 235 GBM patients with complete surgical resection was divided into short-term/long-term survival groups with 1-yr survival time threshold. Each patient received a pre-surgery multi-parametri…
▽ More
Purpose: To develop a novel deep-learning model that integrates radiomics analysis in a multi-dimensional feature fusion workflow for glioblastoma (GBM) post-resection survival prediction. Methods: A cohort of 235 GBM patients with complete surgical resection was divided into short-term/long-term survival groups with 1-yr survival time threshold. Each patient received a pre-surgery multi-parametric MRI exam, and three tumor subregions were segmented by neuroradiologists. The developed model comprises three data source branches: in the 1st radiomics branch, 456 radiomics features (RF) were from each patient; in the 2nd deep learning branch, an encoding neural network architecture was trained for survival group prediction using each single MR modality, and high-dimensional parameters of the last two network layers were extracted as deep features (DF). The extracted radiomics features and deep features were processed by a feature selection procedure to reduce dimension size of each feature space. In the 3rd branch, non-image-based patient-specific clinical features (PSCF) were collected. Finally, data sources from all three branches were fused as an integrated input for a supporting vector machine (SVM) execution for survival group prediction. Different strategies of model design, including 1) 2D/3D-based image analysis, and 2) different data source combinations in SVM input design, were investigated in comparison studies. Results: The model achieved 0.638 prediction accuracy when using PSCF only, which was higher than the results using RF or DF only in both 2D and 3D analysis. The joint use of RF/PSCF improved accuracy results to 0.681 in 3D analysis. The most accurate models in 2D/3D analysis reached the highest accuracy 0.745 with different combinations of RF/DF/ PSCF, and the corresponding ROC AUC results were 0.69(2D) and 0.71(3D), respectively.
△ Less
Submitted 11 March, 2022;
originally announced March 2022.
-
A Radiomics-Boosted Deep-Learning Model for COVID-19 and Non-COVID-19 Pneumonia Classification Using Chest X-ray Image
Authors:
Zongsheng Hu,
Zhenyu Yang,
Kyle J. Lafata,
Fang-Fang Yin,
Chunhao Wang
Abstract:
To develop a deep-learning model that integrates radiomics analysis for enhanced performance of COVID-19 and Non-COVID-19 pneumonia detection using chest X-ray image, two deep-learning models were trained based on a pre-trained VGG-16 architecture: in the 1st model, X-ray image was the sole input; in the 2nd model, X-ray image and 2 radiomic feature maps (RFM) selected by the saliency map analysis…
▽ More
To develop a deep-learning model that integrates radiomics analysis for enhanced performance of COVID-19 and Non-COVID-19 pneumonia detection using chest X-ray image, two deep-learning models were trained based on a pre-trained VGG-16 architecture: in the 1st model, X-ray image was the sole input; in the 2nd model, X-ray image and 2 radiomic feature maps (RFM) selected by the saliency map analysis of the 1st model were stacked as the input. Both models were developed using 812 chest X-ray images with 262/288/262 COVID-19/Non-COVID-19 pneumonia/healthy cases, and 649/163 cases were assigned as training-validation/independent test sets. In 1st model using X-ray as the sole input, the 1) sensitivity, 2) specificity, 3) accuracy, and 4) ROC Area-Under-the-Curve of COVID-19 vs Non-COVID-19 pneumonia detection were 1) 0.90$\pm$0.07 vs 0.78$\pm$0.09, 2) 0.94$\pm$0.04 vs 0.94$\pm$0.04, 3) 0.93$\pm$0.03 vs 0.89$\pm$0.03, and 4) 0.96$\pm$0.02 vs 0.92$\pm$0.04. In the 2nd model, two RFMs, Entropy and Short-Run-Emphasize, were selected with their highest cross-correlations with the saliency maps of the 1st model. The corresponding results demonstrated significant improvements (p<0.05) of COVID-19 vs Non-COVID-19 pneumonia detection: 1) 0.95$\pm$0.04 vs 0.85$\pm$0.04, 2) 0.97$\pm$0.02 vs 0.96$\pm$0.02, 3) 0.97$\pm$0.02 vs 0.93$\pm$0.02, and 4) 0.99$\pm$0.01 vs 0.97$\pm$0.02. The reduced variations suggested a superior robustness of 2nd model design.
△ Less
Submitted 23 October, 2021; v1 submitted 19 July, 2021;
originally announced July 2021.
-
Quantification of lung function on CT images based on pulmonary radiomic filtering
Authors:
Zhenyu Yang,
Kyle J Lafata,
Xinru Chen,
James Bowsher,
Yushi Chang,
Chunhao Wang,
Fang-Fang Yin
Abstract:
Purpose: To develop a radiomics filtering technique for characterizing spatial-encoded regional pulmonary ventilation information on lung CT. Methods: The lung volume was segmented on 46 CT images, and a 3D sliding window kernel was implemented across the lung volume to capture the spatial-encoded image information. Fifty-three radiomic features were extracted within the kernel, resulting in a 4th…
▽ More
Purpose: To develop a radiomics filtering technique for characterizing spatial-encoded regional pulmonary ventilation information on lung CT. Methods: The lung volume was segmented on 46 CT images, and a 3D sliding window kernel was implemented across the lung volume to capture the spatial-encoded image information. Fifty-three radiomic features were extracted within the kernel, resulting in a 4th order tensor object. As such, each voxel coordinate of the original lung was represented as a 53-dimensional feature vector, such that radiomic features could be viewed as feature maps within the lungs. To test the technique as a potential pulmonary ventilation biomarker, the radiomic feature maps were compared to paired functional images (Galligas PET or DTPA-SPECT) based on Spearman correlation (r) analysis. Results: The radiomic feature map GLRLM-based Run-Length Non-Uniformity and GLCOM-based Sum Average are found to be highly correlated with the functional imaging. The achieved r (median [range]) for the two features are 0.46 [0.05, 0.67] and 0.45 [0.21, 0.65] across 46 patients and 2 functional imaging modalities, respectively. Conclusions: The results provide evidence that local regions of sparsely encoded heterogeneous lung parenchyma on CT are associated with diminished radiotracer uptake and measured lung ventilation defects on PET/SPECT imaging. These findings demonstrate the potential of radiomics to serve as a complementary tool to the current lung quantification techniques and provide hypothesis-generating data for future studies.
△ Less
Submitted 24 June, 2022; v1 submitted 24 May, 2021;
originally announced May 2021.
-
Post-Radiotherapy PET Image Outcome Prediction by Deep Learning under Biological Model Guidance: A Feasibility Study of Oropharyngeal Cancer Application
Authors:
Hangjie Ji,
Kyle Lafata,
Yvonne Mowery,
David Brizel,
Andrea L. Bertozzi,
Fang-Fang Yin,
Chunhao Wang
Abstract:
This paper develops a method of biologically guided deep learning for post-radiation FDG-PET image outcome prediction based on pre-radiation images and radiotherapy dose information. Based on the classic reaction-diffusion mechanism, a novel biological model was proposed using a partial differential equation that incorporates spatial radiation dose distribution as a patient-specific treatment info…
▽ More
This paper develops a method of biologically guided deep learning for post-radiation FDG-PET image outcome prediction based on pre-radiation images and radiotherapy dose information. Based on the classic reaction-diffusion mechanism, a novel biological model was proposed using a partial differential equation that incorporates spatial radiation dose distribution as a patient-specific treatment information variable. A 7-layer encoder-decoder-based convolutional neural network (CNN) was designed and trained to learn the proposed biological model. As such, the model could generate post-radiation FDG-PET image outcome predictions with possible time-series transition from pre-radiotherapy image states to post-radiotherapy states. The proposed method was developed using 64 oropharyngeal patients with paired FDG-PET studies before and after 20Gy delivery (2Gy/daily fraction) by IMRT. In a two-branch deep learning execution, the proposed CNN learns specific terms in the biological model from paired FDG-PET images and spatial dose distribution as in one branch, and the biological model generates post-20Gy FDG-PET image prediction in the other branch. The proposed method successfully generated post-20Gy FDG-PET image outcome prediction with breakdown illustrations of biological model components. Time-series FDG-PET image predictions were generated to demonstrate the feasibility of disease response rendering. The developed biologically guided deep learning method achieved post-20Gy FDG-PET image outcome predictions in good agreement with ground-truth results. With break-down biological modeling components, the outcome image predictions could be used in adaptive radiotherapy decision-making to optimize personalized plans for the best outcome in the future.
△ Less
Submitted 22 May, 2021;
originally announced May 2021.
-
Data clustering based on Langevin annealing with a self-consistent potential
Authors:
Kyle Lafata,
Zhennan Zhou,
Jian-Guo Liu,
Fang-Fang Yin
Abstract:
This paper introduces a novel data clustering algorithm based on Langevin dynamics, where the associated potential is constructed directly from the data. To introduce a self-consistent potential, we adopt the potential model from the established Quantum Clustering method. The first step is to use a radial basis function to construct a density distribution from the data. A potential function is the…
▽ More
This paper introduces a novel data clustering algorithm based on Langevin dynamics, where the associated potential is constructed directly from the data. To introduce a self-consistent potential, we adopt the potential model from the established Quantum Clustering method. The first step is to use a radial basis function to construct a density distribution from the data. A potential function is then constructed such that this density distribution is the ground state solution to the time-independent Schrodinger equation. The second step is to use this potential function with the Langevin dynamics at sub-critical temperature to avoid ergodicity. The Langevin equations take a classical Gibbs distribution as the invariant measure, where the peaks of the distribution coincide with minima of the potential surface. The time dynamics of individual data points lead to different metastable states, which are interpreted as cluster centers. Clustering is therefore achieved when subsets of the data aggregate - as a result of the Langevin dynamics for a moderate period of time - in the neighborhood of a particular potential minimum. While the data points are pushed towards potential minima by the potential gradient, Brownian motion allows them to effectively tunnel through local potential barriers and escape saddle points into locations of the potential surface otherwise forbidden. The algorithm's feasibility is first established based on several illustrating examples and theoretical analyses, followed by a stricter evaluation using a standard benchmark dataset.
△ Less
Submitted 20 June, 2018;
originally announced June 2018.
