Generative artificial intelligence in ophthalmology: multimodal retinal images for the diagnosis of Alzheimer's disease with convolutional neural networks
Authors:
I. R. Slootweg,
M. Thach,
K. R. Curro-Tafili,
F. D. Verbraak,
F. H. Bouwman,
Y. A. L. Pijnenburg,
J. F. Boer,
J. H. P. de Kwisthout,
L. Bagheriye,
P. J. González
Abstract:
Background/Aim. This study aims to predict Amyloid Positron Emission Tomography (AmyloidPET) status with multimodal retinal imaging and convolutional neural networks (CNNs) and to improve the performance through pretraining with synthetic data. Methods. Fundus autofluorescence, optical coherence tomography (OCT), and OCT angiography images from 328 eyes of 59 AmyloidPET positive subjects and 108 A…
▽ More
Background/Aim. This study aims to predict Amyloid Positron Emission Tomography (AmyloidPET) status with multimodal retinal imaging and convolutional neural networks (CNNs) and to improve the performance through pretraining with synthetic data. Methods. Fundus autofluorescence, optical coherence tomography (OCT), and OCT angiography images from 328 eyes of 59 AmyloidPET positive subjects and 108 AmyloidPET negative subjects were used for classification. Denoising Diffusion Probabilistic Models (DDPMs) were trained to generate synthetic images and unimodal CNNs were pretrained on synthetic data and finetuned on real data or trained solely on real data. Multimodal classifiers were developed to combine predictions of the four unimodal CNNs with patient metadata. Class activation maps of the unimodal classifiers provided insight into the network's attention to inputs. Results. DDPMs generated diverse, realistic images without memorization. Pretraining unimodal CNNs with synthetic data improved AUPR at most from 0.350 to 0.579. Integration of metadata in multimodal CNNs improved AUPR from 0.486 to 0.634, which was the best overall best classifier. Class activation maps highlighted relevant retinal regions which correlated with AD. Conclusion. Our method for generating and leveraging synthetic data has the potential to improve AmyloidPET prediction from multimodal retinal imaging. A DDPM can generate realistic and unique multimodal synthetic retinal images. Our best performing unimodal and multimodal classifiers were not pretrained on synthetic data, however pretraining with synthetic data slightly improved classification performance for two out of the four modalities.
△ Less
Submitted 26 June, 2024;
originally announced June 2024.
Endo-microscopy beyond the Abbe and Nyquist limits
Authors:
Lyubov V. Amitonova,
Johannes F. de Boer
Abstract:
For several centuries, far-field optical microscopy has remained a key instrument in many scientific disciplines including physical, chemical and biomedical research. Nonetheless, far-field imaging has many limitations: the spatial resolution is controlled by the diffraction of light and the imaging speed follows Nyquist-Shannon sampling theorem. The recent development of super-resolution techniqu…
▽ More
For several centuries, far-field optical microscopy has remained a key instrument in many scientific disciplines including physical, chemical and biomedical research. Nonetheless, far-field imaging has many limitations: the spatial resolution is controlled by the diffraction of light and the imaging speed follows Nyquist-Shannon sampling theorem. The recent development of super-resolution techniques has pushed the limits of spatial resolution. However, these methods typically require complicated setups, long acquisition time and are still not applicable for deep-tissue bioimaging. Here we report imaging through an ultra-thin fiber probe with a spatial resolution beyond the Abbe limit and a temporal resolution beyond the Nyquist limit simultaneously in a simple and compact setup. We use the random nature of mode coupling in a multimode fiber, the sparsity constraint and compressive sensing reconstruction. The new approach of super-resolution endo-microscopy does not use any specific properties of the fluorescent label such as depletion or stochastic activation of the molecular fluorescent state and therefore could be used for label-free imaging. We demonstrate a spatial resolution more than 2 times better than the diffraction limit and an imaging speed 20 times faster than the Nyquist limit. The proposed approach could significantly expand the realm of the application of nanoscopy for bioimaging.
△ Less
Submitted 24 January, 2020;
originally announced January 2020.
