-
Super resolution of histopathological frozen sections via deep learning preserving tissue structure
Authors:
Elad Yoshai,
Gil Goldinger,
Miki Haifler,
Natan T. Shaked
Abstract:
Histopathology plays a pivotal role in medical diagnostics. In contrast to preparing permanent sections for histopathology, a time-consuming process, preparing frozen sections is significantly faster and can be performed during surgery, where the sample scanning time should be optimized. Super-resolution techniques allow imaging the sample in lower magnification and sparing scanning time. In this…
▽ More
Histopathology plays a pivotal role in medical diagnostics. In contrast to preparing permanent sections for histopathology, a time-consuming process, preparing frozen sections is significantly faster and can be performed during surgery, where the sample scanning time should be optimized. Super-resolution techniques allow imaging the sample in lower magnification and sparing scanning time. In this paper, we present a new approach to super resolution for histopathological frozen sections, with focus on achieving better distortion measures, rather than pursuing photorealistic images that may compromise critical diagnostic information. Our deep-learning architecture focuses on learning the error between interpolated images and real images, thereby it generates high-resolution images while preserving critical image details, reducing the risk of diagnostic misinterpretation. This is done by leveraging the loss functions in the frequency domain, assigning higher weights to the reconstruction of complex, high-frequency components. In comparison to existing methods, we obtained significant improvements in terms of Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR), as well as indicated details that lost in the low-resolution frozen-section images, affecting the pathologist's clinical decisions. Our approach has a great potential in providing more-rapid frozen-section imaging, with less scanning, while preserving the high resolution in the imaged sample.
△ Less
Submitted 17 October, 2023;
originally announced October 2023.
-
Joint Privacy Enhancement and Quantization in Federated Learning
Authors:
Natalie Lang,
Elad Sofer,
Tomer Shaked,
Nir Shlezinger
Abstract:
Federated learning (FL) is an emerging paradigm for training machine learning models using possibly private data available at edge devices. The distributed operation of FL gives rise to challenges that are not encountered in centralized machine learning, including the need to preserve the privacy of the local datasets, and the communication load due to the repeated exchange of updated models. Thes…
▽ More
Federated learning (FL) is an emerging paradigm for training machine learning models using possibly private data available at edge devices. The distributed operation of FL gives rise to challenges that are not encountered in centralized machine learning, including the need to preserve the privacy of the local datasets, and the communication load due to the repeated exchange of updated models. These challenges are often tackled individually via techniques that induce some distortion on the updated models, e.g., local differential privacy (LDP) mechanisms and lossy compression. In this work we propose a method coined joint privacy enhancement and quantization (JoPEQ), which jointly implements lossy compression and privacy enhancement in FL settings. In particular, JoPEQ utilizes vector quantization based on random lattice, a universal compression technique whose byproduct distortion is statistically equivalent to additive noise. This distortion is leveraged to enhance privacy by augmenting the model updates with dedicated multivariate privacy preserving noise. We show that JoPEQ simultaneously quantizes data according to a required bit-rate while holding a desired privacy level, without notably affecting the utility of the learned model. This is shown via analytical LDP guarantees, distortion and convergence bounds derivation, and numerical studies. Finally, we empirically assert that JoPEQ demolishes common attacks known to exploit privacy leakage.
△ Less
Submitted 23 August, 2022;
originally announced August 2022.
-
Real-time stain-free classification of cancer cells and blood cells using interferometric phase microscopy and machine learning
Authors:
Noga Nissim,
Matan Dudaie,
Itay Barnea.,
Natan T. Shaked
Abstract:
We present a method for a real time visualization and automatic processing for detection and classification of untouched cancer cells in blood during stain free imaging flow cytometry using digital holographic microscopy and machine learning in throughput of 15 cells per second. As a preliminary model for circulating tumor cells in blood, we automatically classified primary and metastatic colon ca…
▽ More
We present a method for a real time visualization and automatic processing for detection and classification of untouched cancer cells in blood during stain free imaging flow cytometry using digital holographic microscopy and machine learning in throughput of 15 cells per second. As a preliminary model for circulating tumor cells in blood, we automatically classified primary and metastatic colon cancer cells, where the two types of cancer cells were isolated from the same individual, as well as four types of blood cells. We used low-coherence off-axis interferometric phase microscopy and a microfluidic channel to quantitatively image cells during flow. The acquired images were processed and classified based on their morphology and quantitative phase features during the cell flow. We achieved high accuracy of 92.56 percent for distinguishing between the cells, paving the way for future automatic label-free cancer cell classification in blood samples.
△ Less
Submitted 12 June, 2021;
originally announced June 2021.
-
High-resolution 4D acquisition of freely swimming human sperm cells without staining
Authors:
Gili Dardikman-Yoffe,
Simcha K. Mirsky,
Itay Barnea,
Natan T. Shaked
Abstract:
We present a new acquisition method that enables high-resolution, fine-detail full reconstruction of the three-dimensional movement and structure of individual human sperm cells swimming freely. We achieve both retrieval of the three-dimensional refractive-index profile of the sperm head, revealing its fine internal organelles and time-varying orientation, and the detailed four-dimensional localiz…
▽ More
We present a new acquisition method that enables high-resolution, fine-detail full reconstruction of the three-dimensional movement and structure of individual human sperm cells swimming freely. We achieve both retrieval of the three-dimensional refractive-index profile of the sperm head, revealing its fine internal organelles and time-varying orientation, and the detailed four-dimensional localization of the thin, highly-dynamic flagellum of the sperm cell. Live human sperm cells were acquired during free swim using a high-speed off-axis holographic system that does not require any moving elements or cell staining. The reconstruction is based solely on the natural movement of the sperm cell and a novel set of algorithms, enabling the detailed four-dimensional recovery. Using this refractive-index imaging approach, we believe we have detected an area in the cell that is attributed to the centriole. This method has great potential for both biological assays and clinical use of intact sperm cells.
△ Less
Submitted 22 February, 2020;
originally announced February 2020.
-
Integral refractive index imaging of flowing cell nuclei using quantitative phase microscopy combined with fluorescence microscopy
Authors:
Gili Dardikman,
Yoav N. Nygate,
Itay Barnea,
Nir A. Turko,
Gyanendra Singh,
Barham Javidi,
Natan T. Shaked
Abstract:
We suggest a new multimodal imaging technique for quantitatively measuring the integral (thickness-average) refractive index of the nuclei of live biological cells in suspension. For this aim, we combined quantitative phase microscopy with simultaneous 2-D fluorescence microscopy. We used 2-D fluorescence microscopy to localize the nucleus inside the quantitative phase map of the cell, as well as…
▽ More
We suggest a new multimodal imaging technique for quantitatively measuring the integral (thickness-average) refractive index of the nuclei of live biological cells in suspension. For this aim, we combined quantitative phase microscopy with simultaneous 2-D fluorescence microscopy. We used 2-D fluorescence microscopy to localize the nucleus inside the quantitative phase map of the cell, as well as for measuring the nucleus radii. As verified offline by both 3-D confocal fluorescence microscopy and by 2-D fluorescence microscopy while rotating the cells during flow, the nucleus of cells in suspension that are not during division can be assumed to be an ellipsoid. The entire shape of a cell in suspension can be assumed to be a sphere. Then, the cell and nucleus 3-D shapes can be evaluated based on their in-plain radii available from the 2-D phase and fluorescent measurements, respectively. Finally, the nucleus integral refractive index profile is calculated. We demonstrate the new technique on cancer cells, obtaining nucleus refractive index values that are lower than those of the cytoplasm, coinciding with recent findings. We believe that the proposed technique has the potential to be used for flow cytometry, where full 3-D refractive index tomography is too slow to be implemented during flow.
△ Less
Submitted 3 September, 2019;
originally announced October 2019.
-
HoloStain: Holographic virtual staining of individual biological cells
Authors:
Yoav N. Nygate,
Mattan Levi,
Simcha K. Mirsky,
Nir A. Turko,
Moran Rubin,
Itay Barnea,
Gili Dardikman-Yoffe,
Alon Shalev,
Natan T. Shaked
Abstract:
Many medical and biological protocols for analyzing individual biological cells involve morphological evaluation based on cell staining, designed to enhance imaging contrast and enable clinicians and biologists to differentiate between various cell organelles. However, cell staining is not always allowed in certain medical procedures. In other cases, staining may be time consuming or expensive to…
▽ More
Many medical and biological protocols for analyzing individual biological cells involve morphological evaluation based on cell staining, designed to enhance imaging contrast and enable clinicians and biologists to differentiate between various cell organelles. However, cell staining is not always allowed in certain medical procedures. In other cases, staining may be time consuming or expensive to implement. Here, we present a new deep-learning approach, called HoloStain, which converts images of isolated biological cells acquired without staining by holographic microscopy to their virtually stained images. We demonstrate this approach for human sperm cells, as there is a well-established protocol and global standardization for characterizing the morphology of stained human sperm cells for fertility evaluation, but, on the other hand, staining might be cytotoxic and thus is not allowed during human in vitro fertilization (IVF). We use deep convolutional Generative Adversarial Networks (DCGANs) with training that is based on both the quantitative phase images and two gradient phase images, all extracted from the digital holograms of the stain-free cells, with the ground truth of bright-field images of the same cells that subsequently underwent chemical staining. To validate the quality of our virtual staining approach, an experienced embryologist analyzed the unstained cells, the virtually stained cells, and the chemically stained sperm cells several times in a blinded and randomized manner. We obtained a 5-fold recall (sensitivity) improvement in the analysis results. With the introduction of simple holographic imaging methods in clinical settings, the proposed method has a great potential to become a common practice in human IVF procedures, as well as to significantly simplify and facilitate other cell analyses and techniques such as imaging flow cytometry.
△ Less
Submitted 25 September, 2019;
originally announced September 2019.
-
Four dimensional phase unwrap** of dynamic objects in digital holography
Authors:
Gili Dardikman,
Gyanendra Singh,
Natan T. Shaked
Abstract:
We present a new four-dimensional phase unwrap** approach for time-lapse quantitative phase microscopy, which allows reconstruction of optically thick objects that are optically thin in a certain temporal point and angular view. We thus use all four dimensions of the dynamic quantitative phase profile acquired, including the angular dimension and the temporal dimension, in addition to the x-y di…
▽ More
We present a new four-dimensional phase unwrap** approach for time-lapse quantitative phase microscopy, which allows reconstruction of optically thick objects that are optically thin in a certain temporal point and angular view. We thus use all four dimensions of the dynamic quantitative phase profile acquired, including the angular dimension and the temporal dimension, in addition to the x-y dimensions. We first demonstrate the capabilities of this algorithm on simulative data, enabling the quantification of the reconstruction quality relative to both the ground truth and existing unwrap** approaches. Then, we demonstrate the applicability of the proposed four-dimensional phase unwrap** algorithm by experimentally capturing a dual-angular dynamic off-axis hologram with simultaneous recording of two angular views, using multiplexing of two off-axis holograms into a single multiplexed hologram.
△ Less
Submitted 3 September, 2019;
originally announced September 2019.
-
Review on methods of solving the refractive index-thickness coupling problem in digital holographic microscopy of biological cells
Authors:
Gili Dardikman,
Natan T. Shaked
Abstract:
Digital holographic microscopy is a thriving imaging modality that attracted considerable research interest in quantitative biological cell imaging due to its ability to not only create excellent label-free contrast, but also supply valuable physical information regarding the density and dimensions of the sample with nanometer-scale axial sensitivity. This technique records the interference patter…
▽ More
Digital holographic microscopy is a thriving imaging modality that attracted considerable research interest in quantitative biological cell imaging due to its ability to not only create excellent label-free contrast, but also supply valuable physical information regarding the density and dimensions of the sample with nanometer-scale axial sensitivity. This technique records the interference pattern between a sample beam and a reference beam, and by digitally processing it, one can reconstruct the optical path delay between these beams. Per each spatial point, the optical path delay map is proportional to the product of the sample physical thickness and the integral refractive index of the sample. Since the refractive index of the cell indicates its contents without the need for labeling, it is highly beneficial to decouple cell physical thickness from its refractive index profile. This manuscript reviews various approaches of extracting the refractive index from digital holographic microscopy measurements of cells. As soon as the refractive index of the cell is available, it can be used for either biological assays or medical diagnosis, as reviewed in this manuscript.
△ Less
Submitted 2 April, 2019;
originally announced April 2019.
-
Simultaneous off-axis multiplexed holography and regular fluorescence microscopy of biological cells
Authors:
Yoav N. Nygate,
Gyanendra Singh,
Itay Barnea,
Natan T. Shaked
Abstract:
We present a new technique for obtaining simultaneous multimodal quantitative phase and fluorescence microscopy of biological cells, providing both quantitative phase imaging and molecular specificity using a single camera. Our system is based on an interferometric multiplexing module, externally positioned at the exit of an optical microscope. In contrast to previous approaches, the presented tec…
▽ More
We present a new technique for obtaining simultaneous multimodal quantitative phase and fluorescence microscopy of biological cells, providing both quantitative phase imaging and molecular specificity using a single camera. Our system is based on an interferometric multiplexing module, externally positioned at the exit of an optical microscope. In contrast to previous approaches, the presented technique allows conventional fluorescence imaging, rather than interferometric off-axis fluorescence imaging. We demonstrate the presented technique for imaging fluorescent beads and live biological cells.
△ Less
Submitted 2 April, 2019;
originally announced April 2019.
-
Six-pack off-axis holography
Authors:
Moran Rubin,
Gili Dardikman,
Simcha K. Mirsky,
Nir A. Turko,
Natan T. Shaked
Abstract:
We present a new holographic concept, named six-pack holography (6PH), in which we compress six off-axis holograms into a single multiplexed off-axis hologram without loss of magnification or resolution. The multiplexed hologram contains straight off-axis fringes with six different orientations, and can be generated optically or digitally. We show that since the six different complex wavefronts do…
▽ More
We present a new holographic concept, named six-pack holography (6PH), in which we compress six off-axis holograms into a single multiplexed off-axis hologram without loss of magnification or resolution. The multiplexed hologram contains straight off-axis fringes with six different orientations, and can be generated optically or digitally. We show that since the six different complex wavefronts do not overlap in the spatial frequency domain, they can be fully reconstructed. 6PH allows more than 50% improvement in the spatial bandwidth consumption when compared to the best multiplexing method proposed so far. We expect the 6PH concept to be useful for a variety of applications, such as field-of-view multiplexing, wavelength multiplexing, temporal multiplexing, multiplexing for super-resolution imaging, and others.
△ Less
Submitted 2 April, 2019;
originally announced April 2019.
-
Simultanous two-wavelength phase unwrap** using external module for multiplexing off-axis holography
Authors:
Nir A. Turko,
Natan T. Shaked
Abstract:
We present a dual-wavelength external holographic microscopy module for quantitative phase imaging of 3D structures with extended thickness range. This is done by simultaneous acquisition of two off-axis interferograms, each of which at a different wavelength, and generation of a synthetic wavelength, which is larger than the sample optical thickness, allowing two-wavelength unwrap**. The simult…
▽ More
We present a dual-wavelength external holographic microscopy module for quantitative phase imaging of 3D structures with extended thickness range. This is done by simultaneous acquisition of two off-axis interferograms, each of which at a different wavelength, and generation of a synthetic wavelength, which is larger than the sample optical thickness, allowing two-wavelength unwrap**. The simultaneous acquisition is carried out by using optical multiplexing of the two interferograms onto the camera, where each of them has orthogonal off-axis interference fringe direction in relation to the other one. We used the system to quantitatively image a 7.96 um step target and 30.5 um circular copper pillars.
△ Less
Submitted 2 April, 2019;
originally announced April 2019.
-
Quantitative phase microscopy spatial signatures of cancer cells
Authors:
Darina Roitshtain,
Lauren Wolbromsky,
Evgeny Bal,
Hayit Greenspan,
Lisa L. Satterwhite,
Natan T. Shaked
Abstract:
We present cytometric classification of live healthy and cancer cells by using the spatial morphological and textural information found in the label-free quantitative phase images of the cells. We compare both healthy cells to primary tumor cell and primary tumor cells to metastatic cancer cells, where tumor biopsies and normal tissues were isolated from the same individuals. To mimic analysis of…
▽ More
We present cytometric classification of live healthy and cancer cells by using the spatial morphological and textural information found in the label-free quantitative phase images of the cells. We compare both healthy cells to primary tumor cell and primary tumor cells to metastatic cancer cells, where tumor biopsies and normal tissues were isolated from the same individuals. To mimic analysis of liquid biopsies by flow cytometry, the cells were imaged while unattached to the substrate. We used low-coherence off-axis interferometric phase microscopy setup, which allows a single-exposure acquisition mode, and thus is suitable for quantitative imaging of dynamic cells during flow. After acquisition, the optical path delay maps of the cells were extracted, and used to calculate 15 parameters derived from cellular 3-D morphology and texture. Upon analyzing tens of cells in each group, we found high statistical significance in the difference between the groups in most of the parameters calculated, with the same trends for all statistically significant parameters. Furthermore, a specially designed machine learning algorithm, implemented on the phase map extracted features, classified the correct cell type (healthy/cancer/metastatic) with 81%-93% sensitivity and 81%-99% specificity. The quantitative phase imaging approach for liquid biopsies presented in this paper could be the basis for advanced techniques of staging freshly isolated live cancer cells in imaging flow cytometers.
△ Less
Submitted 2 April, 2019;
originally announced April 2019.
-
Is multiplexed off-axis holography for quantitative phase imaging more spatial bandwidth-efficient than on-axis holography?
Authors:
Gili Dardikman,
Natan T. Shaked
Abstract:
Digital holographic microcopy is a thriving imaging modality that attracts considerable research interest due to its ability to not only create excellent label-free contrast, but also supply valuable physical information regarding the density and dimensions of the sample with nanometer-scale axial sensitivity. Three basic holographic recording geometries currently exist, including on-axis, off-axi…
▽ More
Digital holographic microcopy is a thriving imaging modality that attracts considerable research interest due to its ability to not only create excellent label-free contrast, but also supply valuable physical information regarding the density and dimensions of the sample with nanometer-scale axial sensitivity. Three basic holographic recording geometries currently exist, including on-axis, off-axis and slightly off-axis holography, each of them enabling a variety of architectures in terms of bandwidth use and compression capacity. Specifically, off-axis holography and slightly off-axis holography allow spatial hologram multiplexing, enabling compressing more information into the same digital hologram. In this paper, we define an efficiency score used to analyze the various possible architectures, and compare the signal-to-noise ratio and mean squared error obtained using each of them, determining the optimal holographic method.
△ Less
Submitted 28 March, 2019;
originally announced March 2019.
-
Single-exposure full-field multi-depth imaging using low-coherence holographic multiplexing
Authors:
Lauren Wolbromsky,
Nir A. Turko,
Natan T. Shaked
Abstract:
We present a new interferometric imaging approach that allows for multiple-depth imaging in a single acquisition, using off-axis low-coherence holographic multiplexing. This technique enables sectioned imaging of multiple slices within a thick sample, in a single image acquisition. Each slice has a distinct off-axis interference fringe orientation indicative of its axial location, and the camera a…
▽ More
We present a new interferometric imaging approach that allows for multiple-depth imaging in a single acquisition, using off-axis low-coherence holographic multiplexing. This technique enables sectioned imaging of multiple slices within a thick sample, in a single image acquisition. Each slice has a distinct off-axis interference fringe orientation indicative of its axial location, and the camera acquires the multiplexed hologram containing the different slices at once. We demonstrate the proposed technique for amplitude and phase imaging of optically thick samples.
△ Less
Submitted 28 March, 2019;
originally announced March 2019.
-
Simultaneous three-wavelength unwrap** using external digital holographic multiplexing module
Authors:
Nir A. Turko,
Pinkie Jacob Eravuchira,
Itay Barnea,
Natan T. Shaked
Abstract:
We present an external interferometric setup that is able to simultaneously acquire three wavelengths of the same sample instance without scanning or multiple exposures. This setup projects onto the monochrome digital camera three off-axis holograms with rotated fringe orientations, each from a different wavelength channel, without overlap in the spatial-frequency domain, and thus allows the full…
▽ More
We present an external interferometric setup that is able to simultaneously acquire three wavelengths of the same sample instance without scanning or multiple exposures. This setup projects onto the monochrome digital camera three off-axis holograms with rotated fringe orientations, each from a different wavelength channel, without overlap in the spatial-frequency domain, and thus allows the full reconstruction of the three complex wavefronts from the three wavelength channels. We use this new setup for three-wavelength phase unwrap**, allowing phase imaging of thicker objects than possible with a single wavelength, but without the increased level of noise. We demonstrate the proposed technique for micro-channel profiling and label-free cell imaging.
△ Less
Submitted 28 March, 2019;
originally announced March 2019.
-
TOP-GAN: Label-Free Cancer Cell Classification Using Deep Learning with a Small Training Set
Authors:
Moran Rubin,
Omer Stein,
Nir A. Turko,
Yoav Nygate,
Darina Roitshtain,
Lidor Karako,
Itay Barnea,
Raja Giryes,
Natan T. Shaked
Abstract:
We propose a new deep learning approach for medical imaging that copes with the problem of a small training set, the main bottleneck of deep learning, and apply it for classification of healthy and cancer cells acquired by quantitative phase imaging. The proposed method, called transferring of pre-trained generative adversarial network (TOP-GAN), is a hybridization between transfer learning and ge…
▽ More
We propose a new deep learning approach for medical imaging that copes with the problem of a small training set, the main bottleneck of deep learning, and apply it for classification of healthy and cancer cells acquired by quantitative phase imaging. The proposed method, called transferring of pre-trained generative adversarial network (TOP-GAN), is a hybridization between transfer learning and generative adversarial networks (GANs). Healthy cells and cancer cells of different metastatic potential have been imaged by low-coherence off-axis holography. After the acquisition, the optical path delay maps of the cells have been extracted and directly used as an input to the deep networks. In order to cope with the small number of classified images, we have used GANs to train a large number of unclassified images from another cell type (sperm cells). After this preliminary training, and after transforming the last layer of the network with new ones, we have designed an automatic classifier for the correct cell type (healthy/primary cancer/metastatic cancer) with 90-99% accuracy, although small training sets of down to several images have been used. These results are better in comparison to other classic methods that aim at co** with the same problem of a small training set. We believe that our approach makes the combination of holographic microscopy and deep learning networks more accessible to the medical field by enabling a rapid, automatic and accurate classification in stain-free imaging flow cytometry. Furthermore, our approach is expected to be applicable to many other medical image classification tasks, suffering from a small training set.
△ Less
Submitted 17 December, 2018;
originally announced December 2018.
-
Wide & Deep Learning for Recommender Systems
Authors:
Heng-Tze Cheng,
Levent Koc,
Jeremiah Harmsen,
Tal Shaked,
Tushar Chandra,
Hrishi Aradhye,
Glen Anderson,
Greg Corrado,
Wei Chai,
Mustafa Ispir,
Rohan Anil,
Zakaria Haque,
Lichan Hong,
Vihan Jain,
Xiaobing Liu,
Hemal Shah
Abstract:
Generalized linear models with nonlinear feature transformations are widely used for large-scale regression and classification problems with sparse inputs. Memorization of feature interactions through a wide set of cross-product feature transformations are effective and interpretable, while generalization requires more feature engineering effort. With less feature engineering, deep neural networks…
▽ More
Generalized linear models with nonlinear feature transformations are widely used for large-scale regression and classification problems with sparse inputs. Memorization of feature interactions through a wide set of cross-product feature transformations are effective and interpretable, while generalization requires more feature engineering effort. With less feature engineering, deep neural networks can generalize better to unseen feature combinations through low-dimensional dense embeddings learned for the sparse features. However, deep neural networks with embeddings can over-generalize and recommend less relevant items when the user-item interactions are sparse and high-rank. In this paper, we present Wide & Deep learning---jointly trained wide linear models and deep neural networks---to combine the benefits of memorization and generalization for recommender systems. We productionized and evaluated the system on Google Play, a commercial mobile app store with over one billion active users and over one million apps. Online experiment results show that Wide & Deep significantly increased app acquisitions compared with wide-only and deep-only models. We have also open-sourced our implementation in TensorFlow.
△ Less
Submitted 24 June, 2016;
originally announced June 2016.