-
Memory-efficient Learning for High-Dimensional MRI Reconstruction
Authors:
Ke Wang,
Michael Kellman,
Christopher M. Sandino,
Kevin Zhang,
Shreyas S. Vasanawala,
Jonathan I. Tamir,
Stella X. Yu,
Michael Lustig
Abstract:
Deep learning (DL) based unrolled reconstructions have shown state-of-the-art performance for under-sampled magnetic resonance imaging (MRI). Similar to compressed sensing, DL can leverage high-dimensional data (e.g. 3D, 2D+time, 3D+time) to further improve performance. However, network size and depth are currently limited by the GPU memory required for backpropagation. Here we use a memory-effici…
▽ More
Deep learning (DL) based unrolled reconstructions have shown state-of-the-art performance for under-sampled magnetic resonance imaging (MRI). Similar to compressed sensing, DL can leverage high-dimensional data (e.g. 3D, 2D+time, 3D+time) to further improve performance. However, network size and depth are currently limited by the GPU memory required for backpropagation. Here we use a memory-efficient learning (MEL) framework which favorably trades off storage with a manageable increase in computation during training. Using MEL with multi-dimensional data, we demonstrate improved image reconstruction performance for in-vivo 3D MRI and 2D+time cardiac cine MRI. MEL uses far less GPU memory while marginally increasing the training time, which enables new applications of DL to high-dimensional MRI.
△ Less
Submitted 5 March, 2021;
originally announced March 2021.
-
How to do Physics-based Learning
Authors:
Michael Kellman,
Michael Lustig,
Laura Waller
Abstract:
The goal of this tutorial is to explain step-by-step how to implement physics-based learning for the rapid prototy** of a computational imaging system. We provide a basic overview of physics-based learning, the construction of a physics-based network, and its reduction to practice. Specifically, we advocate exploiting the auto-differentiation functionality twice, once to build a physics-based ne…
▽ More
The goal of this tutorial is to explain step-by-step how to implement physics-based learning for the rapid prototy** of a computational imaging system. We provide a basic overview of physics-based learning, the construction of a physics-based network, and its reduction to practice. Specifically, we advocate exploiting the auto-differentiation functionality twice, once to build a physics-based network and again to perform physics-based learning. Thus, the user need only implement the forward model process for their system, speeding up prototy** time. We provide an open-source Pytorch implementation of a physics-based network and training procedure for a generic sparse recovery problem
△ Less
Submitted 28 May, 2020; v1 submitted 27 May, 2020;
originally announced May 2020.
-
Memory-efficient Learning for Large-scale Computational Imaging
Authors:
Michael Kellman,
Kevin Zhang,
Jon Tamir,
Emrah Bostan,
Michael Lustig,
Laura Waller
Abstract:
Critical aspects of computational imaging systems, such as experimental design and image priors, can be optimized through deep networks formed by the unrolled iterations of classical model-based reconstructions (termed physics-based networks). However, for real-world large-scale inverse problems, computing gradients via backpropagation is infeasible due to memory limitations of graphics processing…
▽ More
Critical aspects of computational imaging systems, such as experimental design and image priors, can be optimized through deep networks formed by the unrolled iterations of classical model-based reconstructions (termed physics-based networks). However, for real-world large-scale inverse problems, computing gradients via backpropagation is infeasible due to memory limitations of graphics processing units. In this work, we propose a memory-efficient learning procedure that exploits the reversibility of the network's layers to enable data-driven design for large-scale computational imaging systems. We demonstrate our method on a small-scale compressed sensing example, as well as two large-scale real-world systems: multi-channel magnetic resonance imaging and super-resolution optical microscopy.
△ Less
Submitted 11 March, 2020;
originally announced March 2020.
-
Memory-efficient Learning for Large-scale Computational Imaging -- NeurIPS deep inverse workshop
Authors:
Michael Kellman,
Jon Tamir,
Emrah Boston,
Michael Lustig,
Laura Waller
Abstract:
Computational imaging systems jointly design computation and hardware to retrieve information which is not traditionally accessible with standard imaging systems. Recently, critical aspects such as experimental design and image priors are optimized through deep neural networks formed by the unrolled iterations of classical physics-based reconstructions (termed physics-based networks). However, for…
▽ More
Computational imaging systems jointly design computation and hardware to retrieve information which is not traditionally accessible with standard imaging systems. Recently, critical aspects such as experimental design and image priors are optimized through deep neural networks formed by the unrolled iterations of classical physics-based reconstructions (termed physics-based networks). However, for real-world large-scale systems, computing gradients via backpropagation restricts learning due to memory limitations of graphical processing units. In this work, we propose a memory-efficient learning procedure that exploits the reversibility of the network's layers to enable data-driven design for large-scale computational imaging. We demonstrate our methods practicality on two large-scale systems: super-resolution optical microscopy and multi-channel magnetic resonance imaging.
△ Less
Submitted 11 March, 2020; v1 submitted 10 December, 2019;
originally announced December 2019.
-
Data-Driven Design for Fourier Ptychographic Microscopy
Authors:
Michael Kellman,
Emrah Bostan,
Michael Chen,
Laura Waller
Abstract:
Fourier Ptychographic Microscopy (FPM) is a computational imaging method that is able to super-resolve features beyond the diffraction-limit set by the objective lens of a traditional microscope. This is accomplished by using synthetic aperture and phase retrieval algorithms to combine many measurements captured by an LED array microscope with programmable source patterns. FPM provides simultaneou…
▽ More
Fourier Ptychographic Microscopy (FPM) is a computational imaging method that is able to super-resolve features beyond the diffraction-limit set by the objective lens of a traditional microscope. This is accomplished by using synthetic aperture and phase retrieval algorithms to combine many measurements captured by an LED array microscope with programmable source patterns. FPM provides simultaneous large field-of-view and high resolution imaging, but at the cost of reduced temporal resolution, thereby limiting live cell applications. In this work, we learn LED source pattern designs that compress the many required measurements into only a few, with negligible loss in reconstruction quality or resolution. This is accomplished by recasting the super-resolution reconstruction as a Physics-based Neural Network and learning the experimental design to optimize the network's overall performance. Specifically, we learn LED patterns for different applications (e.g. amplitude contrast and quantitative phase imaging) and show that the designs we learn through simulation generalize well in the experimental setting. Further, we discuss a context-specific loss function, practical memory limitations, and interpretability of our learned designs.
△ Less
Submitted 8 April, 2019;
originally announced April 2019.
-
Physics-based Learned Design: Optimized Coded-Illumination for Quantitative Phase Imaging
Authors:
Michael R. Kellman,
Emrah Bostan,
Nicole Repina,
Laura Waller
Abstract:
Coded-illumination can enable quantitative phase microscopy of transparent samples with minimal hardware requirements. Intensity images are captured with different source patterns and a non-linear phase retrieval optimization reconstructs the image. The non-linear nature of the processing makes optimizing the illumination pattern designs complicated. Traditional techniques for experimental design…
▽ More
Coded-illumination can enable quantitative phase microscopy of transparent samples with minimal hardware requirements. Intensity images are captured with different source patterns and a non-linear phase retrieval optimization reconstructs the image. The non-linear nature of the processing makes optimizing the illumination pattern designs complicated. Traditional techniques for experimental design (e.g. condition number optimization, spectral analysis) consider only linear measurement formation models and linear reconstructions. Deep neural networks (DNNs) can efficiently represent the non-linear process and can be optimized over via training in an end-to-end framework. However, DNNs typically require a large amount of training examples and parameters to properly learn the phase retrieval process, without making use of the known physical models. Here, we aim to use both our knowledge of the physics and the power of machine learning together. We develop a new data-driven approach to optimizing coded-illumination patterns for a LED array microscope for a given phase reconstruction algorithm. Our method incorporates both the physics of the measurement scheme and the non-linearity of the reconstruction algorithm into the design problem. This enables efficient parameterization, which allows us to use only a small number of training examples to learn designs that generalize well in the experimental setting without retraining. We show experimental results for both a well-characterized phase target and mouse fibroblast cells using coded-illumination patterns optimized for a sparsity-based phase reconstruction algorithm. Our learned design results using 2 measurements demonstrate similar accuracy to Fourier Ptychography with 69 measurements.
△ Less
Submitted 5 February, 2019; v1 submitted 10 August, 2018;
originally announced August 2018.
