Showing 1–34 of 34 results for author: Öktem, O

Reconstruction for Sparse View Tomography of Long Objects Applied to Imaging in the Wood Industry

Authors: Buda Bajić, Johannes A. J. Huber, Benedikt Neyses, Linus Olofsson, Ozan Öktem

Abstract: In the wood industry, logs are commonly quality screened by discrete X-ray scans on a moving conveyor belt from a few source positions. Typically, two-dimensional (2D) slice-wise measurements are obtained by a sequential scanning geometry. Each 2D slice alone does not carry sufficient information for a three-dimensional tomographic reconstruction in which biological features of interest in the log… ▽ More In the wood industry, logs are commonly quality screened by discrete X-ray scans on a moving conveyor belt from a few source positions. Typically, two-dimensional (2D) slice-wise measurements are obtained by a sequential scanning geometry. Each 2D slice alone does not carry sufficient information for a three-dimensional tomographic reconstruction in which biological features of interest in the log are well preserved. In the present work, we propose a learned iterative reconstruction method based on the Learned Primal-Dual neural network, suited for sequential scanning geometries. Our method accumulates information between neighbouring slices, instead of only accounting for single slices during reconstruction. Our quantitative and qualitative evaluations with as few as five source positions show that our method yields reconstructions of logs that are sufficiently accurate to identify biological features like knots (branches), heartwood and sapwood. △ Less

Submitted 5 March, 2024; originally announced March 2024.

Fast deep learning based reconstruction for limited angle tomography

Authors: Knut Salomonsson, Eric Oldgren, Emanuel Ström, Ozan Öktem

Abstract: A major challenge in computed tomography is reconstructing objects from incomplete data. An increasingly popular solution for these problems is to incorporate deep learning models into reconstruction algorithms. This study introduces a novel approach by integrating a Fourier neural operator (FNO) into the Filtered Backprojection (FBP) reconstruction method, yielding the FNO back projection (FNO-BP… ▽ More A major challenge in computed tomography is reconstructing objects from incomplete data. An increasingly popular solution for these problems is to incorporate deep learning models into reconstruction algorithms. This study introduces a novel approach by integrating a Fourier neural operator (FNO) into the Filtered Backprojection (FBP) reconstruction method, yielding the FNO back projection (FNO-BP) network. We employ moment conditions for sinogram extrapolation to assist the model in mitigating artefacts from limited data. Notably, our deep learning architecture maintains a runtime comparable to classical filtered back projection (FBP) reconstructions, ensuring swift performance during both inference and training. We assess our reconstruction method in the context of the Helsinki Tomography Challenge 2022 and also compare it against regular FBP methods. △ Less

Submitted 19 February, 2024; originally announced February 2024.

Comments: 11 pages, 4 figures

Riemannian geometry for efficient analysis of protein dynamics data

Authors: Willem Diepeveen, Carlos Esteve-Yagüe, Jan Lellmann, Ozan Öktem, Carola-Bibiane Schönlieb

Abstract: An increasingly common viewpoint is that protein dynamics data sets reside in a non-linear subspace of low conformational energy. Ideal data analysis tools for such data sets should therefore account for such non-linear geometry. The Riemannian geometry setting can be suitable for a variety of reasons. First, it comes with a rich structure to account for a wide range of geometries that can be mode… ▽ More An increasingly common viewpoint is that protein dynamics data sets reside in a non-linear subspace of low conformational energy. Ideal data analysis tools for such data sets should therefore account for such non-linear geometry. The Riemannian geometry setting can be suitable for a variety of reasons. First, it comes with a rich structure to account for a wide range of geometries that can be modelled after an energy landscape. Second, many standard data analysis tools initially developed for data in Euclidean space can also be generalised to data on a Riemannian manifold. In the context of protein dynamics, a conceptual challenge comes from the lack of a suitable smooth manifold and the lack of guidelines for constructing a smooth Riemannian structure based on an energy landscape. In addition, computational feasibility in computing geodesics and related map**s poses a major challenge. This work considers these challenges. The first part of the paper develops a novel local approximation technique for computing geodesics and related map**s on Riemannian manifolds in a computationally feasible manner. The second part constructs a smooth manifold of point clouds modulo rigid body group actions and a Riemannian structure that is based on an energy landscape for protein conformations. The resulting Riemannian geometry is tested on several data analysis tasks relevant for protein dynamics data. It performs exceptionally well on coarse-grained molecular dynamics simulated data. In particular, the geodesics with given start- and end-points approximately recover corresponding molecular dynamics trajectories for proteins that undergo relatively ordered transitions with medium sized deformations. The Riemannian protein geometry also gives physically realistic summary statistics and retrieves the underlying dimension even for large-sized deformations within seconds on a laptop. △ Less

Submitted 26 October, 2023; v1 submitted 15 August, 2023; originally announced August 2023.

MSC Class: 49Q10; 53C22; 53Z10; 53Z50; 65D18; 92-08; 92-10

Neural incomplete factorization: learning preconditioners for the conjugate gradient method

Authors: Paul Häusner, Ozan Öktem, Jens Sjölund

Abstract: Finding suitable preconditioners to accelerate iterative solution methods, such as the conjugate gradient method, is an active area of research. In this paper, we develop a computationally efficient data-driven approach to replace the typically hand-engineered algorithms with neural networks. Optimizing the condition number of the linear system directly is computationally infeasible. Instead, our… ▽ More Finding suitable preconditioners to accelerate iterative solution methods, such as the conjugate gradient method, is an active area of research. In this paper, we develop a computationally efficient data-driven approach to replace the typically hand-engineered algorithms with neural networks. Optimizing the condition number of the linear system directly is computationally infeasible. Instead, our method generates an incomplete factorization of the matrix and is, therefore, referred to as neural incomplete factorization (NeuralIF). For efficient training, we utilize a stochastic approximation of the Frobenius loss which only requires matrix-vector multiplications. At the core of our method is a novel messagepassing block, inspired by sparse matrix theory, that aligns with the objective of finding a sparse factorization of the matrix. By replacing conventional preconditioners used within the conjugate gradient method by data-driven models based on graph neural networks, we accelerate the iterative solving procedure. We evaluate our proposed method on both a synthetic and a real-world problem arising from scientific computing and show its ability to reduce the solving time while remaining computationally efficient. △ Less

Submitted 5 February, 2024; v1 submitted 25 May, 2023; originally announced May 2023.

Comments: Under review. 18 pages, 8 figures

Calibrating Ensembles for Scalable Uncertainty Quantification in Deep Learning-based Medical Segmentation

Authors: Thomas Buddenkotte, Lorena Escudero Sanchez, Mireia Crispin-Ortuzar, Ramona Woitek, Cathal McCague, James D. Brenton, Ozan Öktem, Evis Sala, Leonardo Rundo

Abstract: Uncertainty quantification in automated image analysis is highly desired in many applications. Typically, machine learning models in classification or segmentation are only developed to provide binary answers; however, quantifying the uncertainty of the models can play a critical role for example in active learning or machine human interaction. Uncertainty quantification is especially difficult wh… ▽ More Uncertainty quantification in automated image analysis is highly desired in many applications. Typically, machine learning models in classification or segmentation are only developed to provide binary answers; however, quantifying the uncertainty of the models can play a critical role for example in active learning or machine human interaction. Uncertainty quantification is especially difficult when using deep learning-based models, which are the state-of-the-art in many imaging applications. The current uncertainty quantification approaches do not scale well in high-dimensional real-world problems. Scalable solutions often rely on classical techniques, such as dropout, during inference or training ensembles of identical models with different random seeds to obtain a posterior distribution. In this paper, we show that these approaches fail to approximate the classification probability. On the contrary, we propose a scalable and intuitive framework to calibrate ensembles of deep learning models to produce uncertainty quantification measurements that approximate the classification probability. On unseen test data, we demonstrate improved calibration, sensitivity (in two out of three cases) and precision when being compared with the standard approaches. We further motivate the usage of our method in active learning, creating pseudo-labels to learn from unlabeled images and human-machine collaboration. △ Less

Submitted 20 September, 2022; originally announced September 2022.

Spectral decomposition of atomic structures in heterogeneous cryo-EM

Authors: Carlos Esteve-Yagüe, Willem Diepeveen, Ozan Öktem, Carola-Bibiane Schönlieb

Abstract: We consider the problem of recovering the three-dimensional atomic structure of a flexible macromolecule from a heterogeneous cryo-EM dataset. The dataset contains noisy tomographic projections of the electrostatic potential of the macromolecule, taken from different viewing directions, and in the heterogeneous case, each image corresponds to a different conformation of the macromolecule. Under th… ▽ More We consider the problem of recovering the three-dimensional atomic structure of a flexible macromolecule from a heterogeneous cryo-EM dataset. The dataset contains noisy tomographic projections of the electrostatic potential of the macromolecule, taken from different viewing directions, and in the heterogeneous case, each image corresponds to a different conformation of the macromolecule. Under the assumption that the macromolecule can be modelled as a chain, or discrete curve (as it is for instance the case for a protein backbone with a single chain of amino-acids), we introduce a method to estimate the deformation of the atomic model with respect to a given conformation, which is assumed to be known a priori. Our method consists on estimating the torsion and bond angles of the atomic model in each conformation as a linear combination of the eigenfunctions of the Laplace operator in the manifold of conformations. These eigenfunctions can be approximated by means of a well-known technique in manifold learning, based on the construction of a graph Laplacian using the cryo-EM dataset. Finally, we test our approach with synthetic datasets, for which we recover the atomic model of two-dimensional and three-dimensional flexible structures from noisy tomographic projections. △ Less

Submitted 27 December, 2022; v1 submitted 12 September, 2022; originally announced September 2022.

Comments: 35 pages,20 figures

Regularising orientation estimation in Cryo-EM 3D map refinement through measure-based lifting over Riemannian manifolds

Authors: Willem Diepeveen, Jan Lellmann, Ozan Öktem, Carola-Bibiane Schönlieb

Abstract: Motivated by the trade-off between noise-robustness and data-consistency for joint 3D map reconstruction and rotation estimation in single particle cryogenic-electron microscopy (Cryo-EM), we propose ellipsoidal support lifting (ESL), a measure-based lifting scheme for regularising and approximating the global minimiser of a smooth function over a Riemannian manifold. Under a uniqueness assumption… ▽ More Motivated by the trade-off between noise-robustness and data-consistency for joint 3D map reconstruction and rotation estimation in single particle cryogenic-electron microscopy (Cryo-EM), we propose ellipsoidal support lifting (ESL), a measure-based lifting scheme for regularising and approximating the global minimiser of a smooth function over a Riemannian manifold. Under a uniqueness assumption on the minimiser we show several theoretical results, in particular well-posedness of the method and an error bound due to the induced bias with respect to the global minimiser. Additionally, we use the developed theory to integrate the measure-based lifting scheme into an alternating update method for joint homogeneous 3D map reconstruction and rotation estimation, where typically tens of thousands of manifold-valued minimisation problems have to be solved and where regularisation is necessary because of the high noise levels in the data. The joint recovery method is used to test both the theoretical predictions and algorithmic performance through numerical experiments with Cryo-EM data. In particular, the induced bias due to the regularising effect of ESL empirically estimates better rotations, i.e., rotations closer to the ground truth, than global optimisation would. △ Less

Submitted 31 January, 2023; v1 submitted 7 September, 2022; originally announced September 2022.

MSC Class: 90C26; 92E10; 68U10

Deep Learning for Material Decomposition in Photon-Counting CT

Authors: Alma Eguizabal, Ozan Öktem, Mats U. Persson

Abstract: Photon-counting CT (PCCT) offers improved diagnostic performance through better spatial and energy resolution, but develo** high-quality image reconstruction methods that can deal with these large datasets is challenging. Model-based solutions incorporate models of the physical acquisition in order to reconstruct more accurate images, but are dependent on an accurate forward operator and prese… ▽ More Photon-counting CT (PCCT) offers improved diagnostic performance through better spatial and energy resolution, but develo** high-quality image reconstruction methods that can deal with these large datasets is challenging. Model-based solutions incorporate models of the physical acquisition in order to reconstruct more accurate images, but are dependent on an accurate forward operator and present difficulties with finding good regularization. Another approach is deep-learning reconstruction, which has shown great promise in CT. However, fully data-driven solutions typically need large amounts of training data and lack interpretability. To combine the benefits of both methods, while minimizing their respective drawbacks, it is desirable to develop reconstruction algorithms that combine both model-based and data-driven approaches. In this work, we present a novel deep-learning solution for material decomposition in PCCT, based on an unrolled/unfolded iterative network. We evaluate two cases: a learned post-processing, which implicitly utilizes model knowledge, and a learned gradient-descent, which has explicit model-based components in the architecture. With our proposed techniques, we solve a challenging PCCT simulation case: three-material decomposition in abdomen imaging with low dose, iodine contrast, and a very small training sample support. In this scenario, our approach outperforms a maximum likelihood estimation, a variational method, as well as a fully-learned network. △ Less

Submitted 5 August, 2022; originally announced August 2022.

Learned reconstruction methods with convergence guarantees

Authors: Subhadip Mukherjee, Andreas Hauptmann, Ozan Öktem, Marcelo Pereyra, Carola-Bibiane Schönlieb

Abstract: In recent years, deep learning has achieved remarkable empirical success for image reconstruction. This has catalyzed an ongoing quest for precise characterization of correctness and reliability of data-driven methods in critical use-cases, for instance in medical imaging. Notwithstanding the excellent performance and efficacy of deep learning-based methods, concerns have been raised regarding the… ▽ More In recent years, deep learning has achieved remarkable empirical success for image reconstruction. This has catalyzed an ongoing quest for precise characterization of correctness and reliability of data-driven methods in critical use-cases, for instance in medical imaging. Notwithstanding the excellent performance and efficacy of deep learning-based methods, concerns have been raised regarding their stability, or lack thereof, with serious practical implications. Significant advances have been made in recent years to unravel the inner workings of data-driven image recovery methods, challenging their widely perceived black-box nature. In this article, we will specify relevant notions of convergence for data-driven image reconstruction, which will form the basis of a survey of learned methods with mathematically rigorous reconstruction guarantees. An example that is highlighted is the role of ICNN, offering the possibility to combine the power of deep learning with classical convex regularization theory for devising methods that are provably convergent. This survey article is aimed at both methodological researchers seeking to advance the frontiers of our understanding of data-driven image reconstruction methods as well as practitioners, by providing an accessible description of useful convergence concepts and by placing some of the existing empirical practices on a solid mathematical foundation. △ Less

Submitted 14 September, 2022; v1 submitted 11 June, 2022; originally announced June 2022.

3D helical CT Reconstruction with a Memory Efficient Learned Primal-Dual Architecture

Authors: Jevgenija Rudzusika, Buda Bajić, Thomas Koehler, Ozan Öktem

Abstract: Deep learning based computed tomography (CT) reconstruction has demonstrated outstanding performance on simulated 2D low-dose CT data. This applies in particular to domain adapted neural networks, which incorporate a handcrafted physics model for CT imaging. Empirical evidence shows that employing such architectures reduces the demand for training data and improves upon generalisation. However, th… ▽ More Deep learning based computed tomography (CT) reconstruction has demonstrated outstanding performance on simulated 2D low-dose CT data. This applies in particular to domain adapted neural networks, which incorporate a handcrafted physics model for CT imaging. Empirical evidence shows that employing such architectures reduces the demand for training data and improves upon generalisation. However, their training requires large computational resources that quickly become prohibitive in 3D helical CT, which is the most common acquisition geometry used for medical imaging. Furthermore, clinical data also comes with other challenges not accounted for in simulations, like errors in flux measurement, resolution mismatch and, most importantly, the absence of the real ground truth. The necessity to have a computationally feasible training combined with the need to address these issues has made it difficult to evaluate deep learning based reconstruction on clinical 3D helical CT. This paper modifies a domain adapted neural network architecture, the Learned Primal-Dual (LPD), so that it can be trained and applied to reconstruction in this setting. We achieve this by splitting the helical trajectory into sections and applying the unrolled LPD iterations to those sections sequentially. To the best of our knowledge, this work is the first to apply an unrolled deep learning architecture for reconstruction on full-sized clinical data, like those in the Low dose CT image and projection data set (LDCT). Moreover, training and testing is done on a single GPU card with 24GB of memory. △ Less

Submitted 28 November, 2023; v1 submitted 24 May, 2022; originally announced May 2022.

Deep learning based dictionary learning and tomographic image reconstruction

Authors: Jevgenija Rudzusika, Thomas Koehler, Ozan Öktem

Abstract: This work presents an approach for image reconstruction in clinical low-dose tomography that combines principles from sparse signal processing with ideas from deep learning. First, we describe sparse signal representation in terms of dictionaries from a statistical perspective and interpret dictionary learning as a process of aligning distribution that arises from a generative model with empirical… ▽ More This work presents an approach for image reconstruction in clinical low-dose tomography that combines principles from sparse signal processing with ideas from deep learning. First, we describe sparse signal representation in terms of dictionaries from a statistical perspective and interpret dictionary learning as a process of aligning distribution that arises from a generative model with empirical distribution of true signals. As a result we can see that sparse coding with learned dictionaries resembles a specific variational autoencoder, where the decoder is a linear function and the encoder is a sparse coding algorithm. Next, we show that dictionary learning can also benefit from computational advancements introduced in the context of deep learning, such as parallelism and as stochastic optimization. Finally, we show that regularization by dictionaries achieves competitive performance in computed tomography (CT) reconstruction comparing to state-of-the-art model based and data driven approaches. △ Less

Submitted 26 August, 2021; originally announced August 2021.

Comments: 34 pages, 5 figures

Journal ref: SIAM Journal on Imaging Sciences, Vol 15, Iss 4. (2002)

Deep Microlocal Reconstruction for Limited-Angle Tomography

Authors: Héctor Andrade-Loarca, Gitta Kutyniok, Ozan Öktem, Philipp Petersen

Abstract: We present a deep learning-based algorithm to jointly solve a reconstruction problem and a wavefront set extraction problem in tomographic imaging. The algorithm is based on a recently developed digital wavefront set extractor as well as the well-known microlocal canonical relation for the Radon transform. We use the wavefront set information about x-ray data to improve the reconstruction by requi… ▽ More We present a deep learning-based algorithm to jointly solve a reconstruction problem and a wavefront set extraction problem in tomographic imaging. The algorithm is based on a recently developed digital wavefront set extractor as well as the well-known microlocal canonical relation for the Radon transform. We use the wavefront set information about x-ray data to improve the reconstruction by requiring that the underlying neural networks simultaneously extract the correct ground truth wavefront set and ground truth image. As a necessary theoretical step, we identify the digital microlocal canonical relations for deep convolutional residual neural networks. We find strong numerical evidence for the effectiveness of this approach. △ Less

Submitted 12 August, 2021; originally announced August 2021.

Comments: 43 pages, 8 figures

MSC Class: 35A18; 65T60; 68T10

End-to-end reconstruction meets data-driven regularization for inverse problems

Authors: Subhadip Mukherjee, Marcello Carioni, Ozan Öktem, Carola-Bibiane Schönlieb

Abstract: We propose an unsupervised approach for learning end-to-end reconstruction operators for ill-posed inverse problems. The proposed method combines the classical variational framework with iterative unrolling, which essentially seeks to minimize a weighted combination of the expected distortion in the measurement space and the Wasserstein-1 distance between the distributions of the reconstruction an… ▽ More We propose an unsupervised approach for learning end-to-end reconstruction operators for ill-posed inverse problems. The proposed method combines the classical variational framework with iterative unrolling, which essentially seeks to minimize a weighted combination of the expected distortion in the measurement space and the Wasserstein-1 distance between the distributions of the reconstruction and ground-truth. More specifically, the regularizer in the variational setting is parametrized by a deep neural network and learned simultaneously with the unrolled reconstruction operator. The variational problem is then initialized with the reconstruction of the unrolled operator and solved iteratively till convergence. Notably, it takes significantly fewer iterations to converge, thanks to the excellent initialization obtained via the unrolled operator. The resulting approach combines the computational efficiency of end-to-end unrolled reconstruction with the well-posedness and noise-stability guarantees of the variational setting. Moreover, we demonstrate with the example of X-ray computed tomography (CT) that our approach outperforms state-of-the-art unsupervised methods, and that it outperforms or is on par with state-of-the-art supervised learned reconstruction approaches. △ Less

Submitted 7 June, 2021; originally announced June 2021.

Accelerated Forward-Backward Optimization using Deep Learning

Authors: Sebastian Banert, Jevgenija Rudzusika, Ozan Öktem, Jonas Adler

Abstract: We propose several deep-learning accelerated optimization solvers with convergence guarantees. We use ideas from the analysis of accelerated forward-backward schemes like FISTA, but instead of the classical approach of proving convergence for a choice of parameters, such as a step-size, we show convergence whenever the update is chosen in a specific set. Rather than picking a point in this set usi… ▽ More We propose several deep-learning accelerated optimization solvers with convergence guarantees. We use ideas from the analysis of accelerated forward-backward schemes like FISTA, but instead of the classical approach of proving convergence for a choice of parameters, such as a step-size, we show convergence whenever the update is chosen in a specific set. Rather than picking a point in this set using some predefined method, we train a deep neural network to pick the best update. Finally, we show that the method is applicable to several cases of smooth and non-smooth optimization and show superior results to established accelerated solvers. △ Less

Submitted 11 May, 2021; originally announced May 2021.

Comments: 28 pages, 4 figures

MSC Class: 90C25 (Primary) 90C06; 68T07; 49M37; 65K10 (Secondary)

Adversarially learned iterative reconstruction for imaging inverse problems

Authors: Subhadip Mukherjee, Ozan Öktem, Carola-Bibiane Schönlieb

Abstract: In numerous practical applications, especially in medical image reconstruction, it is often infeasible to obtain a large ensemble of ground-truth/measurement pairs for supervised learning. Therefore, it is imperative to develop unsupervised learning protocols that are competitive with supervised approaches in performance. Motivated by the maximum-likelihood principle, we propose an unsupervised le… ▽ More In numerous practical applications, especially in medical image reconstruction, it is often infeasible to obtain a large ensemble of ground-truth/measurement pairs for supervised learning. Therefore, it is imperative to develop unsupervised learning protocols that are competitive with supervised approaches in performance. Motivated by the maximum-likelihood principle, we propose an unsupervised learning framework for solving ill-posed inverse problems. Instead of seeking pixel-wise proximity between the reconstructed and the ground-truth images, the proposed approach learns an iterative reconstruction network whose output matches the ground-truth in distribution. Considering tomographic reconstruction as an application, we demonstrate that the proposed unsupervised approach not only performs on par with its supervised variant in terms of objective quality measures but also successfully circumvents the issue of over-smoothing that supervised approaches tend to suffer from. The improvement in reconstruction quality comes at the expense of higher training complexity, but, once trained, the reconstruction time remains the same as its supervised counterpart. △ Less

Submitted 30 March, 2021; originally announced March 2021.

Comments: Accepted to the Eighth International Conference on Scale Space and Variational Methods in Computer Vision (SSVM), May-2021

Learned convex regularizers for inverse problems

Authors: Subhadip Mukherjee, Sören Dittmer, Zakhar Shumaylov, Sebastian Lunz, Ozan Öktem, Carola-Bibiane Schönlieb

Abstract: We consider the variational reconstruction framework for inverse problems and propose to learn a data-adaptive input-convex neural network (ICNN) as the regularization functional. The ICNN-based convex regularizer is trained adversarially to discern ground-truth images from unregularized reconstructions. Convexity of the regularizer is desirable since (i) one can establish analytical convergence g… ▽ More We consider the variational reconstruction framework for inverse problems and propose to learn a data-adaptive input-convex neural network (ICNN) as the regularization functional. The ICNN-based convex regularizer is trained adversarially to discern ground-truth images from unregularized reconstructions. Convexity of the regularizer is desirable since (i) one can establish analytical convergence guarantees for the corresponding variational reconstruction problem and (ii) devise efficient and provable algorithms for reconstruction. In particular, we show that the optimal solution to the variational problem converges to the ground-truth if the penalty parameter decays sub-linearly with respect to the norm of the noise. Further, we prove the existence of a sub-gradient-based algorithm that leads to a monotonically decreasing error in the parameter space with iterations. To demonstrate the performance of our approach for solving inverse problems, we consider the tasks of deblurring natural images and reconstructing images in computed tomography (CT), and show that the proposed convex regularizer is at least competitive with and sometimes superior to state-of-the-art data-driven techniques for inverse problems. △ Less

Submitted 1 March, 2021; v1 submitted 6 August, 2020; originally announced August 2020.

Image reconstruction in dynamic inverse problems with temporal models

Authors: Andreas Hauptmann, Ozan Öktem, Carola Schönlieb

Abstract: The paper surveys variational approaches for image reconstruction in dynamic inverse problems. Emphasis is on methods that rely on parametrised temporal models. These are here encoded as diffeomorphic deformations with time dependent parameters, or as motion constrained reconstruction where the motion model is given by a partial differential equation. The survey also includes recent development in… ▽ More The paper surveys variational approaches for image reconstruction in dynamic inverse problems. Emphasis is on methods that rely on parametrised temporal models. These are here encoded as diffeomorphic deformations with time dependent parameters, or as motion constrained reconstruction where the motion model is given by a partial differential equation. The survey also includes recent development in integrating deep learning for solving these computationally demanding variational methods. Examples are given for 2D dynamic tomography, but methods apply to general inverse problems. △ Less

Submitted 20 July, 2020; originally announced July 2020.

A Fast and Adaptive Algorithm to Compute the X-ray Transform

Authors: Chong Chen, Runqian Wang, Chandrajit Bajaj, Ozan Öktem

Abstract: We propose a new algorithm to compute the X-ray transform of an image represented by unit (pixel/voxel) basis functions. The fundamental issue is equivalently calculating the intersection lengths of the ray with associated units. For any given ray, we first derive the sufficient and necessary condition for non-vanishing intersectability. By this condition, we then distinguish the units that produc… ▽ More We propose a new algorithm to compute the X-ray transform of an image represented by unit (pixel/voxel) basis functions. The fundamental issue is equivalently calculating the intersection lengths of the ray with associated units. For any given ray, we first derive the sufficient and necessary condition for non-vanishing intersectability. By this condition, we then distinguish the units that produce valid intersections with the ray. Only for those units rather than all the individuals, we calculate the intersection lengths by the obtained analytic formula. The proposed algorithm is adapted to 2D/3D parallel beam and 2D fan beam. Particularly, we derive the transformation formulas and generalize the algorithm to 3D circular and helical cone beams. Moreover, we discuss the intrinsic ambiguities of the problem itself, and present a solution. The algorithm not only possesses the adaptability with regard to the center position, scale and size of the image, but also is suited to parallelize with optimality. The comparison study demonstrates the proposed algorithm is fast, more complete, and is more flexible with respect to different scanning geometries and different basis functions. Finally, we validate the correctness of the algorithm by the aforementioned scanning geometries. △ Less

Submitted 20 August, 2020; v1 submitted 31 May, 2020; originally announced June 2020.

Comments: 27 pages, 4 figures, 5 tables

MSC Class: 65R10; 92C55; 65D18; 65R32; 68U10; 94A08; 94A12; 92C05

Journal ref: International Journal of Computer Mathematics, 2021

Shearlets as Feature Extractor for Semantic Edge Detection: The Model-Based and Data-Driven Realm

Authors: Héctor Andrade-Loarca, Gitta Kutyniok, Ozan Öktem

Abstract: Semantic edge detection has recently gained a lot of attention as an image processing task, mainly due to its wide range of real-world applications. This is based on the fact that edges in images contain most of the semantic information. Semantic edge detection involves two tasks, namely pure edge detecion and edge classification. Those are in fact fundamentally distinct in terms of the level of a… ▽ More Semantic edge detection has recently gained a lot of attention as an image processing task, mainly due to its wide range of real-world applications. This is based on the fact that edges in images contain most of the semantic information. Semantic edge detection involves two tasks, namely pure edge detecion and edge classification. Those are in fact fundamentally distinct in terms of the level of abstraction that each task requires, which is known as the distracted supervision paradox that limits the possible performance of a supervised model in semantic edge detection. In this work, we will present a novel hybrid method to avoid the distracted supervision paradox and achieve high-performance in semantic edge detection. Our approach is based on a combination of the model-based concept of shearlets, which provides probably optimally sparse approximations of a model-class of images, and the data-driven method of a suitably designed convolutional neural netwok. Finally, we present several applications such as tomographic reconstruction and show that our approach signifiantly outperforms former methods, thereby indicating the value of such hybrid methods for the area in biomedical imaging. △ Less

Submitted 27 November, 2019; originally announced November 2019.

Comments: 30 pages, 12 figures. To appear in Proceedings of the Royal Society. Mathematical, physical and engineering sciences

MSC Class: 42Bxx; 35A18; 65T60; 68T10

Spatiotemporal PET reconstruction using ML-EM with learned diffeomorphic deformation

Authors: Ozan Öktem, Camille Pouchol, Olivier Verdier

Abstract: Patient movement in emission tomography deteriorates reconstruction quality because of motion blur. Gating the data improves the situation somewhat: each gate contains a movement phase which is approximately stationary. A standard method is to use only the data from a few gates, with little movement between them. However, the corresponding loss of data entails an increase of noise. Motion correcti… ▽ More Patient movement in emission tomography deteriorates reconstruction quality because of motion blur. Gating the data improves the situation somewhat: each gate contains a movement phase which is approximately stationary. A standard method is to use only the data from a few gates, with little movement between them. However, the corresponding loss of data entails an increase of noise. Motion correction algorithms have been implemented to take into account all the gated data, but they do not scale well, especially not in 3D. We propose a novel motion correction algorithm which addresses the scalability issue. Our approach is to combine an enhanced ML-EM algorithm with deep learning based movement registration. The training is unsupervised, and with artificial data. We expect this approach to scale very well to higher resolutions and to 3D, as the overall cost of our algorithm is only marginally greater than that of a standard ML-EM algorithm. We show that we can significantly decrease the noise corresponding to a limited number of gates. △ Less

Submitted 26 August, 2019; originally announced August 2019.

Multi-Scale Learned Iterative Reconstruction

Authors: Andreas Hauptmann, Jonas Adler, Simon Arridge, Ozan Öktem

Abstract: Model-based learned iterative reconstruction methods have recently been shown to outperform classical reconstruction algorithms. Applicability of these methods to large scale inverse problems is however limited by the available memory for training and extensive training times, the latter due to computationally expensive forward models. As a possible solution to these restrictions we propose a mult… ▽ More Model-based learned iterative reconstruction methods have recently been shown to outperform classical reconstruction algorithms. Applicability of these methods to large scale inverse problems is however limited by the available memory for training and extensive training times, the latter due to computationally expensive forward models. As a possible solution to these restrictions we propose a multi-scale learned iterative reconstruction scheme that computes iterates on discretisations of increasing resolution. This procedure does not only reduce memory requirements, it also considerably speeds up reconstruction and training times, but most importantly is scalable to large scale inverse problems with non-trivial forward operators, such as those that arise in many 3D tomographic applications. In particular, we propose a hybrid network that combines the multi-scale iterative approach with a particularly expressive network architecture which in combination exhibits excellent scalability in 3D. Applicability of the algorithm is demonstrated for 3D cone beam computed tomography from real measurement data of an organic phantom. Additionally, we examine scalability and reconstruction quality in comparison to established learned reconstruction methods in two dimensions for low dose computed tomography on human phantoms. △ Less

Submitted 20 April, 2020; v1 submitted 1 August, 2019; originally announced August 2019.

Extraction of digital wavefront sets using applied harmonic analysis and deep neural networks

Authors: Héctor Andrade-Loarca, Gitta Kutyniok, Ozan Öktem, Philipp Petersen

Abstract: Microlocal analysis provides deep insight into singularity structures and is often crucial for solving inverse problems, predominately, in imaging sciences. Of particular importance is the analysis of wavefront sets and the correct extraction of those. In this paper, we introduce the first algorithmic approach to extract the wavefront set of images, which combines data-based and model-based method… ▽ More Microlocal analysis provides deep insight into singularity structures and is often crucial for solving inverse problems, predominately, in imaging sciences. Of particular importance is the analysis of wavefront sets and the correct extraction of those. In this paper, we introduce the first algorithmic approach to extract the wavefront set of images, which combines data-based and model-based methods. Based on a celebrated property of the shearlet transform to unravel information on the wavefront set, we extract the wavefront set of an image by first applying a discrete shearlet transform and then feeding local patches of this transform to a deep convolutional neural network trained on labeled data. The resulting algorithm outperforms all competing algorithms in edge-orientation and ramp-orientation detection. △ Less

Submitted 10 July, 2019; v1 submitted 5 January, 2019; originally announced January 2019.

MSC Class: 35A18; 65T60; 68T10

A New Variational Model for Joint Image Reconstruction and Motion Estimation in Spatiotemporal Imaging

Authors: Chong Chen, Barbara Gris, Ozan Öktem

Abstract: We propose a new variational model for joint image reconstruction and motion estimation in spatiotemporal imaging, which is investigated along a general framework that we present with shape theory. This model consists of two components, one for conducting modified static image reconstruction, and the other performs sequentially indirect image registration. For the latter, we generalize the large d… ▽ More We propose a new variational model for joint image reconstruction and motion estimation in spatiotemporal imaging, which is investigated along a general framework that we present with shape theory. This model consists of two components, one for conducting modified static image reconstruction, and the other performs sequentially indirect image registration. For the latter, we generalize the large deformation diffeomorphic metric map** framework into the sequentially indirect registration setting. The proposed model is compared theoretically against alternative approaches (optical flow based model and diffeomorphic motion models), and we demonstrate that the proposed model has desirable properties in terms of the optimal solution. The theoretical derivations and efficient algorithms are also presented for a time-discretized scenario of the proposed model, which show that the optimal solution of the time-discretized version is consistent with that of the time-continuous one, and most of the computational components is the easy-implemented linearized deformation. The complexity of the algorithm is analyzed as well. This work is concluded by some numerical examples in 2D space + time tomography with very sparse and/or highly noisy data. △ Less

Submitted 18 December, 2018; v1 submitted 9 December, 2018; originally announced December 2018.

Comments: 35 pages, 5 figures, 3 tables, revised

MSC Class: 65F22; 65R32; 65R30; 65D18; 65J22; 65J20; 65L09; 68U10; 94A12; 94A08; 92C55; 54C56; 57N25; 47A52

Journal ref: SIAM Journal on Imaging Sciences 2019

A data-driven iteratively regularized Landweber iteration

Authors: Andrea Aspri, Sebastian Banert, Ozan Öktem, Otmar Scherzer

Abstract: We derive and analyse a new variant of the iteratively regularized Landweber iteration, for solving linear and nonlinear ill-posed inverse problems. The method takes into account training data, which are used to estimate the interior of a black box, which is used to define the iteration process. We prove convergence and stability for the scheme in infinite dimensional Hilbert spaces. These theoret… ▽ More We derive and analyse a new variant of the iteratively regularized Landweber iteration, for solving linear and nonlinear ill-posed inverse problems. The method takes into account training data, which are used to estimate the interior of a black box, which is used to define the iteration process. We prove convergence and stability for the scheme in infinite dimensional Hilbert spaces. These theoretical results are complemented by several numerical experiments for solving linear inverse problems for the Radon transform and a nonlinear inverse problem for Schlieren tomography. △ Less

Submitted 18 March, 2020; v1 submitted 1 December, 2018; originally announced December 2018.

MSC Class: 65J20 (65J10; 65J15; 65J22)

Deep Bayesian Inversion

Authors: Jonas Adler, Ozan Öktem

Abstract: Characterizing statistical properties of solutions of inverse problems is essential for decision making. Bayesian inversion offers a tractable framework for this purpose, but current approaches are computationally unfeasible for most realistic imaging applications in the clinic. We introduce two novel deep learning based methods for solving large-scale inverse problems using Bayesian inversion: a… ▽ More Characterizing statistical properties of solutions of inverse problems is essential for decision making. Bayesian inversion offers a tractable framework for this purpose, but current approaches are computationally unfeasible for most realistic imaging applications in the clinic. We introduce two novel deep learning based methods for solving large-scale inverse problems using Bayesian inversion: a sampling based method using a WGAN with a novel mini-discriminator and a direct approach that trains a neural network using a novel loss function. The performance of both methods is demonstrated on image reconstruction in ultra low dose 3D helical CT. We compute the posterior mean and standard deviation of the 3D images followed by a hypothesis test to assess whether a "dark spot" in the liver of a cancer stricken patient is present. Both methods are computationally efficient and our evaluation shows very promising performance that clearly supports the claim that Bayesian inversion is usable for 3D imaging in time critical applications. △ Less

Submitted 14 November, 2018; originally announced November 2018.

Template-Based Image Reconstruction from Sparse Tomographic Data

Authors: Lukas F. Lang, Sebastian Neumayer, Ozan Öktem, Carola-Bibiane Schönlieb

Abstract: We propose a variational regularisation approach for the problem of template-based image reconstruction from indirect, noisy measurements as given, for instance, in X-ray computed tomography. An image is reconstructed from such measurements by deforming a given template image. The image registration is directly incorporated into the variational regularisation approach in the form of a partial diff… ▽ More We propose a variational regularisation approach for the problem of template-based image reconstruction from indirect, noisy measurements as given, for instance, in X-ray computed tomography. An image is reconstructed from such measurements by deforming a given template image. The image registration is directly incorporated into the variational regularisation approach in the form of a partial differential equation that models the registration as either mass- or intensity-preserving transport from the template to the unknown reconstruction. We provide theoretical results for the proposed variational regularisation for both cases. In particular, we prove existence of a minimiser, stability with respect to the data, and convergence for vanishing noise when either of the abovementioned equations is imposed and more general distance functions are used. Numerically, we solve the problem by extending existing Lagrangian methods and propose a multilevel approach that is applicable whenever a suitable downsampling procedure for the operator and the measured data can be provided. Finally, we demonstrate the performance of our method for template-based image reconstruction from highly undersampled and noisy Radon transform data. We compare results for mass- and intensity-preserving image registration, various regularisation functionals, and different distance functions. Our results show that very reasonable reconstructions can be obtained when only few measurements are available and demonstrate that the use of a normalised cross correlation-based distance is advantageous when the image intensities between the template and the unknown image differ substantially. △ Less

Submitted 1 April, 2019; v1 submitted 19 October, 2018; originally announced October 2018.

MSC Class: 65F22; 65R32; 65D18; 65K10; 49J20; 35Q93; 94A08; 92C55

Task adapted reconstruction for inverse problems

Authors: Jonas Adler, Sebastian Lunz, Olivier Verdier, Carola-Bibiane Schönlieb, Ozan Öktem

Abstract: The paper considers the problem of performing a task defined on a model parameter that is only observed indirectly through noisy data in an ill-posed inverse problem. A key aspect is to formalize the steps of reconstruction and task as appropriate estimators (non-randomized decision rules) in statistical estimation problems. The implementation makes use of (deep) neural networks to provide a diffe… ▽ More The paper considers the problem of performing a task defined on a model parameter that is only observed indirectly through noisy data in an ill-posed inverse problem. A key aspect is to formalize the steps of reconstruction and task as appropriate estimators (non-randomized decision rules) in statistical estimation problems. The implementation makes use of (deep) neural networks to provide a differentiable parametrization of the family of estimators for both steps. These networks are combined and jointly trained against suitable supervised training data in order to minimize a joint differentiable loss function, resulting in an end-to-end task adapted reconstruction method. The suggested framework is generic, yet adaptable, with a plug-and-play structure for adjusting both the inverse problem and the task at hand. More precisely, the data model (forward operator and statistical model of the noise) associated with the inverse problem is exchangeable, e.g., by using neural network architecture given by a learned iterative method. Furthermore, any task that is encodable as a trainable neural network can be used. The approach is demonstrated on joint tomographic image reconstruction, classification and joint tomographic image reconstruction segmentation. △ Less

Submitted 27 August, 2018; originally announced September 2018.

Journal ref: Inverse Problems, Vol. 38, Issue 7 (2022)

Data-driven nonsmooth optimization

Authors: Sebastian Banert, Axel Ringh, Jonas Adler, Johan Karlsson, Ozan Öktem

Abstract: In this work, we consider methods for solving large-scale optimization problems with a possibly nonsmooth objective function. The key idea is to first specify a class of optimization algorithms using a generic iterative scheme involving only linear operations and applications of proximal operators. This scheme contains many modern primal-dual first-order solvers like the Douglas-Rachford and hybri… ▽ More In this work, we consider methods for solving large-scale optimization problems with a possibly nonsmooth objective function. The key idea is to first specify a class of optimization algorithms using a generic iterative scheme involving only linear operations and applications of proximal operators. This scheme contains many modern primal-dual first-order solvers like the Douglas-Rachford and hybrid gradient methods as special cases. Moreover, we show convergence to an optimal point for a new method which also belongs to this class. Next, we interpret the generic scheme as a neural network and use unsupervised training to learn the best set of parameters for a specific class of objective functions while imposing a fixed number of iterations. In contrast to other approaches of "learning to optimize", we present an approach which learns parameters only in the set of convergent schemes. As use cases, we consider optimization problems arising in tomographic reconstruction and image deconvolution, and in particular a family of total variation regularization problems. △ Less

Submitted 2 August, 2018; originally announced August 2018.

Comments: 33 pages, 7 figures

MSC Class: 90C25 (Primary) 68T05; 47H05 (Secondary)

Journal ref: SIAM Journal on Optimization, 30(1), 102-131, 2020

Adversarial Regularizers in Inverse Problems

Authors: Sebastian Lunz, Ozan Öktem, Carola-Bibiane Schönlieb

Abstract: Inverse Problems in medical imaging and computer vision are traditionally solved using purely model-based methods. Among those variational regularization models are one of the most popular approaches. We propose a new framework for applying data-driven approaches to inverse problems, using a neural network as a regularization functional. The network learns to discriminate between the distribution… ▽ More Inverse Problems in medical imaging and computer vision are traditionally solved using purely model-based methods. Among those variational regularization models are one of the most popular approaches. We propose a new framework for applying data-driven approaches to inverse problems, using a neural network as a regularization functional. The network learns to discriminate between the distribution of ground truth images and the distribution of unregularized reconstructions. Once trained, the network is applied to the inverse problem by solving the corresponding variational problem. Unlike other data-based approaches for inverse problems, the algorithm can be applied even if only unsupervised training data is available. Experiments demonstrate the potential of the framework for denoising on the BSDS dataset and for computed tomography reconstruction on the LIDC dataset. △ Less

Submitted 11 January, 2019; v1 submitted 29 May, 2018; originally announced May 2018.

Comments: published at NeurIPS 2018

Learning to solve inverse problems using Wasserstein loss

Authors: Jonas Adler, Axel Ringh, Ozan Öktem, Johan Karlsson

Abstract: We propose using the Wasserstein loss for training in inverse problems. In particular, we consider a learned primal-dual reconstruction scheme for ill-posed inverse problems using the Wasserstein distance as loss function in the learning. This is motivated by miss-alignments in training data, which when using standard mean squared error loss could severely degrade reconstruction quality. We prove… ▽ More We propose using the Wasserstein loss for training in inverse problems. In particular, we consider a learned primal-dual reconstruction scheme for ill-posed inverse problems using the Wasserstein distance as loss function in the learning. This is motivated by miss-alignments in training data, which when using standard mean squared error loss could severely degrade reconstruction quality. We prove that training with the Wasserstein loss gives a reconstruction operator that correctly compensates for miss-alignments in certain cases, whereas training with the mean squared error gives a smeared reconstruction. Moreover, we demonstrate these effects by training a reconstruction algorithm using both mean squared error and optimal transport loss for a problem in computerized tomography. △ Less

Submitted 30 October, 2017; originally announced October 2017.

Comments: 11 pages, 3 figures

Learned Primal-dual Reconstruction

Authors: Jonas Adler, Ozan Öktem

Abstract: We propose the Learned Primal-Dual algorithm for tomographic reconstruction. The algorithm accounts for a (possibly non-linear) forward operator in a deep neural network by unrolling a proximal primal-dual optimization method, but where the proximal operators have been replaced with convolutional neural networks. The algorithm is trained end-to-end, working directly from raw measured data and it d… ▽ More We propose the Learned Primal-Dual algorithm for tomographic reconstruction. The algorithm accounts for a (possibly non-linear) forward operator in a deep neural network by unrolling a proximal primal-dual optimization method, but where the proximal operators have been replaced with convolutional neural networks. The algorithm is trained end-to-end, working directly from raw measured data and it does not depend on any initial reconstruction such as FBP. We compare performance of the proposed method on low dose CT reconstruction against FBP, TV, and deep learning based post-processing of FBP. For the Shepp-Logan phantom we obtain >6dB PSNR improvement against all compared methods. For human phantoms the corresponding improvement is 6.6dB over TV and 2.2dB over learned post-processing along with a substantial improvement in the SSIM. Finally, our algorithm involves only ten forward-back-projection computations, making the method feasible for time critical clinical applications. △ Less

Submitted 5 July, 2018; v1 submitted 20 July, 2017; originally announced July 2017.

Comments: 11 pages, 5 figures

Journal ref: IEEE Transactions on Medical Imaging (2018)

Indirect Image Registration with Large Diffeomorphic Deformations

Authors: Chong Chen, Ozan Öktem

Abstract: The paper adapts the large deformation diffeomorphic metric map** framework for image registration to the indirect setting where a template is registered against a target that is given through indirect noisy observations. The registration uses diffeomorphisms that transform the template through a (group) action. These diffeomorphisms are generated by solving a flow equation that is defined by a… ▽ More The paper adapts the large deformation diffeomorphic metric map** framework for image registration to the indirect setting where a template is registered against a target that is given through indirect noisy observations. The registration uses diffeomorphisms that transform the template through a (group) action. These diffeomorphisms are generated by solving a flow equation that is defined by a velocity field with certain regularity. The theoretical analysis includes a proof that indirect image registration has solutions (existence) that are stable and that converge as the data error tends so zero, so it becomes a well-defined regularization method. The paper concludes with examples of indirect image registration in 2D tomography with very sparse and/or highly noisy data. △ Less

Submitted 11 October, 2017; v1 submitted 13 June, 2017; originally announced June 2017.

Comments: 43 pages, 4 figures, 1 table; revised

MSC Class: 65F22; 65R32; 65R30; 65D18; 94A12; 94A08; 92C55; 54C56; 57N25; 47A52

Journal ref: SIAM Journal on Imaging Sciences 2018

Solving ill-posed inverse problems using iterative deep neural networks

Authors: Jonas Adler, Ozan Öktem

Abstract: We propose a partially learned approach for the solution of ill posed inverse problems with not necessarily linear forward operators. The method builds on ideas from classical regularization theory and recent advances in deep learning to perform learning while making use of prior information about the inverse problem encoded in the forward operator, noise model and a regularizing functional. The m… ▽ More We propose a partially learned approach for the solution of ill posed inverse problems with not necessarily linear forward operators. The method builds on ideas from classical regularization theory and recent advances in deep learning to perform learning while making use of prior information about the inverse problem encoded in the forward operator, noise model and a regularizing functional. The method results in a gradient-like iterative scheme, where the "gradient" component is learned using a convolutional network that includes the gradients of the data discrepancy and regularizer as input in each iteration. We present results of such a partially learned gradient scheme on a non-linear tomographic inversion problem with simulated data from both the Sheep-Logan phantom as well as a head CT. The outcome is compared against FBP and TV reconstruction and the proposed method provides a 5.4 dB PSNR improvement over the TV reconstruction while being significantly faster, giving reconstructions of 512 x 512 volumes in about 0.4 seconds using a single GPU. △ Less

Submitted 22 May, 2017; v1 submitted 13 April, 2017; originally announced April 2017.

Journal ref: Inverse Problems 2017

Tunable Ampere phase plate for low dose imaging of biomolecular complexes

Authors: Amir H. Tavabi, Marco Beleggia, Vadim Migunov, Alexey Savenko, Sara Sandin, Ozan Öktem, Rafal E. Dunin-Borkowski, Giulio Pozzi

Abstract: A novel device that can be used as a tunable support-free phase plate for transmission electron microscopy of weakly scattering specimens is described. The device relies on the generation of a controlled phase shift by the magnetic field of a segment of current-carrying wire that is oriented parallel or antiparallel to the electron beam. The validity of the concept is established using both experi… ▽ More A novel device that can be used as a tunable support-free phase plate for transmission electron microscopy of weakly scattering specimens is described. The device relies on the generation of a controlled phase shift by the magnetic field of a segment of current-carrying wire that is oriented parallel or antiparallel to the electron beam. The validity of the concept is established using both experimental electron holographic measurements and a theoretical model based on Ampere's law. Computer simulations are used to illustrate the resulting contrast enhancement for studies of biological cells and macromolecules. △ Less

Submitted 2 February, 2017; originally announced February 2017.