-
Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields
Authors:
Tom Fischer,
Pascal Peter,
Joachim Weickert,
Eddy Ilg
Abstract:
Deep learning has revolutionized the field of computer vision by introducing large scale neural networks with millions of parameters. Training these networks requires massive datasets and leads to intransparent models that can fail to generalize. At the other extreme, models designed from partial differential equations (PDEs) embed specialized domain knowledge into mathematical equations and usual…
▽ More
Deep learning has revolutionized the field of computer vision by introducing large scale neural networks with millions of parameters. Training these networks requires massive datasets and leads to intransparent models that can fail to generalize. At the other extreme, models designed from partial differential equations (PDEs) embed specialized domain knowledge into mathematical equations and usually rely on few manually chosen hyperparameters. This makes them transparent by construction and if designed and calibrated carefully, they can generalize well to unseen scenarios. In this paper, we show how to bring model- and data-driven approaches together by combining the explicit PDE-based approaches with convolutional neural networks to obtain the best of both worlds. We illustrate a joint architecture for the task of inpainting optical flow fields and show that the combination of model- and data-driven modeling leads to an effective architecture. Our model outperforms both fully explicit and fully data-driven baselines in terms of reconstruction quality, robustness and amount of required training data. Averaging the endpoint error across different mask densities, our method outperforms the explicit baselines by 11-27%, the GAN baseline by 47% and the Probabilisitic Diffusion baseline by 42%. With that, our method sets a new state of the art for inpainting of optical flow fields from random masks.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Anisotropic Diffusion Stencils: From Simple Derivations over Stability Estimates to ResNet Implementations
Authors:
Karl Schrader,
Joachim Weickert,
Michael Krause
Abstract:
Anisotropic diffusion processes with a diffusion tensor are important in image analysis, physics, and engineering. However, their numerical approximation has a strong impact on dissipative artefacts and deviations from rotation invariance. In this work, we study a large family of finite difference discretisations on a 3 x 3 stencil. We derive it by splitting 2-D anisotropic diffusion into four 1-D…
▽ More
Anisotropic diffusion processes with a diffusion tensor are important in image analysis, physics, and engineering. However, their numerical approximation has a strong impact on dissipative artefacts and deviations from rotation invariance. In this work, we study a large family of finite difference discretisations on a 3 x 3 stencil. We derive it by splitting 2-D anisotropic diffusion into four 1-D diffusions. The resulting stencil class involves one free parameter and covers a wide range of existing discretisations. It comprises the full stencil family of Weickert et al. (2013) and shows that their two parameters contain redundancy. Furthermore, we establish a bound on the spectral norm of the matrix corresponding to the stencil. This gives time step size limits that guarantee stability of an explicit scheme in the Euclidean norm. Our directional splitting also allows a very natural translation of the explicit scheme into ResNet blocks. Employing neural network libraries enables simple and highly efficient parallel implementations on GPUs.
△ Less
Submitted 8 April, 2024; v1 submitted 11 September, 2023;
originally announced September 2023.
-
Efficient Neural Generation of 4K Masks for Homogeneous Diffusion Inpainting
Authors:
Karl Schrader,
Pascal Peter,
Niklas Kämper,
Joachim Weickert
Abstract:
With well-selected data, homogeneous diffusion inpainting can reconstruct images from sparse data with high quality. While 4K colour images of size 3840 x 2160 can already be inpainted in real time, optimising the known data for applications like image compression remains challenging: Widely used stochastic strategies can take days for a single 4K image. Recently, a first neural approach for this…
▽ More
With well-selected data, homogeneous diffusion inpainting can reconstruct images from sparse data with high quality. While 4K colour images of size 3840 x 2160 can already be inpainted in real time, optimising the known data for applications like image compression remains challenging: Widely used stochastic strategies can take days for a single 4K image. Recently, a first neural approach for this so-called mask optimisation problem offered high speed and good quality for small images. It trains a mask generation network with the help of a neural inpainting surrogate. However, these mask networks can only output masks for the resolution and mask density they were trained for. We solve these problems and enable mask optimisation for high-resolution images through a neuroexplicit coarse-to-fine strategy. Additionally, we improve the training and interpretability of mask networks by including a numerical inpainting solver directly into the network. This allows to generate masks for 4K images in around 0.6 seconds while exceeding the quality of stochastic methods on practically relevant densities. Compared to popular existing approaches, this is an acceleration of up to four orders of magnitude.
△ Less
Submitted 16 May, 2023; v1 submitted 17 March, 2023;
originally announced March 2023.
-
CNN-based Euler's Elastica Inpainting with Deep Energy and Deep Image Prior
Authors:
Karl Schrader,
Tobias Alt,
Joachim Weickert,
Michael Ertel
Abstract:
Euler's elastica constitute an appealing variational image inpainting model. It minimises an energy that involves the total variation as well as the level line curvature. These components are transparent and make it attractive for shape completion tasks. However, its gradient flow is a singular, anisotropic, and nonlinear PDE of fourth order, which is numerically challenging: It is difficult to fi…
▽ More
Euler's elastica constitute an appealing variational image inpainting model. It minimises an energy that involves the total variation as well as the level line curvature. These components are transparent and make it attractive for shape completion tasks. However, its gradient flow is a singular, anisotropic, and nonlinear PDE of fourth order, which is numerically challenging: It is difficult to find efficient algorithms that offer sharp edges and good rotation invariance. As a remedy, we design the first neural algorithm that simulates inpainting with Euler's Elastica. We use the deep energy concept which employs the variational energy as neural network loss. Furthermore, we pair it with a deep image prior where the network architecture itself acts as a prior. This yields better inpaintings by steering the optimisation trajectory closer to the desired solution. Our results are qualitatively on par with state-of-the-art algorithms on elastica-based shape completion. They combine good rotation invariance with sharp edges. Moreover, we benefit from the high efficiency and effortless parallelisation within a neural framework. Our neural elastica approach only requires 3x3 central difference stencils. It is thus much simpler than other well-performing algorithms for elastica inpainting. Last but not least, it is unsupervised as it requires no ground truth training data.
△ Less
Submitted 14 March, 2023; v1 submitted 16 July, 2022;
originally announced July 2022.
-
Learning Sparse Masks for Diffusion-based Image Inpainting
Authors:
Tobias Alt,
Pascal Peter,
Joachim Weickert
Abstract:
Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. Its quality strongly depends on the choice of known data. Optimising their spatial location -- the inpainting mask -- is challenging. A commonly used tool for this task are stochastic optimisation strategies. However, they are slow as they compute multiple inpainting results. We provide a remedy in ter…
▽ More
Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. Its quality strongly depends on the choice of known data. Optimising their spatial location -- the inpainting mask -- is challenging. A commonly used tool for this task are stochastic optimisation strategies. However, they are slow as they compute multiple inpainting results. We provide a remedy in terms of a learned mask generation model. By emulating the complete inpainting pipeline with two networks for mask generation and neural surrogate inpainting, we obtain a model for highly efficient adaptive mask generation. Experiments indicate that our model can achieve competitive quality with an acceleration by as much as four orders of magnitude. Our findings serve as a basis for making diffusion-based inpainting more attractive for applications such as image compression, where fast encoding is highly desirable.
△ Less
Submitted 16 May, 2022; v1 submitted 6 October, 2021;
originally announced October 2021.
-
Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods
Authors:
Tobias Alt,
Karl Schrader,
Joachim Weickert,
Pascal Peter,
Matthias Augustin
Abstract:
Partial differential equation (PDE) models and their associated variational energy formulations are often rotationally invariant by design. This ensures that a rotation of the input results in a corresponding rotation of the output, which is desirable in applications such as image analysis. Convolutional neural networks (CNNs) do not share this property, and existing remedies are often complex. Th…
▽ More
Partial differential equation (PDE) models and their associated variational energy formulations are often rotationally invariant by design. This ensures that a rotation of the input results in a corresponding rotation of the output, which is desirable in applications such as image analysis. Convolutional neural networks (CNNs) do not share this property, and existing remedies are often complex. The goal of our paper is to investigate how diffusion and variational models achieve rotation invariance and transfer these ideas to neural networks. As a core novelty we propose activation functions which couple network channels by combining information from several oriented filters. This guarantees rotation invariance within the basic building blocks of the networks while still allowing for directional filtering. The resulting neural architectures are inherently rotationally invariant. With only a few small filters, they can achieve the same invariance as existing techniques which require a fine-grained sampling of orientations. Our findings help to translate diffusion and variational models into mathematically well-founded network architectures, and provide novel concepts for model-based CNN design.
△ Less
Submitted 18 March, 2022; v1 submitted 31 August, 2021;
originally announced August 2021.
-
Connections between Numerical Algorithms for PDEs and Neural Networks
Authors:
Tobias Alt,
Karl Schrader,
Matthias Augustin,
Pascal Peter,
Joachim Weickert
Abstract:
We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalis…
▽ More
We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for develo** new mathematically well-founded neural building blocks.
△ Less
Submitted 21 March, 2022; v1 submitted 30 July, 2021;
originally announced July 2021.
-
Translating Numerical Concepts for PDEs into Neural Architectures
Authors:
Tobias Alt,
Pascal Peter,
Joachim Weickert,
Karl Schrader
Abstract:
We investigate what can be learned from translating numerical algorithms into neural networks. On the numerical side, we consider explicit, accelerated explicit, and implicit schemes for a general higher order nonlinear diffusion equation in 1D, as well as linear multigrid methods. On the neural network side, we identify corresponding concepts in terms of residual networks (ResNets), recurrent net…
▽ More
We investigate what can be learned from translating numerical algorithms into neural networks. On the numerical side, we consider explicit, accelerated explicit, and implicit schemes for a general higher order nonlinear diffusion equation in 1D, as well as linear multigrid methods. On the neural network side, we identify corresponding concepts in terms of residual networks (ResNets), recurrent networks, and U-nets. These connections guarantee Euclidean stability of specific ResNets with a transposed convolution layer structure in each block. We present three numerical justifications for skip connections: as time discretisations in explicit schemes, as extrapolation mechanisms for accelerating those methods, and as recurrent connections in fixed point solvers for implicit schemes. Last but not least, we also motivate uncommon design choices such as nonmonotone activation functions. Our findings give a numerical perspective on the success of modern neural network architectures, and they provide design criteria for stable networks.
△ Less
Submitted 17 May, 2021; v1 submitted 29 March, 2021;
originally announced March 2021.
-
Multi-frame Super-resolution from Noisy Data
Authors:
Kireeti Bodduna,
Joachim Weickert,
Marcelo Cárdenas
Abstract:
Obtaining high resolution images from low resolution data with clipped noise is algorithmically challenging due to the ill-posed nature of the problem. So far such problems have hardly been tackled, and the few existing approaches use simplistic regularisers. We show the usefulness of two adaptive regularisers based on anisotropic diffusion ideas: Apart from evaluating the classical edge-enhancing…
▽ More
Obtaining high resolution images from low resolution data with clipped noise is algorithmically challenging due to the ill-posed nature of the problem. So far such problems have hardly been tackled, and the few existing approaches use simplistic regularisers. We show the usefulness of two adaptive regularisers based on anisotropic diffusion ideas: Apart from evaluating the classical edge-enhancing anisotropic diffusion regulariser, we introduce a novel non-local one with one-sided differences and superior performance. It is termed sector diffusion. We combine it with all six variants of the classical super-resolution observational model that arise from permutations of its three operators for war**, blurring, and downsampling. Surprisingly, the evaluation in a practically relevant noisy scenario produces a different ranking than the one in the noise-free setting in our previous work (SSVM 2017).
△ Less
Submitted 25 March, 2021;
originally announced March 2021.
-
Learning Integrodifferential Models for Image Denoising
Authors:
Tobias Alt,
Joachim Weickert
Abstract:
We introduce an integrodifferential extension of the edge-enhancing anisotropic diffusion model for image denoising. By accumulating weighted structural information on multiple scales, our model is the first to create anisotropy through multiscale integration. It follows the philosophy of combining the advantages of model-based and data-driven approaches within compact, insightful, and mathematica…
▽ More
We introduce an integrodifferential extension of the edge-enhancing anisotropic diffusion model for image denoising. By accumulating weighted structural information on multiple scales, our model is the first to create anisotropy through multiscale integration. It follows the philosophy of combining the advantages of model-based and data-driven approaches within compact, insightful, and mathematically well-founded models with improved performance. We explore trained results of scale-adaptive weighting and contrast parameters to obtain an explicit modelling by smooth functions. This leads to a transparent model with only three parameters, without significantly decreasing its denoising performance. Experiments demonstrate that it outperforms its diffusion-based predecessors. We show that both multiscale information and anisotropy are crucial for its success.
△ Less
Submitted 17 May, 2021; v1 submitted 21 October, 2020;
originally announced October 2020.
-
Translating Diffusion, Wavelets, and Regularisation into Residual Networks
Authors:
Tobias Alt,
Joachim Weickert,
Pascal Peter
Abstract:
Convolutional neural networks (CNNs) often perform well, but their stability is poorly understood. To address this problem, we consider the simple prototypical problem of signal denoising, where classical approaches such as nonlinear diffusion, wavelet-based methods and regularisation offer provable stability guarantees. To transfer such guarantees to CNNs, we interpret numerical approximations of…
▽ More
Convolutional neural networks (CNNs) often perform well, but their stability is poorly understood. To address this problem, we consider the simple prototypical problem of signal denoising, where classical approaches such as nonlinear diffusion, wavelet-based methods and regularisation offer provable stability guarantees. To transfer such guarantees to CNNs, we interpret numerical approximations of these classical methods as a specific residual network (ResNet) architecture. This leads to a dictionary which allows to translate diffusivities, shrinkage functions, and regularisers into activation functions, and enables a direct communication between the four research communities. On the CNN side, it does not only inspire new families of nonmonotone activation functions, but also introduces intrinsically stable architectures for an arbitrary number of layers.
△ Less
Submitted 7 June, 2020; v1 submitted 7 February, 2020;
originally announced February 2020.
-
Robustness of Brain Tumor Segmentation
Authors:
Sabine Müller,
Joachim Weickert,
Norbert Graf
Abstract:
Purpose: The segmentation of brain tumors is one of the most active areas of medical image analysis. While current methods perform superhuman on benchmark data sets, their applicability in daily clinical practice has not been evaluated. In our work we investigate the generalization behavior of deep neural networks in this scenario.
Approach: We evaluate the performance of three state-of-the-art…
▽ More
Purpose: The segmentation of brain tumors is one of the most active areas of medical image analysis. While current methods perform superhuman on benchmark data sets, their applicability in daily clinical practice has not been evaluated. In our work we investigate the generalization behavior of deep neural networks in this scenario.
Approach: We evaluate the performance of three state-of-the-art methods, a basic U-net architecture and a cascadic Mumford-Shah approach. We also propose two simple modifications (which do not change the topology) to improve generalization performance.
Results: In our experiments we show that a well-trained U-network shows the best generalization behavior and is sufficient to solve this segmentation problem. We illustrate why extensions of this model in a realistic scenario can be not only pointless but even harmful.
Conclusions: We conclude from our experiments that the generalization performance of deep neural networks is severely limited in medical image analysis especially in the area of brain tumor segmentation. In our opinion, current topologies are optimized for the actual benchmark data set, but are not directly applicable in daily clinical practice.
△ Less
Submitted 15 December, 2020; v1 submitted 24 December, 2019;
originally announced December 2019.
-
Variational Coupling Revisited: Simpler Models, Theoretical Connections, and Novel Applications
Authors:
Aaron Wewior,
Joachim Weickert
Abstract:
Variational models with coupling terms are becoming increasingly popular in image analysis. They involve auxiliary variables, such that their energy minimisation splits into multiple fractional steps that can be solved easier and more efficiently. In our paper we show that coupling models offer a number of interesting properties that go far beyond their obvious numerical benefits. We demonstrate t…
▽ More
Variational models with coupling terms are becoming increasingly popular in image analysis. They involve auxiliary variables, such that their energy minimisation splits into multiple fractional steps that can be solved easier and more efficiently. In our paper we show that coupling models offer a number of interesting properties that go far beyond their obvious numerical benefits. We demonstrate that discontinuity-preserving denoising can be achieved even with quadratic data and smoothness terms, provided that the coupling term involves the $L^1$ norm. We show that such an $L^1$ coupling term provides additional information as a powerful edge detector that has remained unexplored so far. While coupling models in the literature approximate higher order regularisation, we argue that already first order coupling models can be useful. As a specific example, we present a first order coupling model that outperforms classical TV regularisation. It also establishes a theoretical connection between TV regularisation and the Mumford-Shah segmentation approach. Unlike other Mumford-Shah algorithms, it is a strictly convex approximation, for which we can guarantee convergence of a split Bregman algorithm.
△ Less
Submitted 12 December, 2019;
originally announced December 2019.
-
Learning a Generic Adaptive Wavelet Shrinkage Function for Denoising
Authors:
Tobias Alt,
Joachim Weickert
Abstract:
The rise of machine learning in image processing has created a gap between trainable data-driven and classical model-driven approaches: While learning-based models often show superior performance, classical ones are often more transparent. To reduce this gap, we introduce a generic wavelet shrinkage function for denoising which is adaptive to both the wavelet scales as well as the noise standard d…
▽ More
The rise of machine learning in image processing has created a gap between trainable data-driven and classical model-driven approaches: While learning-based models often show superior performance, classical ones are often more transparent. To reduce this gap, we introduce a generic wavelet shrinkage function for denoising which is adaptive to both the wavelet scales as well as the noise standard deviation. It is inferred from trained results of a tightly parametrised function which is inherited from nonlinear diffusion. Our proposed shrinkage function is smooth and compact while only using two parameters. In contrast to many existing shrinkage functions, it is able to enhance image structures by amplifying wavelet coefficients. Experiments show that it outperforms classical shrinkage functions by a significant margin.
△ Less
Submitted 14 April, 2020; v1 submitted 21 October, 2019;
originally announced October 2019.
-
Object Segmentation Tracking from Generic Video Cues
Authors:
Amirhossein Kardoost,
Sabine Müller,
Joachim Weickert,
Margret Keuper
Abstract:
We propose a light-weight variational framework for online tracking of object segmentations in videos based on optical flow and image boundaries. While high-end computer vision methods on this task rely on sequence specific training of dedicated CNN architectures, we show the potential of a variational model, based on generic video information from motion and color. Such cues are usually required…
▽ More
We propose a light-weight variational framework for online tracking of object segmentations in videos based on optical flow and image boundaries. While high-end computer vision methods on this task rely on sequence specific training of dedicated CNN architectures, we show the potential of a variational model, based on generic video information from motion and color. Such cues are usually required for tasks such as robot navigation or grasp estimation. We leverage them directly for video object segmentation and thus provide accurate segmentations at potentially very low extra cost. Our simple method can provide competitive results compared to the costly CNN-based methods with parameter tuning. Furthermore, we show that our approach can be combined with state-of-the-art CNN-based segmentations in order to improve over their respective results. We evaluate our method on the datasets DAVIS 16,17 and SegTrack v2.
△ Less
Submitted 14 August, 2020; v1 submitted 5 October, 2019;
originally announced October 2019.
-
Stable Backward Diffusion Models that Minimise Convex Energies
Authors:
Leif Bergerhoff,
Marcelo Cárdenas,
Joachim Weickert,
Martin Welk
Abstract:
The inverse problem of backward diffusion is known to be ill-posed and highly unstable. Backward diffusion processes appear naturally in image enhancement and deblurring applications. It is therefore greatly desirable to establish a backward diffusion model which implements a smart stabilisation approach that can be used in combination with an easy to handle numerical scheme. So far, existing stab…
▽ More
The inverse problem of backward diffusion is known to be ill-posed and highly unstable. Backward diffusion processes appear naturally in image enhancement and deblurring applications. It is therefore greatly desirable to establish a backward diffusion model which implements a smart stabilisation approach that can be used in combination with an easy to handle numerical scheme. So far, existing stabilisation strategies in literature require sophisticated numerics to solve the underlying initial value problem. We derive a class of space-discrete one-dimensional backward diffusion as gradient descent of energies where we gain stability by imposing range constraints. Interestingly, these energies are even convex. Furthermore, we establish a comprehensive theory for the time-continuous evolution and we show that stability carries over to a simple explicit time discretisation of our model. Finally, we confirm the stability and usefulness of our technique in experiments in which we enhance the contrast of digital greyscale and colour images.
△ Less
Submitted 17 June, 2020; v1 submitted 8 March, 2019;
originally announced March 2019.
-
Algorithms for Piecewise Constant Signal Approximations
Authors:
Leif Bergerhoff,
Joachim Weickert,
Yehuda Dar
Abstract:
We consider the problem of finding optimal piecewise constant approximations of one-dimensional signals. These approximations should consist of a specified number of segments (samples) and minimise the mean squared error to the original signal. We formalise this goal as a discrete nonconvex optimisation problem, for which we study two algorithms. First we reformulate a recent adaptive sampling met…
▽ More
We consider the problem of finding optimal piecewise constant approximations of one-dimensional signals. These approximations should consist of a specified number of segments (samples) and minimise the mean squared error to the original signal. We formalise this goal as a discrete nonconvex optimisation problem, for which we study two algorithms. First we reformulate a recent adaptive sampling method by Dar and Bruckstein in a compact and transparent way. This allows us to analyse its limitations when it comes to violations of its three key assumptions: signal smoothness, local linearity, and error balancing. As a remedy, we propose a direct optimisation approach which does not rely on any of these assumptions and employs a particle swarm optimisation algorithm. Our experiments show that for nonsmooth signals or low sample numbers, the direct optimisation approach offers substantial qualitative advantages over the Dar--Bruckstein method. As a more general contribution, we disprove the optimality of the principle of error balancing for optimising data in the l^2 norm.
△ Less
Submitted 11 June, 2019; v1 submitted 4 March, 2019;
originally announced March 2019.
-
Space-Filling Curve Indices as Acceleration Structure for Exemplar-Based Inpainting
Authors:
Tim Dahmen,
Patrick Trampert,
Pascal Peter,
Pinak Bheed,
Joachim Weickert,
Philipp Slusallek
Abstract:
Exemplar-based inpainting is the process of reconstructing missing parts of an image by searching the remaining data for patches that fit seamlessly. The image is completed to a plausible-looking solution by repeatedly inserting the patch that is the best match according to some cost function. We present an acceleration structure that uses a multi-index scheme to accelerate this search procedure d…
▽ More
Exemplar-based inpainting is the process of reconstructing missing parts of an image by searching the remaining data for patches that fit seamlessly. The image is completed to a plausible-looking solution by repeatedly inserting the patch that is the best match according to some cost function. We present an acceleration structure that uses a multi-index scheme to accelerate this search procedure drastically, particularly in the case of very large datasets. The index scheme uses ideas such as dimensionality reduction and k-nearest neighbor search on space-filling curves that are well known in the field of multimedia databases. Our method has a theoretic runtime of O(log2 n) per iteration and reaches a speedup factor of up to 660 over the original method. The approach has the advantage of being agnostic to most modelbased parts of exemplar-based inpainting such as the order in which patches are processed and the cost function used to determine patch similarity. Thus, the acceleration structure can be used in conjunction with most exemplar-based inpainting algorithms.
△ Less
Submitted 22 January, 2020; v1 submitted 18 December, 2017;
originally announced December 2017.
-
Optimising Spatial and Tonal Data for PDE-based Inpainting
Authors:
Laurent Hoeltgen,
Markus Mainberger,
Sebastian Hoffmann,
Joachim Weickert,
Ching Hoo Tang,
Simon Setzer,
Daniel Johannsen,
Frank Neumann,
Benjamin Doerr
Abstract:
Some recent methods for lossy signal and image compression store only a few selected pixels and fill in the missing structures by inpainting with a partial differential equation (PDE). Suitable operators include the Laplacian, the biharmonic operator, and edge-enhancing anisotropic diffusion (EED). The quality of such approaches depends substantially on the selection of the data that is kept. Opti…
▽ More
Some recent methods for lossy signal and image compression store only a few selected pixels and fill in the missing structures by inpainting with a partial differential equation (PDE). Suitable operators include the Laplacian, the biharmonic operator, and edge-enhancing anisotropic diffusion (EED). The quality of such approaches depends substantially on the selection of the data that is kept. Optimising this data in the domain and codomain gives rise to challenging mathematical problems that shall be addressed in our work.
In the 1D case, we prove results that provide insights into the difficulty of this problem, and we give evidence that a splitting into spatial and tonal (i.e. function value) optimisation does hardly deteriorate the results. In the 2D setting, we present generic algorithms that achieve a high reconstruction quality even if the specified data is very sparse. To optimise the spatial data, we use a probabilistic sparsification, followed by a nonlocal pixel exchange that avoids getting trapped in bad local optima. After this spatial optimisation we perform a tonal optimisation that modifies the function values in order to reduce the global reconstruction error. For homogeneous diffusion inpainting, this comes down to a least squares problem for which we prove that it has a unique solution. We demonstrate that it can be found efficiently with a gradient descent approach that is accelerated with fast explicit diffusion (FED) cycles. Our framework allows to specify the desired density of the inpainting mask a priori. Moreover, is more generic than other data optimisation approaches for the sparse inpainting problem, since it can also be extended to nonlinear inpainting operators such as EED. This is exploited to achieve reconstructions with state-of-the-art quality.
We also give an extensive literature survey on PDE-based image compression methods.
△ Less
Submitted 15 June, 2015;
originally announced June 2015.
