-
Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields
Authors:
Tom Fischer,
Pascal Peter,
Joachim Weickert,
Eddy Ilg
Abstract:
Deep learning has revolutionized the field of computer vision by introducing large scale neural networks with millions of parameters. Training these networks requires massive datasets and leads to intransparent models that can fail to generalize. At the other extreme, models designed from partial differential equations (PDEs) embed specialized domain knowledge into mathematical equations and usual…
▽ More
Deep learning has revolutionized the field of computer vision by introducing large scale neural networks with millions of parameters. Training these networks requires massive datasets and leads to intransparent models that can fail to generalize. At the other extreme, models designed from partial differential equations (PDEs) embed specialized domain knowledge into mathematical equations and usually rely on few manually chosen hyperparameters. This makes them transparent by construction and if designed and calibrated carefully, they can generalize well to unseen scenarios. In this paper, we show how to bring model- and data-driven approaches together by combining the explicit PDE-based approaches with convolutional neural networks to obtain the best of both worlds. We illustrate a joint architecture for the task of inpainting optical flow fields and show that the combination of model- and data-driven modeling leads to an effective architecture. Our model outperforms both fully explicit and fully data-driven baselines in terms of reconstruction quality, robustness and amount of required training data. Averaging the endpoint error across different mask densities, our method outperforms the explicit baselines by 11-27%, the GAN baseline by 47% and the Probabilisitic Diffusion baseline by 42%. With that, our method sets a new state of the art for inpainting of optical flow fields from random masks.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Generalised Diffusion Probabilistic Scale-Spaces
Authors:
Pascal Peter
Abstract:
Diffusion probabilistic models excel at sampling new images from learned distributions. Originally motivated by drift-diffusion concepts from physics, they apply image perturbations such as noise and blur in a forward process that results in a tractable probability distribution. A corresponding learned reverse process generates images and can be conditioned on side information, which leads to a wi…
▽ More
Diffusion probabilistic models excel at sampling new images from learned distributions. Originally motivated by drift-diffusion concepts from physics, they apply image perturbations such as noise and blur in a forward process that results in a tractable probability distribution. A corresponding learned reverse process generates images and can be conditioned on side information, which leads to a wide variety of practical applications. Most of the research focus currently lies on practice-oriented extensions. In contrast, the theoretical background remains largely unexplored, in particular the relations to drift-diffusion. In order to shed light on these connections to classical image filtering, we propose a generalised scale-space theory for diffusion probabilistic models. Moreover, we show conceptual and empirical connections to diffusion and osmosis filters.
△ Less
Submitted 6 June, 2024; v1 submitted 15 September, 2023;
originally announced September 2023.
-
Efficient Neural Generation of 4K Masks for Homogeneous Diffusion Inpainting
Authors:
Karl Schrader,
Pascal Peter,
Niklas Kämper,
Joachim Weickert
Abstract:
With well-selected data, homogeneous diffusion inpainting can reconstruct images from sparse data with high quality. While 4K colour images of size 3840 x 2160 can already be inpainted in real time, optimising the known data for applications like image compression remains challenging: Widely used stochastic strategies can take days for a single 4K image. Recently, a first neural approach for this…
▽ More
With well-selected data, homogeneous diffusion inpainting can reconstruct images from sparse data with high quality. While 4K colour images of size 3840 x 2160 can already be inpainted in real time, optimising the known data for applications like image compression remains challenging: Widely used stochastic strategies can take days for a single 4K image. Recently, a first neural approach for this so-called mask optimisation problem offered high speed and good quality for small images. It trains a mask generation network with the help of a neural inpainting surrogate. However, these mask networks can only output masks for the resolution and mask density they were trained for. We solve these problems and enable mask optimisation for high-resolution images through a neuroexplicit coarse-to-fine strategy. Additionally, we improve the training and interpretability of mask networks by including a numerical inpainting solver directly into the network. This allows to generate masks for 4K images in around 0.6 seconds while exceeding the quality of stochastic methods on practically relevant densities. Compared to popular existing approaches, this is an acceleration of up to four orders of magnitude.
△ Less
Submitted 16 May, 2023; v1 submitted 17 March, 2023;
originally announced March 2023.
-
Generalised Scale-Space Properties for Probabilistic Diffusion Models
Authors:
Pascal Peter
Abstract:
Probabilistic diffusion models enjoy increasing popularity in the deep learning community. They generate convincing samples from a learned distribution of input images with a wide field of practical applications. Originally, these approaches were motivated from drift-diffusion processes, but these origins find less attention in recent, practice-oriented publications. We investigate probabilistic d…
▽ More
Probabilistic diffusion models enjoy increasing popularity in the deep learning community. They generate convincing samples from a learned distribution of input images with a wide field of practical applications. Originally, these approaches were motivated from drift-diffusion processes, but these origins find less attention in recent, practice-oriented publications. We investigate probabilistic diffusion models from the viewpoint of scale-space research and show that they fulfil generalised scale-space properties on evolving probability distributions. Moreover, we discuss similarities and differences between interpretations of the physical core concept of drift-diffusion in the deep learning and model-based world. To this end, we examine relations of probabilistic diffusion to osmosis filters.
△ Less
Submitted 18 September, 2023; v1 submitted 14 March, 2023;
originally announced March 2023.
-
A Wasserstein GAN for Joint Learning of Inpainting and Spatial Optimisation
Authors:
Pascal Peter
Abstract:
Image inpainting is a restoration method that reconstructs missing image parts. However, a carefully selected mask of known pixels that yield a high quality inpainting can also act as a sparse image representation. This challenging spatial optimisation problem is essential for practical applications such as compression. So far, it has been almost exclusively adressed by model-based approaches. Fir…
▽ More
Image inpainting is a restoration method that reconstructs missing image parts. However, a carefully selected mask of known pixels that yield a high quality inpainting can also act as a sparse image representation. This challenging spatial optimisation problem is essential for practical applications such as compression. So far, it has been almost exclusively adressed by model-based approaches. First attempts with neural networks seem promising, but are tailored towards specific inpainting operators or require postprocessing.
To address this issue, we propose the first generative adversarial network (GAN) for spatial inpainting data optimisation. In contrast to previous approaches, it allows joint training of an inpainting generator and a corresponding mask optimisation network. With a Wasserstein distance, we ensure that our inpainting results accurately reflect the statistics of natural images. This yields significant improvements in visual quality and speed over conventional stochastic models. It also outperforms current spatial optimisation networks.
△ Less
Submitted 2 December, 2022; v1 submitted 11 February, 2022;
originally announced February 2022.
-
Learning Sparse Masks for Diffusion-based Image Inpainting
Authors:
Tobias Alt,
Pascal Peter,
Joachim Weickert
Abstract:
Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. Its quality strongly depends on the choice of known data. Optimising their spatial location -- the inpainting mask -- is challenging. A commonly used tool for this task are stochastic optimisation strategies. However, they are slow as they compute multiple inpainting results. We provide a remedy in ter…
▽ More
Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. Its quality strongly depends on the choice of known data. Optimising their spatial location -- the inpainting mask -- is challenging. A commonly used tool for this task are stochastic optimisation strategies. However, they are slow as they compute multiple inpainting results. We provide a remedy in terms of a learned mask generation model. By emulating the complete inpainting pipeline with two networks for mask generation and neural surrogate inpainting, we obtain a model for highly efficient adaptive mask generation. Experiments indicate that our model can achieve competitive quality with an acceleration by as much as four orders of magnitude. Our findings serve as a basis for making diffusion-based inpainting more attractive for applications such as image compression, where fast encoding is highly desirable.
△ Less
Submitted 16 May, 2022; v1 submitted 6 October, 2021;
originally announced October 2021.
-
Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods
Authors:
Tobias Alt,
Karl Schrader,
Joachim Weickert,
Pascal Peter,
Matthias Augustin
Abstract:
Partial differential equation (PDE) models and their associated variational energy formulations are often rotationally invariant by design. This ensures that a rotation of the input results in a corresponding rotation of the output, which is desirable in applications such as image analysis. Convolutional neural networks (CNNs) do not share this property, and existing remedies are often complex. Th…
▽ More
Partial differential equation (PDE) models and their associated variational energy formulations are often rotationally invariant by design. This ensures that a rotation of the input results in a corresponding rotation of the output, which is desirable in applications such as image analysis. Convolutional neural networks (CNNs) do not share this property, and existing remedies are often complex. The goal of our paper is to investigate how diffusion and variational models achieve rotation invariance and transfer these ideas to neural networks. As a core novelty we propose activation functions which couple network channels by combining information from several oriented filters. This guarantees rotation invariance within the basic building blocks of the networks while still allowing for directional filtering. The resulting neural architectures are inherently rotationally invariant. With only a few small filters, they can achieve the same invariance as existing techniques which require a fine-grained sampling of orientations. Our findings help to translate diffusion and variational models into mathematically well-founded network architectures, and provide novel concepts for model-based CNN design.
△ Less
Submitted 18 March, 2022; v1 submitted 31 August, 2021;
originally announced August 2021.
-
Connections between Numerical Algorithms for PDEs and Neural Networks
Authors:
Tobias Alt,
Karl Schrader,
Matthias Augustin,
Pascal Peter,
Joachim Weickert
Abstract:
We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalis…
▽ More
We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for develo** new mathematically well-founded neural building blocks.
△ Less
Submitted 21 March, 2022; v1 submitted 30 July, 2021;
originally announced July 2021.
-
Translating Numerical Concepts for PDEs into Neural Architectures
Authors:
Tobias Alt,
Pascal Peter,
Joachim Weickert,
Karl Schrader
Abstract:
We investigate what can be learned from translating numerical algorithms into neural networks. On the numerical side, we consider explicit, accelerated explicit, and implicit schemes for a general higher order nonlinear diffusion equation in 1D, as well as linear multigrid methods. On the neural network side, we identify corresponding concepts in terms of residual networks (ResNets), recurrent net…
▽ More
We investigate what can be learned from translating numerical algorithms into neural networks. On the numerical side, we consider explicit, accelerated explicit, and implicit schemes for a general higher order nonlinear diffusion equation in 1D, as well as linear multigrid methods. On the neural network side, we identify corresponding concepts in terms of residual networks (ResNets), recurrent networks, and U-nets. These connections guarantee Euclidean stability of specific ResNets with a transposed convolution layer structure in each block. We present three numerical justifications for skip connections: as time discretisations in explicit schemes, as extrapolation mechanisms for accelerating those methods, and as recurrent connections in fixed point solvers for implicit schemes. Last but not least, we also motivate uncommon design choices such as nonmonotone activation functions. Our findings give a numerical perspective on the success of modern neural network architectures, and they provide design criteria for stable networks.
△ Less
Submitted 17 May, 2021; v1 submitted 29 March, 2021;
originally announced March 2021.
-
Translating Diffusion, Wavelets, and Regularisation into Residual Networks
Authors:
Tobias Alt,
Joachim Weickert,
Pascal Peter
Abstract:
Convolutional neural networks (CNNs) often perform well, but their stability is poorly understood. To address this problem, we consider the simple prototypical problem of signal denoising, where classical approaches such as nonlinear diffusion, wavelet-based methods and regularisation offer provable stability guarantees. To transfer such guarantees to CNNs, we interpret numerical approximations of…
▽ More
Convolutional neural networks (CNNs) often perform well, but their stability is poorly understood. To address this problem, we consider the simple prototypical problem of signal denoising, where classical approaches such as nonlinear diffusion, wavelet-based methods and regularisation offer provable stability guarantees. To transfer such guarantees to CNNs, we interpret numerical approximations of these classical methods as a specific residual network (ResNet) architecture. This leads to a dictionary which allows to translate diffusivities, shrinkage functions, and regularisers into activation functions, and enables a direct communication between the four research communities. On the CNN side, it does not only inspire new families of nonmonotone activation functions, but also introduces intrinsically stable architectures for an arbitrary number of layers.
△ Less
Submitted 7 June, 2020; v1 submitted 7 February, 2020;
originally announced February 2020.
-
Space-Filling Curve Indices as Acceleration Structure for Exemplar-Based Inpainting
Authors:
Tim Dahmen,
Patrick Trampert,
Pascal Peter,
Pinak Bheed,
Joachim Weickert,
Philipp Slusallek
Abstract:
Exemplar-based inpainting is the process of reconstructing missing parts of an image by searching the remaining data for patches that fit seamlessly. The image is completed to a plausible-looking solution by repeatedly inserting the patch that is the best match according to some cost function. We present an acceleration structure that uses a multi-index scheme to accelerate this search procedure d…
▽ More
Exemplar-based inpainting is the process of reconstructing missing parts of an image by searching the remaining data for patches that fit seamlessly. The image is completed to a plausible-looking solution by repeatedly inserting the patch that is the best match according to some cost function. We present an acceleration structure that uses a multi-index scheme to accelerate this search procedure drastically, particularly in the case of very large datasets. The index scheme uses ideas such as dimensionality reduction and k-nearest neighbor search on space-filling curves that are well known in the field of multimedia databases. Our method has a theoretic runtime of O(log2 n) per iteration and reaches a speedup factor of up to 660 over the original method. The approach has the advantage of being agnostic to most modelbased parts of exemplar-based inpainting such as the order in which patches are processed and the cost function used to determine patch similarity. Thus, the acceleration structure can be used in conjunction with most exemplar-based inpainting algorithms.
△ Less
Submitted 22 January, 2020; v1 submitted 18 December, 2017;
originally announced December 2017.
-
Clustering-Based Quantisation for PDE-Based Image Compression
Authors:
Laurent Hoeltgen,
Pascal Peter,
Michael Breuß
Abstract:
Finding optimal data for inpainting is a key problem in the context of partial differential equation based image compression. The data that yields the most accurate reconstruction is real-valued. Thus, quantisation models are mandatory to allow an efficient encoding. These can also be understood as challenging data clustering problems. Although clustering approaches are well suited for this kind o…
▽ More
Finding optimal data for inpainting is a key problem in the context of partial differential equation based image compression. The data that yields the most accurate reconstruction is real-valued. Thus, quantisation models are mandatory to allow an efficient encoding. These can also be understood as challenging data clustering problems. Although clustering approaches are well suited for this kind of compression codecs, very few works actually consider them. Each pixel has a global impact on the reconstruction and optimal data locations are strongly correlated with their corresponding colour values. These facts make it hard to predict which feature works best.
In this paper we discuss quantisation strategies based on popular methods such as k-means. We are lead to the central question which kind of feature vectors are best suited for image compression. To this end we consider choices such as the pixel values, the histogram or the colour map.
Our findings show that the number of colours can be reduced significantly without impacting the reconstruction quality. Surprisingly, these benefits do not directly translate to a good image compression performance. The gains in the compression ratio are lost due to increased storage costs. This suggests that it is integral to evaluate the clustering on both, the reconstruction error and the final file size.
△ Less
Submitted 20 June, 2017;
originally announced June 2017.
-
SAHER: Secure and Efficient Routing in Sensor Networks
Authors:
Minko Dudev,
Sebastian Gerling,
Philip Peter
Abstract:
As an increasing amount of research is being done on various applications of sensor networks in adversarial environments, ensuring secure routing becomes of critical importance for the success of such deployments. The problem of designing a secure routing protocol for ad hoc networks has been already addressed, yet, there exists no complete solution that meets the specific requirements of sensor n…
▽ More
As an increasing amount of research is being done on various applications of sensor networks in adversarial environments, ensuring secure routing becomes of critical importance for the success of such deployments. The problem of designing a secure routing protocol for ad hoc networks has been already addressed, yet, there exists no complete solution that meets the specific requirements of sensor networks, where nodes are extremely constrained in terms of both power and computational resources. Thus, we propose a new protocol that is not built solely around security but also has efficiency and simplicity among its main goals. We propose the Secure Ad Hoc Efficient Routing protocol (SAHER) which employs a two-tier architecture based on node clustering. Also, we combine mechanisms like localscale geographic routing, per-node reputation tables, credit based alternate route enforcement and cumulative authentication. Using these techniques we examine ways to efficiently defend against the two most common network layer attacks: selective packet drop** and message flooding. Further, we consider join/leave operations which have not yet been studied in sufficient depth for sensor networks from a security standpoint. We provide a description of the protocol along with comprehensive experimental evaluation under different node distributions, different proportions of non-malicious vs. malicious nodes and different types of activity that malicious nodes could exhibit.
△ Less
Submitted 28 June, 2013;
originally announced July 2013.
