-
Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation
Authors:
Christian Reinbacher,
Gottfried Graber,
Thomas Pock
Abstract:
Event cameras or neuromorphic cameras mimic the human perception system as they measure the per-pixel intensity change rather than the actual intensity level. In contrast to traditional cameras, such cameras capture new information about the scene at MHz frequency in the form of sparse events. The high temporal resolution comes at the cost of losing the familiar per-pixel intensity information. In…
▽ More
Event cameras or neuromorphic cameras mimic the human perception system as they measure the per-pixel intensity change rather than the actual intensity level. In contrast to traditional cameras, such cameras capture new information about the scene at MHz frequency in the form of sparse events. The high temporal resolution comes at the cost of losing the familiar per-pixel intensity information. In this work we propose a variational model that accurately models the behaviour of event cameras, enabling reconstruction of intensity images with arbitrary frame rate in real-time. Our method is formulated on a per-event-basis, where we explicitly incorporate information about the asynchronous nature of events via an event manifold induced by the relative timestamps of events. In our experiments we verify that solving the variational model on the manifold produces high-quality images without explicitly estimating optical flow.
△ Less
Submitted 4 August, 2016; v1 submitted 21 July, 2016;
originally announced July 2016.
-
Techniques for Gradient Based Bilevel Optimization with Nonsmooth Lower Level Problems
Authors:
Peter Ochs,
René Ranftl,
Thomas Brox,
Thomas Pock
Abstract:
We propose techniques for approximating bilevel optimization problems with non-smooth lower level problems that can have a non-unique solution. To this end, we substitute the expression of a minimizer of the lower level minimization problem with an iterative algorithm that is guaranteed to converge to a minimizer of the problem. Using suitable non-linear proximal distance functions, the update map…
▽ More
We propose techniques for approximating bilevel optimization problems with non-smooth lower level problems that can have a non-unique solution. To this end, we substitute the expression of a minimizer of the lower level minimization problem with an iterative algorithm that is guaranteed to converge to a minimizer of the problem. Using suitable non-linear proximal distance functions, the update map**s of such an iterative algorithm can be differentiable, notwithstanding the fact that the minimization problem is non-smooth.
△ Less
Submitted 26 April, 2016; v1 submitted 23 February, 2016;
originally announced February 2016.
-
Solving Dense Image Matching in Real-Time using Discrete-Continuous Optimization
Authors:
Alexander Shekhovtsov,
Christian Reinbacher,
Gottfried Graber,
Thomas Pock
Abstract:
Dense image matching is a fundamental low-level problem in Computer Vision, which has received tremendous attention from both discrete and continuous optimization communities. The goal of this paper is to combine the advantages of discrete and continuous optimization in a coherent framework. We devise a model based on energy minimization, to be optimized by both discrete and continuous algorithms…
▽ More
Dense image matching is a fundamental low-level problem in Computer Vision, which has received tremendous attention from both discrete and continuous optimization communities. The goal of this paper is to combine the advantages of discrete and continuous optimization in a coherent framework. We devise a model based on energy minimization, to be optimized by both discrete and continuous algorithms in a consistent way. In the discrete setting, we propose a novel optimization algorithm that can be massively parallelized. In the continuous setting we tackle the problem of non-convex regularizers by a formulation based on differences of convex functions. The resulting hybrid discrete-continuous algorithm can be efficiently accelerated by modern GPUs and we demonstrate its real-time performance for the applications of dense stereo matching and optical flow.
△ Less
Submitted 23 January, 2016;
originally announced January 2016.
-
Acceleration of the PDHGM on strongly convex subspaces
Authors:
Tuomo Valkonen,
Thomas Pock
Abstract:
We propose several variants of the primal-dual method due to Chambolle and Pock. Without requiring full strong convexity of the objective functions, our methods are accelerated on subspaces with strong convexity. This yields mixed rates, $O(1/N^2)$ with respect to initialisation and $O(1/N)$ with respect to the dual sequence, and the residual part of the primal sequence. We demonstrate the efficac…
▽ More
We propose several variants of the primal-dual method due to Chambolle and Pock. Without requiring full strong convexity of the objective functions, our methods are accelerated on subspaces with strong convexity. This yields mixed rates, $O(1/N^2)$ with respect to initialisation and $O(1/N)$ with respect to the dual sequence, and the residual part of the primal sequence. We demonstrate the efficacy of the proposed methods on image processing problems lacking strong convexity, such as total generalised variation denoising and total variation deblurring.
△ Less
Submitted 10 February, 2016; v1 submitted 20 November, 2015;
originally announced November 2015.
-
Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration
Authors:
Yun** Chen,
Thomas Pock
Abstract:
Image restoration is a long-standing problem in low-level computer vision with many interesting applications. We describe a flexible learning framework based on the concept of nonlinear reaction diffusion models for various image restoration problems. By embodying recent improvements in nonlinear diffusion models, we propose a dynamic nonlinear reaction diffusion model with time-dependent paramete…
▽ More
Image restoration is a long-standing problem in low-level computer vision with many interesting applications. We describe a flexible learning framework based on the concept of nonlinear reaction diffusion models for various image restoration problems. By embodying recent improvements in nonlinear diffusion models, we propose a dynamic nonlinear reaction diffusion model with time-dependent parameters (\ie, linear filters and influence functions). In contrast to previous nonlinear diffusion models, all the parameters, including the filters and the influence functions, are simultaneously learned from training data through a loss based approach. We call this approach TNRD -- \textit{Trainable Nonlinear Reaction Diffusion}. The TNRD approach is applicable for a variety of image restoration tasks by incorporating appropriate reaction force. We demonstrate its capabilities with three representative applications, Gaussian image denoising, single image super resolution and JPEG deblocking. Experiments show that our trained nonlinear diffusion models largely benefit from the training of the parameters and finally lead to the best reported performance on common test datasets for the tested applications. Our trained models preserve the structural simplicity of diffusion models and take only a small number of diffusion steps, thus are highly efficient. Moreover, they are also well-suited for parallel computation on GPUs, which makes the inference procedure extremely fast.
△ Less
Submitted 20 August, 2016; v1 submitted 12 August, 2015;
originally announced August 2015.
-
On learning optimized reaction diffusion processes for effective image restoration
Authors:
Yun** Chen,
Wei Yu,
Thomas Pock
Abstract:
For several decades, image restoration remains an active research topic in low-level computer vision and hence new approaches are constantly emerging. However, many recently proposed algorithms achieve state-of-the-art performance only at the expense of very high computation time, which clearly limits their practical relevance. In this work, we propose a simple but effective approach with both hig…
▽ More
For several decades, image restoration remains an active research topic in low-level computer vision and hence new approaches are constantly emerging. However, many recently proposed algorithms achieve state-of-the-art performance only at the expense of very high computation time, which clearly limits their practical relevance. In this work, we propose a simple but effective approach with both high computational efficiency and high restoration quality. We extend conventional nonlinear reaction diffusion models by several parametrized linear filters as well as several parametrized influence functions. We propose to train the parameters of the filters and the influence functions through a loss based approach. Experiments show that our trained nonlinear reaction diffusion models largely benefit from the training of the parameters and finally lead to the best reported performance on common test datasets for image restoration. Due to their structural simplicity, our trained models are highly efficient and are also well-suited for parallel computation on GPUs.
△ Less
Submitted 25 March, 2015; v1 submitted 19 March, 2015;
originally announced March 2015.
-
Total variation on a tree
Authors:
Vladimir Kolmogorov,
Thomas Pock,
Michal Rolinek
Abstract:
We consider the problem of minimizing the continuous valued total variation subject to different unary terms on trees and propose fast direct algorithms based on dynamic programming to solve these problems. We treat both the convex and the non-convex case and derive worst case complexities that are equal or better than existing methods. We show applications to total variation based 2D image proces…
▽ More
We consider the problem of minimizing the continuous valued total variation subject to different unary terms on trees and propose fast direct algorithms based on dynamic programming to solve these problems. We treat both the convex and the non-convex case and derive worst case complexities that are equal or better than existing methods. We show applications to total variation based 2D image processing and computer vision problems based on a Lagrangian decomposition approach. The resulting algorithms are very efficient, offer a high degree of parallelism and come along with memory requirements which are only in the order of the number of image pixels.
△ Less
Submitted 25 April, 2016; v1 submitted 26 February, 2015;
originally announced February 2015.
-
Real-time flare detection in ground-based H$α$ imaging at Kanzelhöhe Observatory
Authors:
Werner Pötzi,
Astrid M. Veronig,
Gernot Riegler,
Ute Amerstorfer,
Thomas Pock,
Manuela Temmer,
Wolfgang Polanec,
Dietmar J. Baumgartner
Abstract:
Kanzelhöhe Observatory (KSO) regularly performs high-cadence full-disk imaging of the solar chromosphere in the H$α$ and CaIIK spectrallines as well as the solar photosphere in white-light. In the frame of ESA's Space Situational Awareness (SSA) programme, a new system for real-time H$α$ data provision and automatic flare detection was developed at KSO. The data and events detected are published i…
▽ More
Kanzelhöhe Observatory (KSO) regularly performs high-cadence full-disk imaging of the solar chromosphere in the H$α$ and CaIIK spectrallines as well as the solar photosphere in white-light. In the frame of ESA's Space Situational Awareness (SSA) programme, a new system for real-time H$α$ data provision and automatic flare detection was developed at KSO. The data and events detected are published in near real-time at ESA's SSA Space Weather portal (http://swe.ssa.esa.int/web/guest/kso-federated). In this paper, we describe the H$α$ instrument, the image recognition algorithms developed, the implementation into the KSO H$α$ observing system and present the evaluation results of the real-time data provision and flare detection for a period of five months. The H$α$ data provision worked in $99.96$% of the images, with a mean time lag between image recording and online provision of 4s. Within the given criteria for the automatic image recognition system (at least three H$α$ images are needed for a positive detection), all flares with an area $\ge$50 micro-hemispheres and located within $60^\circ$ of the Sun's center that occurred during the KSO observing times were detected, in total a number of 87 events. The automatically determined flare importance and brightness classes were correct in $\sim$85%. The mean flare positions in heliographic longitude and latitude were correct within $\sim$1$^\circ$. The median of the absolute differences for the flare start times and peak times from the automatic detections in comparison to the official NOAA (and KSO) visual flare reports were 3 min (1 min).
△ Less
Submitted 18 November, 2014; v1 submitted 14 November, 2014;
originally announced November 2014.
-
A higher-order MRF based variational model for multiplicative noise reduction
Authors:
Yun** Chen,
Wensen Feng,
René Ranftl,
Hong Qiao,
Thomas Pock
Abstract:
The Fields of Experts (FoE) image prior model, a filter-based higher-order Markov Random Fields (MRF) model, has been shown to be effective for many image restoration problems. Motivated by the successes of FoE-based approaches, in this letter, we propose a novel variational model for multiplicative noise reduction based on the FoE image prior model. The resulted model corresponds to a non-convex…
▽ More
The Fields of Experts (FoE) image prior model, a filter-based higher-order Markov Random Fields (MRF) model, has been shown to be effective for many image restoration problems. Motivated by the successes of FoE-based approaches, in this letter, we propose a novel variational model for multiplicative noise reduction based on the FoE image prior model. The resulted model corresponds to a non-convex minimization problem, which can be solved by a recently published non-convex optimization algorithm. Experimental results based on synthetic speckle noise and real synthetic aperture radar (SAR) images suggest that the performance of our proposed method is on par with the best published despeckling algorithm. Besides, our proposed model comes along with an additional advantage, that the inference is extremely efficient. {Our GPU based implementation takes less than 1s to produce state-of-the-art despeckling performance.}
△ Less
Submitted 7 July, 2014; v1 submitted 21 April, 2014;
originally announced April 2014.
-
iPiano: Inertial Proximal Algorithm for Non-Convex Optimization
Authors:
Peter Ochs,
Yun** Chen,
Thomas Brox,
Thomas Pock
Abstract:
In this paper we study an algorithm for solving a minimization problem composed of a differentiable (possibly non-convex) and a convex (possibly non-differentiable) function. The algorithm iPiano combines forward-backward splitting with an inertial force. It can be seen as a non-smooth split version of the Heavy-ball method from Polyak. A rigorous analysis of the algorithm for the proposed class o…
▽ More
In this paper we study an algorithm for solving a minimization problem composed of a differentiable (possibly non-convex) and a convex (possibly non-differentiable) function. The algorithm iPiano combines forward-backward splitting with an inertial force. It can be seen as a non-smooth split version of the Heavy-ball method from Polyak. A rigorous analysis of the algorithm for the proposed class of problems yields global convergence of the function values and the arguments. This makes the algorithm robust for usage on non-convex problems. The convergence result is obtained based on the \KL inequality. This is a very weak restriction, which was used to prove convergence for several other gradient methods. First, an abstract convergence theorem for a generic algorithm is proved, and, then iPiano is shown to satisfy the requirements of this theorem. Furthermore, a convergence rate is established for the general problem class. We demonstrate iPiano on computer vision problems: image denoising with learned priors and diffusion based image compression.
△ Less
Submitted 18 April, 2014;
originally announced April 2014.
-
An inertial forward-backward algorithm for monotone inclusions
Authors:
Dirk A. Lorenz,
Thomas Pock
Abstract:
In this paper, we propose an inertial forward backward splitting algorithm to compute a zero of the sum of two monotone operators, with one of the two operators being co-coercive. The algorithm is inspired by the accelerated gradient method of Nesterov, but can be applied to a much larger class of problems including convex-concave saddle point problems and general monotone inclusions. We prove con…
▽ More
In this paper, we propose an inertial forward backward splitting algorithm to compute a zero of the sum of two monotone operators, with one of the two operators being co-coercive. The algorithm is inspired by the accelerated gradient method of Nesterov, but can be applied to a much larger class of problems including convex-concave saddle point problems and general monotone inclusions. We prove convergence of the algorithm in a Hilbert space setting and show that several recently proposed first-order methods can be obtained as special cases of the general algorithm. Numerical results show that the proposed algorithm converges faster than existing methods, while kee** the computational cost of each iteration basically unchanged.
△ Less
Submitted 12 September, 2014; v1 submitted 14 March, 2014;
originally announced March 2014.
-
A bi-level view of inpainting - based image compression
Authors:
Yun** Chen,
René Ranftl,
Thomas Pock
Abstract:
Inpainting based image compression approaches, especially linear and non-linear diffusion models, are an active research topic for lossy image compression. The major challenge in these compression models is to find a small set of descriptive supporting points, which allow for an accurate reconstruction of the original image. It turns out in practice that this is a challenging problem even for the…
▽ More
Inpainting based image compression approaches, especially linear and non-linear diffusion models, are an active research topic for lossy image compression. The major challenge in these compression models is to find a small set of descriptive supporting points, which allow for an accurate reconstruction of the original image. It turns out in practice that this is a challenging problem even for the simplest Laplacian interpolation model. In this paper, we revisit the Laplacian interpolation compression model and introduce two fast algorithms, namely successive preconditioning primal dual algorithm and the recently proposed iPiano algorithm, to solve this problem efficiently. Furthermore, we extend the Laplacian interpolation based compression model to a more general form, which is based on principles from bi-level optimization. We investigate two different variants of the Laplacian model, namely biharmonic interpolation and smoothed Total Variation regularization. Our numerical results show that significant improvements can be obtained from the biharmonic interpolation model, and it can recover an image with very high quality from only 5% pixels.
△ Less
Submitted 9 May, 2014; v1 submitted 16 January, 2014;
originally announced January 2014.
-
Revisiting loss-specific training of filter-based MRFs for image restoration
Authors:
Yun** Chen,
Thomas Pock,
René Ranftl,
Horst Bischof
Abstract:
It is now well known that Markov random fields (MRFs) are particularly effective for modeling image priors in low-level vision. Recent years have seen the emergence of two main approaches for learning the parameters in MRFs: (1) probabilistic learning using sampling-based algorithms and (2) loss-specific training based on MAP estimate. After investigating existing training approaches, it turns out…
▽ More
It is now well known that Markov random fields (MRFs) are particularly effective for modeling image priors in low-level vision. Recent years have seen the emergence of two main approaches for learning the parameters in MRFs: (1) probabilistic learning using sampling-based algorithms and (2) loss-specific training based on MAP estimate. After investigating existing training approaches, it turns out that the performance of the loss-specific training has been significantly underestimated in existing work. In this paper, we revisit this approach and use techniques from bi-level optimization to solve it. We show that we can get a substantial gain in the final performance by solving the lower-level problem in the bi-level framework with high accuracy using our newly proposed algorithm. As a result, our trained model is on par with highly specialized image denoising algorithms and clearly outperforms probabilistically trained MRF models. Our findings suggest that for the loss-specific training scheme, solving the lower-level problem with higher accuracy is beneficial. Our trained model comes along with the additional advantage, that inference is extremely efficient. Our GPU-based implementation takes less than 1s to produce state-of-the-art performance.
△ Less
Submitted 16 January, 2014;
originally announced January 2014.
-
Learning $\ell_1$-based analysis and synthesis sparsity priors using bi-level optimization
Authors:
Yun** Chen,
Thomas Pock,
Horst Bischof
Abstract:
We consider the analysis operator and synthesis dictionary learning problems based on the the $\ell_1$ regularized sparse representation model. We reveal the internal relations between the $\ell_1$-based analysis model and synthesis model. We then introduce an approach to learn both analysis operator and synthesis dictionary simultaneously by using a unified framework of bi-level optimization. Our…
▽ More
We consider the analysis operator and synthesis dictionary learning problems based on the the $\ell_1$ regularized sparse representation model. We reveal the internal relations between the $\ell_1$-based analysis model and synthesis model. We then introduce an approach to learn both analysis operator and synthesis dictionary simultaneously by using a unified framework of bi-level optimization. Our aim is to learn a meaningful operator (dictionary) such that the minimum energy solution of the analysis (synthesis)-prior based model is as close as possible to the ground-truth. We solve the bi-level optimization problem using the implicit differentiation technique. Moreover, we demonstrate the effectiveness of our leaning approach by applying the learned analysis operator (dictionary) to the image denoising task and comparing its performance with state-of-the-art methods. Under this unified framework, we can compare the performance of the two types of priors.
△ Less
Submitted 16 January, 2014;
originally announced January 2014.
-
Insights into analysis operator learning: From patch-based sparse models to higher-order MRFs
Authors:
Yun** Chen,
René Ranftl,
Thomas Pock
Abstract:
This paper addresses a new learning algorithm for the recently introduced co-sparse analysis model. First, we give new insights into the co-sparse analysis model by establishing connections to filter-based MRF models, such as the Field of Experts (FoE) model of Roth and Black. For training, we introduce a technique called bi-level optimization to learn the analysis operators. Compared to existing…
▽ More
This paper addresses a new learning algorithm for the recently introduced co-sparse analysis model. First, we give new insights into the co-sparse analysis model by establishing connections to filter-based MRF models, such as the Field of Experts (FoE) model of Roth and Black. For training, we introduce a technique called bi-level optimization to learn the analysis operators. Compared to existing analysis operator learning approaches, our training procedure has the advantage that it is unconstrained with respect to the analysis operator. We investigate the effect of different aspects of the co-sparse analysis model and show that the sparsity promoting function (also called penalty function) is the most important factor in the model. In order to demonstrate the effectiveness of our training approach, we apply our trained models to various classical image restoration problems. Numerical experiments show that our trained models clearly outperform existing analysis operator learning approaches and are on par with state-of-the-art image denoising algorithms. Our approach develops a framework that is intuitive to understand and easy to implement.
△ Less
Submitted 13 January, 2014;
originally announced January 2014.
-
A Convex Approach for Image Hallucination
Authors:
Peter Innerhofer,
Thomas Pock
Abstract:
In this paper we propose a global convex approach for image hallucination. Altering the idea of classical multi image super resolution (SU) systems to single image SU, we incorporate aligned images to hallucinate the output. Our work is based on the paper of Tappen et al. where they use a non-convex model for image hallucination. In comparison we formulate a convex primal optimization problem and…
▽ More
In this paper we propose a global convex approach for image hallucination. Altering the idea of classical multi image super resolution (SU) systems to single image SU, we incorporate aligned images to hallucinate the output. Our work is based on the paper of Tappen et al. where they use a non-convex model for image hallucination. In comparison we formulate a convex primal optimization problem and derive a fast converging primal-dual algorithm with a global optimal solution. We use a database with face images to incorporate high-frequency details to the high-resolution output. We show that we can achieve state-of-the-art results by using a convex approach.
△ Less
Submitted 26 April, 2013;
originally announced April 2013.
-
Filament and Flare Detection in Hα image sequences
Authors:
Gernot Riegler,
Thomas Pock,
Werner Pötzi,
Astrid Veronig
Abstract:
Solar storms can have a major impact on the infrastructure of the earth. Some of the causing events are observable from ground in the Hα spectral line. In this paper we propose a new method for the simultaneous detection of flares and filaments in Hα image sequences. Therefore we perform several preprocessing steps to enhance and normalize the images. Based on the intensity values we segment the i…
▽ More
Solar storms can have a major impact on the infrastructure of the earth. Some of the causing events are observable from ground in the Hα spectral line. In this paper we propose a new method for the simultaneous detection of flares and filaments in Hα image sequences. Therefore we perform several preprocessing steps to enhance and normalize the images. Based on the intensity values we segment the image by a variational approach. In a final postprecessing step we derive essential properties to classify the events and further demonstrate the performance by comparing our obtained results to the data annotated by an expert. The information produced by our method can be used for near real-time alerts and the statistical analysis of existing data by solar physicists.
△ Less
Submitted 26 April, 2013;
originally announced April 2013.
