-
Prediction techniques for dynamic imaging with online primal-dual methods
Authors:
Neil Dizon,
Jyrki Jauhiainen,
Tuomo Valkonen
Abstract:
Online optimisation facilitates the solution of dynamic inverse problems, such as image stabilisation, fluid flow monitoring, and dynamic medical imaging. In this paper, we improve upon previous work on predictive online primal-dual methods on two fronts. Firstly, we provide a more concise analysis that symmetrises previously unsymmetric regret bounds, and relaxes previous restrictive conditions o…
▽ More
Online optimisation facilitates the solution of dynamic inverse problems, such as image stabilisation, fluid flow monitoring, and dynamic medical imaging. In this paper, we improve upon previous work on predictive online primal-dual methods on two fronts. Firstly, we provide a more concise analysis that symmetrises previously unsymmetric regret bounds, and relaxes previous restrictive conditions on the dual predictor. Secondly, based on the latter, we develop several improved dual predictors. We numerically demonstrate their efficacy in image stabilisation and dynamic positron emission tomography.
△ Less
Submitted 5 July, 2024; v1 submitted 3 May, 2024;
originally announced May 2024.
-
Proximal methods for point source localisation
Authors:
Tuomo Valkonen
Abstract:
Point source localisation is generally modelled as a Lasso-type problem on measures. However, optimisation methods in non-Hilbert spaces, such as the space of Radon measures, are much less developed than in Hilbert spaces. Most numerical algorithms for point source localisation are based on the Frank-Wolfe conditional gradient method, for which ad hoc convergence theory is developed. We develop ex…
▽ More
Point source localisation is generally modelled as a Lasso-type problem on measures. However, optimisation methods in non-Hilbert spaces, such as the space of Radon measures, are much less developed than in Hilbert spaces. Most numerical algorithms for point source localisation are based on the Frank-Wolfe conditional gradient method, for which ad hoc convergence theory is developed. We develop extensions of proximal-type methods to spaces of measures. This includes forward-backward splitting, its inertial version, and primal-dual proximal splitting. Their convergence proofs follow standard patterns. We demonstrate their numerical efficacy.
△ Less
Submitted 20 September, 2023; v1 submitted 6 December, 2022;
originally announced December 2022.
-
A nonsmooth primal-dual method with interwoven PDE constraint solver
Authors:
Bjørn Jensen,
Tuomo Valkonen
Abstract:
We introduce an efficient first-order primal-dual method for the solution of nonsmooth PDE-constrained optimization problems. We achieve this efficiency through not solving the PDE or its linearisation on each iteration of the optimization method. Instead, we run the method interwoven with a simple conventional linear system solver (Jacobi, Gauss-Seidel, conjugate gradients), always taking only on…
▽ More
We introduce an efficient first-order primal-dual method for the solution of nonsmooth PDE-constrained optimization problems. We achieve this efficiency through not solving the PDE or its linearisation on each iteration of the optimization method. Instead, we run the method interwoven with a simple conventional linear system solver (Jacobi, Gauss-Seidel, conjugate gradients), always taking only one step of the linear system solver for each step of the optimization method. The control parameter is updated on each iteration as determined by the optimization method. We prove linear convergence under a second-order growth condition, and numerically demonstrate the performance on a variety of PDEs related to inverse problems involving boundary measurements.
△ Less
Submitted 23 May, 2024; v1 submitted 9 November, 2022;
originally announced November 2022.
-
Linearly convergent bilevel optimization with single-step inner methods
Authors:
Ensio Suonperä,
Tuomo Valkonen
Abstract:
We propose a new approach to solving bilevel optimization problems, intermediate between solving full-system optimality conditions with a Newton-type approach, and treating the inner problem as an implicit function. The overall idea is to solve the full-system optimality conditions, but to precondition them to alternate between taking steps of simple conventional methods for the inner problem, the…
▽ More
We propose a new approach to solving bilevel optimization problems, intermediate between solving full-system optimality conditions with a Newton-type approach, and treating the inner problem as an implicit function. The overall idea is to solve the full-system optimality conditions, but to precondition them to alternate between taking steps of simple conventional methods for the inner problem, the adjoint equation, and the outer problem. While the inner objective has to be smooth, the outer objective may be nonsmooth subject to a prox-contractivity condition. We prove linear convergence of the approach for combinations of gradient descent and forward-backward splitting with exact and inexact solution of the adjoint equation. We demonstrate good performance on learning the regularization parameter for anisotropic total variation image denoising, and the convolution kernel for image deconvolution.
△ Less
Submitted 29 May, 2023; v1 submitted 10 May, 2022;
originally announced May 2022.
-
Mumford-Shah regularization in electrical impedance tomography with complete electrode model
Authors:
Jyrki Jauhiainen,
Aku Seppänen,
Tuomo Valkonen
Abstract:
In electrical impedance tomography, we aim to solve the conductivity within a target body through electrical measurements made on the surface of the target. This inverse conductivity problem is severely ill-posed, especially in real applications with only partial boundary data available. Thus regularization has to be introduced. Conventionally regularization promoting smooth features is used, howe…
▽ More
In electrical impedance tomography, we aim to solve the conductivity within a target body through electrical measurements made on the surface of the target. This inverse conductivity problem is severely ill-posed, especially in real applications with only partial boundary data available. Thus regularization has to be introduced. Conventionally regularization promoting smooth features is used, however, the Mumford--Shah regularizer familiar for image segmentation is more appropriate for targets consisting of several distinct objects or materials. It is, however, numerically challenging. We show theoretically through $Γ$-convergence that a modification of the Ambrosio--Tortorelli approximation of the Mumford--Shah regularizer is applicable to electrical impedance tomography, in particular the complete electrode model of boundary measurements. With numerical and experimental studies, we confirm that this functional works in practice and produces higher quality results than typical regularizations employed in electrical impedance tomography when the conductivity of the target consists of distinct smoothly-varying regions.
△ Less
Submitted 23 May, 2022; v1 submitted 29 October, 2021;
originally announced October 2021.
-
Electrodeless electrode model for electrical impedance tomography
Authors:
J. Dardé,
N. Hyvönen,
T. Kuutela,
T. Valkonen
Abstract:
Electrical impedance tomography is an imaging modality for extracting information on the interior structure of a physical body from boundary measurements of current and voltage. This work studies a new robust way of modeling the contact electrodes used for driving current patterns into the examined object and measuring the resulting voltages. The idea is to not define the electrodes as strict geom…
▽ More
Electrical impedance tomography is an imaging modality for extracting information on the interior structure of a physical body from boundary measurements of current and voltage. This work studies a new robust way of modeling the contact electrodes used for driving current patterns into the examined object and measuring the resulting voltages. The idea is to not define the electrodes as strict geometric objects on the measurement boundary, but only to assume approximate knowledge about their whereabouts and let a boundary admittivity function determine the actual locations of the current inputs. Such an approach enables reconstructing the boundary admittivity, i.e. the locations and strengths of the contacts, at the same time and with analogous methods as the interior admittivity. The functionality of the new model is verified by two-dimensional numerical experiments based on water tank data.
△ Less
Submitted 3 February, 2021;
originally announced February 2021.
-
Non-planar sensing skins for structural health monitoring based on electrical resistance tomography
Authors:
Jyrki Jauhiainen,
Mohammad Pour-Ghaz,
Tuomo Valkonen,
Aku Seppänen
Abstract:
Electrical resistance tomography (ERT) -based distributed surface sensing systems, or sensing skins, offer alternative sensing techniques for structural health monitoring, providing capabilities for distributed sensing of, for example, damage, strain and temperature. Currently, however, the computational techniques utilized for sensing skins are limited to planar surfaces. In this paper, to overco…
▽ More
Electrical resistance tomography (ERT) -based distributed surface sensing systems, or sensing skins, offer alternative sensing techniques for structural health monitoring, providing capabilities for distributed sensing of, for example, damage, strain and temperature. Currently, however, the computational techniques utilized for sensing skins are limited to planar surfaces. In this paper, to overcome this limitation, we generalize the ERT-based surface sensing to non-planar surfaces covering arbitrarily shaped three-dimensional structures; We construct a framework in which we reformulate the image reconstruction problem of ERT using techniques of Riemannian geometry, and solve the resulting problem numerically. We test this framework in series of numerical and experimental studies. The results demonstrate that the feasibility of the proposed formulation and the applicability of ERT-based sensing skins for non-planar geometries.
△ Less
Submitted 8 December, 2020;
originally announced December 2020.
-
Regularisation, optimisation, subregularity
Authors:
Tuomo Valkonen
Abstract:
Regularisation theory in Banach spaces, and non--norm-squared regularisation even in finite dimensions, generally relies upon Bregman divergences to replace norm convergence. This is comparable to the extension of first-order optimisation methods to Banach spaces. Bregman divergences can, however, be somewhat suboptimal in terms of descriptiveness. Using the concept of (strong) metric subregularit…
▽ More
Regularisation theory in Banach spaces, and non--norm-squared regularisation even in finite dimensions, generally relies upon Bregman divergences to replace norm convergence. This is comparable to the extension of first-order optimisation methods to Banach spaces. Bregman divergences can, however, be somewhat suboptimal in terms of descriptiveness. Using the concept of (strong) metric subregularity, previously used to prove the fast local convergence of optimisation methods, we show norm convergence in Banach spaces and for non--norm-squared regularisation. For problems such as total variation regularised image reconstruction, the metric subregularity reduces to a geometric condition on the ground truth: flat areas in the ground truth have to compensate for the fidelity term not having second-order growth within the kernel of the forward operator. Our approach to proving such regularisation results is based on optimisation formulations of inverse problems. As a side result of the regularisation theory that we develop, we provide regularisation complexity results for optimisation methods: how many steps $N_δ$ of the algorithm do we have to take for the approximate solutions to converge as the corruption level $δ\searrow 0$?
△ Less
Submitted 17 January, 2021; v1 submitted 15 November, 2020;
originally announced November 2020.
-
Inverse problems with second-order Total Generalized Variation constraints
Authors:
Kristian Bredies,
Tuomo Valkonen
Abstract:
Total Generalized Variation (TGV) has recently been introduced as penalty functional for modelling images with edges as well as smooth variations. It can be interpreted as a "sparse" penalization of optimal balancing from the first up to the $k$-th distributional derivative and leads to desirable results when applied to image denoising, i.e., $L^2$-fitting with TGV penalty. The present paper studi…
▽ More
Total Generalized Variation (TGV) has recently been introduced as penalty functional for modelling images with edges as well as smooth variations. It can be interpreted as a "sparse" penalization of optimal balancing from the first up to the $k$-th distributional derivative and leads to desirable results when applied to image denoising, i.e., $L^2$-fitting with TGV penalty. The present paper studies TGV of second order in the context of solving ill-posed linear inverse problems. Existence and stability for solutions of Tikhonov-functional minimization with respect to the data is shown and applied to the problem of recovering an image from blurred and noisy data.
△ Less
Submitted 19 May, 2020;
originally announced May 2020.
-
Relaxed Gauss-Newton methods with applications to electrical impedance tomography
Authors:
Jyrki Jauhiainen,
Petri Kuusela,
Aku Seppänen,
Tuomo Valkonen
Abstract:
As second-order methods, Gauss--Newton-type methods can be more effective than first-order methods for the solution of nonsmooth optimization problems with expensive-to-evaluate smooth components. Such methods, however, often do not converge. Motivated by nonlinear inverse problems with nonsmooth regularization, we propose a new Gauss--Newton-type method with inexact relaxed steps. We prove that t…
▽ More
As second-order methods, Gauss--Newton-type methods can be more effective than first-order methods for the solution of nonsmooth optimization problems with expensive-to-evaluate smooth components. Such methods, however, often do not converge. Motivated by nonlinear inverse problems with nonsmooth regularization, we propose a new Gauss--Newton-type method with inexact relaxed steps. We prove that the method converges to a set of disjoint critical points given that the linearisation of the forward operator for the inverse problem is sufficiently precise. We extensively evaluate the performance of the method on electrical impedance tomography (EIT).
△ Less
Submitted 12 May, 2020; v1 submitted 19 February, 2020;
originally announced February 2020.
-
Predictive online optimisation with applications to optical flow
Authors:
Tuomo Valkonen
Abstract:
Online optimisation revolves around new data being introduced into a problem while it is still being solved; think of deep learning as more training samples become available. We adapt the idea to dynamic inverse problems such as video processing with optical flow. We introduce a corresponding predictive online primal-dual proximal splitting method. The video frames now exactly correspond to the al…
▽ More
Online optimisation revolves around new data being introduced into a problem while it is still being solved; think of deep learning as more training samples become available. We adapt the idea to dynamic inverse problems such as video processing with optical flow. We introduce a corresponding predictive online primal-dual proximal splitting method. The video frames now exactly correspond to the algorithm iterations. A user-prescribed predictor describes the evolution of the primal variable. To prove convergence we need a predictor for the dual variable based on (proximal) gradient flow. This affects the model that the method asymptotically minimises. We show that for inverse problems the effect is, essentially, to construct a new dynamic regulariser based on infimal convolution of the static regularisers with the temporal coupling. We finish by demonstrating excellent real-time performance of our method in computational image stabilisation and convergence in terms of regularisation theory.
△ Less
Submitted 3 August, 2020; v1 submitted 7 February, 2020;
originally announced February 2020.
-
Introduction to Nonsmooth Analysis and Optimization
Authors:
Christian Clason,
Tuomo Valkonen
Abstract:
This book aims to give an introduction to generalized derivative concepts useful in deriving necessary optimality conditions and numerical algorithms for infinite-dimensional nondifferentiable optimization problems that arise in inverse problems, imaging, and PDE-constrained optimization. They cover convex subdifferentials, Fenchel duality, monotone operators and resolvents, Moreau--Yosida regular…
▽ More
This book aims to give an introduction to generalized derivative concepts useful in deriving necessary optimality conditions and numerical algorithms for infinite-dimensional nondifferentiable optimization problems that arise in inverse problems, imaging, and PDE-constrained optimization. They cover convex subdifferentials, Fenchel duality, monotone operators and resolvents, Moreau--Yosida regularization as well as Clarke and (briefly) limiting subdifferentials. Both first-order (proximal point and splitting) methods and second-order (semismooth Newton) methods are treated. In addition, differentiation of set-valued map** is discussed and used for deriving second-order optimality conditions for as well as Lipschitz stability properties of minimizers. The required background from functional analysis and calculus of variations is also briefly summarized.
△ Less
Submitted 26 July, 2023; v1 submitted 1 January, 2020;
originally announced January 2020.
-
Primal-dual block-proximal splitting for a class of non-convex problems
Authors:
Stanislav Mazurenko,
Jyrki Jauhiainen,
Tuomo Valkonen
Abstract:
We develop block structure adapted primal-dual algorithms for non-convex non-smooth optimisation problems whose objectives can be written as compositions $G(x)+F(K(x))$ of non-smooth block-separable convex functions $G$ and $F$ with a non-linear Lipschitz-differentiable operator $K$. Our methods are refinements of the non-linear primal-dual proximal splitting method for such problems without the b…
▽ More
We develop block structure adapted primal-dual algorithms for non-convex non-smooth optimisation problems whose objectives can be written as compositions $G(x)+F(K(x))$ of non-smooth block-separable convex functions $G$ and $F$ with a non-linear Lipschitz-differentiable operator $K$. Our methods are refinements of the non-linear primal-dual proximal splitting method for such problems without the block structure, which itself is based on the primal-dual proximal splitting method of Chambolle and Pock for convex problems. We propose individual step length parameters and acceleration rules for each of the primal and dual blocks of the problem. This allows them to convergence faster by adapting to the structure of the problem. For the squared distance of the iterates to a critical point, we show local $O(1/N)$, $O(1/N^2)$ and linear rates under varying conditions and choices of the step lengths parameters. Finally, we demonstrate the performance of the methods on practical inverse problems: diffusion tensor imaging and electrical impedance tomography.
△ Less
Submitted 22 April, 2020; v1 submitted 14 November, 2019;
originally announced November 2019.
-
First-order primal-dual methods for nonsmooth nonconvex optimisation
Authors:
Tuomo Valkonen
Abstract:
We provide an overview of primal-dual algorithms for nonsmooth and non-convex-concave saddle-point problems. This flows around a new analysis of such methods, using Bregman divergences to formulate simplified conditions for convergence.
We provide an overview of primal-dual algorithms for nonsmooth and non-convex-concave saddle-point problems. This flows around a new analysis of such methods, using Bregman divergences to formulate simplified conditions for convergence.
△ Less
Submitted 5 August, 2020; v1 submitted 30 September, 2019;
originally announced October 2019.
-
Primal-dual proximal splitting and generalized conjugation in non-smooth non-convex optimization
Authors:
Christian Clason,
Stanislav Mazurenko,
Tuomo Valkonen
Abstract:
We demonstrate that difficult non-convex non-smooth optimization problems, such as Nash equilibrium problems and anisotropic as well as isotropic Potts segmentation model, can be written in terms of generalized conjugates of convex functionals. These, in turn, can be formulated as saddle-point problems involving convex non-smooth functionals and a general smooth but non-bilinear coupling term. We…
▽ More
We demonstrate that difficult non-convex non-smooth optimization problems, such as Nash equilibrium problems and anisotropic as well as isotropic Potts segmentation model, can be written in terms of generalized conjugates of convex functionals. These, in turn, can be formulated as saddle-point problems involving convex non-smooth functionals and a general smooth but non-bilinear coupling term. We then show through detailed convergence analysis that a conceptually straightforward extension of the primal--dual proximal splitting method of Chambolle and Pock is applicable to the solution of such problems. Under sufficient local strong convexity assumptions of the functionals -- but still with a non-bilinear coupling term -- we even demonstrate local linear convergence of the method. We illustrate these theoretical results numerically on the aforementioned example problems.
△ Less
Submitted 19 March, 2020; v1 submitted 9 January, 2019;
originally announced January 2019.
-
Inertial, corrected, primal-dual proximal splitting
Authors:
Tuomo Valkonen
Abstract:
We study inertial versions of primal-dual proximal splitting, also known as the Chambolle--Pock method. Our starting point is the preconditioned proximal point formulation of this method. By adding correctors corresponding to the anti-symmetric part of the relevant monotone operator, using a FISTA-style gap unrolling argument, we are able to derive gap estimates instead of merely ergodic gap estim…
▽ More
We study inertial versions of primal-dual proximal splitting, also known as the Chambolle--Pock method. Our starting point is the preconditioned proximal point formulation of this method. By adding correctors corresponding to the anti-symmetric part of the relevant monotone operator, using a FISTA-style gap unrolling argument, we are able to derive gap estimates instead of merely ergodic gap estimates. Moreover, based on adding a diagonal component to this corrector, we are able to combine strong convexity based acceleration with inertial acceleration. We test our proposed method on image processing and inverse problems problems, obtaining convergence improvements for sparse Fourier inversion and Positron Emission Tomography.
△ Less
Submitted 12 February, 2020; v1 submitted 23 April, 2018;
originally announced April 2018.
-
Acceleration and global convergence of a first-order primal--dual method for nonconvex problems
Authors:
Christian Clason,
Stanislav Mazurenko,
Tuomo Valkonen
Abstract:
The primal--dual hybrid gradient method (PDHGM, also known as the Chambolle--Pock method) has proved very successful for convex optimization problems involving linear operators arising in image processing and inverse problems. In this paper, we analyze an extension to nonconvex problems that arise if the operator is nonlinear. Based on the idea of testing, we derive new step length parameter condi…
▽ More
The primal--dual hybrid gradient method (PDHGM, also known as the Chambolle--Pock method) has proved very successful for convex optimization problems involving linear operators arising in image processing and inverse problems. In this paper, we analyze an extension to nonconvex problems that arise if the operator is nonlinear. Based on the idea of testing, we derive new step length parameter conditions for the convergence in infinite-dimensional Hilbert spaces and provide acceleration rules for suitably (locally and/or partially) monotone problems. Importantly, we prove linear convergence rates as well as global convergence in certain cases. We demonstrate the efficacy of these step length rules for PDE-constrained optimization problems.
△ Less
Submitted 8 August, 2018; v1 submitted 9 February, 2018;
originally announced February 2018.
-
Spherical function regularization for parallel MRI reconstruction
Authors:
Yonggui Zhu,
Tuomo Valkonen
Abstract:
From the optimization point of view, a difficulty with parallel MRI with simultaneous coil sensitivity estimation is the multiplicative nature of the non-linear forward operator: the image being reconstructed and the coil sensitivities compete against each other, causing the optimization process to be very sensitive to small perturbations. This can, to some extent, be avoided by regularizing the u…
▽ More
From the optimization point of view, a difficulty with parallel MRI with simultaneous coil sensitivity estimation is the multiplicative nature of the non-linear forward operator: the image being reconstructed and the coil sensitivities compete against each other, causing the optimization process to be very sensitive to small perturbations. This can, to some extent, be avoided by regularizing the unknown in a suitably "orthogonal" fashion. In this paper, we introduce such a regularization based on spherical function bases. To perform this regularization, we represent efficient recurrence formulas for spherical Bessel functions and associated Legendre functions. Numerically, we study the solution of the model with non-linear ADMM. We perform various numerical simulations to demonstrate the efficacy of the proposed model in parallel MRI reconstruction.
△ Less
Submitted 4 February, 2018;
originally announced February 2018.
-
Preconditioned proximal point methods and notions of partial subregularity
Authors:
Tuomo Valkonen
Abstract:
Based on the needs of convergence proofs of preconditioned proximal point methods, we introduce notions of partial strong submonotonicity and partial (metric) subregularity of set-valued maps. We study relationships between these two concepts, neither of which is generally weaker or stronger than the other one. For our algorithmic purposes, the novel submonotonicity turns out to be easier to emplo…
▽ More
Based on the needs of convergence proofs of preconditioned proximal point methods, we introduce notions of partial strong submonotonicity and partial (metric) subregularity of set-valued maps. We study relationships between these two concepts, neither of which is generally weaker or stronger than the other one. For our algorithmic purposes, the novel submonotonicity turns out to be easier to employ than more conventional error bounds obtained from subregularity. Using strong submonotonicity, we demonstrate the linear convergence of the Primal-Dual Proximal splitting method to some strictly complementary solutions of example problems from image processing and data science. This is without the conventional assumption that all the objective functions of the involved saddle point problem are strongly convex.
△ Less
Submitted 28 February, 2020; v1 submitted 14 November, 2017;
originally announced November 2017.
-
Interior-proximal primal-dual methods
Authors:
Tuomo Valkonen
Abstract:
We study preconditioned proximal point methods for a class of saddle point problems, where the preconditioner decouples the overall proximal point method into an alternating primal--dual method. This is akin to the Chambolle--Pock method or the ADMM. In our work, we replace the squared distance in the dual step by a barrier function on a symmetric cone, while using a standard (Euclidean) proximal…
▽ More
We study preconditioned proximal point methods for a class of saddle point problems, where the preconditioner decouples the overall proximal point method into an alternating primal--dual method. This is akin to the Chambolle--Pock method or the ADMM. In our work, we replace the squared distance in the dual step by a barrier function on a symmetric cone, while using a standard (Euclidean) proximal step for the primal variable. We show that under non-degeneracy and simple linear constraints, such a hybrid primal--dual algorithm can achieve linear convergence on originally strongly convex problems involving the second-order cone in their saddle point form. On general symmetric cones, we are only able to show an $O(1/N)$ rate. These results are based on estimates of strong convexity of the barrier function, extended with a penalty to the boundary of the symmetric cone.
△ Less
Submitted 22 August, 2018; v1 submitted 21 June, 2017;
originally announced June 2017.
-
Testing and non-linear preconditioning of the proximal point method
Authors:
Tuomo Valkonen
Abstract:
Employing the ideas of non-linear preconditioning and testing of the classical proximal point method, we formalise common arguments in convergence rate and convergence proofs of optimisation methods to the verification of a simple iteration-wise inequality. When applied to fixed point operators, the latter can be seen as a generalisation of firm non-expansivity or the $α$-averaged property. The ma…
▽ More
Employing the ideas of non-linear preconditioning and testing of the classical proximal point method, we formalise common arguments in convergence rate and convergence proofs of optimisation methods to the verification of a simple iteration-wise inequality. When applied to fixed point operators, the latter can be seen as a generalisation of firm non-expansivity or the $α$-averaged property. The main purpose of this work is to provide the abstract background theory for our companion paper "Block-proximal methods with spatially adapted acceleration". In the present account we demonstrate the effectiveness of the general approach on several classical algorithms, as well as their stochastic variants. Besides, of course, the proximal point method, these method include the gradient descent, forward--backward splitting, Douglas--Rachford splitting, Newton's method, as well as several methods for saddle-point problems, such as the Alternating Directions Method of Multipliers, and the Chambolle--Pock method.
△ Less
Submitted 23 August, 2018; v1 submitted 16 March, 2017;
originally announced March 2017.
-
Block-proximal methods with spatially adapted acceleration
Authors:
Tuomo Valkonen
Abstract:
We study and develop (stochastic) primal--dual block-coordinate descent methods for convex problems based on the method due to Chambolle and Pock. Our methods have known convergence rates for the iterates and the ergodic gap: $O(1/N^2)$ if each block is strongly convex, $O(1/N)$ if no convexity is present, and more generally a mixed rate $O(1/N^2)+O(1/N)$ for strongly convex blocks, if only some b…
▽ More
We study and develop (stochastic) primal--dual block-coordinate descent methods for convex problems based on the method due to Chambolle and Pock. Our methods have known convergence rates for the iterates and the ergodic gap: $O(1/N^2)$ if each block is strongly convex, $O(1/N)$ if no convexity is present, and more generally a mixed rate $O(1/N^2)+O(1/N)$ for strongly convex blocks, if only some blocks are strongly convex. Additional novelties of our methods include blockwise-adapted step lengths and acceleration, as well as the ability to update both the primal and dual variables randomly in blocks under a very light compatibility condition. In other words, these variants of our methods are doubly-stochastic. We test the proposed methods on various image processing problems, where we employ pixelwise-adapted acceleration.
△ Less
Submitted 3 January, 2019; v1 submitted 23 September, 2016;
originally announced September 2016.
-
Primal-dual extragradient methods for nonlinear nonsmooth PDE-constrained optimization
Authors:
Christian Clason,
Tuomo Valkonen
Abstract:
We study the extension of the Chambolle--Pock primal-dual algorithm to nonsmooth optimization problems involving nonlinear operators between function spaces. Local convergence is shown under technical conditions including metric regularity of the corresponding primal-dual optimality conditions. We also show convergence for a Nesterov-type accelerated variant provided one part of the functional is…
▽ More
We study the extension of the Chambolle--Pock primal-dual algorithm to nonsmooth optimization problems involving nonlinear operators between function spaces. Local convergence is shown under technical conditions including metric regularity of the corresponding primal-dual optimality conditions. We also show convergence for a Nesterov-type accelerated variant provided one part of the functional is strongly convex.
We show the applicability of the accelerated algorithm to examples of inverse problems with $L^1$- and $L^\infty$-fitting terms as well as of state-constrained optimal control problems, where convergence can be guaranteed after introducing an (arbitrary small, still nonsmooth) Moreau--Yosida regularization. This is verified in numerical examples.
△ Less
Submitted 15 March, 2017; v1 submitted 20 June, 2016;
originally announced June 2016.
-
Explorations on anisotropic regularisation of dynamic inverse problems by bilevel optimisation
Authors:
Martin Benning,
Carola-Bibiane Schönlieb,
Tuomo Valkonen,
Verner Vlačić
Abstract:
We explore anisotropic regularisation methods in the spirit of [Holler & Kunisch, 14]. Based on ground truth data, we propose a bilevel optimisation strategy to compute the optimal regularisation parameters of such a model for the application of video denoising. The optimisation poses a challenge in itself, as the dependency on one of the regularisation parameters is non-linear such that the stand…
▽ More
We explore anisotropic regularisation methods in the spirit of [Holler & Kunisch, 14]. Based on ground truth data, we propose a bilevel optimisation strategy to compute the optimal regularisation parameters of such a model for the application of video denoising. The optimisation poses a challenge in itself, as the dependency on one of the regularisation parameters is non-linear such that the standard existence and convergence theory does not apply. Moreover, we analyse numerical results of the proposed parameter learning strategy based on three exemplary video sequences and discuss the impact of these results on the actual modelling of dynamic inverse problems.
△ Less
Submitted 3 February, 2016;
originally announced February 2016.
-
Acceleration of the PDHGM on strongly convex subspaces
Authors:
Tuomo Valkonen,
Thomas Pock
Abstract:
We propose several variants of the primal-dual method due to Chambolle and Pock. Without requiring full strong convexity of the objective functions, our methods are accelerated on subspaces with strong convexity. This yields mixed rates, $O(1/N^2)$ with respect to initialisation and $O(1/N)$ with respect to the dual sequence, and the residual part of the primal sequence. We demonstrate the efficac…
▽ More
We propose several variants of the primal-dual method due to Chambolle and Pock. Without requiring full strong convexity of the objective functions, our methods are accelerated on subspaces with strong convexity. This yields mixed rates, $O(1/N^2)$ with respect to initialisation and $O(1/N)$ with respect to the dual sequence, and the residual part of the primal sequence. We demonstrate the efficacy of the proposed methods on image processing problems lacking strong convexity, such as total generalised variation denoising and total variation deblurring.
△ Less
Submitted 10 February, 2016; v1 submitted 20 November, 2015;
originally announced November 2015.
-
Preconditioned ADMM with nonlinear operator constraint
Authors:
Martin Benning,
Florian Knoll,
Carola-Bibiane Schönlieb,
Tuomo Valkonen
Abstract:
We are presenting a modification of the well-known Alternating Direction Method of Multipliers (ADMM) algorithm with additional preconditioning that aims at solving convex optimisation problems with nonlinear operator constraints. Connections to the recently developed Nonlinear Primal-Dual Hybrid Gradient Method (NL-PDHGM) are presented, and the algorithm is demonstrated to handle the nonlinear in…
▽ More
We are presenting a modification of the well-known Alternating Direction Method of Multipliers (ADMM) algorithm with additional preconditioning that aims at solving convex optimisation problems with nonlinear operator constraints. Connections to the recently developed Nonlinear Primal-Dual Hybrid Gradient Method (NL-PDHGM) are presented, and the algorithm is demonstrated to handle the nonlinear inverse problem of parallel Magnetic Resonance Imaging (MRI).
△ Less
Submitted 2 November, 2015;
originally announced November 2015.
-
Stability of saddle points via explicit coderivatives of pointwise subdifferentials
Authors:
Christian Clason,
Tuomo Valkonen
Abstract:
We derive stability criteria for saddle points of a class of nonsmooth optimization problems in Hilbert spaces arising in PDE-constrained optimization, using metric regularity of infinite-dimensional set-valued map**s. A main ingredient is an explicit pointwise characterization of the Fréchet coderivative of the subdifferential of convex integral functionals. This is applied to several stability…
▽ More
We derive stability criteria for saddle points of a class of nonsmooth optimization problems in Hilbert spaces arising in PDE-constrained optimization, using metric regularity of infinite-dimensional set-valued map**s. A main ingredient is an explicit pointwise characterization of the Fréchet coderivative of the subdifferential of convex integral functionals. This is applied to several stability properties for parameter identification problems for an elliptic partial differential equation with non-differentiable data fitting terms.
△ Less
Submitted 10 February, 2017; v1 submitted 22 September, 2015;
originally announced September 2015.
-
Diffusion tensor imaging with deterministic error bounds
Authors:
Artur Gorokh,
Yury Korolev,
Tuomo Valkonen
Abstract:
Errors in the data and the forward operator of an inverse problem can be handily modelled using partial order in Banach lattices. We present some existing results of the theory of regularisation in this novel framework, where errors are represented as bounds by means of the appropriate partial order.
We apply the theory to Diffusion Tensor Imaging, where correct noise modelling is challenging: i…
▽ More
Errors in the data and the forward operator of an inverse problem can be handily modelled using partial order in Banach lattices. We present some existing results of the theory of regularisation in this novel framework, where errors are represented as bounds by means of the appropriate partial order.
We apply the theory to Diffusion Tensor Imaging, where correct noise modelling is challenging: it involves the Rician distribution and the nonlinear Stejskal-Tanner equation. Linearisation of the latter in the statistical framework would complicate the noise model even further. We avoid this using the error bounds approach, which preserves simple error structure under monotone transformations.
△ Less
Submitted 26 January, 2016; v1 submitted 7 September, 2015;
originally announced September 2015.
-
Bilevel parameter learning for higher-order total variation regularisation models
Authors:
J. C. De los Reyes,
C. -B. Schönlieb,
T. Valkonen
Abstract:
We consider a bilevel optimisation approach for parameter learning in higher-order total variation image reconstruction models. Apart from the least squares cost functional, naturally used in bilevel learning, we propose and analyse an alternative cost, based on a Huber regularised TV-seminorm. Differentiability properties of the solution operator are verified and a first-order optimality system i…
▽ More
We consider a bilevel optimisation approach for parameter learning in higher-order total variation image reconstruction models. Apart from the least squares cost functional, naturally used in bilevel learning, we propose and analyse an alternative cost, based on a Huber regularised TV-seminorm. Differentiability properties of the solution operator are verified and a first-order optimality system is derived. Based on the adjoint information, a quasi-Newton algorithm is proposed for the numerical solution of the bilevel problems. Numerical experiments are carried out to show the suitability of our approach and the improved performance of the new cost functional. Thanks to the bilevel optimisation framework, also a detailed comparison between TGV$^2$ and ICTV is carried out, showing the advantages and shortcomings of both regularisers, depending on the structure of the processed images and their noise level.
△ Less
Submitted 28 August, 2015;
originally announced August 2015.
-
Bilevel approaches for learning of variational imaging models
Authors:
Luca Calatroni,
Cao Chung,
Juan Carlos De Los Reyes,
Carola-Bibiane Schönlieb,
Tuomo Valkonen
Abstract:
We review some recent learning approaches in variational imaging, based on bilevel optimisation, and emphasize the importance of their treatment in function space. The paper covers both analytical and numerical techniques. Analytically, we include results on the existence and structure of minimisers, as well as optimality conditions for their characterisation. Based on this information, Newton typ…
▽ More
We review some recent learning approaches in variational imaging, based on bilevel optimisation, and emphasize the importance of their treatment in function space. The paper covers both analytical and numerical techniques. Analytically, we include results on the existence and structure of minimisers, as well as optimality conditions for their characterisation. Based on this information, Newton type methods are studied for the solution of the problems at hand, combining them with sampling techniques in case of large databases. The computational verification of the developed techniques is extensively documented, covering instances with different type of regularisers, several noise models, spatially dependent weights and large image databases.
△ Less
Submitted 8 May, 2015;
originally announced May 2015.
-
The structure of optimal parameters for image restoration problems
Authors:
Juan Carlos De Los Reyes,
Carola-Bibiane Schönlieb,
Tuomo Valkonen
Abstract:
We study the qualitative properties of optimal regularisation parameters in variational models for image restoration. The parameters are solutions of bilevel optimisation problems with the image restoration problem as constraint. A general type of regulariser is considered, which encompasses total variation (TV), total generalized variation (TGV) and infimal-convolution total variation (ICTV). We…
▽ More
We study the qualitative properties of optimal regularisation parameters in variational models for image restoration. The parameters are solutions of bilevel optimisation problems with the image restoration problem as constraint. A general type of regulariser is considered, which encompasses total variation (TV), total generalized variation (TGV) and infimal-convolution total variation (ICTV). We prove that under certain conditions on the given data optimal parameters derived by bilevel optimisation problems exist. A crucial point in the existence proof turns out to be the boundedness of the optimal parameters away from $0$ which we prove in this paper. The analysis is done on the original -- in image restoration typically non-smooth variational problem -- as well as on a smoothed approximation set in Hilbert space which is the one considered in numerical computations. For the smoothed bilevel problem we also prove that it $Γ$ converges to the original problem as the smoothing vanishes. All analysis is done in function spaces rather than on the discretised learning problem.
△ Less
Submitted 8 May, 2015;
originally announced May 2015.
-
Asymptotic behaviour of total generalised variation
Authors:
Konstantinos Papafitsoros,
Tuomo Valkonen
Abstract:
The recently introduced second order total generalised variation functional $\mathrm{TGV}_{β,α}^{2}$ has been a successful regulariser for image processing purposes. Its definition involves two positive parameters $α$ and $β$ whose values determine the amount and the quality of the regularisation. In this paper we report on the behaviour of $\mathrm{TGV}_{β,α}^{2}$ in the cases where the parameter…
▽ More
The recently introduced second order total generalised variation functional $\mathrm{TGV}_{β,α}^{2}$ has been a successful regulariser for image processing purposes. Its definition involves two positive parameters $α$ and $β$ whose values determine the amount and the quality of the regularisation. In this paper we report on the behaviour of $\mathrm{TGV}_{β,α}^{2}$ in the cases where the parameters $α, β$ as well as their ratio $β/α$ becomes very large or very small. Among others, we prove that for sufficiently symmetric two dimensional data and large ratio $β/α$, $\mathrm{TGV}_{β,α}^{2}$ regularisation coincides with total variation ($\mathrm{TV}$) regularisation.
△ Less
Submitted 24 February, 2015;
originally announced February 2015.
-
Limiting aspects of non-convex ${TV}^φ$ models
Authors:
Michael Hintermüller,
Tuomo Valkonen,
Tao Wu
Abstract:
Recently, non-convex regularisation models have been introduced in order to provide a better prior for gradient distributions in real images. They are based on using concave energies $φ$ in the total variation type functional ${TV}^φ(u) := \int φ(|\nabla u(x)|) d x$. In this paper, it is demonstrated that for typical choices of $φ$, functionals of this type pose several difficulties when extended…
▽ More
Recently, non-convex regularisation models have been introduced in order to provide a better prior for gradient distributions in real images. They are based on using concave energies $φ$ in the total variation type functional ${TV}^φ(u) := \int φ(|\nabla u(x)|) d x$. In this paper, it is demonstrated that for typical choices of $φ$, functionals of this type pose several difficulties when extended to the entire space of functions of bounded variation, ${BV}(Ω)$. In particular, if $φ(t)=t^q$ for $q \in (0, 1)$ and ${TV}^φ$ is defined directly for piecewise constant functions and extended via weak* lower semicontinuous envelopes to ${BV}(Ω)$, then still ${TV}^φ(u)=\infty$ for $u$ not piecewise constant. If, on the other hand, ${TV}^φ$ is defined analogously via continuously differentiable functions, then ${TV}^φ\equiv 0$, (!). We study a way to remedy the models through additional multiscale regularisation and area strict convergence, provided that the energy $φ(t)=t^q$ is linearised for high values. The fact, that this kind of energies actually better matches reality and improves reconstructions, is demonstrated by statistics and numerical experiments.
△ Less
Submitted 23 December, 2014;
originally announced December 2014.
-
The jump set under geometric regularisation. Part 2: Higher-order approaches
Authors:
Tuomo Valkonen
Abstract:
In Part 1, we developed a new technique based on Lipschitz pushforwards for proving the jump set containment property $\mathcal{H}^{m-1}(J_u \setminus J_f)=0$ of solutions $u$ to total variation denoising. We demonstrated that the technique also applies to Huber-regularised TV. Now, in this Part 2, we extend the technique to higher-order regularisers. We are not quite able to prove the property fo…
▽ More
In Part 1, we developed a new technique based on Lipschitz pushforwards for proving the jump set containment property $\mathcal{H}^{m-1}(J_u \setminus J_f)=0$ of solutions $u$ to total variation denoising. We demonstrated that the technique also applies to Huber-regularised TV. Now, in this Part 2, we extend the technique to higher-order regularisers. We are not quite able to prove the property for total generalised variation (TGV) based on the symmetrised gradient for the second-order term. We show that the property holds under three conditions: First, the solution $u$ is locally bounded. Second, the second-order variable is of locally bounded variation, $w \in \mbox{BV}_\mbox{loc}(Ω; \mathbb{R}^m)$, instead of just bounded deformation, $w \in \mbox{BD}(Ω)$. Third, $w$ does not jump on $J_u$ parallel to it. The second condition can be achieved for non-symmetric TGV. Both the second and third condition can be achieved if we change the Radon (or $L^1$) norm of the symmetrised gradient $Ew$ into an $L^p$ norm, $p>1$, in which case Korn's inequality holds. We also consider the application of the technique to infimal convolution TV, and study the limiting behaviour of the singular part of $D u$, as the second parameter of $\mbox{TGV}^2$ goes to zero. Unsurprisingly, it vanishes, but in numerical discretisations the situation looks quite different. Finally, our work additionally includes a result on TGV-strict approximation in $\mbox{BV}(Ω)$.
△ Less
Submitted 8 July, 2014;
originally announced July 2014.
-
The jump set under geometric regularisation. Part 1: Basic technique and first-order denoising
Authors:
Tuomo Valkonen
Abstract:
Let $u \in \mbox{BV}(Ω)$ solve the total variation denoising problem with $L^2$-squared fidelity and data $f$. Caselles et al. [Multiscale Model. Simul. 6 (2008), 879--894] have shown the containment $\mathcal{H}^{m-1}(J_u \setminus J_f)=0$ of the jump set $J_u$ of $u$ in that of $f$. Their proof unfortunately depends heavily on the co-area formula, as do many results in this area, and as such is…
▽ More
Let $u \in \mbox{BV}(Ω)$ solve the total variation denoising problem with $L^2$-squared fidelity and data $f$. Caselles et al. [Multiscale Model. Simul. 6 (2008), 879--894] have shown the containment $\mathcal{H}^{m-1}(J_u \setminus J_f)=0$ of the jump set $J_u$ of $u$ in that of $f$. Their proof unfortunately depends heavily on the co-area formula, as do many results in this area, and as such is not directly extensible to higher-order, curvature-based, and other advanced geometric regularisers, such as total generalised variation (TGV) and Euler's elastica. These have received increased attention in recent times due to their better practical regularisation properties compared to conventional total variation or wavelets. We prove analogous jump set containment properties for a general class of regularisers. We do this with novel Lipschitz transformation techniques, and do not require the co-area formula. In the present Part 1 we demonstrate the general technique on first-order regularisers, while in Part 2 we will extend it to higher-order regularisers. In particular, we concentrate in this part on TV and, as a novelty, Huber-regularised TV. We also demonstrate that the technique would apply to non-convex TV models as well as the Perona-Malik anisotropic diffusion, if these approaches were well-posed to begin with.
△ Less
Submitted 14 April, 2015; v1 submitted 6 July, 2014;
originally announced July 2014.
-
Imaging with Kantorovich-Rubinstein discrepancy
Authors:
Jan Lellmann,
Dirk A. Lorenz,
Carola Schönlieb,
Tuomo Valkonen
Abstract:
We propose the use of the Kantorovich-Rubinstein norm from optimal transport in imaging problems. In particular, we discuss a variational regularisation model endowed with a Kantorovich-Rubinstein discrepancy term and total variation regularization in the context of image denoising and cartoon-texture decomposition. We point out connections of this approach to several other recently proposed metho…
▽ More
We propose the use of the Kantorovich-Rubinstein norm from optimal transport in imaging problems. In particular, we discuss a variational regularisation model endowed with a Kantorovich-Rubinstein discrepancy term and total variation regularization in the context of image denoising and cartoon-texture decomposition. We point out connections of this approach to several other recently proposed methods such as total generalized variation and norms capturing oscillating patterns. We also show that the respective optimization problem can be turned into a convex-concave saddle point problem with simple constraints and hence, can be solved by standard tools. Numerical examples exhibit interesting features and favourable performance for denoising and cartoon-texture decomposition.
△ Less
Submitted 1 July, 2014;
originally announced July 2014.
-
A primal-dual hybrid gradient method for non-linear operators with applications to MRI
Authors:
Tuomo Valkonen
Abstract:
We study the solution of minimax problems $\min_x \max_y G(x) + \langle K(x),y\rangle - F^*(y)$ in finite-dimensional Hilbert spaces. The functionals $G$ and $F^*$ we assume to be convex, but the operator $K$ we allow to be non-linear. We formulate a natural extension of the modified primal-dual hybrid gradient method (PDHGM), originally for linear $K$, due to Chambolle and Pock. We prove the loca…
▽ More
We study the solution of minimax problems $\min_x \max_y G(x) + \langle K(x),y\rangle - F^*(y)$ in finite-dimensional Hilbert spaces. The functionals $G$ and $F^*$ we assume to be convex, but the operator $K$ we allow to be non-linear. We formulate a natural extension of the modified primal-dual hybrid gradient method (PDHGM), originally for linear $K$, due to Chambolle and Pock. We prove the local convergence of the method, provided various technical conditions are satisfied. These include in particular the Aubin property of the inverse a monotone operator at the solution. Of particular interest to us is the case arising from reformulation of regularisation problems $\min_x \|f-T(x)\|^2/2 + αR(x)$ with the operator $T$ non-linear. For such problems, we show that our general local convergence result holds when the noise level of the data $f$ is low, and the regularisation parameter $α$ is correspondingly small. We verify the numerical performance of the method by applying it to problems from magnetic resonance imaging (MRI) in chemical engineering and medicine. The specific applications are in diffusion tensor imaging (DTI) and MR velocity imaging. These numerical studies show very promising performance.
△ Less
Submitted 2 July, 2014; v1 submitted 19 September, 2013;
originally announced September 2013.
