-
Sequential Projected Newton method for regularization of nonlinear least squares problems
Authors:
Jeffrey Cornelis,
Wim Vanroose
Abstract:
We develop a computationally efficient algorithm for the automatic regularization of nonlinear inverse problems based on the discrepancy principle. We formulate the problem as an equality constrained optimization problem, where the constraint is given by a least squares data fidelity term and expresses the discrepancy principle. The objective function is a convex regularization function that incor…
▽ More
We develop a computationally efficient algorithm for the automatic regularization of nonlinear inverse problems based on the discrepancy principle. We formulate the problem as an equality constrained optimization problem, where the constraint is given by a least squares data fidelity term and expresses the discrepancy principle. The objective function is a convex regularization function that incorporates some prior knowledge, such as the total variation regularization function. Using the Jacobian matrix of the nonlinear forward model, we consider a sequence of quadratically constrained optimization problems that can all be solved using the Projected Newton method. We show that the solution of such a quadratically constrained sub-problem results in a descent direction for an exact merit function. This merit function can then be used to describe a formal line-search method. We also formulate a slightly more heuristic approach that simplifies the algorithm and allows for an inexact solution of the sequence of sub-problems. We illustrate the behavior of the algorithm using a number of numerical experiments, with Talbot-Lau X-ray phase contrast imaging as the main application. The numerical experiments confirm that the quadratically constrained sub-problems need not be solved with high accuracy in early iterations to make sufficient progress towards the solution. In addition, we show that the proposed method is able to produce reconstructions of similar quality compared to other state-of-the-art approaches with a significant reduction in computational time.
△ Less
Submitted 2 September, 2021; v1 submitted 11 May, 2021;
originally announced May 2021.
-
Convergence analysis of a regularized inexact interior-point method for linear programming problems
Authors:
Jeffrey Cornelis,
Wim Vanroose
Abstract:
Interior-point methods for linear programming problems require the repeated solution of a linear system of equations. Solving these linear systems is non-trivial due to the severe ill-conditioning of the matrices towards convergence. This issue can be alleviated by incorporating suitable regularization terms in the linear programming problem. Regularization also allows us to efficiently handle ran…
▽ More
Interior-point methods for linear programming problems require the repeated solution of a linear system of equations. Solving these linear systems is non-trivial due to the severe ill-conditioning of the matrices towards convergence. This issue can be alleviated by incorporating suitable regularization terms in the linear programming problem. Regularization also allows us to efficiently handle rank deficient constraint matrices. We provide a convergence analysis of a regularized inexact interior-point method. The term `inexact' refers to the fact that we do not need to compute the true solution of the linear system of equations, only an approximation thereof. The formulation of the algorithm is sufficiently general such that specialized linear algebra routines developed in other work on inexact interior-point methods can also be incorporated in our regularized framework. In this work, we exploit the inexactness by using a mixed-precision solver for the linear system of equations. More specifically, we perform a Cholesky factorization in IEEE single precision and use it as a preconditioner for the Conjugate Gradient method. Numerical experiments illustrate the benefits of this approach applied to linear programming problems with a dense constraint matrix.
△ Less
Submitted 4 May, 2021;
originally announced May 2021.
-
Krylov-Simplex method that minimizes the residual in $\ell_1$-norm or $\ell_\infty$-norm
Authors:
Wim Vanroose,
Jeffrey Cornelis
Abstract:
The paper presents two variants of a Krylov-Simplex iterative method that combines Krylov and simplex iterations to minimize the residual $r = b-Ax$. The first method minimizes $\|r\|_\infty$, i.e. maximum of the absolute residuals. The second minimizes $\|r\|_1$, and finds the solution with the least absolute residuals. Both methods search for an optimal solution $x_k$ in a Krylov subspace which…
▽ More
The paper presents two variants of a Krylov-Simplex iterative method that combines Krylov and simplex iterations to minimize the residual $r = b-Ax$. The first method minimizes $\|r\|_\infty$, i.e. maximum of the absolute residuals. The second minimizes $\|r\|_1$, and finds the solution with the least absolute residuals. Both methods search for an optimal solution $x_k$ in a Krylov subspace which results in a small linear programming problem. A specialized simplex algorithm solves this projected problem and finds the optimal linear combination of Krylov basis vectors to approximate the solution. The resulting simplex algorithm requires the solution of a series of small dense linear systems that only differ by rank-one updates. The $QR$ factorization of these matrices is updated each iteration. We demonstrate the effectiveness of the methods with numerical experiments.
△ Less
Submitted 27 January, 2021;
originally announced January 2021.
-
Projected Newton method for noise constrained $\ell_p$ regularization
Authors:
Jeffrey Cornelis,
Wim Vanroose
Abstract:
Choosing an appropriate regularization term is necessary to obtain a meaningful solution to an ill-posed linear inverse problem contaminated with measurement errors or noise. The $\ell_p$ norm covers a wide range of choices for the regularization term since its behavior critically depends on the choice of $p$ and since it can easily be combined with a suitable regularization matrix. We develop an…
▽ More
Choosing an appropriate regularization term is necessary to obtain a meaningful solution to an ill-posed linear inverse problem contaminated with measurement errors or noise. The $\ell_p$ norm covers a wide range of choices for the regularization term since its behavior critically depends on the choice of $p$ and since it can easily be combined with a suitable regularization matrix. We develop an efficient algorithm that simultaneously determines the regularization parameter and corresponding $\ell_p$ regularized solution such that the discrepancy principle is satisfied. We project the problem on a low-dimensional Generalized Krylov subspace and compute the Newton direction for this much smaller problem. We illustrate some interesting properties of the algorithm and compare its performance with other state-of-the-art approaches using a number of numerical experiments, with a special focus of the sparsity inducing $\ell_1$ norm and edge-preserving total variation regularization.
△ Less
Submitted 31 July, 2020; v1 submitted 6 May, 2020;
originally announced May 2020.
-
Projected Newton Method for noise constrained Tikhonov regularization
Authors:
Jeffrey Cornelis,
Nick Schenkels,
Wim Vanroose
Abstract:
Tikhonov regularization is a popular approach to obtain a meaningful solution for ill-conditioned linear least squares problems. A relatively simple way of choosing a good regularization parameter is given by Morozov's discrepancy principle. However, most approaches require the solution of the Tikhonov problem for many different values of the regularization parameter, which is computationally dema…
▽ More
Tikhonov regularization is a popular approach to obtain a meaningful solution for ill-conditioned linear least squares problems. A relatively simple way of choosing a good regularization parameter is given by Morozov's discrepancy principle. However, most approaches require the solution of the Tikhonov problem for many different values of the regularization parameter, which is computationally demanding for large scale problems. We propose a new and efficient algorithm which simultaneously solves the Tikhonov problem and finds the corresponding regularization parameter such that the discrepancy principle is satisfied. We achieve this by formulating the problem as a nonlinear system of equations and solving this system using a line search method. We obtain a good search direction by projecting the problem onto a low dimensional Krylov subspace and computing the Newton direction for the projected problem. This projected Newton direction, which is significantly less computationally expensive to calculate than the true Newton direction, is then combined with a backtracking line search to obtain a globally convergent algorithm, which we refer to as the Projected Newton method. We prove convergence of the algorithm and illustrate the improved performance over current state-of-the-art solvers with some numerical experiments.
△ Less
Submitted 30 April, 2020; v1 submitted 18 October, 2019;
originally announced October 2019.
-
Numerically Stable Recurrence Relations for the Communication Hiding Pipelined Conjugate Gradient Method
Authors:
Siegfried Cools,
Jeffrey Cornelis,
Wim Vanroose
Abstract:
Pipelined Krylov subspace methods (also referred to as communication-hiding methods) have been proposed in the literature as a scalable alternative to classic Krylov subspace algorithms for iteratively computing the solution to a large linear system in parallel. For symmetric and positive definite system matrices the pipelined Conjugate Gradient method outperforms its classic Conjugate Gradient co…
▽ More
Pipelined Krylov subspace methods (also referred to as communication-hiding methods) have been proposed in the literature as a scalable alternative to classic Krylov subspace algorithms for iteratively computing the solution to a large linear system in parallel. For symmetric and positive definite system matrices the pipelined Conjugate Gradient method outperforms its classic Conjugate Gradient counterpart on large scale distributed memory hardware by overlap** global communication with essential computations like the matrix-vector product, thus hiding global communication. A well-known drawback of the pipelining technique is the (possibly significant) loss of numerical stability. In this work a numerically stable variant of the pipelined Conjugate Gradient algorithm is presented that avoids the propagation of local rounding errors in the finite precision recurrence relations that construct the Krylov subspace basis. The multi-term recurrence relation for the basis vector is replaced by two-term recurrences, improving stability without increasing the overall computational cost of the algorithm. The proposed modification ensures that the pipelined Conjugate Gradient method is able to attain a highly accurate solution independently of the pipeline length. Numerical experiments demonstrate a combination of excellent parallel performance and improved maximal attainable accuracy for the new pipelined Conjugate Gradient algorithm. This work thus resolves one of the major practical restrictions for the useability of pipelined Krylov subspace methods.
△ Less
Submitted 15 May, 2019; v1 submitted 8 February, 2019;
originally announced February 2019.
-
Global optimization in inverse problems: A comparison of Kriging and radial basis functions
Authors:
W. Jacquet,
B. Truyen,
P. de Groen,
I. Lemahieu,
J. Cornelis
Abstract:
We study global optimization (GOP) in the framework of non-linear inverse problems with a unique solution. These problems are in general ill-posed. Evaluation of the objective function is often expensive, as it implies the solution of a non-trivial forward problem. The ill-posedness of these problems calls for regularization while the high evaluation cost of the objective function can be address…
▽ More
We study global optimization (GOP) in the framework of non-linear inverse problems with a unique solution. These problems are in general ill-posed. Evaluation of the objective function is often expensive, as it implies the solution of a non-trivial forward problem. The ill-posedness of these problems calls for regularization while the high evaluation cost of the objective function can be addressed with response surface techniques. The global optimization using Radial Basis Function (RBF) as presented by Gutmann (2001) is a response surface global optimization technique with regularizing aspects. Alternatively, several publications put forward global optimization using a probabilistic approach based upon Kriging as an efficient technique for non-linear multi modal objective functions, thereby providing a credible stop** rule (Jones2001). After comparing both concepts, we argue that in case of non-linear inverse problems an adaptation of the RBF algorithm seems to be the most promising approach.
△ Less
Submitted 22 June, 2005;
originally announced June 2005.
