-
Regularized methods via cubic model subspace minimization for nonconvex optimization
Authors:
Stefania Bellavia,
Davide Palitta,
Margherita Porcelli,
Valeria Simoncini
Abstract:
Adaptive cubic regularization methods for solving nonconvex problems need the efficient computation of the trial step, involving the minimization of a cubic model. We propose a new approach in which this model is minimized in a low dimensional subspace that, in contrast to classic approaches, is reused for a number of iterations. Whenever the trial step produced by the low-dimensional minimization…
▽ More
Adaptive cubic regularization methods for solving nonconvex problems need the efficient computation of the trial step, involving the minimization of a cubic model. We propose a new approach in which this model is minimized in a low dimensional subspace that, in contrast to classic approaches, is reused for a number of iterations. Whenever the trial step produced by the low-dimensional minimization process is unsatisfactory, we employ a regularized Newton step whose regularization parameter is a by-product of the model minimization over the low-dimensional subspace. We show that the worst-case complexity of classic cubic regularized methods is preserved, despite the possible regularized Newton steps. We focus on the large class of problems for which (sparse) direct linear system solvers are available and provide several experimental results showing the very large gains of our new approach when compared to standard implementations of adaptive cubic regularization methods based on direct linear solvers. Our first choice as projection space for the low-dimensional model minimization is the polynomial Krylov subspace; nonetheless, we also explore the use of rational Krylov subspaces in case where the polynomial ones lead to less competitive numerical results.
△ Less
Submitted 11 April, 2024; v1 submitted 25 June, 2023;
originally announced June 2023.
-
Accelerated Algorithms for Nonlinear Matrix Decomposition with the ReLU function
Authors:
Giovanni Seraghiti,
Atharva Awari,
Arnaud Vandaele,
Margherita Porcelli,
Nicolas Gillis
Abstract:
In this paper, we study the following nonlinear matrix decomposition (NMD) problem: given a sparse nonnegative matrix $X$, find a low-rank matrix $Θ$ such that $X \approx f(Θ)$, where $f$ is an element-wise nonlinear function. We focus on the case where $f(\cdot) = \max(0, \cdot)$, the rectified unit (ReLU) non-linear activation. We refer to the corresponding problem as ReLU-NMD. We first provide…
▽ More
In this paper, we study the following nonlinear matrix decomposition (NMD) problem: given a sparse nonnegative matrix $X$, find a low-rank matrix $Θ$ such that $X \approx f(Θ)$, where $f$ is an element-wise nonlinear function. We focus on the case where $f(\cdot) = \max(0, \cdot)$, the rectified unit (ReLU) non-linear activation. We refer to the corresponding problem as ReLU-NMD. We first provide a brief overview of the existing approaches that were developed to tackle ReLU-NMD. Then we introduce two new algorithms: (1) aggressive accelerated NMD (A-NMD) which uses an adaptive Nesterov extrapolation to accelerate an existing algorithm, and (2) three-block NMD (3B-NMD) which parametrizes $Θ= WH$ and leads to a significant reduction in the computational cost. We also propose an effective initialization strategy based on the nuclear norm as a proxy for the rank function. We illustrate the effectiveness of the proposed algorithms (available on gitlab) on synthetic and real-world data sets.
△ Less
Submitted 15 May, 2023;
originally announced May 2023.
-
Numerical solution of a class of quasi-linear matrix equations
Authors:
Margherita Porcelli,
Valeria Simoncini
Abstract:
Given the matrix equation ${\bf A X} + {\bf X B} + f({\bf X }) {\bf C} ={\bf D}$ in the unknown $n\times m$ matrix ${\bf X }$, we analyze existence and uniqueness conditions, together with computational solution strategies for $f \,: \mathbb{R}^{n \times m} \to \mathbb{R}$ being a linear or nonlinear function. We characterize different properties of the matrix equation and of its solution, dependi…
▽ More
Given the matrix equation ${\bf A X} + {\bf X B} + f({\bf X }) {\bf C} ={\bf D}$ in the unknown $n\times m$ matrix ${\bf X }$, we analyze existence and uniqueness conditions, together with computational solution strategies for $f \,: \mathbb{R}^{n \times m} \to \mathbb{R}$ being a linear or nonlinear function. We characterize different properties of the matrix equation and of its solution, depending on the considered classes of functions $f$. Our analysis mainly concerns small dimensional problems, though several considerations also apply to large scale matrix equations.
△ Less
Submitted 2 September, 2022;
originally announced September 2022.
-
A semidefinite programming approach for the projection onto the cone of negative semidefinite symmetric tensors with applications to solid mechanics
Authors:
Cristina Padovani,
Margherita Porcelli
Abstract:
We propose an algorithm for computing the projection of a symmetric second-order tensor onto the cone of negative semidefinite symmetric tensors with respect to the inner product defined by an assigned positive definite symmetric fourth-order tensor C. The projection problem is written as a semidefinite programming problem and an algorithm based on a primal-dual path-following interior point metho…
▽ More
We propose an algorithm for computing the projection of a symmetric second-order tensor onto the cone of negative semidefinite symmetric tensors with respect to the inner product defined by an assigned positive definite symmetric fourth-order tensor C. The projection problem is written as a semidefinite programming problem and an algorithm based on a primal-dual path-following interior point method coupled with a Mehrotra's predictor-corrector approach is proposed. Implementations based on direct methods are theoretically and numerically investigated taking into account tensors C arising in the modelling of masonry-like materials.
△ Less
Submitted 3 August, 2022;
originally announced August 2022.
-
An Improved Penalty Algorithm using Model Order Reduction for MIPDECO problems with partial observations
Authors:
Dominik Garmatter,
Margherita Porcelli,
Francesco Rinaldi,
Martin Stoll
Abstract:
This work addresses optimal control problems governed by a linear time-dependent partial differential equation (PDE) as well as integer constraints on the control. Moreover, partial observations are assumed in the objective function. The resulting problem poses several numerical challenges due to the mixture of combinatorial aspects, induced by integer variables, and large scale linear algebra iss…
▽ More
This work addresses optimal control problems governed by a linear time-dependent partial differential equation (PDE) as well as integer constraints on the control. Moreover, partial observations are assumed in the objective function. The resulting problem poses several numerical challenges due to the mixture of combinatorial aspects, induced by integer variables, and large scale linear algebra issues, arising from the PDE discretization. Since classical solution approaches such as the branch-and-bound framework are typically overwhelmed by such large-scale problems, this work extends an improved penalty algorithm proposed by the authors, to the time-dependent setting. The main contribution is a novel combination of an interior point method, preconditioning, and model order reduction yielding a tailored local optimization solver at the heart of the overall solution procedure. A thorough numerical investigation is carried out both for a Poisson problem as well as a convection-diffusion problem demonstrating the versatility of the approach.
△ Less
Submitted 7 October, 2021;
originally announced October 2021.
-
A spectral PALM algorithm for matrix and tensor-train based Dictionary Learning
Authors:
Domitilla Brandoni,
Margherita Porcelli,
Valeria Simoncini
Abstract:
Dictionary Learning (DL) is one of the leading sparsity promoting techniques in the context of image classification, where the "dictionary" matrix D of images and the sparse matrix X are determined so as to represent a redundant image dataset. The resulting constrained optimization problem is nonconvex and non-smooth, providing several computational challenges for its solution. To preserve multidi…
▽ More
Dictionary Learning (DL) is one of the leading sparsity promoting techniques in the context of image classification, where the "dictionary" matrix D of images and the sparse matrix X are determined so as to represent a redundant image dataset. The resulting constrained optimization problem is nonconvex and non-smooth, providing several computational challenges for its solution. To preserve multidimensional data features, various tensor DL formulations have been introduced, adding to the problem complexity. We develop a new alternating algorithm for the solution of the DL problem both in the matrix and tensor frameworks; in the latter case a new formulation based on Tensor-Train decompositions is also proposed. The new method belongs to the Proximal Alternating Linearized Minimization (PALM) algorithmic family, with the inclusion of second order information to enhance efficiency. We discuss a rigorous convergence analysis, and report on the new method performance on the image classification of several benchmark datasets.
△ Less
Submitted 9 March, 2022; v1 submitted 24 July, 2021;
originally announced July 2021.
-
Solving nonlinear systems of equations via spectral residual methods: stepsize selection and applications
Authors:
Enrico Meli,
Benedetta Morini,
Margherita Porcelli,
Cristina Sgattoni
Abstract:
Spectral residual methods are derivative-free and low-cost per iteration procedures for solving nonlinear systems of equations. They are generally coupled with a nonmonotone linesearch strategy and compare well with Newton-based methods for large nonlinear systems and sequences of nonlinear systems. The residual vector is used as the search direction and choosing the steplength has a crucial impac…
▽ More
Spectral residual methods are derivative-free and low-cost per iteration procedures for solving nonlinear systems of equations. They are generally coupled with a nonmonotone linesearch strategy and compare well with Newton-based methods for large nonlinear systems and sequences of nonlinear systems. The residual vector is used as the search direction and choosing the steplength has a crucial impact on the performance. In this work we address both theoretically and experimentally the steplength selection and provide results on a real application such as a rolling contact problem.
△ Less
Submitted 17 September, 2021; v1 submitted 12 May, 2020;
originally announced May 2020.
-
Exploiting problem structure in derivative free optimization
Authors:
Margherita Porcelli,
Philippe L. Toint
Abstract:
A structured version of derivative-free random pattern search optimization algorithms is introduced which is able to exploit coordinate partially separable structure (typically associated with sparsity) often present in unconstrained and bound-constrained optimization problems. This technique improves performance by orders of magnitude and makes it possible to solve large problems that otherwise a…
▽ More
A structured version of derivative-free random pattern search optimization algorithms is introduced which is able to exploit coordinate partially separable structure (typically associated with sparsity) often present in unconstrained and bound-constrained optimization problems. This technique improves performance by orders of magnitude and makes it possible to solve large problems that otherwise are totally intractable by other derivative-free methods. A library of interpolation-based modelling tools is also described, which can be associated to the structured or unstructured versions of the initial pattern search algorithm. The use of the library further enhances performance, especially when associated with structure. The significant gains in performance associated with these two techniques are illustrated using a new freely-available release of the BFO (Brute Force Optimizer) package firstly introduced in [Porcelli,Toint, ACM TOMS, 2017], which incorporates them. An interesting conclusion of the numerical results presented is that providing global structural information on a problem can result in significantly less evaluations of the objective function than attempting to building local Taylor-like models.
△ Less
Submitted 12 January, 2021; v1 submitted 14 January, 2020;
originally announced January 2020.
-
A relaxed interior point method for low-rank semidefinite programming problems with applications to matrix completion
Authors:
Stefania Bellavia,
Jacek Gondzio,
Margherita Porcelli
Abstract:
A new relaxed variant of interior point method for low-rank semidefinite programming problems is proposed in this paper. The method is a step outside of the usual interior point framework. In anticipation to converging to a low-rank primal solution, a special nearly low-rank form of all primal iterates is imposed. To accommodate such a (restrictive) structure, the first order optimality conditions…
▽ More
A new relaxed variant of interior point method for low-rank semidefinite programming problems is proposed in this paper. The method is a step outside of the usual interior point framework. In anticipation to converging to a low-rank primal solution, a special nearly low-rank form of all primal iterates is imposed. To accommodate such a (restrictive) structure, the first order optimality conditions have to be relaxed and are therefore approximated by solving an auxiliary least-squares problem. The relaxed interior point framework opens numerous possibilities how primal and dual approximated Newton directions can be computed. In particular, it admits the application of both the first- and the second-order methods in this context. The convergence of the method is established. A prototype implementation is discussed and encouraging preliminary computational results are reported for solving the SDP-reformulation of matrix-completion problems.
△ Less
Submitted 25 March, 2021; v1 submitted 13 September, 2019;
originally announced September 2019.
-
Improved penalty algorithm for Mixed Integer PDE Constrained Optimization Problems
Authors:
Dominik Garmatter,
Margherita Porcelli,
Francesco Rinaldi,
Martin Stoll
Abstract:
Optimal control problems including partial differential equation (PDE) as well as integer constraints merge the combinatorial difficulties of integer programming and the challenges related to large-scale systems resulting from discretized PDEs. So far, the Branch-and-Bound framework has been the most common solution strategy for such problems. In order to provide an alternative solution approach,…
▽ More
Optimal control problems including partial differential equation (PDE) as well as integer constraints merge the combinatorial difficulties of integer programming and the challenges related to large-scale systems resulting from discretized PDEs. So far, the Branch-and-Bound framework has been the most common solution strategy for such problems. In order to provide an alternative solution approach, especially in a large-scale context, this article investigates penalization techniques. Taking inspiration from a well-known family of existing exact penalty algorithms, a novel improved penalty algorithm is derived, whose key ingredients are a basin hop** strategy and an interior point method, both of which are specialized for the problem class. A thorough numerical investigation is carried out for a standard stationary test problem. Extensions to a convection-diffusion as well as a nonlinear test problem finally demonstrate the versatility of the approach.
△ Less
Submitted 8 September, 2021; v1 submitted 15 July, 2019;
originally announced July 2019.
-
Interior Point Methods and Preconditioning for PDE-Constrained Optimization Problems Involving Sparsity Terms
Authors:
John W. Pearson,
Margherita Porcelli,
Martin Stoll
Abstract:
PDE-constrained optimization problems with control or state constraints are challenging from an analytical as well as numerical perspective. The combination of these constraints with a sparsity-promoting $\rm L^1$ term within the objective function requires sophisticated optimization methods. We propose the use of an Interior Point scheme applied to a smoothed reformulation of the discretized prob…
▽ More
PDE-constrained optimization problems with control or state constraints are challenging from an analytical as well as numerical perspective. The combination of these constraints with a sparsity-promoting $\rm L^1$ term within the objective function requires sophisticated optimization methods. We propose the use of an Interior Point scheme applied to a smoothed reformulation of the discretized problem, and illustrate that such a scheme exhibits robust performance with respect to parameter changes. To increase the potency of this method we introduce fast and efficient preconditioners which enable us to solve problems from a number of PDE applications in low iteration numbers and CPU times, even when the parameters involved are altered dramatically.
△ Less
Submitted 11 February, 2019; v1 submitted 15 June, 2018;
originally announced June 2018.
-
Finite element model updating for structural applications
Authors:
Maria Girardi,
Cristina Padovani,
Daniele Pellegrini,
Margherita Porcelli,
Leonardo Robol
Abstract:
A novel method for performing model updating on finite element models is presented. The approach is particularly tailored to modal analyses of buildings, by which the lowest frequencies, obtained by using sensors and system identification approaches, need to be matched to the numerical ones predicted by the model. This is done by optimizing some unknown material parameters (such as mass density an…
▽ More
A novel method for performing model updating on finite element models is presented. The approach is particularly tailored to modal analyses of buildings, by which the lowest frequencies, obtained by using sensors and system identification approaches, need to be matched to the numerical ones predicted by the model. This is done by optimizing some unknown material parameters (such as mass density and Young's modulus) of the materials and/or the boundary conditions, which are often known only approximately. In particular, this is the case when considering historical buildings.
The straightforward application of a general-purpose optimizer can be impractical, given the large size of the model involved. In the paper, we show that, by slightly modifying the projection scheme used to compute the eigenvalues at the lowest end of the spectrum one can obtain local parametric reduced order models that, embedded in a trust-region scheme, form the basis for a reliable and efficient specialized algorithm.
We describe an optimization strategy based on this approach, and we provide numerical experiments that confirm its effectiveness and accuracy.
△ Less
Submitted 17 July, 2018; v1 submitted 27 January, 2018;
originally announced January 2018.
-
A note on using performance and data profilesfor training algorithms
Authors:
Margherita Porcelli,
Philippe L. Toint
Abstract:
It is shown how to use the performance and data profile benchmarking tools to improve algorithms' performance. An illustration for the BFO derivative-free optimizer suggests that the obtained gains are potentially significant.
It is shown how to use the performance and data profile benchmarking tools to improve algorithms' performance. An illustration for the BFO derivative-free optimizer suggests that the obtained gains are potentially significant.
△ Less
Submitted 26 November, 2017;
originally announced November 2017.
-
Preconditioning PDE-constrained optimization with $\rm L^1$-sparsity and control constraints
Authors:
Margherita Porcelli,
Valeria Simoncini,
Martin Stoll
Abstract:
PDE-constrained optimization aims at finding optimal setups for partial differential equations so that relevant quantities are minimized. Including sparsity promoting terms in the formulation of such problems results in more practically relevant computed controls but adds more challenges to the numerical solution of these problems. The needed $\rm L^1$-terms as well as additional inclusion of box…
▽ More
PDE-constrained optimization aims at finding optimal setups for partial differential equations so that relevant quantities are minimized. Including sparsity promoting terms in the formulation of such problems results in more practically relevant computed controls but adds more challenges to the numerical solution of these problems. The needed $\rm L^1$-terms as well as additional inclusion of box control constraints require the use of semismooth Newton methods. We propose robust preconditioners for different formulations of the Newton's equation. With the inclusion of a line-search strategy and an inexact approach for the solution of the linear systems, the resulting semismooth Newton's method is feasible for practical problems. Our results are underpinned by a theoretical analysis of the preconditioned matrix. Numerical experiments illustrate the robustness of the proposed scheme.
△ Less
Submitted 22 November, 2016;
originally announced November 2016.
-
Preconditioning of Active-Set Newton Methods for PDE-constrained Optimal Control Problems
Authors:
Margherita Porcelli,
Valeria Simoncini,
Mattia Tani
Abstract:
We address the problem of preconditioning a sequence of saddle point linear systems arising in the solution of PDE-constrained optimal control problems via active-set Newton methods, with control and (regularized) state constraints. We present two new preconditioners based on a full block matrix factorization of the Schur complement of the Jacobian matrices, where the active-set blocks are merged…
▽ More
We address the problem of preconditioning a sequence of saddle point linear systems arising in the solution of PDE-constrained optimal control problems via active-set Newton methods, with control and (regularized) state constraints. We present two new preconditioners based on a full block matrix factorization of the Schur complement of the Jacobian matrices, where the active-set blocks are merged into the constraint blocks. We discuss the robustness of the new preconditioners with respect to the parameters of the continuous and discrete problems. Numerical experiments on 3D problems are presented, including comparisons with existing approaches based on preconditioned conjugate gradients in a nonstandard inner product.
△ Less
Submitted 22 May, 2015; v1 submitted 4 July, 2014;
originally announced July 2014.
