Showing 1–50 of 83 results for author: Boţ, R I

Tikhonov regularization of monotone operator flows not only ensures strong convergence of the trajectories but also speeds up the vanishing of the residuals

Authors: Radu Ioan Bot, Dang-Khoa Nguyen

Abstract: In the framework of real Hilbert spaces, we investigate first-order dynamical systems governed by monotone and continuous operators. It has been established that for these systems, only the ergodic trajectory converges to a zero of the operator. A notable example is the counterclockwise $π/2$-rotation operator on $\mathbb{R}^2$, which illustrates that general trajectory convergence cannot be expec… ▽ More In the framework of real Hilbert spaces, we investigate first-order dynamical systems governed by monotone and continuous operators. It has been established that for these systems, only the ergodic trajectory converges to a zero of the operator. A notable example is the counterclockwise $π/2$-rotation operator on $\mathbb{R}^2$, which illustrates that general trajectory convergence cannot be expected. However, trajectory convergence is assured for operators with the stronger property of cocoercivity. For this class of operators, the trajectory's velocity and the opertor values along the trajectory converge in norm to zero at a rate of $o(\frac{1}{\sqrt{t}})$ as $t \rightarrow +\infty$. In this paper, we demonstrate that when the monotone operator flow is augmented with a Tikhonov regularization term, the resulting trajectory converges strongly to the element of the set of zeros with minimal norm. In addition, rates of convergence in norm for the trajectory's velocity and the operator along the trajectory can be derived in terms of the regularization function. In some particular cases, these rates of convergence can outperform the ones of the coercive operator flows and can be as fast as $O(\frac{1}{t})$ as $t \rightarrow +\infty$. In this way, we emphasize a surprising acceleration feature of the Tikhonov regularization. Additionally, we explore these properties for monotone operator flows that incorporate time rescaling and an anchor point. For a specific choice of the Tikhonov regularization function, these flows are closely linked to second-order dynamical systems with a vanishing dam** term. The convergence and convergence rate results we achieve for these systems complement recent findings for the Fast Optimistic Gradient Descent Ascent (OGDA) dynamics, leading to surprising outcomes. △ Less

Submitted 2 June, 2024; originally announced June 2024.

MSC Class: 47J20; 47H05; 65K10; 65K15; 90C25

On a Stochastic Differential Equation with Correction Term Governed by a Monotone and Lipschitz Continuous Operator

Authors: Radu Ioan Bot, Chiara Schindler

Abstract: In our pursuit of finding a zero for a monotone and Lipschitz continuous operator $M : \R^n \rightarrow \R^n$ amidst noisy evaluations, we explore an associated differential equation within a stochastic framework, incorporating a correction term. We present a result establishing the existence and uniqueness of solutions for the stochastic differential equations under examination. Additionally, ass… ▽ More In our pursuit of finding a zero for a monotone and Lipschitz continuous operator $M : \R^n \rightarrow \R^n$ amidst noisy evaluations, we explore an associated differential equation within a stochastic framework, incorporating a correction term. We present a result establishing the existence and uniqueness of solutions for the stochastic differential equations under examination. Additionally, assuming that the diffusion term is square-integrable, we demonstrate the almost sure convergence of the trajectory process $X(t)$ to a zero of $M$ and of $\|M(X(t))\|$ to $0$ as $t \rightarrow +\infty$. Furthermore, we provide ergodic upper bounds and ergodic convergence rates in expectation for $\|M(X(t))\|^2$ and $\langle M(X(t), X(t)-x^*\rangle$, where $x^*$ is an arbitrary zero of the monotone operator. Subsequently, we apply these findings to a minimax problem. Finally, we analyze two temporal discretizations of the continuous-time models, resulting in stochastic variants of the Optimistic Gradient Descent Ascent and Extragradient methods, respectively, and assess their convergence properties. △ Less

Submitted 27 April, 2024; originally announced April 2024.

MSC Class: 34F05; 47H05; 60H10; 68W20

A full splitting algorithm for fractional programs with structured numerators and denominators

Authors: Radu Ioan Boţ, Guoyin Li, Min Tao

Abstract: In this paper, we consider a class of nonconvex and nonsmooth fractional programming problems, which involve the sum of a convex, possibly nonsmooth function composed with a linear operator and a differentiable, possibly nonconvex function in the numerator and a convex, possibly nonsmooth function composed with a linear operator in the denominator. These problems have applications in various field… ▽ More In this paper, we consider a class of nonconvex and nonsmooth fractional programming problems, which involve the sum of a convex, possibly nonsmooth function composed with a linear operator and a differentiable, possibly nonconvex function in the numerator and a convex, possibly nonsmooth function composed with a linear operator in the denominator. These problems have applications in various fields, including CT reconstruction and sparse signal recovery. We propose an adaptive full-splitting proximal subgradient algorithm with an extrapolated step that addresses the challenge of evaluating the composition in the numerator by decoupling the linear operator from the nonsmooth component. We specifically evaluate the nonsmooth function using its proximal operator, while the linear operator is assessed through forward evaluations. Furthermore, the smooth component in the numerator is evaluated through its gradient, the nonsmooth component in the denominator is managed using its subgradient, and the linear operator in the denominator is also assessed through forward evaluations. We demonstrate subsequential convergence toward an approximate lifted stationary point and ensure global convergence under the Kurdyka-Łojasiewicz property, all achieved {\it without relying on any full-row rank assumptions regarding the linear operators}. We further explain the reasoning behind aiming for an approximate lifted stationary point. This is exemplified by constructing a scenario illustrating that the algorithm could diverge when seeking exact solutions. Lastly, we present a practical iteration of the algorithm incorporating a nonmonotone line search, significantly enhancing its convergence performance. Our theoretical findings are validated through simulations involving limited-angle CT reconstruction and the robust sharp ratio minimization problem. △ Less

Submitted 21 December, 2023; originally announced December 2023.

Comments: 27 pages, 4 figures

MSC Class: 90C26; 90C32; 49M27; 65K05

Fast Forward-Backward splitting for monotone inclusions with a convergence rate of the tangent residual of $o(1/k)$

Authors: Radu Ioan Bot, Dang-Khoa Nguyen, Chunxiang Zong

Abstract: We address the problem of finding the zeros of the sum of a maximally monotone operator and a cocoercive operator. Our approach introduces a modification to the forward-backward method by integrating an inertial/momentum term alongside a correction term. We demonstrate that the sequence of iterations thus generated converges weakly towards a solution for the monotone inclusion problem. Furthermore… ▽ More We address the problem of finding the zeros of the sum of a maximally monotone operator and a cocoercive operator. Our approach introduces a modification to the forward-backward method by integrating an inertial/momentum term alongside a correction term. We demonstrate that the sequence of iterations thus generated converges weakly towards a solution for the monotone inclusion problem. Furthermore, our analysis reveals an outstanding attribute of our algorithm: it displays rates of convergence of the order $o(1/k)$ for the discrete velocity and the tangent residual approaching zero. These rates for tangent residuals can be extended to fixed-point residuals frequently discussed in the existing literature. Specifically, when applied to minimize a nonsmooth convex function subject to linear constraints, our method evolves into a primal-dual full splitting algorithm. Notably, alongside the convergence of iterates, this algorithm possesses a remarkable characteristic of nonergodic/last iterate $o(1/k)$ convergence rates for both the function value and the feasibility measure. Our algorithm showcases the most advanced convergence and convergence rate outcomes among primal-dual full splitting algorithms when minimizing nonsmooth convex functions with linear constraints. △ Less

Submitted 19 December, 2023; originally announced December 2023.

Comments: 36 pages, 8 figures

A Fast Optimistic Method for Monotone Variational Inequalities

Authors: Michael Sedlmayer, Dang-Khoa Nguyen, Radu Ioan Bot

Abstract: We study monotone variational inequalities that can arise as optimality conditions for constrained convex optimisation or convex-concave minimax problems and propose a novel algorithm that uses only one gradient/operator evaluation and one projection onto the constraint set per iteration. The algorithm, which we call fOGDA-VI, achieves a $o \left( \frac{1}{k} \right)$ rate of convergence in terms… ▽ More We study monotone variational inequalities that can arise as optimality conditions for constrained convex optimisation or convex-concave minimax problems and propose a novel algorithm that uses only one gradient/operator evaluation and one projection onto the constraint set per iteration. The algorithm, which we call fOGDA-VI, achieves a $o \left( \frac{1}{k} \right)$ rate of convergence in terms of the restricted gap function as well as the natural residual for the last iterate. Moreover, we provide a convergence guarantee for the sequence of iterates to a solution of the variational inequality. These are the best theoretical convergence results for numerical methods for (only) monotone variational inequalities reported in the literature. To empirically validate our algorithm we investigate a two-player matrix game with mixed strategies of the two players. Concluding, we show promising results regarding the application of fOGDA-VI to the training of generative adversarial nets. △ Less

Submitted 20 July, 2023; originally announced July 2023.

Comments: Accepted at ICML 2023

Accelerated Griffin-Lim algorithm: A fast and provably converging numerical method for phase retrieval

Authors: Rossen Nenov, Dang-Khoa Nguyen, Peter Balazs, Radu Ioan Bot

Abstract: The recovery of a signal from the magnitudes of its transformation, like the Fourier transform, is known as the phase retrieval problem and is of big relevance in various fields of engineering and applied physics. In this paper, we present a fast inertial/momentum based algorithm for the phase retrieval problem and we prove a convergence guarantee for the new algorithm and for the Fast Griffin-Lim… ▽ More The recovery of a signal from the magnitudes of its transformation, like the Fourier transform, is known as the phase retrieval problem and is of big relevance in various fields of engineering and applied physics. In this paper, we present a fast inertial/momentum based algorithm for the phase retrieval problem and we prove a convergence guarantee for the new algorithm and for the Fast Griffin-Lim algorithm, whose convergence remained unproven in the past decade. In the final chapter, we compare the algorithm for the Short Time Fourier transform phase retrieval with the Griffin-Lim algorithm and FGLA and to other iterative algorithms typically used for this type of problem. △ Less

Submitted 21 June, 2023; originally announced June 2023.

Fast convex optimization via closed-loop time scaling of gradient dynamics

Authors: Hedy Attouch, Radu Ioan Bot, Dang-Khoa Nguyen

Abstract: In a Hilbert setting, for convex differentiable optimization, we develop a general framework for adaptive accelerated gradient methods. They are based on damped inertial dynamics where the coefficients are designed in a closed-loop way. Specifically, the dam** is a feedback control of the velocity, or of the gradient of the objective function. For this, we develop a closed-loop version of the ti… ▽ More In a Hilbert setting, for convex differentiable optimization, we develop a general framework for adaptive accelerated gradient methods. They are based on damped inertial dynamics where the coefficients are designed in a closed-loop way. Specifically, the dam** is a feedback control of the velocity, or of the gradient of the objective function. For this, we develop a closed-loop version of the time scaling and averaging technique introduced by the authors. We thus obtain autonomous inertial dynamics which involve vanishing viscous dam** and implicit Hessian driven dam**. By simply using the convergence rates for the continuous steepest descent and Jensen's inequality, without the need for further Lyapunov analysis, we show that the trajectories have several remarkable properties at once: they ensure fast convergence of values, fast convergence of the gradients towards zero, and they converge to optimal solutions. Our approach leads to parallel algorithmic results, that we study in the case of proximal algorithms. These are among the very first general results of this type obtained using autonomous dynamics. △ Less

Submitted 2 January, 2023; originally announced January 2023.

MSC Class: 37N40; 46N10; 49M30; 65B99; 65K05; 65K10; 90B50; 90C25

Fast convex optimization via time scale and averaging of the steepest descent

Authors: Hedy Attouch, Radu Ioan Bot, Dang-Khoa Nguyen

Abstract: In a Hilbert setting, we develop a gradient-based dynamic approach for fast solving convex optimization problems. By applying time scaling, averaging, and perturbation techniques to the continuous steepest descent (SD), we obtain high-resolution ODEs of the Nesterov and Ravine methods. These dynamics involve asymptotically vanishing viscous dam** and Hessian driven dam** (either in explicit or… ▽ More In a Hilbert setting, we develop a gradient-based dynamic approach for fast solving convex optimization problems. By applying time scaling, averaging, and perturbation techniques to the continuous steepest descent (SD), we obtain high-resolution ODEs of the Nesterov and Ravine methods. These dynamics involve asymptotically vanishing viscous dam** and Hessian driven dam** (either in explicit or implicit form). Mathematical analysis does not require develo** a Lyapunov analysis for inertial systems. We simply exploit classical convergence results for (SD) and its external perturbation version, then use tools of differential and integral calculus, including Jensen's inequality. The method is flexible and by way of illustration we show how it applies starting from other important dynamics in optimization. We consider the case where the initial dynamics is the regularized Newton method, then the case where the starting dynamics is the differential inclusion associated with a convex lower semicontinuous potential, and finally we show that the technique can be naturally extended to the case of a monotone cocoercive operator. Our approach leads to parallel algorithmic results, which we study in the case of fast gradient and proximal algorithms. Our averaging technique shows new links between the Nesterov and Ravine methods. △ Less

Submitted 3 May, 2023; v1 submitted 17 August, 2022; originally announced August 2022.

Comments: Improved convergence rates and new links between the Nesterov and Ravine methods are also provided

Fast Krasnosel'skii-Mann algorithm with a convergence rate of the fixed point iteration of $o\left(\frac{1}{k}\right)$

Authors: Radu Ioan Bot, Dang-Khoa Nguyen

Abstract: The Krasnosel'skii-Mann (KM) algorithm is the most fundamental iterative scheme designed to find a fixed point of an averaged operator in the framework of a real Hilbert space, since it lies at the heart of various numerical algorithms for solving monotone inclusions and convex optimization problems. We enhance the Krasnosel'skii-Mann algorithm with Nesterov's momentum updates and show that the re… ▽ More The Krasnosel'skii-Mann (KM) algorithm is the most fundamental iterative scheme designed to find a fixed point of an averaged operator in the framework of a real Hilbert space, since it lies at the heart of various numerical algorithms for solving monotone inclusions and convex optimization problems. We enhance the Krasnosel'skii-Mann algorithm with Nesterov's momentum updates and show that the resulting numerical method exhibits a convergence rate for the fixed point residual of $o(1/k)$ while preserving the weak convergence of the iterates to a fixed point of the operator. Numerical experiments illustrate the superiority of the resulting so-called Fast KM algorithm over various fixed point iterative schemes, and also its oscillatory behavior, which is a specific of Nesterov's momentum optimization algorithms. △ Less

Submitted 24 August, 2023; v1 submitted 19 June, 2022; originally announced June 2022.

MSC Class: 47J20; 47H05; 65K15; 65Y20

Fast Optimistic Gradient Descent Ascent (OGDA) method in continuous and discrete time

Authors: Radu Ioan Bot, Ernö Robert Csetnek, Dang-Khoa Nguyen

Abstract: In the framework of real Hilbert spaces we study continuous in time dynamics as well as numerical algorithms for the problem of approaching the set of zeros of a single-valued monotone and continuous operator $V$. The starting poin is a second order dynamical system that combines a vanishing dam** term with the time derivative of $V$ along the trajectory. Our method exhibits fast convergence rat… ▽ More In the framework of real Hilbert spaces we study continuous in time dynamics as well as numerical algorithms for the problem of approaching the set of zeros of a single-valued monotone and continuous operator $V$. The starting poin is a second order dynamical system that combines a vanishing dam** term with the time derivative of $V$ along the trajectory. Our method exhibits fast convergence rates of order $o \left( \frac{1}{tβ(t)} \right)$ for $\|V(z(t))\|$, wher $β(\cdot)$ is a positive nondecreasing function satisfying a growth condition, and also for the restricted gap function. We also prove the weak convergence of the trajectory to a zero of $V$. Temporal discretizations of the dynamical system generate implicit and explicit numerical algorithms, which can be both seen as accelerated versions of the Optimistic Gradient Descent Ascent (OGDA) method, for which we prove that the generated sequence of iterates shares the asymptotic features of the continuous dynamics. In particular we show for the implicit numerical algorithm convergence rates of order $o \left( \frac{1}{kβ_k} \right)$ for $\|V(z^k)\|$ and the restricted gap function, where $(β_k)_{k \geq 0}$ is a positive nondecreasing sequence satisfying a growth condition. For the explicit numerical algorithm we show by additionally assuming that the operator $V$ is Lipschitz continuous convergence rates of order $o \left( \frac{1}{k} \right)$ for $\|V(z^k)\|$ and the restricted gap function. All convergence rate statements are last iterate convergence results; in addition we prove for both algorithms the convergence of the iterates to a zero of $V$. To our knowledge, our study exhibits the best known convergence rate results for monotone equations. Numerical experiments indicate the overwhelming superiority of our explicit numerical algorithm over other methods for monotone equations. △ Less

Submitted 22 February, 2024; v1 submitted 21 March, 2022; originally announced March 2022.

Comments: 43 pages

MSC Class: 47J20; 47H05; 65K10; 65K15; 65Y20; 90C30; 90C52

A fast continuous time approach with time scaling for nonsmooth convex optimization

Authors: Radu Ioan Bot, Mikhail A. Karapetyants

Abstract: In a Hilbert setting we study the convergence properties of a second order in time dynamical system combining viscous and Hessian-driven dam** with time scaling in relation with the minimization of a nonsmooth and convex function. The system is formulated in terms of the gradient of the Moreau envelope of the objective function with time-dependent parameter. We show fast convergence rates for th… ▽ More In a Hilbert setting we study the convergence properties of a second order in time dynamical system combining viscous and Hessian-driven dam** with time scaling in relation with the minimization of a nonsmooth and convex function. The system is formulated in terms of the gradient of the Moreau envelope of the objective function with time-dependent parameter. We show fast convergence rates for the Moreau envelope and its gradient along the trajectory, and also for the velocity of the system. From here we derive fast convergence rates for the objective function along a path which is the image of the trajectory of the system through the proximal operator of the first. Moreover, we prove the weak convergence of the trajectory of the system to a global minimizer of the objective function. Finally, we provide multiple numerical examples which illustrate the theoretical results. △ Less

Submitted 1 March, 2022; originally announced March 2022.

MSC Class: 37N40; 46N10; 49M99; 65K05; 65K10; 90C25

A primal-dual splitting algorithm for composite monotone inclusions with minimal lifting

Authors: Francisco J. Aragón-Artacho, Radu I. Boţ, David Torregrosa-Belén

Abstract: In this work, we study resolvent splitting algorithms for solving composite monotone inclusion problems. The objective of these general problems is finding a zero in the sum of maximally monotone operators composed with linear operators. Our main contribution is establishing the first primal-dual splitting algorithm for composite monotone inclusions with minimal lifting. Specifically, the proposed… ▽ More In this work, we study resolvent splitting algorithms for solving composite monotone inclusion problems. The objective of these general problems is finding a zero in the sum of maximally monotone operators composed with linear operators. Our main contribution is establishing the first primal-dual splitting algorithm for composite monotone inclusions with minimal lifting. Specifically, the proposed scheme reduces the dimension of the product space where the underlying fixed point operator is defined, in comparison to other algorithms, without requiring additional evaluations of the resolvent operators. We prove the convergence of this new algorithm and analyze its performance in a problem arising in image deblurring and denoising. This work also contributes to the theory of resolvent splitting algorithms by extending the minimal lifting theorem recently proved by Malitsky and Tam to schemes with resolvent parameters. △ Less

Submitted 19 February, 2022; originally announced February 2022.

MSC Class: 47H05; 65K10; 90C30

Second order splitting dynamics with vanishing dam** for additively structured monotone inclusions

Authors: Radu Ioan Bot, David Alexander Hulett

Abstract: In the framework of a real Hilbert space, we address the problem of finding the zeros of the sum of a maximally monotone operator $A$ and a cocoercive operator $B$. We study the asymptotic behaviour of the trajectories generated by a second order equation with vanishing dam**, attached to this problem, and governed by a time-dependent forward-backward-type operator. This is a splitting system, a… ▽ More In the framework of a real Hilbert space, we address the problem of finding the zeros of the sum of a maximally monotone operator $A$ and a cocoercive operator $B$. We study the asymptotic behaviour of the trajectories generated by a second order equation with vanishing dam**, attached to this problem, and governed by a time-dependent forward-backward-type operator. This is a splitting system, as it only requires forward evaluations of $B$ and backward evaluations of $A$. A proper tuning of the system parameters ensures the weak convergence of the trajectories to the set of zeros of $A + B$, as well as fast convergence of the velocities towards zero. A particular case of our system allows to derive fast convergence rates for the problem of minimizing the sum of a proper, convex and lower semicontinuous function and a smooth and convex function with Lipschitz continuous gradient. We illustrate the theoretical outcomes by numerical experiments. △ Less

Submitted 4 January, 2022; originally announced January 2022.

Fast Augmented Lagrangian Method in the convex regime with convergence guarantees for the iterates

Authors: Radu Ioan Bot, Ernö Robert Csetnek, Dang-Khoa Nguyen

Abstract: This work aims to minimize a continuously differentiable convex function with Lipschitz continuous gradient under linear equality constraints. The proposed inertial algorithm results from the discretization of the second-order primal-dual dynamical system with asymptotically vanishing dam** term addressed by Bot and Nguyen in [Bot, Nguyen, JDE, 2021], and it is formulated in terms of the Augment… ▽ More This work aims to minimize a continuously differentiable convex function with Lipschitz continuous gradient under linear equality constraints. The proposed inertial algorithm results from the discretization of the second-order primal-dual dynamical system with asymptotically vanishing dam** term addressed by Bot and Nguyen in [Bot, Nguyen, JDE, 2021], and it is formulated in terms of the Augmented Lagrangian associated with the minimization problem. The general setting we consider for the inertial parameters covers the three classical rules by Nesterov, Chambolle-Dossal and Attouch-Cabot used in the literature to formulate fast gradient methods. For these rules, we obtain in the convex regime convergence rates of order ${\cal O}(1/k^{2})$ for the primal-dual gap, the feasibility measure, and the objective function value. In addition, we prove that the generated sequence of primal-dual iterates converges to a primal-dual solution in a general setting that covers the two latter rules. This is the first result which provides the convergence of the sequence of iterates generated by a fast algorithm for linearly constrained convex optimization problems without additional assumptions such as strong convexity. We also emphasize that all convergence results of this paper are compatible with the ones obtained in [Bot, Nguyen, JDE, 2021] in the continuous setting. △ Less

Submitted 1 August, 2022; v1 submitted 17 November, 2021; originally announced November 2021.

Comments: 37 pages

MSC Class: 49M29; 65K05; 68Q25; 90C25; 65B99

Improved convergence rates and trajectory convergence for primal-dual dynamical systems with vanishing dam**

Authors: Radu Ioan Bot, Dang-Khoa Nguyen

Abstract: In this work, we approach the minimization of a continuously differentiable convex function under linear equality constraints by a second-order dynamical system with asymptotically vanishing dam** term. The system is formulated in terms of the augmented Lagrangian associated to the minimization problem. We show fast convergence of the primal-dual gap, the feasibility measure, and the objective f… ▽ More In this work, we approach the minimization of a continuously differentiable convex function under linear equality constraints by a second-order dynamical system with asymptotically vanishing dam** term. The system is formulated in terms of the augmented Lagrangian associated to the minimization problem. We show fast convergence of the primal-dual gap, the feasibility measure, and the objective function value along the generated trajectories. In case the objective function has Lipschitz continuous gradient, we show that the primal-dual trajectory asymptotically weakly converges to a primal-dual optimal solution of the underlying minimization problem. To the best of our knowledge, this is the first result which guarantees the convergence of the trajectory generated by a primal-dual dynamical system with asymptotic vanishing dam**. Moreover, we will rediscover in case of the unconstrained minimization of a convex differentiable function with Lipschitz continuous gradient all convergence statements obtained in the literature for Nesterov's accelerated gradient method. △ Less

Submitted 23 June, 2021; originally announced June 2021.

MSC Class: 37N40; 46N10; 65K10; 90C25

An accelerated minimax algorithm for convex-concave saddle point problems with nonsmooth coupling function

Authors: Radu Ioan Bot, Ernö Robert Csetnek, Michael Sedlmayer

Abstract: In this work we aim to solve a convex-concave saddle point problem, where the convex-concave coupling function is smooth in one variable and nonsmooth in the other and not assumed to be linear in either. The problem is augmented by a nonsmooth regulariser in the smooth component. We propose and investigate a novel algorithm under the name of OGAProx, consisting of an optimistic gradient ascent ste… ▽ More In this work we aim to solve a convex-concave saddle point problem, where the convex-concave coupling function is smooth in one variable and nonsmooth in the other and not assumed to be linear in either. The problem is augmented by a nonsmooth regulariser in the smooth component. We propose and investigate a novel algorithm under the name of OGAProx, consisting of an optimistic gradient ascent step in the smooth variable coupled with a proximal step of the regulariser, and which is alternated with a {proximal step} in the nonsmooth component of the coupling function. We consider the situations convex-concave, convex-strongly concave and strongly convex-strongly concave related to the saddle point problem under investigation. Regarding iterates we obtain (weak) convergence, a convergence rate of order $ \mathcal{O}(\frac{1}{K}) $ and linear convergence like $\mathcal{O}(θ^{K})$ with $ θ< 1 $, respectively. In terms of function values we obtain ergodic convergence rates of order $ \mathcal{O}(\frac{1}{K}) $, $ \mathcal{O}(\frac{1}{K^{2}}) $ and $ \mathcal{O}(θ^{K}) $ with $ θ< 1 $, respectively. We validate our theoretical considerations on a nonsmooth-linear saddle point problem, the training of multi kernel support vector machines and a classification problem incorporating minimax group fairness. △ Less

Submitted 6 August, 2021; v1 submitted 13 April, 2021; originally announced April 2021.

Comments: A new numerical experiment on minimax group fairness has been added

Inertial Proximal Block Coordinate Method for a Class of Nonsmooth Sum-of-Ratios Optimization Problems

Authors: Radu Ioan Boţ, Minh N. Dao, Guoyin Li

Abstract: In this paper, we consider a class of nonsmooth sum-of-ratios fractional optimization problems with block structure. This model class is ubiquitous and encompasses several important nonsmooth optimization problems in the literature. We first propose an inertial proximal block coordinate method for solving this class of problems by exploiting the underlying structure. The global convergence of our… ▽ More In this paper, we consider a class of nonsmooth sum-of-ratios fractional optimization problems with block structure. This model class is ubiquitous and encompasses several important nonsmooth optimization problems in the literature. We first propose an inertial proximal block coordinate method for solving this class of problems by exploiting the underlying structure. The global convergence of our method is guaranteed under the Kurdyka--Lojasiewicz (KL) property and some mild assumptions. We then identify the explicit exponents of the KL property for three important structured fractional optimization problems. In particular, for the sparse generalized eigenvalue problem with either cardinality regularization or sparsity constraint, we show that the KL exponents are 1/2, and so, the proposed method exhibits linear convergence rate. Finally, we illustrate our theoretical results with both analytic and simulated numerical examples. △ Less

Submitted 18 May, 2023; v1 submitted 19 November, 2020; originally announced November 2020.

Journal ref: SIAM Journal on Optimization, 33(2):361--393, 2023

Fast optimization via inertial dynamics with closed-loop dam**

Authors: Hedy Attouch, Radu Ioan Bot, Ernö Robert Csetnek

Abstract: In a Hilbert space $H$, in order to develop fast optimization methods, we analyze the asymptotic behavior, as time $t$ tends to infinity, of inertial continuous dynamics where the dam** acts as a closed-loop control. The function $f: H \to R$ to be minimized (not necessarily convex) enters the dynamic through it gradient, which is assumed to be Lipschitz continuous on the bounded subsets of $H$.… ▽ More In a Hilbert space $H$, in order to develop fast optimization methods, we analyze the asymptotic behavior, as time $t$ tends to infinity, of inertial continuous dynamics where the dam** acts as a closed-loop control. The function $f: H \to R$ to be minimized (not necessarily convex) enters the dynamic through it gradient, which is assumed to be Lipschitz continuous on the bounded subsets of $H$. This gives autonomous dynamical systems with nonlinear dam** and nonlinear driving force. We first consider the case where the dam** term $\partial φ(\dot{x}(t))$ acts as a closed-loop control of the velocity. The dam** potential $φ: H \to [0,+\infty)$ is a convex continuous function which achieves its minimum at the origin. We show the existence and uniqueness of a global solution to the associated Cauchy problem. Then, we analyze the asymptotic convergence properties of the generated trajectories generated. We use techniques from optimization, control theory, and PDE's: Lyapunov analysis based on the decreasing property of an energy-like function, quasi-gradient and Kurdyka-Lojasiewicz theory, monotone operator theory for wave-like equations. Convergence rates are obtained based on the geometric properties of the data $f$ and $φ$. When $f$ is strongly convex, we give general conditions which provide exponential convergence rates. Then, we extend the results to the case where an additional Hessian-driven dam** enters the dynamic, which reduces the oscillations. Finally, we consider an inertial system involving jointly the velocity $\dot{x}(t)$ and the gradient $\nabla f(x(t))$. In addition to its original results, this work surveys the numerous works devoted in recent years to the interaction between continuous damped inertial dynamics and numerical algorithms for optimization, with the emphasis on autonomous systems, closed-loop adaptive procedures, and convergence rates. △ Less

Submitted 11 January, 2021; v1 submitted 5 August, 2020; originally announced August 2020.

Comments: 63 pages, 6 figures, updated introduction, updated list of references

MSC Class: 37N40; 46N10; 49M30; 65K05; 65K10; 90B50; 90C25

Journal ref: Journal of the European Mathematical Society, 2021

Alternating proximal-gradient steps for (stochastic) nonconvex-concave minimax problems

Authors: Radu Ioan Boţ, Axel Böhm

Abstract: Minimax problems of the form $\min_x \max_y Ψ(x,y)$ have attracted increased interest largely due to advances in machine learning, in particular generative adversarial networks. These are typically trained using variants of stochastic gradient descent for the two players. Although convex-concave problems are well understood with many efficient solution methods to choose from, theoretical guarant… ▽ More Minimax problems of the form $\min_x \max_y Ψ(x,y)$ have attracted increased interest largely due to advances in machine learning, in particular generative adversarial networks. These are typically trained using variants of stochastic gradient descent for the two players. Although convex-concave problems are well understood with many efficient solution methods to choose from, theoretical guarantees outside of this setting are sometimes lacking even for the simplest algorithms. In particular, this is the case for alternating gradient descent ascent, where the two agents take turns updating their strategies. To partially close this gap in the literature we prove a novel global convergence rate for the stochastic version of this method for finding a critical point of $g(\cdot) := \max_y Ψ(\cdot,y)$ in a setting which is not convex-concave. △ Less

Submitted 13 April, 2023; v1 submitted 27 July, 2020; originally announced July 2020.

Two steps at a time -- taking GAN training in stride with Tseng's method

Authors: Axel Böhm, Michael Sedlmayer, Ernö Robert Csetnek, Radu Ioan Boţ

Abstract: Motivated by the training of Generative Adversarial Networks (GANs), we study methods for solving minimax problems with additional nonsmooth regularizers. We do so by employing \emph{monotone operator} theory, in particular the \emph{Forward-Backward-Forward (FBF)} method, which avoids the known issue of limit cycling by correcting each update by a second gradient evaluation. Furthermore, we propo… ▽ More Motivated by the training of Generative Adversarial Networks (GANs), we study methods for solving minimax problems with additional nonsmooth regularizers. We do so by employing \emph{monotone operator} theory, in particular the \emph{Forward-Backward-Forward (FBF)} method, which avoids the known issue of limit cycling by correcting each update by a second gradient evaluation. Furthermore, we propose a seemingly new scheme which recycles old gradients to mitigate the additional computational cost. In doing so we rediscover a known method, related to \emph{Optimistic Gradient Descent Ascent (OGDA)}. For both schemes we prove novel convergence rates for convex-concave minimax problems via a unifying approach. The derived error bounds are in terms of the gap function for the ergodic iterates. For the deterministic and the stochastic problem we show a convergence rate of $\mathcal{O}(1/k)$ and $\mathcal{O}(1/\sqrt{k})$, respectively. We complement our theoretical results with empirical improvements in the training of Wasserstein GANs on the CIFAR10 dataset. △ Less

Submitted 16 June, 2020; originally announced June 2020.

Comments: 19 pages, 5 figures

A Relaxed Inertial Forward-Backward-Forward Algorithm for Solving Monotone Inclusions with Application to GANs

Authors: Radu Ioan Bot, Michael Sedlmayer, Phan Tu Vuong

Abstract: We introduce a relaxed inertial forward-backward-forward (RIFBF) splitting algorithm for approaching the set of zeros of the sum of a maximally monotone operator and a single-valued monotone and Lipschitz continuous operator. This work aims to extend Tseng's forward-backward-forward method by both using inertial effects as well as relaxation parameters. We formulate first a second order dynamical… ▽ More We introduce a relaxed inertial forward-backward-forward (RIFBF) splitting algorithm for approaching the set of zeros of the sum of a maximally monotone operator and a single-valued monotone and Lipschitz continuous operator. This work aims to extend Tseng's forward-backward-forward method by both using inertial effects as well as relaxation parameters. We formulate first a second order dynamical system which approaches the solution set of the monotone inclusion problem to be solved and provide an asymptotic analysis for its trajectories. We provide for RIFBF, which follows by explicit time discretization, a convergence analysis in the general monotone case as well as when applied to the solving of pseudo-monotone variational inequalities. We illustrate the proposed method by applications to a bilinear saddle point problem, in the context of which we also emphasize the interplay between the inertial and the relaxation parameters, and to the training of Generative Adversarial Networks (GANs). △ Less

Submitted 22 March, 2020; v1 submitted 17 March, 2020; originally announced March 2020.

MSC Class: 47J20; 90C25; 90C30; 90C52

Extrapolated Proximal Subgradient Algorithms for Nonconvex and Nonsmooth Fractional Programs

Authors: Radu Ioan Boţ, Minh N. Dao, Guoyin Li

Abstract: In this paper, we consider a broad class of nonsmooth and nonconvex fractional programs, where the numerator can be written as the sum of a continuously differentiable convex function whose gradient is Lipschitz continuous and a proper lower semicontinuous (possibly nonconvex) function, and the denominator is weakly convex over the constraint set. This model problem includes the composite optimiza… ▽ More In this paper, we consider a broad class of nonsmooth and nonconvex fractional programs, where the numerator can be written as the sum of a continuously differentiable convex function whose gradient is Lipschitz continuous and a proper lower semicontinuous (possibly nonconvex) function, and the denominator is weakly convex over the constraint set. This model problem includes the composite optimization problems studied extensively lately, and encompasses many important modern fractional optimization problems arising from diverse areas such as the recently proposed scale invariant sparse signal reconstruction problem in signal processing. We propose a proximal subgradient algorithm with extrapolations for solving this optimization model and show that the iterated sequence generated by the algorithm is bounded and any of its limit points is a stationary point of the model problem. The choice of our extrapolation parameter is flexible and includes the popular extrapolation parameter adopted in the restarted Fast Iterative Shrinking-Threshold Algorithm (FISTA). By providing a unified analysis framework of descent methods, we establish the convergence of the full sequence under the assumption that a suitable merit function satisfies the Kurdyka--Łojasiewicz (KL) property. In particular, our algorithm exhibits linear convergence for the scale invariant sparse signal reconstruction problem and the Rayleigh quotient problem over spherical constraint. In the case where the denominator is the maximum of finitely many continuously differentiable weakly convex functions, we also propose an enhanced extrapolated proximal subgradient algorithm with guaranteed convergence to a stronger notion of stationary points of the model problem. Finally, we illustrate the proposed methods by both analytical and simulated numerical examples. △ Less

Submitted 16 October, 2020; v1 submitted 9 March, 2020; originally announced March 2020.

Comments: Revised version: Oct. 16, 2020

Journal ref: Mathematics of Operations Research, 2021

A forward-backward dynamical approach for nonsmooth problems with block structure coupled by a smooth function

Authors: Radu Ioan Bot, Laura Kanzler

Abstract: In this paper we aim to minimize the sum of two nonsmooth (possibly also nonconvex) functions in separate variables connected by a smooth coupling function. To tackle this problem we chose a continuous forward-backward approach and introduce a dynamical system which is formulated by means of the partial gradients of the smooth coupling function and the proximal point operator of the two nonsmooth… ▽ More In this paper we aim to minimize the sum of two nonsmooth (possibly also nonconvex) functions in separate variables connected by a smooth coupling function. To tackle this problem we chose a continuous forward-backward approach and introduce a dynamical system which is formulated by means of the partial gradients of the smooth coupling function and the proximal point operator of the two nonsmooth functions. Moreover, we consider variable rates of implicitness of the resulting system. We discuss the existence and uniqueness of a solution and carry out the asymptotic analysis of its convergence behaviour to a critical point of the optimization problem, when a regularization of the objective function fulfills the Kurdyka-Lojasiewicz property. We further provide convergence rates for the solution trajectory in terms of the Lojasiewicz exponent. We conclude this work with numerical simulations which confirm and validate the analytical results. △ Less

Submitted 27 January, 2020; originally announced January 2020.

MSC Class: 34G25; 37N40; 49J52; 90C26; 90C56

Tikhonov regularization of a second order dynamical system with Hessian driven dam**

Authors: Radu Ioan Bot, Ernö Robert Csetnek, Szilárd Csaba László

Abstract: We investigate the asymptotic properties of the trajectories generated by a second-order dynamical system with Hessian driven dam** and a Tikhonov regularization term in connection with the minimization of a smooth convex function in Hilbert spaces. We obtain fast convergence results for the function values along the trajectories. The Tikhonov regularization term enables the derivation of strong… ▽ More We investigate the asymptotic properties of the trajectories generated by a second-order dynamical system with Hessian driven dam** and a Tikhonov regularization term in connection with the minimization of a smooth convex function in Hilbert spaces. We obtain fast convergence results for the function values along the trajectories. The Tikhonov regularization term enables the derivation of strong convergence results of the trajectory to the minimizer of the objective function of minimum norm. △ Less

Submitted 31 July, 2020; v1 submitted 28 November, 2019; originally announced November 2019.

MSC Class: 34G25; 47J25; 47H05; 90C26; 90C30; 65K10

Journal ref: Mathematical Programming, 2020

A strongly convergent Krasnosel'skiǐ-Mann-type algorithm for finding a common fixed point of a countably infinite family of nonexpansive operators in Hilbert spaces

Authors: Radu Ioan Bot, Dennis Meier

Abstract: In this article, we propose a Krasnosel'skiǐ-Mann-type algorithm for finding a common fixed point of a countably infinite family of nonexpansive operators $(T_n)_{n \geq 0}$ in Hilbert spaces. We formulate an asymptotic property which the family $(T_n)_{n \geq 0}$ has to fulfill such that the sequence generated by the algorithm converges strongly to the element in… ▽ More In this article, we propose a Krasnosel'skiǐ-Mann-type algorithm for finding a common fixed point of a countably infinite family of nonexpansive operators $(T_n)_{n \geq 0}$ in Hilbert spaces. We formulate an asymptotic property which the family $(T_n)_{n \geq 0}$ has to fulfill such that the sequence generated by the algorithm converges strongly to the element in $\bigcap_{n \geq 0} \operatorname{Fix} T_n$ with minimum norm. Based on this, we derive a forward-backward algorithm that allows variable step sizes and generates a sequence of iterates that converge strongly to the zero with minimum norm of the sum of a maximally monotone operator and a cocoercive one. We demonstrate the superiority of the forward-backward algorithm with variable step sizes over the one with constant step size by means of numerical experiments on variational image reconstruction and split feasibility problems in infinite dimensional Hilbert spaces. △ Less

Submitted 26 November, 2019; originally announced November 2019.

Inducing strong convergence of trajectories in dynamical systems associated to monotone inclusions with composite structure

Authors: Radu Ioan Boţ, Sorin-Mihai Grad, Dennis Meier, Mathias Staudigl

Abstract: In this work we investigate dynamical systems designed to approach the solution sets of inclusion problems involving the sum of two maximally monotone operators. Our aim is to design methods which guarantee strong convergence of trajectories towards the minimum norm solution of the underlying monotone inclusion problem. To that end, we investigate in detail the asymptotic behavior of dynamical sys… ▽ More In this work we investigate dynamical systems designed to approach the solution sets of inclusion problems involving the sum of two maximally monotone operators. Our aim is to design methods which guarantee strong convergence of trajectories towards the minimum norm solution of the underlying monotone inclusion problem. To that end, we investigate in detail the asymptotic behavior of dynamical systems perturbed by a Tikhonov regularization where either the maximally monotone operators themselves, or the vector field of the dynamical system is regularized. In both cases we prove strong convergence of the trajectories towards minimum norm solutions to an underlying monotone inclusion problem, and we illustrate numerically qualitative differences between these two complementary regularization strategies. The so-constructed dynamical systems are either of Krasnoselskii-Mann, of forward-backward type or of forward-backward-forward type, and with the help of injected regularization we demonstrate seminal results on the strong convergence of Hilbert space valued evolutions designed to solve monotone inclusion and equilibrium problems. △ Less

Submitted 12 November, 2019; originally announced November 2019.

Comments: 30 pages, 21 figures

MSC Class: 34G25; 37N40; 47H05; 90C25

Journal ref: Advances in Nonlinear Analysis 10:450-476, 2021

A primal-dual dynamical approach to structured convex minimization problems

Authors: Radu Ioan Bot, Ernö Robert Csetnek, Szilard Laszlo

Abstract: In this paper we propose a primal-dual dynamical approach to the minimization of a structured convex function consisting of a smooth term, a nonsmooth term, and the composition of another nonsmooth term with a linear continuous operator. In this scope we introduce a dynamical system for which we prove that its trajectories asymptotically converge to a saddle point of the Lagrangian of the underlyi… ▽ More In this paper we propose a primal-dual dynamical approach to the minimization of a structured convex function consisting of a smooth term, a nonsmooth term, and the composition of another nonsmooth term with a linear continuous operator. In this scope we introduce a dynamical system for which we prove that its trajectories asymptotically converge to a saddle point of the Lagrangian of the underlying convex minimization problem as time tends to infinity. In addition, we provide rates for both the violation of the feasibility condition by the ergodic trajectories and the convergence of the objective function along these ergodic trajectories to its minimal value. Explicit time discretization of the dynamical system results in a numerical algorithm which is a combination of the linearized proximal method of multipliers and the proximal ADMM algorithm. △ Less

Submitted 31 July, 2020; v1 submitted 20 May, 2019; originally announced May 2019.

MSC Class: 37N40; 49N15; 90C25; 90C46

Journal ref: Journal of Differential Equations, 2020

Variable smoothing for convex optimization problems using stochastic gradients

Authors: Radu Ioan Bot, Axel Böhm

Abstract: We aim to solve a structured convex optimization problem, where a nonsmooth function is composed with a linear operator. When opting for full splitting schemes, usually, primal-dual type methods are employed as they are effective and also well studied. However, under the additional assumption of Lipschitz continuity of the nonsmooth function which is composed with the linear operator we can derive… ▽ More We aim to solve a structured convex optimization problem, where a nonsmooth function is composed with a linear operator. When opting for full splitting schemes, usually, primal-dual type methods are employed as they are effective and also well studied. However, under the additional assumption of Lipschitz continuity of the nonsmooth function which is composed with the linear operator we can derive novel algorithms through regularization via the Moreau envelope. Furthermore, we tackle large scale problems by means of stochastic oracle calls, very similar to stochastic gradient techniques. Applications to total variational denoising and deblurring are provided. △ Less

Submitted 16 May, 2019; originally announced May 2019.

Forward-backward-forward methods with variance reduction for stochastic variational inequalities

Authors: Radu Ioan Bot, Panayotis Mertikopoulos, Mathias Staudigl, Phan Tu Vuong

Abstract: We develop a new stochastic algorithm with variance reduction for solving pseudo-monotone stochastic variational inequalities. Our method builds on Tseng's forward-backward-forward (FBF) algorithm, which is known in the deterministic literature to be a valuable alternative to Korpelevich's extragradient method when solving variational inequalities over a convex and closed set governed by pseudo-mo… ▽ More We develop a new stochastic algorithm with variance reduction for solving pseudo-monotone stochastic variational inequalities. Our method builds on Tseng's forward-backward-forward (FBF) algorithm, which is known in the deterministic literature to be a valuable alternative to Korpelevich's extragradient method when solving variational inequalities over a convex and closed set governed by pseudo-monotone, Lipschitz continuous operators. The main computational advantage of Tseng's algorithm is that it relies only on a single projection step and two independent queries of a stochastic oracle. Our algorithm incorporates a variance reduction mechanism and leads to almost sure (a.s.) convergence to an optimal solution. To the best of our knowledge, this is the first stochastic look-ahead algorithm achieving this by using only a single projection at each iteration.. △ Less

Submitted 8 February, 2019; originally announced February 2019.

Comments: 34 pages, 11 figures

MSC Class: Primary 65K15; 62L20; secondary 90C15; 90C33

The Forward-Backward-Forward Method from continuous and discrete perspective for pseudo-monotone variational inequalities in Hilbert spaces

Authors: Radu Ioan Bot, Ernö Robert Csetnek, Phan Tu Vuong

Abstract: Tseng's forward-backward-forward algorithm is a valuable alternative for Korpelevich's extragradient method when solving variational inequalities over a convex and closed set governed by monotone and Lipschitz continuous operators, as it requires in every step only one projection operation. However, it is well-known that Korpelevich's method converges and can therefore be used also for solving var… ▽ More Tseng's forward-backward-forward algorithm is a valuable alternative for Korpelevich's extragradient method when solving variational inequalities over a convex and closed set governed by monotone and Lipschitz continuous operators, as it requires in every step only one projection operation. However, it is well-known that Korpelevich's method converges and can therefore be used also for solving variational inequalities governed by pseudo-monotone and Lipschitz continuous operators. In this paper, we first associate to a pseudo-monotone variational inequality a forward-backward-forward dynamical system and carry out an asymptotic analysis for the generated trajectories. The explicit time discretization of this system results into Tseng's forward-backward-forward algorithm with relaxation parameters, which we prove to converge also when it is applied to pseudo-monotone variational inequalities. In addition, we show that linear convergence is guaranteed under strong pseudo-monotonicity. Numerical experiments are carried out for pseudo-monotone variational inequalities over polyhedral sets and fractional programming problems. △ Less

Submitted 31 July, 2020; v1 submitted 24 August, 2018; originally announced August 2018.

MSC Class: 47J20; 90C25; 90C30; 90C52

Journal ref: European Journal of Operational Research 287, 49-60, 2020

The Proximal Alternating Minimization Algorithm for two-block separable convex optimization problems with linear constraints

Authors: Sandy Bitterlich, Radu Ioan Bot, Ernö Robert Csetnek, Gert Wanka

Abstract: The Alternating Minimization Algorithm (AMA) has been proposed by Tseng to solve convex programming problems with two-block separable linear constraints and objectives, whereby (at least) one of the components of the latter is assumed to be strongly convex. The fact that one of the subproblems to be solved within the iteration process of AMA does not usually correspond to the calculation of a prox… ▽ More The Alternating Minimization Algorithm (AMA) has been proposed by Tseng to solve convex programming problems with two-block separable linear constraints and objectives, whereby (at least) one of the components of the latter is assumed to be strongly convex. The fact that one of the subproblems to be solved within the iteration process of AMA does not usually correspond to the calculation of a proximal operator through a closed formula, affects the implementability of the algorithm. In this paper we allow in each block of the objective a further smooth convex function and propose a proximal version of AMA, called Proximal AMA, which is achieved by equip** the algorithm with proximal terms induced by variable metrics. For suitable choices of the latter, the solving of the two subproblems in the iterative scheme can be reduced to the computation of proximal operators. We investigate the convergence of the proposed algorithm in a real Hilbert space setting and illustrate its numerical performances on two applications in image processing and machine learning. △ Less

Submitted 1 June, 2018; originally announced June 2018.

MSC Class: 47H05; 65K05; 90C25

A proximal minimization algorithm for structured nonconvex and nonsmooth problems

Authors: Radu Ioan Bot, Ernö Robert Csetnek, Dang-Khoa Nguyen

Abstract: We propose a proximal algorithm for minimizing objective functions consisting of three summands: the composition of a nonsmooth function with a linear operator, another nonsmooth function, each of the nonsmooth summands depending on an independent block variable, and a smooth function which couples the two block variables. The algorithm is a full splitting method, which means that the nonsmooth fu… ▽ More We propose a proximal algorithm for minimizing objective functions consisting of three summands: the composition of a nonsmooth function with a linear operator, another nonsmooth function, each of the nonsmooth summands depending on an independent block variable, and a smooth function which couples the two block variables. The algorithm is a full splitting method, which means that the nonsmooth functions are processed via their proximal operators, the smooth function via gradient steps, and the linear operator via matrix times vector multiplication. We provide sufficient conditions for the boundedness of the generated sequence and prove that any cluster point of the latter is a KKT point of the minimization problem. In the setting of the Kurdyka-Łojasiewicz property we show global convergence, and derive convergence rates for the iterates in terms of the Łojasiewicz exponent. △ Less

Submitted 31 July, 2020; v1 submitted 28 May, 2018; originally announced May 2018.

MSC Class: 65K10; 90C26; 90C30

Journal ref: SIAM Journal on Optimization 29(2), 1300-1328, 2019

The proximal alternating direction method of multipliers in the nonconvex setting: convergence analysis and rates

Authors: Radu Ioan Bot, Dang-Khoa Nguyen

Abstract: We propose two numerical algorithms in the fully nonconvex setting for the minimization of the sum of a smooth function and the composition of a nonsmooth function with a linear operator. The iterative schemes are formulated in the spirit of the proximal alternating direction method of multipliers and its linearized variant, respectively. The proximal terms are introduced via variable metrics, a f… ▽ More We propose two numerical algorithms in the fully nonconvex setting for the minimization of the sum of a smooth function and the composition of a nonsmooth function with a linear operator. The iterative schemes are formulated in the spirit of the proximal alternating direction method of multipliers and its linearized variant, respectively. The proximal terms are introduced via variable metrics, a fact which allows us to derive new proximal splitting algorithms for nonconvex structured optimization problems, as particular instances of the general schemes. Under mild conditions on the sequence of variable metrics and by assuming that a regularization of the associated augmented Lagrangian has the Kurdyka-Lojasiewicz property, we prove that the iterates converge to a KKT point of the objective function. By assuming that the augmented Lagrangian has the Lojasiewicz property, we also derive convergence rates for both the augmented Lagrangian and the iterates. △ Less

Submitted 31 July, 2020; v1 submitted 6 January, 2018; originally announced January 2018.

MSC Class: 47H05; 65K05; 90C26

Journal ref: Mathematics of Operations Research 45(2), 682-712, 2020

Approaching nonsmooth nonconvex minimization through second order proximal-gradient dynamical systems

Authors: Radu Ioan Bot, Ernö Robert Csetnek, Szilárd Csaba László

Abstract: We investigate the asymptotic properties of the trajectories generated by a second-order dynamical system of proximal-gradient type stated in connection with the minimization of the sum of a nonsmooth convex and a (possibly nonconvex) smooth function. The convergence of the generated trajectory to a critical point of the objective is ensured provided a regularization of the objective function sati… ▽ More We investigate the asymptotic properties of the trajectories generated by a second-order dynamical system of proximal-gradient type stated in connection with the minimization of the sum of a nonsmooth convex and a (possibly nonconvex) smooth function. The convergence of the generated trajectory to a critical point of the objective is ensured provided a regularization of the objective function satisfies the Kurdyka-Łojasiewicz property. We also provide convergence rates for the trajectory formulated in terms of the Łojasiewicz exponent. △ Less

Submitted 16 November, 2017; originally announced November 2017.

Comments: arXiv admin note: text overlap with arXiv:1507.01416, arXiv:1610.00911, arXiv:1703.01339

MSC Class: 34G25; 47J25; 47H05; 90C26; 90C30; 65K10

ADMM for monotone operators: convergence analysis and rates

Authors: Radu Ioan Bot, Ernö Robert Csetnek

Abstract: We propose in this paper a unifying scheme for several algorithms from the literature dedicated to the solving of monotone inclusion problems involving compositions with linear continuous operators in infinite dimensional Hilbert spaces. We show that a number of primal-dual algorithms for monotone inclusions and also the classical ADMM numerical scheme for convex optimization problems, along with… ▽ More We propose in this paper a unifying scheme for several algorithms from the literature dedicated to the solving of monotone inclusion problems involving compositions with linear continuous operators in infinite dimensional Hilbert spaces. We show that a number of primal-dual algorithms for monotone inclusions and also the classical ADMM numerical scheme for convex optimization problems, along with some of its variants, can be embedded in this unifying scheme. While in the first part of the paper convergence results for the iterates are reported, the second part is devoted to the derivation of convergence rates obtained by combining variable metric techniques with strategies based on suitable choice of dynamical step sizes. △ Less

Submitted 5 May, 2017; v1 submitted 4 May, 2017; originally announced May 2017.

MSC Class: 47H05; 65K05; 90C25

Newton-like dynamics associated to nonconvex optimization problems

Authors: Radu Ioan Bot, Ernö Robert Csetnek

Abstract: We consider the dynamical system \begin{equation*}\left\{ \begin{array}{ll} v(t)\in\partialφ(x(t))\\ λ\dot x(t) + \dot v(t) + v(t) + \nabla ψ(x(t))=0, \end{array}\right.\end{equation*} where $φ:\R^n\to\R\cup\{+\infty\}$ is a proper, convex and lower semicontinuous function, $ψ:\R^n\to\R$ is a (possibly nonconvex) smooth function and $λ>0$ is a parameter which controls the velocity. We show that th… ▽ More We consider the dynamical system \begin{equation*}\left\{ \begin{array}{ll} v(t)\in\partialφ(x(t))\\ λ\dot x(t) + \dot v(t) + v(t) + \nabla ψ(x(t))=0, \end{array}\right.\end{equation*} where $φ:\R^n\to\R\cup\{+\infty\}$ is a proper, convex and lower semicontinuous function, $ψ:\R^n\to\R$ is a (possibly nonconvex) smooth function and $λ>0$ is a parameter which controls the velocity. We show that the set of limit points of the trajectory $x$ is contained in the set of critical points of the objective function $φ+ψ$, which is here seen as the set of the zeros of its limiting subdifferential. If the objective function satisfies the Kurdyka-Łojasiewicz property, then we can prove convergence of the whole trajectory $x$ to a critical point. Furthermore, convergence rates for the orbits are obtained in terms of the Łojasiewicz exponent of the objective function, provided the latter satisfies the Łojasiewicz property. △ Less

Submitted 3 March, 2017; originally announced March 2017.

MSC Class: 34G25; 47J25; 47H05; 90C26; 90C30; 65K10

Second order dynamical systems with penalty terms associated to monotone inclusions

Authors: Radu Ioan Bot, Ernö Robert Csetnek, Szilárd Csaba László

Abstract: In this paper we investigate in a Hilbert space setting a second order dynamical system of the form $$\ddot{x}(t)+\g(t)\dot{x}(t)+x(t)-J_{λ(t) A}\big(x(t)-λ(t) D(x(t))-λ(t)β(t)B(x(t))\big)=0,$$ where $A:{\mathcal H}\toto{\mathcal H}$ is a maximal monotone operator, $J_{λ(t) A}:{\mathcal H}\To{\mathcal H}$ is the resolvent operator of $λ(t)A$ and $D,B: {\mathcal H}\rightarrow{\mathcal H}$ are cocoe… ▽ More In this paper we investigate in a Hilbert space setting a second order dynamical system of the form $$\ddot{x}(t)+\g(t)\dot{x}(t)+x(t)-J_{λ(t) A}\big(x(t)-λ(t) D(x(t))-λ(t)β(t)B(x(t))\big)=0,$$ where $A:{\mathcal H}\toto{\mathcal H}$ is a maximal monotone operator, $J_{λ(t) A}:{\mathcal H}\To{\mathcal H}$ is the resolvent operator of $λ(t)A$ and $D,B: {\mathcal H}\rightarrow{\mathcal H}$ are cocoercive operators, and $λ,β:[0,+\infty)\rightarrow (0,+\infty)$, and $γ:[0,+\infty)\rightarrow (0,+\infty)$ are step size, penalization and, respectively, dam** functions, all depending on time. We show the existence and uniqueness of strong global solutions in the framework of the Cauchy-Lipschitz-Picard Theorem and prove ergodic asymptotic convergence for the generated trajectories to a zero of the operator $A+D+{N}_C,$ where $C=\zer B$ and $N_C$ denotes the normal cone operator of $C$. To this end we use Lyapunov analysis combined with the celebrated Opial Lemma in its ergodic continuous version. Furthermore, we show strong convergence for trajectories to the unique zero of $A+D+{N}_C$, provided that $A$ is a strongly monotone operator. △ Less

Submitted 18 January, 2017; originally announced January 2017.

MSC Class: 34G25; 47J25; 47H05; 90C25

Fixing and extending some recent results on the ADMM algorithm

Authors: Sebastian Banert, Radu Ioan Bot, Ernö Robert Csetnek

Abstract: We investigate the techniques and ideas used in the convergence analysis of two proximal ADMM algorithms for solving convex optimization problems involving compositions with linear operators. Besides this, we formulate a variant of the ADMM algorithm that is able to handle convex optimization problems involving an additional smooth function in its objective, and which is evaluated through its grad… ▽ More We investigate the techniques and ideas used in the convergence analysis of two proximal ADMM algorithms for solving convex optimization problems involving compositions with linear operators. Besides this, we formulate a variant of the ADMM algorithm that is able to handle convex optimization problems involving an additional smooth function in its objective, and which is evaluated through its gradient. Moreover, in each iteration we allow the use of variable metrics, while the investigations are carried out in the setting of infinite dimensional Hilbert spaces. This algorithmic scheme is investigated from the point of view of its convergence properties. △ Less

Submitted 19 December, 2019; v1 submitted 15 December, 2016; originally announced December 2016.

Comments: Updates in Section 2 concerning the derivation of the convergence rates + a unifying convergence theorem for the sequence of iterates

A general double-proximal gradient algorithm for d.c. programming

Authors: Sebastian Banert, Radu Ioan Bot

Abstract: The possibilities of exploiting the special structure of d.c. programs, which consist of optimizing the difference of convex functions, are currently more or less limited to variants of the DCA proposed by Pham Dinh Tao and Le Thi Hoai An in 1997. These assume that either the convex or the concave part, or both, are evaluated by one of their subgradients. In this paper we propose an algorithm wh… ▽ More The possibilities of exploiting the special structure of d.c. programs, which consist of optimizing the difference of convex functions, are currently more or less limited to variants of the DCA proposed by Pham Dinh Tao and Le Thi Hoai An in 1997. These assume that either the convex or the concave part, or both, are evaluated by one of their subgradients. In this paper we propose an algorithm which allows the evaluation of both the concave and the convex part by their proximal points. Additionally, we allow a smooth part, which is evaluated via its gradient. In the spirit of primal-dual splitting algorithms, the concave part might be the composition of a concave function with a linear operator, which are, however, evaluated separately. For this algorithm we show that every cluster point is a solution of the optimization problem. Furthermore, we show the connection to the Toland dual problem and prove a descent property for the objective function values of a primal-dual formulation of the problem. Convergence of the iterates is shown if this objective function satisfies the Kurdyka--Łojasiewicz property. In the last part, we apply the algorithm to an image processing model. △ Less

Submitted 20 October, 2016; originally announced October 2016.

MSC Class: 90C26; 90C30; 65K05

Approaching nonsmooth nonconvex optimization problems through first order dynamical systems with hidden acceleration and Hessian driven dam** terms

Authors: Radu Ioan Bot, Ernö Robert Csetnek

Abstract: In this paper we carry out an asymptotic analysis of the proximal-gradient dynamical system \begin{equation*}\left\{ \begin{array}{ll} \dot x(t) +x(t) = \prox_{γf}\big[x(t)-γ\nablaΦ(x(t))-ax(t)-by(t)\big],\\ \dot y(t)+ax(t)+by(t)=0 \end{array}\right.\end{equation*} where $f$ is a proper, convex and lower semicontinuous function, $Φ$ a possibly nonconvex smooth function and $γ, a$ and $b$ are posit… ▽ More In this paper we carry out an asymptotic analysis of the proximal-gradient dynamical system \begin{equation*}\left\{ \begin{array}{ll} \dot x(t) +x(t) = \prox_{γf}\big[x(t)-γ\nablaΦ(x(t))-ax(t)-by(t)\big],\\ \dot y(t)+ax(t)+by(t)=0 \end{array}\right.\end{equation*} where $f$ is a proper, convex and lower semicontinuous function, $Φ$ a possibly nonconvex smooth function and $γ, a$ and $b$ are positive real numbers. We show that the generated trajectories approach the set of critical points of $f+Φ$, here understood as zeros of its limiting subdifferential, under the premise that a regularization of this sum function satisfies the Kurdyka-Łojasiewicz property. We also establish convergence rates for the trajectories, formulated in terms of the Łojasiewicz exponent of the considered regularization function. △ Less

Submitted 4 October, 2016; originally announced October 2016.

Comments: arXiv admin note: substantial text overlap with arXiv:1507.01416

MSC Class: 34G25; 47J25; 47H05; 90C26; 90C30; 65K10

Inducing strong convergence into the asymptotic behaviour of proximal splitting algorithms in Hilbert spaces

Authors: Radu Ioan Bot, Ernö Robert Csetnek, Dennis Meier

Abstract: Proximal splitting algorithms for monotone inclusions (and convex optimization problems) in Hilbert spaces share the common feature to guarantee for the generated sequences in general weak convergence to a solution. In order to achieve strong convergence, one usually needs to impose more restrictive properties for the involved operators, like strong monotonicity (respectively, strong convexity for… ▽ More Proximal splitting algorithms for monotone inclusions (and convex optimization problems) in Hilbert spaces share the common feature to guarantee for the generated sequences in general weak convergence to a solution. In order to achieve strong convergence, one usually needs to impose more restrictive properties for the involved operators, like strong monotonicity (respectively, strong convexity for optimization problems). In this paper, we propose a modified Krasnosel'skiĭ--Mann algorithm in connection with the determination of a fixed point of a nonexpansive map** and show strong convergence of the iteratively generated sequence to the minimal norm solution of the problem. Relying on this, we derive a forward-backward and a Douglas-Rachford algorithm, both endowed with Tikhonov regularization terms, which generate iterates that strongly converge to the minimal norm solution of the set of zeros of the sum of two maximally monotone operators. Furthermore, we formulate strong convergent primal-dual algorithms of forward-backward and Douglas-Rachford-type for highly structured monotone inclusion problems involving parallel-sums and compositions with linear operators. The resulting iterative schemes are particularized to the solving of convex minimization problems. △ Less

Submitted 18 November, 2017; v1 submitted 6 September, 2016; originally announced September 2016.

MSC Class: 47J25; 47H09; 47H05; 90C25

A second order dynamical system with Hessian-driven dam** and penalty term associated to variational inequalities

Authors: Radu Ioan Bot, Ernö Robert Csetnek

Abstract: We consider the minimization of a convex objective function subject to the set of minima of another convex function, under the assumption that both functions are twice continuously differentiable. We approach this optimization problem from a continuous perspective by means of a second order dynamical system with Hessian-driven dam** and a penalty term corresponding to the constrained function. B… ▽ More We consider the minimization of a convex objective function subject to the set of minima of another convex function, under the assumption that both functions are twice continuously differentiable. We approach this optimization problem from a continuous perspective by means of a second order dynamical system with Hessian-driven dam** and a penalty term corresponding to the constrained function. By constructing appropriate energy functionals, we prove weak convergence of the trajectories generated by this differential equation to a minimizer of the optimization problem as well as convergence for the objective function values along the trajectories. The performed investigations rely on Lyapunov analysis in combination with the continuous version of the Opial Lemma. In case the objective function is strongly convex, we can even show strong convergence of the trajectories. △ Less

Submitted 14 August, 2016; originally announced August 2016.

Comments: arXiv admin note: text overlap with arXiv:1512.04702

MSC Class: 34G25; 47J25; 47H05; 90C25

Conditional stability versus ill-posedness for operator equations with monotone operators in Hilbert space

Authors: Radu Ioan Bot, Bernd Hofmann

Abstract: In the literature on singular perturbation (Lavrentiev regularization) for the stable approximate solution of operator equations with monotone operators in the Hilbert space the phenomena of conditional stability and local well-posedness and ill-posedness are rarely investigated. Our goal is to present some studies which try to bridge this gap. So we discuss the impact of conditional stability on… ▽ More In the literature on singular perturbation (Lavrentiev regularization) for the stable approximate solution of operator equations with monotone operators in the Hilbert space the phenomena of conditional stability and local well-posedness and ill-posedness are rarely investigated. Our goal is to present some studies which try to bridge this gap. So we discuss the impact of conditional stability on error estimates and convergence rates for the Lavrentiev regularization and distinguish for linear problems well-posedness and ill-posedness in a specific manner motivated by a saturation result. The role of the regularization error in the noise-free case, called bias, is a crucial point in the paper for nonlinear and linear problems. In particular, for linear operator equations general convergence rates, including logarithmic rates, are derived by means of the method of approximate source conditions. This allows us to extend well-known convergence rates results for the Lavrentiev regularization that were based on general source conditions to the case of non-selfadjoint linear monotone forward operators for which general source conditions fail. Examples presenting the self-adjoint multiplication operator as well as the non-selfadjoint fractional integral operator and Cesàro operator illustrate the theoretical results. Extensions to the nonlinear case under specific conditions on the nonlinearity structure complete the paper. △ Less

Submitted 19 July, 2016; originally announced July 2016.

Comments: 24 pages

MSC Class: 47A52; 65F22; 47H05; 65J22; 65J15

Levenberg-Marquardt dynamics associated to variational inequalities

Authors: Radu Ioan Bot, Ernö Robert Csetnek

Abstract: In connection with the optimization problem $$\inf_{x\in argmin Ψ}\{Φ(x)+Θ(x)\},$$ where $Φ$ is a proper, convex and lower semicontinuous function and $Θ$ and $Ψ$ are convex and smooth functions defined on a real Hilbert space, we investigate the asymptotic behavior of the trajectories of the nonautonomous Levenberg-Marquardt dynamical system \begin{equation*}\left\{ \begin{array}{ll} v(t)\in\part… ▽ More In connection with the optimization problem $$\inf_{x\in argmin Ψ}\{Φ(x)+Θ(x)\},$$ where $Φ$ is a proper, convex and lower semicontinuous function and $Θ$ and $Ψ$ are convex and smooth functions defined on a real Hilbert space, we investigate the asymptotic behavior of the trajectories of the nonautonomous Levenberg-Marquardt dynamical system \begin{equation*}\left\{ \begin{array}{ll} v(t)\in\partialΦ(x(t))\\ λ(t)\dot x(t) + \dot v(t) + v(t) + \nabla Θ(x(t))+β(t)\nabla Ψ(x(t))=0, \end{array}\right.\end{equation*} where $λ$ and $β$ are functions of time controlling the velocity and the penalty term, respectively. We show weak convergence of the generated trajectory to an optimal solution as well as convergence of the objective function values along the trajectories, provided $λ$ is monotonically decreasing, $β$ satisfies a growth condition and a relation expressed via the Fenchel conjugate of $Ψ$ is fulfilled. When the objective function is assumed to be strongly convex, we can even show strong convergence of the trajectories. △ Less

Submitted 14 March, 2016; originally announced March 2016.

Comments: arXiv admin note: text overlap with arXiv:1512.04702

MSC Class: 34G25; 47J25; 47H05; 90C25

Proximal-gradient algorithms for fractional programming

Authors: Radu Ioan Bot, Ernö Robert Csetnek

Abstract: In this paper we propose two proximal gradient algorithms for fractional programming problems in real Hilbert spaces, where the numerator is a proper, convex and lower semicontinuous function and the denominator is a smooth function, either concave or convex. In the iterative schemes, we perform a proximal step with respect to the nonsmooth numerator and a gradient step with respect to the smooth… ▽ More In this paper we propose two proximal gradient algorithms for fractional programming problems in real Hilbert spaces, where the numerator is a proper, convex and lower semicontinuous function and the denominator is a smooth function, either concave or convex. In the iterative schemes, we perform a proximal step with respect to the nonsmooth numerator and a gradient step with respect to the smooth denominator. The algorithm in case of a concave denominator has the particularity that it generates sequences which approach both the (global) optimal solutions set and the optimal objective value of the underlying fractional programming problem. In case of a convex denominator the numerical scheme approaches the set of critical points of the objective function, provided the latter satisfies the Kurdyka-Łojasiewicz property. △ Less

Submitted 29 January, 2016; originally announced January 2016.

MSC Class: 65K05; 90C25; 90C32

Second order dynamical systems associated to variational inequalities

Authors: Radu Ioan Bot, Ernö Robert Csetnek

Abstract: We investigate the asymptotic convergence of the trajectories generated by the second order dynamical system $\ddot x(t) + γ\dot x(t) + \nabla φ(x(t))+β(t)\nabla ψ(x(t))=0$, where $φ,ψ:{\cal H}\rightarrow \R$ are convex and smooth functions defined on a real Hilbert space ${\cal H}$, $γ>0$ and $β$ is a function of time which controls the penalty term. We show weak convergence of the trajectories t… ▽ More We investigate the asymptotic convergence of the trajectories generated by the second order dynamical system $\ddot x(t) + γ\dot x(t) + \nabla φ(x(t))+β(t)\nabla ψ(x(t))=0$, where $φ,ψ:{\cal H}\rightarrow \R$ are convex and smooth functions defined on a real Hilbert space ${\cal H}$, $γ>0$ and $β$ is a function of time which controls the penalty term. We show weak convergence of the trajectories to a minimizer of the function $φ$ over the (nonempty) set of minima of $ψ$ as well as convergence for the objective function values along the trajectories, provided a condition expressed via the Fenchel conjugate of $ψ$ is fulfilled. When the function $φ$ is assumed to be strongly convex, we can even show strong convergence of the trajectories. The results can be seen as the second order counterparts of the ones given by Attouch and Czarnecki (Journal of Differential Equations 248(6), 1315--1344, 2010) for first order dynamical systems associated to constrained variational inequalities. At the same time we give a positive answer to an open problem posed in \cite{att-cza-16} by the same authors. △ Less

Submitted 17 February, 2016; v1 submitted 15 December, 2015; originally announced December 2015.

MSC Class: 34G25; 47J25; 47H05; 90C25

Penalty schemes with inertial effects for monotone inclusion problems

Authors: Radu Ioan Bot, Ernö Robert Csetnek

Abstract: We introduce a penalty term-based splitting algorithm with inertial effects designed for solving monotone inclusion problems involving the sum of maximally monotone operators and the convex normal cone to the (nonempty) set of zeros of a monotone and Lipschitz continuous operator. We show weak ergodic convergence of the generated sequence of iterates to a solution of the monotone inclusion problem… ▽ More We introduce a penalty term-based splitting algorithm with inertial effects designed for solving monotone inclusion problems involving the sum of maximally monotone operators and the convex normal cone to the (nonempty) set of zeros of a monotone and Lipschitz continuous operator. We show weak ergodic convergence of the generated sequence of iterates to a solution of the monotone inclusion problem, provided a condition expressed via the Fitzpatrick function of the operator describing the underlying set of the normal cone is verified. Under strong monotonicity assumptions we can even show strong nonergodic convergence of the iterates. This approach constitutes the starting point for investigating from a similar perspective monotone inclusion problems involving linear compositions of parallel-sum operators and, further, for the minimization of a complexly structured convex objective function subject to the set of minima of another convex and differentiable function. △ Less

Submitted 14 December, 2015; originally announced December 2015.

Comments: arXiv admin note: text overlap with arXiv:1306.0352

MSC Class: 47H05; 65K05; 90C25

A forward-backward dynamical approach to the minimization of the sum of a nonsmooth convex with a smooth nonconvex function

Authors: Radu Ioan Bot, Ernö Robert Csetnek

Abstract: We address the minimization of the sum of a proper, convex and lower semicontinuous with a (possibly nonconvex) smooth function from the perspective of an implicit dynamical system of forward-backward type. The latter is formulated by means of the gradient of the smooth function and of the proximal point operator of the nonsmooth one. The trajectory generated by the dynamical system is proved to a… ▽ More We address the minimization of the sum of a proper, convex and lower semicontinuous with a (possibly nonconvex) smooth function from the perspective of an implicit dynamical system of forward-backward type. The latter is formulated by means of the gradient of the smooth function and of the proximal point operator of the nonsmooth one. The trajectory generated by the dynamical system is proved to asymptotically converge to a critical point of the objective, provided a regularization of the latter satisfies the Kurdyka-Łojasiewicz property. Convergence rates for the trajectory in terms of the Łojasiewicz exponent of the regularized objective function are also provided. △ Less

Submitted 6 July, 2015; originally announced July 2015.

MSC Class: 34G25; 47J25; 47H05; 90C26; 90C30; 65K10

Convergence rates for forward-backward dynamical systems associated with strongly monotone inclusions

Authors: Radu Ioan Bot, Ernö Robert Csetnek

Abstract: We investigate the convergence rates of the trajectories generated by implicit first and second order dynamical systems associated to the determination of the zeros of the sum of a maximally monotone operator and a monotone and Lipschitz continuous one in a real Hilbert space. We show that these trajectories strongly converge with exponential rate to a zero of the sum, provided the latter is stron… ▽ More We investigate the convergence rates of the trajectories generated by implicit first and second order dynamical systems associated to the determination of the zeros of the sum of a maximally monotone operator and a monotone and Lipschitz continuous one in a real Hilbert space. We show that these trajectories strongly converge with exponential rate to a zero of the sum, provided the latter is strongly monotone. We derive from here convergence rates for the trajectories generated by dynamical systems associated to the minimization of the sum of a proper, convex and lower semicontinuous function with a smooth convex one provided the objective function fulfills a strong convexity assumption. In the particular case of minimizing a smooth and strongly convex function, we prove that its values converge along the trajectory to its minimum value with exponential rate, too. △ Less

Submitted 8 April, 2015; originally announced April 2015.

Comments: arXiv admin note: text overlap with arXiv:1503.04652

MSC Class: 34G25; 47J25; 47H05; 90C25

A forward-backward-forward differential equation and its asymptotic properties

Authors: Sebastian Banert, Radu Ioan Bot

Abstract: In this paper, we approach the problem of finding the zeros of the sum of a maximally monotone operator and a monotone and Lipschitz continuous one in a real Hilbert space via an implicit forward-backward-forward dynamical system with nonconstant relaxation parameters and stepsizes of the resolvents. Besides proving existence and uniqueness of strong global solutions for the differential equation… ▽ More In this paper, we approach the problem of finding the zeros of the sum of a maximally monotone operator and a monotone and Lipschitz continuous one in a real Hilbert space via an implicit forward-backward-forward dynamical system with nonconstant relaxation parameters and stepsizes of the resolvents. Besides proving existence and uniqueness of strong global solutions for the differential equation under consideration, we show weak convergence of the generated trajectories and, under strong monotonicity assumptions, strong convergence with exponential rate. In the particular setting of minimizing the sum of a proper, convex and lower semicontinuous function with a smooth convex one, we provide a rate for the convergence of the objective function along the ergodic trajectory to its minimum value. △ Less

Submitted 22 April, 2015; v1 submitted 26 March, 2015; originally announced March 2015.

MSC Class: 34G25; 47H05; 90C25