-
A Stochastic Variance Reduction Algorithm with Bregman Distances for Structured Composite Problems
Authors:
Nguyen Van Dung,
Băng Công Vũ
Abstract:
We develop a novel stochastic primal dual splitting method with Bregman distances for solving a structured composite problems involving infimal convolutions in non-Euclidean spaces. The sublinear convergence in expectation of the primal-dual gap is proved under mild conditions on stepsize for the general case. The linear convergence rate is obtained under additional condition like the strong conve…
▽ More
We develop a novel stochastic primal dual splitting method with Bregman distances for solving a structured composite problems involving infimal convolutions in non-Euclidean spaces. The sublinear convergence in expectation of the primal-dual gap is proved under mild conditions on stepsize for the general case. The linear convergence rate is obtained under additional condition like the strong convexity relative to Bregman functions.
△ Less
Submitted 15 March, 2021;
originally announced March 2021.
-
Convergence analysis of the stochastic reflected forward-backward splitting algorithm
Authors:
Nguyen Van Dung,
Bang Cong Vu
Abstract:
We propose and analyze the convergence of a novel stochastic algorithm for solving monotone inclusions that are the sum of a maximal monotone operator and a monotone, Lipschitzian operator. The propose algorithm requires only unbiased estimations of the Lipschitzian operator. We obtain the rate $\mathcal{O}(log(n)/n)$ in expectation for the strongly monotone case, as well as almost sure convergenc…
▽ More
We propose and analyze the convergence of a novel stochastic algorithm for solving monotone inclusions that are the sum of a maximal monotone operator and a monotone, Lipschitzian operator. The propose algorithm requires only unbiased estimations of the Lipschitzian operator. We obtain the rate $\mathcal{O}(log(n)/n)$ in expectation for the strongly monotone case, as well as almost sure convergence for the general case. Furthermore, in the context of application to convex-concave saddle point problems, we derive the rate of the primal-dual gap. In particular, we also obtain $\mathcal{O}(1/n)$ rate convergence of the primal-dual gap in the deterministic setting.
△ Less
Submitted 17 February, 2021;
originally announced February 2021.
-
Finding the forward-Douglas-Rachford-forward method
Authors:
Ernest K. Ryu,
Bang Cong Vu
Abstract:
We consider the monotone inclusion problem with a sum of 3 operators, in which 2 are monotone and 1 is monotone-Lipschitz. The classical Douglas--Rachford and Forward-backward-forward methods respectively solve the monotone inclusion problem with a sum of 2 monotone operators and a sum of 1 monotone and 1 monotone-Lipschitz operators. We first present a method that naturally combines Douglas--Rach…
▽ More
We consider the monotone inclusion problem with a sum of 3 operators, in which 2 are monotone and 1 is monotone-Lipschitz. The classical Douglas--Rachford and Forward-backward-forward methods respectively solve the monotone inclusion problem with a sum of 2 monotone operators and a sum of 1 monotone and 1 monotone-Lipschitz operators. We first present a method that naturally combines Douglas--Rachford and Forward-backward-forward and show that it solves the 3 operator problem under further assumptions, but fails in general. We then present a method that naturally combines Douglas--Rachford and forward-reflected-backward, a recently proposed alternative to Forward-backward-forward by Malitsky and Tam [arXiv:1808.04162, 2018]. We show that this second method solves the 3 operator problem generally, without further assumptions.
△ Less
Submitted 16 October, 2019; v1 submitted 20 September, 2019;
originally announced September 2019.
-
A reflected forward-backward splitting method for monotone inclusions involving Lipschitzian operators
Authors:
Volkan Cevher,
Bang Cong Vu
Abstract:
The proximal extrapolated gradient method \cite{Malitsky18a} is an extension of the projected reflected gradient method \cite{Malitsky15}. Both methods were proposed for solving the classic variational inequalities. In this paper, we investigate the projected reflected gradient method, in the general setting, for solving monotone inclusions involving Lipschitzian operators. As a result, we obtain…
▽ More
The proximal extrapolated gradient method \cite{Malitsky18a} is an extension of the projected reflected gradient method \cite{Malitsky15}. Both methods were proposed for solving the classic variational inequalities. In this paper, we investigate the projected reflected gradient method, in the general setting, for solving monotone inclusions involving Lipschitzian operators. As a result, we obtain a simple method for finding a zero point of the sum of two monotone operators where one of them is Lipschizian. We also show that one can improve the range of the stepsize of this method for the case when the Lipschitzian operator is restricted to be cocoercive. A nice combination of this method and the forward-backward splitting was proposed. As a result, we obtain a new splitting method for finding a zero point of the sum of three operators ( maximally monotone + monotone Lipschitzian + cocoercive). Application to composite monotone inclusions are demonstrated.
△ Less
Submitted 16 August, 2019;
originally announced August 2019.
-
Inertial Three-Operator Splitting Method and Applications
Authors:
Volkan Cevher,
Bang Cong Vu,
Alp Yurtsever
Abstract:
We introduce an inertial variant of the forward-Douglas-Rachford splitting and analyze its convergence. We specify an instance of the proposed method to the three-composite convex minimization template. We provide practical guidance on the selection of the inertial parameter based on the adaptive starting idea. Finally, we illustrate the practical performance of our method in various machine learn…
▽ More
We introduce an inertial variant of the forward-Douglas-Rachford splitting and analyze its convergence. We specify an instance of the proposed method to the three-composite convex minimization template. We provide practical guidance on the selection of the inertial parameter based on the adaptive starting idea. Finally, we illustrate the practical performance of our method in various machine learning applications.
△ Less
Submitted 29 April, 2019;
originally announced April 2019.
-
On the linear convergence of the stochastic gradient method with constant step-size
Authors:
Volkan Cevher,
Bang Cong Vu
Abstract:
The strong growth condition (SGC) is known to be a sufficient condition for linear convergence of the stochastic gradient method using a constant step-size $γ$ (SGM-CS). In this paper, we provide a necessary condition, for the linear convergence of SGM-CS, that is weaker than SGC. Moreover, when this necessary is violated up to a additive perturbation $σ$, we show that both the projected stochasti…
▽ More
The strong growth condition (SGC) is known to be a sufficient condition for linear convergence of the stochastic gradient method using a constant step-size $γ$ (SGM-CS). In this paper, we provide a necessary condition, for the linear convergence of SGM-CS, that is weaker than SGC. Moreover, when this necessary is violated up to a additive perturbation $σ$, we show that both the projected stochastic gradient method using a constant step-size (PSGM-CS) and the proximal stochastic gradient method exhibit linear convergence to a noise dominated region, whose distance to the optimal solution is proportional to $γσ$.
△ Less
Submitted 17 June, 2018; v1 submitted 5 December, 2017;
originally announced December 2017.
-
Stochastic Three-Composite Convex Minimization
Authors:
Alp Yurtsever,
Bang Cong Vu,
Volkan Cevher
Abstract:
We propose a stochastic optimization method for the minimization of the sum of three convex functions, one of which has Lipschitz continuous gradient as well as restricted strong convexity. Our approach is most suitable in the setting where it is computationally advantageous to process smooth term in the decomposition with its stochastic gradient estimate and the other two functions separately wit…
▽ More
We propose a stochastic optimization method for the minimization of the sum of three convex functions, one of which has Lipschitz continuous gradient as well as restricted strong convexity. Our approach is most suitable in the setting where it is computationally advantageous to process smooth term in the decomposition with its stochastic gradient estimate and the other two functions separately with their proximal operators, such as doubly regularized empirical risk minimization problems. We prove the convergence characterization of the proposed algorithm in expectation under the standard assumptions for the stochastic gradient estimate of the smooth term. Our method operates in the primal space and can be considered as a stochastic extension of the three-operator splitting method. Numerical evidence supports the effectiveness of our method in real-world problems.
△ Less
Submitted 31 January, 2017;
originally announced January 2017.
-
A first-order stochastic primal-dual algorithm with correction step
Authors:
Lorenzo Rosasco,
Silvia Villa,
Bang Cong Vu
Abstract:
We investigate the convergence properties of a stochastic primal-dual splitting algorithm for solving structured monotone inclusions involving the sum of a cocoercive operator and a composite monotone operator. The proposed method is the stochastic extension to monotone inclusions of a proximal method studied in {\em Y. Drori, S. Sabach, and M. Teboulle, A simple algorithm for a class of nonsmooth…
▽ More
We investigate the convergence properties of a stochastic primal-dual splitting algorithm for solving structured monotone inclusions involving the sum of a cocoercive operator and a composite monotone operator. The proposed method is the stochastic extension to monotone inclusions of a proximal method studied in {\em Y. Drori, S. Sabach, and M. Teboulle, A simple algorithm for a class of nonsmooth convex-concave saddle-point problems, 2015} and {\em I. Loris and C. Verhoeven, On a generalization of the iterative soft-thresholding algorithm for the case of non-separable penalty, 2011} for saddle point problems. It consists in a forward step determined by the stochastic evaluation of the cocoercive operator, a backward step in the dual variables involving the resolvent of the monotone operator, and an additional forward step using the stochastic evaluation of the cocoercive introduced in the first step. We prove weak almost sure convergence of the iterates by showing that the primal-dual sequence generated by the method is stochastic quasi Fejér-monotone with respect to the set of zeros of the considered primal and dual inclusions. Additional results on ergodic convergence in expectation are considered for the special case of saddle point models.
△ Less
Submitted 25 February, 2016;
originally announced February 2016.
-
Stochastic inertial primal-dual algorithms
Authors:
Lorenzo Rosasco,
Silvia Villa,
Bang Cong Vu
Abstract:
We propose and study a novel stochastic inertial primal-dual approach to solve composite optimization problems. These latter problems arise naturally when learning with penalized regularization schemes. Our analysis provide convergence results in a general setting, that allows to analyze in a unified framework a variety of special cases of interest. Key in our analysis is considering the framework…
▽ More
We propose and study a novel stochastic inertial primal-dual approach to solve composite optimization problems. These latter problems arise naturally when learning with penalized regularization schemes. Our analysis provide convergence results in a general setting, that allows to analyze in a unified framework a variety of special cases of interest. Key in our analysis is considering the framework of splitting algorithm for solving a monotone inclusions in suitable product spaces and for a specific choice of preconditioning operators.
△ Less
Submitted 3 July, 2015;
originally announced July 2015.
-
A stochastic inertial forward-backward splitting algorithm for multivariate monotone inclusions
Authors:
Lorenzo Rosasco,
Silvia Villa,
Bang Cong Vu
Abstract:
We propose an inertial forward-backward splitting algorithm to compute the zero of a sum of two monotone operators allowing for stochastic errors in the computation of the operators. More precisely, we establish almost sure convergence in real Hilbert spaces of the sequence of iterates to an optimal solution. Then, based on this analysis, we introduce two new classes of stochastic inertial primal-…
▽ More
We propose an inertial forward-backward splitting algorithm to compute the zero of a sum of two monotone operators allowing for stochastic errors in the computation of the operators. More precisely, we establish almost sure convergence in real Hilbert spaces of the sequence of iterates to an optimal solution. Then, based on this analysis, we introduce two new classes of stochastic inertial primal-dual splitting methods for solving structured systems of composite monotone inclusions and prove their convergence. Our results extend to the stochastic and inertial setting various types of structured monotone inclusion problems and corresponding algorithmic solutions. Application to minimization problems is discussed.
△ Less
Submitted 3 July, 2015;
originally announced July 2015.
-
A new splitting method for solving composite monotone inclusions involving parallel-sum operators
Authors:
Quoc Tran-Dinh,
Bang Cong Vu
Abstract:
We propose a new primal-dual splitting method for solving composite inclusions involving Lipschitzian, and parallel-sum-type monotone operators. Our approach extends the framework in \cite{Siopt4} to a more general class of monotone inclusions in a nontrivial fashion. The main idea is to represent the solution set of both the primal and dual problems using their associated Kuhn-Tucker set, and the…
▽ More
We propose a new primal-dual splitting method for solving composite inclusions involving Lipschitzian, and parallel-sum-type monotone operators. Our approach extends the framework in \cite{Siopt4} to a more general class of monotone inclusions in a nontrivial fashion. The main idea is to represent the solution set of both the primal and dual problems using their associated Kuhn-Tucker set, and then develop a projected method to successively approximate a feasible point of the Kuhn-Tucker set. We propose a splitting algorithm based on the resolvent of each maximally monotone operator to construct a primal-dual sequence that weakly converges to a solution of the original problem. The key feature of our method is that it only employes the resolvent of each monotone operator separately, which is different from existing methods in the literature. As a byproduct, our algorithm can be specialized to solve composite convex minimization problems that uses the proximal-operator of each objective component independently, and is equipped with a weakly convergence guarantee.
△ Less
Submitted 25 July, 2015; v1 submitted 29 May, 2015;
originally announced May 2015.
-
Almost sure convergence of the forward-backward-forward splitting algorithm
Authors:
Bang Cong Vũ
Abstract:
In this paper, we propose a stochastic forward-backward-forward splitting algorithm and prove its almost sure weak convergence in real separable Hilbert spaces. Applications to composite monotone inclusion and minimization problems are demonstrated.
In this paper, we propose a stochastic forward-backward-forward splitting algorithm and prove its almost sure weak convergence in real separable Hilbert spaces. Applications to composite monotone inclusion and minimization problems are demonstrated.
△ Less
Submitted 19 May, 2015;
originally announced May 2015.
-
A forward-backward view of some primal-dual optimization methods in image recovery
Authors:
Patrick L. Combettes,
Laurent Condat,
Jean-Christophe Pesquet,
Bang Cong Vu
Abstract:
A wide array of image recovery problems can be abstracted into the problem of minimizing a sum of composite convex functions in a Hilbert space. To solve such problems, primal-dual proximal approaches have been developed which provide efficient solutions to large-scale optimization problems. The objective of this paper is to show that a number of existing algorithms can be derived from a general f…
▽ More
A wide array of image recovery problems can be abstracted into the problem of minimizing a sum of composite convex functions in a Hilbert space. To solve such problems, primal-dual proximal approaches have been developed which provide efficient solutions to large-scale optimization problems. The objective of this paper is to show that a number of existing algorithms can be derived from a general form of the forward-backward algorithm applied in a suitable product space. Our approach also allows us to develop useful extensions of existing algorithms by introducing a variable metric. An illustration to image restoration is provided.
△ Less
Submitted 20 June, 2014;
originally announced June 2014.
-
A Stochastic forward-backward splitting method for solving monotone inclusions in Hilbert spaces
Authors:
Lorenzo Rosasco,
Silvia Villa,
Bang Công Vũ
Abstract:
We propose and analyze the convergence of a novel stochastic forward-backward splitting algorithm for solving monotone inclusions given by the sum of a maximal monotone operator and a single-valued maximal monotone cocoercive operator. This latter framework has a number of interesting special cases, including variational inequalities and convex minimization problems, while stochastic approaches ar…
▽ More
We propose and analyze the convergence of a novel stochastic forward-backward splitting algorithm for solving monotone inclusions given by the sum of a maximal monotone operator and a single-valued maximal monotone cocoercive operator. This latter framework has a number of interesting special cases, including variational inequalities and convex minimization problems, while stochastic approaches are practically relevant to account for perturbations in the data. The algorithm we propose is a stochastic extension of the classical deterministic forward-backward method, and is obtained considering the composition of the resolvent of the maximal monotone operator with a forward step based on a stochastic estimate of the single-valued operator. Our study provides a non-asymptotic error analysis in expectation for the strongly monotone case, as well as almost sure convergence under weaker assumptions. The approach we consider allows to avoid averaging, a feature critical when considering methods based on sparsity, and, for minimization problems, it allows to obtain convergence rates matching those obtained by stochastic extensions of so called accelerated methods. Stochastic quasi Fejer's sequences are a key technical tool to prove almost sure convergence.
△ Less
Submitted 20 February, 2015; v1 submitted 31 March, 2014;
originally announced March 2014.
-
Convergence of Stochastic Proximal Gradient Algorithm
Authors:
Lorenzo Rosasco,
Silvia Villa,
Bang Công Vũ
Abstract:
We prove novel convergence results for a stochastic proximal gradient algorithm suitable for solving a large class of convex optimization problems, where a convex objective function is given by the sum of a smooth and a possibly non-smooth component. We consider the iterates convergence and derive $O(1/n)$ non asymptotic bounds in expectation in the strongly convex case, as well as almost sure con…
▽ More
We prove novel convergence results for a stochastic proximal gradient algorithm suitable for solving a large class of convex optimization problems, where a convex objective function is given by the sum of a smooth and a possibly non-smooth component. We consider the iterates convergence and derive $O(1/n)$ non asymptotic bounds in expectation in the strongly convex case, as well as almost sure convergence results under weaker assumptions. Our approach allows to avoid averaging and weaken boundedness assumptions which are often considered in theoretical studies and might not be satisfied in practice.
△ Less
Submitted 16 September, 2014; v1 submitted 20 March, 2014;
originally announced March 2014.
-
A splitting algorithm for system of composite monotone inclusions
Authors:
Dinh Dung,
Bang Cong Vu
Abstract:
We propose a splitting algorithm for solving a system of composite monotone inclusions formulated in the form of the extended set of solutions in real Hilbert spaces. The resluting algorithm is a an extension of the algorithm in [4]. The weak convergence of the algorithm proposed is proved. Applications to minimization problems is demonstrated.
We propose a splitting algorithm for solving a system of composite monotone inclusions formulated in the form of the extended set of solutions in real Hilbert spaces. The resluting algorithm is a an extension of the algorithm in [4]. The weak convergence of the algorithm proposed is proved. Applications to minimization problems is demonstrated.
△ Less
Submitted 13 August, 2013;
originally announced August 2013.
-
A variable metric extension of the forward--backward--forward algorithm for monotone operators
Authors:
B. C. Vũ
Abstract:
We propose a variable metric extension of the forward--backward-forward algorithm for finding a zero of the sum of a maximally monotone operator and a Lipschitzian monotone operator in Hilbert spaces. In turn, this framework provides a variable metric splitting algorithm for solving monotone inclusions involving sums of composite operators. Several splitting algorithms recently proposed in the lit…
▽ More
We propose a variable metric extension of the forward--backward-forward algorithm for finding a zero of the sum of a maximally monotone operator and a Lipschitzian monotone operator in Hilbert spaces. In turn, this framework provides a variable metric splitting algorithm for solving monotone inclusions involving sums of composite operators. Several splitting algorithms recently proposed in the literature are recovered as special cases.
△ Less
Submitted 31 October, 2012; v1 submitted 10 October, 2012;
originally announced October 2012.
-
Variable Metric Forward-Backward Splitting with Applications to Monotone Inclusions in Duality
Authors:
Patrick L. Combettes,
Bang C. Vũ
Abstract:
We propose a variable metric forward-backward splitting algorithm and prove its convergence in real Hilbert spaces. We then use this framework to derive primal-dual splitting algorithms for solving various classes of monotone inclusions in duality. Some of these algorithms are new even when specialized to the fixed metric case. Various applications are discussed.
We propose a variable metric forward-backward splitting algorithm and prove its convergence in real Hilbert spaces. We then use this framework to derive primal-dual splitting algorithms for solving various classes of monotone inclusions in duality. Some of these algorithms are new even when specialized to the fixed metric case. Various applications are discussed.
△ Less
Submitted 28 June, 2012;
originally announced June 2012.
-
Variable Metric Quasi-Fejér Monotonicity
Authors:
Patrick L. Combettes,
Bang C. Vu
Abstract:
The notion of quasi-Fejér monotonicity has proven to be an efficient tool to simplify and unify the convergence analysis of various algorithms arising in applied nonlinear analysis. In this paper, we extend this notion in the context of variable metric algorithms, whereby the underlying norm is allowed to vary at each iteration. Applications to convex optimization and inverse problems are demonstr…
▽ More
The notion of quasi-Fejér monotonicity has proven to be an efficient tool to simplify and unify the convergence analysis of various algorithms arising in applied nonlinear analysis. In this paper, we extend this notion in the context of variable metric algorithms, whereby the underlying norm is allowed to vary at each iteration. Applications to convex optimization and inverse problems are demonstrated.
△ Less
Submitted 31 August, 2012; v1 submitted 25 June, 2012;
originally announced June 2012.
-
A splitting algorithm for dual monotone inclusions involving cocoercive operators
Authors:
Bang Cong Vu
Abstract:
We consider the problem of solving dual monotone inclusions involving sums of composite parallel-sum type operators. A feature of this work is to exploit explicitly the cocoercivity of some of the operators appearing in the model. Several splitting algorithms recently proposed in the literature are recovered as special cases.
We consider the problem of solving dual monotone inclusions involving sums of composite parallel-sum type operators. A feature of this work is to exploit explicitly the cocoercivity of some of the operators appearing in the model. Several splitting algorithms recently proposed in the literature are recovered as special cases.
△ Less
Submitted 8 October, 2011;
originally announced October 2011.
-
Proximity for Sums of Composite Functions
Authors:
Patrick L. Combettes,
Dinh Dung,
Bang Cong Vu
Abstract:
We propose an algorithm for computing the proximity operator of a sum of composite convex functions in Hilbert spaces and investigate its asymptotic behavior. Applications to best approximation and image recovery are described.
We propose an algorithm for computing the proximity operator of a sum of composite convex functions in Hilbert spaces and investigate its asymptotic behavior. Applications to best approximation and image recovery are described.
△ Less
Submitted 20 July, 2010;
originally announced July 2010.
-
Dualization of Signal Recovery Problems
Authors:
Patrick L. Combettes,
Dinh Dung,
Bang Cong Vu
Abstract:
In convex optimization, duality theory can sometimes lead to simpler solution methods than those resulting from direct primal analysis. In this paper, this principle is applied to a class of composite variational problems arising in particular in signal recovery. These problems are not easily amenable to solution by current methods but they feature Fenchel-Moreau-Rockafellar dual problems that can…
▽ More
In convex optimization, duality theory can sometimes lead to simpler solution methods than those resulting from direct primal analysis. In this paper, this principle is applied to a class of composite variational problems arising in particular in signal recovery. These problems are not easily amenable to solution by current methods but they feature Fenchel-Moreau-Rockafellar dual problems that can be solved by forward-backward splitting. The proposed algorithm produces simultaneously a sequence converging weakly to a dual solution, and a sequence converging strongly to the primal solution. Our framework is shown to capture and extend several existing duality-based signal recovery methods and to be applicable to a variety of new problems beyond their scope.
△ Less
Submitted 20 June, 2010; v1 submitted 2 July, 2009;
originally announced July 2009.
