Showing 1–40 of 40 results for author: Moursi, W M

Projection, Degeneracy, and Singularity Degree for Spectrahedra

Authors: Haesol Im, Woosuk L. Jung, Walaa M. Moursi, David Torregrosa-Belin, Henry Wolkowicz

Abstract: Facial reduction, FR, is a regularization technique for convex programs where the strict feasibility constraint qualification, CQ, fails. Though this CQ holds generically, failure is pervasive in applications such as semidefinite relaxations of hard discrete optimization problems. In this paper we relate FR to the analysis of the convergence behaviour of a semi-smooth Newton root finding method fo… ▽ More Facial reduction, FR, is a regularization technique for convex programs where the strict feasibility constraint qualification, CQ, fails. Though this CQ holds generically, failure is pervasive in applications such as semidefinite relaxations of hard discrete optimization problems. In this paper we relate FR to the analysis of the convergence behaviour of a semi-smooth Newton root finding method for the projection onto a spectrahedron, i.e., onto the intersection of a linear manifold and the semidefinite cone. In the process, we derive and use an elegant formula for the projection onto a face of the semidefinite cone. We show further that the ill-conditioning of the Jacobian of the Newton method near optimality characterizes the degeneracy of the nearest point in the spectrahedron. We apply the results, both theoretically and empirically, to the problem of finding nearest points to the sets of: (i) correlation matrices or the elliptope; and (ii) semidefinite relaxations of permutation matrices or the vontope, i.e., the feasible sets for the semidefinite relaxations of the max-cut and quadratic assignment problems, respectively. △ Less

Submitted 8 July, 2024; originally announced July 2024.

MSC Class: 90C22; 90C25; 90C27; 90C59

Infeasible and Critically Feasible Optimal Control

Authors: Regina S. Burachik, C. Yalçın Kaya, Walaa M. Moursi

Abstract: We consider optimal control problems involving two constraint sets: one comprised of linear ordinary differential equations with the initial and terminal states specified and the other defined by the control variables constrained by simple bounds. When the intersection of these two sets is empty, typically because the bounds on the control variables are too tight, the problem becomes infeasible. I… ▽ More We consider optimal control problems involving two constraint sets: one comprised of linear ordinary differential equations with the initial and terminal states specified and the other defined by the control variables constrained by simple bounds. When the intersection of these two sets is empty, typically because the bounds on the control variables are too tight, the problem becomes infeasible. In this paper, we prove that, under a controllability assumption, the ``best approximation'' optimal control minimizing the distance (and thus finding the ``gap'') between the two sets is of bang--bang type, with the ``gap function'' playing the role of a switching function. The critically feasible control solution (the case when one has the smallest control bound for which the problem is feasible) is also shown to be of bang--bang type. We present the full analytical solution for the critically feasible problem involving the (simple but rich enough) double integrator. We illustrate the overall results numerically on various challenging example problems. △ Less

Submitted 16 January, 2024; originally announced January 2024.

Comments: 22 pages, 3 figures

Exact Solutions for the NP-hard Wasserstein Barycenter Problem using a Doubly Nonnegative Relaxation and a Splitting Method

Authors: Abdo Alfakih, Jeffery Cheng, Woosuk L. Jung, Walaa M. Moursi, Henry Wolkowicz

Abstract: The simplified Wasserstein barycenter problem consists in selecting one point from $k$ given sets, each set consisting of $n$ points, with the aim of minimizing the sum of distances to the barycenter of the $k$ points chosen. This problem is known to be NP-hard. We compute the Wasserstein barycenter by exploiting the Euclidean distance matrix structure to obtain a facially reduced doubly nonnegati… ▽ More The simplified Wasserstein barycenter problem consists in selecting one point from $k$ given sets, each set consisting of $n$ points, with the aim of minimizing the sum of distances to the barycenter of the $k$ points chosen. This problem is known to be NP-hard. We compute the Wasserstein barycenter by exploiting the Euclidean distance matrix structure to obtain a facially reduced doubly nonnegative relaxation (DNN). The facial reduction provides a natural splitting for applying the symmetric alternating directions method of multipliers (sADMM) to the DNN relaxation. The sADMM method exploits structure in the subproblems to find strong upper and lower bounds. The purpose of this paper is twofold. First we want to illustrate the strength of this DNN relaxation with a splitting approach. Our numerical tests then illustrate the surprising success on random problems, as we generally, efficiently, find the provable exact solution of this NP-hard problem. Comparisons with current commercial software illustrate this surprising efficiency. However, we demonstrate and prove that there is a duality gap for problems with enough multiple optimal solutions, and that this arises from problems with highly symmetrized structure. △ Less

Submitted 8 November, 2023; originally announced November 2023.

MSC Class: 90C26; 65K10; 90C27; 90C22

Accelerated gradient descent: A guaranteed bound for a heuristic restart strategy

Authors: Walaa M. Moursi, Viktor Pavlovic, Stephen A. Vavasis

Abstract: The $O(1/k^2)$ convergence rate in function value of accelerated gradient descent is optimal, but there are many modifications that have been used to speed up convergence in practice. Among these modifications are restarts, that is, starting the algorithm with the current iteration being considered as the initial point. We focus on the adaptive restart techniques introduced by O'Donoghue and Candè… ▽ More The $O(1/k^2)$ convergence rate in function value of accelerated gradient descent is optimal, but there are many modifications that have been used to speed up convergence in practice. Among these modifications are restarts, that is, starting the algorithm with the current iteration being considered as the initial point. We focus on the adaptive restart techniques introduced by O'Donoghue and Candès, specifically their gradient restart strategy. While the gradient restart strategy is a heuristic in general, we prove that applying gradient restarts preserves and in fact improves the $O(1/k^2)$ bound, hence establishing function value convergence, for one-dimensional functions. Applications of our results to separable and nearly separable functions are presented. △ Less

Submitted 11 October, 2023; originally announced October 2023.

MSC Class: Primary 90C25; 65K05; Secondary 65K10; 49M27

Range of the displacement operator of PDHG with applications to quadratic and conic programming

Authors: Tao Jiang, Walaa M. Moursi, Stephen A. Vavasis

Abstract: Primal-dual hybrid gradient (PDHG) is a first-order method for saddle-point problems and convex programming introduced by Chambolle and Pock. Recently, Applegate et al.\ analyzed the behavior of PDHG when applied to an infeasible or unbounded instance of linear programming, and in particular, showed that PDHG is able to diagnose these conditions. Their analysis hinges on the notion of the infimal… ▽ More Primal-dual hybrid gradient (PDHG) is a first-order method for saddle-point problems and convex programming introduced by Chambolle and Pock. Recently, Applegate et al.\ analyzed the behavior of PDHG when applied to an infeasible or unbounded instance of linear programming, and in particular, showed that PDHG is able to diagnose these conditions. Their analysis hinges on the notion of the infimal displacement vector in the closure of the range of the displacement map** of the splitting operator that encodes the PDHG algorithm. In this paper, we develop a novel formula for this range using monotone operator theory. The analysis is then specialized to conic programming and further to quadratic programming (QP) and second-order cone programming (SOCP). A consequence of our analysis is that PDHG is able to diagnose infeasible or unbounded instances of QP and of the ellipsoid-separation problem, a subclass of SOCP. △ Less

Submitted 26 September, 2023; originally announced September 2023.

MSC Class: Primary 49M27; 65K10; 90C25; Secondary 47H14; 49M29; 49N15

On the Bredies-Chenchene-Lorenz-Naldi algorithm

Authors: Heinz H. Bauschke, Walaa M. Moursi, Shambhavi Singh, Xianfu Wang

Abstract: Monotone inclusion problems occur in many areas of optimization and variational analysis. Splitting methods, which utilize resolvents or proximal map**s of the underlying operators, are often applied to solve these problems. In 2022, Bredies, Chenchene, Lorenz, and Naldi introduced a new elegant algorithmic framework that encompasses various well known algorithms including Douglas-Rachford and C… ▽ More Monotone inclusion problems occur in many areas of optimization and variational analysis. Splitting methods, which utilize resolvents or proximal map**s of the underlying operators, are often applied to solve these problems. In 2022, Bredies, Chenchene, Lorenz, and Naldi introduced a new elegant algorithmic framework that encompasses various well known algorithms including Douglas-Rachford and Chambolle-Pock. They obtained powerful weak and strong convergence results, where the latter type relies on additional strong monotonicity assumptions. In this paper, we complement the analysis by Bredies et al. by relating the projections of the fixed point sets of the underlying operators that generate the (reduced and original) preconditioned proximal point sequences. We also obtain strong convergence results in the case of linear relations. Various examples are provided to illustrate the applicability of our results. △ Less

Submitted 19 July, 2023; originally announced July 2023.

How averaged is the composition of two linear projections?

Authors: Heinz H. Bauschke, Theo Bendit, Walaa M. Moursi

Abstract: Projection operators are fundamental algorithmic operators in Analysis and Optimization. It is well known that these operators are firmly nonexpansive; however, their composition is generally only averaged and no longer firmly nonexpansive. In this note, we introduce the modulus of averagedness and provide an exact result for the composition of two linear projection operators. As a consequence, we… ▽ More Projection operators are fundamental algorithmic operators in Analysis and Optimization. It is well known that these operators are firmly nonexpansive; however, their composition is generally only averaged and no longer firmly nonexpansive. In this note, we introduce the modulus of averagedness and provide an exact result for the composition of two linear projection operators. As a consequence, we deduce that the Ogura-Yamada bound for the modulus of the composition is sharp. △ Less

Submitted 23 March, 2023; originally announced March 2023.

MSC Class: 47H09 (Primary) 65K05; 90C25 (Secondary)

Optimal Control Duality and the Douglas-Rachford Algorithm

Authors: Regina S. Burachik, Bethany I Caldwell, C. Yalçın Kaya, Walaa M. Moursi

Abstract: We explore the relationship between the dual of a weighted minimum-energy control problem, a special case of linear-quadratic optimal control problems, and the Douglas-Rachford (DR) algorithm. We obtain an expression for the fixed point of the DR operator as applied to solving the optimal control problem, which in turn devises a certificate of optimality that can be employed for numerical verifica… ▽ More We explore the relationship between the dual of a weighted minimum-energy control problem, a special case of linear-quadratic optimal control problems, and the Douglas-Rachford (DR) algorithm. We obtain an expression for the fixed point of the DR operator as applied to solving the optimal control problem, which in turn devises a certificate of optimality that can be employed for numerical verification. The fixed point and the optimality check are illustrated in an example optimal control problem. △ Less

Submitted 21 October, 2023; v1 submitted 11 March, 2023; originally announced March 2023.

MSC Class: 49N10; 49N15; 49M29

Regularized Nonsmooth Newton Algorithms for Best Approximation

Authors: Yair Censor, Walaa M. Moursi, Tyler Weames, Henry Wolkowicz

Abstract: We consider the problem of finding the best approximation point from a polyhedral set, and its applications, in particular to solving large-scale linear programs. The classical projection problem has many various and many applications. We study a regularized nonsmooth Newton type solution method where the Jacobian is singular; and we compare the computational performance to that of the classical p… ▽ More We consider the problem of finding the best approximation point from a polyhedral set, and its applications, in particular to solving large-scale linear programs. The classical projection problem has many various and many applications. We study a regularized nonsmooth Newton type solution method where the Jacobian is singular; and we compare the computational performance to that of the classical projection method of Halperin-Lions-Wittmann-Bauschke (HLWB). We observe empirically that the regularized nonsmooth method significantly outperforms the HLWB method. However, the HLWB has a convergence guarantee while the nonsmooth method is not monotonic and does not guarantee convergence due in part to singularity of the generalized Jacobian. Our application to solving large-scale linear programs uses a parametrized projection problem. This leads to a \emph{step** stone external path following} algorithm. Other applications are finding triangles from branch and bound methods, and generalized constrained linear least squares. We include scaling methods that improve the efficiency and robustness. △ Less

Submitted 8 June, 2023; v1 submitted 26 December, 2022; originally announced December 2022.

Comments: 38 pages, 7 tables, 8 figures

MSC Class: 46N10; 49J52; 65K10; 90C05; 90C46; 90C59; 65F10

How to project onto the intersection of a closed affine subspace and a hyperplane

Authors: Heinz H. Bauschke, Dayou Mao, Walaa M. Moursi

Abstract: Let $A$ be a closed affine subspace and let $B$ be a hyperplane in a Hilbert space. Suppose we are given their associated nearest point map**s $P_A$ and $P_B$, respectively. We present a formula for the projection onto their intersection $A\cap B$. As a special case, we derive a formula for the projection onto the intersection of two hyperplanes. These formulas provides useful information even i… ▽ More Let $A$ be a closed affine subspace and let $B$ be a hyperplane in a Hilbert space. Suppose we are given their associated nearest point map**s $P_A$ and $P_B$, respectively. We present a formula for the projection onto their intersection $A\cap B$. As a special case, we derive a formula for the projection onto the intersection of two hyperplanes. These formulas provides useful information even if $A\cap B$ is empty. Examples and numerical experiments are also provided. △ Less

Submitted 22 June, 2022; originally announced June 2022.

MSC Class: 15A04; 41A50; 47A50 (Primary) 47H09; 90C25 (Secondary)

The range of the Douglas-Rachford operator in infinite-dimensional Hilbert spaces

Authors: Walaa M. Moursi

Abstract: The Douglas-Rachford algorithm is one of the most prominent splitting algorithms for solving convex optimization problems. Recently, the method has been successful in finding a generalized solution (provided that one exists) for optimization problems in the inconsistent case, i.e., when a solution does not exist. The convergence analysis of the inconsistent case hinges on the study of the range of… ▽ More The Douglas-Rachford algorithm is one of the most prominent splitting algorithms for solving convex optimization problems. Recently, the method has been successful in finding a generalized solution (provided that one exists) for optimization problems in the inconsistent case, i.e., when a solution does not exist. The convergence analysis of the inconsistent case hinges on the study of the range of the displacement operator associated with the Douglas-Rachford splitting operator and the corresponding minimal displacement vector. In this paper, we provide a formula for the range of the Douglas-Rachford splitting operator in (possibly) infinite-dimensional Hilbert space under mild assumptions on the underlying operators. Our new results complement known results in finite-dimensional Hilbert spaces. Several examples illustrate and tighten our conclusions. △ Less

Submitted 14 June, 2022; originally announced June 2022.

MSC Class: 47H05; 47H09; 90C25; 90C46; 47H14; 49M27; 49M29; 49N15

Strongly nonexpansive map**s revisited: uniform monotonicity and operator splitting

Authors: Leon Liu, Walaa M. Moursi, Jon Vanderwerff

Abstract: The correspondence between the class of nonexpansive map**s and the class of maximally monotone operators via the reflected resolvents of the latter has played an instrumental role in the convergence analysis of the splitting methods. Indeed, the performance of some of these methods, e.g., Douglas-Rachford and Peaceman-Rachford methods hinges on iterating the so-called splitting operator associa… ▽ More The correspondence between the class of nonexpansive map**s and the class of maximally monotone operators via the reflected resolvents of the latter has played an instrumental role in the convergence analysis of the splitting methods. Indeed, the performance of some of these methods, e.g., Douglas-Rachford and Peaceman-Rachford methods hinges on iterating the so-called splitting operator associated with the individual operators. This splitting operator is a function of the composition of the reflected resolvents of the underlying operators. In this paper, we provide a comprehensive study of the class of uniformly monotone operators and their corresponding reflected resolvents. We show that the latter is closely related to the class of the strongly nonexpansive operators introduced by Bruck and Reich. Connections to duality via inverse operators are systematically studied. We provide applications to Douglas-Rachford and Peaceman-Rachford methods. Examples that illustrate and tighten our results are presented. △ Less

Submitted 18 May, 2022; originally announced May 2022.

On the Douglas-Rachford and Peaceman-Rachford algorithms in the presence of uniform monotonicity and the absence of minimizers

Authors: Regina S. Burachik, Bethany I. Caldwell, C. Yalçın Kaya, Walaa M. Moursi, Matthew Saurette

Abstract: The Douglas-Rachford and Peaceman-Rachford algorithms have been successfully employed to solve convex optimization problems, or more generally find zeros of monotone inclusions. Recently, the behaviour of these methods in the inconsistent case, i.e., in the absence of solutions has triggered significant consideration. It has been shown that under mild assumptions the shadow sequence of the Douglas… ▽ More The Douglas-Rachford and Peaceman-Rachford algorithms have been successfully employed to solve convex optimization problems, or more generally find zeros of monotone inclusions. Recently, the behaviour of these methods in the inconsistent case, i.e., in the absence of solutions has triggered significant consideration. It has been shown that under mild assumptions the shadow sequence of the Douglas-Rachford algorithm converges weakly to a generalized solution when the underlying operators are subdifferentials of proper lower semicontinuous convex functions. However, no convergence behaviour has been proved in the case of Peaceman-Rachford algorithm. In this paper, we prove the convergence of the shadow sequences associated with the Douglas-Rachford algorithm and Peaceman-Rachford algorithm when one of the operators is uniformly monotone and $3^*$ monotone but not necessarily a subdifferential. Several examples illustrate and strengthen our conclusion. We carry out numerical experiments using example instances of optimization problems. △ Less

Submitted 24 May, 2024; v1 submitted 17 January, 2022; originally announced January 2022.

MSC Class: Primary 49M27; 65K10; 90C25; Secondary 47H14; 49M29

On the Douglas-Rachford algorithm for solving possibly inconsistent optimization problems

Authors: Heinz H. Bauschke, Walaa M. Moursi

Abstract: More than 40 years ago, Lions and Mercier introduced in a seminal paper the Douglas-Rachford algorithm. Today, this method is well recognized as a classical and highly successful splitting method to find minimizers of the sum of two (not necessarily smooth) convex functions. While the underlying theory has matured, one case remains a mystery: the behaviour of the shadow sequence when the given fun… ▽ More More than 40 years ago, Lions and Mercier introduced in a seminal paper the Douglas-Rachford algorithm. Today, this method is well recognized as a classical and highly successful splitting method to find minimizers of the sum of two (not necessarily smooth) convex functions. While the underlying theory has matured, one case remains a mystery: the behaviour of the shadow sequence when the given functions have disjoint domains. Building on previous work, we establish for the first time weak and value convergence of the shadow sequence generated by the Douglas-Rachford algorithm in a setting of unprecedented generality. The weak limit point is shown to solve the associated normal problem which is a minimal perturbation of the original optimization problem. We also present new results on the geometry of the minimal displacement vector. △ Less

Submitted 22 June, 2021; originally announced June 2021.

MSC Class: 65K10; 90C25 (Primary); 47H05 (Secondary)

Edelstein's Astonishing Affine Isometry

Authors: Heinz H. Bauschke, Sylvain Gretchko, Walaa M. Moursi, Matthew Saurette

Abstract: In 1964, Michael Edelstein presented an amazing affine isometry acting on the space of square-summable sequences. This operator has no fixed points, but a suborbit that converges to 0 while another escapes in norm to infinity! We revisit, extend and sharpen his construction. Moreover, we sketch a connection to modern optimization and monotone operator theory. In 1964, Michael Edelstein presented an amazing affine isometry acting on the space of square-summable sequences. This operator has no fixed points, but a suborbit that converges to 0 while another escapes in norm to infinity! We revisit, extend and sharpen his construction. Moreover, we sketch a connection to modern optimization and monotone operator theory. △ Less

Submitted 15 September, 2020; originally announced September 2020.

On compositions of special cases of Lipschitz continuous operators

Authors: Pontus Giselsson, Walaa M. Moursi

Abstract: Many iterative optimization algorithms involve compositions of special cases of Lipschitz continuous operators, namely firmly nonexpansive, averaged and nonexpansive operators. The structure and properties of the compositions are of particular importance in the proofs of convergence of such algorithms. In this paper, we systematically study the compositions of further special cases of Lipschitz co… ▽ More Many iterative optimization algorithms involve compositions of special cases of Lipschitz continuous operators, namely firmly nonexpansive, averaged and nonexpansive operators. The structure and properties of the compositions are of particular importance in the proofs of convergence of such algorithms. In this paper, we systematically study the compositions of further special cases of Lipschitz continuous operators. Applications of our results include compositions of scaled conically nonexpansive map**s, as well as the Douglas--Rachford and forward-backward operators, when applied to solve certain structured monotone inclusion and optimization problems. Several examples illustrate and tighten our conclusions. △ Less

Submitted 30 December, 2019; originally announced December 2019.

MSC Class: Primary 47H05; 90C25; Secondary 47H09; 49M27; 65K05; 65K10

On the behaviour of the Douglas-Rachford algorithm for minimizing a convex function subject to a linear constraint

Authors: Heinz H. Bauschke, Walaa M. Moursi

Abstract: The Douglas-Rachford algorithm (DRA) is a powerful optimization method for minimizing the sum of two convex (not necessarily smooth) functions. The vast majority of previous research dealt with the case when the sum has at least one minimizer. In the absence of minimizers, it was recently shown that for the case of two indicator functions, the DRA converges to a best approximation solution. In thi… ▽ More The Douglas-Rachford algorithm (DRA) is a powerful optimization method for minimizing the sum of two convex (not necessarily smooth) functions. The vast majority of previous research dealt with the case when the sum has at least one minimizer. In the absence of minimizers, it was recently shown that for the case of two indicator functions, the DRA converges to a best approximation solution. In this paper, we present a new convergence result on the the DRA applied to the problem of minimizing a convex function subject to a linear constraint. Indeed, a normal solution may be found even when the domain of the objective function and the linear subspace constraint have no point in common. As an important application, a new parallel splitting result is provided. We also illustrate our results through various examples. △ Less

Submitted 9 July, 2020; v1 submitted 14 August, 2019; originally announced August 2019.

MSC Class: 49M27; 65K10; 90C25; 47H14; 49M29

Maximally monotone operators with ranges whose closures are not convex and an answer to a recent question by Stephen Simons

Authors: Heinz H. Bauschke, Walaa M. Moursi, Xianfu Wang

Abstract: In his recent Proceedings of the AMS paper "Gossez's skew linear map and its pathological maximally monotone multifunctions", Stephen Simons proved that the closure of the range of the sum of the Gossez operator and a multiple of the duality map is nonconvex whenever the scalar is between 0 and 4. The problem of the convexity of that range when the scalar is equal to 4 was explicitly stated. In th… ▽ More In his recent Proceedings of the AMS paper "Gossez's skew linear map and its pathological maximally monotone multifunctions", Stephen Simons proved that the closure of the range of the sum of the Gossez operator and a multiple of the duality map is nonconvex whenever the scalar is between 0 and 4. The problem of the convexity of that range when the scalar is equal to 4 was explicitly stated. In this paper, we answer this question in the negative for any scalar greater than or equal to 4. We derive this result from an abstract framework that allows us to also obtain a corresponding result for the Fitzpatrick-Phelps integral operator. △ Less

Submitted 14 September, 2019; v1 submitted 29 April, 2019; originally announced April 2019.

MSC Class: Primary 47H05; Secondary 46B20; 47A05

Differentiating Through a Cone Program

Authors: Akshay Agrawal, Shane Barratt, Stephen Boyd, Enzo Busseti, Walaa M. Moursi

Abstract: We consider the problem of efficiently computing the derivative of the solution map of a convex cone program, when it exists. We do this by implicitly differentiating the residual map for its homogeneous self-dual embedding, and solving the linear systems of equations required using an iterative method. This allows us to efficiently compute the derivative operator, and its adjoint, evaluated at a… ▽ More We consider the problem of efficiently computing the derivative of the solution map of a convex cone program, when it exists. We do this by implicitly differentiating the residual map for its homogeneous self-dual embedding, and solving the linear systems of equations required using an iterative method. This allows us to efficiently compute the derivative operator, and its adjoint, evaluated at a vector. These correspond to computing an approximate new solution, given a perturbation to the cone program coefficients (i.e., perturbation analysis), and to computing the gradient of a function of the solution with respect to the coefficients. Our method scales to large problems, with numbers of coefficients in the millions. We present an open-source Python implementation of our method that solves a cone program and returns the derivative and its adjoint as abstract linear maps; our implementation can be easily integrated into software systems for automatic differentiation. △ Less

Submitted 19 May, 2020; v1 submitted 18 April, 2019; originally announced April 2019.

Comments: Correct sign error on page 6

Generalized monotone operators and their averaged resolvents

Authors: Heinz H. Bauschke, Walaa M. Moursi, Xianfu Wang

Abstract: The correspondence between the monotonicity of a (possibly) set-valued operator and the firm nonexpansiveness of its resolvent is a key ingredient in the convergence analysis of many optimization algorithms. Firmly nonexpansive operators form a proper subclass of the more general - but still pleasant from an algorithmic perspective - class of averaged operators. In this paper, we introduce the new… ▽ More The correspondence between the monotonicity of a (possibly) set-valued operator and the firm nonexpansiveness of its resolvent is a key ingredient in the convergence analysis of many optimization algorithms. Firmly nonexpansive operators form a proper subclass of the more general - but still pleasant from an algorithmic perspective - class of averaged operators. In this paper, we introduce the new notion of conically nonexpansive operators which generalize nonexpansive map**s. We characterize averaged operators as being resolvents of comonotone operators under appropriate scaling. As a consequence, we characterize the proximal point map**s associated with hypoconvex functions as cocoercive operators, or equivalently; as displacement map**s of conically nonexpansive operators. Several examples illustrate our analysis and demonstrate tightness of our results. △ Less

Submitted 26 February, 2019; originally announced February 2019.

MSC Class: 47H05; 47H09; 49N15; 90C25

Numerical explorations of feasibility algorithms for finding points in the intersection of finite sets

Authors: Heinz H. Bauschke, Sylvain Gretchko, Walaa M. Moursi

Abstract: Projection methods are popular algorithms for iteratively solving feasibility problems in Euclidean or even Hilbert spaces. They employ (selections of) nearest point map**s to generate sequences that are designed to approximate a point in the intersection of a collection of constraint sets. Theoretical properties of projection methods are fairly well understood when the underlying constraint set… ▽ More Projection methods are popular algorithms for iteratively solving feasibility problems in Euclidean or even Hilbert spaces. They employ (selections of) nearest point map**s to generate sequences that are designed to approximate a point in the intersection of a collection of constraint sets. Theoretical properties of projection methods are fairly well understood when the underlying constraint sets are convex; however, convergence results for the nonconvex case are more complicated and typically only local. In this paper, we explore the perhaps simplest instance of a feasibility algorithm, namely when each constraint set consists of only finitely many points. We numerically investigate four constellations: either few or many constraint sets, with either few or many points. Each constellation is tackled by four popular projection methods each of which features a tuning parameter. We examine the behaviour for a single and for a multitude of orbits, and we also consider local and global behaviour. Our findings demonstrate the importance of the choice of the algorithm and that of the tuning parameter. △ Less

Submitted 23 January, 2019; originally announced January 2019.

MSC Class: 49M20; 49M27; 49M37; 65K05; 65K10; 90C25; 90C26; 90C30

On the minimal displacement vector of compositions and convex combinations of nonexpansive map**s

Authors: Heinz H. Bauschke, Walaa M. Moursi

Abstract: Monotone operators and (firmly) nonexpansive map**s are fundamental objects in modern analysis and computational optimization. Five years ago, it was shown that if finitely many firmly nonexpansive map**s have or "almost have" fixed points, then the same is true for compositions and convex combinations. More recently, sharp information about the minimal displacement vector of compositions and… ▽ More Monotone operators and (firmly) nonexpansive map**s are fundamental objects in modern analysis and computational optimization. Five years ago, it was shown that if finitely many firmly nonexpansive map**s have or "almost have" fixed points, then the same is true for compositions and convex combinations. More recently, sharp information about the minimal displacement vector of compositions and of convex combinations of firmly nonexpansive map**s was obtained in terms of the displacement vectors of the underlying operators. Using a new proof technique based on the Brezis-Haraux theorem and reflected resolvents, we extend these results from firmly nonexpansive to general averaged nonexpansive map**s. Various examples illustrate the tightness of our results. △ Less

Submitted 4 September, 2018; originally announced September 2018.

MSC Class: 47H05; 47H09 (Primary); 47H10; 90C25 (Secondary)

A note on the equivalence of operator splitting methods

Authors: Walaa M. Moursi, Yuriy Zinchenko

Abstract: This paper provides a comprehensive discussion of the equivalences between splitting methods. These equivalences have been studied over the past few decades and, in fact, have proven to be very useful. In this paper, we survey known results and also present new ones. In particular, we provide simplified proofs of the equivalence of the ADMM and the Douglas-Rachford method and the equivalence of th… ▽ More This paper provides a comprehensive discussion of the equivalences between splitting methods. These equivalences have been studied over the past few decades and, in fact, have proven to be very useful. In this paper, we survey known results and also present new ones. In particular, we provide simplified proofs of the equivalence of the ADMM and the Douglas-Rachford method and the equivalence of the ADMM with intermediate update of multipliers and the Peaceman-Rachford method. Other splitting methods are also considered. △ Less

Submitted 8 June, 2018; originally announced June 2018.

MSC Class: Primary 47H05; 47H09; 49M27; Secondary 49M29; 49N15; 90C25

On the asymptotic behaviour of the Aragon Artacho-Campoy algorithm

Authors: Salihah Alwadani, Heinz H. Bauschke, Walaa M. Moursi, X. Wang

Abstract: Aragón Artacho and Campoy recently proposed a new method for computing the projection onto the intersection of two closed convex sets in Hilbert space; moreover, they proposed in 2018 a generalization from normal cone operators to maximally monotone operators. In this paper, we complete this analysis by demonstrating that the underlying curve converges to the nearest zero of the sum of the two ope… ▽ More Aragón Artacho and Campoy recently proposed a new method for computing the projection onto the intersection of two closed convex sets in Hilbert space; moreover, they proposed in 2018 a generalization from normal cone operators to maximally monotone operators. In this paper, we complete this analysis by demonstrating that the underlying curve converges to the nearest zero of the sum of the two operators. We also provide a new interpretation of the underlying operators in terms of the resolvent and the proximal average. △ Less

Submitted 28 May, 2018; originally announced May 2018.

MSC Class: Primary 47H05; 90C25; Secondary 47H09; 49M27; 65K05; 65K10

Douglas-Rachford splitting for a Lipschitz continuous and a strongly monotone operator

Authors: Walaa M. Moursi, Lieven Vandenberghe

Abstract: The Douglas-Rachford method is a popular splitting technique for finding a zero of the sum of two subdifferential operators of proper closed convex functions; more generally two maximally monotone operators. Recent results concerned with linear rates of convergence of the method require additional properties of the underlying monotone operators, such as strong monotonicity and cocoercivity. In thi… ▽ More The Douglas-Rachford method is a popular splitting technique for finding a zero of the sum of two subdifferential operators of proper closed convex functions; more generally two maximally monotone operators. Recent results concerned with linear rates of convergence of the method require additional properties of the underlying monotone operators, such as strong monotonicity and cocoercivity. In this paper, we study the case when one operator is Lipschitz continuous but not necessarily a subdifferential operator and the other operator is strongly monotone. This situation arises in optimization methods which involve primal-dual approaches. We provide new linear convergence results in this setting. △ Less

Submitted 23 May, 2018; originally announced May 2018.

MSC Class: Primary 47H05; 47H09; 49M27; 90C25. Secondary 49M29

Intriguing maximally monotone operators derived from nonsunny nonexpansive retractions

Authors: Heinz H. Bauschke, Levi Miller, Walaa M. Moursi

Abstract: Monotone operator theory and fixed point theory for nonexpansive map**s are central areas in modern nonlinear analysis and optimization. Although these areas are fairly well developed, almost all examples published are based on subdifferential operators, linear relations, or combinations thereof. In this paper, we construct an intriguing maximally monotone operator induced by a certain nonexpa… ▽ More Monotone operator theory and fixed point theory for nonexpansive map**s are central areas in modern nonlinear analysis and optimization. Although these areas are fairly well developed, almost all examples published are based on subdifferential operators, linear relations, or combinations thereof. In this paper, we construct an intriguing maximally monotone operator induced by a certain nonexpansive retraction. We analyze this operator, which does not appear to be assembled from subdifferential operators or linear relations, in some detail. Particular emphasis is placed on duality and strong monotonicity. △ Less

Submitted 23 May, 2018; originally announced May 2018.

MSC Class: Primary 47H05; 47H09; Secondary 90C25

The magnitude of the minimal displacement vector for compositions and convex combinations of firmly nonexpansive map**s

Authors: Heinz H. Bauschke, Walaa M. Moursi

Abstract: Maximally monotone operators and firmly nonexpansive map**s play key roles in modern optimization and nonlinear analysis. Five years ago, it was shown that if finitely many firmly nonexpansive operators are all asymptotically regular (i.e., the have or "almost have" fixed points), then the same is true for compositions and convex combinations. In this paper, we derive bounds on the magnitude of… ▽ More Maximally monotone operators and firmly nonexpansive map**s play key roles in modern optimization and nonlinear analysis. Five years ago, it was shown that if finitely many firmly nonexpansive operators are all asymptotically regular (i.e., the have or "almost have" fixed points), then the same is true for compositions and convex combinations. In this paper, we derive bounds on the magnitude of the minimal displacement vectors of compositions and of convex combinations in terms of the displacement vectors of the underlying operators. Our results completely generalize earlier works. Moreover, we present various examples illustrating that our bounds are sharp. △ Less

Submitted 1 December, 2017; originally announced December 2017.

MSC Class: Primar 47H05; 47H09 Secondary 47H10; 90C25

The forward-backward algorithm and the normal problem

Authors: Walaa M. Moursi

Abstract: The forward-backward splitting technique is a popular method for solving monotone inclusions that has applications in optimization. In this paper we explore the behaviour of the algorithm when the inclusion problem has no solution. We present a new formula to define the normal solutions using the forward-backward operator. We also provide a formula for the range of the displacement map of the forw… ▽ More The forward-backward splitting technique is a popular method for solving monotone inclusions that has applications in optimization. In this paper we explore the behaviour of the algorithm when the inclusion problem has no solution. We present a new formula to define the normal solutions using the forward-backward operator. We also provide a formula for the range of the displacement map of the forward-backward operator. Several examples illustrate our theory. △ Less

Submitted 7 August, 2016; originally announced August 2016.

MSC Class: Primary 47H09; 49M27; 65K10; Secondary 47H05; 47H14; 49M29; 49N19

On the Douglas-Rachford algorithm

Authors: Heinz H. Bauschke, Walaa M. Moursi

Abstract: The Douglas-Rachford algorithm is a very popular splitting technique for finding a zero of the sum of two maximally monotone operators. However, the behaviour of the algorithm remains mysterious in the general inconsistent case, i.e., when the sum problem has no zeros. More than a decade ago, however, it was shown that in the (possibly inconsistent) convex feasibility setting, the shadow sequence… ▽ More The Douglas-Rachford algorithm is a very popular splitting technique for finding a zero of the sum of two maximally monotone operators. However, the behaviour of the algorithm remains mysterious in the general inconsistent case, i.e., when the sum problem has no zeros. More than a decade ago, however, it was shown that in the (possibly inconsistent) convex feasibility setting, the shadow sequence remains bounded and it is weak cluster points solve a best approximation problem. In this paper, we advance the understanding of the inconsistent case significantly by providing a complete proof of the full weak convergence in the convex feasibility setting. In fact, a more general sufficient condition for the weak convergence in the general case is presented. Several examples illustrate the results. △ Less

Submitted 15 April, 2016; originally announced April 2016.

MSC Class: 47H05; 47H09; 49M27 (Primary); 49M29; 49N15; 90C25 (Secondary)

Affine nonexpansive operators, Attouch-Théra duality and the Douglas-Rachford algorithm

Authors: Heinz H. Bauschke, Brett Lukens, Walaa M. Moursi

Abstract: The Douglas-Rachford splitting algorithm was originally proposed in 1956 to solve a system of linear equations arising from the discretization of a partial differential equation. In 1979, Lions and Mercier brought forward a very powerful extension of this method suitable to solve optimization problems. In this paper, we revisit the original affine setting. We provide a powerful convergence resul… ▽ More The Douglas-Rachford splitting algorithm was originally proposed in 1956 to solve a system of linear equations arising from the discretization of a partial differential equation. In 1979, Lions and Mercier brought forward a very powerful extension of this method suitable to solve optimization problems. In this paper, we revisit the original affine setting. We provide a powerful convergence result for finding a zero of the sum of two maximally monotone affine relations. As a by product of our analysis, we obtain results concerning the convergence of iterates of affine nonexpansive map**s as well as Attouch-Théra duality. Numerous examples are presented. △ Less

Submitted 30 March, 2016; originally announced March 2016.

MSC Class: Primary 47H05; 47H09; 49M27; Secondary 49M29; 49N15; 90C25

Nearly convex sets: fine properties and domains or ranges of subdifferentials of convex functions

Authors: Sarah M. Moffat, Walaa M. Moursi, Xianfu Wang

Abstract: Nearly convex sets play important roles in convex analysis, optimization and theory of monotone operators. We give a systematic study of nearly convex sets, and construct examples of subdifferentials of lower semicontinuous convex functions whose domain or ranges are nonconvex. Nearly convex sets play important roles in convex analysis, optimization and theory of monotone operators. We give a systematic study of nearly convex sets, and construct examples of subdifferentials of lower semicontinuous convex functions whose domain or ranges are nonconvex. △ Less

Submitted 25 July, 2015; originally announced July 2015.

MSC Class: 52A41; 26A51; 47H05

On Fejér monotone sequences and nonexpansive map**s

Authors: H. H. Bauschke, M. N. Dao, W. M. Moursi

Abstract: The notion of Fejér monotonicity has proven to be a fruitful concept in fixed point theory and optimization. In this paper, we present new conditions sufficient for convergence of Fejér monotone sequences and we also provide applications to the study of nonexpansive map**s. Various examples illustrate our results. The notion of Fejér monotonicity has proven to be a fruitful concept in fixed point theory and optimization. In this paper, we present new conditions sufficient for convergence of Fejér monotone sequences and we also provide applications to the study of nonexpansive map**s. Various examples illustrate our results. △ Less

Submitted 20 July, 2015; originally announced July 2015.

MSC Class: Primary 47H09; Secondary 47H05; 90C25

Journal ref: Linear and Nonlinear Analysis, 1(2):287--295, 2015

The Douglas-Rachford algorithm in the affine-convex case

Authors: Heinz H. Bauschke, Minh N. Dao, Walaa M. Moursi

Abstract: The Douglas-Rachford algorithm is a simple yet effective method for solving convex feasibility problems. However, if the underlying constraints are inconsistent, then the convergence theory is incomplete. We provide convergence results when one constraint is an affine subspace. As a consequence, we extend a result by S**arn from halfspaces to general closed convex sets admitting least-squares so… ▽ More The Douglas-Rachford algorithm is a simple yet effective method for solving convex feasibility problems. However, if the underlying constraints are inconsistent, then the convergence theory is incomplete. We provide convergence results when one constraint is an affine subspace. As a consequence, we extend a result by S**arn from halfspaces to general closed convex sets admitting least-squares solutions. △ Less

Submitted 24 May, 2015; originally announced May 2015.

Journal ref: Operations Research Letters, 44(3):379--382, 2016

On a result of Pazy concerning the asymptotic behaviour of nonexpansive map**s

Authors: Heinz H. Bauschke, Graeme R. Douglas, Walaa M. Moursi

Abstract: In 1971, Pazy presented a beautiful trichotomy result concerning the asymptotic behaviour of the iterates of a nonexpansive map**. In this note, we analyze the fixed-point free case in more detail. Our results and examples give credence to the conjecture that the iterates always converge cosmically. In 1971, Pazy presented a beautiful trichotomy result concerning the asymptotic behaviour of the iterates of a nonexpansive map**. In this note, we analyze the fixed-point free case in more detail. Our results and examples give credence to the conjecture that the iterates always converge cosmically. △ Less

Submitted 15 May, 2015; originally announced May 2015.

Comments: 10 pages

MSC Class: Primary 47H09; Secondary 90C25

On the order of the operators in the Douglas-Rachford algorithm

Authors: Heinz H. Bauschke, Walaa M. Moursi

Abstract: The Douglas-Rachford algorithm is a popular method for finding zeros of sums of monotone operators. By its definition, the Douglas-Rachford operator is not symmetric with respect to the order of the two operators. In this paper we provide a systematic study of the two possible Douglas-Rachford operators. We show that the reflectors of the underlying operators act as bijections between the fixed po… ▽ More The Douglas-Rachford algorithm is a popular method for finding zeros of sums of monotone operators. By its definition, the Douglas-Rachford operator is not symmetric with respect to the order of the two operators. In this paper we provide a systematic study of the two possible Douglas-Rachford operators. We show that the reflectors of the underlying operators act as bijections between the fixed points sets of the two Douglas-Rachford operators. Some elegant formulae arise under additional assumptions. Various examples illustrate our results. △ Less

Submitted 11 May, 2015; originally announced May 2015.

Comments: 10 pages

MSC Class: Primary 47H09; 90C25; Secondary 47H05; 49M27; 65K05

The Douglas-Rachford algorithm for two (not necessarily intersecting) affine subspaces

Authors: Heinz H. Bauschke, Walaa M. Moursi

Abstract: The Douglas--Rachford algorithm is a classical and very successful splitting method for finding the zeros of the sums of monotone operators. When the underlying operators are normal cone operators, the algorithm solves a convex feasibility problem. In this paper, we provide a detailed study of the Douglas--Rachford iterates and the corresponding {shadow sequence} when the sets are affine subspaces… ▽ More The Douglas--Rachford algorithm is a classical and very successful splitting method for finding the zeros of the sums of monotone operators. When the underlying operators are normal cone operators, the algorithm solves a convex feasibility problem. In this paper, we provide a detailed study of the Douglas--Rachford iterates and the corresponding {shadow sequence} when the sets are affine subspaces that do not necessarily intersect. We prove strong convergence of the shadows to the nearest generalized solution. Our results extend recent work from the consistent to the inconsistent case. Various examples are provided to illustrates the results. △ Less

Submitted 14 April, 2015; originally announced April 2015.

MSC Class: Primary 47H09; 49M27; 65K05; 65K10; Secondary 47H05; 47H14; 49M29; 49N15

On the range of the Douglas-Rachford operator

Authors: Heinz H. Bauschke, Warren L. Hare, Walaa M. Moursi

Abstract: The problem of finding a minimizer of the sum of two convex functions - or, more generally, that of finding a zero of the sum of two maximally monotone operators - is of central importance in variational analysis. Perhaps the most popular method of solving this problem is the Douglas-Rachford splitting method. Surprisingly, little is known about the range of the Douglas-Rachford operator. In thi… ▽ More The problem of finding a minimizer of the sum of two convex functions - or, more generally, that of finding a zero of the sum of two maximally monotone operators - is of central importance in variational analysis. Perhaps the most popular method of solving this problem is the Douglas-Rachford splitting method. Surprisingly, little is known about the range of the Douglas-Rachford operator. In this paper, we set out to study this range systematically. We prove that for 3* monotone operators a very pleasing formula can be found that reveals the range to be nearly equal to a simple set involving the domains and ranges of the underlying operators. A similar formula holds for the range of the corresponding displacement map**. We discuss applications to subdifferential operators, to the infimal displacement vector, and to firmly nonexpansive map**s. Various examples and counter-examples are presented, including some concerning the celebrated Brezis-Haraux theorem. △ Less

Submitted 31 July, 2014; v1 submitted 15 May, 2014; originally announced May 2014.

MSC Class: 47H05; 47H09; 90C25 (Primary) 90C46; 47H14; 49M27; 49M29; 49N15 (Secondary)

Generalized solutions for the sum of two maximally monotone operators

Authors: Heinz H. Bauschke, Warren L. Hare, Walaa M. Moursi

Abstract: A common theme in mathematics is to define generalized solutions to deal with problems that potentially do not have solutions. A classical example is the introduction of least squares solutions via the normal equations associated with a possibly infeasible system of linear equations. In this paper, we introduce a "normal problem" associated with finding a zero of the sum of two maximally monotone… ▽ More A common theme in mathematics is to define generalized solutions to deal with problems that potentially do not have solutions. A classical example is the introduction of least squares solutions via the normal equations associated with a possibly infeasible system of linear equations. In this paper, we introduce a "normal problem" associated with finding a zero of the sum of two maximally monotone operators. If the original problem admits solutions, then the normal problem returns this same set of solutions. The normal problem may yield solutions when the original problem does not admit any; furthermore, it has attractive variational and duality properties. Several examples illustrate our theory. △ Less

Submitted 7 June, 2013; originally announced June 2013.

MSC Class: 47H05; 47H09; 90C46 (Primary) 49M27; 49N15; 90C25 (Secondary)

A Derivative-Free CoMirror Algorithm

Authors: Heinz H. Bauschke, Warren L. Hare, Walaa M. Moursi

Abstract: We consider $\min\{f(x):g(x) \le 0, ~x\in X\},$ where $X$ is a compact convex subset of $\RR^m$, and $f$ and $g$ are continuous convex functions defined on an open neighbourhood of $X$. We work in the setting of derivative-free optimization, assuming that $f$ and $g$ are available through a black-box that provides only function values for a lower-$\mathcal{C}^2$ representation of the functions. We… ▽ More We consider $\min\{f(x):g(x) \le 0, ~x\in X\},$ where $X$ is a compact convex subset of $\RR^m$, and $f$ and $g$ are continuous convex functions defined on an open neighbourhood of $X$. We work in the setting of derivative-free optimization, assuming that $f$ and $g$ are available through a black-box that provides only function values for a lower-$\mathcal{C}^2$ representation of the functions. We present a derivative-free optimization variant of the $\eps$-comirror algorithm \cite{BBTGBT2010}. Algorithmic convergence hinges on the ability to accurately approximate subgradients of lower-$\mathcal{C}^2$ functions, which we prove is possible through linear interpolation. We provide convergence analysis that quantifies the difference between the function values of the iterates and the optimal function value. We find that the DFO algorithm we develop has the same convergence result as the original gradient-based algorithm. We present some numerical testing that demonstrate the practical feasibility of the algorithm, and conclude with some directions for further research. △ Less

Submitted 23 October, 2012; originally announced October 2012.

MSC Class: 90C25; 90C56 (Primary) 49M30; 65K10 (Secondary)

Attouch-Théra duality revisited: paramonotonicity and operator splitting

Authors: Heinz H. Bauschke, Radu I. Bot, Warren L. Hare, Walaa M. Moursi

Abstract: The problem of finding the zeros of the sum of two maximally monotone operators is of fundamental importance in optimization and variational analysis. In this paper, we systematically study Attouch-Théra duality for this problem. We provide new results related to Passty's parallel sum, to Eckstein and Svaiter's extended solution set, and to Combettes' fixed point description of the set of primal s… ▽ More The problem of finding the zeros of the sum of two maximally monotone operators is of fundamental importance in optimization and variational analysis. In this paper, we systematically study Attouch-Théra duality for this problem. We provide new results related to Passty's parallel sum, to Eckstein and Svaiter's extended solution set, and to Combettes' fixed point description of the set of primal solutions. Furthermore, paramonotonicity is revealed to be a key property because it allows for the recovery of all primal solutions given just one arbitrary dual solution. As an application, we generalize the best approximation results by Bauschke, Combettes and Luke [J. Approx. Theory 141 (2006), 63-69] from normal cone operators to paramonotone operators. Our results are illustrated through numerous examples. △ Less

Submitted 21 October, 2011; originally announced October 2011.

MSC Class: 41A50; 47H05; 90C25 (Primary) 47J05; 47H09; 47N10; 49M27; 49N15; 65K05; 65K10; 90C46 (Secondary)