-
Provably Faster Gradient Descent via Long Steps
Abstract: This work establishes new convergence guarantees for gradient descent in smooth convex optimization via a computer-assisted analysis technique. Our theory allows nonconstant stepsize policies with frequent long steps potentially violating descent by analyzing the overall effect of many iterations at once rather than the typical one-iteration inductions used in most first-order method analyses. We… ▽ More
Submitted 4 February, 2024; v1 submitted 12 July, 2023; originally announced July 2023.
Comments: 20 pages
-
Some Primal-Dual Theory for Subgradient Methods for Strongly Convex Optimization
Abstract: We consider (stochastic) subgradient methods for strongly convex but potentially nonsmooth non-Lipschitz optimization. We provide new equivalent dual descriptions (in the style of dual averaging) for the classic subgradient method, the proximal subgradient method, and the switching subgradient method. These equivalences enable $O(1/T)$ convergence guarantees in terms of both their classic primal g… ▽ More
Submitted 26 June, 2024; v1 submitted 26 May, 2023; originally announced May 2023.
Comments: 24 pages, major revision shortened the write-up and unified the analysis to be done just once in a single "super" setting
-
Gauges and Accelerated Optimization over Smooth and/or Strongly Convex Sets
Abstract: We consider feasibility and constrained optimization problems defined over smooth and/or strongly convex sets. These notions mirror their popular function counterparts but are much less explored in the first-order optimization literature. We propose new scalable, projection-free, accelerated first-order methods in these settings. Our methods avoid linear optimization or projection oracles, only us… ▽ More
Submitted 31 March, 2023; v1 submitted 9 March, 2023; originally announced March 2023.
Comments: 22pages (32pages with references and appendix)
MSC Class: 90C25; 90C52
-
Limiting Behaviors of Nonconvex-Nonconcave Minimax Optimization via Continuous-Time Systems
Abstract: Unlike nonconvex optimization, where gradient descent is guaranteed to converge to a local optimizer, algorithms for nonconvex-nonconcave minimax optimization can have topologically different solution paths: sometimes converging to a solution, sometimes never converging and instead following a limit cycle, and sometimes diverging. In this paper, we study the limiting behaviors of three classic min… ▽ More
Submitted 4 March, 2021; v1 submitted 20 October, 2020; originally announced October 2020.
MSC Class: 65K05; 65K10; 90C26; 90C15; 90C30
-
The Landscape of the Proximal Point Method for Nonconvex-Nonconcave Minimax Optimization
Abstract: Minimax optimization has become a central tool in machine learning with applications in robust optimization, reinforcement learning, GANs, etc. These applications are often nonconvex-nonconcave, but the existing theory is unable to identify and deal with the fundamental difficulties this poses. In this paper, we study the classic proximal point method (PPM) applied to nonconvex-nonconcave minimax… ▽ More
Submitted 1 April, 2021; v1 submitted 15 June, 2020; originally announced June 2020.
Comments: Notably updated version that connects our theory with that of Attouch and Wets from the 80s and notably expands on our first posting to apply to generic minimax problems (rather than requiring bilinear interaction)
MSC Class: 65K05; 65K10; 90C26; 90C15; 90C30
-
arXiv:1712.04104 [pdf, ps, other]
Convergence Rates for Deterministic and Stochastic Subgradient Methods Without Lipschitz Continuity
Abstract: We extend the classic convergence rate theory for subgradient methods to apply to non-Lipschitz functions. For the deterministic projected subgradient method, we present a global $O(1/\sqrt{T})$ convergence rate for any convex function which is locally Lipschitz around its minimizers. This approach is based on Shor's classic subgradient analysis and implies generalizations of the standard converge… ▽ More
Submitted 26 February, 2018; v1 submitted 11 December, 2017; originally announced December 2017.
Comments: Update 2/26/18: Major revision improving the convergence results to no longer need an exponential upper bound on function growth in the convex case. Now local Lipschitz continuity around a minimizer suffices for a global convergence rate. Update 12/21/17: Added three more references on weakening strong convexity and minorly changed some wording. 16 pages
MSC Class: 65K05; 65K10; 90C25; 90C15; 90C30
-
Proximally Guided Stochastic Subgradient Method for Nonsmooth, Nonconvex Problems
Abstract: In this paper, we introduce a stochastic projected subgradient method for weakly convex (i.e., uniformly prox-regular) nonsmooth, nonconvex functions---a wide class of functions which includes the additive and convex composite classes. At a high-level, the method is an inexact proximal point iteration in which the strongly convex proximal subproblems are quickly solved with a specialized stochasti… ▽ More
Submitted 17 September, 2018; v1 submitted 11 July, 2017; originally announced July 2017.
Comments: Updated 9/17/2018: Major Revision -added high probability bounds, improved convergence analysis in general, new experimental results. Updated 7/26/2017: Added references to introduction and a couple simple extensions as Sections 3.2 and 4. Updated 8/23/2017: Added NSF acknowledgements. Updated 10/16/2017: Added experimental results
MSC Class: 65K05; 65K10; 90C26; 90C15; 90C30
-
Dual-Based Approximation Algorithms for Cut-Based Network Connectivity Problems
Abstract: We consider a variety of NP-Complete network connectivity problems. We introduce a novel dual-based approach to approximating network design problems with cut-based linear programming relaxations. This approach gives a $3/2$-approximation to Minimum 2-Edge-Connected Spanning Subgraph that is equivalent to a previously proposed algorithm. One well-studied branch of network design models ad hoc netw… ▽ More
Submitted 20 July, 2017; v1 submitted 23 August, 2015; originally announced August 2015.
Comments: 7/20/2017: Changed Title to be more accurate. Improved presentation and clarity throughout the document (i.e. adding references and fixing typos)
ACM Class: G.1.6; G.2.2; F.2.2