Showing 1–2 of 2 results for author: de Ruisselet, T B

Adaptive First- and Second-Order Algorithms for Large-Scale Machine Learning

Authors: Sanae Lotfi, Tiphaine Bonniot de Ruisselet, Dominique Orban, Andrea Lodi

Abstract: In this paper, we consider both first- and second-order techniques to address continuous optimization problems arising in machine learning. In the first-order case, we propose a framework of transition from deterministic or semi-deterministic to stochastic quadratic regularization methods. We leverage the two-phase nature of stochastic optimization to propose a novel first-order algorithm with ada… ▽ More In this paper, we consider both first- and second-order techniques to address continuous optimization problems arising in machine learning. In the first-order case, we propose a framework of transition from deterministic or semi-deterministic to stochastic quadratic regularization methods. We leverage the two-phase nature of stochastic optimization to propose a novel first-order algorithm with adaptive sampling and adaptive step size. In the second-order case, we propose a novel stochastic damped L-BFGS method that improves on previous algorithms in the highly nonconvex context of deep learning. Both algorithms are evaluated on well-known deep learning datasets and exhibit promising performance. △ Less

Submitted 29 November, 2021; originally announced November 2021.

Comments: 29 pages, 8 figures. arXiv admin note: text overlap with arXiv:2012.05783

MSC Class: 68T07; 90C15; 90C30; 90C53 ACM Class: G.1.6; G.3; G.4; I.2.6

Stochastic Damped L-BFGS with Controlled Norm of the Hessian Approximation

Authors: Sanae Lotfi, Tiphaine Bonniot de Ruisselet, Dominique Orban, Andrea Lodi

Abstract: We propose a new stochastic variance-reduced damped L-BFGS algorithm, where we leverage estimates of bounds on the largest and smallest eigenvalues of the Hessian approximation to balance its quality and conditioning. Our algorithm, VARCHEN, draws from previous work that proposed a novel stochastic damped L-BFGS algorithm called SdLBFGS. We establish almost sure convergence to a stationary point a… ▽ More We propose a new stochastic variance-reduced damped L-BFGS algorithm, where we leverage estimates of bounds on the largest and smallest eigenvalues of the Hessian approximation to balance its quality and conditioning. Our algorithm, VARCHEN, draws from previous work that proposed a novel stochastic damped L-BFGS algorithm called SdLBFGS. We establish almost sure convergence to a stationary point and a complexity bound. We empirically demonstrate that VARCHEN is more robust than SdLBFGS-VR and SVRG on a modified DavidNet problem -- a highly nonconvex and ill-conditioned problem that arises in the context of deep learning, and their performance is comparable on a logistic regression problem and a nonconvex support-vector machine problem. △ Less

Submitted 10 December, 2020; originally announced December 2020.

Comments: 14 pages, 4 figures

Report number: Cahier du GERAD G-2020-52 MSC Class: 68T07; 90C15; 90C30; 90C53 ACM Class: G.1.6; G.3; G.4; I.2.6