Bounding the expected run-time of nonconvex optimization with early stop**
Authors:
Thomas Flynn,
Kwang Min Yu,
Abid Malik,
Nicolas D'Imperio,
Shinjae Yoo
Abstract:
This work examines the convergence of stochastic gradient-based optimization algorithms that use early stop** based on a validation function. The form of early stop** we consider is that optimization terminates when the norm of the gradient of a validation function falls below a threshold. We derive conditions that guarantee this stop** rule is well-defined, and provide bounds on the expecte…
▽ More
This work examines the convergence of stochastic gradient-based optimization algorithms that use early stop** based on a validation function. The form of early stop** we consider is that optimization terminates when the norm of the gradient of a validation function falls below a threshold. We derive conditions that guarantee this stop** rule is well-defined, and provide bounds on the expected number of iterations and gradient evaluations needed to meet this criterion. The guarantee accounts for the distance between the training and validation sets, measured with the Wasserstein distance. We develop the approach in the general setting of a first-order optimization algorithm, with possibly biased update directions subject to a geometric drift condition. We then derive bounds on the expected running time for early stop** variants of several algorithms, including stochastic gradient descent (SGD), decentralized SGD (DSGD), and the stochastic variance reduced gradient (SVRG) algorithm. Finally, we consider the generalization properties of the iterate returned by early stop**.
△ Less
Submitted 22 July, 2020; v1 submitted 20 February, 2020;
originally announced February 2020.
