Showing 1–2 of 2 results for author: Ahmed, M O
-
Stop Wasting My Gradients: Practical SVRG
Authors:
Reza Babanezhad,
Mohamed Osama Ahmed,
Alim Virani,
Mark Schmidt,
Jakub Konečný,
Scott Sallinen
Abstract:
We present and analyze several strategies for improving the performance of stochastic variance-reduced gradient (SVRG) methods. We first show that the convergence rate of these methods can be preserved under a decreasing sequence of errors in the control variate, and use this to derive variants of SVRG that use growing-batch strategies to reduce the number of gradient calculations required in the…
▽ More
We present and analyze several strategies for improving the performance of stochastic variance-reduced gradient (SVRG) methods. We first show that the convergence rate of these methods can be preserved under a decreasing sequence of errors in the control variate, and use this to derive variants of SVRG that use growing-batch strategies to reduce the number of gradient calculations required in the early iterations. We further (i) show how to exploit support vectors to reduce the number of gradient computations in the later iterations, (ii) prove that the commonly-used regularized SVRG iteration is justified and improves the convergence rate, (iii) consider alternate mini-batch selection strategies, and (iv) consider the generalization error of the method.
△ Less
Submitted 5 November, 2015;
originally announced November 2015.
-
Non-Uniform Stochastic Average Gradient Method for Training Conditional Random Fields
Authors:
Mark Schmidt,
Reza Babanezhad,
Mohamed Osama Ahmed,
Aaron Defazio,
Ann Clifton,
Anoop Sarkar
Abstract:
We apply stochastic average gradient (SAG) algorithms for training conditional random fields (CRFs). We describe a practical implementation that uses structure in the CRF gradient to reduce the memory requirement of this linearly-convergent stochastic gradient method, propose a non-uniform sampling scheme that substantially improves practical performance, and analyze the rate of convergence of the…
▽ More
We apply stochastic average gradient (SAG) algorithms for training conditional random fields (CRFs). We describe a practical implementation that uses structure in the CRF gradient to reduce the memory requirement of this linearly-convergent stochastic gradient method, propose a non-uniform sampling scheme that substantially improves practical performance, and analyze the rate of convergence of the SAGA variant under non-uniform sampling. Our experimental results reveal that our method often significantly outperforms existing methods in terms of the training objective, and performs as well or better than optimally-tuned stochastic gradient methods in terms of test error.
△ Less
Submitted 16 April, 2015;
originally announced April 2015.
