Stochastic gradient algorithms have been the main focus of large-scale learning problems and they led to impressive successes in machine learning. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose a new adaptive learning rate algorithm, which utilizes curvature information for automatically tuning the learning rates. The information about the element-wise curvature of the loss function is estimated from the local statistics of the stochastic first order gradients. We further propose a new variance reduction technique which automatically reduces the variance of noise in the local gradient estimates to speed up the convergence. In our preliminary experiments with deep neural networks, we obtained better performance compared to the popular stochastic gradient algorithms.
Gulcehre and Bengio REF propose an adaptive learning rate algorithm, AdaSecant, that utilizes the root mean square statistics and variance of the gradients.
ADASECANT: Robust Adaptive Secant Method for Stochastic Gradient
ArXiv
ArXiv
Mathematics Computer Science