Stochastic gradient methods (SGMs) have been extensively used for solving stochastic problems or large-scale machine learning problems. Recent works employ various techniques to improve the convergence rate of SGMs for both convex and nonconvex cases. Most of them require a large number of samples in some or all iterations of the improved SGMs. In this paper, we propose a new SGM, named PStorm, for solving nonconvex nonsmooth stochastic problems. With a momentum-based variance reduction technique, PStorm can achieve a near-optimal complexity result (\varepsilon^{-3}) to produce a stochastic \varepsilon-stationary solution, if a mean-squared smoothness condition holds  . Different from existing near-optimal methods, PStorm requires  only one or O(1) samples in every update. With this property, PStorm can be applied to online learning problems that favor real-time decisions based on one or O(1) new observations. In addition, for large-scale machine learning problems, PStorm can generalize better by small-batch training than other near-optimal methods that require large-batch training and the vanilla SGM, as we demonstrate on training a sparse fully-connected  neural network.
Stochastic gradient methods (SGMs) have been extensively used for solving stochastic problems or large-scale machine learning problems. Recent works employ various techniques to improve the convergence rate of SGMs for both convex and nonconvex cases. Most of them require a large number of samples in some or all iterations of the improved SGMs. In this paper, we propose a new SGM, named PStorm, for solving nonconvex nonsmooth stochastic problems. With a momentum-based variance reduction technique, PStorm can achieve the optimal complexity result O (\varepsilon^{-3}) to produce a stochastic \varepsilon-stationary solution, if a mean-squared smoothness condition holds and \Theta(\varepsilon^{-1 . Different from existing optimal methods, PStorm can still achieve a near-optimal complexity result (\varepsilon^{-3 only one or O(1) samples in every update. With this property, PStorm can be applied to online learning problems that favor real-time decisions based on one or O(1) new observations. In addition, for large-scale machine learning problems, PStorm can generalize better by small-batch training than other optimal methods that require large-batch training and the vanilla SGM, as we demonstrate on training a sparse fully-connected neural network and a sparse convolutional neural network.
R
a near-optimal complexity result
the optimal complexity result O
A
and \Theta(\varepsilon^{-1
R
near-optimal
optimal
R
requires
can still achieve a near-optimal complexity result
A
(\varepsilon^{-3
R
near-optimal
optimal
A
neural network and a sparse convolutional