Stochastic gradient descent (SGD) is of fundamental importance in deep learning. Despite its simplicity, elucidating its efficacy remains challenging. Conventionally, the success of SGD is attributed to thestochastic gradient noise (SGN) incurred in the training process. Based on this general consensus, SGD is frequently treated and analyzed as the Euler-Maruyama discretization of astochastic differential equation (SDE ) driven by either Brownian or L\'evy stable motion. In this study, we argue that SGN is neither Gaussian nor  stable. Instead, inspired by the long-time correlation emerging in  SGN series, we propose that SGD can be viewed as a discretization of an SDE driven by \textit{fractional Brownian motion (FBM). Accordingly, the different convergence behavior of SGD dynamics is well grounded . Moreover, the first passage time of an SDE driven by FBM is approximately derived. This indicates a lower escaping rate for a larger Hurst parameter, and thus SGD stays longer in flat minima. This happens to coincide with the well-known phenomenon that SGD favors flat minima that generalize well. Four groups of experiments are conducted to validate our conjecture, and it is demonstrated that long-range memory effects persist across various model architectures, datasets, and training strategies. Our study opens up a new perspective and may contribute to a better understanding of SGD.
Stochastic gradient descent (SGD) is of fundamental importance in deep learning. Despite its simplicity, elucidating its efficacy remains challenging. Conventionally, the success of SGD is ascribed to the stochastic gradient noise (SGN) incurred in the training process. Based on this consensus, SGD is frequently treated and analyzed as the Euler-Maruyama discretization of stochastic differential equations (SDEs ) driven by either Brownian or Levy stable motion. In this study, we argue that SGN is neither Gaussian nor Levy stable. Instead, inspired by the short-range correlation emerging in the SGN series, we propose that SGD can be viewed as a discretization of an SDE driven by \textit{fractional Brownian motion (FBM). Accordingly, the different convergence behavior of SGD dynamics is well-grounded . Moreover, the first passage time of an SDE driven by FBM is approximately derived. The result suggests a lower escaping rate for a larger Hurst parameter, and thus SGD stays longer in flat minima. This happens to coincide with the well-known phenomenon that SGD favors flat minima that generalize well. Extensive experiments are conducted to validate our conjecture, and it is demonstrated that short-range memory effects persist across various model architectures, datasets, and training strategies. Our study opens up a new perspective and may contribute to a better understanding of SGD.
R
Stochastic gradient descent
Stochastic gradient descent
D
attributed to the
R
stochastic gradient noise
ascribed to the stochastic gradient noise
D
general
D
a
D
stochastic differential equation
R
(SDE
stochastic differential equations (SDEs
R
L\'evy
Levy
A
Levy
R
long-time
short-range
A
the
R
fractional Brownian motion
fractional Brownian motion
R
well grounded
well-grounded
R
This indicates
The result suggests
R
Four groups of
Extensive
R
long-range
short-range