T HE LIMITING DYNAMICS OF SGD: MODIFIED LOSS , PHASE SPACE OSCILLATIONS , ANOMALOUS DIFFUSION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work we explore the limiting dynamics of deep neural networks trained with stochastic gradient descent (SGD).
As observed previously, long after performance has converged, networks continue to move through parameter space by a process of anomalous diffusion in which distance travelled grows as a power law in the number of gradient updates with a nontrivial exponent.
We reveal an intricate interaction between the hyperparameters of optimization, the structure in the gradient noise, and the Hessian matrix at the end of training that explains this anomalous diffusion.
To build this understanding, we ﬁrst derive a continuoustime model for SGD with ﬁnite learning rates and batch sizes as an underdamped
Langevin equation.
We study this equation in the setting of linear regression, where we can derive exact, analytic expressions for the phase space dynamics of the parameters and their instantaneous velocities from initialization to stationarity.
Using the Fokker-Planck equation, we show that the key ingredient driving these dynamics is not the original training loss, but rather the combination of a modiﬁed loss, which implicitly regularizes the velocity, and probability currents, which cause oscillations in phase space.
We identify qualitative and quantitative predictions of this theory in the dynamics of a ResNet-18 model trained on ImageNet.
Through the lens of statistical physics, we uncover a mechanistic origin for the anomalous limiting dynamics of deep neural networks trained with SGD.
I NTRODUCTION
Deep neural networks have demonstrated remarkable generalization across a variety of datasets and tasks.
Essential to their success has been a collection of good practices on how to train these models with stochastic gradient descent (SGD).
Yet, despite their importance, these practices are mainly based on heuristic arguments and trial and error search.
Without a general theory connecting the hyperparameters of optimization, the architecture of the network and the geometry of the dataset, theory-driven design of deep learning systems is impossible.
Existing theoretical works studying this interaction have leveraged the random structure of neural networks at initialization [1, 2, 3] and in their inﬁnite width limits in order to study their dynamics [4, 5, 6, 7, 8].
Here we take a different approach and study the training dynamics of pre-trained networks that are ready to be used for inference.
By leveraging the mathematical structures found at the end of training, we uncover an intricate interaction between the hyperparameters of optimization, the structure in the gradient noise, and the Hessian matrix that corroborates previously identiﬁed empirical behavior such as anomalous limiting dynamics.
Not only is understanding the limiting dynamics of SGD a critical stepping stone to building a complete theory for the learning dynamics of neural networks, but recently there have been a series of works demonstrating that the performance of pre-trained networks can be improved through averaging and ensembling [9, 10, 11].
Combining empirical exploration and theoretical tools from statistical physics, we identify and uncover a mechanistic explanation for the limiting dynamics of neural networks trained with SGD.
D IFFUSIVE B EHAVIOR IN THE L IMITING D YNAMICS OF SGD
A network that has converged in performance will continue to move through parameter space [12, 13, 14, 15].
To demonstrate this behavior, we resume training of pre-trained convolutional networks while tracking the network trajectory through parameter space.
Let θ ∗ ∈ R m be the parameter vector for a pre-trained network and θ k ∈ R m be the parameter vector after k steps of resumed training.
We track two metrics of the training trajectory, namely the local parameter displacement δ k between consecutive steps, and the global displacement ∆ k after k steps from the pre-trained initialization: …
VGG-16 with BN
VGG-11 with BN
Step
ResNetResNet-
VGGc = 1.
c
= 1.
c
= 1.
c = 1.
Step c=1.
x10 4 3 4 0.
[Equation]
[Figure]
[Equation]
[Figure] [Equation] As shown in Fig.
1, neither of these differences converge to zero across a variety of architectures, indicating that despite performance convergence, the networks continue to move through parameter space, both locally and glob-
The norm local displacement (cid:107) δ [Equation]
near is instantaneous
This observation is quite similar to the “equilibrium" phenomenon or “constant angular update" observed in Li et al.
[17] and Wan et al.
[13] respectively.
However, these works only studied the displacement for parameters immediately preceding a normalization layer.
The constant instantaneous speed behavior we observe is for all parameters in the model and is even present in models without normalization layers.
While the squared norm of the local displacement is essentially constant, the squared norm of the global displacement (cid:107) ∆ k (cid:107) 22 is monotonically growing for all net- works, implying even once trained, the network continues to diverge from where it has been.
Indeed Fig.
1 indicates a power law relationship between global displacement and number of steps, given by (cid:107) ∆ k (cid:107) 22 ∝ k c .
As we’ll see in section 8, this relationship is indicative of anomalous diffusion where c corresponds to the anomalous diffusion exponent.
Standard Brownian motion corresponds to c = 1 .
Similar observation were made by Baity-Jesi et al.
[14] who noticed distinct phases of the training trajectory evident in the dynamics of the global displacement and Chen et al.
[15] who found that the exponent of diffusion changes through the course of training.
A parallel observation is given by Hoffer et al.
[18] for the beginning of training, where they measure the global displacement from the initialization of an untrained network and observe a rate ∝ log( k ) , a form of ultra-slow diffusion.
These empirical observations raise the natural questions, where is the network moving to and why?
To answer these questions we will build a diffusion based theory of SGD, study these dynamics in the setting of linear regression, and use lessons learned in this fundamental setting to understand the limiting dynamics of neural networks.
R ELATED W ORK
There is a long line of literature studying both theoretically and empirically the learning dynamics of deep neural networks trained with SGD.
Our analysis and experiments build upon this literature.
Continuous models for SGD.
Many works consider how to improve the classic gradient ﬂow model for SGD to more realistically reﬂect momentum [19], discretization due to ﬁnite learning rates [20, 21], and stochasticity due to random batches [22, 23].
One line of work has studied the dynamics of networks in their inﬁnite width limits through dynamical mean ﬁeld theory [24, 25, 26, 27], while a different approach has used stochastic differential equations (SDEs) to model SGD directly, the approach we take in this work. However, recently, the validity of this approach has been questioned.
The main argument, as nicely explained in Yaida [28], is that most SDE approximations simultaneously assume that ∆ t → 0 + , while maintaining that the learning rate η = ∆ t is ﬁnite.
The works Simsekli et al.
[29] and Li et al. [30] have questioned the correctness of the using the central limit theorem (CLT) to model the gradient noise as Gaussian, arguing respectively that the heavy-tailed structure in the gradient noise and the weak dependence between batches leads the CLT to break down.
In our work, we maintain the CLT assumption holds, which we discuss fur- ther in appendix A, but importantly we avoid the pitfalls of many previous SDE approximations by simultaneously modeling the effect of ﬁnite learning rates and stochasticity.
Limiting dynamics.
A series of works have applied SDE models of SGD to study the limiting dynamics of neural networks.
In the seminal work by Mandt et al.
[31], the limiting dynamics were modeled with a multivariate Ornstein-Uhlenbeck process by combining a ﬁrst-order SDE model for SGD with assumptions on the geometry of the loss and covariance matrix for the gradient noise.
This analysis was extended by Jastrz˛ebski et al.
[12] through additional assumptions on the covariance matrix to gain tractable insights and applied by Ali et al.
[32] to the simpler setting of linear regression, which has a quadratic loss.
A different approach was taken by Chaudhari and Soatto [33], which did not formulate the dynamics as an OU process, nor assume directly a structure on the loss or gradient noise.
Rather, this analysis studied the same ﬁrst-order SDE via the Fokker-Planck equation to propose the existence of a modiﬁed loss and probability currents driving the limiting dynamics, but did not provide explicit expressions.
Our analysis deepens and combines ideas from all these works, where our key insight is to lift the dynamics into phase space.
By studying the dynamics of the parameters and their velocities, and by applying the analysis ﬁrst in the setting of linear regression where assumptions are provably true, we are able to identify analytic expressions and explicit insights which lead to concrete predictions and testable hypothesis.
Stationary dynamics.
A different line of work avoids modeling the limiting dynamics of SGD with an SDE and instead chooses to leverage the property of stationarity.
These works [28, 34, 35, 36] assume that eventually the probability distribution governing the model parameters reaches stationarity such that the discrete SGD process is simply sampling from this distribution.
Yaida [28] used this approach to derive ﬂuctuation-dissipation relations that link measurable quantities of the parameters and hyperparameters of SGD.
Liu et al.
[35] used this approach to derive properties for the stationary distribution of SGD with a quadratic loss.
Similar to our analysis, this work identiﬁes that the stationary distribution for the parameters reﬂects a modiﬁed loss function dependent on the relationship between the covariance matrix of the gradient noise and the Hessian matrix for the original loss.
Empirical exploration.
Another set of works analyzing the limiting dynamics of SGD has taken a purely empirical approach.
Building on the intuition that ﬂat minima generalize better than sharp minima, Keskar et al.
[37] demonstrated empirically that the hyperparameters of optimization inﬂuence the eigenvalue spectrum of the Hessian matrix at the end of training.
Many subsequent works have studied the Hessian eigenspectrum during and at the end of training.
Jastrz˛ebski et al.
[38], Cohen et al.
[39] studied the dynamics of the top eigenvalues during training. Sagun et al. [40], Papyan [41], Ghorbani et al. [42] demonstrated the spectrum has a bulk of values near zero plus a small number of larger outliers.
Gur-Ari et al.
[43] demonstrated that the learning dynamics are constrained to the subspace spanned by the top eigenvectors, but found no special properties of the dynamics within this subspace.
In our work we also determine that the top eigensubspace of the Hessian plays a crucial role in the limiting dynamics and by projecting the dynamics into this subspace in phase space, we see that the motion is not random, but consists of incoherent oscillations leading to anomalous diffusion.
M ODELING SGD AS AN U NDERDAMPED L ANGEVIN E QUATION
Following the route of previous works [31, 12, 33] studying the limiting dynamics of neural networks, we ﬁrst seek to model SGD as a continuous stochastic process.
We consider a network parameterized by θ ∈ R m , a training dataset { x 1 , . . .
, x N } of size N , and a training loss
L ( θ ) =
1 N (cid:80) Ni =1 (cid:96) ( θ, x i ) with corresponding gradient g ( θ ) =
∂ L ∂θ .
The state of the network at the k th step of training is deﬁned by the position vector θ k and velocity vector v k of the same dimension.
The gradient descent update with learning rate η , momentum β , and weight decay λ is given by
[Equation]
where we initialize the network such that v 0 = 0 and θ 0 is the parameter initialization.
In order to understand the dynamics of the network through position and velocity space, which we will refer to as phase space , we express these discrete recursive equations as the discretization of some unknown ordinary differential equation (ODE), sometimes referred to as a modiﬁed equation as in [44, 20].
While this ODE models the gradient descent process even at ﬁnite learning rates, it fails to account for the stochasticity introduced by choosing a random batch B of size S drawn uniformly from the set of N training points.
This sampling yields the stochastic gradient g B ( θ )
= 1 S (cid:80)
i ∈B ∇ (cid:96) ( θ, x i ) .
To model this effect, we make the following assumption:
Assumption 1 (CLT) .
We assume the batch gradient is a noisy version of the true gradient such that g B ( θ ) − g ( θ ) is a Gaussian random variable with mean 0 and covariance 1 S Σ( θ ) .
Incorporating this model of stochastic gradients into the previous ﬁnite difference equation and applying the stochastic counterparts to Euler discretizations, results in the standard drift-diffusion stochastic differential equation (SDE), referred to as an underdamped Langevin equation ,
[Equation]
where W t is a standard Wiener process.
This is the continuous model we will study in this work:
Assumption 2 (SDE) .
We assume the underdamped Langevin equation (3) accurately models the trajectory of the network driven by SGD through phase space such that θ ( ηk )
≈ θ k
and v ( ηk )
≈ v k .
See appendix A for further discussion on the nuances of modeling SGD with an SDE.
L INEAR R EGRESSION WITH SGD IS AN O RNSTEIN -U HLENBECK P ROCESS
Equipped with a model for SGD, we seek to understand its dynamics in the fundamental setting of linear regression, one of the few cases where we have a complete model for the interaction of the dataset, architecture, and optimizer.
Let X ∈ R N ×
d
be the input data, Y ∈ R N be the output labels, and θ ∈ R
d be our vector of regression coefﬁcients.
The least squares loss is the convex quadratic loss L ( θ ) = 12 N (cid:107)
Y − Xθ (cid:107) 2 with gradient g ( θ ) =
Hθ − b , where H = X (cid:124) XN and b = X (cid:124) YN .
Plugging this expression for the gradient into the underdamped Langevin equation (3), and rearranging terms, results in the multivariate Ornstein-Uhlenbeck (OU) process,
[Equation]
[Equation]
[Figure]
We train a linear network to perform regression on the
CIFAR-10 dataset by using an MSE loss on the one-hot encoding of the labels.
We compute the hessian of the loss, as well as its top eigenvectors.
The position and velocity trajectories are projected onto the ﬁrst eigenvector of the hessian and visualized in black.
The theoretically derived mean, equation (5), is shown in red.
The top and bottom panels demonstrate the effect of varying momentum on the oscillation mode.
where A and D are the drift and diffusion matrices respectively, κ = S (1 − β 2 ) is an inverse temperature constant, and µ = ( H + λI ) − 1 b is the ridge regression solution.
The solution to an OU process is a Gaussian process.
By solving for the temporal dynamics of the ﬁrst and second moments of the process, we can obtain an analytic expression for the trajectory at any time t .
In particular, we can decompose the trajectory as the sum of a deterministic and stochastic component deﬁned by the ﬁrst and second moments respectively.
Deterministic component.
Using the form of A we can decompose the expectation as a sum of harmonic oscillators in the eigenbasis { q 1 , . . .
, q m } of the Hessian,
[Equation]
Here the coefﬁcients a i ( t ) and b i ( t ) depend on the optimization hyperparameters η, β, λ, S and the respective eigenvalue of the Hessian ρ i as further explained in appendix
F.
We verify this expression nearly perfectly matches empirics on complex datasets under various hyperparameter settings as shown in Fig.
Stochastic component.
The cross-covariance of the process between two points in time t ≤ s , is
[Equation]
where B solves the Lyapunov equation AB + BA (cid:124)
= 2 D .
In order to gain analytic expressions for
B in terms of the optimization hyperparameters, eigendecomposition of the Hessian, and covariance of the gradient noise, we must introduce the following assumption:
Assumption 3 (Simultaneously Diagonalizable) .
We assume the covariance of the gradient noise is spatially independent Σ( θ )
= Σ and commutes with the Hessian H Σ = Σ H , therefore sharing a common eigenbasis.
U NDERSTANDING S TATIONARITY VIA THE F OKKER -P LANCK E QUATION
The OU process is unique in that it is one of the few SDEs which we can solve exactly.
As shown in section 5, we were able to derive exact expressions for the dynamics of linear regression trained with
SGD from initialization to stationarity by simply solving for the ﬁrst and second moments.
While the expression for the ﬁrst moment provides an understanding of the intricate oscillatory relationship in the deterministic component of the process, the second moment, driving the stochastic component, is much more opaque.
An alternative route to solving the OU process that potentially provides more insight is the Fokker-Planck equation.
The Fokker-Planck (FP) equation is a PDE describing the time evolution for the probability distribution of a particle governed by Langevin dynamics.
For an arbitrary potential Φ and diffusion matrix
D , the Fokker-Planck equation (under an Itô integration prescription) is
[Equation]
where p represents the time-dependent probability distribution, and J is a vector ﬁeld commonly referred to as the probability current.
The FP equation is especially useful for explicitly solving for the stationary solution, assuming one exists, of the Langevin dynamics.
The stationary solution p ss by deﬁnition obeys ∂ t p ss = 0 or equivalently ∇ · J ss = 0 .
From this second deﬁnition we see that there are two distinct settings of stationarity: detailed balance when J ss = 0 everywhere, or broken detailed balance when ∇ · J ss = 0 and J ss (cid:54) = 0 .
For a general OU process, the potential is a convex quadratic function Φ( x ) = x (cid:124)
Ax deﬁned by the drift matrix A .
When the diffusion matrix is isotropic ( D ∝ I ) and spatially independent ( ∇ · D = 0 )
the resulting stationary solution is a Gibbs distribution p ss ( x ) ∝ e − κ Φ( x ) determined by the original loss Φ( x ) and is in detailed balance.
Lesser known properties of the OU process arise when the diffusion matrix is anisotropic or spatially dependent [45, 46].
In this setting the solution is still a Gaussian process, but the stationary solution, if it exists, is no longer deﬁned by the Gibbs distribution of the original loss Φ( x ) , but actually a modiﬁed loss Ψ( x ) .
Furthermore, the stationary solution may be in broken detailed balance leading to a non-zero probability current
J ss ( x ) .
Depending on the relationship between the drift matrix A and the diffusion matrix D the resulting dynamics of the OU process can have very nontrivial behavior.
In the setting of linear regression, anisotropy in the data distribution will lead to anisotropy in the gradient noise and thus an anisotropic diffusion matrix.
This implies that for most datasets we should expect that the SGD trajectory is not driven by the original least squares loss, but by a modiﬁed loss and converges to a stationary solution with broken detailed balance, as predicted by Chaudhari and
Soatto [33].
Using the explicit expressions for the drift A and diffusion D matrices we can compute analytically the modiﬁed loss and stationary probability current,
[Equation]
where Q is a skew-symmetric matrix and U is a positive deﬁnite matrix deﬁned as,
[Equation]
These new fundamental matrices, Q and U , relate to the original drift A and diffusion D matrices through the unique decomposition
A
=
( D + Q ) U , introduced by Ao [47] and Kwon et al.
Using this decomposition we can easily show that B
= U − 1 solves the Lyapunov equation and indeed the stationary solution p ss is the Gibbs distribution deﬁned by the modiﬁed loss Ψ( θ, v ) in equation (8).
Further, the stationary cross-covariance solved in section 5 reﬂects the oscillatory dynamics introduced by the stationary probability currents J ss ( θ, v ) in equation (8).
Taken together, we gain the intuition that the limiting dynamics of SGD in linear regression are driven by a modiﬁed loss subject to oscillatory probability currents.
E VIDENCE OF A M ODIFIED L OSS AND O SCILLATIONS IN D EEP L EARNING
Does the theory derived in the linear regression setting (sections 5, 6) help explain the empirical phenomena observed in the non-linear setting of deep neural networks (section 2)?
In order for the theory built in the previous sections to apply to the limiting dynamics of neural networks, we must introduce simplifying assumptions on the loss landscape and gradient noise at the end of training:
Assumption 4 (Quadratic Loss) .
We assume that at the end of training the loss for a neural network can be approximated by the quadratic loss L ( θ )
=
( θ − µ ) (cid:124) (cid:0) H 2 (cid:1) ( θ − µ ) , where H (cid:23) 0 is the training loss Hessian and µ is some unknown mean vector, corresponding to a local minimum.
Assumption 5 (Covariance Structure) .
We assume the covariance of the gradient noise is proportional to the Hessian of the quadratic loss Σ( θ ) =
σ 2 H where σ ∈ R
+ is some unknown scalar.
Under these simpliﬁcations, then the expressions derived in the linear regression setting would apply to the limiting dynamics of deep neural networks and depend only on quantities that we can easily estimate empirically.
Of course, these simpliﬁcations are quite strong, but without arguing their theoretical validity, we can empirically test their qualitative implications: (1) a modiﬁed isotropic loss driving the limiting dynamics through parameter space, (2) implicit regularization of the velocity trajectory, and (3) oscillatory phase space dynamics determined by the Hessian eigen-structure.
Modiﬁed loss.
As discussed in section 6, due to the anisotropy of the diffusion matrix, the loss landscape driving the dynamics at the end of training is not the original training loss
L ( θ ) , but a modiﬁed loss Ψ( θ, v ) in phase space.
As shown in equation (8), the modiﬁed loss decouples into a term Ψ θ that only depends on the parameters θ and a term Ψ v that only depends on the velocities v .
Under assumption 5, the parameter dependent component is proportional to the convex quadratic,
[Equation]
This quadratic function has the same mean µ as the training loss, but a different curvature.
Using this expression, notice that when λ ≈ 0 , the modiﬁed loss is isotropic in the column space of H , regardless of what the nonzero eigenspectrum of H is.
This striking prediction suggests that no matter how anisotropic the original training loss – as reﬂected by poor conditioning of the Hessian eigenspectrum – the training trajectory of the network will behave isotropically, since it is driven not by the original anisotropic loss, but a modiﬁed isotropic loss.
We test this prediction by studying the limiting dynamics of a pre-trained ResNet-18 model with batch normalization that we continue to train on ImageNet according to the last setting of its hyperparameters [49].
Let θ
∗
represent the initial pre-trained parameters of the network, depicted with the white dot in ﬁgures 3 and 4. [Table] [Figure]
The shape of the projected velocity trajectory closely resembles the contours of the modiﬁed loss Ψ v .
We estimate 1 the top thirty eigenvectors q 1 , . . .
, q 30 of the Hessian matrix H ∗ evaluated at θ ∗ and project the limiting trajectory for the parameters onto the plane spanned by the top q 1 and bottom q 30 eigenvectors to maximize the illustrated anisotropy with our estimates.
We sample the train and test loss in this subspace for a region around the projected trajectory.
Additionally, using the hyperparameters of the optimization, the eigenvalues ρ 1 and ρ 30 , and the estimate for the mean µ = θ ∗ − H − 1 ∗ g ∗ ( g ∗ is the gradient evaluated at θ ∗ ), we also sample from the modiﬁed loss equation (10) in the same region.
Figure 3 shows the projected parameter trajectory on the sampled train, test and modiﬁed losses.
Contour lines of both
the train and test loss exhibit anisotropic structure, with sharper curvature along eigenvector q 1 compared to eigenvector q 30 , as expected.
However, as predicted, the trajectory appears to cover both directions equally.
This striking isotropy of the trajectory within a highly anisotropic slice of the loss landscape indicates qualitatively that the trajectory evolves in a modiﬁed isotropic loss landscape. Step V e l o c i t y
Position
Step V e l o c i t y Position the trajectories.
Implicit velocity regularization.
A second qualitative prediction of the theory is that the velocity is regulated by the inverse Hessian of the training loss.
Of course there are no explicit terms in either the train or test losses that depend on the velocity.
Yet, the modiﬁed loss contains a component, Ψ v ∝ v (cid:124)
H − 1 v , that only depends on the velocities This additional term can be understood as a form of implicit regularization on the velocity trajectory.
Indeed, when we project the velocity trajectory onto the plane spanned by the q 1 and q 30 eigenvectors, as shown in Fig. 4, we see that the trajectory closely resembles the curvature of the inverse Hessian H − 1 .
The modiﬁed loss is effectively penalizing SGD for moving in eigenvectors of the Hessian with small eigenvalues.
A similar qualitative effect was recently proposed by Barrett and Dherin [21] as a consequence of the discretization error due to ﬁnite learning rates.
Phase space oscillations.
A ﬁnal implication of the theory is that at stationarity the network is in broken detailed balance leading to non-zero probability currents ﬂowing through phase space:
[Equation]
These probability currents encourage oscillatory dynamics in the phase space planes characterized by the eigenvectors of the Hessian, at rates proportional to their eigenvalues.
We consider the same projected trajectory of the ResNet-18 model visualized in ﬁgures 3 and 4, but plot the trajectory in phase space for the two eigenvectors q 1 and q 30 separately.
Shown in Fig.
5, we see that both trajectories look like noisy clockwise rotations.
Qualitatively, the trajectories for the different eigenvectors appear to be rotating at different rates.
The integral curves of the stationary probability current are one-dimensional paths conﬁned to level sets of the modiﬁed loss.
These paths might cross themselves, in which case they are limit cycles, or they could cover the entire surface of the level sets, in which case they are space-ﬁlling curves.
This distinction depends on the relative frequencies of the oscillations, as determined by the pairwise ratios of the eigenvalues of the Hessian.
For real-world datasets, with a large spectrum of incommensurate frequencies, we expect to be in the latter setting, thus contradicting the suggestion that SGD in deep networks converges to limit cycles, as claimed in Chaudhari and Soatto [33].
U NDERSTANDING THE D IFFUSIVE B EHAVIOUR OF THE L IMITING D YNAMICS
Taken together the empirical results shown in section 7 indicate that many of the same qualitative behaviors of SGD identiﬁed theoretically for linear regression are evident in the limiting dynamics of neural networks.
Can this theory quantitatively explain the results we identiﬁed in section 2?
Constant instantaneous speed.
As noted in section 2, we observed that at the end of training, across various architectures, the squared norm of the local displacement (cid:107) δ t (cid:107) 22 remains essentially constant.
Assuming the limiting dynamics are described by the stationary solution the expectation of the local displacement is
[Equation]
(12) as derived in appendix G. We cannot test this prediction directly as we do not know σ 2 and computing tr( H ) is computationally prohibitive.
However, we can estimate σ 2 tr( H ) by resuming training for a model, measuring the average (cid:107) δ t (cid:107) 2 , and then inverting equation (12).
Using this single estimate, we ﬁnd that for a sweep of models with varying hyperparameters, equation (12) accurately predicts their instantaneous speed.
Indeed, Fig.
6 shows an exact match between the empirics and theory, which strongly suggests that despite changing hyperparameters at the end of training, the model remains in the same quadratic basin.
[Figure]
Exponent of anomalous diffusion.
The expected value for the global displacement under the stationary solution can also be analytically expressed in terms of the optimization hyperparameters and the eigendecomposition of the Hessian as,
[Equation]
where C l ( k ) is a trigonometric function describing the velocity of a harmonic oscillator with damping ratio ζ l = (1 − β ) / (cid:112) 2 η (1 + β ) ( p l + λ ) , see appendix G for details.
As shown empirically in section 2, the squared norm (cid:107) ∆ t (cid:107) 2 monotonically increases as a power law in the number of steps, suggesting its expectation is proportional to t c for some unknown, constant c .
The exponent c determines the regime of diffusion for the process.
When c = 1 , the process corresponds to standard
Brownian diffusion.
For c > 1 or c < 1 the process corresponds to anomalous super-diffusion or sub-diffusion respectively.
Unfortunately, it is not immediately clear how to extract the explicit exponent c from equation (13).
However, by exploring the functional form of C l ( k ) and its relationship to the hyperparameters of optimization through the damping ratio ζ l , we can determine overall trends in the diffusion exponent c .
Akin to how the exponent c determines the regime of diffusion, the damping ratio ζ l determines the regime for the harmonic oscillator describing the stationary velocity-velocity correlation in the l th eigenvector of the Hessian.
When ζ l = 1 , the oscillator is critically damped implying the velocity correlations converge to zero as quickly as possible.
In the extreme setting of C l ( k ) = 0 for all l, k , then equation (13) simpliﬁes to standard Brownian diffusion, E ss (cid:2) (cid:107) ∆ t (cid:107) 2 (cid:3) ∝ t .
When ζ l > 1 , the oscillator is overdamped implying the velocity correlations dampen slowly and remain positive even over long temporal lags.
Such long lasting temporal correlations in velocity lead to faster global displacement.
Indeed, in the extreme setting of C l ( k ) = 1 for all l, k , then equation (13) simpliﬁes to a form of anomalous super-diffusion, E ss (cid:2) (cid:107) ∆ t (cid:107) 2 (cid:3) ∝ t 2 .
When ζ l < 1 , the oscillator is underdamped implying the velocity correlations will oscillate quickly between positive and negative values.
Indeed, the only way equation (13) could describe anomalous sub-diffusion is if C l ( k ) took on negative values for certain l, k .
Using the same sweep of models described previously, we can empirically conﬁrm that the optimization hyperparameters each inﬂuence the diffusion exponent c .
As shown in Fig.
6, the learning rate, batch size, and momentum can each independently drive the exponent c into different regimes of anomalous diffusion.
Notice how the inﬂuence of the learning rate and momentum on the diffusion exponent c closely resembles their respective inﬂuences on the damping ratio ζ l .
Interestingly, a larger learning rate leads to underdamped oscillations, and the resultant temporal velocities’ anti-correlations reduce the exponent of anomalous diffusion.
Thus contrary to intuition, a larger learning rate actually leads to slower global transport in parameter space.
The batch size on the other hand, has no inﬂuence on the damping ratio, but leads to an interesting, non-monotonic inﬂuence on the diffusion exponent.
Overall, the hyperparameters of optimization and eigenspectrum of the
Hessian all conspire to govern the degree of anomalous diffusion at the end of training.
D ISCUSSION
Through combined empirics and theory based on statistical physics, we uncovered an intricate interplay between the optimization hyperparameters, structure in the gradient noise, and the Hessian matrix at the end of training.
Signiﬁcance.
The signiﬁcance of our work lies in (1) the identiﬁcation/veriﬁcation of multiple empirical phenomena (constant instantaneous speed, anomalous diffusion in global displacement, isotropic parameter exploration despite anisotopic loss, velocity regularization, and slower global parameter exploration with faster learning rates) present in the limiting dynamics of deep neural networks, (2) the emphasis on studying the dynamics in velocity space in addition to parameter space, and (3) concrete quantitative as well as qualitative predictions of an SDE based theory that we empirically veriﬁed in deep networks trained on large scale datasets (indeed some of the above nontrivial phenomena were predictions of this theory).
Of course, these contributions directly build upon a series of related works studying the immensely complex process of deep learning.
To this end, we further clarify the originality of our contributions with respect to some relevant works.
Originality.
The empirical phenomena we present provide novel insight with respect to the works of Wan et al.
[13], Hoffer et al.
[18], and Chen et al.
We observe that all parameters in the network (not just those with scale symmetry) move at a constant instantaneous speed at the end of training and diffuse anomalously at rates determined by the hyperparameters of optimization.
In contrast to the work by Liu et al.
[35], we modeled the entire SGD process as an OU process which allows us to provide insight into the transient dynamics and identify oscillations in parameter and velocity space.
We build on the theoretical framework used by Chaudhari and Soatto [33] and provide explicit expressions for the limiting dynamics in the simpliﬁed linear regression setting and conclude that the oscillations present in the limiting dynamics are more likely to be space-ﬁlling curves (and not limit cycles) in deep learning due to many incommensurate oscillations.
Overall, by identifying key phenomena, explaining them in a simpler setting, deriving predictions of new phenomena, and providing evidence for these predictions at scale, we are furthering the scientiﬁc study of deep learning.
We hope our newly derived understanding of the limiting dynamics of SGD, and its dependence on various important hyperparameters like batch size, learning rate, and momentum, can serve as a basis for future work that can turn these insights into algorithmic gains.
A M ODELING SGD WITH AN SDE
As explained in section 4, in order to understand the dynamics of stochastic gradient descent we build a continuous Langevin equation in phase space modeling the effect of discrete updates and stochastic batches simultaneously.
A. M ODELING D ISCRETIZATION
To model the discretization effect we assume that the system of update equations (2) is actually a discretization of some unknown ordinary differential equation.
To uncover this ODE, we combine the two update equations in (2), by incorporating a previous time step θ k − 1 , and rearrange into the form of a ﬁnite difference discretization, as shown in equation ( ?? ).
Like all discretizations, the Euler discretizations introduce error terms proportional to the step size, which in this case is the learning rate η .
Taylor expanding θ k +1 and θ k − 1 around θ k , its easy to show that both Euler discretizations introduce a second-order error term proportional to η 2 ¨ θ .
[Equation]
Notice how the momentum coefﬁcient β ∈ [0 , 1] regulates the amount of backward Euler incorporated into the discretization.
When β = 0 , we remove all backward Euler discretization leaving just the forward Euler discretization.
When β = 1 , we have equal amounts of backward Euler as forward Euler resulting in a central second-order discretization 2 as noticed in [19].
A. M ODELING S TOCHASTICITY
In order to model the effect of stochastic batches, we ﬁrst model a batch gradient with the following assumption:
Assumption 1 (CLT).
We assume the batch gradient is a noisy version of the true gradient such that g B ( θ ) − g ( θ ) is a Gaussian random variable with mean 0 and covariance 1 S Σ( θ ) .
The two conditions needed for the CLT to hold are not exactly met in the setting of SGD.
Independent and identically distributed.
Generally we perform SGD by making a complete pass through the entire dataset before using a sample again which introduces a weak dependence between samples.
While the covariance matrix without replacement more accurately models the dependence between samples within a batch, it fails to account for the dependence between batches.
Finite variance.
A different line of work has questioned the Gaussian assumption entirely because of the need for ﬁnite variance random variables.
This work instead suggests using the generalized central limit theorem implying the noise would be a heavy-tailed α -stable random variable [29].
Thus, the previous assumption is implicitly assuming the i.i.d.
and ﬁnite variance conditions apply for large enough datasets and small enough batches.
Under the CLT assumption, we must also replace the Euler discretizations with Euler–Maruyama discretizations.
For a general stochastic process, dX t = µdt + σdW t , the Euler–Maruyama method extends the Euler method for ODEs to SDEs, resulting in the update equation X k
+
=
X k + ∆ tµ + √ ∆ tσξ , where ξ ∼ N (0 , 1) .
Notice, the key difference is that if the temporal step size is ∆ t = η , then the noise is scaled by the square root √ η .
In fact, the main argument against modeling SGD with an SDE, as nicely explained in Yaida [28], is that most SDE approximations simultaneously assume that ∆ t → 0 + , while maintaining that the square root of the learning rate √ η is ﬁnite.
However, by modeling the discretization and stochastic effect simultaneously we can avoid this argument, bringing us to our second assumption:
Assumption 2 (SDE).
We assume the underdamped Langevin equation (3) accurately models the trajectory of the network driven by SGD through phase space such that θ ( ηk )
≈ θ k
and v ( ηk )
≈ v k .
This approach of modeling discretization and stochasticity simultaneously is called stochastic modiﬁed equations, as further explained in Li et al. [Table]
B S TRUCTURE IN THE C OVARIANCE OF THE G RADIENT N OISE
As we’ve mentioned before, SGD introduces highly structured noise into an optimization process, often assumed to be an essential ingredient for its ability to avoid local minima.
Assumption 5 (Covariance Structure).
We assume the covariance of the gradient noise is proportional to the Hessian of the quadratic loss Σ( θ ) =
σ 2 H where σ ∈ R
+ is some unknown scalar.
In the setting of linear regression, this is a very natural assumption.
If we assume the classic generative model for linear regression data y i
= x (cid:124)
i ¯ θ + σ(cid:15) where, ¯ θ ∈ R
d is the true model and (cid:15) ∼ N (0 , 1) , then provably Σ( θ ) ≈ σ 2 H .
Proof.
We can estimate the covariance as Σ( θ ) ≈ 1 N (cid:80)
{+Ni =
+}
g i g (cid:124)
i − gg (cid:124) .
Near stationarity gg (cid:124) (cid:28) 1 N (cid:80) Ni
=
g i g (cid:124) i , and thus,
[Equation]
Under the generative model y i
=
x (cid:124)
i ¯ θ + σ(cid:15) where (cid:15) ∼ N (0 , 1) and σ ∈ R + , then the gradient g i is g i = [Equation] i ( [Equation]
− ¯
[Equation]
) −
[Equation]
) x i , and the matrix g i g (cid:124)
i is g
[Equation]
Assuming θ
≈ ¯ θ at stationarity, then ( x (cid:124) i
( θ − ¯ θ ) − σ(cid:15) )
2 ≈ σ 2 .
Thus, Σ( θ
[Equation]
Also notice that weight decay is independent of the data or batch and thus simply shifts the gradient distribution, but leaves the covariance of the gradient noise unchanged.
While the above analysis is in the linear regression setting, for deep neural networks it is reasonable to make the same assumption.
See the appendix of Jastrz˛ebski et al.
[12] for a discussion on this assumption in the non-linear setting.
Recent work by Ali et al.
[32] also studies the dynamics of SGD (without momentum) in the setting of linear regression.
This work, while studying the classic ﬁrst-order stochastic differential equation, made a point to not introduce an assumption on the diffusion matrix.
In particular, they make the point that even in the setting of linear regression, a constant covariance matrix will fail to capture the actual dynamics.
To illustrate this point they consider the univariate responseless least squares problem, minimize
[Equation]
As they explain, the SGD update for this problem would be
[Equation]
from which they conclude for a small enough learning rate η , then with probability one θ k
→ 0 .
They contrast this with the Ornstein-Uhlenbeck process given by a constant covariance matrix where while the mean for θ k converges to zero its variance converges to a positive constant.
So is this discrepancy evidence that an Ornstein-Uhlenbeck process with a constant covariance matrix fails to capture the updates of SGD?
In many ways this problem is not a simple example, rather a pathological edge case.
Consider the generative model that would give rise to this problem,
[Equation]
In otherwords, the true model ¯ θ = 0 and the standard deviation for the noise σ = 0 .
This would imply by the assumption used in our paper that there would be zero diffusion and the resulting SDE would simplify to a deterministic ODE that exponentially converges to zero.
C A Q UADRATIC L OSS AT THE E ND OF T RAINING
Assumption 4 (Quadratic Loss).
We assume that at the end of training the loss for a neural network can be approximated by the quadratic loss L ( θ )
=
( θ − µ ) (cid:124) (cid:0) H 2 (cid:1) ( θ − µ ) , where H (cid:23) 0 is the training loss Hessian and µ is some unknown mean vector, corresponding to a local minimum.
This assumption has been amply used in previous works such as Mandt et al.
[31], Jastrz˛ebski et al.
[12], and Poggio et al.
Particularly, Mandt et al.
[31] discuss how this assumption makes sense for smooth loss functions for which the stationary solution to the stochastic process reaches a deep local minimum from which it is difﬁcult to escape.
It is a well-studied fact, both empirically and theoretically, that the Hessian is low-rank near local minima as noted by Sagun et al.
[51], and Kunin et al.
This degeneracy results in ﬂat directions of equal loss.
Kunin et al.
[20] discuss how differentiable symmetries, architectural features that keep the loss constant under certain weight transformations, give rise to these ﬂat directions.
Importantly, the Hessian and the covariance matrix share the same null space, and thus we can always restrict ourselves to the image space of the Hessian, where the drift and diffusion matrix will be full rank.
Further discussion on the relationship between the Hessian and the covariance matrix can be found in Thomas et al.
It is also a well known empirical fact that even at the end of training the Hessian can have negative eigenvalues [41].
This empirical observation is at odds with our assumption that the Hessian is positive semi-deﬁnite H (cid:23) 0 .
Further analysis is needed to alleviate this inconsistency.
D S OLVING AN O RNSTEIN -U HLENBECK P ROCESS WITH A NISOTROPIC N OISE
We will study the multivariate Ornstein-Uhlenbeck process described by the stochastic differential equation
[Equation]
where A ∈ S m ++ is a positive deﬁnite drift matrix, µ ∈ R m is a mean vector, κ ∈ R + is some positive constant, and D ∈ S m ++ is a positive deﬁnite diffusion matrix.
This OU process is unique in that it is one of the few SDEs we can solve explicitly.
We can derive an expression for X T as,
[Equation]
Proof.
Consider the function f
( t, x ) =
e At x where e A is a matrix exponential.
Then by Itô’s
Lemma 3 we can evaluate the derivative of f ( t, X t ) as
[Equation]
Integrating this expression from t = 0 to t =
T gives
[Equation]
which rearranged gives the expression for X T .
From this expression it is clear that X T is a Gaussian process.
The mean of the process is
[Equation]
and the covariance and cross-covariance of the process are
[Equation]
These last two expressions are derived by Itô Isometry 4 .
D. T HE L YAPUNOV E QUATION
We can explicitly solve the integral expressions for the covariance and cross-covariance exactly by solving for the unique matrix B ∈ S m ++ that solves the Lyapunov equation ,
[Equation]
If B solves the Lyapunov equation
[Equation]
notice
[Equation]
Using this derivative
[Equation]
the integral expressions for the covariance and cross-covariance simplify as,
[Equation]
[Equation]
(21) where we implicitly assume T ≤ S .
[Table]
[Table] [Table]
[Table]
D. D ECOMPOSING THE D RIFT M ATRIX
While the Lyapunov equation simpliﬁes the expressions for the covariance and cross-covariance, it does not explain how to actually solve for the unknown matrix B .
Following a method proposed by
Kwon et al.
[48], we will show how to solve for B explicitly in terms of the drift A and diffusion D .
The drift matrix A can be uniquely decomposed as,
[Equation]
22) where D is our symmetric diffusion matrix, Q is a skew-symmetric matrix (i.e. Q = − Q (cid:124) ), and U is a positive deﬁnite matrix.
Using this decomposition, then B
= U − 1 , solves the Lyapunov equation.
Proof.
Plug B = U − 1 into the left-hand side of equation (19),
[Equation]
Here we used the symmetry of A, D, U and the skew-symmetry of Q .
All that is left is to do is solve for the unknown matrices Q and U .
First notice the following identity,
[Equation]
Proof.
Multiplying A =
( D + Q ) U on the right by ( D − Q ) gives,
[Equation]
which rearranged and using A
=
A (cid:124) gives the desired equation.
Let V Λ V (cid:124) be the eigendecomposition of A and deﬁne the matrices (cid:101)
D = V (cid:124) DV and (cid:101) Q = V (cid:124) QV .
These matrices observe the following relationship,
[Equation]
Proof.
Replace A in the previous equality with its eigendecompsoition,
[Equation]
Multiply this equation on the right by V and on the left by V (cid:124) ,
[Equation]
Looking at this equality element-wise and using the fact that Λ is diagonal gives the scalar equality for any i, j , (
[Equation]
which rearranged gives the desired expression.
Thus, Q and U are given by,
[Equation]
This decomposition always holds uniquely when A, D
[Equation]
, as λ i − λ
j λ i + λ j exists and ( D + Q ) is invertible.
See [48] for a discussion on the singularities of this decomposition.
D. S TATIONARY S OLUTION
Using the Lyapunov equation and the drift decomposition, then X T ∼ p T , where
[Equation]
In the limit as T → ∞ , then e − AT → 0 and p T → p ss where
[Equation]
Similarly, the cross-covariance converges to the stationary cross-covariance,
[Equation]
E A V ARIATIONAL F ORMULATION OF THE OU P ROCESS WITH A NISOTROPIC N OISE
In this section we will describe an alternative, variational, route towards solving the dynamics of the
OU process studied in appendix D.
Let Φ : R n → R be an arbitrary, non-negative potential and consider the stochastic differential equation describing the Langevin dynamics of a particle in this potential ﬁeld,
[Equation]
where D ( X t ) is an arbitrary, spatially-dependent, diffusion matrix, κ is a temperature constant, and x 0 ∈ R m is the particle’s initial position.
The Fokker-Planck equation describes the time evolution for the probability distribution p of the particle’s position such that p ( x, t ) =
P ( X t = x ) .
The FP equation is the partial differential equation 5 ,
[Equation]
where ∇· denotes the divergence and δ ( x 0 ) is a dirac delta distribution centered at the initialization x 0 .
To assist in the exploration of the FP equation we deﬁne the vector ﬁeld,
[Equation]
(31) which is commonly referred to as the probability current .
Notice, that this gives an alternative expression for the FP equation, ∂ t p = −∇· J , demonstrating that J ( x, t ) deﬁnes the ﬂow of probability mass through space and time.
This interpretation is especially useful for solving for the stationary solution p ss , which is the unique distribution that satisﬁes,
[Equation]
(32) where J ss is the probability current for p ss .
The stationary condition can be obtained in two distinct ways:
Detailed balance.
This is when J ss ( x ) = 0 for all x ∈ Ω .
This is analogous to reversibility for discrete Markov chains, which implies that the probability mass ﬂowing from a state i to any state j is the same as the probability mass ﬂowing from state j to state i .
Broken detailed balance.
This is when ∇ · J ss ( x ) =
but J ss ( x ) (cid:54) =
for all x ∈ Ω .
This is analogous to irreversibility for discrete Markov chains, which only implies that the total probability mass ﬂowing out of state i equals to the total probability mass ﬂowing into state i .
The distinction between these two cases is critical for understanding the limiting dynamics of the process.
E. T HE V ARIATIONAL F ORMULATION OF THE F OKKER -P LANCK E QUATION WITH I SOTROPIC D IFFUSION
We will now consider the restricted setting of standard, isotropic diffusion ( D = I ).
It is easy enough to check that in this setting the stationary solution is
[Equation]
33) where p ss is called a Gibbs distribution and Z is the partition function.
Under this distribution, the stationary probability current is zero ( J ss ( x ) = 0 ) and thus the process is in detailed balance.
Interestingly, the Gibbs distribution p ss has another interpretation as the unique minimizer of the the
Gibbs free energy functional,
[Equation]
(34) where E [Φ] is the expectation of the potential Φ under the distribution p and H ( p ) =
− (cid:82) Ω p ( x ) log ( p ( x ))
dx is the Shannon entropy of p .
Proof.
To prove that indeed p ss is the unique minimizer of the Gibbs free energy functional, consider the following equivalent expression
[Equation]
From this expressions, it is clear that the Kullback–Leibler divergence is uniquely minimized when p = p ss .
In other words, with isotropic diffusion the stationary solution p ss can be thought of as the limiting distribution given by the Fokker-Planck equation or the unique minimizer of an energetic-entropic functional.
Seminal work by Jordan et al.
[53] deepened this connection between the Fokker-Planck equation and the Gibbs free energy functional.
In particular, their work demonstrates that the solution p ( x, t ) to the Fokker-Planck equation is the Wasserstein gradient ﬂow trajectory on the Gibbs free energy functional.
Steepest descent is always deﬁned with respect to a distance metric.
For example, the update equation, x k +
= x k − η ∇ Φ( x k ) , for classic gradient descent on a potential Φ( x ) , can be formulated as the solution to the minimization problem x k +1 =
argmin x
η Φ( x ) + 12 d ( x, x k ) 2 where d ( x, x k )
= (cid:107) x − x k (cid:107) is the Euclidean distance metric.
Gradient ﬂow is the continuous-time limit of gradient descent where we take η → 0 + .
Similarly, Wasserstein gradient ﬂow is the continuous-time limit of steepest descent optimization deﬁned by the Wasserstein metric .
The Wasserstein metric is a distance metric between probability measures deﬁned as,
[Equation]
where µ 1 and µ 2 are two probability measures on R n with ﬁnite second moments and Π( µ 1 , µ 2 ) deﬁnes the set of joint probability measures with marginals µ 1 and µ 2 .
Thus, given an initial distribution and learning rate η , we can use the Wasserstein metric to derive a sequence of distributions minimizing some functional in the sense of steepest descent.
In the continuous-time limit as η → 0 + this sequence deﬁnes a continuous trajectory of probability distributions minimizing the functional.
Jordan et al.
[54] proved, through the following theorem, that this process applied to the Gibbs free energy functional converges to the solution to the Fokker-Planck equation with the same initialization:
Theorem 1 (JKO) .
Given an initial condition p 0 with ﬁnite second moment and an η > 0 , deﬁne the iterative scheme p η with iterates deﬁned by
[Equation]
As η → 0 + , then p η → p weakly in L 1 where p is the solution to the Fokker-Planck equation with the same initial condition.
See [54] for further explanation and [53] for a complete derivation.
E. E XTENDING THE V ARIATIONAL F ORMULATION TO THE S ETTING OF A NISOTROPIC D IFFUSION
While the JKO theorem provides a very powerful lens through which to view solutions to the FokkerPlanck equation, and thus distributions for particles governed by Langevin dynamics, it only applies in the very restricted setting of isotropic diffusion.
In this section we will review work by Chaudhari and Soatto [33] extending the variational interpretation to the setting of anisotropic diffusion.
Consider when D ( X t ) is an anisotropic, spatially-dependent diffusion matrix.
In this setting, the original Gibbs distribution given in equation (33) does not necessarily satisfy the stationarity condition equation (32).
In fact, it is not immediately clear what the stationary solution is or if the dynamics even have one.
Thus, Chaudhari and Soatto [33] make the following assumption:
Stationary Assumption.
Assume there exists a unique distribution p ss that is the stationary solution to the Fokker-Planck equation irregardless of initial conditions.
Under this assumption we can implicitly deﬁne the potential Ψ( x ) = − κ − 1 log ( p ss ( x )) .
Using this modiﬁed potential we can express the stationary solution as a Gibbs distribution,
[Equation]
Under this implicit deﬁnition we can deﬁne the stationary probability current as J ss ( x ) =
j ( x ) p ss ( x ) where
[Equation]
The vector ﬁeld j ( x ) reﬂects the discrepancy between the original potential Φ and the modiﬁed potential Ψ according to the diffusion D ( x ) .
Notice that in the isotropic case, when D ( x ) =
I , then Φ = Ψ and j ( x )
= 0 .
Chaudhari and Soatto [33] introduce another property of j ( x ) through assumption,
Conservative Assumption.
Assume that the force j ( x ) is conservative (i.e. ∇ · j ( x ) = 0 ).
Using this assumption, Chaudhari and Soatto [33] extends the variational formulation provided by the JKO theorem to the anisotropic setting,
Theorem 2 (CS) .
Given an initial condition p 0 with ﬁnite second moment, then the energeticentropic functional,
[Equation]
monotonically decreases throughout the trajectory given by the solution to the Fokker-Planck equation with the given initial condition.
In other words, the Fokker-Plank equation (30) with anisotropic diffusion can be interpreted as minimizing the expectation of a modiﬁed loss Ψ , while being implicitly regularized towards distributions that maximize entropy.
The derivation requires we assume a stationary solution p ss exists and that the force j ( x ) implicitly deﬁned by p ss is conservative.
However, rather than implicitly deﬁne Ψ( x ) and j ( x ) through assumption, if we can explicitly construct a modiﬁed loss Ψ( x ) such that the resulting j ( x ) satisﬁes certain conditions, then the stationary solution exists and the variational formulation will apply as well.
We formalize this statement with the following theorem,
Theorem 3 (Explicit Construction) .
If there exists a potential Ψ( x ) such that either j ( x ) = 0 or ∇ · j ( x ) = 0 and ∇ Ψ( x ) ⊥ j ( x ) , then p ss is the Gibbs distribution ∝ e − κ Ψ( x ) and the variational formulation given in Theorem 2 applies.
E. A PPLYING THE V ARIATIONAL F ORMULATION TO THE OU P ROCESS
Through explicit construction we now seek to ﬁnd analytic expressions for the modiﬁed loss Ψ( x ) and force j ( x ) hypothesised by Chaudhari and Soatto [33] in the fundamental setting of an OU process with anisotropic diffusion, as described in section
D.
We assume the diffusion matrix is anisotropic, but spatially independent, ∇ · D ( x )
= 0 .
For the OU process the original potential generating the drift is
[Equation]
Recall, that in order to extend the variational formulation we must construct some potential Ψ( x ) such that ∇ · j ( x ) = 0 and ∇ Ψ ⊥ j ( x ) .
It is possible to construct Ψ( x ) using the unique decomposition of the drift matrix A
=
( D + Q ) U discussed in appendix D. Deﬁne the modiﬁed potential,
[Equation]
Using this potential, the force j ( x ) is
[Equation]
Notice that j ( x ) is conservative, ∇ · j ( x ) = ∇ · − QU ( x − µ )
= 0 because Q is skew-symmetric.
Additionally, j ( x ) is orthogonal, j ( x ) (cid:124) ∇
[Equation]
x ) =
[Equation] [Equation]
U
[Equation]
QU ( x − µ ) = 0 , again because Q is skew-symmetric.
Thus, we have determined a modiﬁed potential Ψ( x ) that results in a conservative orthogonal force j ( x ) satisfying the conditions for Theorem 3.
Indeed the stationary Gibbs distribution given by Theorem 3 agrees with equation (27) derived via the ﬁrst and second moments in appendix D,
[Equation]
In addition to the variational formulation, this interpretation further details explicitly the stationary probability current, J ss ( x ) =
j ( x ) p ss , and whether or not the the stationary solution is in broken detailed balance.
F E XPLICIT EXPRESSIONS FOR THE OU PROCESS G ENERATED BY SGD
We will now consider the speciﬁc OU process generated by SGD with linear regression.
Here we repeat the setup as explained in section 5.
Let X ∈ R N × d , Y ∈ R N be the input data, output labels respectively and θ ∈ R d be our vector of regression coefﬁcients.
The least squares loss is the convex quadratic loss L ( θ ) = 12 N (cid:107) Y − Xθ
(cid:107)with gradient g ( θ ) =
Hθ − b , where H = X (cid:124) XN and b = X (cid:124) YN .
Plugging this expression for the gradient into the underdamped Langevin equation (3), and rearranging terms, results in the multivariate Ornstein-Uhlenbeck (OU) process,
[Equation]
where A and D are the drift and diffusion matrices respectively,
[Equation]
κ = S (1 − β 2 ) is a temperature constant, and µ = ( H + λI ) − 1 b is the ridge regression solution.
F. S OLVING FOR THE M ODIFIED L OSS AND C ONSERVATIVE F ORCE
In order to apply the expressions derived for a general OU process in appendix D and E, we must ﬁrst decompose the drift as A =
( D + Q ) U .
Under the simpliﬁcation Σ( θ )
=
σ 2 H discussed in appendix B, then the matrices Q and U , as deﬁned below, achieve this,
[Equation]
Using these matrices we can now derive explicit expressions for the modiﬁed loss Ψ( θ, v ) and conservative force j ( θ, v ) .
First notice that the least squares loss with L 2 regularization is proportional to the convex quadratic,
[Equation]
[Equation]
The modiﬁed loss Ψ is composed of two terms, one that only depends on the position,
[Equation] [Equation]
and another that only depends on the velocity,
[Equation]
The conservative force j ( θ, v ) is
[Equation]
47) and
thus the stationary probability current is J ss ( θ, v
[Equation]
= j ( θ, v ) p ss .
F. D ECOMPOSING THE T RAJECTORY INTO THE E IGENBASIS OF THE H ESSIAN
As shown in appendix D, the temporal distribution for the OU process at some time T ≥ 0 is, p T (cid:18)(cid:20)
[Equation]
Here we will now use the eigenbasis { q 1 , . . .
q m } of the Hessian with eigenvalues { ρ 1 , . .
, ρ m } to derive explicit expressions for the mean and covariance of the process through time.
Deterministic component.
We can rearrange the expectation as
[Equation]
Notice that the second, time-dependent term is actually the solution to the system of ODEs
[Equation]
with initial condition [ θ 0 − µ v 0 ] (cid:124) .
This system of ODEs can be block diagonalized by factorizing
A = OSO (cid:124) where O is orthogonal and S is block diagonal deﬁned as
O =
[Equation]


In otherwords in the plane spanned by [ q i 0] (cid:124) and [0 q i ] (cid:124) the system of ODEs decouples into the 2D system
[Equation]
This system has a simple physical interpretation as a damped harmonic oscillator.
If we let b i = ˙
a i , then we can unravel this system into the second order ODE
[Equation]
which is in standard form (i.e. ¨ x + 2 γ ˙ x + ω 2 x = 0 ) for γ = 1 − β η (1+ β ) and ω i = (cid:113) 2 η
(1+ β ) ( ρ i + λ ) .
Let a i (0) =
(cid:104) θ 0 − µ, q i (cid:105) and b i (0) =
(cid:104) v 0 , q i (cid:105) , then the solution in terms of γ and ω i is a i ( t ) =
[Equation]
γ
[Equation]
Differentiating these equations gives us solutions for b
i ( t )
[Equation]
Combining all these results, we can now analytically decompose the expectation as the sum,
[Equation]
Intuitively, this equation describes a damped rotation (spiral) around the OLS solution in the planes deﬁned by the the eigenvectors of the Hessian at a rate proportional to the respective eigenvalue.
Stochastic component.
Using the previous block diagonal decomposition A = OSO (cid:124)
we can simplify the variance as
[Equation]
Interestingly, the matrix O (cid:124) U − 1 O is also block diagonal,
[Equation]

[Equation] [Equation]
[Equation]

Thus, similar to the mean
[Equation]
we can simply consider the variance in each of the planes spanned by [ q i 0] (cid:124) and [0 q i ] (cid:124) .
If we deﬁne the block matrices,
[Equation]
then the projected variance matrix in this plane simpliﬁes as
Var
[Equation]
Using the solution to a damped harmonic osccilator discussed previously
[Equation]
we can express the matrix exponential e − S i T explicitly in terms of γ = 1 − β η
(1+ β ) and ω i = (cid:113) 2 η
(1+ β ) ( ρ i + λ ) .
If we let α i =
[Equation]
| γ 2 − ω 2 i | , then the matrix exponential is
[Equation]
 
[Equation]
G A NALYZING P ROPERTIES OF THE S TATIONARY S OLUTION
Assuming the stationary solution is given by equation ( ?? )
we can solve for the expected value of the norm of the local displacement and gain some intuition for the expected value of the norm of global displacement.
G. I NSTANTANEOUS S PEED
[Equation]
Note that this follows directly from the deﬁnition of δ k in equation (1) and the mean and variance of the stationary solution in equation ( ?? ), as well as the follow-up derivation in appendix F.
G. A NOMALOUS D IFFUSION
Notice, that the global movement ∆ t = θ t − θ 0 can be broken up into the sum of the local movements ∆ t = (cid:80) ti =
δ i , where δ i
= θ i − θ
i − 1 .
Applying this decomposition,
[Equation]

[Equation]
(cid:88) i =
[Equation]
i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
As we solved for previously,
[Equation]
By a similar simpliﬁcation, we can express the second term in terms of the stationary crosscovariance,
[Equation]
Thus, to simplify this expression we just need to consider the velocity-velocity covariance
Cov ss ( v i , v j ) .
At stationarity, the cross-covariance for the system in phase space, z i =
[ θ i v i ] is
[Equation]
where κ = S (1 − β 2 ) , and
[Equation]
β ) η (1+ β ) I (cid:21)
As discussed when solving for the mean of the OU trajectory, the drift matrix A can be block diagonalized as A =
OSO (cid:124) where O is orthogonal and S is block diagonal deﬁned as
O =  
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]

[Equation]
[Equation]
[Equation]
−
[Equation]
− β
[Equation]
β
[Equation]

 .
Notice also that O diagonalizes U − 1 such that,
[Equation]
Applying these decompositions, properties of matrix exponentials, and the cyclic invariance of the trace, allows us to express the trace of the cross-covariance as
[Equation]
where Λ k and S k are the blocks associated with each eigenvector of H .
As solved for previously in the variance of the OU process, we can express the matrix exponential e − S k | i − j | explicitly in terms of γ = 1 − β η
(1+ β ) and ω k = (cid:113) 2 η
[Equation]
1+ β ) ( ρ k + λ ) .
If we let τ = | i − j | and α k = (cid:112) | γ 2 − ω 2 k | , then the matrix exponential is
[Equation]
Plugging in these expressions into previous expression and restricting to just the k th velocity component, we see
[Equation]
Pulling it all together,
[Equation]
where C l ( k ) is deﬁned as
[Equation]
−
[Equation]
) γ
[Equation]
e
[Equation]
cos
[Equation]
γ <
[Equation]
for γ = 1 − β η (1+ β ) , ω l = (cid:113) 2 η
(1+ β ) ( ρ l + λ ) , and α l = (cid:112) | γ
− ω 2 l | .
G. T RAINING L OSS AND E QUIPARTITION T HEOREM
In addition to solving for the expected values of the local and global displacements, we can consider the expected training loss and ﬁnd an interesting relationship to the equipartition theorem from classical statistical mechanics.
The regularized training loss is L λ ( θ )
=
12 ( θ − µ )
[Equation]
H ( θ − µ ) + λ 2 (cid:107) θ (cid:107) 2 , where H is the Hessian matrix and µ is the true mean.
Taking the expectation with respect to the stationary distribution,
[Equation]
The ﬁrst and second moments of the stationary solution are
[Equation]
Plugging these expressions in and canceling terms we get
[Equation]
Deﬁne the kinetic energy of the network as K ( v ) = 12 m (cid:107) v (cid:107) 2 , where m =
η 2 (1 + β ) is the perparameter “mass" of the network according to our previously derived Langevin dynamics.
At stationarity,
[Equation]
where we used the fact that E ss [ vv (cid:124) ]
=
[Equation]
σ 2 H .
In otherwords, at stationarity,
[Equation]
This relationship between the expected potential and kinetic energy can be understood as a form of the equipartition theorem.
H E XPERIMENTAL D ETAILS H. C OMPUTING THE H ESSIAN EIGENDECOMPOSITION
Computing the full Hessian of the loss with respect to the parameters is computationally intractable for large models.
However, equipped with an autograd engine, we can compute Hessian-vector products.
We use the subspace iteration on Hessian-vector products computed on a variety of datasets.
For Cifar-10 we use the entire train dataset to compute the Hessian-vector products.
For Imagenet, we use a subset 40,000 images sampled from the train dataset to keep the computation within reasonable limits.
For experiments on linear regression, the Hessian is independent of the model (it only depends on the data) and can be computed using any model checkpoint.
For all other experiments, the Hessian eigenvectors were computed using the model at its initial pre-trained state
. H. F IGURE
We resumed training for a variety of ImageNet pre-trained models from Torchvision [16] forepochs, with the hyperparameters used at the end of training, shown in table 1.
We kept track of the norms of the local and global displacement, (cid:107) δ k (cid:107)
and (cid:107) ∆ k (cid:107) 22 , every 250 steps in the training process, to keep the length of the trajectories within reasonable limits.
(cid:107) δ k (cid:107) 22 is visualized directly, along with its 15 step moving average.
We then ﬁt a power law of the form αk c to the (cid:107) ∆ k (cid:107) 22 trajectories for each model, using the last 2/3 of the saved trajectories.
We visualize the (cid:107) ∆ k (cid:107) 22 trajectories along with their ﬁts on a log-log plot.
[Table]
H. F IGURE
For this ﬁgure we trained a linear regression model on Cifar-10, using MSE loss on the one-hot encoded labels.
The hyperparameters used during training are shown in table 2.
At every step in training the full set of model weights and velocities were stored.
The top 30 eigenvectors of the hessian were computed as described in appendix H.1, using 10 subspace iterations.
The saved weight and velocity trajectories were then projected onto the top eigenvector of the hessian and were visualized in black.
Using the initial weights and velocities, the red trajectories were computed according to equation (5)
[Table]
H. F IGURE
For this ﬁgure we constructed an arbitrary Ornstein-Uhlenbeck process with anisotropic noise which would help contrast the original and modiﬁed potentials.
We sampled from a 2 dimensional OU process of the form
[Equation]
where we set b = [ − 0 . 1 , 0 . 05] (cid:124) and arbitrarily construct A such that it’s eigenvectors are aligned with q 1 =
[ − 1 , 1] (cid:124) and q 2 =
[1 , 1] (cid:124)
and it’s eigenvalues are 4 and 1 as follows:
[Equation]
The background for the left panel was computed from the convex quadratic potential Φ( x ) =
12 x (cid:124)
Ax − bx .
The background for the right panel was computed from the modiﬁed quadratic Ψ( x ) =
12 x (cid:124)
Ux − ux , with U
=
( D + Q ) − 1 A and u = UA − 1 b
(see equation (25)).
Both were sampled in a regular 40 × 40 grid in [ − 0 .
1] × [ − 0 .
H. F IGURES 4, 5, AND
Starting with the ImageNet pre-trained ResNet-18 from Torchvision [16], we resumed training for 5 epochs with the hyperparameters used at the end of training, shown in table3.
The top 30 Hessian eigenvectors were computed as described in appendix H.1, using 10 subspace iterations.
During training, we tracked the projection of the weights and velocities onto eigenvectors q 1 and q 30 .
For Figure 3, we show the projection of the position trajectory onto eigenvectors q 1 and q 30 in 2D in black.
The background for the left and center panels was computed taking the ImageNet pre-trained
ResNet-18 from Torchvision [16] and perturbing its weights in the q 1 and q 30 directions in a region close to the projected trajectory.
The training and test loss were computed for a grid of 20 ×perturbed models.
The background for the right panel was computed according to equation (10).
[Table]
H. F IGURE
We resumed training for an ImageNet pre-trained ResNet-18 from Torchvision [16] for 2 epochs, using the sweeps of hyperparameters shown in table 4.
We indicate a sweep in a particular hyperparameter by [ A, B ] , which denotes 20 evenly spaced numbers between A and B , inclusive.
We kept track of the norms of the local and global displacement, (cid:107) δ k (cid:107) 2
and (cid:107) ∆ k (cid:107) 2 2 , every step in the training process.
The value for (cid:107) δ k (cid:107) 22 at the end of the two epochs is shown in the top row of the ﬁgure.
We ﬁtted power law of the form αk c to the (cid:107) ∆ k (cid:107) 22 trajectories for each model on the full trajectories.
The ﬁtted exponent c for each model is plotted in the bottom row of the ﬁgure.
[Table]
[Table]
I NCREASING RATE OF ANOMALOUS DIFFUSION
Upon further experimentation with the ﬁtting procedure for the rate of anomalous diffusion explained in Figure 1, we observed an interesting phenomenon.
The ﬁtted exponent c for the power law relationship (cid:107) ∆ k (cid:107) 2 2 ∝ k c increases as a function of the length of the trajectory we ﬁt to.
As can be seen in Figure 7, c increases at a diminishing rate with the length of the trajectory.
This could be indicative of (cid:107) ∆ k (cid:107) 22 being governed by a sum of power laws where the leading term becomes dominant for longer trajectories.
[Figure]