Original: A D IFFUSION T HEORY F OR D EEP L EARNING D YNAMICS : S TOCHASTIC G RADIENT D ESCENT E XPONENTIALLY F AVORS F LAT M INIMA
Revised: A D IFFUSION T HEORY F OR D EEP L EARNING D YNAMICS : S TOCHASTIC G RADIENT D ESCENT E XPONENTIALLY F AVORS F LAT M INIMA

Original: A BSTRACT
Revised: A BSTRACT

Original: Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice.
Revised: Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice.

Original: SGD is known to ﬁnd a ﬂat minimum that often generalizes well.
Revised: SGD is known to ﬁnd a ﬂat minimum that often generalizes well.

Original: However, it is mathematically unclear how deep learning can select a ﬂat minimum among so many minima.
Revised: However, it is mathematically unclear how deep learning can select a ﬂat minimum among so many minima.

Original: To answer the question quantitatively, we develop a density diffusion theory to reveal how minima selection quantitatively depends on the minima sharpness and the hyperparameters.
Revised: To answer the question quantitatively, we develop a density diffusion theory to reveal how minima selection quantitatively depends on the minima sharpness and the hyperparameters.

Original: To the best of our knowledge, we are the ﬁrst to theoretically and empirically prove that, beneﬁted from the Hessian-dependent covariance of stochastic gradient noise, SGD favors ﬂat minima exponentially more than sharp minima, while Gradient Descent (GD) with injected white noise favors ﬂat minima only polynomially more than sharp minima.
Revised: To the best of our knowledge, we are the ﬁrst to theoretically and empirically prove that, beneﬁted from the Hessian-dependent covariance of stochastic gradient noise, SGD favors ﬂat minima exponentially more than sharp minima, while Gradient Descent (GD) with injected white noise favors ﬂat minima only polynomially more than sharp minima.

Original: We also reveal that either a small learning rate or large-batch training requires exponentially many iterations to escape from minima in terms of the ratio of the batch size and learning rate.
Revised: We also reveal that either a small learning rate or large-batch training requires exponentially many iterations to escape from minima in terms of the ratio of the batch size and learning rate.

Original: Thus, large-batch training cannot search ﬂat minima efﬁciently in a realistic computational time.
Revised: Thus, large-batch training cannot search ﬂat minima efﬁciently in a realistic computational time.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, deep learning (LeCun et al., 2015) has achieved great empirical success in various application areas.
Revised: In recent years, deep learning (LeCun et al., 2015) has achieved great empirical success in various application areas.

Original: Due to the over-parametrization and the highly complex loss landscape of deep networks, optimizing deep networks is a difﬁcult task.
Revised: Due to the over-parametrization and the highly complex loss landscape of deep networks, optimizing deep networks is a difﬁcult task.

Original: Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks.
Revised: Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks.

Original: Empirically, SGD can usually ﬁnd ﬂat minima among a large number of sharp minima and local minima (Hochreiter & Schmidhuber, 1995; 1997).
Revised: Empirically, SGD can usually ﬁnd ﬂat minima among a large number of sharp minima and local minima (Hochreiter & Schmidhuber, 1995; 1997).

Original: More papers reported that learning ﬂat minima closely relate to generalization (Hardt et al., 2016; Zhang et al., 2017a; Arpit et al., 2017; Hoffer et al., 2017;
Revised: More papers reported that learning ﬂat minima closely relate to generalization (Hardt et al., 2016; Zhang et al., 2017a; Arpit et al., 2017; Hoffer et al., 2017;

Original: Dinh et al., 2017; Neyshabur et al., 2017; Wu et al., 2017; Dziugaite & Roy, 2017; Kleinberg et al., 2018).
Revised: Dinh et al., 2017; Neyshabur et al., 2017; Wu et al., 2017; Dziugaite & Roy, 2017; Kleinberg et al., 2018).

Original: Some researchers speciﬁcally study ﬂatness itself.
Revised: Some researchers speciﬁcally study ﬂatness itself.

Original: They try to measure ﬂatness (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017;
Revised: They try to measure ﬂatness (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017;

Original: Sagun et al., 2017; Yao et al., 2018), rescale ﬂatness (Tsuzuku et al., 2019; Xie et al., 2020b), and ﬁnd ﬂatter minima (Hoffer et al., 2017; Chaudhari et al., 2017; He et al., 2019b; Xie et al., 2020a).
Revised: Sagun et al., 2017; Yao et al., 2018), rescale ﬂatness (Tsuzuku et al., 2019; Xie et al., 2020b), and ﬁnd ﬂatter minima (Hoffer et al., 2017; Chaudhari et al., 2017; He et al., 2019b; Xie et al., 2020a).

Original: However, we still lack a quantitative theory that answers why deep learning dynamics selects a ﬂat minimum.
Revised: However, we still lack a quantitative theory that answers why deep learning dynamics selects a ﬂat minimum.

Original: The diffusion theory is an important theoretical tool to understand how deep learning dynamics works.
Revised: The diffusion theory is an important theoretical tool to understand how deep learning dynamics works.

Original: It helps us model the diffusion process of probability densities of parameters instead of model parameters themselves.
Revised: It helps us model the diffusion process of probability densities of parameters instead of model parameters themselves.

Original: The density diffusion process of Stochastic Gradient Langevin Dynamics (SGLD) under injected isotropic noise has been discussed by (Sato & Nakagawa, 2014;
Revised: The density diffusion process of Stochastic Gradient Langevin Dynamics (SGLD) under injected isotropic noise has been discussed by (Sato & Nakagawa, 2014;

Original: Raginsky et al., 2017; Zhang et al., 2017b; Xu et al., 2018).
Revised: Raginsky et al., 2017; Zhang et al., 2017b; Xu et al., 2018).

Original: Zhu et al.
Revised: Zhu et al.

Original: (2019) revealed that anisotropic diffusion of SGD often leads to ﬂatter minima than isotropic diffusion.
Revised: (2019) revealed that anisotropic diffusion of SGD often leads to ﬂatter minima than isotropic diffusion.

Original: A few papers has quantitatively studied the diffusion process of SGD under the isotropic gradient noise assumption.
Revised: A few papers has quantitatively studied the diffusion process of SGD under the isotropic gradient noise assumption.

Original: Jastrz˛ebski et al.
Revised: Jastrz˛ebski et al.

Original: ﬁrst studied the minima selection probability of SGD.
Revised: ﬁrst studied the minima selection probability of SGD.

Original: Smith & Le (2018) presented a Beyesian perspective on generalization of SGD.
Revised: Smith & Le (2018) presented a Beyesian perspective on generalization of SGD.

Original: Wu et al.
Revised: Wu et al.

Original: (2018) studied the escape problems of
Revised: (2018) studied the escape problems of

Original: SGD from a dynamical perspective, and obtained the qualitative conclusion on the effects of batch size, learning rate, and sharpness.
Revised: SGD from a dynamical perspective, and obtained the qualitative conclusion on the effects of batch size, learning rate, and sharpness.

Original: Hu et al.
Revised: Hu et al.

Original: (2019) quantitatively showed that the mean escape time of SGD exponentially depends on the inverse learning rate.
Revised: (2019) quantitatively showed that the mean escape time of SGD exponentially depends on the inverse learning rate.

Original: Achille & Soatto (2019) also obtained a related proposition that describes the mean escape time in terms of a free energy that depends on the Fisher Information.
Revised: Achille & Soatto (2019) also obtained a related proposition that describes the mean escape time in terms of a free energy that depends on the Fisher Information.

Original: Li et al.
Revised: Li et al.

Original: (2017) analyzed Stochastic Differential Equation (SDE) of adaptive gradient methods.
Revised: (2017) analyzed Stochastic Differential Equation (SDE) of adaptive gradient methods.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: (2019) mainly contributed to closing the theoretical gap between continuous-time dynamics and discrete-time dynamics under isotropic heavy-tailed noise.
Revised: (2019) mainly contributed to closing the theoretical gap between continuous-time dynamics and discrete-time dynamics under isotropic heavy-tailed noise.

Original: However, the related papers mainly analyzed the diffusion process under parameter-independent and isotropic gradient noise, while stochastic gradient noise (SGN) is highly parameter-dependent and anisotropic in deep learning dynamics.
Revised: However, the related papers mainly analyzed the diffusion process under parameter-independent and isotropic gradient noise, while stochastic gradient noise (SGN) is highly parameter-dependent and anisotropic in deep learning dynamics.

Original: Thus, they failed to quantitatively formulate how SGD selects ﬂat minima, which closely depends on the Hessian-dependent structure of SGN.
Revised: Thus, they failed to quantitatively formulate how SGD selects ﬂat minima, which closely depends on the Hessian-dependent structure of SGN.

Original: We try to bridge the gap between the qualitative knowledge and the quantitative theory for SGD in the presence of parameter-dependent and anisotropic SGN.
Revised: We try to bridge the gap between the qualitative knowledge and the quantitative theory for SGD in the presence of parameter-dependent and anisotropic SGN.

Original: Mainly based on Theorem 3.2 , we have four contributions:
Revised: Mainly based on Theorem 3.2 , we have four contributions:

Original: • The proposed theory formulates the fundamental roles of gradient noise, batch size, the learning rate, and the Hessian in minima selection.
Revised: • The proposed theory formulates the fundamental roles of gradient noise, batch size, the learning rate, and the Hessian in minima selection.

Original: • The SGN covariance is approximately proportional to the Hessian and inverse to batch size.
Revised: • The SGN covariance is approximately proportional to the Hessian and inverse to batch size.

Original: • Either a small learning rate or large-batch training requires exponentially many iterations to escape minima in terms of ratio of batch size and learning rate.
Revised: • Either a small learning rate or large-batch training requires exponentially many iterations to escape minima in terms of ratio of batch size and learning rate.

Original: • To the best of our knowledge, we are the ﬁrst to theoretically and empirically reveal that SGD favors ﬂat minima exponentially more than sharp minima.
Revised: • To the best of our knowledge, we are the ﬁrst to theoretically and empirically reveal that SGD favors ﬂat minima exponentially more than sharp minima.

Original:  S TOCHASTIC G RADIENT N OISE AND SGD D YNAMICS
Revised:  S TOCHASTIC G RADIENT N OISE AND SGD D YNAMICS

Original: We mainly introduce the necessary foundation for the proposed diffusion theory in this section.
Revised: We mainly introduce the necessary foundation for the proposed diffusion theory in this section.

Original: We denote the data samples as { x j } mj =1 , the model parameters as θ and the loss function over data samples x as L ( θ, x ) .
Revised: We denote the data samples as { x j } mj =1 , the model parameters as θ and the loss function over data samples x as L ( θ, x ) .

Original: For simplicity, we denote the training loss as L ( θ ) .
Revised: For simplicity, we denote the training loss as L ( θ ) .

Original: Following Mandt et al.
Revised: Following Mandt et al.

Original: (2017), we may write SGD dynamics as
Revised: (2017), we may write SGD dynamics as

Original: [Equation]
Revised: [Equation]

Original: where ˆ L ( θ ) is the loss of one minibatch, ζ t ∼ N (0 , I ) , and C ( θ ) represents the gradient noise covariance matrix.
Revised: where ˆ L ( θ ) is the loss of one minibatch, ζ t ∼ N (0 , I ) , and C ( θ ) represents the gradient noise covariance matrix.

Original: The classic approach is to model SGN by Gaussian noise, N (0 , C ( θ ))
Revised: The classic approach is to model SGN by Gaussian noise, N (0 , C ( θ ))

Original: (Mandt et al., 2017; Smith & Le, 2018; Chaudhari & Soatto, 2018).
Revised: (Mandt et al., 2017; Smith & Le, 2018; Chaudhari & Soatto, 2018).

Original: Stochastic Gradient Noise Analysis.
Revised: Stochastic Gradient Noise Analysis.

Original: We ﬁrst note that the SGN we study is introduced by minibatch training, C ( θ t ) 12 ζ t = ∂L ( θ t )
Revised: We ﬁrst note that the SGN we study is introduced by minibatch training, C ( θ t ) 12 ζ t = ∂L ( θ t )

Original: ∂θ t − ∂
Revised: ∂θ t − ∂

Original: ˆ
Revised: ˆ

Original: L ( θ t )
Revised: L ( θ t )

Original: ∂θ t , which is the difference between gradient descent and stochastic gradient descent.
Revised: ∂θ t , which is the difference between gradient descent and stochastic gradient descent.

Original: According to Generalized Central Limit Theorem (Gnedenko et al., 1954), the mean of many inﬁnite-variance random variables converges to a stable distribution, while the mean of many ﬁnite-variance random variables converges to a Gaussian distribution.
Revised: According to Generalized Central Limit Theorem (Gnedenko et al., 1954), the mean of many inﬁnite-variance random variables converges to a stable distribution, while the mean of many ﬁnite-variance random variables converges to a Gaussian distribution.

Original: As SGN is ﬁnite in practice, we believe the Gaussian approximation of SGN is reasonable.
Revised: As SGN is ﬁnite in practice, we believe the Gaussian approximation of SGN is reasonable.

Original: Simsekli et al.
Revised: Simsekli et al.

Original: (2019) argued that SGN is Lévy noise (stable variables), rather than Gaussian noise.
Revised: (2019) argued that SGN is Lévy noise (stable variables), rather than Gaussian noise.

Original: They presented empirical evidence showing that SGN seems heavy-tailed, and the heavy-tailed distribution looks closer to a stable distribution than a Gaussian distribution.
Revised: They presented empirical evidence showing that SGN seems heavy-tailed, and the heavy-tailed distribution looks closer to a stable distribution than a Gaussian distribution.

Original: However, this research line (Simsekli et al., 2019; Nguyen et al., 2019) relies on a hidden strict assumption that SGN must be isotropic and obey the same distribution across dimensions.
Revised: However, this research line (Simsekli et al., 2019; Nguyen et al., 2019) relies on a hidden strict assumption that SGN must be isotropic and obey the same distribution across dimensions.

Original: Simsekli et al.
Revised: Simsekli et al.

Original: (2019) computed “SGN” across n model parameters and regarded “SGN" as n samples drawn from a single-variant distribution.
Revised: (2019) computed “SGN” across n model parameters and regarded “SGN" as n samples drawn from a single-variant distribution.

Original: This is why one tail-index for all parameters was studied in Simsekli et al.
Revised: This is why one tail-index for all parameters was studied in Simsekli et al.

Original: The arguments in Simsekli et al.
Revised: The arguments in Simsekli et al.

Original: (2019) did not necessarily hold for parameter-dependent and anisotropic
Revised: (2019) did not necessarily hold for parameter-dependent and anisotropic

Original: Gaussian noise.
Revised: Gaussian noise.

Original: In our paper, SGN computed over different minibatches obeys a n -variant Gaussian distribution, which can be parameter-dependent and anisotropic .
Revised: In our paper, SGN computed over different minibatches obeys a n -variant Gaussian distribution, which can be parameter-dependent and anisotropic .

Original: In Figure 1, we empirically verify that SGN is highly similar to Gaussian noise instead of heavy-tailed
Revised: In Figure 1, we empirically verify that SGN is highly similar to Gaussian noise instead of heavy-tailed

Original: Lévy noise.
Revised: Lévy noise.

Original: We recover the experiment of Simsekli et al.
Revised: We recover the experiment of Simsekli et al.

Original: (2019) to show that gradient noise is approximately Lévy noise only if it is computed across parameters.
Revised: (2019) to show that gradient noise is approximately Lévy noise only if it is computed across parameters.

Original: Figure 1 actually suggests that
Revised: Figure 1 actually suggests that

Original: [Figure]
Revised: [Figure]

Original: the contradicted  are from the different formulations of gradient noise
Revised: the contradicted observations are from the different formulations of gradient noise.

Original: Simsekli et al
Revised: Simsekli et al

Original: (2019) studied the distribution of SGN as a single-variant distribution, while we relax it as a n -variant distribution.
Revised: (2019) studied the distribution of SGN as a single-variant distribution, while we relax it as a n -variant distribution.

Original: Our empirical analysis in Figure 1 holds well at least when the batch size B is larger than 16 , which is common in practice.
Revised: Our empirical analysis in Figure 1 holds well at least when the batch size B is larger than 16 , which is common in practice.

Original: Similar empirical evidence can be observed for training ResNet (He et al., 2016) on CIFAR-10 (Krizhevsky et al., 2009), seen in Appendix C. 
Revised: Similar empirical evidence can be observed for training ResNet (He et al., 2016) on CIFAR-10 (Krizhevsky et al., 2009), seen in Appendix C. Panigrahi et al.

Revised: (2019) also observed that for batch sizes 256 and above, the distribution of SGN is best described as Gaussian at-least in the early phases of training.

Revised: Comparing our results with

Revised: Panigrahi et al.

Revised: (2019), we noticed that the Gaussianity of SGN may depend on more unknown factors.

Original:  The isotropic gradient noise assumption is too rough to capture the Hessian-dependent covariance structure of SGN, which we will study in Figure 2 later.
Revised: First, SGN on random models is more Gaussian than well-trained models. Second, the layer/network matters. Because SGN on some layers/networks is more Gaussian than other layers/networks. The isotropic gradient noise assumption is too rough to capture the Hessian-dependent covariance structure of SGN, which we will study in Figure 2 later.

Original: Our theory that focuses on parameterdependent and anisotropic SGN brings a large improvement over existing parameter-independent and isotropic noise, although Simsekli et al.
Revised: Our theory that focuses on parameterdependent and anisotropic SGN brings a large improvement over existing parameter-independent and isotropic noise, although Simsekli et al.

Original: (2019) brought an improvement over more conventional parameter-independent and isotropic Gaussian noise.
Revised: (2019) brought an improvement over more conventional parameter-independent and isotropic Gaussian noise.

Original: A more sophisticated theory is interesting under parameter-independent anisotropic heavy-tailed noise, when the batch size is too small ( B ∼ 1 ) to apply Central Limit Theorem.
Revised: A more sophisticated theory is interesting under parameter-independent anisotropic heavy-tailed noise, when the batch size is too small ( B ∼ 1 ) to apply Central Limit Theorem.

Original: We will leave it as future work.
Revised: We will leave it as future work.

Original: SGD Dynamics.
Revised: SGD Dynamics.

Original: Let us replace η by dt as unit time.
Revised: Let us replace η by dt as unit time.

Original: Then the continuous-time dynamics of SGD (Coffey & Kalmykov, 2012) is written as
Revised: Then the continuous-time dynamics of SGD (Coffey & Kalmykov, 2012) is written as

Original: [Equation]
Revised: [Equation]

Original: where dW t ∼ N (0 , Idt ) and D ( θ ) =
Revised: where dW t ∼ N (0 , Idt ) and D ( θ ) =

Original: η 2 C ( θ ) .
Revised: η 2 C ( θ ) .

Original: We note that the dynamical time t in the continuoustime dynamics is equal to the product of the number of iterations T and the learning rate η : t = ηT .
Revised: We note that the dynamical time t in the continuoustime dynamics is equal to the product of the number of iterations T and the learning rate η : t = ηT .

Original: The associated Fokker-Planck Equation is written as
Revised: The associated Fokker-Planck Equation is written as

Original: [Equation]
Revised: [Equation]

Original: where ∇ is a nabla operator, and D ij is the element in the i th row and j th column of D .
Revised: where ∇ is a nabla operator, and D ij is the element in the i th row and j th column of D .

Original: In standard
Revised: In standard

Original: SGLD, the injected gradient noise is ﬁxed and isotropic Gaussian, D
Revised: SGLD, the injected gradient noise is ﬁxed and isotropic Gaussian, D

Original: = I .
Revised: = I .

Original: The next question is how to formulate the SGN covariance C ( θ ) for SGD?

Original: Based on Smith & Le (2018), we can express the SGN covariance as

Original: [Equation]

Original: The approximation is true near critical points, due to the fact that the gradient noise variance dominates the gradient mean near critical points.

Original: We know the observed ﬁsher information matrix satisﬁes

Original: [Table]
Revised: [Table]

Original: 0.5 Hessian 0.
Revised: 0.5 Hessian 0.

Original: C o v a r × B a t c h
Revised: C o v a r × B a t c h

Original: S i z e
Revised: S i z e

Original: B=1, Coef:0.8578, Pearson: 0.9709 B=10, Coef:0.8542, Pearson: 0.9717 B=30, Coef:0.8921, Pearson: 0.9671 B=100, Coef:0.8569, Pearson: 0.
Revised: B=1, Coef:0.8578, Pearson: 0.9709 B=10, Coef:0.8542, Pearson: 0.9717 B=30, Coef:0.8921, Pearson: 0.9671 B=100, Coef:0.8569, Pearson: 0.

Original: [Figure]
Revised: [Figure]

Revised: The next question is how to formulate the SGN covariance C ( θ ) for SGD?

Revised: Based on Smith & Le (2018), we can express the SGN covariance as

Revised: [Equation]

Revised: The approximation is true near critical points, due to the fact that the gradient noise variance dominates the gradient mean near critical points.

Revised: We know the observed ﬁsher information matrix satisﬁes

Original: FIM( θ ) ≈ H ( θ ) near minima, referring to Chapter 8 of (Pawitan, 2001).
Revised: FIM( θ ) ≈ H ( θ ) near minima, referring to Chapter 8 of (Pawitan, 2001).

Original: Following Jastrz˛ebski et al.
Revised: Following Jastrz˛ebski et al.

Original: (2017); Zhu et al.
Revised: (2017); Zhu et al.

Original: (2019), we obtain
Revised: (2019), we obtain

Original: [Equation]
Revised: [Equation]

Original: 6) which approximately gives
Revised: which approximately gives

Original: [Equation]
Revised: [Equation]

Original: near minima.
Revised: near minima.

Original: It indicates that the SGN covariance C ( θ ) is approximately proportional to the
Revised: It indicates that the SGN covariance C ( θ ) is approximately proportional to the

Original: Hessian H ( θ ) and inverse to the batch size B .
Revised: Hessian H ( θ ) and inverse to the batch size B .

Original: Obviously, we can generalize Equation 7 by
Revised: Obviously, we can generalize Equation 7 by

Original: D ( θ ) =
Revised: D ( θ ) =

Original: ηC ( θ )=
Revised: ηC ( θ )=

Original: η 2 B [ H ( θ )] + near critical points, when there exist negative eigenvalues in H along some directions.
Revised: η 2 B [ H ( θ )] + near critical points, when there exist negative eigenvalues in H along some directions.

Original: We use [ · ] + to denote the positive semideﬁnite transformation of a symmetric matrix: if we have the eigendecomposation H
Revised: We use [ · ] + to denote the positive semideﬁnite transformation of a symmetric matrix: if we have the eigendecomposation H

Original: =
Revised: =

Original: U diag( H 1 , · · · , H n − 1 , H n ) U (cid:62) , then [ H ] + = U diag( | H 1 | , · · · , | H n − 1 | , | H n | ) U (cid:62) .
Revised: U diag( H 1 , · · · , H n − 1 , H n ) U (cid:62) , then [ H ] + = U diag( | H 1 | , · · · , | H n − 1 | , | H n | ) U (cid:62) .

Original: We empirically verify this relation in Figure 2 for pretrained fully-connected networks, and a followup paper Xie et al.
Revised: We empirically verify this relation in Figure 2 for pretrained fully-connected networks, and a followup paper Xie et al.

Original: (2020c)
Revised: (2020c)

Original: ﬁrst veriﬁed this relation for randomly initialized fully-connected networks on real-world datasets.
Revised: ﬁrst veriﬁed this relation for randomly initialized fully-connected networks on real-world datasets.

Original: The Pearson Correlation is up to 0 .
Revised: The Pearson Correlation is up to 0 .

Original: 999 for pretrained networks.
Revised: 999 for pretrained networks.

Original: We note that, the relation still approximately holds for even the randomly network, which is far from critical points.
Revised: We note that, the relation still approximately holds for even the randomly network, which is far from critical points.

Original: The correlation is especially high along the ﬂat directions with small-magnitude eigenvalues of the
Revised: The correlation is especially high along the ﬂat directions with small-magnitude eigenvalues of the

Original: Hessian (Xie et al., 2020c).
Revised: Hessian (Xie et al., 2020c).

Original: We emphasize that previous papers with the isotropic Lévy or Gaussian noise approximation all failed to capture this core relation in deep learning dynamics.
Revised: We emphasize that previous papers with the isotropic Lévy or Gaussian noise approximation all failed to capture this core relation in deep learning dynamics.

Original:  SGD D IFFUSION T HEORY
Revised:  SGD D IFFUSION T HEORY

Original: We start the theoretical analysis from the classical Kramers Escape Problem (Kramers, 1940).
Revised: We start the theoretical analysis from the classical Kramers Escape Problem (Kramers, 1940).

Original: We assume there are two valleys, Sharp Valley a 1 and Flat Valley a 2 , seen in Figure 3.
Revised: We assume there are two valleys, Sharp Valley a 1 and Flat Valley a 2 , seen in Figure 3.

Original: Also Col b is the boundary between two valleys.
Revised: Also Col b is the − −           ab c a Energy Landscape

Revised: [Figure]

Original:  What is the mean escape time for a particle governed by Equationto escape from Sharp Valley a 1 to Flat Valley a 2 ?
Revised: ab a [Figure] boundary between two valleys. What is the mean escape time for a particle governed by Equationto escape from Sharp Valley a 1 to Flat Valley a 2 ?

Original: The mean escape time is widely used in related statistical physics and stochastic process (Van Kampen, 1992; Nguyen et al., 2019).
Revised: The mean escape time is widely used in related statistical physics and stochastic process (Van Kampen, 1992; Nguyen et al., 2019).

Original: Gauss’s Divergence Theorem (Arfken & Weber, 1999; Lipschutz et al., 2009) states that the surface integral of a vector ﬁeld over a closed surface, which is called the ﬂux through the surface, is equal to the volume integral of the divergence over the region inside the surface.
Revised: Gauss’s Divergence Theorem (Arfken & Weber, 1999; Lipschutz et al., 2009) states that the surface integral of a vector ﬁeld over a closed surface, which is called the ﬂux through the surface, is equal to the volume integral of the divergence over the region inside the surface.

Original: We respectively denote the mean escape time as τ , the escape rate as γ , and the probability current as J .
Revised: We respectively denote the mean escape time as τ , the escape rate as γ , and the probability current as J .

Original: We apply Gauss’s
Revised: We apply Gauss’s

Original: Divergence Theorem to the Fokker-Planck Equation resulting in
Revised: Divergence Theorem to the Fokker-Planck Equation resulting in

Original: [Equation]
Revised: [Equation]

Original: − −           ab c a

Original: Energy Landscape (a)

Original: [Figure]

Original: ab a [Figure] The mean escape time is expressed (Van Kampen, 1992) as 
Revised: The mean escape time is expressed (Van Kampen, 1992) as [Equation]

Original: [Equation]
Revised: [Equation]

Original: where P ( θ ∈ V a )
Revised: where P ( θ ∈ V a )

Original: =
Revised: =

Original: (cid:82) V a P ( θ ) dV is the current probability inside Valley a, J is the probability current produced by the probability source P ( θ ∈ V a ) , j = (cid:82) S a J · dS is the probability ﬂux (surface integrals of probability current), S a is the surface (boundary) surrounding Valley a, and V a is the volume surrounded by S a .
Revised: (cid:82) V a P ( θ ) dV is the current probability inside Valley a, J is the probability current produced by the probability source P ( θ ∈ V a ) , j = (cid:82) S a J · dS is the probability ﬂux (surface integrals of probability current), S a is the surface (boundary) surrounding Valley a, and V a is the volume surrounded by S a .

Original: We have j = J in the case of one-dimensional escape.
Revised: We have j = J in the case of one-dimensional escape.

Original: Classical Assumptions.
Revised: Classical Assumptions.

Original: We state three classical assumptions ﬁrst for the density diffusion theory.
Revised: We state three classical assumptions ﬁrst for the density diffusion theory.

Original: Assumption 1 is the common second order Taylor approximation, which was also used by (Mandt et al., 2017; Zhang et al., 2019).
Revised: Assumption 1 is the common second order Taylor approximation, which was also used by (Mandt et al., 2017; Zhang et al., 2019).

Original: Assumptions 2 and 3 are widely used in many ﬁelds’ Kramers
Revised: Assumptions 2 and 3 are widely used in many ﬁelds’ Kramers

Original: Escape Problems, including statistical physics (Kramers, 1940; Hanggi, 1986), chemistry (Eyring, 1935;
Revised: Escape Problems, including statistical physics (Kramers, 1940; Hanggi, 1986), chemistry (Eyring, 1935;

Original: Hänggi et al., 1990), biology (Zhou, 2010), electrical engineering (Coffey & Kalmykov, 2012), and stochastic process (Van Kampen, 1992; Berglund, 2013).
Revised: Hänggi et al., 1990), biology (Zhou, 2010), electrical engineering (Coffey & Kalmykov, 2012), and stochastic process (Van Kampen, 1992; Berglund, 2013).

Original: Related machine learning papers (Jastrz˛ebski et al., 2017) usually used Assumptions 2 and 3 as the background of Kramers Escape
Revised: Related machine learning papers (Jastrz˛ebski et al., 2017) usually used Assumptions 2 and 3 as the background of Kramers Escape

Original: Problems.
Revised: Problems.

Original: Assumption 1 (The Second Order Taylor Approximation) .
Revised: Assumption 1 (The Second Order Taylor Approximation) .

Original: The loss function around critical points θ (cid:63) can be approximately written as
Revised: The loss function around critical points θ (cid:63) can be approximately written as

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: Assumption 2 (Quasi-Equilibrium Approximation) .
Revised: Assumption 2 (Quasi-Equilibrium Approximation) .

Original: The system is in quasi-equilibrium near minima.
Revised: The system is in quasi-equilibrium near minima.

Original: Assumption 3 (Low Temperature Approximation) .
Revised: Assumption 3 (Low Temperature Approximation) .

Original: The gradient noise is small (low temperature).
Revised: The gradient noise is small (low temperature).

Original: We will dive into these two assumptions deeper than previous papers for SGD dynamics.
Revised: We will dive into these two assumptions deeper than previous papers for SGD dynamics.

Original: Assumptions 2 and 3 both mean that our diffusion theory can better describe the escape processes that cost more iterations.
Revised: Assumptions 2 and 3 both mean that our diffusion theory can better describe the escape processes that cost more iterations.

Original: As this class of “slow” escape processes takes main computational time compared with “fast” escape processes, this class of “slow” escape process is more interesting for training of deep neural networks.
Revised: As this class of “slow” escape processes takes main computational time compared with “fast” escape processes, this class of “slow” escape process is more interesting for training of deep neural networks.

Original: Our empirical analysis in Section 4 supports that the escape processes in the wide range of iterations (50 to 100,000 iterations) can be modeled by our theory very well.
Revised: Our empirical analysis in Section 4 supports that the escape processes in the wide range of iterations (50 to 100,000 iterations) can be modeled by our theory very well.

Original: Thus,
Revised: Thus,

Original: Assumption 2 and 3 are reasonable in practice.
Revised: Assumption 2 and 3 are reasonable in practice.

Original: More discussion can be found in Appendix B.
Revised: More discussion can be found in Appendix B.

Original: Escape paths.
Revised: Escape paths.

Original: We generalize the concept of critical points into critical paths as the path where 1) the gradient perpendicular to the path direction must be zero, and 2) the second order directional derivatives perpendicular to the path direction must be nonnegative.
Revised: We generalize the concept of critical points into critical paths as the path where 1) the gradient perpendicular to the path direction must be zero, and 2) the second order directional derivatives perpendicular to the path direction must be nonnegative.

Original: The Most Possible Paths (MPPs) for escaping must be critical paths.
Revised: The Most Possible Paths (MPPs) for escaping must be critical paths.

Original: The most possible escape direction at one point must be the direction of one eigenvector of the Hessian at the point.
Revised: The most possible escape direction at one point must be the direction of one eigenvector of the Hessian at the point.

Original: Under Assumption 3, the probability density far from critical points and MPPs is very small.
Revised: Under Assumption 3, the probability density far from critical points and MPPs is very small.

Original: Thus, the density diffusion will concentrate around
Revised: Thus, the density diffusion will concentrate around

Original: MPPs.
Revised: MPPs.

Original: Draxler et al.
Revised: Draxler et al.

Original: (2018) reported that minima in the loss landscape of deep networks are connected by Minimum Energy Paths (MEPs) that are essentially ﬂat and Local MEPs that have high-loss saddle points.
Revised: (2018) reported that minima in the loss landscape of deep networks are connected by Minimum Energy Paths (MEPs) that are essentially ﬂat and Local MEPs that have high-loss saddle points.

Original: Obviously, MPPs in our paper correspond to Local MEPs.
Revised: Obviously, MPPs in our paper correspond to Local MEPs.

Original: The density diffusion along MEPs, which are strictly ﬂat, is ignorable according to our following analysis.
Revised: The density diffusion along MEPs, which are strictly ﬂat, is ignorable according to our following analysis.

Original: The boundary between Sharp Valley a 1 and Flat Valley a 2 is the saddle point b .
Revised: The boundary between Sharp Valley a 1 and Flat Valley a 2 is the saddle point b .

Original: The Hessian at b , H b , must have only one negative eigenvalue and the corresponding eigenvector is the escape direction.
Revised: The Hessian at b , H b , must have only one negative eigenvalue and the corresponding eigenvector is the escape direction.

Original: Without losing generality, we ﬁrst assume that there is only one most possible path through Col b existing between Sharp Valley a 1 and Flat Valley a 2 .
Revised: Without losing generality, we ﬁrst assume that there is only one most possible path through Col b existing between Sharp Valley a 1 and Flat Valley a 2 .

Original: SGLD diffusion.
Revised: SGLD diffusion.

Original: We ﬁrst analyze a simple case: how does SGLD escape sharp minima?
Revised: We ﬁrst analyze a simple case: how does SGLD escape sharp minima?

Original: Researchers are interested in SGLD, when the injected noise dominates SGN as η → 0 in ﬁnal epochs.
Revised: Researchers are interested in SGLD, when the injected noise dominates SGN as η → 0 in ﬁnal epochs.

Original: Because
Revised: Because

Original: SGLD may work as a Bayesian inference method in this limit (Welling & Teh, 2011).
Revised: SGLD may work as a Bayesian inference method in this limit (Welling & Teh, 2011).

Original: SGLD is usually simpliﬁed as Gradient Descent with injected white noise, whose behavior is identical to
Revised: SGLD is usually simpliﬁed as Gradient Descent with injected white noise, whose behavior is identical to

Original: Kramers Escape Problem with thermo noise in statistical physics.
Revised: Kramers Escape Problem with thermo noise in statistical physics.

Original: We present Theorem 3.1.
Revised: We present Theorem 3.1.

Original: We leave the proof in Appendix A.1.
Revised: We leave the proof in Appendix A.1.

Original: We also note that more precise SGLD diffusion analysis should study a mixture of injected white noise and SGN.
Revised: We also note that more precise SGLD diffusion analysis should study a mixture of injected white noise and SGN.

Original: Theorem 3.1 (SGLD Escapes Minima) .
Revised: Theorem 3.1 (SGLD Escapes Minima) .

Original: The loss function L ( θ ) is of class C 2 and n-dimensional.
Revised: The loss function L ( θ ) is of class C 2 and n-dimensional.

Original: Only one most possible path exists between Valley a and the outside of Valley a.
Revised: Only one most possible path exists between Valley a and the outside of Valley a.

Original: If Assumption 1, 2, and 3 hold, and the dynamics is governed by SGLD, then the mean escape time from Valley a to the outside of Valley a is
Revised: If Assumption 1, 2, and 3 hold, and the dynamics is governed by SGLD, then the mean escape time from Valley a to the outside of Valley a is

Original: [Equation]
Revised: [Equation]

Original: We denote that H a
Revised: We denote that H a

Original: and H b are the Hessians of the loss function at the minimum a and the saddle point b , ∆ L = L ( b ) − L ( a ) is the loss barrier height, e indicates the escape direction, and H be is the eigenvalue of the Hessian H b corresponding to the escape direction.
Revised: and H b are the Hessians of the loss function at the minimum a and the saddle point b , ∆ L = L ( b ) − L ( a ) is the loss barrier height, e indicates the escape direction, and H be is the eigenvalue of the Hessian H b corresponding to the escape direction.

Original: The diffusion coefﬁcient D is usually set to 1 in SGLD.
Revised: The diffusion coefﬁcient D is usually set to 1 in SGLD.

Original: SGD diffusion.
Revised: SGD diffusion.

Original: However, SGD diffusion is essentially different from SGLD diffusion in several aspects: 1) anisotropic noise, 2) parameter-dependent noise, and 3) the stationary distribution of SGD is far from the Gibs-Boltzmann distribution, P ( θ )
Revised: However, SGD diffusion is essentially different from SGLD diffusion in several aspects: 1) anisotropic noise, 2) parameter-dependent noise, and 3) the stationary distribution of SGD is far from the Gibs-Boltzmann distribution, P ( θ )

Original: = 1 Z exp (cid:16) − L ( θ ) D (cid:17) .
Revised: = 1 Z exp (cid:16) − L ( θ ) D (cid:17) .

Original: These different characteristics make SGD diffusion behave differently from known physical dynamical systems and much less studied than SGLD diffusion.
Revised: These different characteristics make SGD diffusion behave differently from known physical dynamical systems and much less studied than SGLD diffusion.

Original: We formulate Theorem 3.2 for SGD.
Revised: We formulate Theorem 3.2 for SGD.

Original: We leave the proof in Appendix
Revised: We leave the proof in Appendix

Original: A.2.The theoretical analysis of SGD can be easily generalized to the dynamics with a mixture of SGN and injected white noise, as long as the eigenvectors of D ( θ ) are closely aligned with the eigenvectors of H ( θ ) .
Revised: A.2.The theoretical analysis of SGD can be easily generalized to the dynamics with a mixture of SGN and injected white noise, as long as the eigenvectors of D ( θ ) are closely aligned with the eigenvectors of H ( θ ) .

Original: Theorem 3.2 (SGD Escapes Minima) .
Revised: Theorem 3.2 (SGD Escapes Minima) .

Original: The loss function L ( θ ) is of class C 2 and n-dimensional.
Revised: The loss function L ( θ ) is of class C 2 and n-dimensional.

Original: Only one most possible path exists between Valley a and the outside of Valley a.
Revised: Only one most possible path exists between Valley a and the outside of Valley a.

Original: If Assumption 1, 2, and 3 hold, and the dynamics is governed by SGD, then the mean escape time from Valley a to the outside of Valley a is
Revised: If Assumption 1, 2, and 3 hold, and the dynamics is governed by SGD, then the mean escape time from Valley a to the outside of Valley a is

Original: [Equation]
Revised: [Equation]

Original: where s ∈ (0 , 1) is a path-dependent parameter, and H ae and H be are, respectively, the eigenvalues of the Hessians at the minimum a
Revised: where s ∈ (0 , 1) is a path-dependent parameter, and H ae and H be are, respectively, the eigenvalues of the Hessians at the minimum a

Original: and the saddle point b corresponding to the escape direction e .
Revised: and the saddle point b corresponding to the escape direction e .

Original: Multiple-path escape.
Revised: Multiple-path escape.

Original: Each escape path contributes to the total escape rate.
Revised: Each escape path contributes to the total escape rate.

Original: Multiple paths combined together have a total escape rate.
Revised: Multiple paths combined together have a total escape rate.

Original: If there are multiple parallel from the start valley to the end valley, we can compute the total escape rate easily based on the following computation rule.
Revised: If there are multiple parallel from the start valley to the end valley, we can compute the total escape rate easily based on the following computation rule.

Original: The computation  rule is based on the fact that the probability ﬂux integrals are additive.
Revised: The computation [Figure] rule is based on the fact that the probability ﬂux integrals are additive.

Original: We can easily generalize the mean escape time analysis into the cases that there are multiple parallel escape paths indexed by p .
Revised: We can easily generalize the mean escape time analysis into the cases that there are multiple parallel escape paths indexed by p .

Original: As for multiple-valley escape problems, we can always reduce a multiple-valley escape problem into multiple two-valley escape problems.
Revised: As for multiple-valley escape problems, we can always reduce a multiple-valley escape problem into multiple two-valley escape problems.

Original: We also note that, while Theorem A.2 does not depend the dimensionality directly, higher dimensionality may increase the number of escape paths and loss valleys, and change the spectrum of the Hessians.
Revised: We also note that, while Theorem A.2 does not depend the dimensionality directly, higher dimensionality may increase the number of escape paths and loss valleys, and change the spectrum of the Hessians.

Original: Rule 1.
Revised: Rule 1.

Original: If there are multiple MPPs between the start valley and the end valley, then γ total = (cid:80) p γ p .
Revised: If there are multiple MPPs between the start valley and the end valley, then γ total = (cid:80) p γ p .

Original: Thus, we only need to ﬁnd the saddle points that connect two valleys as we analyzed in the paper and analyze the escape rates.
Revised: Thus, we only need to ﬁnd the saddle points that connect two valleys as we analyzed in the paper and analyze the escape rates.

Original: Minima selection.
Revised: Minima selection.

Original: Now, we may formulate the probability of minima selection as Proposition 1.
Revised: Now, we may formulate the probability of minima selection as Proposition 1.

Original: We leave the proof in Appendix A.3.
Revised: We leave the proof in Appendix A.3.

Original: In deep learning, one loss valley represents one mode and[Figure] [Figure] the landscape contain many good modes and bad modes.
Revised: In deep learning, one loss valley represents one mode and the landscape contain many good modes and bad modes.

Original: SGD transits from one mode to another mode during training.
Revised: SGD transits from one mode to another mode during training.

Original: The mean escape time of one mode corresponds to the number of iterations which SGD spends on this mode during training, which is naturally proportional to the probability of selecting this mode after training.
Revised: The mean escape time of one mode corresponds to the number of iterations which SGD spends on this mode during training, which is naturally proportional to the probability of selecting this mode after training.

Original: Proposition 1.
Revised: Proposition 1.

Original: Suppose there are two valleys connected by an escape path.
Revised: Suppose there are two valleys connected by an escape path.

Original: If all assumptions of
Revised: If all assumptions of

Original: Theorem 3.2 hold, then the stationary distribution of locating these valleys is given by
Revised: Theorem 3.2 hold, then the stationary distribution of locating these valleys is given by

Original: [Equation]
Revised: [Equation]

Original: where v is the index of valleys, and τ v is the mean escape time from Valley v to the outside of Valley v .
Revised: where v is the index of valleys, and τ v is the mean escape time from Valley v to the outside of Valley v .

Original:  E MPIRICAL A NALYSIS
Revised:  E MPIRICAL A NALYSIS

Original: In this section, we try to directly validate the escape formulas on real-world datasets.
Revised: In this section, we try to directly validate the escape formulas on real-world datasets.

Original: Each escape process,from the inside of loss valleys to the outside of loss valleys, are repeatedly simulated fortimes under various gradient noise scales,batch sizes, learning rates, and sharpness.
Revised: Each escape process, noise scales,

Original: How to compare the escape rates under the same settings with various minima sharpness?
Revised: How to compare the escape rates under the same settings with various minima sharpness?

Original: Our method is to multiply a rescaling factor √ k to each parameter, and the Hessian will be proportionally rescaled by a factor k .
Revised: Our method is to multiply a rescaling factor √ k to each parameter, and the Hessian will be proportionally rescaled by a factor k .

Original: If we let L ( θ ) =
Revised: If we let L ( θ ) =

Original: f ( θ ) →
Revised: f ( θ ) →

Original: L ( θ ) =
Revised: L ( θ ) =

Original: f ( √ kθ ) , then H ( θ )
Revised: f ( √ kθ ) , then H ( θ )

Original: = ∇ 2 f ( θ ) → H ( θ )
Revised: = ∇ 2 f ( θ ) → H ( θ )

Original: =
Revised: =

Original: k ∇ 2 f ( θ ) .
Revised: k ∇ 2 f ( θ ) .

Original: Thus, we can use k to indicate the minima sharpness.
Revised: Thus, we can use k to indicate the minima sharpness.

Original: The theoretical relations of SGD we try to validate can be formulated as: (1) − log( γ ) =
Revised: The theoretical relations of SGD we try to validate can be formulated as: (1) − log( γ ) =

Original: O ( 1 k ) , (2) − log( γ ) =
Revised: O ( 1 k ) , (2) − log( γ ) =

Original: O ( B ) , and (3) − log( γ ) =
Revised: O ( B ) , and (3) − log( γ ) =

Original: O ( 1 η ) .
Revised: O ( 1 η ) .

Original: [Figure]

Original: The mean escape time analysis of SGD.
Revised: The mean escape time analysis of SGD.

Original: Styblinski-Tang Function, which has multiple minima and saddle points, is a common test function for nonconvex optimization.
Revised: Styblinski-Tang Function, which has multiple minima and saddle points, is a common test function for nonconvex optimization.

Original: We conduct an intuitional 10-dimensional experiment, where the simulations start from a given minimum and terminate when reaching the boundary of the loss valley.
Revised: We conduct an intuitional 10-dimensional experiment, where the simulations start from a given minimum and terminate when reaching the boundary of the loss valley.

Original: The number of iterations is recorded for calculating  the escape rate.
Revised: The number of iterations is recorded for calculating [Figure] [Figure] the escape rate.

Original: We also train fully connected networks on four real-world datasets, including a)
Revised: We also train fully connected networks on four real-world datasets, including a)

Original: Avila, b) Banknote Authentication, c) Cardiotocography, d) Dataset for Sensorless Drive Diagnosis (De Stefano et al., 2018; Dua & Graff, 2017).
Revised: Avila, b) Banknote Authentication, c) Cardiotocography, d) Dataset for Sensorless Drive Diagnosis (De Stefano et al., 2018; Dua & Graff, 2017).

Original: Figure 4 and Figure 5 clearly veriﬁes that the escape rate exponentially depends on the minima sharpness (reﬂected by k ), the batch size, and the learning rate on both test functions and real-world training, which fully supports our theoretical results.
Revised: Figure 4 and Figure 5 clearly veriﬁes that the escape rate exponentially depends on the minima sharpness (reﬂected by k ), the batch size, and the learning rate on both test functions and real-world training, which fully supports our theoretical results.

Original: Model architecture and details
Revised: Model architecture and details

Original: : We used fully-connected networks with the depth 2 and the widthin Figure 5.
Revised: : We used fully-connected networks with the depth 2 and the widthin Figure 5.

Original: The experiments using Logistic Regression and Fully-connected networks with the depth 3 are presented in
Revised: The experiments using Logistic Regression and Fully-connected networks with the depth 3 are presented in

Original: Appendix E. We leave more experimental details and results in Appendix D.1 and
Revised: Appendix E. We leave more experimental details and results in Appendix D.1 and

Original: Appendix E.
Revised: Appendix E.

Original: The mean escape time analysis of SGLD.
Revised: The mean escape time analysis of SGLD.

Original: We try to validate γ
Revised: We try to validate γ

Original: =
Revised: =

Original: O ( k ) and − log( γ )
Revised: O ( k ) and − log( γ )

Original: =
Revised: =

Original: O ( 1 D ) for
Revised: O ( 1 D ) for

Original: SGLD (dominated by injected Gaussian noise).
Revised: SGLD (dominated by injected Gaussian noise).

Original: Figure 6 shows that SGLD only favors ﬂat minima polynomially more than sharp minima as Theorem 3.1 indicates.
Revised: Figure 6 shows that SGLD only favors ﬂat minima polynomially more than sharp minima as Theorem 3.1 indicates.

Original: Figure 6 also veriﬁes that the injected gradient noise scale exponentially affects ﬂat minima selection.
Revised: Figure 6 also veriﬁes that the injected gradient noise scale exponentially affects ﬂat minima selection.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: SGD favors ﬂat minima exponentially more than sharp minima.
Revised: SGD favors ﬂat minima exponentially more than sharp minima.

Original: We can discover a few interesting insights about SGD by Theorem 3.2.
Revised: We can discover a few interesting insights about SGD by Theorem 3.2.

Original: Most importantly, the mean escape time exponentially depends on the eigenvalue of the Hessian at minima along the escape direction, H ae .
Revised: Most importantly, the mean escape time exponentially depends on the eigenvalue of the Hessian at minima along the escape direction, H ae .

Original: Thus, SGD favors ﬂat minima exponentially more than sharp minima.
Revised: Thus, SGD favors ﬂat minima exponentially more than sharp minima.

Original: We claim one main advantage of SGD comes from the exponential relation of the mean escape time and the minima sharpness.
Revised: We claim one main advantage of SGD comes from the exponential relation of the mean escape time and the minima sharpness.

Original: The measure of “sharpness” has reformed in contexts of SGLD and SGD.
Revised: The measure of “sharpness” has reformed in contexts of SGLD and SGD.

Original: In the context of SGLD, the “sharpness” is quantiﬁed by the determinant of the Hessian.
Revised: In the context of SGLD, the “sharpness” is quantiﬁed by the determinant of the Hessian.

Original: In the context of SGD, the “sharpness” is quantiﬁed by the top eigenvalues of the Hessian along the escape direction.
Revised: In the context of SGD, the “sharpness” is quantiﬁed by the top eigenvalues of the Hessian along the escape direction.

Original: Based on the proposed diffusion theory, recent work (Xie et al., 2020c) successfully proved that SGD favors ﬂat minima signiﬁcantly more than Adam.
Revised: Based on the proposed diffusion theory, recent work (Xie et al., 2020c) successfully proved that SGD favors ﬂat minima signiﬁcantly more than Adam.

Original: The ratio of the batch size and the learning rate exponentially matters.
Revised: The ratio of the batch size and the learning rate exponentially matters.

Original: Theorem 3.2 explains why large-batch training can easily get trapped near sharp minima, and increasing the learning rate proportionally is helpful for large-batch training (Krizhevsky, 2014; Keskar et al., 2017;
Revised: Theorem 3.2 explains why large-batch training can easily get trapped near sharp minima, and increasing the learning rate proportionally is helpful for large-batch training (Krizhevsky, 2014; Keskar et al., 2017;

Original: Sagun et al., 2017; Smith et al., 2018; Yao et al., 2018;
Revised: Sagun et al., 2017; Smith et al., 2018; Yao et al., 2018;

Original: He et al., 2019a).
Revised: He et al., 2019a).

Original: We argue that the main cause is large-batch training expects exponentially longer time to escape minima.
Revised: We argue that the main cause is large-batch training expects exponentially longer time to escape minima.

Original: Note that, as the mean escape time in the theorems is equivalent to the product of the learning rate and the number of iterations, both the number of iterations and dynamical time exponentially depend on the ratio of the batch size and the learning rate.
Revised: Note that, as the mean escape time in the theorems is equivalent to the product of the learning rate and the number of iterations, both the number of iterations and dynamical time exponentially depend on the ratio of the batch size and the learning rate.

Original: The practical computational time in large-batch training is usually too short to search many enough ﬂat minima.
Revised: The practical computational time in large-batch training is usually too short to search many enough ﬂat minima.

Original: We conjecture that exponentially increasing training iterations may be helpful for large batch training, while this is often too expensive in practice.
Revised: We conjecture that exponentially increasing training iterations may be helpful for large batch training, while this is often too expensive in practice.

Original: Low dimensional diffusion.
Revised: Low dimensional diffusion.

Original: Most eigenvalues of the Hessian at the loss landscape of overparametrized deep networks are close to zero, while only a small number of eigenvalues are large (Sagun et al., 2017; Li et al., 2018).
Revised: Most eigenvalues of the Hessian at the loss landscape of overparametrized deep networks are close to zero, while only a small number of eigenvalues are large (Sagun et al., 2017; Li et al., 2018).

Original: Zero eigenvalues indicate zero diffusion along the corresponding directions.
Revised: Zero eigenvalues indicate zero diffusion along the corresponding directions.

Original: Thus, we may theoretically ignore these zero-eigenvalue directions.
Revised: Thus, we may theoretically ignore these zero-eigenvalue directions.

Original: This also indicates that the density diffusion is ignorable along an essentially ﬂat MEP in Draxler et al.
Revised: This also indicates that the density diffusion is ignorable along an essentially ﬂat MEP in Draxler et al.

Original: As the escape rate exponentially depends the corresponding eigenvalues, a small number of large eigenvalues means that the process of minima selection mainly happens in the relatively low dimensional subspace corresponding to top eigenvalues of the Hessian.
Revised: As the escape rate exponentially depends the corresponding eigenvalues, a small number of large eigenvalues means that the process of minima selection mainly happens in the relatively low dimensional subspace corresponding to top eigenvalues of the Hessian.

Original: Gur-Ari et al.
Revised: Gur-Ari et al.

Original: (2018) also reported a similar ﬁnding.
Revised: (2018) also reported a similar ﬁnding.

Original: Although the parameter space is very high-dimensional, SGD dynamics hardly depends on those “meaningless” dimensions with small second order directional derivatives.
Revised: Although the parameter space is very high-dimensional, SGD dynamics hardly depends on those “meaningless” dimensions with small second order directional derivatives.

Original: This novel characteristic of SGD signiﬁcantly reduces the explorable parameter space around one minimum into a much lower dimensional space.
Revised: This novel characteristic of SGD signiﬁcantly reduces the explorable parameter space around one minimum into a much lower dimensional space.

Original: High-order effects.
Revised: High-order effects.

Original: As we have applied the second-order Taylor approximation near critical points, our SGD diffusion theory actually excludes the third-order and higher-order effect.
Revised: As we have applied the second-order Taylor approximation near critical points, our SGD diffusion theory actually excludes the third-order and higher-order effect.

Original: The asymmetric valley in He et al.
Revised: The asymmetric valley in He et al.

Original: (2019b), which only appears in high-order analysis, is beyond the scope of this paper.
Revised: (2019b), which only appears in high-order analysis, is beyond the scope of this paper.

Original: However, we also argue that the third-order effect is much smaller than the second-order effect under the low temperature assumption in Kramers Escape Problems.
Revised: However, we also argue that the third-order effect is much smaller than the second-order effect under the low temperature assumption in Kramers Escape Problems.

Original: We will leave the more reﬁned high-order theory as future work.
Revised: We will leave the more reﬁned high-order theory as future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we demonstrate that one essential advantage of SGD is selecting ﬂat minima with an exponentially higher probability than sharp minima.
Revised: In this paper, we demonstrate that one essential advantage of SGD is selecting ﬂat minima with an exponentially higher probability than sharp minima.

Original: To the best of our knowledge, we are the ﬁrst to formulate the exponential relation of minima selection to the minima sharpness, the batch size, and the learning rate.
Revised: To the best of our knowledge, we are the ﬁrst to formulate the exponential relation of minima selection to the minima sharpness, the batch size, and the learning rate.

Original: Our work bridges the gap between the qualitative knowledge and the quantitative theoretical knowledge on the minima selection mechanism of SGD.
Revised: Our work bridges the gap between the qualitative knowledge and the quantitative theoretical knowledge on the minima selection mechanism of SGD.

Original: We believe the proposed theory not only helps us understand how SGD selects ﬂat minima, but also will provide researchers a powerful theoretical tool to analyze more learning behaviors and design better optimizers in future.
Revised: We believe the proposed theory not only helps us understand how SGD selects ﬂat minima, but also will provide researchers a powerful theoretical tool to analyze more learning behaviors and design better optimizers in future.

Original: A CKNOWLEDGEMENT
Revised: A CKNOWLEDGEMENT

Original: We thanks Dr. Yuanqian Tang for helpful discussion.
Revised: We thanks Dr. Yuanqian Tang for helpful discussion

Original: MS was supportedby the International Research 
Revised: MS was supported Research Center for

Original: Center for Neurointelligence (WPI-IRCN) at The University ofTokyo Institutes for Advanced Study.
Revised: University of Institutes for Study

Original: A P ROOFS A. P ROOF OF T HEOREM 3. 
Revised: A P ROOFS A. P ROOF OF T HEOREM 3.

Original: Proof.
Revised: Proof.

Original: This proposition is a well known conclusion in statistical physics under Assumption 1, 2 and 3.
Revised: This proposition is a well known conclusion in statistical physics under Assumption 1, 2 and 3.

Original: We still provide an intuitional proof here, and the following proof of SGD Diffusion will closely relate to this proof.
Revised: We still provide an intuitional proof here and the following proof of SGD Diffusion will closely relate to this proof.

Original: We decompose the proof into two steps: 1) compute the probability of locating in valley a, P ( θ ∈ V a ), and 2) compute the probability ﬂux j = (cid:82) S a J · dS .
Revised: We decompose the proof into two steps: 1) compute the probability of locating in valley a, P ( θ ∈ V a ) and 2) compute the probability ﬂux j = (cid:82) S a J · dS .

Original: Without losing generality, we ﬁrst prove the one-dimensional case.
Revised: Without losing generality we ﬁrst prove the one-dimensional case

Original: Step 1: Under Assumption 1, the stationary distribution around minimum
Revised: Step 1: Under Assumption 1, the stationary distribution around minimum

Original: a is P ( θ ) =
Revised: a is P ( θ ) =

Original: P ( a ) exp[ − L ( θ ) − L ( a ) T ] , where T = D .
Revised: P ( a ) exp[ − L ( θ ) − L ( a ) T ] , where T = D .

Original: Under Assumption 3, we may only consider the second order Taylor approximation of the density function around critical points.
Revised: Under Assumption 3, we may only consider the second order Taylor approximation of the density function around critical points.

Original: We use the T notation as the temperature parameter in the stationary distribution, and use the D notation as the diffusion coefﬁcient in the dynamics, for their different roles. 
Revised: We use the T notation as the temperature parameter in the stationary distribution, and use the D notation as the diffusion coefﬁcient in the dynamics, for their different roles. [Equation] ∈ V a P

Original: [Equation]
Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: [Equation]

Revised: θ ∈

Revised: [Equation]

Revised: (cid:34)

Revised: [Equation]

Revised: T (cid:35) dθ (13) =

Revised: P ( a )(2 πT )

Revised: H 12 a .

Original: Step 2:
Revised: Step 2:

Original: [Equation]
Revised: [Equation]

Original: Apply this result to the Fokker-Planck Equation 4, we have
Revised: Apply this result to the Fokker-Planck Equation 4, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: And thus we obtain the Smoluchowski equation and a new form of J
Revised: And thus we obtain the Smoluchowski equation and a new form of J

Original: [Equation]
Revised: [Equation]

Original: We note that the probability density outside Valley a must be zero, P ( c ) = 0 .
Revised: We note that the probability density outside Valley a must be zero, P ( c ) = 0 .

Original: As we want to compute the probability ﬂux escaping from Valley a in the proof, the probability ﬂux escaping from other valleys into Valley a should be ignored.
Revised: As we want to compute the probability ﬂux escaping from Valley a in the proof, the probability ﬂux escaping from other valleys into Valley a should be ignored.

Original: Under Assumption 2, we integrate the equation from Valley a to the outside of Valley a along the most possible escape path
Revised: Under Assumption 2, we integrate the equation from Valley a to the outside of Valley a along the most possible escape path

Original: [Equation]
Revised: [Equation]

Original: We move J to the outside of integral based on Gauss’s Divergence Theorem [Equation] because J is ﬁxed on the escape path from one minimum to another.
Revised: We move J to the outside of integral based on Gauss’s Divergence Theorem, because J is ﬁxed on the escape path from one minimum to another.

Original: As there is no ﬁeld source on the escape path, (cid:82)
Revised: As there is no ﬁeld source on the escape path, (cid:82)

Original: V ∇ · J ( θ ) dV = 0 .
Revised: V ∇ · J ( θ ) dV = 0 .

Original: Then ∇ J ( θ ) = 0 .
Revised: Then ∇ J ( θ ) = 0 .

Original: Obviously, only minima are probability sources in deep learning.
Revised: Obviously, only minima are probability sources in deep learning.

Original: Under Assumption 3 and the second-order Taylor approximation
Revised: Under Assumption 3 and the second-order Taylor approximation, we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: Based on the results of Step 1 and Step

Original: [Equation]
Revised: [Equation]

Original: Based on the results of Step 1 and Step 2, we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: We generalize the proof of one-dimensional diffusion to high-dimensional diffusion
Revised: We generalize the proof of one-dimensional diffusion to high-dimensional diffusion

Original: Step 1:
Revised: Step 1:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Step 2:
Revised: Step 2:

Original: Based on the formula of the one-dimensional probability current and ﬂux, we obtain 
Revised: Based on the formula of the one-dimensional probability current and ﬂux, we obtain (cid:90)

Original: [Equation]
Revised: [Equation]

Original: So we have
Revised: So we have

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: Proof.
Revised: Proof.

Original: We decompose the proof into two steps and analyze the one-dimensional case like before.
Revised: We decompose the proof into two steps and analyze the one-dimensional case like before.

Original: The following proof is similar to the proof of SGLD except that we make T a the temperature near the minimum a and T b the temperature near the saddle point b.
Revised: The following proof is similar to the proof of SGLD except that we make T a the temperature near the minimum a and T b the temperature near the saddle point b.

Original: One-dimensional SGD Diffusion:
Revised: One-dimensional SGD Diffusion:

Original: Step 1: Under Assumption 3, we may only consider the second order Taylor approximation of the density function around critical points.
Revised: Step 1: Under Assumption 3, we may only consider the second order Taylor approximation of the density function around critical points.

Original: [Equation]
Revised: [Equation]

Original: Step 2:
Revised: Step 2:

Original: [Equation]
Revised: [Equation]

Original: According to Equation 7, ∇ (cid:0) 1 T (cid:1) is ignorable near the minimum a and the col b, thus
Revised: According to Equation 7, ∇ (cid:0) 1 T (cid:1) is ignorable near the minimum a and the col b, thus

Original: [Equation]
Revised: [Equation]

Original: Apply this result to the Fokker-Planck Equation 4, we have
Revised: Apply this result to the Fokker-Planck Equation 4, we have

Original: [Equation]
Revised: [Equation]

Original: And thus we obtain the Smoluchowski equation and a new form of J
Revised: And thus we obtain the Smoluchowski equation and a new form of J

Original: [Equation]
Revised: [Equation]

Original: We note that the Smoluchowski equation is true only near critical points.
Revised: We note that the Smoluchowski equation is true only near critical points.

Original: We assume the point s is the midpoint on the most possible path between a and b, where L ( s ) =
Revised: We assume the point s is the midpoint on the most possible path between a and b, where L ( s ) =

Original: (1 − s )
Revised: (1 − s )

Original: L ( a ) + sL ( b ) .
Revised: L ( a ) + sL ( b ) .

Original: The temperature T a dominates the path a → s , while temperature T b dominates the path s
Revised: The temperature T a dominates the path a → s , while temperature T b dominates the path s

Original: → b .
Revised: → b .

Original: So we have
Revised: So we have

Original: [Equation]
Revised: [Equation]

Original: Under Assumption [Equation] we integrate the equation from Valley a to the outside of Valley a along the most possible escape path
Revised: Under Assumption 2, we integrate the equation from Valley a to the outside of Valley a along the most possible escape path

Original: [Equation]
Revised: [Equation]

Original: We move J to the outside of integral based on Gauss’s Divergence Theorem, because J is ﬁxed on the escape path from one minimum to another.
Revised: We move J to the outside of integral based on Gauss’s Divergence Theorem, because J is ﬁxed on the escape path from one minimum to another.

Original: As there is no ﬁeld source on the escape path, (cid:82)
Revised: As there is no ﬁeld source on the escape path, (cid:82)

Original: V ∇· J ( θ ) dV = 0 and ∇ J ( θ ) = 0 .
Revised: V ∇· J ( θ ) dV = 0 and ∇ J ( θ ) = 0 .

Original: Obviously, only minima are probability sources in deep learning.
Revised: Obviously, only minima are probability sources in deep learning.

Original: So we obtain
Revised: So we obtain

Original: [Equation]
Revised: [Equation]

Original:  Under Assumption 
Revised: [Equation] Under Assumption 3, we have [Equation] Based on the results of Step 1 and Step

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Based on the results of Step 1 and Step 2, we have

Original: [Equation]
Revised: [Equation]

Original: So we have
Revised: So we have

Original: [Equation]
Revised: [Equation]

Original: In the case of pure SGN, T a = η 2 B H a and T b = − η 2 B H b gives
Revised: In the case of pure SGN [Equation] T a = η 2 B H a and T b = − η 2 B H b gives

Original: [Equation]
Revised: [Equation]

Original: We generalize the proof above into the high-dimensional SGD diffusion.
Revised: We generalize the proof above into the high-dimensional SGD diffusion

Original: [Equation]
Revised: [Equation]

Revised: Step 1:

Revised: [Equation]

Original: Step 2:
Revised: Step 2:

Original: Based on the formula of the one-dimensional probability current and ﬂux, we obtain the high-dimensional ﬂux escaping through Col b:
Revised: Based on the formula of the one-dimensional probability current and ﬂux, we obtain the high-dimensional ﬂux escaping through Col b:

Original: [Equation]
Revised: [Equation]

Original: where [ · ] ⊥ e indicates the directions perpendicular to the escape direction e .
Revised: where [ · ] ⊥ e indicates the directions perpendicular to the escape direction e .

Original: So we have
Revised: So we have

Original: [Equation]
Revised: [Equation]

Original: T a and T b are the eigenvalues of H − 1 a D a and H − 1 b D b corresponding to the escape direction.
Revised: T a and T b are the eigenvalues of H − 1 a D a and H − 1 b D b corresponding to the escape direction.

Original: We know D a = η 2 B H a and D b = η 2 B [ H b ] + .
Revised: We know D a = η 2 B H a and D b = η 2 B [ H b ] + .

Original: As D must be positive semideﬁnite, we replace H b
Revised: As D must be positive semideﬁnite, we replace H b

Original: = U (cid:62) b diag ( H b 1 , · · · , H b ( n − 1) , H be ) U b by its positive semideﬁnite analog [ H b ] + =
Revised: = U (cid:62) b diag ( H b 1 , · · · , H b ( n − 1) , H be ) U b by its positive semideﬁnite analog [ H b ] + =

Original: U (cid:62) b diag ( H b 1 , · · · , H b ( n − 1) ,
Revised: U (cid:62) b diag ( H b 1 , · · · , H b ( n − 1) ,

Original: | H be | ) U b .
Revised: | H be | ) U b .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: A stationary distribution must have a balanced probability ﬂux between valleys.
Revised: A stationary distribution must have a balanced probability ﬂux between valleys.

Original: So the probability ﬂux of each valley must be equivalent,
Revised: So the probability ﬂux of each valley must be equivalent,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: As τ = γ − 1 , it leads to P ( θ ∈ V v )
Revised: As τ = γ − 1 , it leads to P ( θ ∈ V v )

Original: ∝ τ v .
Revised: ∝ τ v .

Original: We normalize the total probability to 1, then we obtain the result.
Revised: We normalize the total probability to 1, then we obtain the result.

Original: B A SSUMPTIONS
Revised: B A SSUMPTIONS

Original: Assumption 2 indicates that the dynamical system is in equilibrium near minima but not necessarily near saddle points.
Revised: Assumption 2 indicates that the dynamical system is in equilibrium near minima but not necessarily near saddle points.

Original: It means that ∂P ( θ,t ) ∂t 
Revised: It means that ∂P ( θ,t ) ∂t = −∇ · J

Original: [Equation]
Revised: [Equation]

Original:  0 holds near minima a 1 and a 2 , but not necessarily holds near saddle point b .
Revised: θ, t ) ≈ 0 holds near minima a 1 and a 2 , but not necessarily holds near saddle point b .

Original: Quasi-Equilibrium Assumption is actually weaker but more useful than the conventional stationary assumption for deep learning (Welling & Teh, 2011;
Revised: Quasi-Equilibrium Assumption is actually weaker but more useful than the conventional stationary assumption for deep learning (Welling & Teh, 2011;

Original: Mandt et al., 2017).
Revised: Mandt et al., 2017).

Original: Under Assumption 2, the probability density P can behave like a stationary distribution only inside valleys, but density transportation through saddle points can be busy.
Revised: Under Assumption 2, the probability density P can behave like a stationary distribution only inside valleys, but density transportation through saddle points can be busy.

Original: Quasi-Equilibrium is more like: stable lakes (loss valleys) is connected by rapid Rivers (escape paths).
Revised: Quasi-Equilibrium is more like: stable lakes (loss valleys) is connected by rapid Rivers (escape paths).

Original: In contrast, the stationary assumption requires strictly zero ﬂux between lakes (loss valleys).
Revised: In contrast, the stationary assumption requires strictly zero ﬂux between lakes (loss valleys).

Original: Little knowledge about density motion can be obtained under the stationary assumption.
Revised: Little knowledge about density motion can be obtained under the stationary assumption.

Original: Low Temperature Assumption is common (Van Kampen, 1992; Zhou, 2010; Berglund, 2013; Jastrz˛ebski et al., 2017), and is always justiﬁed when ηB is small.
Revised: Low Temperature Assumption is common (Van Kampen, 1992; Zhou, 2010; Berglund, 2013; Jastrz˛ebski et al., 2017), and is always justiﬁed when ηB is small.

Original: Under Assumption 3, the probability densities will concentrate around minima and MPPs.
Revised: Under Assumption 3, the probability densities will concentrate around minima and MPPs.

Original: Numerically, the 6-sigma rule may often provide good approximation for a Gaussian distribution.
Revised: Numerically, the 6-sigma rule may often provide good approximation for a Gaussian distribution.

Original: Assumption 3 will make the second order Taylor approximation, Assumption 1, even more reasonable in SGD diffusion.
Revised: Assumption 3 will make the second order Taylor approximation, Assumption 1, even more reasonable in SGD diffusion.

Original: Here, we try to provide a more intuitive explanation about Low Temperature Assumption in the domain of deep learning.
Revised: Here, we try to provide a more intuitive explanation about Low Temperature Assumption in the domain of deep learning.

Original: Without loss of generality, we discuss it in one-dimensional dynamics.
Revised: Without loss of generality, we discuss it in one-dimensional dynamics.

Original: The temperature can be interpreted as a real number D .
Revised: The temperature can be interpreted as a real number D .

Original: In SGD, we have the temperature as D = η 2 B H .
Revised: In SGD, we have the temperature as D = η 2 B H .

Original: In statistical physics, if ∆ LD is large, then we call it Low Temperature Approximation.
Revised: In statistical physics, if ∆ LD is large, then we call it Low Temperature Approximation.

Original: Note that ∆ LD appears insides an exponential function in the theoretical analysis.
Revised: Note that ∆ LD appears insides an exponential function in the theoretical analysis.

Original: People usually believe that, numerically, ∆ LD > 6 can make a good approximation, for a similar reason of the 6-sigma rule in statistics.
Revised: People usually believe that, numerically, ∆ LD > 6 can make a good approximation, for a similar reason of the 6-sigma rule in statistics.

Original: In the ﬁnal training phase of deep networks, a common setting is η = 0 .
Revised: In the ﬁnal training phase of deep networks, a common setting is η = 0 .

Original: 01 and B = 128 .
Revised: 01 and B = 128 .

Original: Thus, we may safely apply Assumption 3 to the loss valleys which satisfy the very mild condition ∆
Revised: Thus, we may safely apply Assumption 3 to the loss valleys which satisfy the very mild condition ∆

Original: LH > 2 .
Revised: LH > 2 .

Original: 3 × 10 − 4 .
Revised: 3 × 10 − 4 .

Original: Empirically, the condition ∆ LH > 2 .
Revised: Empirically, the condition ∆ LH > 2 .

Original: 3 × 10 − 4 holds well in SGD dynamics.
Revised: 3 × 10 − 4 holds well in SGD dynamics.

Original: It also suggests that, we can adjust the learning rate to let SGD search among loss valleys with certain barrier heights.
Revised: It also suggests that, we can adjust the learning rate to let SGD search among loss valleys with certain barrier heights.

Original: C T HE S TOCHASTIC G RADIENT N OISE A NALYSIS
Revised: C T HE S TOCHASTIC G RADIENT N OISE A NALYSIS

Original: Figure 7 demonstrates that the stochastic gradient noise is also approximately Gaussian for training
Revised: Figure 7 demonstrates that the SGN is also approximately Gaussian on a randomly initialized ResNet with B = 50 on CIFAR-10. We also note that the SGN on ResNet seems less Gaussian than SGN on [Figure] [Figure]

Original: ResNet on CIFAR-10.
Revised: [Figure] [Figure]

Original: By Figure 8, we validate C = HB in the original coordinates on MNIST.
Revised: By Figure [Figure] we validate C = HB in the original coordinates on MNIST.

Original: By Figure 9, we also validate
Revised: By Figure 9, we also validate

Original: C = HB on another dataset, Avila, in the space spanned by the eigenvectors of Hessian.
Revised: C = HB on another dataset, Avila, in the space spanned by the eigenvectors of Hessian.

Original: The relation
Revised: The relation

Original: C = HB can still be observed in these two cases.
Revised: C = HB can still be observed in these two cases.

Original: Data Precessing : We perform the usual per-pixel zero-mean and unit-variance normalization on
Revised: Data Precessing : We perform the usual per-pixel zero-mean and unit-variance normalization on

Original: MNIST.
Revised: MNIST.

Original: We leave the preprocessing of Avila in D. Model : Fully-connected networks.
Revised: We leave the preprocessing of Avila in D. Model : Fully-connected networks.

Original: D M AIN E XPERIMENTS
Revised: D M AIN E XPERIMENTS

Original: Figure 10, 11, and 12 respectively validate that the exponential relation of the escape rate with the
Revised: Figure 10, 11, and 12 respectively validate that the exponential relation of the escape rate with the

Original: Hessian, the batch size and the learning rate.
Revised: Hessian, the batch size and the learning rate.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: D. E XPERIMENTAL S ETTINGS
Revised: D. E XPERIMENTAL S ETTINGS

Original: Datasets : a) Avila, b) Banknote Authentication, c) Cardiotocography, d) Dataset for Sensorless Drive
Revised: Datasets : a) Avila, b) Banknote Authentication, c) Cardiotocography, d) Dataset for Sensorless Drive

Original: Diagnosis.
Revised: Diagnosis.

Original: Data Precessing : We perform per-pixel zero-mean and unit-variance normalization on input data.
Revised: Data Precessing : We perform per-pixel zero-mean and unit-variance normalization on input data.

Original: For simplicity, we also transform multi-class problems into binary-class problems by grouping labels, although this is unnecessary.
Revised: For simplicity, we also transform multi-class problems into binary-class problems by grouping labels, although this is unnecessary.

Original: Model : Two-layer fully-connected networks with one hidden layer and 10 neurons per hidden layer.
Revised: Model : Two-layer fully-connected networks with one hidden layer and 10 neurons per hidden layer.

Original: Initializations : To ensure the initialized models are near minima, we ﬁrst pretrain models with 200-1000 epochs to ﬁt each data set as well as possible.
Revised: Initializations : To ensure the initialized models are near minima, we ﬁrst pretrain models with 200-1000 epochs to ﬁt each data set as well as possible.

Original: We set the pretrained models’ parameters as the initialized θ t =0 .
Revised: We set the pretrained models’ parameters as the initialized θ t =0 .

Original: Valleys’ Boundary : In principle, any small neighborhood around θ t =0 can be regarded as the inside of the start valleys.
Revised: Valleys’ Boundary : In principle, any small neighborhood around θ t =0 can be regarded as the inside of the start valleys.

Original: In our experiments, we set each dimension’s distance from θ t =0 should be less than 0 .
Revised: In our experiments, we set each dimension’s distance from θ t =0 should be less than 0 .

Original: 05 , namely | ∆ θ i | ≤ 0 .
Revised: 05 , namely | ∆ θ i | ≤ 0 .

Original: for each dimension i .
Revised: for each dimension i .

Original: If we rescale the landscape by a factor k , the neighborhood will also be rescaled by k .
Revised: If we rescale the landscape by a factor k , the neighborhood will also be rescaled by k .

Original: Although we don’t know which loss valleys exist inside the neighborhood, we know the landscape of the neighborhood is invariant in each simulation.
Revised: Although we don’t know which loss valleys exist inside the neighborhood, we know the landscape of the neighborhood is invariant in each simulation.

Original: Hyperparameters :
Revised: Hyperparameters :

Original: In Figure 10: (a) η = 0 .
Revised: In Figure 10: (a) η = 0 .

Original: 001 , B = 1 , (b) η = 0 .
Revised: 001 , B = 1 , (b) η = 0 .

Original: 015 , B = 1 , (c) η = 0 .
Revised: 015 , B = 1 , (c) η = 0 .

Original: 005 , B = 1 , (d) η = 0 .
Revised: 005 , B = 1 , (d) η = 0 .

Original: 0005 , B = 1 .
Revised: 0005 , B = 1 .

Original: In Figure 11: (a) η = 0 .
Revised: In Figure 11: (a) η = 0 .

Original: 02 , (b) η = 0 .
Revised: 02 , (b) η = 0 .

Original: 6 , (c) η = 0 .
Revised: 6 , (c) η = 0 .

Original: 18 , (d) η = 0 .
Revised: 18 , (d) η = 0 .

Original: In
Revised: In

Original: Figure 12: (a) B = 1 , (b) B = 1 , (c) B = 1 , (d) B = 1 .
Revised: Figure 12: a) B = 1 , b) B = 1 , c) B = 1 , d) B = 1 .

Original: In Figure 13: (a) η = 0 . 0002 , B = 100 , (b) η = 0 .
Revised: In Figure 13: a) η = 0 . 0002 , B = 100 , (b) η = 0 .

Original: 001 , B = 100 , (c) η = 0 . 0002 , B = 100 , (d) η = 0 . 0001 , B = 100 .
Revised: 001 , B = 100 , (c) η = 0 . 0002 , B = 100 , (d) η = 0 . 0001 , B = 100 .

Original: In Figure 14: (a) η = 0 . 0002 , B = 100 , D = 0 . 0002 , (b) η = 0 .
Revised: In Figure 14: (a) η = 0 . 0002 , B = 100 , D = 0 . 0002 , (b) η = 0 .

Original: 001 , B = 100 , D = 0 . 0001 , (c) η = 0 . 0002 , B = 100 , D = 0 .
Revised: 001 , B = 100 , D = 0 . 0001 , (c) η = 0 . 0002 , B = 100 , D = 0 .

Original: 0005 , (d) η = 0 . 0001 , B = 100 , D= 0 .
Revised: 0005 , (d) η = 0 . 0001 , B = 100 , D

Original: We notethat the hyperparameters need be tuned foreach initialized pretrained models, dueto the stochastic property of deep learning. 
Revised: We note the hyperparameters need be tuned for pretrained models, due property of deep learning. [Figure] [Figure] [Figure]

Original:  According to our experience, we can always ﬁnd the hyperparameters to discover the quantitative relations as long as the pretrained model ﬁts the data set well enough.
Revised: [Figure] According to our experience, we can always ﬁnd the hyperparameters to discover the quantitative relations as long as the pretrained model ﬁts the data set well enough.

Original: The ﬁned-tuned requirement can be avoided in Section E, because the models in Section E are artiﬁcially initialized.
Revised: The ﬁned-tuned requirement can be avoided in Section E, because the models in Section E are artiﬁcially initialized.

Original: Observation : we observe the number of iterations from the initialized position to the terminated position.
Revised: Observation : we observe the number of iterations from the initialized position to the terminated position.

Original: We repeat experiments 100 times to estimate the escape rate γ and the mean escape time τ .
Revised: We repeat experiments 100 times to estimate the escape rate γ and the mean escape time τ .

Original: As the escape time is a random variable obeying an exponential distribution, t ∼ Exponential ( γ ) , the estimated escape rate can be written as
Revised: As the escape time is a random variable obeying an exponential distribution, t ∼ Exponential ( γ ) , the estimated escape rate can be written as

Original: [Equation]
Revised: [Equation]

Original: The 95% conﬁdence interval of this estimator is
Revised: The 95% conﬁdence interval of this estimator is

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: D. E XPERIMENTS ON SGLD
Revised: D. E XPERIMENTS ON SGLD

Original: Experimental Results : Figure 13 shows a highly precise exponential relation of the escape rate and the diffusion coefﬁcient in the ﬁgure.
Revised: Experimental Results : Figure 13 shows a highly precise exponential relation of the escape rate and the diffusion coefﬁcient in the ﬁgure.

Original: Figure 14 shows a proportional relation of the escape rate and the Hessian determinant in the ﬁgure.
Revised: Figure 14 shows a proportional relation of the escape rate and the Hessian determinant in the ﬁgure.

Original: Overall, the empirical results support the density diffusion theory in the dynamics of white noise.
Revised: Overall, the empirical results support the density diffusion theory in the dynamics of white noise.

Original: In experiments on SGLD, we carefully adjust the injected gradient noise scale in experiment to ensure that D is signiﬁcantly smaller than the loss barrier’ height and large enough to dominate SGN scale.
Revised: In experiments on SGLD, we carefully adjust the injected gradient noise scale in experiment to ensure that D is signiﬁcantly smaller than the loss barrier’ height and large enough to dominate SGN scale.

Original: If D is too large, learning dynamics will be reduced to Free Brownian Motion.
Revised: If D is too large, learning dynamics will be reduced to Free Brownian Motion.

Revised: [Figure]

Original: E E XPERIMENTS ON M ORE M ODELS
Revised: E E XPERIMENTS ON M ORE M ODELS

Original: We supply experiments of training three models on artiﬁcial Gaussian datasets.
Revised: We supply experiments of training three models on artiﬁcial Gaussian datasets.

Original: In these experiments, we can analytically know the locations of the minima, Hessians and loss barriers, as each input feature is Gaussian noise.[Figure] [Figure]
Revised: In these experiments, we can analytically know the locations of the minima, Hessians and loss barriers, as each input feature is Gaussian noise.

Original: E. E XPERIMENTS S ETTINGS
Revised: E. E XPERIMENTS S ETTINGS

Original: Data Set: We generate 50000 Gaussian samples and random two-class labels as the training data set, { ( x ( i ) , y ( i ) )
Revised: Data Set: We generate 50000 Gaussian samples and random two-class labels as the training data set, { ( x ( i ) , y ( i ) )

Original: | x ( i ) ∼ N (0 , I ) , y ( i ) ∈ { 0 , 1 } , i ∈ { 1 , 2 , · · · , 50000 }} .
Revised: | x ( i ) ∼ N (0 , I ) , y ( i ) ∈ { 0 , 1 } , i ∈ { 1 , 2 , · · · , 50000 }} .

Original: Hyperparameters :
Revised: Hyperparameters :

Original: In Figure 15: (a) η = 0 . 0001 , B = 100 , (b) η = 0 .
Revised: In Figure 15: (a) η = 0 . 0001 , B = 100 , (b) η = 0 .

Original: 001 , B = 100 , (c) η = 0 .
Revised: 001 , B = 100 , (c) η = 0 .

Original: 0003 , B = 100 .
Revised: 0003 , B = 100 .

Original: In Figure 16: (a) η = 0 .
Revised: In Figure 16: (a) η = 0 .

Original: 0001 , B = 50 ,
Revised: 0001 , B = 50 ,

Original: D = 0 .
Revised: D = 0 .

Original: 2 , (b) η = 0 .
Revised: 2 , (b) η = 0 .

Original: 001 , B = 50 , D = 0 .
Revised: 001 , B = 50 , D = 0 .

Original: 0005 , (c) η = 0 .
Revised: 0005 , (c) η = 0 .

Original: 0003 , B = 1 , D = 0 .
Revised: 0003 , B = 1 , D = 0 .

Original: In Figure 17: (a) η = 0 .
Revised: In Figure 17: (a) η = 0 .

Original: 006 , B = 50 , (b) η = 0 .
Revised: 006 , B = 50 , (b) η = 0 .

Original: 05 , B = 50 , (c) η = 0 .
Revised: 05 , B = 50 , (c) η = 0 .

Original: 005 , B = 1 .
Revised: 005 , B = 1 .

Original: In Figure 18: (a) η = 0 .
Revised: In Figure 18: (a) η = 0 .

Original: 006 , (b) η = 0 .
Revised: 006 , (b) η = 0 .

Original: 06 , (c) η = 0 .
Revised: 06 , (c) η = 0 .

Original: In
Revised: In

Original: Figure 19: (a) B = 1 , (b) B = 1 , (c) B = 1 .
Revised: Figure 19: (a) B = 1 , (b) B = 1 , (c) B = 1 .

Original: We note that the hyperparameters are recommended and needn’t be ﬁne tuned again.
Revised: We note that the hyperparameters are recommended and needn’t be ﬁne tuned again.

Original: The artiﬁcially initialized parameters avoids the stochastic property of the initial states.
Revised: The artiﬁcially initialized parameters avoids the stochastic property of the initial states.

Original: Experiment Setting 1: Styblinski-Tang Function is a commonly used function in nonconvex optimization, written as
Revised: Experiment Setting 1: Styblinski-Tang Function is a commonly used function in nonconvex optimization, written as

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original:  We use high-dimensional Styblinski-Tang Function as the test function, and Gaussian samples as training data.
Revised: [Equation] + 5 θ [Equation] We use high-dimensional Styblinski-Tang Function as the test function, and Gaussian samples as training data.

Original: [Equation]
Revised: [Equation]

Original: where data samples x ∼ N (0 , I ) .
Revised: where data samples x ∼ N (0 , I ) .

Original: The one-dimensional Styblinski-Tang Function has one global minimum located at a = − 2 . 903534 , one local minimum located at d , and one saddle point b = 0 .
Revised: The one-dimensional Styblinski-Tang Function has one global minimum located at a = − 2 . 903534 , one local minimum located at d , and one saddle point b = 0 .

Original: 156731 as the boundary separating Valley a 1 and Valley a 2 .
Revised: 156731 as the boundary separating Valley a 1 and Valley a 2 .

Original: For a n-dimensional Styblinski-Tang
Revised: For a n-dimensional Styblinski-Tang

Original: Function, we initialize parameters θ t =0 = 1 √ k ( − 2 .
Revised: Function, we initialize parameters θ t =0 = 1 √ k ( − 2 .

Original: 903534 , · · · , − 2 . 903534) , and set the valley’s boundary as θ i < 1 √ k 0 .
Revised: 903534 , · · · , − 2 . 903534) , and set the valley’s boundary as θ i < 1 √ k 0 .

Original: 156731 , where i is the dimension index.
Revised: 156731 , where i is the dimension index.

Original: We record the number of iterations required to escape from the valley to the outside of valley.
Revised: We record the number of iterations required to escape from the valley to the outside of valley.

Original: The setting 1 does not need labels.
Revised: The setting 1 does not need labels.

Original: Experiment Setting 2: We study the learning dynamics of Logistic Regression.
Revised: Experiment Setting 2: We study the learning dynamics of Logistic Regression.

Original: Parameters Initialization: θ t =0 =
Revised: Parameters Initialization: θ t =0 =

Original: Valley Boundary: − 0 .
Revised: Valley Boundary: − 0 .

Original: 1 < θ i < 0 .
Revised: 1 < θ i < 0 .

Original: Due to the randomness of training data and the symmetry of dimension, the origin must be a minimum and there are a lot unknown valleys neighboring the origin valley.
Revised: Due to the randomness of training data and the symmetry of dimension, the origin must be a minimum and there are a lot unknown valleys neighboring the origin valley.

Original: And we can set an arbitrary boundary surrounding the origin valley group, and study the mean escape time from the group of valleys.
Revised: And we can set an arbitrary boundary surrounding the origin valley group, and study the mean escape time from the group of valleys.

Original: Experiment Setting 3: We study the learning dynamics of MLP with ReLu activations, cross entropy losses, depth as 3, and hidden layers’ width as 10.
Revised: Experiment Setting 3: We study the learning dynamics of MLP with ReLu activations, cross entropy losses, depth as 3, and hidden layers’ width as 10.

Original: Parameters Initialization: θ t =0 =
Revised: Parameters Initialization: θ t =0 =

Original: (0 . 1 , · · · , 0 . 1) with a small Gaussian noise (cid:15) =
Revised: (0 . 1 , · · · , 0 . 1) with a small Gaussian noise (cid:15) =

Original: 01 I ) .
Revised: 01 I ) .

Original: Valley Boundary: 0 .
Revised: Valley Boundary: 0 .

Original: 05 <
Revised: 05 <

Original: θ i < 0 .
Revised: θ i < 0 .

Original: To prevent the gradient disappearance problem of deep learning, we move the starting point from the origin.
Revised: To prevent the gradient disappearance problem of deep learning, we move the starting point from the origin.

Original: For symmetry breaking of deep learning, we add a small Gaussian noise to each parameter’s initial value.
Revised: For symmetry breaking of deep learning, we add a small Gaussian noise to each parameter’s initial value.

Original: Due to the complex loss landscape of deep networks, we can hardly know the exact information about valleys and cols.
Revised: Due to the complex loss landscape of deep networks, we can hardly know the exact information about valleys and cols.

Original: However, the escape formula can still approximately hold even if an arbitrary boundary surrounding an arbitrary group of valleys.
Revised: However, the escape formula can still approximately hold even if an arbitrary boundary surrounding an arbitrary group of valleys.

Original: We set the batch size as 1 in this setting.
Revised: We set the batch size as 1 in this setting.

Original: When the batch size is small, the gradient noise is more like a heavy-tailed noise.
Revised: When the batch size is small, the gradient noise is more like a heavy-tailed noise.

Original: We can validate whether or not the propositions can hold with very-small-batch gradient noise in practice.
Revised: We can validate whether or not the propositions can hold with very-small-batch gradient noise in practice.

Original: E. E XPERIMENTS R ESULTS
Revised: E. E XPERIMENTS R ESULTS

Original: Figure 15 shows the relation of the escape rate and the isotropic diffusion coefﬁcient D. Figureshows the relation of the escape rate and the Hessian determinant in the dynamics of white noise.
Revised: Figure 15 shows the relation of the escape rate and the isotropic diffusion coefﬁcient D. Figureshows the relation of the escape rate and the Hessian determinant in the dynamics of white noise.

Original: Figure 17 shows the relation of the escape rate and the second order directional derivative in the dynamics of SGD.
Revised: Figure 17 shows the relation of the escape rate and the second order directional derivative in the dynamics of SGD.

Original: Figure 18 shows the relation of the escape rate and the batch size in the dynamics of SGD.
Revised: Figure 18 shows the relation of the escape rate and the batch size in the dynamics of SGD.

Original: Figure 19 shows the relation of the escape rate and the learning rate in the dynamics of SGD.
Revised: Figure 19 shows the relation of the escape rate and the learning rate in the dynamics of SGD.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: formula that γ is linear with k is validated in
Revised: formula that γ is linear with k is validated in

Original: Styblinski
Revised: Styblinski

Original: Tang
Revised: Tang

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]