Original: O N G ENERALIZATION E RROR B OUNDS OF N OISY G RADIENT M ETHODS FOR N ON -C ONVEX L EARNINGAnonymous authors
Revised: O N G ENERALIZATION E RROR B OUNDS OF N OISY G RADIENT M ETHODS FOR N ON -C ONVEX L EARNING

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generalization error (also known as the out-of-sample error) measures how well the hypothesis learned from training data generalizes to previously unseen data.
Revised: Generalization error (also known as the out-of-sample error) measures how well the hypothesis learned from training data generalizes to previously unseen data.

Original: Proving tight generalization error bounds is a central question in statistical learning theory.
Revised: Proving tight generalization error bounds is a central question in statistical learning theory.

Original: In this paper, we obtain generalization error bounds for learning general non-convex objectives, which has attracted signiﬁcant attention in recent years.
Revised: In this paper, we obtain generalization error bounds for learning general non-convex objectives, which has attracted signiﬁcant attention in recent years.

Original: We develop a new framework, termed Bayes-Stability , for proving algorithm-dependent generalization error bounds.
Revised: We develop a new framework, termed Bayes-Stability , for proving algorithm-dependent generalization error bounds.

Original: The new framework combines ideas from both the PAC-Bayesian theory and the notion of algorithmic stability.
Revised: The new framework combines ideas from both the PAC-Bayesian theory and the notion of algorithmic stability.

Original: Applying the Bayes-Stability method, we obtain new data-dependent generalization bounds for stochastic gradient Langevin dynamics (SGLD) and several other noisy gradient methods (e.g., with momentum, mini-batch and acceleration,
Revised: Applying the Bayes-Stability method, we obtain new data-dependent generalization bounds for stochastic gradient Langevin dynamics (SGLD) and several other noisy gradient methods (e.g., with momentum, mini-batch and acceleration,

Original: Entropy-SGD).
Revised: Entropy-SGD).

Original: Our result recovers (and is typically tighter than) a recent result in
Revised: Our result recovers (and is typically tighter than) a recent result in

Original: Mou et al.
Revised: Mou et al.

Original: and improves upon the results in Pensia et al.
Revised: and improves upon the results in Pensia et al.

Original: Our experiments demonstrate that our data-dependent bounds can distinguish randomly labelled data from normal data, which provides an explanation to the intriguing phenomena observed in Zhang et al.
Revised: Our experiments demonstrate that our data-dependent bounds can distinguish randomly labelled data from normal data, which provides an explanation to the intriguing phenomena observed in Zhang et al.

Original: (2017a).
Revised: (2017a).

Original: We also study the setting where the total loss is the sum of a bounded loss and an additional (cid:96) 2 regularization term.
Revised: We also study the setting where the total loss is the sum of a bounded loss and an additional (cid:96) 2 regularization term.

Original: We obtain new generalization bounds for the continuous Langevin dynamic in this setting by developing a new Log-Sobolev inequality for the parameter distribution at any time.
Revised: We obtain new generalization bounds for the continuous Langevin dynamic in this setting by developing a new Log-Sobolev inequality for the parameter distribution at any time.

Original: Our new bounds are more desirable when the noise level of the process is not very small, and do not become vacuous even when T tends to inﬁnity.
Revised: Our new bounds are more desirable when the noise level of the process is not very small, and do not become vacuous even when T tends to inﬁnity.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Non-convex stochastic optimization is the major workhorse of modern machine learning.
Revised: Non-convex stochastic optimization is the major workhorse of modern machine learning.

Original: For instance, the standard supervised learning on a model class parametrized by R d can be formulated as the following optimization problem:min
Revised: For instance, the standard supervised learning on a model class parametrized by R d can be formulated as the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where w denotes the model parameter, D is an unknown data distribution over the instance space Z , and F : R d × Z
Revised: where w denotes the model parameter, D is an unknown data distribution over the instance space Z , and F : R d × Z

Original: → R is a given objective function which may be non-convex.
Revised: → R is a given objective function which may be non-convex.

Original: A learning algorithm takes as input a sequence S
Revised: A learning algorithm takes as input a sequence S

Original: = ( z 1 , z 2 , . . .
Revised: = ( z 1 , z 2 , . . .

Original: , z n ) of n data points sampled i.i.d.
Revised: , z n ) of n data points sampled i.i.d.

Original: from D , and outputs a (possibly randomized) parameter conﬁguration ˆ w ∈ R d .
Revised: from D , and outputs a (possibly randomized) parameter conﬁguration ˆ w ∈ R d .

Original: A fundamental problem in learning theory is to understand the generalization performance of learning algorithms—is the algorithm guaranteed to output a model that generalizes well to the data distribution D ?
Revised: A fundamental problem in learning theory is to understand the generalization performance of learning algorithms—is the algorithm guaranteed to output a model that generalizes well to the data distribution D ?

Original: Speciﬁcally, we aim to prove upper bounds on the generalization error err gen ( S ) =
Revised: Speciﬁcally, we aim to prove upper bounds on the generalization error err gen ( S ) =

Original: L ( ˆ w, D ) − L ( ˆ w, S ) , where L ( ˆ w, D ) =
Revised: L ( ˆ w, D ) − L ( ˆ w, S ) , where L ( ˆ w, D ) =

Original: E z
Revised: E z

Original: ∼D
Revised: ∼D

Original: [ L ( ˆ w, z )] and L ( ˆ w, S )
Revised: [ L ( ˆ w, z )] and L ( ˆ w, S )

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =

Revised: =

Original: L ( ˆ w, z i ) are the population and empirical losses, respectively.
Revised: L ( ˆ w, z i ) are the population and empirical losses, respectively.

Original: We note that the loss function L (e.g., the 0/1 loss) could be different from the objective function F (e.g., the cross-entropy loss) used in the training process (which serves as a surrogate for the loss L ).
Revised: We note that the loss function L (e.g., the 0/1 loss) could be different from the objective function F (e.g., the cross-entropy loss) used in the training process (which serves as a surrogate for the loss L ).

Original: Classical learning theory relates the generalization error to various complexity measures (e.g., the
Revised: Classical learning theory relates the generalization error to various complexity measures (e.g., the

Original: VC-dimension and Rademacher complexity) of the model class.
Revised: VC-dimension and Rademacher complexity) of the model class.

Original: Directly applying these classical complexity measures, however, often fails to explain the recent success of over-parametrized neural networks, where the model complexity signiﬁcantly exceeds the amount of available training data 
Revised: Directly applying these classical complexity measures, however, often fails to explain the recent success of over-parametrized neural networks, where the model complexity signiﬁcantly exceeds the amount of available training data (see e.g., Zhang et al.

Original: (see e.g., Zhang et al. (2017a)).
Revised: (2017a)).

Original: By incorporating certain data-dependent quantities such as margin and compressibility into the classical framework, some recent work (e.g., Bartlett et al.
Revised: By incorporating certain data-dependent quantities such as margin and compressibility into the classical framework, some recent work (e.g., Bartlett et al.

Original: ; Arora et al.
Revised: ; Arora et al.

Original: ; Wei & Ma (2019)) obtains more meaningful generalization bounds in the deep learning context.
Revised: ; Wei & Ma (2019)) obtains more meaningful generalization bounds in the deep learning context.

Original: An alternative approach to generalization is to prove algorithm-dependent bounds.
Revised: An alternative approach to generalization is to prove algorithm-dependent bounds.

Original: One celebrated example along this line is the algorithmic stability framework initiated by Bousquet & Elisseeff (2002).
Revised: One celebrated example along this line is the algorithmic stability framework initiated by Bousquet & Elisseeff (2002).

Original: Roughly speaking, the generalization error can be bounded by the stability of the algorithm (see Section 2 for the details).
Revised: Roughly speaking, the generalization error can be bounded by the stability of the algorithm (see Section 2 for the details).

Original: Using this framework, Hardt et al.
Revised: Using this framework, Hardt et al.

Original: (2016) study the stability (hence the generalization) of stochastic gradient descent (SGD) for both convex and non-convex functions.
Revised: (2016) study the stability (hence the generalization) of stochastic gradient descent (SGD) for both convex and non-convex functions.

Original: Their work motivates recent study of the generalization performance of several other gradient-based optimization methods: Kuzborskij & Lampert (2018); London (2016);
Revised: Their work motivates recent study of the generalization performance of several other gradient-based optimization methods: Kuzborskij & Lampert (2018); London (2016);

Original: Chaudhari et al.
Revised: Chaudhari et al.

Original: ; Raginsky et al.
Revised: ; Raginsky et al.

Original: ; Mou et al.
Revised: ; Mou et al.

Original: ; Pensia et al.
Revised: ; Pensia et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: In this paper, we study the algorithmic stability and generalization performance of various iterative gradient-based method, with certain continuous noise injected in each iteration, in a non-convex setting.
Revised: In this paper, we study the algorithmic stability and generalization performance of various iterative gradient-based method, with certain continuous noise injected in each iteration, in a non-convex setting.

Original: As a concrete example, we consider the stochastic gradient Langevin dynamics (SGLD) (see Raginsky et al.
Revised: As a concrete example, we consider the stochastic gradient Langevin dynamics (SGLD) (see Raginsky et al.

Original: ; Mou et al.
Revised: ; Mou et al.

Original: ; Pensia et al.
Revised: ; Pensia et al.

Original: Viewed as a variant of SGD,
Revised: Viewed as a variant of SGD,

Original: SGLD adds an isotropic Gaussian noise at every update step:
Revised: SGLD adds an isotropic Gaussian noise at every update step:

Original: [Equation]
Revised: [Equation]

Original: where g t ( W t − 1 ) denotes either the full gradient or the gradient over a mini-batch sampled from training dataset.
Revised: where g t ( W t − 1 ) denotes either the full gradient or the gradient over a mini-batch sampled from training dataset.

Original: We also study a continuous version of (1), which is the dynamic deﬁned by the following stochastic differential equation (SDE):
Revised: We also study a continuous version of (1), which is the dynamic deﬁned by the following stochastic differential equation (SDE):

Original: [Equation]
Revised: [Equation]

Original: where B t is the standard Brownian motion.
Revised: where B t is the standard Brownian motion.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Most related to our work is the study of algorithm-dependent generalization bounds of stochastic gradient methods.
Revised: Most related to our work is the study of algorithm-dependent generalization bounds of stochastic gradient methods.

Original: Hardt et al.
Revised: Hardt et al.

Original: ﬁrst study the generalization performance of SGD via algorithmic stability.
Revised: ﬁrst study the generalization performance of SGD via algorithmic stability.

Original: They prove a generalization bound that scales linearly with T , the number of iterations, when the loss function is convex, but their results for general non-convex optimization are more restricted.
Revised: They prove a generalization bound that scales linearly with T , the number of iterations, when the loss function is convex, but their results for general non-convex optimization are more restricted.

Original: London (2017) presents a generalization bound that also combines PAC-Bayesian analysis with stability.
Revised: London (2017) and Rivasplata et al. (2018) also combine ideas from both PAC-Bayesian and algorithm stability.

Original: However, their prior and posterior areprobability distributions on the hyperparameter space, while ours are distributions on the hypothesis space.
Revised: However, these works are essentially different from ours. In London (2017), the prior and posterior are distributions on the hyperparameter space instead of distributions on the hypothesis space.

Revised: Rivasplata et al.

Revised: (2018) study the hypothesis stability measured by the distance on the hypothesis space in a setting where the returned hypothesis (model parameter) is perturbed by a Gaussian noise.

Original: Our work is a follow-up of the recent work by Mou et al.
Revised: Our work is a follow-up of the recent work by Mou et al.

Original: (2018), in which they provide generalization bounds for SGLD from both stability and PAC-Bayesian perspectives.
Revised: (2018), in which they provide generalization bounds for SGLD from both stability and PAC-Bayesian perspectives.

Original: Another closely related work by Pensia et al.
Revised: Another closely related work by Pensia et al.

Original: (2018) derives similar bounds for noisy stochastic gradient methods, based on the information theoretic framework of Xu & Raginsky (2017).
Revised: (2018) derives similar bounds for noisy stochastic gradient methods, based on the information theoretic framework of Xu & Raginsky (2017).

Original: However, their bounds scale as O ( (cid:112) T/n ) ( n is the size of the training dataset) and are sub-optimal even for SGLD.
Revised: However, their bounds scale as O ( (cid:112) T/n ) ( n is the size of the training dataset) and are sub-optimal even for SGLD.

Original: We acknowledge that besides the algorithm-dependent approach that we follow, recent advances in learning theory aim to explain the generalization performance of neural networks from many other perspectives.
Revised: We acknowledge that besides the algorithm-dependent approach that we follow, recent advances in learning theory aim to explain the generalization performance of neural networks from many other perspectives.

Original: Some of the most prominent ideas include bounding the network capacity by the norms of weight matrices Neyshabur et al.
Revised: Some of the most prominent ideas include bounding the network capacity by the norms of weight matrices Neyshabur et al.

Original: ; Liang et al.
Revised: ; Liang et al.

Original: (2019), margin theory Bartlett et al.
Revised: (2019), margin theory Bartlett et al.

Original: (2017); Wei et al.
Revised: (2017); Wei et al.

Original: (2018), PAC-Bayesian theory Dziugaite & Roy (2017)
Revised: (2019), PAC-Bayesian theory Dziugaite & Roy (2017)

Original: ; Neyshabur et al.
Revised: ; Neyshabur et al.

Original: Dziugaite & Roy (2018), network compressibility Arora et al.
Revised: Dziugaite & Roy (2018), network compressibility Arora et al.

Original: (2018), and over-parametrization Du et al.
Revised: (2018), and over-parametrization Du et al.

Original: Allen-Zhu et al.
Revised: Allen-Zhu et al.

Original: ; Zou et al.
Revised: ; Zou et al.

Original: ; Chizat & Bach (2018).
Revised: ; Chizat et al.

Original: Most of these results are stated in the context of neural networks (some are tailored to networks with speciﬁc architecture), whereas our work addresses generalization in non-convex stochastic optimization in general.
Revised: Most of these results are stated in the context of neural networks (some are tailored to networks with speciﬁc architecture), whereas our work addresses generalization in non-convex stochastic optimization in general.

Original: We also note that some recent work provides explanations for the phenomenon reported in 
Revised: We also note that some recent work provides explanations for the phenomenon reported in Zhang et al.

Original: Zhang et al. (2017a) from a variety of different perspectives (e.g., Bartlett et al.
Revised: (2017a) from a variety of different perspectives (e.g., Bartlett et al.

Original: ; Arora et al.
Revised: ; Arora et al.

Original: Welling & Teh (2011) ﬁrst consider stochastic gradient Langevin dynamics (SGLD) as a sampling algorithm in the Bayesian inference context.
Revised: Welling & Teh (2011) ﬁrst consider stochastic gradient Langevin dynamics (SGLD) as a sampling algorithm in the Bayesian inference context.

Original: Raginsky et al.
Revised: Raginsky et al.

Original: (2017) give a non-asymptotic analysis and establish the ﬁnite-time convergence guarantee of SGLD to an approximate global minimum.
Revised: (2017) give a non-asymptotic analysis and establish the ﬁnite-time convergence guarantee of SGLD to an approximate global minimum.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2017b) analyze the hitting time of SGLD and prove that SGLD converges to an approximate local minimum.
Revised: (2017b) analyze the hitting time of SGLD and prove that SGLD converges to an approximate local minimum.

Original: These results are further improved and generalized to a family of Langevin dynamics based algorithms by the subsequent work of Xu et al.
Revised: These results are further improved and generalized to a family of Langevin dynamics based algorithms by the subsequent work of Xu et al.

Original: 1. O VERVIEW OF O UR R ESULTS
Revised: 1. O VERVIEW OF O UR R ESULTS

Original: In this paper, we provide generalization guarantees for the noisy variants of several popular stochastic gradient methods.
Revised: In this paper, we provide generalization guarantees for the noisy variants of several popular stochastic gradient methods.

Original: The Bayes-Stability method and data-dependent generalization bounds.
Revised: The Bayes-Stability method and data-dependent generalization bounds.

Original: We develop a new method for proving generalization bounds, termed as Bayes-Stability, by incorporating ideas from the PAC-Bayesian theory into the stability framework.
Revised: We develop a new method for proving generalization bounds, termed as Bayes-Stability, by incorporating ideas from the PAC-Bayesian theory into the stability framework.

Original: In particular, assuming the loss takes value in [0 , C ] , our method shows that the generalization error is bounded by both 2 C E
Revised: In particular, assuming the loss takes value in [0 , C ] , our method shows that the generalization error is bounded by both 2 C E

Original: z [ (cid:112) 2KL
Revised: z [ (cid:112) 2KL

Original: ( P, Q z )] and 2 C E z
Revised: ( P, Q z )] and 2 C E z

Original: [ (cid:112) 2KL( Q z , P )] , where P is a prior distribution independent of the training set S , and
Revised: [ (cid:112) 2KL( Q z , P )] , where P is a prior distribution independent of the training set S , and

Original: Q z is the expected posterior distribution conditioned on z n = z (i.e., the last training data is z ).
Revised: Q z is the expected posterior distribution conditioned on z n = z (i.e., the last training data is z ).

Original: The formal deﬁnition and the results can be found in Deﬁnition 5 and Theorem 7.
Revised: The formal deﬁnition and the results can be found in Deﬁnition 5 and Theorem 7.

Original: Inspired by Lever et al.
Revised: Inspired by Lever et al.

Original: (2013), instead of using a ﬁxed prior distribution, we bound the KLdivergence from the posterior to a distribution-dependent prior.
Revised: (2013), instead of using a ﬁxed prior distribution, we bound the KLdivergence from the posterior to a distribution-dependent prior.

Original: This enables us to derive the following generalization error bound that depends on the expected norm of the gradient along the optimization path:
Revised: This enables us to derive the following generalization error bound that depends on the expected norm of the gradient along the optimization path:

Original: [Equation]
Revised: [Equation]

Original: Here S is the dataset and g e ( t ) =
Revised: Here S is the dataset and g e ( t ) =

Original: E W t − 1 [ 1 n (cid:80) ni
Revised: E W t − 1 [ 1 n (cid:80) ni

Original: =1 (cid:107)∇ F ( W t − 1 , z i ) (cid:107) 2 ] is the expected empirical squared gradient norm at step t ; see Theorem 11 for the details.
Revised: =1 (cid:107)∇ F ( W t − 1 , z i ) (cid:107) 2 ] is the expected empirical squared gradient norm at step t ; see Theorem 11 for the details.

Original: Compared with the previous O (cid:16) LCn (cid:113)(cid:80) t γ 2 t σ 2 t (cid:17) bound in (Mou et al., 2018, Theorem 1), where L is the global Lipschitz constant of the loss, our new bound (3) depends on the data distribution and is typically tighter (as the gradient norm is at most L ).
Revised: Compared with the previous O (cid:16) LCn (cid:113)(cid:80) t γ 2 t σ 2 t (cid:17) bound in (Mou et al., 2018, Theorem 1), where L is the global Lipschitz constant of the loss, our new bound (3) depends on the data distribution and is typically tighter (as the gradient norm is at most L ).

Original: In modern deep neural networks, the worstcase Lipschitz constant L can be quite large, and typically much larger than the expected empirical gradient norm along the optimization trajectory.
Revised: In modern deep neural networks, the worstcase Lipschitz constant L can be quite large, and typically much larger than the expected empirical gradient norm along the optimization trajectory.

Original: Speciﬁcally, in the later stage of the training, the expected empirical gradient is small (see Figure 1(d) for the details).
Revised: Speciﬁcally, in the later stage of the training, the expected empirical gradient is small (see Figure 1(d) for the details).

Original: Hence, our generalization bound does not grow much even if we train longer at this stage.
Revised: Hence, our generalization bound does not grow much even if we train longer at this stage.

Original: Our new bound also offers an explanation to the difference between training on correct and random labels observed by Zhang et al.
Revised: Our new bound also offers an explanation to the difference between training on correct and random labels observed by Zhang et al.

Original: (2017a).
Revised: (2017a).

Original: In particular, we show empirically that the sum of expected squared gradient norm (along the optimization path) is signiﬁcantly higher when the training labels are replaced with random labels (Section 3, Remark 13, Figure 1, Appendix C.2).
Revised: In particular, we show empirically that the sum of expected squared gradient norm (along the optimization path) is signiﬁcantly higher when the training labels are replaced with random labels (Section 3.1, Figure 1, Appendix C.2).

Original: We would also like to mention the PAC-Bayesian bound (for SGLD with (cid:96) 2 -regularization) proposed by Mou et al.
Revised: We would also like to mention the PAC-Bayesian bound (for SGLD with (cid:96) 2 -regularization) proposed by Mou et al.

Original: (This bound is different from what we mentioned before; see Theorem 2 in their paper.)
Revised: (This bound is different from what we mentioned before; see Theorem 2 in their paper.)

Original: Their bound scales as O (1 / √ n ) and the numerator of their bound has a similar sum of gradient norms (with a decaying weight if the regularization coefﬁcient λ > 0 ).
Revised: Their bound scales as O (1 / √ n ) and the numerator of their bound has a similar sum of gradient norms (with a decaying weight if the regularization coefﬁcient λ > 0 ).

Original: Their bound is based on the PAC-Bayesian approach and holds with high probability, while our bound only holds in expectation.
Revised: Their bound is based on the PAC-Bayesian approach and holds with high probability, while our bound only holds in expectation.

Original: Extensions.
Revised: Extensions.

Original: We remark that our technique allows for an arguably simpler proof of (Mou et al., 2018,
Revised: We remark that our technique allows for an arguably simpler proof of (Mou et al., 2018,

Original: Theorem 1); the original proof is based on SDE and Fokker-Planck equation.
Revised: Theorem 1); the original proof is based on SDE and Fokker-Planck equation.

Original: More importantly, our technique can be easily extended to handle mini-batches and a variety of general settings as follows.
Revised: More importantly, our technique can be easily extended to handle mini-batches and a variety of general settings as follows.

Original:  Extension to other gradient-based methods.
Revised: 1. Extension to other gradient-based methods.

Original: Our results naturally extends to other noisy stochastic gradient methods including momentum due to Polyak 
Revised: Our results naturally extends to other noisy stochastic gradient methods including momentum due to Polyak (1964) (Theorem 26), Nes-

Original: (1964) (Theorem 26), Nesterov’s accelerated gradient method in Nesterov (1983)
Revised: terov’s accelerated gradient method in Nesterov (

Original: (Theorem 26), and Entropy-SGD proposed by Chaudhari et al.
Revised: Theorem 26) , and Entropy-SGD proposed by Chaudhari et al.

Original: (Theorem 27).
Revised: (Theorem 27).

Original: Extension to general noises.
Revised: Extension to general noises.

Original: The proof of the generalization bound in Mou et al.
Revised: The proof of the generalization bound in Mou et al.

Original: (2018) relies heavily on that the noise is Gaussian 1 , which makes it difﬁcult to generalize to other noise distributions such as the Laplace distribution.
Revised: 2018) relies heavily on that the noise is Gaussian 1 , which makes it difﬁcult to generalize to other noise distributions such as the Laplace distribution.

Original: In contrast, our analysis easily carries over to the class of log-Lipschitz noises (i
Revised: In contrast, our analysis easily carries over to the class of log-Lipschitz noises (i

Original: e., noises drawn from distributions with Lipschitz log densities).
Revised: e., noises drawn from distributions with Lipschitz log densities).

Original: Pathwise stability.
Revised: Pathwise stability.

Revised: In practice, it

Revised: is also natural to output a

Revised: certain function

Revised: of

Original: In practice, it is also natural to output a certain function of the entire optimization path, e.g., the one with the smallest empiricalrisk or a weighted average.
Revised: the entire optimization path, e.g., the one with the smallest empirical

Revised: risk

Revised: or a weighted average.

Revised: We

Original: We show that the same generalization bound holds for all such variants (Remark 12).
Revised: show that the same generalization bound holds for all such variants (

Original:  We note that the analysis in an independent work of Pensia et al.
Revised: Remark We note that the analysis in an independent work of Pensia et al.

Original: (2018) also satisﬁes this property, yet their bound is O (cid:18)(cid:113) C 2 L 2 n − 1 (cid:80)
Revised: (2018) also satisﬁes this property, yet their bound is O (cid:18)(cid:113) C 2 L 2 n − 1 (cid:80)

Original: Tt =1 η 2 t /σ 2 t (cid:19) (see Corollary 1 in their work), which scales at a slower O (1 / √ n ) rate (instead of O (1 /n ) )
Revised: Tt =1 η 2 t /σ 2 t (cid:19) (see Corollary 1 in their work), which scales at a slower O (1 / √ n ) rate (instead of O (1 /n ) )

Original: when dealing with C -bounded loss.
Revised: when dealing with C -bounded loss.

Original: Generalization bounds with (cid:96) 2 regularization via Log-Sobolev inequalities.
Revised: Generalization bounds with (cid:96) 2 regularization via Log-Sobolev inequalities.

Original: We also study the setting where the total objective function F is the sum of a C -bounded differentiable objective Fand an additional (cid:96) 2 regularization term λ 2 (cid:107) w (cid:107) 22 .
Revised: We also study the setting where the total objective function F is the sum of a C -bounded differentiable objective Fand an additional (cid:96) 2 regularization term λ 2 (cid:107) w (cid:107) 22 .

Original: In this case, F can be treated as a perturbation of a quadratic function, and the continuous Langevin dynamics (CLD) is well understood for quadratic functions.
Revised: In this case, F can be treated as a perturbation of a quadratic function, and the continuous Langevin dynamics (CLD) is well understood for quadratic functions.

Original: We obtain two generalization bounds for CLD, both via the technique of Log-Sobolev inequalities, a powerful tool for proving the convergence rate of CLD.
Revised: We obtain two generalization bounds for CLD, both via the technique of Log-Sobolev inequalities, a powerful tool for proving the convergence rate of CLD.

Original: One of our bounds is as follows (Theorem 15):
Revised: One of our bounds is as follows (Theorem 15):

Original: [Equation]
Revised: [Equation]

Original: The above bound has the following advantages:
Revised: The above bound has the following advantages:

Original: Applying e − x ≥ 1 − x , one can see that our bound is at most O ( √ T/n ) , which matches the previous bound in (Mou et al., 2018, Proposition 8) 3 . 2.
Revised: Applying e − x ≥ 1 − x , one can see that our bound is at most O ( √ T/n ) , which matches the previous bound in (Mou et al., 2018, Proposition 8) 3 . 2.

Original: As time T grows, the bound is upper bounded by and approaches to 2 e 4 βC CLn − 1 (cid:112) β/λ (unlike the previous O ( √ T/n ) bound that goes to inﬁnity as T → + ∞ ).
Revised: As time T grows, the bound is upper bounded by and approaches to 2 e 4 βC CLn − 1 (cid:112) β/λ (unlike the previous O ( √ T/n ) bound that goes to inﬁnity as T → + ∞ ).

Original: If the noise level is not so small (i.e., β is not very large), the generalization bound is quite desirable.
Revised: If the noise level is not so small (i.e., β is not very large), the generalization bound is quite desirable.

Original: Our analysis is based on a Log-Sobolev inequality (LSI) for the parameter distribution at time t , whereas most known LSIs only hold for the stationary distribution of the Markov process.
Revised: Our analysis is based on a Log-Sobolev inequality (LSI) for the parameter distribution at time t , whereas most known LSIs only hold for the stationary distribution of the Markov process.

Original: We prove the new LSI by exploiting the variational formulation of the entropy formula.
Revised: We prove the new LSI by exploiting the variational formulation of the entropy formula.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Notations.
Revised: Notations.

Original: We use D to denote the data distribution.
Revised: We use D to denote the data distribution.

Original: The training dataset S =
Revised: The training dataset S =

Original: ( z 1 , . . .
Revised: ( z 1 , . . .

Original: , z n ) is a sequence of n independent samples drawn from D .
Revised: , z n ) is a sequence of n independent samples drawn from D .

Original: S, S (cid:48)
Revised: S, S (cid:48)

Original: ∈
Revised: ∈

Original: Z n are called neighboring datasets if and only if they differ at exactly one data point (we could assume without loss of generality that z n (cid:54) =
Revised: Z n are called neighboring datasets if and only if they differ at exactly one data point (we could assume without loss of generality that z n (cid:54) =

Original: z (cid:48) n ).
Revised: z (cid:48) n ).

Original: Let F ( w, z ) and L ( w, z ) be the objective and the loss functions, respectively, where w ∈ R d denotes a model parameter and z ∈ Z is a data point.
Revised: Let F ( w, z ) and L ( w, z ) be the objective and the loss functions, respectively, where w ∈ R d denotes a model parameter and z ∈ Z is a data point.

Original: Deﬁne F ( w, S )
Revised: Deﬁne F ( w, S )

Original: = 1 n (cid:80) ni =1 F ( w, z i ) and F ( w, D ) =
Revised: = 1 n (cid:80) ni =1 F ( w, z i ) and F ( w, D ) =

Original: E z
Revised: E z

Original: ∼D
Revised: ∼D

Original: [ F ( w, z )] ; L ( w, S ) and L ( w, D ) are deﬁned similarly.
Revised: [ F ( w, z )] ; L ( w, S ) and L ( w, D ) are deﬁned similarly.

Original: A learning algorithm
Revised: A learning algorithm

Original: A takes as input a dataset S , and outputs a parameter w ∈ R d randomly.
Revised: A takes as input a dataset S , and outputs a parameter w ∈ R d randomly.

Original: Let G be the set of all possible mini-batches.
Revised: Let G be the set of all possible mini-batches.

Original: G n = { B ∈ G : n ∈ B } denotes the collection of mini-batches that contain the n -th data point, while G n =
Revised: G n = { B ∈ G : n ∈ B } denotes the collection of mini-batches that contain the n -th data point, while G n =

Original: G \ G n .
Revised: G \ G n .

Original: Let diam( A ) =
Revised: Let diam( A ) =

Original: sup x,y ∈ A (cid:107) x − y (cid:107) 2 denote the diameter of a set A .
Revised: sup x,y ∈ A (cid:107) x − y (cid:107) 2 denote the diameter of a set A .

Original: Deﬁnition 1 ( L -lipschitz) .
Revised: Deﬁnition 1 ( L -lipschitz) .

Original: A function F : R d × Z
Revised: A function F : R d × Z

Original: → R is L -lipschitz if and only if | F ( w 1 , z ) −
Revised: → R is L -lipschitz if and only if | F ( w 1 , z ) −

Original: F ( w 2 , z ) | ≤ L (cid:107) w 1 − w 2 (cid:107) 2 holds for any w 1 , w 2 ∈ R d and z ∈ Z .
Revised: F ( w 2 , z ) | ≤ L (cid:107) w 1 − w 2 (cid:107) 2 holds for any w 1 , w 2 ∈ R d and z ∈ Z .

Original: Deﬁnition 2 (Expected generalization error) .
Revised: Deﬁnition 2 (Expected generalization error) .

Original: The expected generalization error of a learning algorithm
Revised: The expected generalization error of a learning algorithm

Original: A is deﬁned as
Revised: A is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Algorithmic Stability.
Revised: Algorithmic Stability.

Original: Intuitively, a learning algorithm that is stable (i.e., a small perturbation of the training data does not affect its output too much) can generalize well.
Revised: Intuitively, a learning algorithm that is stable (i.e., a small perturbation of the training data does not affect its output too much) can generalize well.

Original: In the seminal work of Bousquet & Elisseeff (2002) (see also Hardt et al.
Revised: In the seminal work of Bousquet & Elisseeff (2002) (see also Hardt et al.

Original: , the authors formally deﬁned algorithmic stability and established a close connection between the stability of a learning algorithm and its generalization performance.
Revised: , the authors formally deﬁned algorithmic stability and established a close connection between the stability of a learning algorithm and its generalization performance.

Original: Deﬁnition 3 (Uniform stability) .
Revised: Deﬁnition 3 (Uniform stability) .

Original: (Bousquet & Elisseeff (2002); Elisseeff et al.
Revised: (Bousquet & Elisseeff (2002); Elisseeff et al.

Original: A randomized algorithm A is (cid:15) n -uniformly stable w.r.t.
Revised: A randomized algorithm A is (cid:15) n -uniformly stable w.r.t.

Original: loss L , if for all neighboring sets S, S (cid:48) ∈
Revised: loss L , if for all neighboring sets S, S (cid:48) ∈

Original: Z n , it holds that
Revised: Z n , it holds that

Original: [Equation]
Revised: [Equation]

Original: where w S and w S (cid:48) denote the outputs of A on S and S (cid:48) respectively.
Revised: where w S and w S (cid:48) denote the outputs of A on S and S (cid:48) respectively.

Original: Lemma 4 (Generalization in expectation) .
Revised: Lemma 4 (Generalization in expectation) .

Original: (Hardt et al.
Revised: (Hardt et al.

Original: Suppose a randomized algorithm
Revised: Suppose a randomized algorithm

Original: A is (cid:15) n -uniformly stable.
Revised: A is (cid:15) n -uniformly stable.

Original: Then, | err gen | ≤ (cid:15) n .
Revised: Then, | err gen | ≤ (cid:15) n .

Original:  B AYES -S TABILITY M ETHOD
Revised:  B AYES -S TABILITY M ETHOD

Original: In this section, we incorporate ideas from the PAC-Bayesian theory (see e.g., Lever et al.
Revised: In this section, we incorporate ideas from the PAC-Bayesian theory (see e.g., Lever et al.

Original: into the algorithmic stability framework.
Revised: into the algorithmic stability framework.

Original: Combined with the technical tools introduced in previous sections, the new framework enables us to prove tighter data-dependent generalization bounds.
Revised: Combined with the technical tools introduced in previous sections, the new framework enables us to prove tighter data-dependent generalization bounds.

Original: First, we deﬁne the posterior of a dataset and the posterior of a single data point.
Revised: First, we deﬁne the posterior of a dataset and the posterior of a single data point.

Original: Deﬁnition 5 (Single-point posterior) .
Revised: Deﬁnition 5 (Single-point posterior) .

Original: Let Q S be the posterior distribution of the parameter for a given training dataset S
Revised: Let Q S be the posterior distribution of the parameter for a given training dataset S

Original: =
Revised: =

Original: ( z 1 , . . .
Revised: ( z 1 , . . .

Original: , z n ) .
Revised: , z n ) .

Original: In other words, it is the probability distribution of the output of the learning algorithm on dataset S (e.g., for T iterations of SGLD in (1) , Q S is the pdf of
Revised: In other words, it is the probability distribution of the output of the learning algorithm on dataset S (e.g., for T iterations of SGLD in (1) , Q S is the pdf of

Original: W T ).
Revised: W T ).

Original: The single-point posterior Q ( i,z ) is deﬁned as
Revised: The single-point posterior Q ( i,z ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: For convenience, we make the following natural assumption on the learning algorithm:
Revised: For convenience, we make the following natural assumption on the learning algorithm:

Original: Assumption 6 (Order-independent) .
Revised: Assumption 6 (Order-independent) .

Original: For any ﬁxed dataset S = ( z 1 , . . .
Revised: For any ﬁxed dataset S = ( z 1 , . . .

Original: , z n ) and any permutation p , Q S is the same as Q S p , where S p =
Revised: , z n ) and any permutation p , Q S is the same as Q S p , where S p =

Original: ( z p 1 , . . .
Revised: ( z p 1 , . . .

Original: , z p n ) .
Revised: , z p n ) .

Original: Assumption 6 implies Q (1 ,z ) = · · · = Q ( n,z ) , so we use Q z as a shorthand for Q ( i,z ) in the following.
Revised: Assumption 6 implies Q (1 ,z ) = · · · = Q ( n,z ) , so we use Q z as a shorthand for Q ( i,z ) in the following.

Original: Note that this assumption can be easily satisﬁed by letting the learning algorithm randomly permute the training data at the beginning.
Revised: Note that this assumption can be easily satisﬁed by letting the learning algorithm randomly permute the training data at the beginning.

Original: It is also easy to verify that both SGD and SGLD satisfy the order-independent assumption.
Revised: It is also easy to verify that both SGD and SGLD satisfy the order-independent assumption.

Original: Now, we state our new Bayes-stability framework, which holds for any prior distribution P over the parameter space that is independent of the training dataset S .
Revised: Now, we state our new Bayes-stability framework, which holds for any prior distribution P over the parameter space that is independent of the training dataset S .

Original: Theorem 7 (Bayes-Stability) .
Revised: Theorem 7 (Bayes-Stability) .

Original: Suppose the loss function L ( w, z ) is C -bounded
Revised: Suppose the loss function L ( w, z ) is C -bounded

Original: and the learning algorithm is order-independent (Assumption 6).
Revised: and the learning algorithm is order-independent (Assumption 6).

Original: Then for any prior distribution P not depending on S , the generalization error is bounded by both 2 C E z (cid:104)(cid:112) 2KL( P, Q z ) (cid:105) and 2 C E z (cid:104)(cid:112) 2KL
Revised: Then for any prior distribution P not depending on S , the generalization error is bounded by both 2 C E z (cid:104)(cid:112) 2KL( P, Q z ) (cid:105) and 2 C E z (cid:104)(cid:112) 2KL

Original: ( Q z , P ) (cid:105) .
Revised: ( Q z , P ) (cid:105) .

Original: Remark 8.
Revised: Remark 8.

Original: Our Bayes-Stability framework originates from the algorithmic stability framework, and hence is similar to the notions of uniform stability and leave-one-out error (see Elisseeff et al.
Revised: Our Bayes-Stability framework originates from the algorithmic stability framework, and hence is similar to the notions of uniform stability and leave-one-out error (see Elisseeff et al.

Original: However, there are important differences.
Revised: However, there are important differences.

Original: Uniform stability is a distribution-independent property, while Bayes-Stability can incorporate the information of the data distribution (through the prior P ).
Revised: Uniform stability is a distribution-independent property, while Bayes-Stability can incorporate the information of the data distribution (through the prior P ).

Original: Leave-one-out error measures the loss of a learned model on an unseen data point, yet
Revised: Leave-one-out error measures the loss of a learned model on an unseen data point, yet

Original: Bayes-Stability focuses on the extent to which a single data point affects the outcome of the learning algorithm (compared to the prior).
Revised: Bayes-Stability focuses on the extent to which a single data point affects the outcome of the learning algorithm (compared to the prior).

Original: To establish an intuition, we ﬁrst apply this framework to obtain an expectation generalization bound for (full) gradient Langevin dynamics (GLD), which is a special case of SGLD in (1) (i.e., GLD uses the full gradient ∇ w F ( W t − 1 , S ) as g t ( W t − 1 ) ).
Revised: To establish an intuition, we ﬁrst apply this framework to obtain an expectation generalization bound for (full) gradient Langevin dynamics (GLD), which is a special case of SGLD in (1) (i.e., GLD uses the full gradient ∇ w F ( W t − 1 , S ) as g t ( W t − 1 ) ).

Original: Theorem 9.
Revised: Theorem 9.

Original: Suppose that the loss function L is C -bounded.
Revised: Suppose that the loss function L is C -bounded.

Original: Then we have the following expected generalization bound for T iterations of GLD:
Revised: Then we have the following expected generalization bound for T iterations of GLD:

Original: [Equation]
Revised: [Equation]

Original: where g e ( t ) =
Revised: where g e ( t ) =

Original: E w ∼ W t − 1 [ 1 n (cid:80) ni
Revised: E w ∼ W t − 1 [ 1 n (cid:80) ni

Original: =1 (cid:107)∇ F ( w, z i ) (cid:107) 22 ] is the empirical squared gradient norm, and W t is the parameter at step t of GLD.
Revised: =1 (cid:107)∇ F ( w, z i ) (cid:107) 22 ] is the empirical squared gradient norm, and W t is the parameter at step t of GLD.

Original: Proof The proof builds upon the following technical lemma, which we prove in Appendix A.2.
Revised: Proof The proof builds upon the following technical lemma, which we prove in Appendix A.2.

Original: Lemma 10.
Revised: Lemma 10.

Original: Let ( W 0 , . . .
Revised: Let ( W 0 , . . .

Original: , W T ) and ( W (cid:48) 0 , . . .
Revised: , W T ) and ( W (cid:48) 0 , . . .

Original: , W (cid:48) T ) be two independent sequences of random variables such that for each t ∈ { 0 , . .
Revised: , W (cid:48) T ) be two independent sequences of random variables such that for each t ∈ { 0 , . .

Original: , T } , W t and W (cid:48) t have the same support.
Revised: , T } , W t and W (cid:48) t have the same support.

Original: Suppose W 0 and W (cid:48)follow the same distribution.
Revised: Suppose W 0 and W (cid:48)follow the same distribution.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where W ≤ t denotes ( W 0 , . .
Revised: where W ≤ t denotes ( W 0 , . .

Original: . , W t ) and W <t denotes W ≤ t − 1 .
Revised: . , W t ) and W <t denotes W ≤ t − 1 .

Original: [Equation]
Revised: [Equation]

Original: ] , where 0 denotes the zero data point (i.e., f ( w, 0 ) = 

Revised: , where 0 denotes the zero data point (i.e., F ( w, 0 ) =

Original: for any w ).
Revised: for any w ).

Original: Theorem 7 shows that
Revised: Theorem 7 shows that

Original: [Equation]
Revised: [Equation]

Original: By the convexity of KL-divergence, for a ﬁxed z ∈ Z , we have
Revised: By the convexity of KL-divergence, for a ﬁxed z ∈ Z , we have

Original: [Equation]
Revised: [Equation]

Original: Let ( W t ) t ≥ 0 and ( W (cid:48) t ) t ≥ 0 be the training process of GLD for S
Revised: Let ( W t ) t ≥ 0 and ( W (cid:48) t ) t ≥ 0 be the training process of GLD for S

Original: = ( S, z ) and S (cid:48) =
Revised: = ( S, z ) and S (cid:48) =

Original: ( S, 0 ) , respectively.
Revised: ( S, 0 ) , respectively.

Original: Note that for a ﬁxed w <t , both W t | W <t = w <t and W (cid:48) t | W (cid:48) <
Revised: Note that for a ﬁxed w <t , both W t | W <t = w <t and W (cid:48) t | W (cid:48) <

Original: t = w <t are Gaussian distributions.
Revised: t = w <t are Gaussian distributions.

Original: Since KL( N ( µ 1 , σ 2 I ) , N ( µ 2 , σ 2 I ))
Revised: Since KL( N ( µ 1 , σ 2 I ) , N [Equation] µ [Equation]

Original: [Equation]
Revised: [Equation]

Original: (see Lemma 18 in Appendix A.2).
Revised: (see Lemma 18 in Appendix A.2).

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma 10 and KL( W T , W (cid:48) T ) ≤ KL( W ≤ T , W (cid:48)≤ T ) gives
Revised: Applying Lemma 10 and KL( W T , W (cid:48) T ) ≤ KL( W ≤ T , W (cid:48)≤ T ) gives

Original: [Equation]
Revised: [Equation]

Original: Recall that W t − 1 is the parameter at step t − 1 using S =
Revised: Recall that W t − 1 is the parameter at step t − 1 using S =

Original: ( S, z ) as dataset.
Revised: ( S, z ) as dataset.

Original: In this case, we can rewrite z as z n since it is the n -th data point of S .
Revised: In this case, we can rewrite z as z n since it is the n -th data point of S .

Original: Note that SGLD satisﬁes the order-independent assumption, we can rewrite z as z i for all i ∈ [ n ] .
Revised: Note that SGLD satisﬁes the order-independent assumption, we can rewrite z as z i for all i ∈ [ n ] .

Original: Together with (5), (6), and using 1 n (cid:80) ni =1 √ x i ≤ (cid:113) 1 n (cid:80) ni =1 x i , we can prove this theorem.
Revised: Together with (5), (6), and using 1 n (cid:80) ni =1 √ x i ≤ (cid:113) 1 n (cid:80) ni =1 x i , we can prove this theorem.

Original: More generally, we give the following bound for SGLD.
Revised: More generally, we give the following bound for SGLD.

Original: The proof is similar to that of Theorem 9; the difference is that we need to bound the KL-divergence between two Gaussian mixtures instead of two Gaussians.
Revised: The proof is similar to that of Theorem 9; the difference is that we need to bound the KL-divergence between two Gaussian mixtures instead of two Gaussians.

Original: This proof is more technical and deferred to Appendix A.3.
Revised: This proof is more technical and deferred to Appendix A.3.

Original: Theorem 11.
Revised: Theorem 11.

Original: Suppose that the loss function L is C -bounded and the objective function f is Llipschitz.
Revised: Suppose that the loss function L is C -bounded and the objective function f is Llipschitz.

Original: Assume that the following conditions hold:
Revised: Assume that the following conditions hold:

Original: Batch size b ≤ n/ 2 .
Revised: Batch size b ≤ n/ 2 .

Original: Learning rate γ t ≤ σ t / (20 L ).
Revised: Learning rate γ t ≤ σ t / (20 L )

Original:  Then, the following expected generalization error bound holds for T iterations of SGLD (1) :
Revised: [Equation] Then, the following expected generalization error bound holds for T iterations of SGLD (1) :

Original: [Equation]
Revised: [Equation]

Original: where g e ( t ) =
Revised: where g e ( t ) =

Original: E w ∼ W t − 1 [ 1 n [Equation]
Revised: E w ∼ W t − 1 [ 1 n (cid:80) ni

Original: ni = [Equation] F ( w, z i ) (cid:107) 22 ] is the empirical squared gradient norm, and W t is the parameter at step t of SGLD.
Revised: =1 (cid:107)∇ F ( w, z i ) (cid:107) 22 ] is the empirical squared gradient norm, and W t is the parameter at step t of SGLD.

Original: Furthermore, based on essentially the same proof, we can obtain the following bound that depends on the population gradient norm :
Revised: Furthermore, based on essentially the same proof, we can obtain the following bound that depends on the population gradient norm :

Original: [Equation]
Revised: [Equation]

Original: The full proofs of the above results are postponed to Appendix A, and we provide some remarks about the new bounds.[Figure] r [Figure]
Revised: The full proofs of the above results are postponed to Appendix A, and we provide some remarks about the new bounds.

Original: i

Original: [Figure]

Original: n t n

Original: [Figure]

Original: r m (d)

Original: Remark 12.
Revised: Remark 12.

Original: In fact, our proof establishes that the above upper bound holds for the two sequences W ≤ T and W (cid:48)≤ T : KL( W ≤ T , W (cid:48)≤ T ) ≤ 8 .
Revised: In fact, our proof establishes that the above upper bound holds for the two sequences W ≤ T and W (cid:48)≤ T : KL( W ≤ T , W (cid:48)≤ T ) ≤ 8 .

Original: 23 n 2 (cid:80) Tt =1 γ 2 t σ 2 t g e ( t ) .
Revised: 12 n 2 (cid:80) Tt =1 γ 2 t σ 2 t g e ( t ) .

Original: Hence, our bound holds for any sufﬁciently regular function over the parameter sequences:
Revised: Hence, our bound holds for any sufﬁciently regular function over the parameter sequences:

Original: KL( f ( W ≤ T ) , f ( W (cid:48)≤ T )) ≤ 8 . 
Revised: KL( f ( W ≤ T ) , f ( W (cid:48)≤ T )) ≤ 8 . 12 n

Original: 23 n 2 (cid:80) Tt =1 γ 2 t σ 2 t g e ( t ) .
Revised: [Equation] Tt = [Equation] t ) .

Original: In particular, our generalization error bound automatically extends to several variants of SGLD, such as outputting the average of the trajectory, the average of the sufﬁx of certain length, or the exponential moving average. 
Revised: In particular, our generalization error bound automatically extends to several variants of SGLD, such as outputting the average of the trajectory, the average of the sufﬁx of certain length, or the exponential moving average. Remark 13 (High-probability bounds) .

Revised: By relaxing the expected squared gradient norm term to Land using the uniform stability framework, our proof can be adapted to recover the O ( LC √ T/n ) bound in (Mou et al., 2018, Theorem 1).

Revised: Then, we can apply the recent results of Feldman &

Original: Remark 13.
Revised: Vondrak (2019) to provide a generalization error bound of ˜ O ( LC √ T/n + 1 / √ n ) that holds with high probability. (Here ˜ O hides poly-logarithmic factors.) When T is at least linear in n , the additional 1 / √ n term is not dominating.

Original:  Inspired by Zhang et al. (2017a), we run both GLD (Figure 1) and SGLD (Appendix C.2) to ﬁt both normal data and randomly labelled data (see Appendix C for more experiment details).
Revised: Distinguish random from normal. Inspired by Zhang et al. (2017a), we run both GLD (Figure 1) and SGLD (Appendix C.2) to ﬁt both normal data and randomly labeled data (see Appendix C for more experiment details).

Original: As shown in Figure 1 and Figure 2 in Appendix C.2,  larger random label portion p leads to both much higher generalization error and much larger generalization error bound.
Revised: As shown in Figure 1 and Figure 3 in Appendix C.2, a larger random label portion p leads to both much higher generalization error and much larger generalization error bound.

Original: Moreover, the shapes of the curves  our bounds look quite similar to that of the generalization error curves.
Revised: Moreover, the shapes of the curves of our bounds look quite similar to those of the generalization error curves.

Revised: Note that in (b) and (c) of Figure 1, the scales in the y -axis are different.

Revised: We list some possible reasons that may explain why our bound is larger than the actual generalization error.

Revised: (1) as we explained in Remark 12, our bounds (Theorem 9 and 11) hold for any trajectory-based output, and are much stronger than upper bounds for the last point on the trajectory.

Revised: The constant we can prove in Lemma 21 may not be very tight.

Original: G ENERALIZATION OF CLD AND GLD WITH (cid:96) 2 REGULARIZATION
Revised: The variance of Gaussian noise σ 2 t is not large enough in our experiment. However, if we choose a larger variance, ﬁtting the random labeled training data becomes quite slow. Hence, we use a small data size ( n = 10000 ) for the above reason. We also run an extra experiment for GLD on the full MNIST dataset ( n = 60000 ) without label corruption (see Figure 2 in the Appendix C).

Revised: We can see that our bound is non-vacuous (since GLD—which computes the full gradients—took a long time to converge, we stopped when we achieved 90% training accuracy).

Revised: [Figure]

Revised: Relax the step size constraint.

Revised: The condition on the step size in Theorem 11 may seem restrictive in the practical use.

Revised: 5 We provide several ways to relax this constraint:

Revised: The proof

Revised: of

Revised: Theorem still goes through if we replace

Revised: L with

Revised: max i ∈ [ n ] (cid:107)∇ F ( W t − 1 , z i ) (cid:107) 2 in the constraint.

Revised: The maximum gradient norm can be controlled by gradient clipping, i.e., multiplying min( C L , (cid:107)∇ F ( W t − 1 ,z i ) (cid:107) 2 ) (cid:107)∇ F ( W t − 1 ,z i ) (cid:107)to each ∇ F ( W t − 1 , z i ) .

Revised: Replacing the constant 20 with 2 in this constraint will only increase the constant of our bound from 8 .

Revised: 12 to 84 .

Revised: We also provide an experiment combining the above ideas to make our Theorem 11 applicable in the practical use (see Figure 4 in Appendix C).

Revised:  G ENERALIZATION OF CLD AND GLD WITH (cid:96) 2 REGULARIZATION

Original: In this section, we study the generalization error of Continuous Langevin Dynamics (CLD) with (cid:96)regularization.
Revised: In this section, we study the generalization error of Continuous Langevin Dynamics (CLD) with (cid:96)regularization.

Original: Throughout this section, we assume that the objective function over training set S is deﬁned as F ( w, S )
Revised: Throughout this section, we assume that the objective function over training set S is deﬁned as F ( w, S )

Original: =
Revised: =

Original: F 0 ( w, S ) + λ 2 (cid:107) w (cid:107) 22 , and moreover, the following assumption holds.
Revised: F 0 ( w, S ) + λ 2 (cid:107) w (cid:107) 22 , and moreover, the following assumption holds.

Original: Assumption 14.
Revised: Assumption 14.

Original: The loss function L and the original objective F 0 are C -bounded.
Revised: The loss function L and the original objective F 0 are C -bounded.

Original: Moreover, F 0 is differentiable and L -lipschitz.
Revised: Moreover, F 0 is differentiable and L -lipschitz.

Original: The Continuous Langevin Dynamics is deﬁned by the following SDE:
Revised: The Continuous Langevin Dynamics is deﬁned by the following SDE:

Original: [Equation]
Revised: [Equation]

Original: where ( B t ) t ≥ 0 is the standard Brownian motion on R d and the initial distribution µ 0 is the centered
Revised: where ( B t ) t ≥ 0 is the standard Brownian motion on R d and the initial distribution µ 0 is the centered

Original: Gaussian distribution in R d with covariance 1 λβ I d .
Revised: Gaussian distribution in R d with covariance 1 λβ I d .

Original: We show that the generalization error of CLD is upper bounded by O (cid:16) e 4 βC n − 1 (cid:112) β/λ (cid:17) , which is independent of the training time T (Theorem 15).
Revised: We show that the generalization error of CLD is upper bounded by O (cid:16) e 4 βC n − 1 (cid:112) β/λ (cid:17) , which is independent of the training time T (Theorem 15).

Original: Furthermore, as T goes to inﬁnity, we have a tighter generalization error bound O (cid:0) βC 2 n − 1 (cid:1) (Theorem 39 in Appendix B).
Revised: Furthermore, as T goes to inﬁnity, we have a tighter generalization error bound O (cid:0) βC 2 n − 1 (cid:1) (Theorem 39 in Appendix B).

Original: We also study the generalization of Gradient Langevin Dynamics (GLD), which is the discretization of CLD:
Revised: We also study the generalization of Gradient Langevin Dynamics (GLD), which is the discretization of CLD:

Original: [Equation]
Revised: [Equation]

Original: where ξ k is the standard Gaussian random vector in R d .
Revised: where ξ k is the standard Gaussian random vector in R d .

Original: By leveraging a result developed in Raginsky et al.
Revised: By leveraging a result developed in Raginsky et al.

Original: , we show that, as Kη 2 tends to zero, GLD has the same generalization as CLD (see
Revised: , we show that, as Kη 2 tends to zero, GLD has the same generalization as CLD (see

Original: Theorems 15 and 39).
Revised: Theorems 15 and 39).

Original: We ﬁrst formally state our ﬁrst main result in this section.
Revised: We ﬁrst formally state our ﬁrst main result in this section.

Original: Theorem 15.
Revised: Theorem 15.

Original: Under Assumption 14, CLD (with initial probability measure d µ [Equation] = [Equation]
Revised: Under Assumption 14, CLD (with initial probability measure d µ 0 = 1 Z e

Original: e

Original: [Equation]
Revised: [Equation]

Original: has the following expected generalization error bound:
Revised: has the following expected generalization error bound:

Original: [Equation]
Revised: [Equation]

Original: In addition, if L is M -smooth and non-negative, by setting λβ > 2 , λ > 12 and η ∈ [0 ,[Equation]
Revised: In addition, if L is M -smooth and non-negative, by setting λβ > 2 , λ > 0 and η ∈ (cid:2) 0 , 1 ∧ λ 8 M 2 (cid:1) ,

Original: GLD (running K iterations with the same µ 0 as CLD) has the expected generalization error bound:
Revised: GLD (running K iterations with the same µ 0 as CLD) has the expected generalization error bound:

Original: [Equation]
Revised: [Equation]

Original: where C 1 is a constant that only depends on M , λ , β , b , L and d .
Revised: where C 1 is a constant that only depends on M , λ , β , b , L and d .

Original: The following lemma is crucial for establishing the above generalization bound for CLD.
Revised: The following lemma is crucial for establishing the above generalization bound for CLD.

Original: In particular, we need to establish a Log-Sobolev inequality for µ t , the parameter distribution at time t , for every time step t > 0 .
Revised: In particular, we need to establish a Log-Sobolev inequality for µ t , the parameter distribution at time t , for every time step t > 0 .

Original: In contrast, most known LSIs only characterize the stationary distribution of the Markov process.
Revised: In contrast, most known LSIs only characterize the stationary distribution of the Markov process.

Original: The proof of the lemma can be found in Appendix B.
Revised: The proof of the lemma can be found in Appendix B.

Original: Lemma 16.
Revised: Lemma 16.

Original: Under Assumption 14, let µ t be the probability measure of W t in CLD (with d µ 0 =1 Z e
Revised: Under Assumption 14, let µ t be the probability measure of W t in CLD (with d µ 0 =

Original: [Equation]
Revised: [Equation]

Original: d w ).
Revised: d w ).

Original: Let ν be a probability measure that is absolutely continuous with respect to µ t .
Revised: Let ν be a probability measure that is absolutely continuous with respect to µ t .

Original: Suppose d µ t =
Revised: Suppose d µ t =

Original: π t ( w ) d w and d ν = γ ( w ) d w .
Revised: π t ( w ) d w and d ν = γ ( w ) d w .

Original: Then, it holds that
Revised: Then, it holds that

Original: [Equation]
Revised: [Equation]

Original: We sketch the proof of Theorem 15, and the complete proof is relegated to Appendix B.
Revised: We sketch the proof of Theorem 15, and the complete proof is relegated to Appendix B.

Original: Proof Sketch of Theorem 15 Suppose S and S (cid:48) are two neighboring datasets.
Revised: Proof Sketch of Theorem 15 Suppose S and S (cid:48) are two neighboring datasets.

Original: Let ( W t ) t ≥ 0 and ( W (cid:48) t ) t ≥ 0 be the process of CLD running on S and S (cid:48) , respectively.
Revised: Let ( W t ) t ≥ 0 and ( W (cid:48) t ) t ≥ 0 be the process of CLD running on S and S (cid:48) , respectively.

Original: Let γ t and π t be the pdf of W (cid:48) t and W t .
Revised: Let γ t and π t be the pdf of W (cid:48) t and W t .

Original: Let F S ( w ) denote F ( w, S ) .
Revised: Let F S ( w ) denote F ( w, S ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: n(Lemma 16)
Revised: (Lemma 16)

Original: Solving this inequality gives KL( γ t , π t ) ≤ 1 n 2 λ 2 βL 2 e 8 βC (1 − e − λt/e 8 βC ) .
Revised: Solving this inequality gives KL( γ t , π t ) ≤ 1 n 2 λ 2 βL 2 e 8 βC (1 − e − λt/e 8 βC ) .

Original: Hence the generalization error of CLD can be bounded by 2 C (cid:113) 12 KL( γ T , π T ) , which proves the ﬁrst part.
Revised: Hence the generalization error of CLD can be bounded by 2 C (cid:113) 12 KL( γ T , π T ) , which proves the ﬁrst part.

Original: The second part of the theorem follows from Lemma 36 in Appendix B.
Revised: The second part of the theorem follows from Lemma 36 in Appendix B.

Original: Our second generalization bound for CLD (Theorem 39 in Appendix B) is
Revised: Our second generalization bound for CLD (Theorem 39 in Appendix B) is

Original: [Equation]
Revised: [Equation]

Original: The high level idea to prove this bound is very similar to that in Raginsky et al.
Revised: The high level idea to prove this bound is very similar to that in Raginsky et al.

Original: We ﬁrst observe that the (stationary) Gibbs distribution µ has a small generalization error.
Revised: We ﬁrst observe that the (stationary) Gibbs distribution µ has a small generalization error.

Original: Then, we bound the distance from µ t to µ .
Revised: Then, we bound the distance from µ t to µ .

Original: In our setting, we can use the Holley-Stroock perturbation lemma which allows us to bound the Logarithmic Sobolev constant, and we can thus bound the above distance easily.
Revised: In our setting, we can use the Holley-Stroock perturbation lemma which allows us to bound the Logarithmic Sobolev constant, and we can thus bound the above distance easily.

Original:  F UTURE D IRECTIONS
Revised:  F UTURE D IRECTIONS

Original: In this paper, we prove new generalization bounds for a variety of noisy gradient-based methods.
Revised: In this paper, we prove new generalization bounds for a variety of noisy gradient-based methods.

Original: Our current techniques can only handle continuous noises for which we can bound the KL-divergence.
Revised: Our current techniques can only handle continuous noises for which we can bound the KL-divergence.

Original: One future direction is to study the discrete noise introduced in SGD (in this case the KL-divergence may not be well deﬁned).
Revised: One future direction is to study the discrete noise introduced in SGD (in this case the KL-divergence may not be well deﬁned).

Original: For either SGLD or CLD, if the noise level is small (i.e., β is large), it may take a long time for the diffusion process to reach the stable distribution.
Revised: For either SGLD or CLD, if the noise level is small (i.e., β is large), it may take a long time for the diffusion process to reach the stable distribution.

Original: Hence, another interesting future direction is to consider the local behavior and generalization of the diffusion process in ﬁnite time through the techniques developed in the studies of metastability (see e.g., Bovier et al.
Revised: Hence, another interesting future direction is to consider the local behavior and generalization of the diffusion process in ﬁnite time through the techniques developed in the studies of metastability (see e.g., Bovier et al.

Original: Bovier & den Hollander (2006); Tzen et al.
Revised: Bovier & den Hollander (2006); Tzen et al.

Original: In particular, the technique may be helpful for further improving the bounds in Theorems 15 and 39 (when T is not very large).  
Revised: In particular, the technique may be helpful for further improving the bounds in Theorems 15 and 39 (when T is not very large).  A CKNOWLEDGEMENT

Revised: We would like to thank Liwei Wang for several helpful discussions during various stages of the work.

Revised: The research is supported in part by the National Natural Science Foundation of China Grant 61822203, 61772297, 61632016, and the Zhongguancun Haihua Institute for Frontier

Revised: Information Technology and Turing AI Institute of Nanjing

Original: A P ROOFS IN S ECTIONA. B AYES -S TABILITY F RAMEWORK
Revised: A P ROOFS IN S ECTIONA. B AYES -S TABILITY F RAMEWORK

Original: Lemma 17.
Revised: Lemma 17.

Original: Under Assumption 6, for any prior distribution P not depending on the dataset S = ( z 1 , . . .
Revised: Under Assumption 6, for any prior distribution P not depending on the dataset S = ( z 1 , . . .

Original: , z n ) , the generalization error is upper bounded by
Revised: , z n ) , the generalization error is upper bounded by

Original: [Equation]
Revised: [Equation]

Original: where L ( w ) denotes the population loss L ( w, D ) .
Revised: where L ( w ) denotes the population loss L ( w, D ) .

Original: Proof of Lemma 17 Let err train =
Revised: Proof of Lemma 17 Let err train =

Original: E S E w ∼ Q S L ( w, S ) and err test =
Revised: E S E w ∼ Q S L ( w, S ) and err test =

Original: E S E w ∼ Q S L ( w ) .
Revised: E S E w ∼ Q S L ( w ) .

Original: We can rewrite generalization error as err gen = err test − err train , where
Revised: We can rewrite generalization error as err gen = err test − err train , where

Original: [Equation]
Revised: [Equation]

Original: E 
Revised: E z

Original: [Equation]
Revised: [Equation]

Original: and err 
Revised: and err train = 1 n n (cid:88)

Original: [Equation]
Revised: [Equation]

Revised: n

Original: [Equation]
Revised: [Equation]

Revised: Assumption 6)

Original: [Equation]
Revised: [Equation]

Original: (deﬁnition of f ( w [Equation]
Revised: (deﬁnition of f ( w ) )

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now we are ready to prove Theorem
Revised: Now we are ready to prove Theorem

Original: [Equation]
Revised: [Equation]

Original: which we restate in the following.
Revised: which we restate in the following.

Original: Theorem 7 (Bayes-Stability) .
Revised: Theorem 7 (Bayes-Stability) .

Original: Suppose the loss function L ( w, z ) is C -bounded
Revised: Suppose the loss function L ( w, z ) is C -bounded

Original: and the learning algorithm is order-independent (Assumption 6), then for any prior distribution P not depending on S , the generalization error is bounded by both 2 C E z (cid:104)(cid:112) 2KL( P, Q z ) (cid:105) and 2 C E z (cid:104)(cid:112) 2KL
Revised: and the learning algorithm is order-independent (Assumption 6), then for any prior distribution P not depending on S , the generalization error is bounded by both 2 C E z (cid:104)(cid:112) 2KL( P, Q z ) (cid:105) and 2 C E z (cid:104)(cid:112) 2KL

Original: ( Q z , P ) (cid:105) .
Revised: ( Q z , P ) (cid:105) .

Original: Proof By Lemma 17,
Revised: Proof By Lemma 17,

Original: [Equation]
Revised: [Equation]

Original: Pinsker’s inequality
Revised: inequality

Original: [Equation]
Revised: [Equation]

Original: The other bound follows from a similar argument
Revised: The other bound follows from a similar argument

Original: [Equation]
Revised: [Equation]

Original: A. T ECHNICAL L EMMAS
Revised: A. T ECHNICAL L EMMAS

Original: Now we turn to the proof of Theorem 11.
Revised: Now we turn to the proof of Theorem 11.

Original: The following lemma allows us to reduce the proof of algorithmic stability to the analysis of a single update step.
Revised: The following lemma allows us to reduce the proof of algorithmic stability to the analysis of a single update step.

Original: Lemma 10.
Revised: Lemma 10.

Original: Let ( W 0 , . . .
Revised: Let ( W 0 , . . .

Original: , W T ) and ( W (cid:48) 0 , . . .
Revised: , W T ) and ( W (cid:48) 0 , . . .

Original: , W (cid:48) T ) be two independent sequences of random variables such that for each t ∈ { 0 , . .
Revised: , W (cid:48) T ) be two independent sequences of random variables such that for each t ∈ { 0 , . .

Original: , T } , W t and W (cid:48) t have the same support.
Revised: , T } , W t and W (cid:48) t have the same support.

Original: Suppose W 0 and W (cid:48)follow the same distribution.
Revised: Suppose W 0 and W (cid:48)follow the same distribution.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where W ≤ t denotes ( W 0 , . .
Revised: where W ≤ t denotes ( W 0 , . .

Original: . , W t ) and W <t denotes W ≤ t − 1 .
Revised: . , W t ) and W <t denotes W ≤ t − 1 .

Original: Proof By the chain rule of the KL-divergence,
Revised: Proof By the chain rule of the KL-divergence,

Original: [Equation]
Revised: [Equation]

Original: The lemma follows from a summation over t = 1 , . .
Revised: The lemma follows from a summation over t = 1 , . .

Original: , T .
Revised: , T .

Original: The following lemma (see e.g., (Duchi, 2007, Section 9)) gives a closed-form formula for the KLdivergence between two Gaussian distributions.
Revised: The following lemma (see e.g., (Duchi, 2007, Section 9)) gives a closed-form formula for the KLdivergence between two Gaussian distributions.

Original: Lemma 18.
Revised: Lemma 18.

Original: Suppose that P = N ( µ 1 , Σ 1 ) and Q = N ( µ 2 , Σ 2 ) are two Gaussian distributions on R d .
Revised: Suppose that P = N ( µ 1 , Σ 1 ) and Q = N ( µ 2 , Σ 2 ) are two Gaussian distributions on R d .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: The following lemma (Topsoe, 2000, Theorem 3) helps us to upper bound the KL-divergence.
Revised: The following lemma (Topsoe, 2000, Theorem 3) helps us to upper bound the KL-divergence.

Original: Deﬁnition 19.
Revised: Deﬁnition 19.

Original: Let P and Q be two probability distributions on R d .
Revised: Let P and Q be two probability distributions on R d .

Original: The directional triangular discrimination from P to Q is deﬁned as
Revised: The directional triangular discrimination from P to Q is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Lemma 20.
Revised: Lemma 20.

Original: For any two probability distributions P and Q on R d ,
Revised: For any two probability distributions P and Q on R d ,

Original: [Equation]
Revised: [Equation]

Original: Recall that G is the set of all possible mini-batches.
Revised: Recall that G is the set of all possible mini-batches.

Original: G n = { B ∈ G : n ∈ B } denotes the collection of mini-batches that contain n , while G n =
Revised: G n = { B ∈ G : n ∈ B } denotes the collection of mini-batches that contain n , while G n =

Original: G \ G n .
Revised: G \ G n .

Original: diam( A ) =
Revised: diam( A ) =

Original: sup x,y ∈ A (cid:107) x − y (cid:107) denotes the diameter of set A .
Revised: sup x,y ∈ A (cid:107) x − y (cid:107) denotes the diameter of set A .

Original: The following technical lemma upper bounds the KL-divergence between two
Revised: The following technical lemma upper bounds the KL-divergence between two

Original: Gaussian mixtures induced by sampling a mini-batch from neighbouring datasets.
Revised: Gaussian mixtures induced by sampling a mini-batch from neighbouring datasets.

Original: Lemma 21.
Revised: Lemma 21.

Original: Suppose that batch size b ≤ n/ 2 .
Revised: Suppose that batch size b ≤ n/ 2 .

Original: { µ B : B ∈ G } and { µ (cid:48) B : B ∈ G } are two collections of points in R d labeled by mini-batches of size b
Revised: { µ B : B ∈ G } and { µ (cid:48) B : B ∈ G } are two collections of points in R d labeled by mini-batches of size b

Original: that satisfy the following conditions for some constant β ∈ [0 , σ ] :
Revised: that satisfy the following conditions for some constant β ∈ [0 , σ ] :

Original: 1. (cid:107) µ B − µ (cid:48) B (cid:107) ≤ β for B ∈ G n and µ B = µ (cid:48) B for B ∈ G n . 2.
Revised: 1. (cid:107) µ B − µ (cid:48) B (cid:107) ≤ β for B ∈ G n and µ B = µ (cid:48) B for B ∈ G n . 2.

Original: diam( { µ B : B ∈ G } ∪ { µ (cid:48)
Revised: diam( { µ B : B ∈ G } ∪ { µ (cid:48)

Original: B : B ∈ G } ) ≤ σ/ 10 .
Revised: B : B ∈ G } ) ≤ σ/ 10 .

Original: Let p µ,σ denote the Gaussian distribution N ( µ, σ 2 2 I d ) .
Revised: Let p µ,σ denote the Gaussian distribution N ( µ, σ 2 2 I d ) .

Original: Let P = 1 | G | (cid:80) B ∈ G p µ B ,σ and P (cid:48)
Revised: Let P = 1 | G | (cid:80) B ∈ G p µ B ,σ and P (cid:48)

Original: =
Revised: =

Original: 1 | G | (cid:80) B ∈ G p µ (cid:48) B ,σ be two mixture distributions over all mini-batches.
Revised: 1 | G | (cid:80) B ∈ G p µ (cid:48) B ,σ be two mixture distributions over all mini-batches.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 21 By Lemma 20, KL( P, P (cid:48) ) is bounded by
Revised: Proof of Lemma 21 By Lemma 20, KL( P, P (cid:48) ) is bounded by

Original: [Equation]
Revised: [Equation]

Original: The numerator of the above integrand is upper bounded by
Revised: The numerator of the above integrand is upper bounded by

Original: [Equation]
Revised: [Equation]

Original: while the denominator can be lower bounded as follows:
Revised: while the denominator can be lower bounded as follows:

Original: [Equation]
Revised: [Equation]

Original:  B
Revised: for B

Original: [Equation]
Revised: [Equation]

Original: {+which implies, by the convexity of 
+}
Revised: {+which implies, by the convexity of 
+}

Original: B
Revised: /x

Original: [Equation]
Revised: [Equation]

Original: which implies, by the convexity of 1 /x , that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: Inequalities (9) and (10) together imply
Revised: Inequalities (9) and (10) together imply

Original: [Equation]
Revised: [Equation]

Original: Now we bound the right-hand side of (11) for ﬁxed A and B .
Revised: Now we bound the right-hand side of (11) for ﬁxed A and B .

Original: By applying a translation and a rotation, we can assume without loss of generality that µ A = 0 , and the last d − 2 coordinates of µ B and µ (cid:48)
Revised: By applying a translation and a rotation, we can assume without loss of generality that µ A = 0 , and the last d − 2 coordinates of µ B and µ (cid:48)

Original: B are all zero.
Revised: B are all zero.

Original: Note that the integral is unchanged when we project the space to the twodimensional subspace corresponding to the ﬁrst two coordinates.
Revised: Note that the integral is unchanged when we project the space to the twodimensional subspace corresponding to the ﬁrst two coordinates.

Original: Thus, it sufﬁces to prove a bound for d = 2 .
Revised: Thus, it sufﬁces to prove a bound for d = 2 .

Original: We rewrite (11) as
Revised: We rewrite (11) as

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Let I be the integral in the right-hand side of (12).
Revised: Let I be the integral in the right hand side of ( Note that µ [Equation]

Original: Note that (cid:13)(cid:13) µ B σ µ  σ and  σ (cid:13)(cid:13)(cid:13) ≤ βσ .
Revised: σ µ B σ and B σ (cid:13)(cid:13)(cid:13) ≤ βσ .

Original: Let δ = βσ and r = (cid:13)(cid:13) wσ
Revised: Let δ = βσ and r = (cid:13)(cid:13) wσ

Original: (cid:13)(cid:13) .
Revised: (cid:13)(cid:13) .

Original: Our goal is to bound max δ ∈ [0 , 0 .
Revised: Our goal is to bound max δ ∈ [0 , 0 .

Original: ( Iδ − 2 ) .
Revised: ( Iδ − 2 ) .

Original: Let ( x ) + = max( x, 0) .
Revised: Let ( x ) + = max( x, 0) .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: We have max
Revised: We have max

Original: [Equation]
Revised: [Equation]

Original: Let φ ( y, δ ) =
Revised: Let φ ( y, δ ) =

Original: ( e − y 2 − e − ( y + δ )2 δ ) 2 , we make two claims which we will prove later:
Revised: ( e − y 2 − e − ( y + δ )2 δ ) 2 , we make two claims which we will prove later:

Original: For all y, δ ≥ 0 , φ ( y, δ ) ≤ 2 e . 2.
Revised: For all y, δ ≥ 0 , φ ( y, δ ) ≤ 2 e . 2.

Original: For all y ≥ 1 √ 2 , φ ( y, δ ) is non-increasing in δ .
Revised: For all y ≥ 1 √ 2 , φ ( y, δ ) is non-increasing in δ .

Original: The above claims imply that:
Revised: The above claims imply that:

Original: For any r ∈ (cid:104) 0 , 1 √ 2 + 0 .
Revised: For any r ∈ (cid:104) 0 , 1 √ 2 + 0 .

Original: 1 (cid:105) , max y ∈
Revised: 1 (cid:105) , max y ∈

Original: [( r − 0 . [Equation] ) + ,r +0 . 1] ,δ ∈ [0 , 0 .1]
Revised: [( r − 0 . 1) + ,r +0 . 1] ,δ ∈ [0 , 0 .

Original: [ φ ( y, δ )] ≤
Revised: φ ( y, δ )] ≤

Original: [Equation]
Revised: [Equation]

Original: For any r ∈ (cid:16) 1 √ 2 + 0 . 1 , + ∞ (cid:17) , we have
Revised: For any r ∈ (cid:16) 1 √ 2 + 0 . 1 , + ∞ (cid:17) , we have

Original: [Equation]
Revised: [Equation]

Original: The last step holds since y (cid:55)→ y 2 e − 2 y 2 is decreasing on (cid:104) 1 √ 2 , + ∞ (cid:17) .
Revised: The last step holds since y (cid:55)→ y 2 e − 2 y 2 is decreasing on (cid:104) 1 √ 2 , + ∞ (cid:17) .

Original: Thus we have
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: Plugging the above into (12) gives
Revised: Plugging the above into (12) gives

Original: [Equation]
Revised: [Equation]

Original: We conclude that
Revised: We conclude that

Original: [Equation]
Revised: [Equation]

Original: Finally, we prove the two claims used above:
Revised: Finally, we prove the two claims used above:

Original: For all y, δ ≥ 0 , let h ( x ) =
Revised: For all y, δ ≥ 0 , let h ( x ) =

Original: e − x 2 , we have e − y 2 − e − ( y + δ ) 2 = (cid:82)
Revised: e − x 2 , we have e − y 2 − e − ( y + δ ) 2 = (cid:82)

Original: y y
Revised: y y

Original: + δ h (cid:48)
Revised: + δ h (cid:48)

Original: ( t ) d t .
Revised: ( t ) d t .

Original: Since | h (cid:48) ( t ) | = | e − t 2 ( − 2 t ) | =
Revised: Since | h (cid:48) ( t ) | = | e − t 2 ( − 2 t ) | =

Original: e − t 2 (2 t ) .
Revised: e − t 2 (2 t ) .

Original: Let ∂∂t | h (cid:48)
Revised: Let ∂∂t | h (cid:48)

Original: ( t ) |
Revised: ( t ) |

Original: =
Revised: =

Original: 0 , we have t = 1 / √ 2 .
Revised: 0 , we have t = 1 / √ 2 .

Original: Thus, | h (cid:48) ( t ) | ≤ e 1 / 2 √ 2 and e − y 2 − e − ( y + δ ) 2 ≤ δ (cid:112) 2 /e .
Revised: Thus, | h (cid:48) ( t ) | ≤ e 1 / 2 √ 2 and e − y 2 − e − ( y + δ ) 2 ≤ δ (cid:112) 2 /e .

Original: Suppose y ≥1 / √2 , we have
Revised: Suppose y ≥ / √ , we have

Original: [Equation]
Revised: [Equation]

Original: Let g ( y, δ ) =
Revised: Let g ( y, δ ) =

Original: e − 2 yδ − δ 2 (2 yδ + 2 δ 2 + 1) − 1 .
Revised: e − 2 yδ − δ 2 (2 yδ + 2 δ 2 + 1) − 1 .

Original: Note that
Revised: Note that

Original: (a) lim δ
Revised: (a) lim δ

Original: → 0 ∂∂δ ( e − y 
Revised: → 0 ∂∂δ ( e − y 2 − e − ( y +

Original: [Equation]
Revised: [Equation]

Original: − (

Original: [Equation]
Revised: [Equation]

Original: (b) ∂∂δ g (1 / √ 2 , δ ) = − 4 e − δ
Revised: (b) ∂∂δ g (1 / √ 2 , δ ) = − 4 e − δ

Original: ( δ + √ 2) δ 2 ( δ {++ √ 2) < 
+}
Revised: ( δ + √ 2) δ 2 ( δ {++ √ 2) < 
+}

Original: [Equation] √ 2) < 0 and g (1 / √ 2 , 0) =0 .
Revised: and g (1 / √ 2 , 0) =

Original: It implies that ∂∂δ ( e 
Revised: It implies that ∂∂δ ( e − (1 / √

Original: − (1 / √ 2))2 − e −(1 / √ 2+ δ )2 δ
Revised: 2 − e − / √ 2+ δ )

Original:  ) ≤ 0 for δ > 0 .
Revised: 2 δ ) ≤ 0 for δ > 0 .

Original: Since ∂∂y g ( y, δ ) = − 4 δ 2 e − δ ( δ +2 y )
Revised: Since ∂∂y g ( y, δ ) = − 4 δ 2 e − δ ( δ +2 y )

Original: ( δ + y ) < 0 for y ≥ 0 , we conclude that for any y ≥ 1 / √ 2 :
Revised: ( δ + y ) < 0 for y ≥ 0 , we conclude that for any y ≥ 1 / √ 2 :

Original: [Equation]
Revised: [Equation]

Original: A. M AIN T HEOREM
Revised: A. M AIN T HEOREM

Original: Recall that SGLD on dataset S is deﬁned as
Revised: Recall that SGLD on dataset S is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Here γ t is the step size.
Revised: Here γ t is the step size.

Original: B t = { i 1 , . . .
Revised: B t = { i 1 , . . .

Original: , i b } is a subset of { 1 , . . . , n } of size b , and S B t
Revised: , i b } is a subset of { 1 , . . . , n } of size b , and S B t

Original: =
Revised: =

Original: ( z i 1 , . . .
Revised: ( z i 1 , . . .

Original: , z i b ) is the mini-batch indexed by B t .
Revised: , z i b ) is the mini-batch indexed by B t .

Original: Recall that F ( w, S ) denotes 1 | S | (cid:80) | S | i =1 F ( w, z i ) .
Revised: Recall that F ( w, S ) denotes 1 | S | (cid:80) | S | i =1 F ( w, z i ) .

Original: We restate and prove Theorem 11 in the following.
Revised: We restate and prove Theorem 11 in the following.

Original: Theorem 11.
Revised: Theorem 11.

Original: Suppose that the loss function L is C -bounded and the objective function F is Llipschitz.
Revised: Suppose that the loss function L is C -bounded and the objective function F is Llipschitz.

Original: Assume that the following conditions hold:
Revised: Assume that the following conditions hold:

Original: Batch size b ≤ n/ 2 . 2.
Revised: Batch size b ≤ n/ 2 . 2.

Original: Learning rate γ t ≤ σ t / (20 L ) .
Revised: Learning rate γ t ≤ σ t / (20 L ) .

Original: Then, the following expected generalization error bound holds for T iterations of SGLD (1) :
Revised: Then, the following expected generalization error bound holds for T iterations of SGLD [Equation]

Original: [Equation]
Revised: [Equation]

Original: where g e ( t ) =
Revised: where g e ( t ) =

Original: E w ∼ W t − 1 [ 1 n (cid:80) ni
Revised: E w ∼ W t − 1 [ 1 n (cid:80) ni

Original: =1 (cid:107)∇ F ( w, z i ) (cid:107) 22 ] is the empirical squared gradient norm, and W t is the parameter at step t of SGLD.
Revised: =1 (cid:107)∇ F ( w, z i ) (cid:107) 22 ] is the empirical squared gradient norm, and W t is the parameter at step t of SGLD.

Original: Proof By Theorem 7, we have
Revised: Proof By Theorem 7, we have

Original: [Equation]
Revised: [Equation]

Original: for any prior distribution P .
Revised: for any prior distribution P .

Original: In particular, we deﬁne the prior as P ( w ) =
Revised: In particular, we deﬁne the prior as P ( w ) =

Original: E S ∼D n − 1 [ P S ( w )] , where
Revised: E S ∼D n − 1 [ P S ( w )] , where

Original: P S ( w ) =
Revised: P S ( w ) =

Original: Q ( S, 0 ) .
Revised: Q ( S, 0 ) .

Original: By the convexity of KL-divergence,
Revised: By the convexity of KL-divergence,

Original: [Equation]
Revised: [Equation]

Original: Fix a data point z ∈ Z .
Revised: Fix a data point z ∈ Z .

Original: Let ( W t ) t ≥ 0 and ( W (cid:48) t ) t ≥ 0 be the training process of SGLD for S
Revised: Let ( W t ) t ≥ 0 and ( W (cid:48) t ) t ≥ 0 be the training process of SGLD for S

Original: =
Revised: =

Original: ( S, z ) and S (cid:48) =
Revised: ( S, z ) and S (cid:48) =

Original: ( S, 0 ) , respectively.
Revised: ( S, 0 ) , respectively.

Original: Fix a time step t and w <t = ( w 0 , . . .
Revised: Fix a time step t and w <t = ( w 0 , . . .

Original: , w t − 1 ) .
Revised: , w t − 1 ) .

Original: Let P t and P (cid:48)
Revised: Let P t and P (cid:48)

Original: t denote the distribution of W t and W (cid:48) t conditioned on W <t = w <t and W (cid:48) <
Revised: t denote the distribution of W t and W (cid:48) t conditioned on W <t = w <t and W (cid:48) <

Original: t = w <t , respectively.
Revised: t = w <t , respectively.

Original: By the deﬁnition of SGLD, we have P t = 1 | G | (cid:80) B ∈ G p µ B and P (cid:48) t
Revised: By the deﬁnition of SGLD, we have P t = 1 | G | (cid:80) B ∈ G p µ B and P (cid:48) t

Original: = 1 | G | (cid:80) B ∈ G p µ (cid:48) B , where µ B = w t − 1 − γ t ∇ w F ( w t − 1 , S B ) , µ (cid:48)
Revised: = 1 | G | (cid:80) B ∈ G p µ (cid:48) B , where µ B = w t − 1 − γ t ∇ w F ( w t − 1 , S B ) , µ (cid:48)

Original: B = w t − 1 − γ t ∇ w F ( w t − 1 , S (cid:48) B ) , and p µ denotes the Gaussian distribution
Revised: B = w t − 1 − γ t ∇ w F ( w t − 1 , S (cid:48) B ) , and p µ denotes the Gaussian distribution

Original: N ( µ, σ 2 t 2 I d ) .
Revised: N ( µ, σ 2 t 2 I d ) .

Original: We note that:
Revised: We note that:

Original: 1. (cid:107) µ (cid:48)
Revised: 1. (cid:107) µ (cid:48)

Original: B − µ B (cid:107) ≤ γ t (cid:107)∇ F ( w t − 1 ,z ) (cid:107) 2 b for B ∈ G n and µ B = µ (cid:48) B for B ∈ G n .
Revised: B − µ B (cid:107) ≤ γ t (cid:107)∇ F ( w t − 1 ,z ) (cid:107) 2 b for B ∈ G n and µ B = µ (cid:48) B for B ∈ G n .

Original: diam
Revised: diam

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By applying Lemma 21 with β = γ t (cid:107)∇ F ( w t
Revised: By applying Lemma 21 with β = γ t (cid:107)∇ F ( w t

Original: [Equation]
Revised: [Equation]

Original: 1 ,z ) (cid:107) 2 b and
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: and σ = σ

Original: [Equation]
Revised: [Equation]

Original: By Lemma
Revised: By Lemma

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: Together with (13) and (14)
Revised: Together with (13) and (14)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: (concavity of √x )
Revised: (concavity of √

Original:  Since SGLD is order-independent, we can replace ∇ F ( w, z n ) with ∇ F ( w, z i ) for any i ∈ [ n ] in the right-hand side of the above bound.
Revised: [Equation] Since SGLD is order-independent, we can replace ∇ F ( w, z n ) with ∇ F ( w, z i ) for any i ∈ [ n ] in the right-hand side of the above bound.

Original: Our theorem then follows from the concavity of √ x .
Revised: Our theorem then follows from the concavity of √ x .

Original: Furthermore, if we bound KL( P, Q z ) instead of KL( Q z , P ) in the above proof, we obtain the following bound that depends on the population squared gradient norm :
Revised: Furthermore, if we bound KL( P, Q z ) instead of KL( Q z , P ) in the above proof, we obtain the following bound that depends on the population squared gradient norm :

Original: [Equation]
Revised: [Equation]

Original: A. E XTENSION TO G ENERAL N OISES
Revised: A. E XTENSION TO G ENERAL N OISES

Original: We can extend the generalization bounds in previous sections, which require the noise to be Gaussian, to other general noises, namely the family of log-lipschitz noises.
Revised: We can extend the generalization bounds in previous sections, which require the noise to be Gaussian, to other general noises, namely the family of log-lipschitz noises.

Original: Deﬁnition 22 (Log-Lipschitz Noises) .
Revised: Deﬁnition 22 (Log-Lipschitz Noises) .

Original: A probability distribution on R d with density p is L -loglipschitz if and only if (cid:107)∇ ln p ( w ) (cid:107) ≤ L holds for any w ∈ R d .
Revised: A probability distribution on R d with density p is L -loglipschitz if and only if (cid:107)∇ ln p ( w ) (cid:107) ≤ L holds for any w ∈ R d .

Original: A random variable ζ is called an
Revised: A random variable ζ is called an

Original: L -log-lipschitz noise if and only if it is drawn from an L -log-lipschitz distribution.
Revised: L -log-lipschitz noise if and only if it is drawn from an L -log-lipschitz distribution.

Original: The analog of SGLD, noisy momentum method (Deﬁnition 24), and noisy NAG (Deﬁnition 25) can be naturally deﬁned by replacing the Gaussian noise ζ t at eachiteration with an independent
Revised: The analog of SGLD, noisy momentum method (Deﬁnition 24), and noisy NAG (Deﬁnition 25) can be naturally deﬁned by replacing the Gaussian noise ζ t at each

Original:  L -log-lipschitz noise in the deﬁnition.
Revised: iteration with an independent L -log-lipschitz noise in the deﬁnition

Original: The following lemma is an analog of Lemma 21 under L -log-lipschitz noises.
Revised: The following lemma is an analog of Lemma 21 under L -log-lipschitz noises.

Original: Recall that G denotes a collection of mini-batches of size b .
Revised: Recall that G denotes a collection of mini-batches of size b .

Original: Lemma 23 readily implies the analogs of Theorems 11, 26 and 27 under more general noise distributions.
Revised: Lemma 23 readily implies the analogs of Theorems 11, 26 and 27 under more general noise distributions.

Original: Lemma 23.
Revised: Lemma 23.

Original: Suppose that batch size b ≤ n/ 2 and N is an L noise -log-lipschitz distribution on R d .
Revised: Suppose that batch size b ≤ n/ 2 and N is an L noise -log-lipschitz distribution on R d .

Original: { µ B : B ∈ G } and { µ (cid:48) B : B ∈ G } are two collections of points in R d that satisfy the following conditions for some constant β ∈ (cid:104) 0 , 1 L noise (cid:105) :
Revised: { µ B : B ∈ G } and { µ (cid:48) B : B ∈ G } are two collections of points in R d that satisfy the following conditions for some constant β ∈ (cid:104) 0 , 1 L noise (cid:105) :

Original: 1. (cid:107) µ B − µ (cid:48) B (cid:107) ≤ β for B ∈ G n and µ B = µ (cid:48) B for B ∈ G n . 2.
Revised: 1. (cid:107) µ B − µ (cid:48) B (cid:107) ≤ β for B ∈ G n and µ B = µ (cid:48) B for B ∈ G n . 2.

Original: diam( { µ B : B ∈ G } ∪ { µ (cid:48)
Revised: diam( { µ B : B ∈ G } ∪ { µ (cid:48)

Original: B : B ∈ G } ) ≤ 1 .
Revised: B : B ∈ G } ) ≤ 1 .

Original: For µ ∈ R d , let p µ denote the distribution of ζ + µ when ζ is drawn from N .
Revised: For µ ∈ R d , let p µ denote the distribution of ζ + µ when ζ is drawn from N .

Original: Let P = 1 | G | (cid:80) B ∈ G p µ B and P (cid:48)
Revised: Let P = 1 | G | (cid:80) B ∈ G p µ B and P (cid:48)

Original: =
Revised: =

Original: 1 | G | (cid:80) B ∈ G p µ (cid:48)
Revised: 1 | G | (cid:80) B ∈ G p µ (cid:48)

Original: B be mixture distributions over all mini-batches.
Revised: B be mixture distributions over all mini-batches.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: for some constant C 0 that only depends on L noise .
Revised: for some constant C 0 that only depends on L noise .

Original: Proof of Lemma 

Revised: Proof of Lemma

Original: Following the same argument as in the proof of Lemma 21, we have
Revised: Following the same argument as in the proof of Lemma 21, we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Fixed A ∈ G n and B ∈ G n .
Revised: Fixed A ∈ G n and B ∈ G n .

Original: Let p noise denote the density of the noise distribution N . 
Revised: Let p noise denote the density of the noise distribution N . Since

Original: Since (cid:107) µ A − µ B (cid:107) ≤ 1 and p noise is L noise -log-lipschitz, we have
Revised: [Equation] µ A − µ B [Equation] ≤ 1 and p noise is L noise -log-lipschitz, we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, since (cid:107) µ B − µ (cid:48) B (cid:107) ≤ β , we have
Revised: Similarly, since (cid:107) µ B − µ (cid:48) B (cid:107) ≤ β , we have

Original: [Equation]
Revised: [Equation]

Original: Then, it follows from βL noise ≤ 1 that
Revised: Then, it follows from βL noise ≤ 1 that

Original: [Equation]
Revised: [Equation]

Original: Therefore, the integral on the righthand side of (16) can be upper bounded as follows:
Revised: Therefore, the integral on the righthand side of (16) can be upper bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: Plugging the above inequality into (15) and (16) gives
Revised: Plugging the above inequality into (15) and (16) gives

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: KL(P, P
Revised: KL( P

Original: [Equation]
Revised: [Equation]

Revised: P (cid:48)

Revised: [Equation]

Original: ≤ ln
Revised: ≤ ln

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  A. E XTENSION TO O THER G RADIENT -B ASED M ETHODS A.5. S TABILITY B OUND FOR M OMENTUM AND N ESTEROVS A CCELERATED G RADIENT
Revised: k [Equation] [Equation] A. E XTENSION TO O THER G RADIENT -B ASED M ETHODS A.5. S TABILITY B OUND FOR M OMENTUM AND N ESTEROVS A CCELERATED G RADIENT

Original: We adopt the formulation of Classical Momentum and Nesterov’s Accelerated Gradient (NAG) methods in Sutskever et al.
Revised: We adopt the formulation of Classical Momentum and Nesterov’s Accelerated Gradient (NAG) methods in Sutskever et al.

Original: and consider the noisy versions of them.
Revised: and consider the noisy versions of them.

Original: Deﬁnition 24 (Noisy Momentum Method) .
Revised: Deﬁnition 24 (Noisy Momentum Method) .

Original: Noisy Momentum Method on objective function F ( w, z ) and dataset S is deﬁned as
Revised: Noisy Momentum Method on objective function F ( w, z ) and dataset S is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 25 (Noisy Nesterovs Accelerated Gradient) .
Revised: Deﬁnition 25 (Noisy Nesterovs Accelerated Gradient) .

Original: Noisy Nesterovs Accelerated Gradient (NAG) on objective function F ( w, z ) and dataset S is deﬁned as
Revised: Noisy Nesterovs Accelerated Gradient (NAG) on objective function F ( w, z ) and dataset S is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: In both deﬁnitions, γ t is the step size, mini-batch B t is drawn uniformly from G , ζ t is a Gaussian noise drawn from N (0 , σ 2 t 2 I d ) , and η ∈ [0 , 1] is the momentum coefﬁcient.
Revised: In both deﬁnitions, γ t is the step size, mini-batch B t is drawn uniformly from G , ζ t is a Gaussian noise drawn from N (0 , σ 2 t 2 I d ) , and η ∈ [0 , 1] is the momentum coefﬁcient.

Original: Theorem 26.
Revised: Theorem 26.

Original: Under the same assumptions on the loss function, objective function, batch size and learning rate as in Theorem 11, the generalization bounds in Theorem 11 still hold for noisy momentum method and noisy NAG.
Revised: Under the same assumptions on the loss function, objective function, batch size and learning rate as in Theorem 11, the generalization bounds in Theorem 11 still hold for noisy momentum method and noisy NAG.

Original: Proof of Theorem 26 For any time step t and w <t = ( w 0 , w 1 , ..., w t − 1 ) , let P t and P (cid:48)
Revised: Proof of Theorem 26 For any time step t and w <t = ( w 0 , w 1 , ..., w t − 1 ) , let P t and P (cid:48)

Original: t denote the distribution of W t and W (cid:48) t conditioned on W <t = w <t and W (cid:48) <
Revised: t denote the distribution of W t and W (cid:48) t conditioned on W <t = w <t and W (cid:48) <

Original: t = w <t , respectively.
Revised: t = w <t , respectively.

Original: By deﬁnition, we have P t = 1 | G | (cid:80) B ∈ G p µ B and P (cid:48)
Revised: By deﬁnition, we have P t = 1 | G | (cid:80) B ∈ G p µ B and P (cid:48)

Original: [Equation]
Revised: [Equation]

Original: If t = 1 , for both noisy momentum method and noisy NAG, we have
Revised: If t = 1 , for both noisy momentum method and noisy NAG, we have

Original: [Equation]
Revised: [Equation]

Original: For t > 1 , if noisy momentum method is used, we have
Revised: For t > 1 , if noisy momentum method is used, we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, the following holds under noisy NAG:
Revised: Similarly, the following holds under noisy NAG:

Original: [Equation]
Revised: [Equation]

Original: In either case, it can be veriﬁed that the conditions of Lemma 21 hold for β = 2 γ t Lb and σ = σ t .
Revised: In either case, it can be veriﬁed that the conditions of Lemma 21 hold for β = 2 γ t Lb and σ = σ t .

Original: The rest of the proof is the same as the proof of Theorem 11.
Revised: The rest of the proof is the same as the proof of Theorem 11.

Original: A.5. S TABILITY B OUND FOR E NTROPY -SGD
Revised: A.5. S TABILITY B OUND FOR E NTROPY -SGD

Original: In the Entropy-SGD algorithm due to Chaudhari et al.
Revised: In the Entropy-SGD algorithm due to Chaudhari et al.

Original: (2017), instead of directly optimizing the original objective F ( w ) , we minimize the negative local entropy deﬁned as follows:
Revised: (2017), instead of directly optimizing the original objective F ( w ) , we minimize the negative local entropy deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, a wider local minimum has a lower loss (i.e., − E ( w, γ ) ) than sharper local minima.
Revised: Intuitively, a wider local minimum has a lower loss (i.e., − E ( w, γ ) ) than sharper local minima.

Original: See Chaudhari et al.
Revised: See Chaudhari et al.

Original: (2017) for more details.
Revised: (2017) for more details.

Original: The Entropy-SGD algorithm invokes standard SGD to minimize the negative local entropy.
Revised: The Entropy-SGD algorithm invokes standard SGD to minimize the negative local entropy.

Original: However, the gradient of negative local entropy −∇
Revised: However, the gradient of negative local entropy −∇

Original: [Equation]
Revised: [Equation]

Original: is hard to compute.
Revised: is hard to compute.

Original: Thus, the algorithm uses exponential averaging to estimate the gradient in the
Revised: Thus, the algorithm uses exponential averaging to estimate the gradient in the

Original: SGLD loop; see Algorithm 1 for more details.
Revised: SGLD loop; see Algorithm 1 for more details.

Original: We have the following generalization bound for Entropy-SGD.
Revised: We have the following generalization bound for Entropy-SGD.

Original: W t,k +1 ← W t,k
Revised: ← mini batch

Original: [Equation] (cid:48) ∇ w g ( W t,k , S B t,k ) + η (cid:48) γ ( W t − 1 ,K +1 − W t,k ) + √ η ε N
Revised: W t,k +1 ← W t,k − η (cid:48) ∇ w g ( W t,k , S B t,k ) + η (cid:48) γ ( W t − 1 ,K +1 − W t,k ) + √ η (cid:48)

Original: 0 , 12 I d
Revised: ε N (0 , 12 I d

Original: ; µ 
Revised: ; µ t,k +1 ← α µ t,k + αW t k ; W t K +1 ← W t,K − ηγ t

Original: + k t t 
Revised: K − µ t K

Original: Theorem 27.
Revised: Theorem 27.

Original: Suppose that the loss function L is C -bounded and the objective function F is Llipschitz.
Revised: Suppose that the loss function L is C -bounded and the objective function F is Llipschitz.

Original: If batch size b ≤ n/ 2 and √ η (cid:48) ≤ ε/ (20 L ) , the following expected generalization error bound holds for Entropy-SGD:
Revised: If batch size b ≤ n/ 2 and √ η (cid:48) ≤ ε/ (20 L ) , the following expected generalization error bound holds for Entropy-SGD:

Original: [Equation]
Revised: [Equation]

Original: empirical
Revised: empirical

Original: [Equation]
Revised: [Equation]

Original: where g e ( t, k ) =
Revised: where g e ( t, k ) =

Original: E w ∼ W t,k
Revised: E w ∼ W t,k

Original: [ 1 n (cid:80) ni =1 (cid:107)∇ F ( w, z i ) (cid:107) 22 ] is the empirical squared gradient norm, and
Revised: [ 1 n (cid:80) ni =1 (cid:107)∇ F ( w, z i ) (cid:107) 22 ] is the empirical squared gradient norm, and

Original: W t,k denotes the training process with respect to S .
Revised: W t,k denotes the training process with respect to S .

Original: Since g e ( t, k ) is at most L 2 , it further implies the generalization error of Entropy-SGD is bounded by O (cid:16) C √ η (cid:48)
Revised: Since g e ( t, k ) is at most L 2 , it further implies the generalization error of Entropy-SGD is bounded by O (cid:16) C √ η (cid:48)

Original: L εn √ TK (cid:17) .
Revised: L εn √ TK (cid:17) .

Original: Proof of Theorem 27 Deﬁne the history before time step ( t, k ) as follows:
Revised: Proof of Theorem 27 Deﬁne the history before time step ( t, k ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Since µ is only determined by W , we only need to focus on W .
Revised: Since µ is only determined by W , we only need to focus on W .

Original: This proof is similar to the proof of
Revised: This proof is similar to the proof of

Original: Theorem 11.
Revised: Theorem 11.

Original: By setting P = E S [ Q ( S, 0 ) ] .
Revised: By setting P = E S [ Q ( S, 0 ) ] .

Original: Suppose S =
Revised: Suppose S =

Original: ( S, z ) and S (cid:48) =
Revised: ( S, z ) and S (cid:48) =

Original: ( S, 0 ) are ﬁxed, let W and
Revised: ( S, 0 ) are ﬁxed, let W and

Original: W (cid:48) denote their training process, respectively.
Revised: W (cid:48) denote their training process, respectively.

Original: Considering the following 3 cases:
Revised: Considering the following 3 cases:

Original: W t, 0 ← W t − 1 ,K +1 :
Revised: W t, 0 ← W t − 1 ,K +1 :

Original: In this case, for a ﬁxed w ≤ ( t − 1 ,K +1) ,
Revised: In this case, for a ﬁxed w ≤ ( t − 1 ,K +1) ,

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  In
Revised: : In

Original: this case, ﬁx a w ≤ ( t,k ) , applying Lemma 21 gives
Revised: this case, ﬁx a w ≤ ( t,k ) , applying Lemma 21 gives

Original: [Equation]
Revised: [Equation]

Original: W t,K +1 ← W t,K − ηγ ( W t , K − µ t,K ) :
Revised: W t,K +1 ← W t,K − ηγ ( W t , K − µ t,K ) :

Original: In this case, for a ﬁxed w ≤ ( t,K ) , we have
Revised: In this case, for a ﬁxed w ≤ ( t,K ) , we have

Original: [Equation]
Revised: [Equation]

Original: By applying Lemma 10, we have
Revised: By applying Lemma 10, we have

Original: [Equation]
Revised: [Equation]

Original: and Where g e ( t, k ) is the empirical squared gradient norm of the k -th SGLD iteration in the t -th
Revised: and Where g e ( t, k ) is the empirical squared gradient norm of the k -th SGLD iteration in the t -th

Original: SGD iteration, respectively.
Revised: SGD iteration, respectively.

Original: The rest of the proof is the same as the proof of Theorem 11.
Revised: The rest of the proof is the same as the proof of Theorem 11.

Original: B P ROOFS IN S ECTIONB. M ARKOV S EMIGROUP AND L OG -S OBOLEV I NEQUALITY
Revised: B P ROOFS IN S ECTIONB. M ARKOV S EMIGROUP AND L OG -S OBOLEV I NEQUALITY

Original: The continuous version of the noisy gradient descent method is the Langevin dynamics, described by the following stochastic differential equation:
Revised: The continuous version of the noisy gradient descent method is the Langevin dynamics, described by the following stochastic differential equation:

Original: [Equation]
Revised: [Equation]

Original: where B t is the standard Brownian motion.
Revised: where B t is the standard Brownian motion.

Original: To analyze the above Langevin dynamics, we need some preliminary knowledge about Log-Sobolev inequalities.
Revised: To analyze the above Langevin dynamics, we need some preliminary knowledge about Log-Sobolev inequalities.

Original: Let p t ( w, y ) denote the probability density function (i.e., probability kernel) describing the distribution of W t starting from w .
Revised: Let p t ( w, y ) denote the probability density function (i.e., probability kernel) describing the distribution of W t starting from w .

Original: For a given SDE such as (20), we can deﬁne the associated diffusion semigroup P :
Revised: For a given SDE such as (20), we can deﬁne the associated diffusion semigroup P :

Original: Deﬁnition 28 (Diffusion Semigroup) .
Revised: Deﬁnition 28 (Diffusion Semigroup) .

Original: (see e.g., (Bakry et al., 2013, p. 39))
Revised: (see e.g., (Bakry et al., 2013, p. 39))

Original: Given a stochastic differential equation (SDE), the associated diffusion semigroup P =
Revised: Given a stochastic differential equation (SDE), the associated diffusion semigroup P =

Original: ( P t ) t ≥ 0 is a family of operators that satisfy for every t ≥ 0 , P t is a linear operator sending any real-valued bounded measurable function f on R d to
Revised: ( P t ) t ≥ 0 is a family of operators that satisfy for every t ≥ 0 , P t is a linear operator sending any real-valued bounded measurable function f on R d to

Original: [Equation]
Revised: [Equation]

Original: The semigroup property P t + s = P t ◦ P s holds for every t, s ≥ 0 .
Revised: The semigroup property P t + s = P t ◦ P s holds for every t, s ≥ 0 .

Original: Another useful property of P t is that it maps a nonnegative function to a nonnegative function.
Revised: Another useful property of P t is that it maps a nonnegative function to a nonnegative function.

Original: The carr ´ e du champ operator Γ of this diffusion semigroup (w.r.t (20)) is (Bakry et al., 2013, p. 42)
Revised: The carr ´ e du champ operator Γ of this diffusion semigroup (w.r.t (20)) is (Bakry et al., 2013, p. 42)

Original: [Equation]
Revised: [Equation]

Original: ( f [Equation] ) =
Revised: (cid:104)∇

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:104)∇ f, ∇ g (cid:105) .
Revised: (cid:105) .

Original: We use the shorthand notation Γ( f ) =
Revised: We use the shorthand notation Γ( f ) =

Original: Γ( f, f ) =
Revised: Γ( f, f ) =

Original: β − 1 (cid:107)∇ f (cid:107) 22 , and deﬁne (with the convention that 0 log 0 = 
Revised: β − 1 (cid:107)∇ f (cid:107) 22 , and deﬁne (with the convention that 0 log 0 = 0 )

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 29 (Logarithmic Sobolev Inequality) .
Revised: Deﬁnition 29 (Logarithmic Sobolev Inequality) .

Original: (see e.g., (Bakry et al., 2013, p. 237))
Revised: (see e.g., (Bakry et al., 2013, p. 237))

Original: A probability measure µ is said to satisfy a logarithmic Sobolev inequality LS( α ) (with respect to Γ ), if for all functions f : R d → R + in the Dirichlet domain D ( E ) ,
Revised: A probability measure µ is said to satisfy a logarithmic Sobolev inequality LS( α ) (with respect to Γ ), if for all functions f : R d → R + in the Dirichlet domain D ( E ) ,

Original: [Equation]
Revised: [Equation]

Original: D ( E ) is the set of functions f ∈ L 2 ( µ ) for which the quantity 1 t (cid:82)
Revised: D ( E ) is the set of functions f ∈ L 2 ( µ ) for which the quantity 1 t (cid:82)

Original: R d f ( f − P t f )
Revised: R d f ( f − P t f )

Original: d µ has a ﬁnite (decreasing) limit as t decreases to 0.
Revised: d µ has a ﬁnite (decreasing) limit as t decreases to 0.

Original: A well-known Logarithmic Sobolev Inequality is the following result for Gaussian measures.
Revised: A well-known Logarithmic Sobolev Inequality is the following result for Gaussian measures.

Original: Lemma 30 (Logarithmic Sobolev Inequality for Gaussian measure) .
Revised: Lemma 30 (Logarithmic Sobolev Inequality for Gaussian measure) .

Original: (Bakry et al., 2013, p. 258)
Revised: (Bakry et al., 2013, p. 258)

Original: Let µ be the centered Gaussian measure on R d with covariance matrix σ 2 I d .
Revised: Let µ be the centered Gaussian measure on R d with covariance matrix σ 2 I d .

Original: Then µ satisﬁes the following LSI:
Revised: Then µ satisﬁes the following LSI:

Original: [Equation]
Revised: [Equation]

Original: Lemma 30 states that the centered Gaussian measure with covariance matrix σ 2 I d satisﬁes LS( βσ 2 ) (with respect to Γ ), where Γ = β − 1 (cid:104)∇ f, ∇ g (cid:105) is the carr´e du champ operator of the diffusion semigroup deﬁned above.
Revised: Lemma 30 states that the centered Gaussian measure with covariance matrix σ 2 I d satisﬁes LS( βσ 2 ) (with respect to Γ ), where Γ = β − 1 (cid:104)∇ f, ∇ g (cid:105) is the carr´e du champ operator of the diffusion semigroup deﬁned above.

Original: Before proving our results, we need some known results from Markov diffusion process.
Revised: Before proving our results, we need some known results from Markov diffusion process.

Original: It is well known that the invariant measure (Bakry et al., 2013, p. 10) of the above CLD is the Gibbs measure
Revised: It is well known that the invariant measure (Bakry et al., 2013, p. 10) of the above CLD is the Gibbs measure

Original: [Equation]
Revised: [Equation]

Original: ) d w (Menz et al., 2014, (1.3)).
Revised: d w (Menz et al., 2014, (1.3)).

Original: In other words, µ satisﬁes (cid:82) R d P t fdµ = (cid:82) R d fdµ for every bounded positive measurable function f , where P t is the Markov semigroup in Deﬁnition 28.
Revised: In other words, µ satisﬁes (cid:82) R d P t fdµ = (cid:82) R d fdµ for every bounded positive measurable function f , where P t is the Markov semigroup in Deﬁnition 28.

Original: The following lemma by Holley and Stroock Holley & Stroock (1987) (see also (Bakry et al., 2013, p. 240)) allows us to determine the Logarithmic Sobolev constant of the invariant measure µ .
Revised: The following lemma by Holley and Stroock Holley & Stroock (1987) (see also (Bakry et al., 2013, p. 240)) allows us to determine the Logarithmic Sobolev constant of the invariant measure µ .

Original: Lemma 31 (Bounded perturbation) .
Revised: Lemma 31 (Bounded perturbation) .

Original: Assume that the probability measure ν satisﬁes LS( α ) (with respect to Γ ).
Revised: Assume that the probability measure ν satisﬁes LS( α ) (with respect to Γ ).

Original: Let µ be a probability measure such that 1 /b ≤ d µ/ d ν ≤ b for some constant b > 1 .
Revised: Let µ be a probability measure such that 1 /b ≤ d µ/ d ν ≤ b for some constant b > 1 .

Original: Then µ satisﬁes LS( b 2 α ) (with respect to Γ ).
Revised: Then µ satisﬁes LS( b 2 α ) (with respect to Γ ).

Original: In fact, Lemma 31 is a simple consequence of the following variational formula in the special case that φ ( x ) = x log x , which we will also need in our proof:
Revised: In fact, Lemma 31 is a simple consequence of the following variational formula in the special case that φ ( x ) = x log x , which we will also need in our proof:

Original: Lemma 32 (Variational formula) .
Revised: Lemma 32 (Variational formula) .

Original: (see .g.
Revised: (see .g.

Original: , (Bakry et al., 2013, p. 240)) Let φ : I → R on some open interval I ⊂
Revised: , (Bakry et al., 2013, p. 240)) Let φ : I → R on some open interval I ⊂

Original: R be convex of class C 2 .
Revised: R be convex of class C 2 .

Original: For every (bounded or suitably integrable) measurable function f : R d → R with values in I , 
Revised: For every (bounded or suitably integrable) measurable function f : R d → R with values in I , [Equation]

Original: [Equation]
Revised: [Equation]

Original: It is worth noting the integrand of the right-hand side is nonnegative due to the convexity of φ .
Revised: It is worth noting the integrand of the right-hand side is nonnegative due to the convexity of φ .

Original: B.
Revised: B.

Original: L OGARITHMIC S OBOLEV I NEQUALITY FOR CLD
Revised: L OGARITHMIC S OBOLEV I NEQUALITY FOR CLD

Original: Recall that F S ( w ) =
Revised: Recall that F S ( w ) =

Original: F ( w, S ) :
Revised: F ( w, S ) :

Original: = F 0 ( w, S ) + λ (cid:107) w (cid:107) 2 2 / 2 is the sum of the empirical original objective F 0 ( w, S ) and (cid:96) 2 regularization.
Revised: = F 0 ( w, S ) + λ (cid:107) w (cid:107) 2 2 / 2 is the sum of the empirical original objective F 0 ( w, S ) and (cid:96) 2 regularization.

Original: Let d µ = 1 Z µ exp( − βF S ( w ))
Revised: Let d µ = 1 Z µ exp( − βF S ( w ))

Original: d w be the invariant (Gibbs) measure of CLD, and ν is the centered Gaussian measure
Revised: d w be the invariant (Gibbs) measure of CLD, and ν is the centered Gaussian measure

Original: d ν = 1 Z ν exp( − βλ (cid:107) w (cid:107) 22 / 2) d w .
Revised: d ν = 1 Z ν exp( − βλ (cid:107) w (cid:107) 22 / 2) d w .

Original: Invoking Lemma 30 with σ 2 = 1 λβ shows that ν satisﬁes LS( 1 /λ )
Revised: Invoking Lemma 30 with σ 2 = 1 λβ shows that ν satisﬁes LS( 1 /λ )

Original: (with respect to Γ ).
Revised: (with respect to Γ ).

Original: Consider the density h ( w ) =
Revised: Consider the density h ( w ) =

Original: d µ d ν = Z ν Z µ exp( − βF 0 ( w, S )) .
Revised: d µ d ν = Z ν Z µ exp( − βF 0 ( w, S )) .

Original: If the original objective function F 0 is C -bounded, we have exp( − 2 βC ) ≤ h ( w ) ≤ exp(2 βC ) .
Revised: If the original objective function F 0 is C -bounded, we have exp( − 2 βC ) ≤ h ( w ) ≤ exp(2 βC ) .

Original: By applying Lemma 31 with b = exp(2 βC ) , we have the following lemma.
Revised: By applying Lemma 31 with b = exp(2 βC ) , we have the following lemma.

Original: Lemma 33.
Revised: Lemma 33.

Original: Under Assumption 14, let Γ( f, g ) =
Revised: Under Assumption 14, let Γ( f, g ) =

Original: β − 1 (cid:104)∇ f, ∇ g (cid:105) be the carr´e du champ operator of the diffusion semigroup associated to CLD, and µ be the invariant measure of the SDE.
Revised: β − 1 (cid:104)∇ f, ∇ g (cid:105) be the carr´e du champ operator of the diffusion semigroup associated to CLD, and µ be the invariant measure of the SDE.

Original: Then, µ satisﬁes LS( e 4 βC /λ ) with respect to Γ .
Revised: Then, µ satisﬁes LS( e 4 βC /λ ) with respect to Γ .

Original: Let µ t be the probability measure of W t .
Revised: Let µ t be the probability measure of W t .

Original: By deﬁnition of P t , for any real-valued bounded measurable function f on R d and any s, t ≥ 0 ,
Revised: By deﬁnition of P t , for any real-valued bounded measurable function f on R d and any s, t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: In particular, if the invariant measure µ = µ ∞ exists, we have
Revised: In particular, if the invariant measure µ = µ ∞ exists, we have

Original: [Equation]
Revised: [Equation]

Original: The following lemma is crucial for establishing the ﬁrst generalization bound for CLD.
Revised: The following lemma is crucial for establishing the ﬁrst generalization bound for CLD.

Original: In fact, we establish a Log-Sobolev inequality for µ t , the parameter distribution at time t , for any time t > 0 .
Revised: In fact, we establish a Log-Sobolev inequality for µ t , the parameter distribution at time t , for any time t > 0 .

Original: Note that our choice of the initial distribution µ 0 is important for the proof.
Revised: Note that our choice of the initial distribution µ 0 is important for the proof.

Original: Lemma 34.
Revised: Lemma 34.

Original: Under Assumption 14, let µ t be the probability measure of W t in (CLD) with initial probability measure d
Revised: Under Assumption 14, let µ t be the probability measure of W t in (CLD) with initial probability measure d

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]

Original: Z e

Original: [Equation]
Revised: [Equation]

Original: d w .
Revised: d w .

Original: Let Γ be the carr´e du champ operator of diffusion semigroup associated to (CLD) .
Revised: Let Γ be the carr´e du champ operator of diffusion semigroup associated to (CLD) .

Original: Then, for any f : R d → R + in D ( E ) 
Revised: Then, for any f : R d → R + in D ( E ) :

Original: [Equation]
Revised: [Equation]

Original: Proof Let µ be the invariant measure of CLD.
Revised: Proof Let µ be the invariant measure of CLD.

Original: By Lemma 33 and Deﬁnition 29,
Revised: By Lemma 33 and Deﬁnition 29,

Original: [Equation]
Revised: [Equation]

Original: By applying Lemma 32 with φ ( x ) = x log x , we rewrite the left-hand side as
Revised: By applying Lemma 32 with φ ( x ) = x log x , we rewrite the left-hand side as

Original: [Equation]
Revised: [Equation]

Original: where the last equation holds by the deﬁnition of invariant measure (cid:82) P t f d µ = (cid:82)
Revised: where the last equation holds by the deﬁnition of invariant measure (cid:82) P t f d µ = (cid:82)

Original: f d µ .
Revised: f d µ .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: Let µ t be the probability measure of W t .
Revised: Let µ t be the probability measure of W t .

Original: Lemma 32 and (22) together imply that
Revised: Lemma 32 and (22) together imply that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since P t ( φ ( f ) − φ ( r ) − φ (cid:48)
Revised: Since P t ( φ ( f ) − φ ( r ) − φ (cid:48)

Original: ( r )( f − r )) ≥ 0 5 and d µ 0 d µ ≤ exp(2 βC ) , we have
Revised: ( r )( f − r )) ≥ 0 7 and d µ 0 d µ ≤ exp(2 βC ) , we have

Original: [Equation]
Revised: [Equation]

Original: Since d µ d µ 0 ≤ exp(2 βC ) and µ is the invariant measure
Revised: Since d µ d µ 0 ≤ exp(2 βC ) and µ is the invariant measure

Original: [Equation]
Revised: [Equation]

Original: we conclude that
Revised: we conclude that

Original: [Equation]
Revised: [Equation]

Original: Lemma 16.
Revised: Lemma 16.

Original: Under Assumption
Revised: Under Assumption

Original: [Equation]
Revised: [Equation]

Original: let µ t be the probability measure of W t in CLD (with d µ 0 = 1 Z e − λβ (cid:107) w (cid:107) 2d w ).
Revised: let µ t be the probability measure of W t in CLD (with d µ 0 = 1 Z e − λβ (cid:107) w (cid:107) 2d w ).

Original: Let ν be a probability measure that is absolutely continuous with respect to µ t .
Revised: Let ν be a probability measure that is absolutely continuous with respect to µ t .

Original: Suppose d µ t =
Revised: Suppose d µ t =

Original: π t ( w ) d w and d ν = γ ( w ) d w .
Revised: π t ( w ) d w and d ν = γ ( w ) d w .

Original: Then it holds that:
Revised: Then it holds that:

Original: [Equation]
Revised: [Equation]

Original: Proof Let f ( w ) =
Revised: Proof Let f ( w ) =

Original: γ ( w ) /π t ( w ) , by Lemma 34 and (cid:82) R d f d µ t = 1 , we have
Revised: γ ( w ) /π t ( w ) , by Lemma 34 and (cid:82) R d f d µ t = 1 , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: B. T HE D ISCRETIZATION L EMMA FROM R AGINSKY ET AL . (2017)
Revised: B. T HE D ISCRETIZATION L EMMA FROM R AGINSKY ET AL . (2017)

Original: Let h ( w, z ) =
Revised: Let h ( w, z ) =

Original: F 0 ( w, z ) + λ (cid:107) w (cid:107) 22.
Revised: F 0 ( w, z ) + λ (cid:107) w (cid:107) 22.

Original: We can rewrite F S ( w )
Revised: We can rewrite F S ( w )

Original: = 1 n (cid:80) ni =1 h ( w, z i ) .
Revised: = 1 n (cid:80) ni =1 h ( w, z i ) .

Original: Deﬁne µ S,k and ν S,t as the probability measure of W k (in GLD) and W t (in CLD), respectively.
Revised: Deﬁne µ S,k and ν S,t as the probability measure of W k (in GLD) and W t (in CLD), respectively.

Original: Raginsky et al.
Revised: Raginsky et al.

Original: (2017) provided a bound of KL( µ S,k , ν S,ηK ) under Assumption 35.
Revised: (2017) provided a bound of KL( µ S,k , ν S,ηK ) under Assumption 35.

Original: This bound enables us to derive a generalization error bound for the discrete GLD from the bound for the continuous CLD.
Revised: This bound enables us to derive a generalization error bound for the discrete GLD from the bound for the continuous CLD.

Original: We use the assumption from Raginsky et al.
Revised: We use the assumption from Raginsky et al.

Original: Their work considers the following SGLD:
Revised: Their work considers the following SGLD:

Original: [Equation]
Revised: [Equation]

Original: Where g S ( w k ) is a conditionally unbiased estimate of the gradient ∇ F S ( w k ) .
Revised: Where g S ( w k ) is a conditionally unbiased estimate of the gradient ∇ F S ( w k ) .

Original: In our GLD setting, g S ( W k ) is equal to ∇ F S ( w k ) .
Revised: In our GLD setting, g S ( W k ) is equal to ∇ F S ( w k ) .

Original: Assumption 35.
Revised: Assumption 35.

Original: Let F S ( w ) = 1 n (cid:80)
Revised: Let F S ( w ) = 1 n (cid:80)

Original: [Equation]
Revised: [Equation]

Original: The function h takes non-negative real values, and there exist constants A, B ≥ 0 , such that 
Revised: The function h takes non-negative real values, and there exist constants A, B ≥ 0 , such that [Equation] 0 , z )

Original: [Equation]
Revised: [Equation]

Original:  For each z ∈ Z , the function h ( · , z ) is M -smooth: for some M > 0 ,
Revised: ≤ A and [Equation] For each z ∈ Z , the function h ( · , z ) is M -smooth: for some M > 0 ,

Original: [Equation]
Revised: [Equation]

Original: For each z ∈ Z , the function h ( · , z ) is ( m, b ) -dissipative: for some m > 0 and b ≥ 0 ,
Revised: For each z ∈ Z , the function h ( · , z ) is ( m, b ) -dissipative: for some m > 0 and b ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: There exists a constant δ ∈ [0 , 1) , such that, for each S ∈ Z n ,
Revised: There exists a constant δ ∈ [0 , 1) , such that, for each S ∈ Z n ,

Original: [Equation]
Revised: [Equation]

Original: The probability law µ 0 of the initial hypothesis W 0 has a bounded and strictly positive density p 0 with respect to the Lebesgue measure on R d , and
Revised: The probability law µ 0 of the initial hypothesis W 0 has a bounded and strictly positive density p 0 with respect to the Lebesgue measure on R d , and

Original: [Equation]
Revised: [Equation]

Original: Lemma 36.
Revised: Lemma 36.

Original: (Raginsky et al., 2017, Lemma 7) Suppose that Assumption 35 holds and set µ S, 

Revised: (Raginsky et al., 2017, Lemma 7) Suppose that Assumption 35 holds and set µ S,

Original: = ν S, 0 = µ 0 .
Revised: = ν S, 0 = µ 0 .

Original: Then, for any k ∈ N and any η ∈ (0 , 1 ∧ m 4 M 2 ) , the following inequality holds
Revised: Then, for any k ∈ N and any η ∈ (0 , 1 ∧ m 4 M 2 ) , the following inequality holds

Original: [Equation]
Revised: [Equation]

Original: where C 0 and C 1 are constants that only depend on M , κ 0 , m , b , β , B and d .
Revised: where C 0 and C 1 are constants that only depend on M , κ 0 , m , b , β , B and d .

Original: B. P ROOFS FOR M AIN T HEOREMS
Revised: B. P ROOFS FOR M AIN T HEOREMS

Original: Theorem 15.
Revised: Theorem 15.

Original: Under Assumption 14, CLD (with initial probability measure d µ 0 = 1 Z e − λβ (cid:107) w (cid:107)
Revised: Under Assumption 14, CLD (with initial probability measure d µ 0 = 1 Z e − λβ (cid:107) w [Equation] d

Original: [Equation]
Revised: [Equation]

Original: d w ) has the following expected generalization error bound:
Revised: ) has the following expected generalization error bound:

Original: [Equation]
Revised: [Equation]

Original: [Equation] In addition, if F 0 is also M -smooth and non-negative, by setting λβ > 2 , λ > 12 and η ∈ [0 , 1 ∧2 λ− 1 8 M 2 ) , the GLD (running K iterations with the same µ 0 as CLD) has the expected generalization error bound:
Revised: In addition, if F 0 is also M -smooth and non-negative, by setting λβ > 2 , λ > 0 and η ∈ [0 , 1 ∧ λ 8 M 2 ) , the GLD (running K iterations with the same µ 0 as CLD) has the expected generalization error bound:

Original: [Equation]
Revised: [Equation]

Original: where C 1 is a constant that only depends on M , λ , β , b , L and d .
Revised: where C 1 is a constant that only depends on M , λ , β , b , L and d .

Original: Proof of Theorem 15 We apply the uniform stability framework.
Revised: Proof of Theorem 15 We apply the uniform stability framework.

Original: Suppose S and S (cid:48) are two neighboring datasets that differ on exactly one data point.
Revised: Suppose S and S (cid:48) are two neighboring datasets that differ on exactly one data point.

Original: Let ( W t ) t ≥ 0 and ( W (cid:48) t ) t ≥ 0 be the process of
Revised: Let ( W t ) t ≥ 0 and ( W (cid:48) t ) t ≥ 0 be the process of

Original: CLD running on S and S (cid:48) , respectively.
Revised: CLD running on S and S (cid:48) , respectively.

Original: Let γ t and π t be the pdf of W (cid:48) t and W t .
Revised: Let γ t and π t be the pdf of W (cid:48) t and W t .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: According to Fokker-Planck equation (see Risken (1996)) for CLD, we know that
Revised: According to Fokker-Planck equation (see Risken (1996)) for CLD, we know that

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: integration by parts) and
Revised: (integration by parts) and

Original: [Equation]
Revised: [Equation]

Original: (integration by parts)
Revised: (integration by parts)

Original: Together with (33)
Revised: Together with (33), we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: d (cid:18) [Equation] = [Equation]

Revised: β

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The last step holds because (cid:104)
Revised: The last step holds because (cid:104)

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107)∇ F S − ∇ F S (cid:48) (cid:107) 22 ≤ 4 L 2 n 2 , by
Revised: Since (cid:107)∇ F S − ∇ F S (cid:48) (cid:107) 22 ≤ 4 L 2 n 2 , by

Original: Lemma 16, we have
Revised: Lemma 16, we have

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Hence, d
Revised: Hence, d

Original: [Equation]
Revised: [Equation]

Original: with KL
Revised: with KL( γ

Original: [Equation]
Revised: [Equation]

Revised: , π

Revised: [Equation]

Revised: [Equation]

Original: Solving this differential inequality gives
Revised: Solving this differential inequality gives

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By Pinsker’s inequality, we can ﬁnally see that
Revised: By Pinsker’s inequality, we can ﬁnally see that

Original: [Equation]
Revised: [Equation]

Original: By Lemma 4, the generalization error of CLD is bounded by the right-hand side of the above inequality.
Revised: By Lemma 4, the generalization error of CLD is bounded by the right-hand side of the above inequality.

Original: Now, we prove the second part of the theorem.
Revised: Now, we prove the second part of the theorem.

Original: Let ( W k ) k ≥ 0 and ( W (cid:48) k ) k ≥ 0 be the (discrete)
Revised: Let ( W k ) k ≥ 0 and ( W (cid:48) k ) k ≥ 0 be the (discrete)

Original: GLD processes training on S and S (cid:48) , respectively.
Revised: GLD processes training on S and S (cid:48) , respectively.

Original: Then for any z ∈ Z :
Revised: Then for any z ∈ Z :

Original: [Equation]
Revised: [Equation]

Original: Since λβ > 2 and λ > 12 , Assumption 35 holds with A = C , B = L , m =2 λ− 1 2 , b
Revised: Since λβ > 2 and λ > 12 , Assumption 35 holds with A = C , B = L , m = λ 2 , b

Original: = L 2 2  , δ =and κ 0 = d 2 log (cid:16) 1 + 2 λβ − 2 (cid:17) .
Revised: = L 2 2 λ , δ = 0 and κ 0 = d 2 log (cid:16) 1 + 2 λβ − 2 (cid:17) .

Original: By applying Pinskers inequality and Lemma 36, we have
Revised: By applying Pinskers inequality and Lemma 36, we have

Original: [Equation]
Revised: [Equation]

Original: 36) and
Revised: ) and

Original: [Equation]
Revised: [Equation]

Original: From (35), we have
Revised: From (35), we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining (36), (37) and (38), we have
Revised: Combining (36), (37) and (38), we have

Original: [Equation]
Revised: [Equation]

Original: By Deﬁnition 3, GLD is (cid:15) n -uniformly stable.
Revised: By Deﬁnition 3, GLD is (cid:15) n -uniformly stable.

Original: Applying Lemma 4 gives the generalization bound of
Revised: Applying Lemma 4 gives the generalization bound of

Original: GLD.
Revised: GLD.

Original: Lemma 37 (Exponential decay in entropy) .
Revised: Lemma 37 (Exponential decay in entropy) .

Original: (Bakry et al., 2013, Theorem 5.2.1)
Revised: (Bakry et al., 2013, Theorem 5.2.1)

Original: The logarithmic
Revised: The logarithmic

Original: Sobolev inequality LS( α ) for the probability measure µ is equivalent to saying that for every positive function ρ in L 1 ( µ ) (with ﬁnite entropy),
Revised: Sobolev inequality LS( α ) for the probability measure µ is equivalent to saying that for every positive function ρ in L 1 ( µ ) (with ﬁnite entropy),

Original: [Equation]
Revised: [Equation]

Original: for every t ≥ 0 .
Revised: for every t ≥ 0 .

Original: The 
Revised: The following Lemma shows that P t ( d µ 0 d µ ) = µ t in our diffusion process.

Original: Lemma 38.
Revised: Lemma 38.

Original: Let P denote the diffusion semigroup of CLD.
Revised: Let P denote the diffusion semigroup of CLD.

Original: Let µ denote the invariant measure of P and let µ t denote the probability measure of W t .
Revised: Let µ denote the invariant measure of P and let µ t denote the probability measure of W t .

Original: Then P t ( d µ 0 d µ ) = µ t .
Revised: Then P t ( d µ 0 d µ ) = µ t .

Original: Proof Let d µ = µ ( x ) d x and d µ t = µ t ( x ) d x .
Revised: Proof Let d µ = µ ( x ) d x and d µ t = µ t ( x ) d x .

Original: As shown in (Pavliotis, 2014, page 118), our diffusion process (Smoluchowski dynamics) is reversible, which means µ ( x ) p t ( x, y )
Revised: As shown in (Pavliotis, 2014, page 118), our diffusion process (Smoluchowski dynamics) is reversible, which means µ ( x ) p t ( x, y )

Original: =
Revised: =

Original: µ ( y ) p t
Revised: µ ( y ) p t

Original: ( y, x ) .
Revised: ( y, x ) .

Original: Thus for any g ( x ) , we have
Revised: Thus for any g ( x ) , we have

Original: [Equation]
Revised: [Equation]

Original: Since g is arbitrary, P t ( d µ 0 d µ ) and µ t must be the same.
Revised: Since g is arbitrary, P t ( d µ 0 d µ ) and µ t must be the same.

Original: Theorem 39.
Revised: Theorem 39.

Original: Suppose that n > 8 βC .
Revised: Suppose that n > 8 βC .

Original: Under Assumption 14, CLD (with initial distribution d µ 0 =
Revised: Under Assumption 14, CLD (with initial distribution d µ 0 =

Original: [Equation]
Revised: [Equation]

Original: d w ) has the following expected generalization error bound:
Revised: d w ) has the following expected generalization error bound:

Original: [Equation]
Revised: [Equation]

Original: In addition, if F 0 is also M -smooth and non-negative, by setting λβ > 2 , λ > 12 and η ∈ [0 , 1 ∧ 2 λ − 1 8 M 2 ) , the GLD process (running K iterations with the same µ 0 as CLD) has the expected generalization error bound:
Revised: In addition, if F 0 is also M -smooth and non-negative, by setting λβ > 2 , λ > 12 and η ∈ [0 , 1 ∧ 2 λ − 1 8 M 2 ) , the GLD process (running K iterations with the same µ 0 as CLD) has the expected generalization error bound:

Original: [Equation]
Revised: [Equation]

Original: where C 1 is a constant that only depends on M , λ , β , b , L and d .
Revised: where C 1 is a constant that only depends on M , λ , β , b , L and d .

Original: Proof of Theorem 39 Suppose S and S (cid:48) are two datasets that differ on exactly one data point.
Revised: Proof of Theorem 39 Suppose S and S (cid:48) are two datasets that differ on exactly one data point.

Original: Let ( W t ) t ≥ 0 and ( W (cid:48) t ) t ≥ 0 be their processes, respectively.
Revised: Let ( W t ) t ≥ 0 and ( W (cid:48) t ) t ≥ 0 be their processes, respectively.

Original: Let d µ t =
Revised: Let d µ t =

Original: π t ( w ) d w and d µ (cid:48) t
Revised: π t ( w ) d w and d µ (cid:48) t

Original: = π (cid:48) t ( w ) d w be the probability measure of W t and W (cid:48) t , respectively.
Revised: = π (cid:48) t ( w ) d w be the probability measure of W t and W (cid:48) t , respectively.

Original: The invariant measure of CLD for S and S (cid:48) are denoted as µ and µ (cid:48) , respectively.
Revised: The invariant measure of CLD for S and S (cid:48) are denoted as µ and µ (cid:48) , respectively.

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: The total variation distance of µ and µ (cid:48) is
Revised: The total variation distance of µ and µ (cid:48) is

Original: [Equation]
Revised: [Equation]

Original: Since Z µ Z µ (cid:48) exp( − β ( F S (cid:48) ( w ) − F S ( w )))
Revised: Since Z µ Z µ (cid:48) exp( − β ( F S (cid:48) ( w ) − F S ( w )))

Original: ∈ (cid:104)
Revised: ∈ (cid:104)

Original: e − 4 βCn , e 4 βCn (cid:105) and 4 βCn < 1 / 2 , we have
Revised: e − 4 βCn , e 4 βCn (cid:105) and 4 βCn < 1 / 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Since µ and µ (cid:48) satisfy LS ( e 4 βC/λ ) (Lemma 33), applying Lemma 37 with ρ = d µ 0 d µ and ρ (cid:48)
Revised: Since µ and µ (cid:48) satisfy LS ( e 4 βC/λ ) (Lemma 33), applying Lemma 37 with ρ = d µ 0 d µ and ρ (cid:48)

Original: =
Revised: =

Original: d µ (cid:48) 

Revised: d µ (cid:48)

Original: d µ (cid:48) and Lemma 38 yields:
Revised: d µ (cid:48) and Lemma 38 yields:

Original: [Equation]
Revised: [Equation]

Original: Since KL( µ 0 , µ ) and KL( µ (cid:48) 0 , µ (cid:48) ) are upper bounded by 2 βC , Pinsker’s inequality implies that
Revised: Since KL( µ 0 , µ ) and KL( µ (cid:48) 0 , µ (cid:48) ) are upper bounded by 2 βC , Pinsker’s inequality implies that

Original: TV( µ t , µ ) and TV( µ (cid:48) t , µ (cid:48) ) are upper bounded by (cid:113) exp (cid:0) − 2 λt e 4 βC (cid:1) βC .
Revised: TV( µ t , µ ) and TV( µ (cid:48) t , µ (cid:48) ) are upper bounded by (cid:113) exp (cid:0) − 2 λt e 4 βC (cid:1) βC .

Original: Combining with (40) and note that TV( µ t , µ (cid:48) t ) ≤ TV( µ t , µ ) + TV( µ, µ (cid:48) )
Revised: Combining with (40) and note that TV( µ t , µ (cid:48) t ) ≤ TV( µ t , µ ) + TV( µ, µ (cid:48) )

Original: + TV( µ (cid:48) t , µ (cid:48) ) , we have 
Revised: + TV( µ (cid:48) t , µ (cid:48) ) , we have sup z | E A [ L ( W T , z ) − L ( W (cid:48)

Original: [Equation]
Revised: T , z )] | ≤ 2 C · TV( µ t , µ (cid:48) t ) ≤ 4 C (cid:115) exp (cid:18) − 2 λt e 4 βC (cid:19) βC + 8 βCn .

Original: By Lemma 4, the generalization error of CLD is bounded by the right-hand side.
Revised: By Lemma 4, the generalization error of CLD is bounded by the right-hand side.

Original: The proof for GLD proceeds in the same way as the second part of the proof of Theorem 15.
Revised: The proof for GLD proceeds in the same way as the second part of the proof of Theorem 15.

Original: C E XPERIMENT 
Revised: C E XPERIMENT D ETAILS

Original: We ﬁrst present the general setup of our experiments:
Revised: We ﬁrst present the general setup of our experiments:

Original: Dataset: We use MNIST (LeCun et al., 1998) and CIFAR10 (Krizhevsky & Hinton, 2009) in our experiments.
Revised: Dataset: We use MNIST (LeCun et al., 1998) and CIFAR10 (Krizhevsky & Hinton, 2009) in our experiments.

Original: Neural network:
Revised: Neural network:

Original: In our experiments, we test two different neural networks: a smaller version of AlexNet (Krizhevsky et al., 2012) and MLP.
Revised: In our experiments, we test two different neural networks: a smaller version of AlexNet (Krizhevsky et al., 2012) and MLP.

Original: The structures of the networks are similar to what are used in Zhang et al.
Revised: The structures of the networks are similar to what are used in Zhang et al.

Original: (2017a).
Revised: (2017a).

Original: • Small AlexNet: k is the kernel size, d is the depth of a convolution layer, fc( m ) is the fullyconnected layer that has m neurons.
Revised: • Small AlexNet: k is the kernel size, d is the depth of a convolution layer, fc( m ) is the fullyconnected layer that has m neurons.

Original: The ReLU activation are used in the ﬁrst 6 layers.
Revised: The ReLU activation are used in the ﬁrst 6 layers.

Original:       [Equation]
Revised:       conv(k:5,d:64) pool( k:3) conv(k

Original: [Equation]
Revised: [Equation]

Original: conv(k:5,d:64) pool(k:3) conv(k:5,d:192)
Revised: d

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: • MLP:
Revised: • MLP:

Original: The MLP used in our experiment has 3 hidden layers, each having width 512.
Revised: The MLP used in our experiment has 3 hidden layers, each having width 512.

Original: We also use ReLU as the activation function in MLP.
Revised: We also use ReLU as the activation function in MLP.

Original: Objective function: For a data point z =
Revised: Objective function: For a data point z =

Original: ( x, y ) in MNIST, the objective function is
Revised: ( x, y ) in MNIST, the objective function is

Original: [Equation]
Revised: [Equation]

Original:  where softmax( a )[ i ] =
Revised: , where softmax( a )[ i ] =

Original: e a [ i ] (cid:80)j =1 e a
Revised: e a [ i ] (cid:80)j =1 e a

Original: [ j ] , and net W ( x ) is the output of the neural network (10 dimensional vector).
Revised: [ j ] , and net W ( x ) is the output of the neural network (10 dimensional vector).

Original: Note that the objective function F is exactly the cross-entropy loss.
Revised: Note that the objective function F is exactly the cross-entropy loss.

Original: 0/1 loss : The 0-1 loss L 01 is deﬁned as:
Revised: 0/1 loss : The 0-1 loss L 01 is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Random labels: Suppose the dataset contains n datapoint, and the corruption portion is p .
Revised: Random labels: Suppose the dataset contains n datapoint, and the corruption portion is p .

Original: We randomly select n · p data points, and replace their labels with random labels, as in Zhang et al.
Revised: We randomly select n · p data points, and replace their labels with random labels, as in Zhang et al.

Original: (2017a).
Revised: (2017a).

Original: C. E XPERIMENTAL RESULTS FOR GLD
Revised: C. E XPERIMENTAL RESULTS FOR GLD

Original: The result of this experiment (see Figure 1) is discussed in Section 3, Remark 13.
Revised: The result of this experiment (see Figure 1) is discussed in Section 3.1.

Original: Here we present our implementation details.
Revised: Here we present our implementation details.

Original: We repeat our experiment 5 times.
Revised: We repeat our experiment 5 times.

Original: At every individual run, we ﬁrst randomly sample 10000 data points from the complete MNIST training data.
Revised: At every individual run, we ﬁrst randomly sample 10000 data points from the complete MNIST training data.

Original: The initial learning rate γ 0 = 0 .
Revised: The initial learning rate γ 0 = 0 .

Original: It decays 0.
Revised: It decays 0.

Original: after every 60 steps, and it stops decaying when it is lower than 0.0005.
Revised: after every 60 steps, and it stops decaying when it is lower than 0.0005.

Original: During the training, we keep σ t = 0 .
Revised: During the training, we keep σ t = 0 .

Original: 2 √ 2 γ t .
Revised: 2 √ 2 γ t .

Original: Recall that the empirical squared gradient norm g e ( t ) in our bound (Theorem 9) is E W t − 1 [ 1 n (cid:80) ni
Revised: Recall that the empirical squared gradient norm g e ( t ) in our bound (Theorem 9) is E W t − 1 [ 1 n (cid:80) ni

Original: =1 (cid:107)∇ f ( W t − 1 , z ) (cid:107) 2 ] .
Revised: =1 (cid:107)∇ f ( W t − 1 , z ) (cid:107) 2 ] .

Original: Since it is time-consuming to compute the exact g e ( t ) , in our experiment, we use an unbiased estimation instead.
Revised: Since it is time-consuming to compute the exact g e ( t ) , in our experiment, we use an unbiased estimation instead.

Original: At every step, we randomly sample a minibatch B with batch size 200 from the training data, and use 1200 (cid:80) i ∈ B (cid:107)∇ f ( W t − 1 , z i ) (cid:107) 2 as g e ( t ) to compute our bound in Figure 1.
Revised: At every step, we randomly sample a mini-batch B with batch size 200 from the training data, and use 1200 (cid:80) i ∈ B (cid:107)∇ f ( W t − 1 , z i ) (cid:107) 2 as g e ( t ) to compute our bound in Figure 1.

Original: The estimation of g e ( t ) at every step t is shown in Figure 1(d).
Revised: The estimation of g e ( t ) at every step t is shown in Figure 1(d).

Original: Since g e ( t ) is not very stable, in our ﬁgure, we plot its moving average over a window of sizeto make the curve smoother (i.e., g avg ( t )
Revised: Since g e ( t ) is not very stable, in our ﬁgure, we plot its moving average over a window of size 100 to make the curve smoother (i.e., g avg ( t )

Original: = 1100 (cid:80) t
Revised: = 1100 (cid:80) t

Original: +100 τ = t g e ( τ ) ).
Revised: +100 τ = t g e ( τ ) ).

Original: C. E XPERIMENTAL R ESULTS FOR SGLD
Revised: C. E XPERIMENTAL R ESULTS FOR SGLD

Original: In this subsection, we present some experiment results for running SGLD on both MNIST and
Revised: In this subsection, we present some experiment results for running SGLD on both MNIST and

Original: CIFAR10 datasets, to demonstrate that our bound (see Theorem 11), in particular the sum of the empirical squared gradient norms along the training path, can distinguish normal dataset from dataset that contains random labels.
Revised: CIFAR10 datasets, to demonstrate that our bound (see Theorem 11), in particular the sum of the empirical squared gradient norms along the training path, can distinguish normal dataset from dataset that contains random labels.

Revised: As shown in Figure 3, the curves of our bounds look quite similar to the generalization curves.

Revised: Due to the sub-optimal constants in our bound, the bound is currently greater than 1, and hence we omit the numbers on the y-axis.

Original: We note that in our experiments, the learn rate  we choose is larger than thatis required by the  condition of Theorem 11.
Revised: We note that in our experiments presented in Figure 3, the learning rate that we choose is larger than that required by the second condition of Theorem 11.

Original: Due to the (non-optimal) constant in our bound, the bound is currently greater than 1, andhence we ignore the numbers on the y-axis.
Revised: This is because the global Lipschitz constant L is hard to estimate and the model is not able to ﬁt training data under a very large noise. As discussed in Section 3.1, we can relax σ t ≥ (20 L ) γ t to σ t ≥ (2 max i ∈ [ n ] (cid:107)∇ F ( W t − 1 , z i ) (cid:107) ) γ t . By applying gradient clipping trick, we can further relax this condition to

Original: However, again, the curves of our bounds look quite similar to the generalization curves (see Figure 2).
Revised: [Equation] where C L is deﬁned in Section 3.1. In order to show that our observation (“random > normal”) still holds when the step size satisﬁes the requirement of our theory, we run an experiment that using gradient clipping trick with C L = 1 . The model is trained on a small subset of MNIST as ﬁtting the original data set with random labels under such a large Gaussian noise is extremely slow.

Revised: As shown in Figure 4, the experimental results remain unchanged when all the conditions of our bound are met.

Original: This indicates that the sum of squared empirical gradient norms is highly related to the generalization performance, and we believe by further optimizing the constants in our bound, we can achieve a generalization bound that is close to the real generalization error. 
Revised: These experiments indicate that the sum of squared empirical gradient norms is highly related to the generalization performance, and we believe by further optimizing the constants in our bound, it is possible to achieve a generalization bound that is much closer to the real generalization error. [Figure] Learning rate γ t = 0 .

Revised: 95 (cid:98) t/ 60 (cid:99) .

Revised: Note that in the early stage, the testing accuracy is even higher than the training accuracy, thus we plot the absolute value of generalization error.

Revised: As shown in this ﬁgure, even when the training accuracy approaches 90%, our bound is still relatively small.

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: (i)

Original: [Figure]
Revised: [Figure]

Revised: s q u a r e d g r a d i e n t n o r m p=0 p=0.

Original: [Figure]
Revised: [Figure]