Generalization Bounds for Gradient Methods via Discrete and Continuous Prior
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Proving algorithm-dependent generalization error bounds for gradient-type optimization methods has attracted signiﬁcant attention recently in learning theory.
However, most existing trajectory-based analyses require either restrictive assumptions on the learning rate (e.g., fast decreasing learning rate), or continuous injectednoise (such as the Gaussian noise in Langevin dynamics).
In this paper, we introduce a new discrete data-dependent prior to the PAC-Bayesian framework, andprove a high probability generalization bound of order O ( 1 n · (cid:80)
Tt =1 ( γ t /ε t ) [-
 ∥-]2 (cid:107) g t (cid:107) 2 )for Floored GD (i.e. a version of gradient descent with precision level ε t ), wheren is the number of training samples, γ t is the learning rate at step t , g t is roughlythe difference of the gradient computed using all samples and that using only priorsamples.
(cid:107) g t (cid:107) is upper bounded by and and typical much smaller than the gradientnorm (cid:107)∇ f ( W t ) (cid:107) .
We remark that our bound holds for nonconvex and nonsmoothscenarios.
Moreover, our theoretical results provide numerically favorable upperbounds of testing errors (e.g., 0 . 037 on MNIST).
Using similar technique, we canalso obtain new generalization bounds for certain variant of SGD.
Furthermore, westudy the generalization bounds for gradient Langevin Dynamics (GLD).
Using thesame framework with a carefully constructed continuous prior, we show a new highprobability generalization bound of order O ( 1 n + L 2 n 2 (cid:80) Tt =1 ( γ t /σ t ) 2 ) for GLD.
Thenew 1 /n 2 rate is due to the concentration of the difference between the gradient oftraining samples and that of the prior.
Introduction
Bounding generalization error of learning algorithms is one of the most central problems in machinelearning theory.
Formally, for a supervised learning problem, the generalization error is deﬁned asthe testing error (or population error) minus the training error (or empirical error).
In particular,we denote R ( w, ( x, y )) :
=
[ h w ( x ) (cid:54) = y ] as the error of a single data point ( x, y ) , where h w ( x )is the output of a model with parameter w ∈ R d .
Suppose S is the set of training data, eachi.i.d.
sampled from the population distribution D , and we use R ( w, S ) :
= 1 | S | (cid:80) z ∈ S R ( w, z ) and
R ( w, D ) :=
E z ∼ D
[ R ( w, z )] to denote the training error and the testing error, respectively.
Thegeneralization error of w is formally deﬁned as err gen ( w ) =
R ( w, D ) − R ( w, S ) .
Proving tighter generalization bounds for general nonconvex learning and in particular deep learninghas attracted signiﬁcant attention recently.
While the classical learning theory (uniform convergence theory) which bounds the generalization error by various complexity measures (e.g., the
VC-dimension and Rademacher complexity) of the hypothesis class has been successful in severalclassical convex learning models, however, they become vacuous and hence fail to explain thesuccess of modern nonconvex over-parametrized neural networks (i.e., the number of parameters the
Kolter [2019]).
Recently, learning theorists have tried to understand and explain the generalizationability of deep learning from several other perspectives, such as margin theory [Bartlett et al., 2017,
Wei et al., 2019], algorithmic stability [Hardt et al., 2016, Mou et al., 2018, Bousquet et al., 2020],
PAC-bayeisan [London, 2017, Bartlett et al., 2017, Neyshabur et al., 2018, Zhou et al., 2019, Yanget al., 2019], neural tangent kernel [Arora et al., 2019, Cao and Gu, 2019],
information theory [Pensiaet al., 2018, Negrea et al., 2019], model compression [Arora et al., 2018], differential privacy [Onetoet al., 2017] and so on.
In this paper, we aim to obtain tighter generalization error bounds that depend on both the trainingdata and the optimization algorithms (a.k.a.
gradient-type methods) for general nonconvex learningproblems.
In particular, we prove algorithm-dependent generalization bounds for close variantsof the most commonly used algorithms such as gradient descent (GD), stochastic gradient descent(SGD) and stochastic gradient Langevin dynamics (SGLD).
Our proof is based on the classic Catoni’s
PAC-Bayesian framework [Catoni, 2007] and also has a ﬂavor of algorithmic stability Bousquet and Elisseeff [2002].
Several prior works have obtained generalization bounds for SGD and SGLD byanalyzing trajectory through either the PAC-Bayesian or the algorithmic stability framework Mouet al. [2018], Li et al.
[2020] (or closely related information theoretic arguments). However, mostexisting results which are based on analysing the optimization trajectories require either restrictiveassumptions on the learning rate (e.g., in Hardt et al.
[2016], the learning rate η t scales with 1 /t ), orcontinuous noise (such as the Gaussian noise in Langevin dynamics) in order to bound the stability orthe KL-divergence.
In this paper, we resolve the above restrictions by combining the PAC-Bayesianframework with a few simple (yet effective) ideas, so that we can obtain new high probability andnon-vacuous generalization bounds for several gradient-based optimization methods with eitherdiscrete or continuous noise (in particular certain variants of GD and SGD, either being deterministicor with discrete noise, which cannot be handled by existing techniques).
1.1 Prior work
We ﬁrst brieﬂy mention some recent work on bounding the generalization error of gradient-basedmethods.
Hardt et al.
ﬁrst studied the uniform stability (hence the generalization) of stochasticgradient descent (SGD) for both convex and non-convex functions.
Their results for non-convexfunctions requires that the learning rate η t scales with 1 /t .
Their work motivates a long line ofsubsequent work on generalization error bounds of gradient-based optimization methods: Kuzborskijand Lampert [2018], London [2016], Chaudhari et al.
[2019], Raginsky et al.
[2017], Mou et al.[2018], Chen et al.
PAC-Bayesian bounds.
The PAC-Bayesian framework [McAllester, 1999] is a powerful methodfor proving high probability generalization bound [Bartlett et al., 2017, Zhou et al., 2019, Mou et al.
Roughly speaking, it bounds the generalization error by the KL divergence KL (cid:0) Q (cid:12)(cid:12)(cid:12)(cid:12) P (cid:1) , where Q is the distribution of learned output and P is a prior distribution which is typically independentof dataset S .
In this framework, bounding KL (cid:0) Q (cid:12)(cid:12)(cid:12)(cid:12) P (cid:1) is the most crucial part for obtaining tighter PAC-Bayesian bounds.
In order to bound the KL divergence, both the prior P and posterior Q arerequired to be continuous distributions (typically Gaussians so that KL can be computed in closedform).
Hence, most prior work either consider gradient methods with continuous noise (such asgradient Langevin Dynamics)
[Mou et al., 2018], or add a Gaussian noise to the ﬁnal parameter at theend [Neyshabur et al., 2018, Zhou et al., 2019] (so Q is a Gaussian distribution).
We also note thatdesigning effective prior P can be also very important.
For example, Lever et al.
[2013] proposedto use the population distribution to compute the prior.
In fact, the prior can even partially dependon the training data [Parrado-Hernández et al., 2012, Negrea et al., 2019], and our Theorem 4.1 ispartially inspired by this idea.
1.2 Our contributions
First, we provide high probability generalization bounds for discrete gradient methods.
In particular,we study the generalization of Floored Gradient Descent (FGD), which is a variant of GD, and Floored
Gradient Descent (FSGD), a variant of SGD.
We obtain our bound by an interesting constructionof discrete priors.
Secondly, we consider well studied gradient methods with continuous noise,(stochastic) gradient Langvin dynamics (GLD and SGLD).
We show sharper generalization boundsby carefully bounding the concentration of the sample gradients.
Now, we summarize our results.
FGD and FSGD.
We ﬁrst study a close variant of GD called FGD (Algorithm 1).
The update ruleof FGD is deﬁned as follows:
[Equation]
where S J is a subset of training dataset S with size m , J is its index-set chosen before the training,γ t is the learning rate, ε t is the precision level, and g t := ∇ f ( W t − 1 , S ) − ∇ f ( W t − 1 , S J ) is thegradient difference.
The ﬂooring operation on a real number is deﬁned by ﬂoor( x ) :
= sign( x ) (cid:98)| x |(cid:99) .
It can viewed as gradient descent with given precision limits ε t .
We can see if we ignore the ﬂooroperatio or let ε t be very small, FGD reduces to GD.
Empirically, its behavior and optimizationcapability is very close to GD (See Figure 2a and 2b).
By constructing a discrete data-dependent prior and incorporate it into Catoni’s PAC-Bayesianframework, we prove that the following bound (Theorem 5.2) holds with high probability:
[Equation]
where d is the dimension of parameter space.
Now we make a few remarks about our results.1.
Our result holds for nonconvex and nonsmooth learning problems (replacing the gradientswith subgradients for nonsmooth cases).
There is no additional requirement on the learningrate γ t .2.
The gradient difference g t is typical much smaller than the worst case gradient norm.
Itusually decreases when m = | J | grows (see Figure 1c).
Thus, it enables us to obtainnon-vacuous bound on classical datasets.
Speciﬁcally, our theoretical test error upper boundon MNIST and CIFAR10 are 0 . 0369 and 0 . 8589 , respectively.3.
The prior P is a carefully constructed discrete random processes, so that we can bound the
KL between P and the deterministic process of FGD.
We hope it may inspire future researchon tackling deterministic optimization algorithms or discrete noise.4.
The result can be extended to a variant of SGD, called FSGD (see Theorem 5.3 for details).
GLD, SGLD and CLD.
We also provide new a generalization bound for Gradient Langevin
Dynamics (GLD) whose update rule is deﬁned below.
[Equation]
In this paper, we show that the following generalization bound (Theorem 6.2) holds with highprobability over the randomness of S ∼ D n and random subset J :
[Equation]
where L ( W t − 1 ) :
= max z ∈ S (cid:107) f
( W t − 1 , z ) (cid:107) is the longest gradient norm of any training samplein S at step t , m is the size of J .
Since W T is independent of the prior index J , the ﬁrst term R ( W T , S [ n ] \ J ) is upper bounded by R ( W T , S )
+ O ( 1 √ n − m ) with high probability.
By settingm = n/ 2 , our generalization bound has an O ( 1 √ n + 1 n + Tn 2 ) rate.
We also prove a high probabilitygeneralization bound for SGLD (see Theorem 6.3):
[Equation]
2 t σ 2 t L ( W t − 1 )(cid:33) .
We compare our bounds with existing GLD/SGLD generalization bounds in prior work. For GLD, Mou et al.
[2018] provide a generalization bound in expectation based on the uniform stabilityframework, which is of rate O
( L √ Tn ) , where L is the global Lipschitz constant (ignoring the factorsdepending on γ t and σ t ).
Their bound can be tighten to O ( 1 n (cid:113)(cid:80)
Tt = 1 L 2 t ) where L t is the gradientnorm at time t (which is always less than L ) [Li et al., 2020]. Their bounds can be converted to highprobability bound with an additional factor O (1 / √ n ) using the technique developed in [Feldmanand Vondrak, 2019].
Bounds of similar orders have been also obtained through information theory[Wang et al., 2021, Haghifam et al., 2020] and differential privacy [Wu et al., 2021].
Note that themain term in our bound is of order O
( 1 n 2 (cid:80) Tt = 1 L 2 t ) , which is quadratically better than theirs if thebound is in (0 , 1) . For SGLD, Mou et al. [2018] and Li et al.
[2020] obtained similar bounds, butrequires the assumption that the learning rate should be of order
O (1 /L ) .
Our bound for SGLD does not require such assumption and is more favorable for large minibatch size b .
For small value of b(say b = O (1) , our bound can be worse.
Another closely related work is Negrea et al.
[2019], which also use a data-dependent prior.
They present an in-expectation bound based on information theory.
They introduce a quantitycalled “incoherence” ξ t that is deﬁned somewhat similar to (cid:107) g t (cid:107) (it is also the norm of the different between two gradients deﬁned by two different subsets of samples).
They obtained an
O ( (cid:113) 1 n − m (cid:80) Tt =1 γ 2 t σ 2 t E [ (cid:107) ξ t (cid:107) 2 ]) bound for SGLD.
By taking expectation, it can be further boundedby O ( (cid:113) 1 n (cid:80) Tt
=1 ( 1 b + ( n − m ) nm )
γ 2 t σ 2 t V t ) , where V t is a quantity about the same size as the variance of training gradients.
However, directly plugging their worst case O ( ( n − m ) 2 n(cid:80)
[Equation]
σ
[Equation]
)bound for KL (cid:0) Q (cid:12)(cid:12)(cid:12)(cid:12) P ( S J ) (cid:1) into Catoni’s PAC-Bayesian bound, one can only obtain an O ( 1 n − m +( n − m ) n(cid:80)
Tt =1 γ 2 t σ 2 t L 2 ) high probability bound.
To make the 2nd term in their bound have the same
O (1 /n 2 ) rate as ours, one needs to set n − m =
O (1) which would result in a large ﬁrst term1 n − m = Ω(1) .
Moreover, that our construction of data-dependent prior P ( S J ) is also very differentfrom theirs.
Their idea is using the gradients in S J to cancel out the gradients in S while ours isbased on the property that the mean gradient on S J is concentrated around the mean gradient of thewhole dataset S .
Mou et al.
[2018] also obtain a high probability PAC-Bayesian bound of rate O ( (cid:113) 1 n (cid:80)
Tt =1 e − r t L 2 t )if there is an (cid:96) 2 -regularization in the loss (ignoring other factors depending on γ t and σ t ).
Heree − r t < 1 is a decay factor depending on the regularization coefﬁcient.
There is a similar decay factorin Wang et al.
[2021]’s bound (the fact comes from strong data processing inequalities).
Note thatwithout (cid:96) 2 -regularization, there is no such decay factor, and Mou et al.
[2018]’s bound becomes
O ( (cid:112) T/n ) , which is looser than ours.
From technical perspective, they use Fokker Planck equationto track the time derivative of KL and Logarithmic Sobolev inequality to related KL with Fisherinformation.
We also use these tools for our generalization bound of Continuous Langevin dynamics(CLD) (see Appendix E), but the general proof idea is very different.
Other Related Work
Stochastic Langevin Dynamics Stochastic Langevin dynamics is popular sampling and optimization method used in machine learning Welling and Teh [2011].
Zhang et al.
[2017b], Chen et al.[2020] show a polynomial hitting time (hitting a stationary point) of SGLD in general non-convexsetting.
Raginsky et al.
[2017] study the generalization and excess risk of SGLD in nonconvexsettings and their bound depends polynomially on a certain spectral gap parameter which may beexponential in the dimension.
Continuous Langevin dynamics (SDE) with various noise structure hasalso been used extensively as approximations of SGD (see e.g., [Li et al., 2017, 2021]).
Nonvacuous PAC-Bayesian Generalization Bounds.
Dziugaite and Roy [2017] ﬁrst present anon-vacuous PAC-Bayesian generalization bound on MNIST (0.20 for a 3 layer MLP).
They usea somewhat different training algorithm that explicitly optimizes the PAC-Bayesian bound and theoutput distribution is a multivariate normal distribution N ( w, diag( s )) .
To facilitate computing theclosed form of KL, they chose a zero-mean Gaussian distribution as the prior.
Zhou et al.
[2018]obtain non-vacuous bound for ImageNet via a quite different approach.
Their method does not requireany continuous noise injected but assumes that the network can be signiﬁcantly compressed (so thatthe prior distribution is supported over the set of discrete parameters with ﬁnite precisions).
To ourbest knowledge, it is the only work that use a discrete prior for proving generalization bounds of deepneural networks.
Our result for Floored gradient descent has a similar ﬂavor in a high level, that isthe optimization method has a ﬁnite precision.
However, our results do not need any assumption oncompressibility of the model and can be applied to other nonconvex learning problems.
Generalization bound via Information theory.
Raginsky et al.
[2017] ﬁrst shows that the expectedgeneralization error E S ∼D n [ R ( W, D ) −R ( W, S )] is bounded by (cid:112)
2 I ( S ; W ) /n , where I ( S ; W ) :=
KL (cid:0) P ( S, W ) (cid:12)(cid:12)(cid:12)(cid:12) P ( S ) ⊗ P ( W ) (cid:1) is the mutual information.
This work motivates many subsequentstudies [Pensia et al., 2018, Negrea et al., 2019, Bu et al., 2020, Wang et al., 2021].
The main goal inthis line of work is to obtain a tight bound on the mutual information I ( S ; W ) .
This is again reduced to bound the KL divergence and thus typically requires continuous injected noise Wang et al.
Negrea et al.
Preliminaries
Notations.
We assume the training dataset S =
( z 1 , .., z n ) is sampled from D n , where D isthe population distribution.
The model parameter w is in R d .
The risk function R : R d × Ω →[0 , 1] measures the error of a model on a datapoint.
The loss function f : R d × Ω → R is usedfor training models.
The empirical risk is R ( w, S )
= 1 | S | (cid:80) z ∈ S R ( w, z ) and population risk is R ( w, D ) =
E z
∼D
[ R ( w, z )] .
Similarly, we can deﬁne the empirical loss f ( w, S ) and populationloss f ( w, D ) .
For any J =
( j 1 , .., j m ) , we use S J to denote the sequence ( S j 1 , ..., S j m ) .
Thesubsequence ( A i , A i +1 , ..., A j ) is denoted by A ji .
We use ( A n 1 , B m 1 ) to denote the merged sequence( A 1 , A 2 , ..., A n , B 1 , ..., B m ) .
When the elements in sequence J are different from each other, wealso use itself to represent the set consisting of all of its elements.
We also use a random variable itselfto denote its distribution.
For example, E x ∼ X [ f ( x )] is equivalent to E x ∼ P X [ f ( x )] , and KL (cid:0) X (
cid:12)(cid:12)(cid:12)(cid:12) Y (cid:1)means KL (cid:0) P X (cid:12)(cid:12)(cid:12)(cid:12)
P Y (cid:1) .
For a random variable W , we deﬁne R ( W, S ) =
E w ∼ W
[ R ( w, S )] and R ( W, D ) =
E w ∼ W
[ R ( w, Q )] .
The index set { 1 , 2 , ..., n } is denoted by [ n ] .
KL-divergence.
Let P and Q be two probability distributions.
The Kullback–Leibler divergence
KL (cid:0) P (cid:12)(cid:12)(cid:12)(cid:12) Q (cid:1) from P to Q is deﬁned only when P is absolute continuous with respect to Q (i.e.for any x , Q ( x ) = 0 implies P ( x ) = 0 ).
In particular, if P and Q are discrete distributions, then
KL (cid:0) P (cid:12)(cid:12)(cid:12)(cid:12) Q (cid:1)
=
(cid:80) x P ( x ) ln P ( x ) Q ( x ) .
Otherwise, if P and Q are continuous distributions, it is deﬁnedas (cid:82) P ( x ) ln P ( x ) Q ( x ) d x .
The following Lemma 3.1 is frequently used in this paper and is a well knownproperty of KL divergence (see Cover [1999, Theorem 2.5.3], Li et al.
[2020], Negrea et al.
Lemma 3.1 (Chain Rule of KL) .
We are given two random sequences W =
( W 0 , ..., W T ) and
W (cid:48) =
( W (cid:48) 0 , ..., W (cid:48) T ) .
Then, the following equation holds (given all KLs are well deﬁned):
[Equation]
w
[Equation]
t − {+
+}
[Equation]
= w (cid:17)(cid:105) .
Here W t | W t − 1= w denotes the distribution of W t conditioning on W t − 1= ( W 0 , . .
, W t − 1 )
= w .
PAC-Bayesian.
In this paper, we use the PAC-Bayesian bound presented in Catoni [2007] whichenjoys a tighter O ( KL/n ) rate compared to the traditional O ( (cid:112) KL/n ) bound, but with a slightlylarger constant factor on the empirical error.
We restate their bound as follows.
Lemma 3.2 (Catoni’s Bound) .
(see e.g., Lever et al.
For any prior distribution P independentof S , any δ ∈ (0 , 1) , and any η > 0 , we have the following bound holds w.p. ≥ 1 − δ over S ∼ D n :
[Equation]
where C η = 1 1 − e − η is an absolute constant.
Concentration inequality.
We will use the following generalized McDiramid inequality to provethe concentration of cumulative gradient difference in Section 6. Deﬁnition 3.3 (Order-Independent) .
A function Φ is said to be order independent if and only ifΦ( x 1 , x 2 , ..., x m ) = Φ( x π 1 , x π 2 , ..., x π m ) holds for any input X = ( x 1 , x 2 , ..., x m
) ∈ Ω m and anypermutation π ∈ S m . To simplify notation, in the following we use Φ( j i 1 , J mi +1 ) to denote Φ( j 1 , j 2 , ..., j i , J i +1 , ..., J m ) .
Lemma 3.4.
Suppose Φ : [ n ] m → R + is order-independent and | Φ( J ) − Φ( J (cid:48) )
| ≤ c holds for anyadjacent J, J (cid:48) satisfying | set( J ) (cid:84) set( J (cid:48) )
| = m − 1 .
Let J be m indices sampled uniformly from[ n ] without replacement.
Then Pr J [Φ( J ) − E J
[Φ( J )] > (cid:15) ] ≤ exp( − 2 (cid:15) 2 mc 2 ) .
Data-Dependent PAC-Bayesian Bound
The dominating term in traditional PAC-Bayesian Bound is KL (cid:0) Q (cid:12)(cid:12)(cid:12)(cid:12) P (cid:1) /n , where P is a prior distribution independent of the training dataset S .
Intuitively, one can hardly predict the output distribution Q S without knowing any information from S . Thus the best possible bound for KL (cid:0) Q (cid:12)(cid:12)(cid:12)(cid:12) P (cid:1) isΘ(1) w.r.t. n .
However, if we are allowed to see m data points from S when constructing our prior,we may produce better prediction on posterior Q S .
The following Theorem 4.1 enables us to usedata-dependent prior in PAC-Bayesian bound.
Theorem 4.1 (Data-Dependent PAC-Bayesian) .
Suppose J is a random sequence including mindices uniformly sampled from [ n ] without replacement.
For any δ ∈ (0 , 1) and η > 0 , we have w.p.≥ 1 − δ over S ∼ D n and J :
[Equation]
where I =
[ n ] \ J are indices not in J , P ( S J ) is the prior distribution only depending on theinformation of S J , and C η := 1 1 − e − η is a constant.
Remarks.
Note that the above bound holds for arbitrary Q .
Usually, we apply this bound to theposterior Q S deﬁned by our learning algorithm such as GD or GLD. We remark that for most of ourlearning algorithms are independent of J , that is, changing J does not change the output Q S . In thiscase, by standard Chernoff-Hoeffding inequality, the ﬁrst term R ( Q, S I ) in the RHS can be boundedby R ( Q, S )
+ O (1 / √ n − m ) with high probability over the randomness of J .
However, we pointout a subtle point that FGD (Algorithm 1) studied in this paper depends on J .
It may be the case thatby knowing J , the algorithm extracts more information from S J but not much from S I , unintendedlymaking R ( Q, S I ) an validation error, rather than training error as it should be.
However, for thealgorithm FGD we study, it does not really discriminate samples from J and from I . From ourexperiment (Figure 2a and 2d) on CIFAR10, we can see that the S I error R ( W T , S I ) is close tothe training error R ( W T , S ) and both are signiﬁcantly smaller than the testing error R ( W T , D ) .
So R ( Q, S I ) can be considered as a genuine training error in our study of FGD.
Finally, note that GLD,
SGLD and CLD we study are the standard versions and independent of J , hence R ( Q, S I ) can bereplaced by R ( Q, S )
+ O (1 / √ n − m ) .
Floored Gradient Descent
In this section, we study the generalization of a variant of gradient descent: Floored Gradient Descent(FGD).
First we need to deﬁne the “ﬂooring” operation which will be used in our algorithm.
Deﬁnition 5.1 (Flooring) .
For any vector X ∈ R d , let Y = ﬂoor( X ) deﬁned as:
Y i = ﬂoor( X i ) =
(cid:98) X i (cid:99) if X i ≥ 0 , = −(cid:98)− X i (cid:99) if X i < 0 , for all i ∈ [ d ] .
The Floored Gradient Descent algorithm is deﬁned in Algorithm 1, where ( γ t ) t ≥ 0 and ( ε t ) t ≥ {+
 arethe+} step size and precision sequences, respectively.
Input: Training dataset S =
( z 1 , .., z n ) .
Index set J .
Momentum coefﬁcient α .
Result:
Parameter W T ∈ R d . One can turn off the momentum by setting α to 0 .
We remark that the FGD is a deterministicalgorithm.
The following Theorem 5.2 gives the generalization error bound for FGD.
Theorem 5.2.
Suppose J is a random sequence consisting of m indices uniformly sampled from[ n ] without replacement.
Then for any δ ∈ (0 , 1) , the FGD (Algorithm 1) satisﬁes the followinggeneralization bound w.p. at least 1 − δ over S ∼ D n and J :
[Equation]
where d is the dimension of parameter space, I =
[ n ] \ J are indices that are not in J , C η := 1 1 − e − ηis a constant, and g t :=
∇ f ( W t − 1 , S ) − ∇ f ( W t − 1 , S J ) is the gradient difference.
Proof.
We’ll use Theorem 4.1 to prove our theorem.
The key is to construct the prior distribution
P ( S J ) such that KL (cid:0) W T (cid:12)(cid:12)(cid:12)(cid:12) P ( S J ) (cid:1) is tractable.
We deﬁne P ( S J ) as the distribution of W (cid:48) T obtainedby the following update rule ( W (cid:48) 0 :
= w 0 ):
[Equation]
where ξ t is a discrete random variable such that for all ( a 1 , .., a d ) ∈ Z d :
[Equation]
It is easy to verify that the sum of the probability equals to 1 .
Recall that W t
= ( W 0 , ..., W t ) is theparameter sequence of FGD (Algorithm 1).
Applying the chain rule of KL-divergence, we have:
[Equation]
The last equation is because FGD is deterministic.
Let w =
W
t − 1, the distribution of W t | W t − 1= wis a point mass on
[Equation]
t · ﬂoor
[Equation]
γ
[Equation]
, S
[Equation]
f
[Equation]
w t − 1 , S J )) ε t ) . Let a = ﬂoor( γ t ε t ( ∇ f ( w t − 1 , S ) − ∇ f ( w t − 1 , S J ))) .
By deﬁnition of W (cid:48) t , we have
[Equation]
Since | i | ≤ i 2 and p ∈ (0 , 1 / 3) , we have ln (cid:16) ( (cid:80) ∞ i =
−∞ p i 2 ) d (cid:17) is at most d ln (cid:0) 1 + 2 (cid:80) ∞
i =1 p i (cid:1) .
Itcan be further bounded by d ln (1 + 3 p ) .
Moreover, it can be bounded by 3 dp as ln(1 + x ) ≤ x .
Thus, the above KL-divergence can be bounded by 3 dp + (cid:80) dk
=1 ln(1 /p )
a 2 k . Recall that a k :=(cid:98) γ t ε t · ( ∇ k f ( w t − 1 , S ) − ∇ k f ( w t − 1 , S J )) (cid:99) which is less than or equal to γ t ε t · ( ∇ k f ( w t − 1 , S ) −∇ k f ( w t − 1 , S J )) .
Therefore we have
[Equation]
Plugging the above inequality into (1), we have
[Equation]
We conclude our proof by plugging it into Theorem 4.1 (setting p = 1 / ( Td ) ).
[Figure] [Figure] [Equation] FSGD We can use a similar approach to prove a generalization bound for FSGD (a variant of SGD).
It is identical to Algorithm 1 except for the deﬁnitions of g 1 and g 2 replaced with:
[Equation]
where B t ⊆ [ n ] is a random batch index set independent of S, J and W t − 1.
The following Theorem 5.3 gives the generalization bound for FSGD.
Theorem 5.3.
Suppose J is a random sequence consisting of m indices uniformly sampled from [ n ]without replacement.
Then for any δ ∈ (0 , 1) , ε ∈ (0 , 1) , FSGD satisﬁes the following generalizationbound: w.p. at least 1 − δ over S ∼ D n and J :
[Equation]
where d is the dimension of parameter space, I =
[ n ] \ J includes indices out of J , C η := 1 1 − e − η is aconstant, and g t :
= f ( W t − 1 , S B t ) − ∇ f ( W t − 1 , S J ∩ B t ) is the gradient difference.
5.1 Experiment
We run experiments for FGD and FSGD on MNIST [LeCun et {+al., 1998] and CIFAR
+}
[Krizhevskyet al., 2009] to investigate the performance of our bound as well as the properties of the algorithms.
Non-vacuous bound.
For MNIST, we train a CNN ( d = 1 . 4 · 10 6 ) by FGD with γ t = ε t = 0 .
01and momentum α = 0 . 9 ).
The size m = |
J | is set to n/ 2 = 30000 .
As shown in Figure 1a and 1b,our bound (Theorem 5.2 with η = 1 , δ = 0 . 1 ) tracks the testing error closely.
At step T = 990 , ourbound is 0 .
while the testing error is 0 . 0101 .
For CIFAR10, we train a SimpleNet [Hasanpouret al., 2016] without BatchNorm and Dropout.
The number of parameters d is nearly 18 · 10 6 .
Weuse FSGD to train our model.
The learning rate γ t is set to 0 .
001 , the precision ε t is set to 0 .
004 , andthe momentum α is set to 0 .
The batch size is 2000 .
m =
| J | is set to n/ 5 = 10000 .
The result isshown in Figure 2c.
We stop training at step t = 5000 when the training
Decline of gradient difference.
Intuitively, the cumulative squared norm of gradient differenceg t := ∇ f ( W t , S ) −∇ f ( W t , S J ) should decrease when increasing the size of J .
Although we cannotprove a concentration like Lemma 6.1 that (cid:107) g t (cid:107) 2 is O (1 /m ) since W t of FGD depends on J , we stillobserve a decreasing phenomena when increasing m .
The result is depicted in Figure 1c.
Gradient Langevin Dynamics
In this section, we present a O ( 1 n + Tn 2 ) generalization bound for the gradient Langevin Dynamics(GLD).
The GLD learning algorithm can be viewed as gradient descent plus a gaussian noise.
Formally, for a given S ∼ D n , the update rule of GLD is deﬁned as follows:
[Equation]
Here the gradient ∇ f
( W t , z i ) can be replaced with any gradient-like vector such as a clipped gradient.
The output of GLD is the last step parameter W T or any function of the whole training trajectory
W T 0 .
We still use the data-dependent PAC-Bayesian framework (Theorem 4.1) to prove the generalizationbound for GLD.
Unlike FGD (Algorithm 1), GLD doesn’t depend on the prior indices J , whichenables us to prove the following concentration bound (Lemma 6.1) for the gradient difference.
Theproof is based on Lemma 3.4.
Lemma 6.1.
Let S =
( z 1 , ...
z n ) be any ﬁxed training set.
J is a random sequence including mindices uniformly sampled from [ n ] without replacement, and W =
( W 0 , ..., W T ) be any randomsequence independent of J .
Then the following bound holds with probability at least 1 − δ over therandomness of J :
[Equation]
, where C δ = 4 + 2 ln(1 /δ )
+ 5 .
(cid:112) ln(1 /δ ) , and L ( w )
= max i ∈ [ n ] (cid:107)∇ f ( w, z i ) (cid:107) .
Now we are ready to present our main result.
Theorem 6.2.
Suppose J is a random sequence consisting for m indices uniformly sampled from [ n ]without replacement.
Let W T be the output of GLD.
Then for any δ ∈ (0 , 12 ) and η > 0 , we have w.p.≥ 1 − 2 δ over S ∼ D n and J , the following holds:
[Equation]
where L ( w ) := max z ∈ S (cid:107) f ( w, z ) (cid:107) , C δ = 4 + 2 ln(1 /δ )
+ 5 .
(cid:112) ln(1 /δ ) , and C η = 1 1 − e − η .
Stochastic Gradient Langevin Dynamics For a given training data set S , the the update rule of SGLD is deﬁned as:
[Equation]
where B t is a mini-batch with batch size b . Each batch is i.i.d. sampled from uniform( S ) b .
Note thatthe B t is a sequence instead of a set, thus it may include duplicate elements.
The output of SGLD isthe whole training sequence W
=
( W 0 , ..., W T ) .
Similar to the analysis of GLD, we can prove thefollowing bounds for SGLD.
Theorem 6.3.
Let Q S be the distribution of SGLD’s output when the training set is S .
For anyδ ∈ (0 , 1) and m ≥ 1 , we have w.p. ≥ 1 − 2 δ over S ∼ D n and J ∼ Uniform ([ n ]) m , the followingholds:
[Equation]
where L ( w ) :=
max z ∈ S (cid:107) f ( w, z ) (cid:107) , C δ = 4 + 2 ln(1 /δ )
+ 5 .
(cid:112) ln(1 /δ ) , b is the batch size, and
I =
[ n ] \ J .
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the conclusion section.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
[Table]
mental results (either in the supplemental material or as a URL)?
[Yes]
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- [Table] ments multiple times)?
[Yes]
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] It can be found in our supplemental [Table] material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
[Table]
able information or offensive content?
[N/A]
We only use the standard MNIST andCIFAR10 datasets.
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]