A New Family of Generalization Bounds Using Samplewise Evaluated CMI
Anonymous Author(s)
AffiliationAddress
Abstract
We present a new family of information-theoretic generalization bounds, in whichthe training loss and the population loss are compared through a jointly convexfunction.
This function is upper-bounded in terms of the disintegrated, samplewise, evaluated conditional mutual information (CMI), an information measurethat depends on the losses incurred by the selected hypothesis rather than on thehypothesis itself, as is common in probably approximately correct (PAC)-Bayesianresults.
We demonstrate the generality of this framework by recovering and extending previously known information-theoretic bounds.
Furthermore, using theevaluated CMI, we derive a samplewise, average version of Seeger’s PAC-Bayesianbound, where the convex function is the binary KL divergence.
In some scenarios,this novel bound results in a tighter characterization of the population loss of deepneural networks than previous bounds.
Finally, we derive high-probability versionsof some of these average bounds.
We demonstrate the unifying nature of theevaluated CMI bounds by showing that they recover average and high-probabilitygeneralization bounds for multiclass classification with finite Natarajan dimension.
Introduction
Information-theoretic generalization bounds, i.e., generalization bounds that are expressed in termsof information-theoretic metrics such as Kullback-Leibler (KL) divergence, mutual information,and conditional mutual information (CMI), have emerged as useful tools to obtain an accuratecharacterization of the performance of deep neural networks.
To obtain such bounds, one comparesa posterior , that is, the distribution over the hypotheses induced by the learning algorithm, to areference distribution called the prior , an approach first introduced to provide probably approximatelycorrect (PAC) bounds for Bayesian classifiers [1, 2, 3, 4].
The connection between these resultsand classic information-theoretic metrics was clarified in [5, 6], where the generalization gap,averaged with respect to the joint posterior and data distribution, is bounded in terms of the mutualinformation between training data and hypothesis.
These bounds have since been extended in severalways [7, 8, 9, 10, 11, 12, 13, 14, 15].
A major step was taken in [16], which considered a setting where the training set is randomly selectedfrom a larger supersample.
We refer to this setup as the CMI setting, as it leads to generalizationbounds in terms of the CMI between the hypothesis and training set selection, given the supersample.
It turns out that these bounds can be further tightened by observing that the selected hypothesis enters the derivation only through the loss that it induces on the supersample.
Using this observation, [16]also derives bounds in terms of the information encoded in these losses rather than in the hypothesis itself, a quantity they refer to as evaluated CMI (e-CMI).
Due to the data-processing inequality, these bounds are always tighter than the regular CMI bounds.
This observation was recently further exploited by [17], which derived samplewise versions of these bounds.
1 Intriguingly, these boundsare both easier to evaluate than their hypothesis-based counterparts, due to the lower dimensionalityof the random variables involved, and quantitatively tighter for deep neural networks.
In particular,while information measures based on the hypothesis space tend to continually increase as trainingprogresses [11, 18], the bound of [17] remains stable.
The results that have so far been derived for the CMI setting pertain only to the (weighted) differencebetween population loss and training loss, or to its squared or absolute value [8, 9, 14, 16, 17, 18].
In the PAC-Bayesian literature, other types of discrepancy measures have been considered, andshown to result in tighter bounds on the population loss.
For example, [19, 20, 21, 22] consider thebinary KL divergence (i.e., the KL divergence between two Bernoulli distributions) with parametersgiven by the training and population loss, respectively, while [23, 24] consider any jointly convexfunction.
Finally, [25] allows for arbitrary functions.
It should be noted that in all of these results, amoment-generating function that depends on the selected function has to be controlled for the boundto be computable.
Recently, the e-CMI framework has also been shown to be expressive enough to allow one to rederiveknown results in learning theory, e.g., generalization bounds expressed in terms of algorithmicstability, VC dimension, and related complexity measures [17, 26].
Tightening and extending CMIbounds, which is the main objective of this paper, has the potential to further increase the unifyingnature of the e-CMI framework.
Contributions Leveraging a basic inequality involving a generic convex function of two randomvariables (Lemma 1), we establish several novel disintegrated, samplewise, e-CMI bounds on theaverage generalization error.
Specifically, we present i) a square-root bound (Theorem 1) on thegeneralization error, together with a mean-squared error extension, which tightens the bound recentlyreported in [17]; ii) a linear bound (Theorem 3) that tightens the bound given in [18]; iii) a binary
KL bound (Theorem 4), which is a natural extension to the e-CMI setting of a well-known bound inthe PAC-Bayes literature [22].
While the derivation of the first two bounds involves an adaptationof results available in the literature, to obtain the binary KL bound we need a novel concentrationinequality involving independent but not identically distributed random variables (Lemma 2).
Numerical experiments conducted on MNIST and CIFAR10 reveal that the binary KL bound results in atighter characterization of the population loss compared to other two bounds for some deep learningscenarios.
As an additional contribution, we show how to adapt the techniques presented in the paperto obtain high-probability (rather than average) e-CMI bounds (Theorem 7).
Finally, we illustrate theexpressiveness of the e-CMI framework by using our bounds to recover average and high-probabilitygeneralization bounds for multiclass classification with finite Natarajan dimension (Theorem 8).
Preliminaries and Notation
We let D ( P || Q ) denote the KL divergence between the two probability measures P and Q .
Thisquantity is well-defined if P is absolutely continuous with respect to Q , denoted by P ≪ Q .
When P and Q are Bernoulli distributions with parameters p and q respectively, we let D ( P || Q ) =
d ( p || q ) =
p log( p/q )+(1 − p ) log((
− p ) / (1 − q )) , and we refer to d ( p || q ) as the binary KL divergence.
For two random variables X and Y with joint distribution P XY and respective marginals P Xand P Y , we let I ( X ; Y ) =
D ( P XY ||
P X P Y ) denote their mutual information.
Throughout thepaper, we use uppercase letters to denote random variables and lowercase letters to denote theirrealizations.
We denote the joint distribution of X and Y given an instance Z
=
z by P XY | Z = zand the corresponding conditional distribution for the product of marginals by P X | Z
=
z P Y | Z
= z .
Furthermore, we let I z
( X ; Y )
=
D ( P XY | Z
=
z
|| P X
| Z = z P Y | Z = z ) .
This quantity is referred to asthe disintegrated mutual information in [11].
Its expectation is the conditional mutual information I ( X ; Y | Z ) =
E Z (cid:2)
I Z ( X ; Y ) (cid:3) .
CMI Setting Let Z be the sample space and let D denote the data-generating distribution.
Considera supersample (cid:101)
Z ∈
Z n × 2 , where each entry is generated i.i.d.
according to D .
For convenience, we index the columns of (cid:101)
Z starting from 0 and the rows starting from 1 .
Furthermore, we denote the i th rowof (cid:101)
Z as (cid:101)
Z i .
Let S =
( S 1 , . . .
, S n ) denote a membership vector, with entries generated independently according to a Bern(1 / 2) distribution, independently of (cid:101) Z .
Let ¯ S
=
(1 − S 1 , . . .
, 1 − S n ) denote themodulo-2 complement of S .
We refer to S as a membership vector because it is used to divide the supersample into an n -dimensional training vector (cid:101) Z S with entries [ (cid:101) Z S ]
i = (cid:101)
Z i,S i and an n -dimensionaltest vector (cid:101)
Z ¯ S with entries [ (cid:101)
Z ¯ S ]
i = (cid:101)
Z i, ¯ S i .
To be able to handle arbitrary learning settings, we considerlearning algorithms as maps A : Z n ×R → F , where R ∈ R is a random variable (independent of (cid:101) Zand S ) that captures the stochasticity of the algorithm and F is a space, for instance, a parameter spaceor function space.
For a fixed R = r and (cid:101) Z S =
˜
z s ,
A (˜ z s , r ) is a deterministic function of ˜ z s .
The quality of the learning algorithm’s output is evaluated using a bounded loss function ℓ : F × Z → [0 , 1] .
We let U be a vector of size m , the elements of which are sampled without replacement uniformlyat random from (1 , . . . , n ) .
For a given realization U =
u = ( u 1 , . . .
, u m ) , we let ˜ z
u denotethe m × 2 matrix obtained by stacking the vectors ˜
z u i for i = 1 , . . .
, m .
Similarly, ℓ ( A (˜ z s , R ) , ˜ z u )denotes the m × 2 matrix of losses obtained by applying ℓ ( A (˜ z s , R ) , · ) elementwise to ˜ z u .
Wedenote the population loss as L D ( A ,
˜ z s , r ) =
E Z ′
[ ℓ ( A (˜ z s , r ) , Z ′ )] ,
where Z ′ ∼ D , and the training loss as
L ˜ z
s ( A , ˜
z s , r )
= 1 n (cid:80) ni
=1 ℓ ( A (˜ z s , r ) , [˜ z s ] i ) .
In general, for any p -dimensional vectorof data points ˆ z , we let L ˆ z
( A , ˜
z s , r ) = 1 p (cid:80) pi =1 ℓ ( A (˜ z s , r ) , ˆ z i ) .
Since A (˜ z s , r ) does not dependon ˜ z ¯ s , L ˜
z
¯ s ( A , ˜
z s , r ) is a test loss.
Finally, we let L D = E (cid:101)
Z,S,R
[ L D ( A , (cid:101) Z S , R )] denote the averagepopulation loss and ˆ L = E (cid:101)
Z,S,R
[ L (cid:101) Z S
( A , (cid:101) Z S , R )] denote the average training loss.
Average Generalization Bounds
In this section, we present the main results of this paper: a new family of disintegrated samplewisee-CMI bounds for the average population loss.
High-probability versions of these bounds are givenin Section 3.
.1 Main
Lemma
We now present the generic inequality upon which the bounds in this section are based.
This inequality,which is similar in nature to the ones provided in [23] and [25], gives us a generic framework to derivegeneralization bounds that allows for a wide choice of functions for comparing training loss andpopulation loss.
A crucial difference compared to [23] and [25] is that our focus in this section is onaverage rather than PAC-Bayesian generalization bounds, and on bounds based on the disintegrated,samplewise e-CMI rather than traditional KL-based bounds.
Lemma 1.
Let f γ : [0 , 1] 2 → R be a function that is jointly convex in its arguments and isparameterized by γ .
Let X
and Y be two random variables, and let Y ′ be a random variablewith the same marginal distribution as Y such that Y ′
and X are independent.
Assume that thejoint distribution of X, Y is absolutely continuous with respect to the joint distribution of X, Y ′ .
Let g 1 ( X, Y ) and g 2 ( X, Y ) be measurable functions with range [0 , 1] and finite first moments, suchthat, for all γ , E X,Y [ f γ ( g 1 ( X, Y ) , g 2 ( X, Y ))] is finite.
Let
[Equation]
and assume that ξ γ is finite.
Then,
[Equation]
The proof, which is an application of Jensen’s inequality and Donsker-Varadhan’s variational representation of the KL divergence, is given in Appendix A, along with the proofs of all other resultsprovided in this paper.
Note that we are allowed to optimize γ because, in this section, we only consider average bounds.
Incontrast, when we derive high-probability bounds in Section 3, we need to fix γ .
Optimizing γ over aset of candidate values in the high-probability scenario incurs a union bound cost [27, Sec. 1.2.2].
To apply Lemma 1, we need to identify functions f γ ( · , · ) for which the moment generating functionin (1) can be controlled.
A theme throughout this section is that we identify functions f γ ( · , · ) for which there exist concentration results that imply ξ γ ≤ 0 .
This allows us to loosen the bound in (2)by discarding ξ γ .
Throughout this section, we assume that the loss is bounded, so that ℓ ( · , · ) ∈ [0 , 1] .
Note that thereported results can be extended to more general losses by use of scaling.
In particular, assume thatthere is a function K : F → R + such that, for all f ∈ F , sup z ℓ ( f, z ) ≤ K ( f ) (referred to as thehypothesis-dependent range condition in [28]).
Then, all of the results that we present for boundedlosses also hold for the scaled loss ℓ ( f, z ) /K
( f ) .
2.2 Extending ( f )-CMI Bounds to e-CMI
We now apply Lemma 1 to recover the average bounds of [17, 18].
These works derive bounds usingthe information contained either in the hypothesis itself or the resulting predictions, i.e., the CMI orthe f -CMI.
We instead derive bounds in terms of the information captured by the vector of losses,i.e., the e-CMI.
For parametric supervised learning algorithms, we can recover the original boundsvia the data-processing inequality.
This is formalized in the following remark.
Remark 1.
Consider a parametric supervised learning setting, where Z = X × Y and A (˜ z S , R ) =
W ∈ W are the parameters of a function ϕ
W : X → Y .
Let ˜ x denote the m × 2 matrix obtained byprojecting each element of ˜ z onto X , i.e., the matrix of unlabeled examples.
Let ϕ W (˜ x u ) denote thematrix of predictions obtained by elementwise application of ϕ W to ˜ x u .
Then, for any fixed ˜ z and u ,
I ˜ z,u ( ℓ ( W, ˜ z ); S u ) ≤ I ˜ z,u ( ϕ W (˜ x u ); S u ) ≤ I ˜ z,u ( W ; S u ) .
[Equation]
We now use our framework to recover some of the results reported in [17].
Let ∆ =
L ˜ z S ′
i ( A , ˜ z
s , R ) −
L ˜ z ¯ S
i
( A , ˜
z
s , R ) , where S ′ i is an independent copy of S i .
For a fixed ˜ z , symmetry impliesthat E S
i
[∆] = 0 .
Furthermore, since ℓ ( · , · ) ∈ [0 , 1] , we have ∆ ∈ [ − 1 , 1] .
Thus, ∆ is 1 -subGaussian.
By using properties of sub-Gaussian random variables, we can control ξ γ in Lemma 1when choosing f γ ( · , · ) as γ ∆ or γ ∆ 2 .
The resulting bounds are given in the following theorem.
Theorem 1 (Square-root bound and squared bound) .
Consider the CMI setting.
Then,
[Equation]
Furthermore,
[Equation]
In (4), the expectation over (cid:101) Z is taken outside of the square root, and the generalization bound is interms of the disintegrated mutual information.
By Jensen’s inequality, this is tighter than (5).
As shown in Appendix A, one can further generalize (4) to obtain an upper bound of the form
[Equation]
However, since this expression is increasing in the size m of the random subset U [17, Prop. 1], wefocus only on the case m = 1 as this leads to the tightest bound.
The same holds for (5), as wellas for all of the bounds that we present in the remainder of this section, where we also focus onlyon m = 1 .
In contrast, the choice of m = 1 in the squared bound in (6) is suboptimal and actuallyyields a vacuous bound due to the 2 /m term.
It is possible to obtain a bound similar to (4), but where an expectation over R is taken outsidethe square root and the mutual information is also conditioned on R .
We present this bound in thefollowing theorem.
Theorem 2 ( R -conditioned square-root bound) .
Consider the CMI setting.
Then,
[Equation]
A similar bound, but given in terms of mutual information rather than e-CMI, is reported in [11,
Thm. 2.4].
The randomness of the learning algorithm can reduce the mutual information between itsoutput and the selection
variable
S
i .
Specifically, one can show that
[Equation]
which implies that the square-root bound in (5) is tighter than the R -conditioned square-root bound in(9).
However, the ordering between the disintegrated square-root bound in (4) and the disintegrated Rconditioned square-root bound in (8) is unclear.
Next, we generalize the linear bounds of [18], which are samplewise extensions of the bounds in [16].
We consider two scenarios.
For the first one, we assume that γ = ( γ 1 , γ 2 ) are positive constants thatsatisfy a certain constraint, and let f γ (ˆ L, L D )
= γ 1 n ( L D − γ 2 ˆ L ) .
For the second one, we assumethat ˆ L = 0 , the so-called interpolating setting, and let f γ (ˆ L, L D ) = n log(2)
L D .
For both of thesescenarios, it can be shown that ξ γ ≤ 0 , yielding the following.
Theorem 3 (Linear bound and interpolation bound) .
Consider the CMI setting.
Let Γ ⊂ R 2+ denotethe set of parameters γ
=
( γ 1 , γ 2 ) that satisfy γ 1 (1 − γ 2 )
+ ( e γ 1 − 1 − γ 1 )(1 + γ 22 ) ≤ 0 .
Then,
[Equation]
Furthermore, if ˆ L = 0 ,
[Equation]
The interpolation bound in (12) improves on the linear bound in (11) when ˆ L = 0 , as the constraintimplies that γ 21 − 4( e γ 1 − 1)( e γ 1 − 1 − γ 1 ) ≥ 0 .
This means that γ 1 < 0 .
37 , whereas log 2 > 0 .
2.3 Binary KL Bound with Samplewise e-CMI
We now derive bounds in terms of the binary KL divergence between the training and population loss.
To this end, similar to [27, 29], we define
[Equation]
An important property of this function is that
[Equation]
Note that both d γ ( · || · ) and d ( · || · ) are jointly convex in their arguments.
For our next result, we needthe following lemma.
Lemma 2.
For i = 1 , . . . , n , let X i ∼ P X
i , E [ X i ] = µ i , ˆ µ = 1 n (cid:80) ni
=
X i , and ¯ µ = 1 n (cid:80) ni =1 µ i .
Assume that X i ∈ [0 , 1] almost surely and that all X i are independent.
Then, for every fixed γ > 0 ,
[Equation]
This inequality, which to the best of our knowledge has previously been reported only for identicallydistributed random variables, follows by combining [30, Lemma 1], [31, Thm. 3], and [29, Eq.
(17)](which is a generalization of [27, Lemma 1.1.1] from binary to bounded random variables).
Consider a fixed ˜ z .
For each i , E S
′ i (cid:2) L [˜ z S ′ ]
i ( A , ˜ z s , R
[Equation]
(cid:3)
=
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
+
[Equation]
( A (˜ z s , R ) , ˜ z i, 1 )) / 2 ,where S ′ i again is an independent copy of S i .
Thus, we have
[Equation]
This observation allows us to bound the binary KL divergence between the training loss ˆ L and thearithmetic mean of the training and population loss, (ˆ L + L D ) / 2 .
Combining (15) and (2), withappropriate choices for the variables and functions therein, we obtain the following result.
Theorem 4 (Binary KL bound) .
Consider the CMI setting.
Then,
[Equation]
which implies that L D can be bounded as
[Equation]
where
[Equation]
Similar to Theorem 1, we can obtain a disintegrated version of (18) where the expectation over (cid:101)
Z isoutside of the inversion of the binary KL divergence.
We present this result in the following theorem.
Theorem 5 (Disintegrated binary KL bound) .
Consider the CMI setting.
Then,
[Equation]
By using Pinsker’s inequality, which implies that 2( q − p ) 2 ≤ d ( q || p ) , one can weaken (18) to obtain
[Equation]
However, since the average over i is inside the square root, this bound is weaker than (5) by Jensen’sinequality.
To establish Theorem 4, it is crucial that the supremum over γ in Lemma 1 is outside the expectationthat defines ξ γ .
Indeed, sup γ d γ ( q || p ) =
d ( q || p ) , and with the notation from Lemma 2, we have [22,
Thm. 1]
[Equation] [Equation]
Therefore, having the supremum inside of the expectation would unavoidably lead to an additionalterm greater than log( √ n ) /n in the upper bound of (18).
The binary KL bound in (18) is an average, samplewise, CMI analogue of the PAC-Bayesian boundwith a binary KL divergence on the left-hand side, sometimes referred to as Seeger’s bound [4].
However, as noted after Lemma 2, the concentration inequality needed to establish Theorem 4 is arefinement of the result used in [20].
Also, the dependence on ˆ L and L D in the binary KL bound isnonstandard.
As we show in Section 5, the binary KL bound in (18) is sometimes tighter than thesquare-root bound in Theorem 1 and the linear and interpolation bounds in Theorem 3.
The binary KL bounds in (18) and (20) can be tightened by considering affine transformations of thearguments of d ( · || · ) .
However, this does not lead to improvements for the low training loss scenariosthat we focus on in this paper.
Therefore, we relegate these extensions to Appendix B, where wealso present an analogue of Theorem 4 in terms of the samplewise mutual information between thelearning algorithm’s output and the training data, rather than the e-CMI.
While d − 1 ( · , · ) does not admit an analytical expression, d − 1 (0 , · ) is tractable.
This leads to thefollowing simplified form of the disintegrated binary KL bound in (20) when ˆ L = 0 .
Theorem 6 (Disintegrated interpolation binary KL bound) .
Assume that ˆ L = 0 .
Then, (20) becomes
[Equation]
High-Probability Bounds
The techniques presented in Section 2 can be adapted to allow for the derivation of high-probabilitybounds, in particular through the use of exponential inequalities [33, 34, 35].
To demonstrate this, wenow present two such high-probability bounds.
Theorem 7 (High-probability square-root and binary KL bounds) .
Let λ =
ℓ
( A ( (cid:101) Z S , R )
[Equation]
(cid:101) Z ) .
Furthermore, let P λ | (cid:101) ZS denote the conditional distribution of λ given (cid:101) Z and S
[Equation]
and let P λ | (cid:101) Z denote theconditional distribution of λ given (cid:101) Z .
Then, with probability at least 1 − δ over the draw of (cid:101) Z and S ,
[Equation]
δ
[Equation]
Furthermore, also with probability at least 1 − δ over the draw of (cid:101) Z and S ,d
[Equation]
These high-probability bounds involve the empirical test loss of the learner
[Equation]
and not the populationloss.
However, a simple use of the triangle inequality, as detailed in [34, Thm. 3], allows one tomake (24) and (25) explicit in the population loss.
Note that the information measures that appearin the bounds depend on the full training set, rather than individual samples.
While it is possibleto obtain tail bounds with samplewise information measures [36, Lemma 8], such bounds have adetrimental linear dependence on 1 /δ , instead of the benign logarithmic dependence in (24) and (25).
Finally, under a stronger technical assumption of absolute continuity, one can obtain high-probabilitybounds not only with respect to the draw of (cid:101) Z and S , but also the randomness R of the learningalgorithm.
We present this result in Appendix B.
Expressiveness of the e-CMI Framework
We now illustrate the unifying nature of the e-CMI framework by demonstrating its expressiveness.
Specifically, proceeding similarly to [17, 26], we use the e-CMI framework to rederive known boundsin learning theory.
In particular, we consider multiclass classification with finite Natarajan dimension.
In the next theorem, we provide a bound on the e-CMI appearing in the results reported in Section 2,as well as a bound on the data-dependent KL divergence appearing in (24) and (25).
Theorem 8.
Consider a multiclass classification setting, for which Z = X
× Y , where X is theinstance space and Y the label space, and assume that |Y| = N .
Furthermore, assume that thelearning algorithm implements a function f : X → Y where f ∈ F belongs to a class of finite
Natarajan dimension d N [37].
Finally, assume that 2 n > d N + 1 .
Then,
[Equation]
Furthermore, with probability at least 1 − δ under the draw of (cid:101) Z and S ,
[Equation]
To obtain generalization bounds from Theorem 8, we need to upper-bound the information termsthat appear in the bounds of Sections 2 and 3 by using either (26) or (27).
The resulting boundsrecover known minimax optimal generalization guarantees in terms of the Natarajan dimension up toa logarithmic term.
We discuss this in more detail and present explicit examples in Appendix B. log(2) B 0.
B o un d o n L 
Square root Linear
Interpolation Binary KL (a) log(2)/ log(2)
B 1.
̂ L
Binary KL
Linear
Square root
Trivial (b)
Comparing the Bounds
We now perform some comparisons between the bounds obtained in Section 2.
As it turns out,there is no clear ordering between the different bounds in general.
For a first comparison, weconsider the interpolating setting, where the training loss is zero.
We focus on the case wherethe size m of the random subset U equals 1 .
Since the squared bound in (6) is vacuous for thischoice, it is not considered for this comparison.
Moreover, we assume that the learning algorithmis, on average, indifferent to permutations of the training set.
Specifically, we assume that thevalue of I ( ℓ ( A ( (cid:101) Z S , R ) , (cid:101) Z i ); S i | (cid:101) Z ) equals a constant B that does not depend on the index i .
For amore straightforward comparison, we also exclude the disintegrated bounds in (4) and (20) from thecomparison, and postpone their evaluation to Section 6.
Thus, in this section, we compare the square-root bound in (5), the linear bound in (11), the interpolation bound in (12), and the binary KL bound in (18).
In the following proposition, we establish aqualitative ordering between these bounds.
The result follows by straightforward arithmetic.
Proposition 1.
Assume that I ( ℓ ( A ( (cid:101) Z S , R ) , (cid:101) Z i ); S i | (cid:101) Z )
=
B for all i and that ˆ L = 0 .
Let γ 1 , opt bethe largest γ 1 such that γ 21 − 4( e γ 1 − 1)( e γ 1 − 1 − γ 1 ) ≥ 0 , and assume that B < 2 γ 21 , opt .
Then,the upper bounds on L D are, in increasing order, the interpolation bound in (12) , the binary KLbound in (18) , the linear bound in (11) , and the square-root bound in (5) .
If B > 2 γ 21 , opt ≈ 0 . 27 , theordering between the square-root bound in (11) and the linear bound in (5) is inverted.
While Proposition 1 gives the order between the bounds for the interpolating scenario, the quantitativedifference between them is not clear.
A numerical illustration is given in Figure 1a, under the sameassumptions as in Proposition 1.
As established in Proposition 1, the interpolation bound in (12)is superior across the range of values for B , and the binary KL bound in (18) is slightly looser.
However, while the constant log 2 in (12) is sharp, the derivation is valid only for interpolatinglearning algorithms.
In contrast, the other bounds are only marginally affected when we allow for asmall, non-zero training loss.
This added flexibility shows the usefulness of (18) as compared to (12).
To get a more complete picture, we numerically evaluate the bounds for a range of values for ˆ Land B .
This is illustrated in Figure 1b.
While the binary KL bound in (18) is tightest for most smallvalues of B and ˆ L , a region of this space is dominated by the linear bound in (11).
When both Band ˆ L grow larger, the square-root bound in (5) is the tightest.
Thus, the picture that emerges fromthese comparisons is that, when the training loss is zero, it is best to use (12), while we need (18) forthe case of a small, but non-zero, training loss.
However, if the training loss is high, one needs toevaluate all three bounds and take the minimum.
[Figure]
Numerical Results
We now consider three deep learning settings and evaluate the bounds derived in Section 2.
To thebest of our knowledge, the tightest average generalization bounds available in the literature for typicaldeep learning scenarios, such as CNNs trained on MNIST [38] or CIFAR10 [39], are found in [17].
These numerical results are based on [17, Eq. 22], which is a version of (4) where the e-CMI isreplaced by the f-CMI.
In order to perform a direct comparison, we consider the same experimentalsetups.
The code for our experiments is largely based on the code from [17].
3 Additional numericalresults for a setting with randomized labels are given in Appendix C. A detailed description of thetraining methods, network architectures, and experimental setup is given in Appendix D.
Since the previous section indicated that the interpolation bound in (12) and the binary KL boundin (20) are the superior bounds for low-training loss settings, which is common for deep learning, wefocus on these bounds, as well as the square-root bound in (4) as a comparison.
When evaluating thebounds, we use the classification error as the loss function.
We consider two learning algorithms,namely stochastic gradient descent (SGD) with a fixed random seed and stochastic gradient Langevindynamics (SGLD).
The first is a deterministic learning algorithm with high relevance for currentpractice.
For deterministic learning algorithms, generalization bounds based on the informationcaptured by the weights of a neural network are vacuous, but as noted by [17], the informationcontained in the predictions (or the losses they induce) can lead to nonvacuous bounds.
The second isa randomized learning algorithm, which allows for comparison to weight-based information-theoreticgeneralization bounds.
For this setting, we also evaluate the SGLD bound of [11, Eq. 6].
First, we consider a CNN trained with Adam (a variant of SGD) on a binarized version of MNIST,where we only consider the digits 4 and 9 .
This is shown in Figure 2a.
In binary classification there isa one-to-one mapping between predictions and losses, so for this scenario, the information capturedby the vector of losses is the same as that in the vector of predictions.
For this scenario, the binary
KL bound in (20) significantly improves on the square-root bound in (4).
For n = 75 , 250 , and 1000 ,all of the neural networks that we trained achieved zero training error, making the interpolationbound in (12) a valid bound that leads to the tightest characterization of the test error.
However,for n = 4000 , the training error was non-zero for some neural networks, albeit very small.
Next, in Figure 2b, we look at ResNet-50 pretrained on ImageNet and fine-tuned using SGD on
CIFAR10.
In multi-class classification, the map from predictions to losses is no longer one-to-one.
Thus, this is a setting where the e-CMI potentially is lower than the f-CMI.
Again, the binary KLbound in (20) significantly improves on the square-root bound in (4).
Finally, as an example of a randomized learning algorithm, we consider a CNN trained with SGLD onthe binarized MNIST data set.
This is shown in Figure 2c.
As training progresses, both (20) and (4)improve on the SGLD bound in [11, Eq. 6].
However, the latter is lower for the initial epochs.
While(20) reduces the magnitude of this discrepancy, which was also observed by [17], it does not fullyclose the gap.
This may be due to the fact that the bound in [11, Eq. 6] has an expectation over Routside of the square root, similar to (8), which may be beneficial during early epochs.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] We clarify our contributions by specifically referencing
where they can be found in the paper.
(b) Did you describe the limitations of your work?
[Yes] We specify the assumptions andsettings for which our results hold.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] The assump-
tions are stated in Section 1 or in the Theorem statements.
(b) Did you include complete proofs of all theoretical results?
[Yes] Full proofs areprovided in Appendix A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Yes, this isdescribed in the file readme .
txt in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] This is specified in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] All random results include error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] This is specified in the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] The assets that we
use are cited in Section 6 and Appendix D.
(b) Did you mention the license of the assets?
[Yes] The license is discussed in Appendix D.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
The data used are standard data sets that only include images of
objects and animals.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
The data used are standard data sets that onlyinclude images of objects and animals.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]