PAC-B AYES AND I NFORMATION C OMPLEXITY
Pradeep Kr Banerjee
A
BSTRACT
We point out that a number of well-known PAC-Bayesian-style and informationtheoretic generalization bounds for randomized learning algorithms can be derived under a common framework starting from a fundamental information exponential inequality.
We also obtain new bounds for data-dependent priors and unbounded loss functions.
Optimizing these bounds naturally gives rise to a method called
Information Complexity Minimization for which we discuss two practical examples for learning with neural networks, namely Entropy- and PAC-Bayes- SGD
I NTRODUCTION
A fundamental observation in statistical learning theory is that information compression and learning are intrinsically related in the sense that both entail identifying statistical regularities and patterns in the data.
This relation has been formalized over the years in various ways, e.g., via sample compression schemes (Littlestone & Warmuth, 1986; Moran & Yehudayoff, 2016), Occam’s razor (Blumer et al., 1987) and minimum description length arguments (Li et al., 2003; Blum & Langford, 2003), and more recently via different notions of information stability (Russo & Zou, 2016; Xu & Raginsky, 2017; Jiao et al., 2017; Bassily et al., 2018; Bu et al., 2019; Steinke & Zakynthinou, 2020a;b).
Information stability quantiﬁes the sensitivity of a learning algorithm to local perturbations of its input, i.e., the training data, and draws on a rich tradition of earlier work on algorithmic (Bousquet & Elisseeff, 2002; Shalev-Shwartz et al., 2010; Bassily et al., 2016) and distributional (Dwork et al., 2015; Rogers et al., 2016; Feldman & Steinke, 2018) stability in adaptive data analysis.
On the other hand, an important approach to data-dependent generalization bounds is PAC-Bayes, originally due to McAllester (1999a;b; 2013).
While PAC-Bayes and information stability have evolved independently of each other, a principal objective of this note is to present them under a uniﬁed framework.
We show that the information exponential inequality (IEI) due to Zhang (2006a) gives a general recipe for constructing bounds of both ﬂavors.
Besides recovering several important bounds such as the mutual information-bound due to Xu & Raginsky (2017) and the classical PAC-Bayesian bounds due to Catoni (2007) and McAllester (2013), we also obtain new bounds for data-dependent priors and unbounded loss functions.
Optimizing these bounds gives rise to variants of the Gibbs algorithm, for which we discuss two examples for learning with neural networks, namely, Entropy-SGD (Chaudhari et al., 2017) and PAC-Bayes-SGD (Dziugaite & Roy, 2017).
We also show a PAC-Bayes bound motivated by an Occam’s factor argument, which can be interpreted in relation to ﬂat minima (Hochreiter & Schmidhuber, 1997).
Three key ideas guide our discussion, namely, (1) the lesser the information revealed by an algorithm about its input, the better the generalization, (2) data-dependent priors entail tighter generalization bounds, and (3) optimizing such bounds is a natural recipe for designing new learning algorithms.
T HE I NFORMATION E XPONENTIAL I NEQUALITY AND A PPLICATIONS
We consider the standard apparatus of statistical learning theory (Shalev-Shwartz & Ben-David, 2014).
We have an example domain Z = X
× Y of the instances and labels, a hypothesis space W , a ﬁxed loss function (cid:96) :
W ×Z → [0 , ∞ ) , and a training sample S , which is an n -tuple ( Z 1 , . . .
, Z n ) of i.i.d.
random elements of Z drawn according to some unknown distribution µ .
A learning algorithm is a
Markov kernel P W | S that maps input training samples S to conditional distributions of hypotheses W in W .
For given n , this deﬁnes a joint distribution P SW =
P S P W | S , P S = µ ⊗ n , and a corresponding marginal distribution P W .
The true risk of a hypothesis w ∈ W on µ is L µ ( w ) :
= E µ [ (cid:96) ( w, Z )] , and its empirical risk on the training sample S is L S ( w ) :
= 1 n (cid:80) ni =1 (cid:96) ( w, Z i ) .
Our goal is to control the generalization error , g( W, S ) :
= L µ ( W ) − L S ( W ) , either in expectation or with high probability.
For controlling the generalization error in expectation, we can rewrite the true risk of a given hypothesis w as L µ ( w ) =
E S (cid:48) ∼ µ ⊗ n [ L S (cid:48) ( w )] , where S (cid:48) =
( Z (cid:48) 1 , . . .
, Z (cid:48) n ) is an i.i.d.
sample.
Then the expected generalization error can be written as a difference of two expectations of the same loss function, E SW [g( W, S )]
=
E P S ⊗ P W [ L S ( W )] − E P SW [ L S ( W )] , where the second expectation is taken w.r.t.
the joint distribution of the training sample and the output hypothesis, while the ﬁrst expectation is taken w.r.t.
the product of the two marginal distributions.
Hence the expected generalization error reﬂects the dependence of the output W on the input S .
This dependence can also be measured by their mutual information as has been shown in recent works (Xu & Raginsky, 2017; Jiao et al., 2017; Bassily et al., 2018).
We refer to such bounds as mutual information-based generalization bounds.
Alternatively, we may wish to control the generalization error of the learning algorithm P W | S with high probability over the training sample S .
The expected generalization error over hypotheses chosen from the distribution P ( posterior ) output by the learning algorithm, i.e., E P [g( W, S )] , can be upper-bounded with high probability under P S by the KL divergence between P and an arbitrary reference distribution Q ( prior ), that is selected before the draw of the training sample S .
For any Q , these bounds hold uniformly for all P , and are called PAC-Bayesian bounds (McAllester, 1999a;b; 2013; Maurer, 2004; Zhang, 2006a; Catoni, 2007;
Alquier et al., 2016;
Germain et al., 2009; 2016;
London, 2017; Thiemann et al., 2017; Gr ¨ unwald & Mehta, 2020; Rivasplata et al., 2020), where PAC stands for Probably Approximately Correct .
Bounds of this type are useful when we have a ﬁxed dataset s ∈ Z n and a new hypothesis is sampled from P every time the algorithm is used.
Choosing the posterior to minimize a PAC-Bayesian bound leads to the well-known Gibbs algorithm (Zhang, 2006a; Xu & Raginsky, 2017; Alquier et al., 2016; Kuzborskij et al., 2019).
On the other hand, for a ﬁxed posterior P , E S [ D ( P (cid:107) Q )] is minimized by the oracle prior , Q (cid:63) =
E S
[ P W | S ( ·| S )] .
Note E
S [ D ( P (cid:107) Q (cid:63) )] is just the mutual information
I ( S ; W ) , which is the key quantity controlling the expected generalization error in (Xu & Raginsky, 2017; Jiao et al., 2017; Bassily et al., 2018).
For any β > 0 , we deﬁne the annealed expectation , M β ( w ) =
− β − 1 ln E µ [ e − β(cid:96)
( w,Z ) ] , which acts as a surrogate for L µ ( w ) .
We write M ( W ) to denote the family of probability measures over a set W .
The following fundamental inequality is due to Tong Zhang:
Lemma 1 (Information exponential inequality (IEI), Zhang 2006a) .
For any prior Q ∈ M ( W ) , any real-valued loss function (cid:96) on W × Z , and any posterior distribution P (cid:28) Q over W that depends on an i.i.d.
training sample S , we have E S exp (cid:8)
nβ E P [ M β ( W ) − L S ( W )] − D ( P (cid:107) Q ) (cid:9) ≤ 1 .
The IEI implies bounds both in probability and in expectation for the quantity nβ E P [ M β ( W ) − L S ( W )] − D ( P (cid:107) Q ) , and is the key tool for showing the following result:
Theorem 2.
Let µ be a distribution over Z , and let S be an i.i.d.
training sample from µ .
Let
Q ∈ M ( W ) be a prior distribution that does not depend on S , and let (cid:96) be a real-valued loss function on W × Z .
Suppose that there exist a convex function ψ : R ≥
→ R satisfying ψ (0) = ψ (cid:48) (0) = 0 , such that sup w ∈W [ L µ ( w ) − M β ( w )] ≤ ψ ( β ) β , ∀ β > 0 .
Then, for any β > 0 , and δ ∈ (0 , 1] , with probability of at least 1 − δ over the choice of S ∼ µ ⊗ n , for all distributions P (cid:28) Q over W (even such that depend on S ), we have
[Equation]
Moreover, we have the following bound in expectation:
[Equation]
where ψ ∗− 1 is the inverse of the Fenchel-Legendre dual of ψ .
The proof of Theorem 2 is given in Appendix A.2.
Under the oracle prior Q (cid:63) =
E S
[ P W | S ] , E
S [ D ( P (cid:107) Q (cid:63) )]
= I ( S ; W ) .
If (cid:96) ( w, Z ) is σ -sub-Gaussian under µ for all w ∈ W , then we can take ψ ( β ) =
β 2 σ 2 / 2 and ψ ∗− 1 ( y ) =
(cid:112) 2 σ 2 y (Boucheron et al., 2013, §2.3), in which case we recover the bound in expectation due to Xu & Raginsky (2017):
Corollary 3.
If (cid:96) ( w, Z ) is σ -sub-Gaussian under µ for all w ∈ W , then E SW [g( W, S )] ≤ (cid:112) 2 σ 2 I ( S ; W ) /n .
Corollary 3 shows that an algorithm that reveals a small amount of information about its input generalizes well .
This observation, for instance, forms the basis for the Gibbs algorithm, which can be thought of as “stabilizing” the empirical risk minimization (ERM) algorithm by controlling the input-output mutual information
I ( S ; W ) (Xu & Raginsky, 2017).
We discuss extensions of this idea in Section 3.
In Appendix A.3, we highlight a functional characterization of the mutual information in relation to the “single-draw” bound due to Bassily et al.
In Appendix A.4, we show how several well-known PAC-Bayesian inequalities can be derived starting from the IEI.
These include, for instance, the following classical bound due to Catoni:
Corollary 4 (Catoni 2007, Theorem 1.2.6) .
For any { 0 , 1 } -valued loss (cid:96) , any distribution µ , prior
Q ∈ M ( W ) , any real β > 0 , and any δ ∈ (0 , 1] , with probability of at least 1 − δ over S ∼ µ ⊗ n , we have for all P (cid:28) Q over W :
[Equation]
Other bounds include, for example, the “Linear PAC-Bayes bound” due to McAllester (2013, Theorem 2) and the “PAC-Bayes-KL inequality” (Seeger, 2002; Maurer, 2004) for [0 , 1] -valued loss functions, as well as the bounds for sub-Gaussian and sub-gamma loss functions due to Germain et al.
Differentially private data-dependent priors.
A PAC-Bayesian bound such as (1) stipulates that the prior Q be chosen before the draw of the training sample S .
Q may, however, depend on the data generating distribution µ (Lever et al., 2013).
To have a good control over the KL term in (1), it is desirable that Q be “aligned” with the data-dependent posterior P .
One way to achieve this goal is to choose Q based on S in a differentially private fashion so that Q is stable to local perturbations in S (Dziugaite & Roy, 2018b).
We can then treat Q “as if” it is independent of S .
The next result formalizes this argument and gives a PAC-Bayesian-style bound that is valid for data-dependent priors and unbounded loss functions.
Proposition 5.
Let K ( S , W ) denote the set of Markov kernels from S to W .
Let µ be a distribution over Z , and let S be an i.i.d.
training sample from µ .
Let Q 0 ∈ K ( S , W ) be an ( (cid:15), 0)
-differentially private algorithm and let (cid:96) be a real-valued loss function on W × Z .
Let β > 0 , and let δ ∈ (0 , 1] .
Then with probability of at least 1 − δ over the choice of S ∼ µ ⊗ n , for all P ∈ M ( W ) ,
[Equation]
The proof of Proposition 5 is given in Appendix A.5.
Proposition 5 is valid for any loss function and is similar in spirit to the traditional PAC-Bayesian bounds in (Dziugaite & Roy, 2018b, Theorem 4.2), and (Rivasplata et al., 2020, Eq. 7), which, however, either apply only when the loss is bounded in [0 , 1] , or entails approximating an exponential moment involving the true risk.
I NFORMATION COMPLEXITY MINIMIZATION
Given a prior distribution, choosing a posterior to minimize a PAC-Bayesian bound naturally gives rise to a method called Information Complexity Minimization (ICM) (Zhang, 2006a;b).
Concretely, for a given prior Q and hypothesis set G ⊆ M ( W ) , deﬁne the Optimal Information Complexity ( OIC ) at a given β > 0 as
[Equation]
When G = M ( W ) , we recover the Gibbs algorithm, in which case the OIC evaluates to the (extended) stochastic complexity , − 1 β ln E Q [ e −
βL S
( W ) ]
(Rissanen, 1989; Yamanishi, 1998).
The latter in turn coincides with the log-Bayesian marginal likelihood for β = 1 and the log loss (Zhang, 2006a;b;
Barron & Cover, 1991).
We brieﬂy discuss two examples of ICM for learning with neural networks.
PAC-Bayes-SGD.
PAC-Bayes-SGD is an approach to computing nonvacuous generalization bounds for overparameterized neural network classiﬁers trained with stochastic gradient descent (SGD) (Dziugaite & Roy, 2017; Langford & Caruana, 2002; Hinton & van Camp, 1993).
These bounds are obtained by retraining the network using an objective derived from a PAC-Bayes bound, starting from the solution found by SGD (or in fact any other procedure) for the training loss L S ( w ) w.r.t.
w .
We show how Catoni’s bound in Corollary 4 can be used to derive a PAC-Bayes-SGD objective.
Consider a binary classiﬁcation setting with examples domain Z
= X ×{ 0 , 1 } and loss (cid:96) : R k ×Z → { 0 , 1 } .
Each w ∈ W corresponds to a classiﬁer f w : X → { 0 , 1 } that can be interpreted as a deterministic neural network with parameters in R k .
For trainable parameters w P ∈ R k , γ ∈ R k + , λ ∈ R + , let G be the set of all Gaussian posteriors of the form P = N ( w P , diag( γ )) and let
Q = N ( w 0 , λI k ) be a prior centered at a non-trainable random initialization, w 0 ∈ R k .
We can use a convex surrogate of the 0-1 loss, and the reparameterization trick w = w P
+ ν (cid:12) √ γ , ν ∼ N (0 , I k ) (Kingma & Welling, 2014; Blundell et al., 2015) to compute an unbiased estimate of the gradient of the PAC-Bayes bound in Corollary 4 w.r.t.
the parameters w P , γ, λ and β .
Computing the expectation E P [ L S ( f W )] is difﬁcult in practice.
Instead, we use a Monte Carlo estimate ˆ L S ( f W ) =
1 m (cid:80) mi =1 L S ( f W i ) , where W i i.i.d.
∼ P .
Then, for any δ, δ (cid:48) ∈ (0 , 1) , ﬁxed α > 1 , c ∈ (0 , 1) , b ∈ N , and m, n ∈ N , with probability of at least 1 − δ − δ (cid:48) over a draw of S ∼ µ ⊗ n and W ∼ ( P ) ⊗ m , we have
E
[Equation]
where R = 2 αnβ ln (cid:104)
[Equation]
βn
[Equation]
+ αnβ ln
[Equation]
ln cλ
[Equation]
2 (cid:105) + (cid:113) 12 m ln 2 δ (cid:48) accounts for the cost of optimizing the parameters β, λ , and using the Monte Carlo estimate of the empirical risk.
For large n, m , R is negligible, and the optimization is dominated by the IC term, ˆ L S ( f W ) + α ( nβ ) − 1 D ( P (cid:107) Q ) .
Entropy-SGD.
A related approach is Entropy-SGD (Chaudhari et al., 2017), which instead directly minimizes the stochastic complexity, − 1 β
ln E Q e − βL S ( W ) .
This entails optimizing the prior Q , when a PAC-Bayesian bound such as (1) stipulates that the prior be ﬁxed before the draw of the training sample S .
A way out is to sample Q in a differentially private fashion, and this forms the basis of the Entropy-SGLD algorithm (Dziugaite & Roy, 2018a).
For
Q = N (cid:0) w, ( βγ ) − 1 I k (cid:1) , the stochastic complexity can be equivalently written as (up to constant terms) −
[Equation]
w
[Equation]
, which can be interpreted as a measure of ﬂatness of the loss landscape that measures the log-volume of low-loss parameter conﬁgurations around w .
More generally, from the perspective of ICM, both Entropy- and PAC-Bayes- SGD can be viewed as optimization schemes that search for ﬂat minima solutions (Hochreiter & Schmidhuber, 1997).
PAC-Bayes and Occam’s factor.
Lemma 6 gives the form of the optimal posterior under a quadratic approximation of the loss around a local minimizer:
Lemma 6.
Consider a quadratic approximation of the training loss around a local minimizer w P , ˜ L S ( w ) = 12 ( w − w P ) (cid:62) H ( w − w P ) , a ﬁxed prior Q = N ( w Q , λ − 1 I k ) , and a posterior distribution of the form P = N ( w P , Σ P ) .
Then the solution to the convex optimization problem min Σ P E P [˜ L S ( W )] + ( nβ ) − 1 D
( P (cid:107) Q ) , is given by Σ (cid:63)P =
H − 1 λ , where H λ :=
( nβH + λI k ) .
Here we assume λ > 0 is sufﬁciently large so that H λ is positive deﬁnite.
We can use a posterior of the form P = N ( w P , H − 1 λ ) to get the following PAC-Bayesian-style bound that incorporates second-order curvature information of the training loss:
Proposition 7.
Let { λ i } ki =1 be the eigenvalues of H λ and suppose that
λ i ≥ λ > 0 for all i .
Let
Q = N ( w Q , λ − 1 I k ) be a prior, and let P = N ( w P , H − 1 λ ) .
Then with probability of at least 1 − δ over a draw of the sample S , we have
[Equation]
The proof of Proposition 7 is given in Appendix A.6.
Notably, the log-ratio term 12 (cid:80) ki
[Equation] 1 ln λ i λ = − ln (cid:113) det λH λ in (5) is the negative logarithm of the Occam factor (MacKay, 1992; Smith & Le, 2018).
The log-Occam factor is the differential entropy of the Gaussian posterior with a scaled covariance λ
( H λ ) − 1 , and can be interpreted as the amount of information we gain about the model’s parameters after seeing the training data.
From the perspective of ICM, minimizing the right hand side of (5) w.r.t.
the posterior leads to solutions with higher entropy and hence wider minima.
D ISCUSSION
We presented a uniﬁed treatment of PAC-Bayesian and information-theoretic generalization bounds, and obtained a few new results for data-dependent priors and unbounded losses.
The bounds we studied are along the notion that bounded information (between the training data and the output hypothesis) implies learning.
On the other hand, Bassily et al.
and Nachum & Yehudayoff (2019) showed that learning does not imply bounded information.
In particular, the information revealed by a learning algorithm about its input can be unbounded even for hypothesis classes of VC dimension 1.
Livni & Moran (2020) showed a result in a similar vein for the PAC-Bayesian framework.
Identifying the common structural properties of these negative results in the information-theoretic and PAC-Bayesian frameworks is an important avenue for further investigation. A CKNOWLEDGMENTS
This project has received funding from the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation programme (grant agreement n o 757983).
A A PPENDIX A. T ECHNICAL L EMMAS
Proposition 8 collects some well-known facts about the cumulant generating function Λ X ( β ) =
ln E [ e βX ] of a random variable X for β > 0 ; see, e.g., (Boucheron et al., 2013, §2), and (Zhang, 2006a).
Proposition 8 (Facts about cumulant generating function) .
Λ X ( β ) is convex in β . 2.
1 β Λ X ( β ) is an increasing function of β . 3.
Λ X − E
[ X ] (0)
=
Λ (cid:48)
X − E
[ X ] (0)
= 0 .
For real constants a, b , Λ aX + b ( β ) =
Λ X ( aβ )
+ b . 5.
Λ X ( β ) ≤ β E
[ X ] + β 2 2 Var( X ) + O ( β 3 ) , where Var( X ) = E (cid:2) ( X − E
[ X ]) 2 (cid:3) .
If X ∈ [0 , 1] , then 1 β Λ X ( β ) ≤ 1 β ln (cid:0) 1 − (1 − e β ) E [ X ] (cid:1) , with equality when X ∈ { 0 , 1 } is Bernoulli.
X is σ -sub-Gaussian if Λ X − E
[ X ] ( β ) ≤ β 2 σ 2 2 . 8.
X is ( σ, c ) -sub-gamma if Λ X − E
[ X ] ( β ) ≤ β 2 σ 2 2(1 − c ) for all β ∈ (0 , 1 c ) .
Recall that the annealed expectation is M β ( w ) = − β − 1 ln E µ [ e − β(cid:96)
( w,Z ) ]
= − β − 1 Λ − (cid:96) ( w,Z ) ( β ) .
By
Proposition 8 item 1) and Jensen’s inequality, we have M β ( w ) ≤ L µ ( w ) .
For general loss functions,
Proposition 8 item 5) is useful for getting bounds in the opposite direction.
By items 4) , 7) and 8) of
Proposition 8, if for all w ∈ W , (cid:96) ( w, Z ) is σ -sub-Gaussian, resp.
, ( σ, c ) -sub-gamma under µ , then we have for all w ∈ W and β > 0 , L µ ( w ) ≤ M β ( w ) + β 2 σ 2 , resp.,
L µ ( w ) ≤ M β ( w ) + β 2(1 − c ) σ 2 .
We will need the following variational characterization of the KL divergence:
Lemma 9 (Donsker-Varadhan, Boucheron et al.
2013, Corollary 4.15) .
Let P, Q be probability measures on W , and let F denote the set of measurable functions f : W
→ R such that E Q [ e f ( W ) ] < ∞ .
If D ( P (cid:107) Q ) < ∞ , then for every f ∈ F , we have
[Equation]
where the supremum is attained when f = ln d P d Q .
We note the following characterization of the inverse of the Fenchel-Legendre dual of a smooth convex function:
Lemma 10 (Boucheron et al.
2013, Lemma 2.4) .
Let ψ be a convex and continuously differentiable function deﬁned on the interval [0 , b ) , where 0 < b ≤ ∞ .
Assume that ψ (0) =
ψ (cid:48) (0)
= 0 .
Then, the Legendre dual of ψ , deﬁned as ψ ∗ ( t ) :
= sup β ∈ [0 ,b ) { βt − ψ ( β ) } , is a nonnegative convex and nondecreasing function on [0 , ∞ ) with ψ ∗ (0) = 0 .
Moreover, for every y ≥ 0 , the set { t ≥ 0 : ψ ∗ ( t ) > y } is non-empty and the generalized inverse of ψ ∗ deﬁned by ψ ∗− 1 ( y )
=
inf { t ≥ 0 : ψ ∗ ( t ) > y } can also be written as ψ ∗− 1 ( y ) =
inf β ∈ (0 ,b ) y + ψ ( β ) β .
We will need the following property of a Gibbs distribution:
Lemma 11 (Zhang 2006a, Proposition 3.1) .
For any real-valued measurable function f on W , any real β > 0 , and any P, Q ∈ M ( W ) such that D ( P (cid:107) Q ) < ∞ , we have β − 1 D ( P (cid:107) P ∗ ) =
E P [ f ( W )]
+ β − 1 D ( P (cid:107) Q ) + β − 1 ln E Q
[ e − βf ( W ) ] , where P ∗ is the Gibbs distribution P ∗ (d w ) :
= e − βf ( w ) E
Q [ e [Equation]
) ] Q (d w ) .
Consequently,
[Equation]
Finally, we recall the golden formula : For all Q ∈ M ( W ) such that D ( P W (cid:107) Q ) < ∞ , we have
[Equation]
where D ( P W | S (cid:107) Q | P S ) = (cid:82) Z n D ( P W | S
= s (cid:107) Q ) µ ⊗ n (d s ) .
By the golden formula (6), under the oracle prior Q (cid:63) =
E S
[ P W | S ] , E S
[ D ( P (cid:107) Q (cid:63) )]
= I ( S ; W ) .
All information-theoretic quantities are expressed in nats , unless speciﬁed otherwise.
A. O MITTED DETAILS IN S ECTION
We include a proof of Lemma 1 since we will use the arguments.
Proof of Lemma 1.
Applying the Donsker-Varadhan Lemma 9 to the function,
[Equation]
we obtain,
[Equation]
Exponentiating both sides of (8) and taking expectations w.r.t.
S ∼ µ ⊗ n , we have
[Equation]
Since Z i i.i.d.
∼ µ , for any w ∈ W and β > 0 , we have e − nβM
[Equation]
This observation and Fubini’s theorem implies that the right hand side of (9) is equal to one.
This proves the IEI.
Proof of Theorem 2.
Letting R ( S ) :
= nβ E P [ M β ( W ) − L S ( W )] − D ( P (cid:107) Q ) , by Lemma 1, we have
E S [ e R ( S ) ] ≤ 1 .
By Markov’s inequality,
[Equation]
Therefore, with probability of at least 1 − δ over the choice of S ∼ µ ⊗ n , we have for all P (cid:28) Q ,
[Equation]
By assumption,
[Equation]
(11) when (1) follows.
For the bound in expectation, note that by Jensen’s inequality, e E S [ R ( S )] ≤ E S
[ e R ( S ) ] ≤ 1 , which implies E S [ R ( S )] ≤ 0 , when we have
[Equation]
Using (11), and rearranging and optimizing, we have
[Equation]
where the second equality follows from Lemma 10.
We can also optimize β in (1) at a small cost using the union bound:
Proposition 12.
Consider the setting in Theorem 2.
If (cid:96) ( w, Z ) is σ -sub-Gaussian under µ for all w ∈ W , then for any constants α > 1 and v > 0 , and any δ ∈ (0 , 1] , for all β ∈ (0 , v ] , with probability of at least 1 − δ , we have
[Equation]
where K = max { log α (cid:0) vσ √ 2 α (cid:1) , 0 } + e .
The proof follows that of van Erven (2014, Lemma 8), extending it to sub-Gaussian losses.
Proof.
For 0 < u < v , and i = 0 , . . .
, (cid:100) log α vu (cid:101) − 1 , for all i let β
i =
uα i be selected before the draw of the training sample.
Then for every β ∈ [ u, v ] , there is a β i such that β i ≤ β ≤ αβ i .
We can extend (10) by applying a union bound over the β i ’s, so that for all P with probability of at least 1 − δ over the draw of S , the following holds simultaneously for all β i :
[Equation]
By Proposition 8(2)
, for any w ∈ W , M β ( w ) is a nonincreasing of β .
Thus for any β ∈ [ u, v ] and β i
such that β i ≤ β ≤ αβ i , M β ( w ) ≤ M β
i
w ) and 1 β i ≤
αβ .
Moreover, since (cid:96) ( w, Z ) is σ -sub-Gaussian under µ by assumption, we have for all w ∈ W and β > 0 , L µ ( w ) ≤ M β ( w )+ β 2 σ 2 .
Hence, with probability of at least 1 − δ we have,
[Equation]
Letting J = D ( P (cid:107) Q ) + ln log α √ n + K δ , we ﬁnd that the value for β that optimizes the right hand side of the bound in the statement of the proposition is bounded from below by (cid:113) 2 αnσ 2 .
Letting u = 1 √ n min (cid:110)(cid:113) 2 α σ 2 , v (cid:111) and plugging it in (14) completes the proof.
Remark 13 (Related work) .
A variation of the IEI for the special case of the 0-1 loss appears in the monograph by Catoni (Catoni, 2007, Eq. 1.2), and has been rediscovered more recently for the sub-Gaussian loss in (Hellstr¨om & Durisi, 2020b;a).
The statements of (Hellstr¨om & Durisi, 2020b,
Corollary 3, Eq. 20) and (Hellstr¨om & Durisi, 2020a, Corollary 2, Eq. 26) which are analogues of our Proposition 12 are incorrect as they assume that β can be optimized “for free,” when in fact we have to pay a union bound price for optimizing β , which is selected before the draw of the training sample.
We also note two related works that focus exclusively on unifying either PAC-Bayesian and
Occam’s razor-based bounds for the 0-1 loss (Blum & Langford, 2003), or information-theoretic bounds for the sub-Gaussian loss (Hafez-Kolahi et al., 2020).
A. T HE S TRONG F UNCTIONAL R EPRESENTATION L EMMA AND S INGLE - DRAW BOUNDS
In this section, we highlight a functional characterization of the mutual information in relation to the “single-draw” bound due to Bassily et al.
A randomized learning algorithm P W | S can be viewed as a noisy channel that maps the input sample
S to conditional distributions of hypotheses W in W .
Consider a communication task where Alice and Bob share a common random string R , possibly of unbounded length, generated in advance.
Alice observes a sample s ∈ Z n drawn according to P S and communicates a preﬁx-free message
M to Bob via a noiseless channel such that Bob can output a hypothesis w ∈ W that is distributed according to P W | S = s .
Harsha et al.
(2009) showed that the minimum expected description length of
M (in bits) needed to accomplish this task is roughly equal to the input-output mutual information
I ( S ; W ) .
Variations on this theme have appeared in a learning-theoretic setting (Blum & Langford, 2003), and by way of the bits-back argument due to Hinton & van Camp (1993); see, e.g., Havasi et al.
More generally, we note the following functional characterization of the mutual information:
Theorem 14 (Strong functional representation lemma (SFRL), Li & El Gamal 2018) .
For any pair of jointly distributed random variables ( S, W ) with I ( S ; W ) < ∞ , there exists a random variable R independent of S such that W can be represented as a deterministic function of S and R , and
[Equation]
The SFRL implies the existence of a random variable R ⊥⊥ S such that H ( W | R )
≈
I ( S ; W ) .
Consider the case for the { 0 , 1 } -valued loss.
If the algorithm P W | S is deterministic, then we have
I :=
I ( S ; W ) =
H ( W ) − H
( W | S )
=
H ( W ) .
By Markov’s inequality, with probability of at least 1 − δ , we have P W ( w ) ≥ e − I/δ .
Let W 0 ⊆
W be the set of hypotheses so that P W ( w ) ≥ e − I/δ .
The size of W 0 is at most e I/δ since 1 =
Pr( W ) ≥ Pr( W 0 ) =
(cid:80) w ∈W 0 P W ( w ) ≥ |W 0 | e − I/δ .
By the Chernoff-Hoeffding bound, for every w in W ,
[Equation]
Applying the union bound over all w ∈ W 0 , the probability of error for the algorithm is 2 |W 0 | e − 2 n(cid:15) 2 + δ , where the second summand is for the case where the algorithm outputs a function outside W 0 .
Hence, for every w ∈ W 0 , the empirical risk is close to the true risk for n
= Ω( Iδ(cid:15) 2 ) with probability of at least 1 − δ .
By the SFRL, any randomized algorithm can be simulated by randomly sampling a deterministic algorithm from some distribution R before observing the input S .
These algorithms have the property that on average (over R ), H ( W | R )
≈
I ( S ; W ) .
Using the argument for the deterministic case and integrating over R , we can bound the probability of error for the randomized case as
[Equation]
An analogous bound for the sub-Gaussian loss appears in (Xu & Raginsky, 2017, Theorem 3).
A. R ECOVERING CLASSICAL PAC-B AYESIAN BOUNDS
By Proposition 8 item 6) , for a { 0 , 1 } -valued loss, we have
[Equation]
Φ β is an increasing one-to-one mapping of the unit interval onto itself, and is convex for β >(Catoni, 2007).
The inverse of Φ β is given by Φ − 1 β ( x ) = 1 − e − βx 1 − e − β , and we recover Catoni’s PACBayesian bound in Corollary 4.
Starting from Catoni’s bound in Corollary 4, using 1 ≤ β (1 − e − β ) − 1 ≤ (1 − β 2 ) − 1 , we have [Equation]
[Equation]
(16) and (17) recover, resp., the bounds due to Catoni (2007, Theorem 1.2.1) and McAllester (2013,
Theorem 2), where for the latter we additionally require that β < 2 .
We now show how inequality (10) relates to other well-known PAC-Bayesian inequalities such as the “PAC-Bayes-KL-inequality” (Seeger, 2002; Maurer, 2004).
Applying the Donsker-Varadhan lemma to the function f ( w ) =
nβ ( L µ ( w ) − L S ( w )) , which involves the true risk L µ ( w ) instead of the annealed expectation M β ( w ) (see 7), and following the same steps as in the proof of (10) in
Theorem 2, we arrive at the following PAC-Bayesian bound:
[Equation]
For an explicit comparison of (18) with (10), we write the latter as
Pr
[Equation]
where the last term in the right hand side of the bound in (19) vanishes since
[Equation]
for any w ∈ W and β > 0 .
In contrast, the term ln E Q E S [Equation] ∼ [Equation]
⊗ n e
[Equation] ) involving the true risk in (18) is, in general, positive.
Specializing to the case of a { 0 , 1 } -valued loss, ﬁx β = 1 , and let ∆ : [0 , 1] × [0 , 1] → R be a convex function.
Applying the Donsker-Varadhan lemma to the function, f ( w ) = n ∆( L S ( w ) , L µ ( w )) , following the same steps as in the proof of (10) in Theorem 2, and by noting that ∆( E P [ L S ( W )] , E P [ L µ ( W )])) ≤ E P [∆( L S ( W ) , L µ ( W ))] , we arrive at the following PACBayesian bound (see, e.g., Maurer (2004, Lemma 3), Germain et al.
(2009, Theorem 2.1), Rivasplata et al.
(2020, Eq. 4)):
[Equation]
For x, y ∈ [0 , 1] , the binary KL divergence is kl( y (cid:107) x ) =
y ln yx
+ (1 − y )
ln 1 − y 1 − x .
The PAC-BayesKL-inequality comes about by upper-bounding the log-exponential-moment term involving the true risk in the right hand side of the bound in (20): For ∆( y, x ) =
kl( y (cid:107) x ) , Maurer (2004) showed that for n ≥ 8 , E Q E S (cid:48)
[ e n ∆( L S (cid:48)
( W ) ,L µ ( W )) ] ≤ 2 √ n , when we have Pr
[Equation]
Letting ∆( y, x ) =
( y − x ) 2 leads to the bound in (McAllester, 1999b), while letting ∆( y, x ) =
( y − x ) 2 / (2 x ) leads to that in (Thiemann et al., 2017).
Fixing β = 1 in (1), we recover (Germain et al., 2016, Corollary 5):
Corollary 15.
Consider the setting in Theorem 2.
If the loss (cid:96) is ( σ, c ) -sub-gamma with c < 1 , then with probability of at least 1 − δ over the choice of S ∼ µ ⊗ n , for all distributions P (cid:28) Q over W ,
E P [g( W, S )] ≤ 1 n ( D ( P (cid:107) Q ) + ln(1 /δ ))
+ σ 2 2(1 − c ) .
The condition c < 1 guarantees that β = 1 ∈ (0 , 1 c ) when the sub-gamma condition in Proposition8) is satisﬁed.
In the limit c → 0 + , a sub-gamma loss reduces to the sub-Gaussian loss (Boucheron et al., 2013, §2.4), and we recover (Germain et al., 2016, Corollary 4).
A. P ROOF OF P ROPOSITION
For the proof of Proposition 5, the key quantity of interest is the approximate max-information between the input S and the data-dependent prior.
We note the following deﬁnitions.
For α ≥ 0 , the α -approximate max-divergence is deﬁned as
[Equation]
The max-divergence D ∞ ( P (cid:107) Q ) is deﬁned as D α ∞ ( P (cid:107) Q ) for α = 0 .
For a pair of variables ( X, Y ) with joint law P XY and marginals P X and P Y , the α -approximate max-information between X and
Y is deﬁned as
[Equation]
The max-information I ∞ ( X ; Y ) is deﬁned to be I α ∞ ( X ; Y ) for α = 0 .
I ∞ ( X ; Y ) is an upper bound on the ordinary mutual information
I ( X ; Y ) (Dwork et al., 2015).
Deﬁnition 16 (Differential Privacy, Dwork & Roth 2014) .
For any (cid:15) ≥ 0 and δ ∈ [0 , 1] , a learning algorithm P W | S is said to be ( (cid:15), δ ) -differentially private if for all pairs of datasets s, s (cid:48) ∈
Z n that differ in a single element, D δ ∞ ( P W | S = s (cid:107) P W | S = s (cid:48) ) ≤ (cid:15) .
The case δ = 0 is called pure differential privacy.
Deﬁnition 17 (Max-Information of an algorithm, Dwork et al. 2015) .
We say that an algorithm P W | S has α -approximate max-information of k , denoted as I α ∞ ,µ ( P W | S , n ) ≤ k , if for every distribution µ over Z , we have I α ∞ ( S ; W ) ≤ k when S ∼ µ ⊗ n .
It follows from the deﬁnition of α -approximate max-information that if an algorithm P W | S has bounded approximate max-information, then we can control the probability of “bad events” that may arise as a result of the dependence of the output W on the input S (Dwork et al., 2015).
Let S (cid:48) ⊥⊥ W be an independent sample with the same distribution as S .
If for some α ≥ 0 , I α ∞ ( S ; W ) ≤ k , then for any event O ⊆
Z n × W , we have
[Equation]
Pure differential privacy implies a bound on the approximate max-information:
Theorem 18 (Pure differential privacy and α -approximate max-information, Dwork et al.
Theorem 20) .
If P W | S is an ( (cid:15), 0) -differentially
private algorithm, then I ∞ ,µ ( P W | S , n ) ≤ (cid:15)n , and for any α > 0 , I α ∞ ,µ ( P W | S , n ) ≤ n(cid:15) 2 / 2 + (cid:15) (cid:112) n ln(2 /α ) / 2 .
Remark 19.
The result above is extended to ( (cid:15), δ ) -differential privacy in (Rogers et al., 2016,
Theorem 3.1): If P W | S is an ( (cid:15), δ ) -differentially private algorithm for (cid:15) ∈ (0 , 1 / 2] and δ ∈ (0 , (cid:15) ) , then for
α = O ( n (cid:112) δ/(cid:15) ) , I α ∞ ,µ ( A , n )
=
O ( n(cid:15) 2 + n (cid:112) δ/(cid:15) ) .
The proof of Proposition 5 follows closely that of (Dziugaite & Roy, 2018b, Theorem 4.2).
Proof of Proposition 5.
For every Q ∈ M ( W ) , let
F ( [Equation] =
[Equation]
By (10), we have Pr S (cid:48) ∼ µ ⊗ n (cid:0) S (cid:48) ∈ F ( Q ) (cid:1) ≤ δ (cid:48) .
From (22), we have
Pr S ∼ µ ⊗ n
[Equation]
δ
[Equation]
Letting δ :=
e I α ∞ ,µ ( Q 0 ,n ) · δ (cid:48) + α , for α ∈ (0 , δ ) we have,
[Equation]
The proof is complete by supplanting I α ∞ ,µ ( Q 0 , n ) with the bound in Theorem 18, and choosing α = δ 2 .
By Remark 19, Proposition 5 can be extended to ( (cid:15), δ ) -differentially private priors.
We can also bound the expected generalization error:
Proposition 20.
Consider the setting in Theorem 2.
Let Q 0 ∈ K ( S , W ) be an ( (cid:15), 0)
-differentially private algorithm.
Then with probability of at least 1 − δ over a draw of the sample S , for all P , we have
[Equation]
The main advantage of the max-information formulation in Propositions 5 and 20 is that we can get high probability guarantees at the cost of a O ( n(cid:15) 2 + (cid:15) (cid:112) n ln 1 /δ ) correction term.
This cost is compensated for by a lower KL complexity since the prior is more “aligned” with the data-dependent posterior than when chosen independently of the data.
As is well-known (Dwork et al., 2015; Feldman & Steinke, 2018), a small mutual information between the data and the prior will not ensure that bad events will happen with low probability.
A. P ROOFS FOR S ECTION
When G = M ( W ) , applying Lemma 11 to f ( w ) = nL S ( w ) , and writing β for nβ , we recover the
Gibbs algorithm, P (cid:63) , in which case OIC β G in (3) evaluates to the (extended) stochastic complexity , − 1 β ln E Q [ e − βL S
( W ) ] .
We show how to optimize the bound in Corollary 4 w.r.t.
the parameters β and λ .
Proof of (4) .
First, note that the bound in Corollary 4 holds uniformly for all β > 1 at an additional cost arising from a union bound argument (Catoni, 2007, Theorem 1.2.7):
For α > 1 ,
[Equation]
Second, we select λ before the draw of the training sample from a ﬁnite grid of possible values:
Following (Langford & Caruana, 2002; Dziugaite & Roy, 2017) [Equation]
let λ = ce − j/b for some j ∈ N and ﬁxed b ∈ N , c ∈ (0 , 1) , where b and c control, resp., the resolution and size of the grid.
If (24) holds for each j ∈ N with probability of at least 1 − 6 δ π 2 j 2 , then by the union bound, it holds for all j ∈ N simultaneously with probability of at least 1 − δ , since (cid:80) ∞
j =1 6 π 2 j 2 = 1 .
Solving for j in terms of λ , we have
[Equation]
Finally, we account for the cost of using a Monte Carlo estimate of the empirical risk, ˆ L S ( f W ) = 1 m (cid:80) mi =1 L S ( f W i ) , where W i i.i.d.
∼ P .
By an application of the Chernoff bound (Langford & Caruana, 2002, Theorem 2.5) and Pinsker’s inequality, for any δ (cid:48) ∈ (0 , 1) , we have with probability of at least 1 − δ (cid:48) , E P
[ L S ( f W )]
≤ ˆ L S ( f W ) + (cid:113) 12 m ln 2 δ (cid:48) .
By another application of the union bound, Corollary 4 ﬁnally takes the form: For any δ, δ (cid:48) ∈ (0 , 1) , ﬁxed α > 1 , c ∈ (0 , 1) , b ∈ N , and m, n ∈ N , with probability of at least 1 − δ − δ (cid:48) over a draw of
S ∼ µ ⊗ n and W ∼ ( P ) ⊗ m ,
[Equation]
where R ( λ, β ; δ, δ (cid:48) )
= 2 αnβ ln
[Equation]
Proof of Lemma 6.
Letting θ = w − w P , and P (cid:48)
=
P − w P , note that θ (cid:62)
Hθ =
Tr( θ (cid:62) Hθ ) =
Tr( Hθθ (cid:62) ) .
Hence
[Equation]
For Q ∼ N ( w Q , Σ Q ) and P ∼ N ( w P , Σ P ) , we have E P
[Equation]
The derivative of the RHS w.r.t.
Σ P is 12 (cid:104) H − ( nβ ) − 1 Σ − 1 P + ( nβ ) − 1 Σ − 1 Q (cid:105) (cid:62) , where we have used the fact that ∇ A Tr( AB ) =
B (cid:62) , and ∇ A ln det( A ) =
( A − 1 ) (cid:62) .
Setting the derivative to zero and Σ Q = λ − 1 I k yields the result.
Proof of Proposition 7.
The proof follows from (10), and the fact that for Q = N ( w Q , λ − 1 I k ) ,
P = N ( w P , H − 1 λ ) such that
λ
i ≥ λ > 0 for all i , we have
[Equation]
(cid:88) i =
ln
λ i λ (cid:33) .