N ONVACUOUS L OSS B OUNDS WITH F AST R ATES FOR N EURAL N ETWORKS VIA C ONDITIONAL I NFORMATION M EASURES
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a framework to derive bounds on the test loss of randomized learning algorithms for the case of bounded loss functions.
This framework leads to bounds that depend on the conditional information density between the the output hypothesis and the choice of the training set, given a larger set of data samples from which the training set is formed.
Furthermore, the bounds pertain to the average test loss as well as to its tail probability, both for the PAC-Bayesian and the single-draw settings.
If the conditional information density is bounded uniformly in the size n of the training set, our bounds decay as 1 /n , which is referred to as a fast rate.
This is in contrast with the tail bounds involving conditional information measures available in the literature, which have a less benign 1 / √ n dependence.
We demonstrate the usefulness of our tail bounds by showing that they lead to estimates of the test loss achievable with several neural network architectures trained on MNIST and
Fashion-MNIST that match the state-of-the-art bounds available in the literature.
I NTRODUCTION
In recent years, there has been a surge of interest in the use of information-theoretic techniques for bounding the loss of learning algorithms.
While the ﬁrst results of this ﬂavour can be traced to the probably approximately correct (PAC)-Bayesian approach (McAllester, 1998; Catoni, 2007) (see also (Guedj, 2019) for a recent review), the connection between loss bounds and classical information-theoretic measures was made explicit in the works of Russo & Zou (2016) and Xu &
Raginsky (2017), where bounds on the average population loss were derived in terms of the mutual information between the training data and the output hypothesis.
Since then, these average loss bounds have been tightened (Bu et al., 2019; Asadi et al., 2018; Negrea et al., 2019).
Furthermore, the information-theoretic framework has also been successfully applied to derive tail probability bounds on the population loss (Bassily et al., 2018; Esposito et al., 2019; Hellstr ¨ om & Durisi, 2020a).
Of particular relevance to the present paper is the random-subset setting , introduced by Steinke &
Zakynthinou (2020) and further studied in (Hellstr¨om & Durisi, 2020b; Haghifam et al., 2020).
In this setting, a random vector S is used to select n training samples Z from a larger set (cid:101) Z of 2 n samples.
Then, bounds on the average population loss are derived in terms of the conditional mutual information (CMI)
I
( W ; S
| (cid:101) Z ) between the chosen hypothesis W and the random vector S given the larger training set (cid:101) Z .
The bounds obtained in Xu & Raginsky (2017) depend on the mutual information
I ( W ; Z ) , a quantity that can be unbounded if W reveals too much about the training set Z .
In contrast, bounds for the random-subset setting are always ﬁnite, since I ( W ; S | (cid:101) Z ) is never larger than n bits.
Most of the information-theoretic population loss bounds mentioned thus far are given by the training loss plus a term with a (cid:112) IM ( P W Z ) /n
-dependence, where IM ( P W Z ) denotes an appropriate information measure, such as mutual information or maximal leakage (Issa et al., 2020).
Assuming that the information measure grows at most polylogarithmically with n , the convergence rate of the population loss to the training loss is ˜ O
(1 / √ n ) , where the ˜ O -notation hides logarithmic factors.
This is sometimes referred to as a slow rate .
In the context of bounds on the excess risk, deﬁned as the difference between the achieved population loss for a chosen hypothesis w and its inﬁmum over the hypothesis class, it is known that slow rates are optimal for worst-case distributions and hypothesis classes (Talagrand, 1994).
However, it is also known that under the assumption of realizability (i.e., the existence of a w in the hypothesis class such that the population loss L P Z ( w ) = 0 ) and when the hypothesis class is ﬁnite, the dependence on the sample size can be improved to ˜ O
(1 /n )
(Vapnik, 1998, Chapter 4).
This is referred to as a fast rate .
Excess risk bounds with fast rates for randomized classiﬁers have also been derived, under certain additional conditions, for both bounded losses (Van Erven et al., 2015) and unbounded losses (Gr¨unwald & Mehta, 2020).
Notably, Steinke & Zakynthinou (2020, Thm. 2(3)) derive a population loss bound whose dependence on n is I ( W ; S | (cid:101) Z )
/n .
The price for this improved dependence is that the training loss that is added to the n -dependent term is multiplied by a constant larger than 1 .
Furthermore, (Steinke & Zakynthinou, 2020, Thm.
8) shows that if the Vapnik-Chervonenkis (VC) dimension of the hypothesis class is ﬁnite, there exists an empirical risk minimizer (ERM) whose CMI grows at most logarithmically with n .
This implies that the CMI approach leads to fast-rate bounds in certain scenarios.
However, the result in (Steinke & Zakynthinou, 2020, Thm. 2(3)) pertains only to the average population loss: no tail bounds on the population loss are provided.
Throughout the paper, we will, with an abuse of terminology, refer to bounds with an n -dependence of the form IM ( P W Z ) /n
as fast-rate bounds.
Such bounds are also known as linear bounds (Dziugaite et al., 2020).
Note that the n -dependence of the information measure IM ( P W Z ) has to be at most polylogarithmic for such bounds to actually achieve a fast rate in the usual sense.
An intriguing open problem in statistical learning is to ﬁnd a theoretical justiﬁcation for the capability of overparameterized neural networks (NNs) to achieve good generalization performance despite being able to memorize randomly labeled training data sets (Zhang et al., 2017).
As a consequence of this behavior, classical population loss bounds that hold uniformly over a given hypothesis class, such as VC bounds, are vacuous when applied to overparameterized NNs.
This has stimulated recent efforts aimed at obtaining tighter population loss bounds that are algorithm-dependent or data-dependent.
In the past few years, several studies have shown that promising bounds are attainable by using techniques from the PAC-Bayesian literature (Zhou et al., 2019; Dziugaite & Roy, 2017; Dziugaite et al., 2020).
The PAC-Bayesian approach entails using the Kullback-Leibler (KL) divergence to compare the distribution on the weights of the NN induced by training to some reference distribution.
These distributions are referred to as the posterior and the prior, respectively.
Recently, Dziugaite et al.
(2020) used data-dependent priors to obtain state-of-the-art bounds for LeNet-5 trained on MNIST and Fashion-MNIST.
In their approach, the available data is used both for training the network and for choosing the prior.
This leads to a bound that is tighter than previously available bounds.
Furthermore, the bound can be further improved by minimizing the KL divergence between the posterior and the chosen prior during training.
One drawback of the PAC-Bayesian approach is that it applies only to stochastic NNs, whose weights are randomly chosen each time the network is used, and not to deterministic NNs with ﬁxed weights.
Information-theoretic bounds have also been derived for iterative, noisy training algorithms such as stochastic gradient Langevin dynamics (SGLD)
(Bu et al., 2019).
These bounds lead to nonvacuous estimates of the population loss of overparameterized NNs that are trained using SGLD through the use of data-dependent priors (Negrea et al., 2019).
However, these bounds do not apply to deterministic NNs, nor to standard stochastic gradient descent (SGD) training.
Furthermore, the bounds pertain to the average population loss, and not to its tails.
Although the techniques yielding these estimates can be adapted to the PAC-Bayesian setting, as discussed by Negrea et al.
App.
I), the resulting bounds are generally loose.
1. C ONTRIBUTIONS
In this paper, we extend the fast-rate average loss bound by Steinke & Zakynthinou (2020) to the
PAC-Bayesian and the single-draw settings.
We then use the resulting PAC-Bayesian and single-draw bounds to characterize the test loss of NNs used to classify images from the MNIST and FashionMNIST data sets.
The single-draw bounds can be applied to deterministic NNs trained through SGD but with Gaussian noise added to the ﬁnal weights, whereas the PAC-Bayesian bounds apply only to randomized neural networks, whose weights are drawn from a Gaussian distribution each time the network is used.
For the same setup, we also evaluate the slow-rate PAC-Bayesian and single-draw bounds from (Hellstr¨om & Durisi, 2020b).
Our numerical experiments reveal that both the slow-rate bounds from (Hellstr¨om & Durisi, 2020b) and the newly derived fast-rate bounds are nonvacuous.
Furthermore, for some settings, the fast-rate bounds presented in this paper are quantitatively stronger than the corresponding slow-rate ones from (Hellstr¨om & Durisi, 2020b), and essentially match the best bounds available in the literature for SGD-trained NNs (Dziugaite et al., 2020).
1. P RELIMINARIES
We now detail some notation and describe the random-subset setting introduced in (Steinke &
Zakynthinou, 2020).
Let Z be the instance space, W be the hypothesis space, and (cid:96) : W × Z
→ R + be the loss function.
Throughout the paper, we will assume that the range of (cid:96) ( w, z ) is restricted to [0 , 1] for all w ∈ W and all z ∈ Z .
A typical example of such a loss function is the classiﬁcation error.
In this setting, the sample Z consists of an example X ∈ X and a corresponding label Y ∈ Y .
Then, the loss is given by (cid:96) ( W, Z )
= 1 { f W ( X ) (cid:54)
= Y } , where f W ( · ) is the map from X to Y induced by the hypothesis W .
We note that, when applying our bounds to NNs, the function (cid:96) ( · , · ) used to characterize the performance of the network does not necessarily need to coincide with the loss function used when training the NN.
For instance, one could use the (unbounded) cross-entropy loss when training the NN, and apply the bounds for the scenario in which (cid:96) ( · , · ) is the classiﬁcation error.
In the random-subset setting, 2 n training samples (cid:101) Z = ( ˜ Z 1 , . .
˜ Z 2 n ) are available, with all entries of (cid:101)
Z being drawn independently from some distribution P Z on Z .
However, only a randomly selected subset of cardinality n is actually used for training.
Following (Steinke & Zakynthinou, 2020), we assume that the training data Z ( S ) is selected as follows.
Let S =
( S 1 , . . .
, S n ) be an n -dimensional random vector, the elements of which are drawn independently from a Bern(1 / 2) distribution and are independent of (cid:101) Z .
Then, for i = 1 , . .
. , n , the i th training sample in Z ( S ) is Z i ( S i ) =
˜
Z i + S i n .
Thus, the binary variable S i determines whether the training set Z ( S ) will contain the sample ˜ Z i or the sample ˜ Z i + n . The selected training procedure, including the loss function used for training, will determine the conditional distribution P W | Z ( S ) on the hypothesis class given the training data.
For a given W ∼ P W | Z ( S ) , we let L Z ( S ) ( W ) =
1 n (cid:80) ni =1 (cid:96) ( W, Z i ) denote the training loss.
Furthermore, we let ¯ S denote the modulo-2 complement of S .
Then L Z ( ¯ S ) ( W ) can be interpreted as a test loss, since W is conditionally independent of Z ( ¯ S ) given Z ( S ) .
Finally note that the average over ( (cid:101) Z , S ) of the test loss is the population loss L P Z ( W ) =
E P (cid:101) ZS
[ L Z ( ¯ S ) ( W )] =
E P Z [ (cid:96) ( W, Z )] .
Our bounds will depend on several different information-theoretic quantities, which we shall introduce next.
The information density ı
( W, Z ) between W and Z is deﬁned as ı ( W, Z ) =
log d P W Z d P W P Z , where d P W Z d P W P Z is the Radon-Nikodym derivative of P W Z with respect to P W P Z .
The information density is well-deﬁned if P W Z is absolutely continuous with respect to P W P Z , denoted by P W Z (cid:28) P W P Z .
The conditional information density ı
( W, S | (cid:101) Z ) between W and S given (cid:101) Z is deﬁned as ı ( W, S | (cid:101) Z ) = log d P W (cid:101)
ZS d P W | (cid:101) Z P (cid:101) ZS , provided that P W (cid:101) ZS (cid:28) P W | (cid:101) Z P (cid:101) ZS .
The mutual information can be obtained as I ( W ; Z ) =
E P W Z [ ı ( W, Z )] and the conditional mutual information as I ( W ; S | (cid:101) Z ) =
E P W (cid:101) ZS (cid:104) ı
( W, S | (cid:101) Z ) (cid:105) .
We will also need the KL divergences D ( P W | Z || P W ) =
E P W | Z
[ ı ( W, Z )] and D ( P W | (cid:101)
ZS || P W | (cid:101) Z ) =
E P W | (cid:101) ZS
(cid:104) ı
( W, S | (cid:101) Z ) (cid:105) .
In practical applications, the marginal distribution P W is not available, since P Z is unknown.
Furthermore, P W | (cid:101) Z is also difﬁcult to compute, since marginalizing P S P W | (cid:101)
ZS over S involves performing training 2 n times.
Hence, bounds depending on ı ( W, Z ) or on ı ( W, S | (cid:101) Z ) cannot typically be evaluated.
Therefore, it will be convenient to replace the information density ı
( W, Z ) with the proxy log d P W Z d Q W P Z
and ı ( W, S | (cid:101) Z )
with log
[Equation]
Z P (cid:101) ZS .
Here, Q W and Q W | (cid:101) Z are suitably chosen auxiliary distributions (priors) that are used in place of the intractable, true marginals.
B ACKGROUND
We next review the bounds available in the literature that are relevant for this paper.
Then, in Section 3, we will present novel fast-rate bounds.
The canonical PAC-Bayesian population-loss bound for loss functions bounded between 0 and 1 states that the following holds with probability at least 1 − δ under P Z (Guedj & Pujol, 2019, Prop. 3):
[Equation]
Here, Q W is the prior on the hypothesis space W , which has to be independent of Z .
The version of the bound given in (1) slightly improves the dependence on the sample size n as compared to the bound reported in (McAllester, 2003, Thm. 1).
We note that, due to the square root, this is a slow-rate bound.
In the vein of (Catoni, 2007, Thm. 1.2.6) and (Seeger, 2002, Thm. 1), McAllester derived the following alternative bound (McAllester, 2013, Eq. (21)): for all γ ∈ R , and with probability at least 1 − δ under P Z , [Equation] Here, d γ ( q || p ) =
γq − log(1 − p + pe γ ) , and one can show that sup γ d γ ( q || p ) =
d ( q || p ) , where d ( q || p ) indicates the KL divergence between two Bernoulli distributions with parameters p and q respectively.
Let q = E P W | Z [ L Z ( W )] and let c denote the right-hand side of (2).
To use the result in (2) to bound the population loss, we need to ﬁnd
[Equation]
This is the largest population loss that satisﬁes the inequality (2). For small q and c , we have p ∗ ( q, c ) ≈ c , which gives us a fast-rate bound. More generally, for any permissible values of q and c , the bound in (2) can be weakened to obtain the following fast-rate bound (McAllester, 2013, Thm. 2): for all λ ∈ (0 , 1) , and with probability at least 1 − δ under P Z ,
[Equation]
Note that the faster decay in n of this bound comes at the price of a multiplication of the training loss and the KL term by a constant that is larger than 1 . As a consequence, if the training loss or the KL term are large, this multiplicative constant may make the fast-rate bound in (4) quantitatively worse than the slow-rate bound in (1) for a ﬁxed n .
In the so called interpolating setting , where the training loss is 0 , we can set λ = 1 / 2 in (4) and conclude that it is enough for the square-root term in (1) to be smaller than 1 / 4 for the fast-rate bound (4) to be tighter than the slow-rate bound (1). Additional insights on the tightness of these bounds are provided in (Letarte et al., 2019, Thm. 3).
We now turn to the random-subset setting introduced by Steinke & Zakynthinou (2020), and described in Section 1.2.
In (Steinke & Zakynthinou, 2020, Thm. 2), several bounds on the average population loss are derived for loss functions bounded between 0 and 1 , including the following slow-rate and fast-rate bounds:
[Equation]
Similar to the bound in (4), the price for a fast rate is a multiplicative constant in front of the training loss and the mutual information term.
The slow-rate bound in (5) was improved in (Haghifam et al., 2020, Thm. 3.4) by combining the samplewise approach from (Bu et al., 2019) with the disintegration approach in (Negrea et al., 2019), whereby the expectation over (cid:101) Z , which is implicit in the deﬁnition of CMI, is pulled outside of the square root.
As we detail in the following proposition, the bound in (Haghifam et al., 2020, Thm. 3.4) can be further tightened by also pulling out the expectation with respect to the conditional distribution P W | (cid:101)
Z from the square root.
The proof of the resulting bound, which is novel, is deferred to Appendix A.1.
Proposition 1.
Consider the random-subset setting described in Section 1.2.
Then,
[Equation]
We recover (Haghifam et al., 2020, Thm. 3.4) by applying Jensen’s inequality to move the expectation with respect to P W | (cid:101)
Z inside the square root.
Furthermore, we recover (5) by applying Jensen’s inequality once more to move the remaining expectation over P (cid:101) Z and the the empirical average over i inside the square root and by upper-bounding the resulting sum of samplewise CMIs by I ( W ; S | (cid:101) Z ) .
In (Hellstr¨om & Durisi, 2020b), the slow-rate average loss bound (5) was extended to the PACBayesian setting and the single-draw setting through the use of an exponential inequality.
Speciﬁcally, the following bounds on the test loss L Z ( ¯ S )
( W ) are derived: with probability at least 1 − δ under P (cid:101) ZS ,
[Equation]
Furthermore, with probability at least 1 − δ under P W (cid:101) ZS ,
[Equation]
While the bounds in (8) and (9) pertain to the test loss instead of the population loss, one can obtain population loss bounds by adding a penalty term to (8) and (9), as discussed in (Hellstr ¨ om & Durisi, 2020b, Thm. 2).
However, when comparing the bounds to the empirical performance of learning algorithms, the population loss is unknown.
Thus, in practice, one has to resort to evaluating a test loss.
In Section 3 below, we will derive fast-rate analogues of the tail bounds (8) and (9), again at the price of multiplicative constants.
F AST -R ATE R ANDOM -S UBSET B OUNDS
We now present an exponential inequality from which several test loss bounds can be derived, in a similar manner as was done in (Hellstr ¨ om & Durisi, 2020b).
The derivation, which echos part of the proof of (Steinke & Zakynthinou, 2020, Thm. 2.(3)), is provided in Appendix A.2.
This result and its proof illustrate how to combine the exponential-inequality approach from (Hellstr¨om & Durisi, 2020b) with fast-rate derivations, like those presented in (Steinke & Zakynthinou, 2020, Thm. 2.(3)) and (McAllester, 2013, Thm. 2).
Theorem 1.
Let W ∈ W be distributed according to P W | Z ( S ) =
P W | (cid:101) ZS .
Also, assume that the joint distribution P W (cid:101) ZS
= P W | (cid:101) ZS P (cid:101)
Z P S is absolutely continuous with respect to Q W | (cid:101) Z P (cid:101) Z P S for some conditional prior Q W | (cid:101) Z .
Then, the following holds:
[Equation]
Note that the exponential function in (10) depends linearly on the population loss.
In contrast, the exponential inequality derived in (Hellstr¨om & Durisi, 2020b, Thm. 4) to establish slow-rate generalization bounds for the random-subset setting depends quadratically on the population loss (after the parameter λ therein is suitably optimized).
This difference explains why Theorem 1 allows for the derivation of fast-rate bounds, whereas (Hellstr¨om & Durisi, 2020b, Thm. 4) unavoidably leads to slow-rate bounds.
Also note that, since in the random-subset setting W and (cid:101) Z are dependent both before and after any change of measure argument, the proof technique used in (McAllester, 2013,
App.
A) and, previously in (Seeger, 2002, Thm. 1), to derive (2) cannot be used in the random-subset setting.
By simple applications of Jensen’s inequality and Markov’s inequality, the exponential inequality (10) can be used to derive bounds on the population loss or test loss.
In particular, as detailed in the proof of Corollary 1 below (see Appendix A.3), it can be used to recover (6), but also to establish novel
PAC-Bayesian and single-draw versions of (6).
Corollary 1.
Consider the setting of Theorem 1.
Then, the average population loss is bounded by
[Equation]
Furthermore, with probability at least 1 − δ over P (cid:101) ZS , the PAC-Bayesian test loss is bounded by
[Equation]
Finally, with probability at least 1 − δ over P W (cid:101) ZS , the single-draw test loss is bounded by
[Equation]
Setting Q W | (cid:101)
Z = P W | (cid:101) Z in (11), we recover the CMI bound in (Steinke & Zakynthinou, 2020) since E P (cid:101) ZS (cid:104) D
( P W | (cid:101)
ZS || P W | (cid:101) Z ) (cid:105) = I ( W
[Equation] S | (cid:101) Z ) .
As illustrated in Corollary 2 below, the bound on the average population loss in (11) can be tightened by replacing the CMI with a sum of samplewise
CMIs.
The proof of this result, which involves the same argument used to establish Proposition 1, is presented in Appendix A.4.
Corollary 2.
Consider the setting of Theorem 1.
Then, the average population loss is bounded by
[Equation]
The bounds in (12) and (13) are data-dependent, i.e., they depend on the speciﬁc instances of (cid:101) Z and S .
They can be turned into data-independent bounds that are functions of the average of the information measures appearing in (12) and (13), at the cost of a less benign polynomial dependence on the conﬁdence parameter δ .
Alternatively, one can obtain bounds that have a more benign dependence on δ if one allows the bounds to depend on sufﬁciently high moments of the information measures appearing in (12) and (13), or if one replaces these measures by quantities such as conditional maximal leakage or conditional α -divergence.
See (Hellstr ¨ om & Durisi, 2020b) for a more detailed discussion.
We conclude by noting that for the interpolating case where L Z ( S ) ( W ) = 0 , and under the additional assumption that Q W | (cid:101)
Z = P W | (cid:101)
Z , one can obtain a different exponential inequality than the one reported in Theorem 1, which leads to tighter bounds than the ones reported in Corollary 1.
Speciﬁcally, in these alternative bounds, the factor 3 is replaced with a factor of 1 / log(2)
≈ 1 . 44 .
These bounds are presented in Appendix B.
E XPERIMENTS
To assess the ability of the bounds just discussed to predict the performance of overparameterized
NNs, we next present the result of several numerical experiments for different NN architectures.
Speciﬁcally, we consider fully connected NNs (FCNNs) and convolutional NNs (CNNs).
The performance of the networks is evaluated on the MNIST and Fashion-MNIST data sets.
The bounds that we will consider are (8), (9), (12) and (13), where we set the loss function to be the classiﬁcation error deﬁned in Section 1.2.
The following procedure is used to evaluate the bounds: from the 2 n available training samples (cid:101) Z (from MNIST or Fashion-MNIST), the training set Z ( S ) is constructed by selecting n training samples uniformly at random.
A network is then trained on this data set using a standard SGD procedure, which is described in more detail in Appendix C.2.
Let µ 1 be the the vector containing the weights of the network after training.
The posterior distribution P W | (cid:101) ZS is then chosen to be a Gaussian distribution with mean µ 1 and covariance matrix equal to σ 21 I d , where d is the number of parameters in the network.
The standard deviation σ 1 is chosen as the largest real number, determined to some ﬁnite precision (see Appendix C.2), for which the absolute value of the difference between the training loss of the network with weights µ 1 and the empirical average of the training loss achieved by 5 NNs with weights randomly drawn from N ( W | µ 1 , σ 21 I d ) is less than some speciﬁed threshold. Unless otherwise speciﬁed, we use a threshold of 0 . for MNIST and 0 . 10 for Fashion-MNIST.
Note that this procedure is performed for a ﬁxed Z ( S ) .
Consequently, σ 1 depends on (cid:101) Z and S .
To select the prior Q W | (cid:101)
Z , we proceed as follows.
We form 10 subsets of (cid:101) Z , each of size n .
The ﬁrst subset contains the ﬁrst n samples in (cid:101) Z , the last contains the last n samples in (cid:101) Z , and the remaining subsets contain the linearly spaced sequences in between.
We then train one NN on each subset and denote the average of the ﬁnal weights of these networks by µ 2 .
Finally, we choose Q W | (cid:101)
Z as a Gaussian distribution with mean µ 2 and covariance matrix σ 22 I d .
To select σ 2 , we proceed as follows.
First, we determine the largest real number ˜ σ 2 , again to some ﬁnite precision, for which the absolute value of the difference between the training loss of a NN with weights µ 2 and the empirical average of the training loss of 5 NNs with weights drawn from N ( W | µ 2 , ˜ σ 22 I d ) is below the selected threshold.
Note that this time the training loss is evaluated over the entire data set (cid:101)
Z , so that there is no dependence on S .
We then use ˜ σ 2 to form a set of 27 candidate values for σ 2 , from which we pick the one that results in the tightest bound on the test loss.
This procedure, the details of which are given in Appendix C.2, typically results in σ 2 = σ 1 .
Note that the prior and the posterior distribution satisfy the assumptions needed for the bounds (8), (9), (12) and (13) to hold.
Indeed, ( µ 1 , σ 1 ) depend on (cid:101)
Z only through Z ( S ) , while ( µ 2 , σ 2 ) are independent of S but depend on (cid:101) Z .
Equipped with these Gaussian distributions, we evaluate the bounds by noting that, for the chosen prior and posterior, the Radon-Nikodym derivatives in (9) and (13) reduce to likelihood ratios, and the KL divergences in (8) and (12) can be evaluated as
[Equation]
Since the MNIST and Fashion-MNIST data sets are ﬁxed and we are unable to draw several data sets from some underlying data distribution, we evaluate our bounds for these particular instances of (cid:101) Z .
We do however have control over S , so we run experiments for 10 instances of S and present the resulting mean as well as standard deviation.
Note that since we pick the training set uniformly at random from (cid:101)
Z , we implicitly randomize over the ordering of the elements of (cid:101) Z .
Our results are obtained by setting δ ≈ 0 .
001 as the conﬁdence parameter.
However, since the bounds are optimized over the choice of σ 2 , we need to use a union bound argument (Dziugaite & Roy, 2017; Dziugaite et al., 2020) to guarantee that the ﬁnal slow-rate and fast-rate bounds hold for all of these candidates simultaneously.
As a consequence, the bounds depicted in the ﬁgure hold with probability at least 95% .
The test loss and training loss are computed empirically by averaging the performance of 5 NNs whose weights are sampled from N ( W | µ 1 , σ 21 I d ) .
For the FCNNs, we use the notation W L to denote an architecture consisting of L hidden layers with width W , in addition to the required input and output layers.
For the case of CNNs, we consider the modiﬁed LeNetarchitecture used in (Zhou et al., 2019) and (Dziugaite et al., 2020).
Detailed descriptions of these architectures are provided in Appendix C.1.
In Figure 1, we study the dependence of the bounds on the number of training epochs.
The shaded areas around the curves correspond to two standard deviations.
The differences between the PACBayesian and the single-draw bounds turn out to be negligible, so we include only the PAC-Bayesian bounds in the ﬁgure.
In Figures 1a–d, we look at the early training epochs, while in Figures 1e–f, we train the networks until a small training loss (on the order of 0 .
001 ) is achieved. The networks are optimized using SGD either with or without momentum. Speciﬁcally, in Figures 1a–d, we use SGD without momentum, while in Figures 1e–f, we use SGD with momentum. More details about the training procedures are given in Appendix C.2.
As seen in Figures 1a–d, the bounds on the test loss are fairly accurate for both architectures, and both MNIST and Fashion-MNIST.
We also see that, as previously discussed, the relative ordering of the slow-rate bounds and the fast-rate bounds from a quantitative standpoint depends on the details of the learning setup.
In particular, higher values for the training loss and information measures tend to make the slow-rate bounds tighter due to their smaller constant factors. As a consequence, the fast-rate
[Figure] we perform training using SGD without momentum with a decaying learning rate. In (e)–(f), we use SGD with momentum and a ﬁxed learning rate. Further details on the experimental setup are given in Appendix C. bounds are tighter for the MNIST data set, for which low training loss and information measures are achieved, while the slow-rate bounds are superior for the more challenging Fashion-MNIST data set.
To shed light on the quantitative relationship between the results presented in Fig. 1a–d and previously obtained bounds, we compare our results on the estimated test loss with the ones reported in (Dziugaite et al., 2020), which established the best available PAC-Bayesian bounds for the settings considered in this section.
The approach used in Dziugaite et al.
(2020) is similar to the random-subset setting considered in this paper, in that the authors make use of a data-dependent prior.
The key differences with respect to the framework considered in this paper is that the posterior in Dziugaite et al.
(2020) is allowed to depend on the entire data set (cid:101)
Z , whereas the training loss and the prior depend on randomly selected disjoint subsets of (cid:101) Z .
In contrast, in the random-subset setting considered in this paper, the prior is allowed to depend on the entire data set (cid:101)
Z , whereas the training loss and the posterior depend only on a randomly selected portion of the data set.
For the case in which training is performed using SGD, the minimum test loss bounds (averaged over 50 runs) for MNIST reported in (Dziugaite et al., 2020, Fig. 4) are approximately 0 .
13 for
LeNet-5 and 0 .
18 for the 600 2 FCNN.
These values are similar to our best bounds, which are 0 .
15 for LeNet-5 and 0 .
19 for the 600 2 FCNN. For LeNet-5 trained on Fashion-MNIST, our tightest bound on the test loss is 0 . 35 , whereas the corresponding one in (Dziugaite et al., 2020, Fig. 4) is approximately 0 .
Taking error bars into account, our bounds are not clearly distinguishable from those reported in (Dziugaite et al., 2020, Fig. 4).
It is important to mention that signiﬁcantly tighter bounds are reported in (Dziugaite et al., 2020, Fig. 5) for the case in which the PAC-Bayesian bound considered therein is used as a regularizer during the training process.
Such a direct optimization of the bound does not appear to be feasible for the random-subset setting considered in this paper.
Next, we discuss the results presented in Figures 1e–f. As shown in the ﬁgure, while our bounds become tighter in the initial phase of training, they loose tightness as training progresses.
This is similar to what noted in (Dziugaite et al., 2020, p. 12) for the PAC-Bayesian bound considered therein.
Speciﬁcally, when the underlying deterministic NN therein is trained to achieve very low errors (or equivalently, is trained for many epochs), the PAC-Bayesian bound considered in Dziugaite et al.
(2020) become loose, and the corresponding stochastic NN yields a signiﬁcantly higher test error than the underlying deterministic NN.
Finally, the different behaviors of our bounds in Figure 1e and Figure 1f illustrates the role played by the variances σ 1 and σ 2 . In Figure 1e, we set the threshold used to determine σ 1 and σ 2 to 0 . 05 , which leads to small values for σ 1 and σ 2 . In Figure 1f we use instead a threshold of 0 . 20 , which allows for larger variances.
The results illustrate the intuitive fact that larger variances yield better generalization performance at the cost of a higher test error.
Further numerical experiments, in which we study how the bounds evolve as a function of the training-set size, and how they are affected by randomized labels, are reported in Appendix D.
C ONCLUSION
We have studied information-theoretic bounds on the test loss in the random-subset setting, in which the posterior and the training loss depend on a randomly selected subset of the available data set, and the prior is allowed to depend on the entire data set.
In particular, we derived new fast-rate bounds for the PAC-Bayesian and single-draw settings.
Provided that the information measures appearing in the bounds scale sublinearly with n , these fast-rate bounds have a better asymptotic dependence on n than the slow-rate PAC-Bayesian and single-draw bounds previously reported in (Hellstr¨om & Durisi, 2020b), at the price of larger multiplicative constants.
We also presented improvements on previously presented bounds on the average loss by using samplewise information measures and disintegration.
Through numerical experiments, we show that our novel fast-rate PAC-Bayesian bound, as well as its slow-rate counterpart, result in test-loss bounds for some overparameterized NNs trained through SGD that essentially match the best available bounds in the literature (Dziugaite et al., 2020).
Furthermore, the single-draw counterparts of these bounds, which are as tight as the PAC-Bayesian bounds, are applicable also to deterministic NNs trained through SGD and with Gaussian noise added to the ﬁnal weights.
These results demonstrate the value of the random-subset approach in studying the generalization capabilities of NNs, and show that fast-rate versions of the available information-theoretic bounds can be beneﬁcial in this setting.
In particular, the random-subset setting provides a natural way to select data-dependent priors, namely by marginalizing the learning algorithm P W | (cid:101) ZS over S , either exactly or approximately. Such data-dependent priors are a key element in obtaining tight information-theoretic generalization bounds (Dziugaite et al., 2020).
A P ROOFS A. P ROOF OF P ROPOSITION
Consider a ﬁxed hypothesis w and a supersample (cid:101) z .
Due to the boundedness of (cid:96) ( · , · ) , the random variable (cid:99) gen i ( w, (cid:101) z , S i ) = (cid:96) ( w, Z i ( ¯ S i )) − (cid:96) ( w, Z i ( S i )) is bounded to [ − 1 , 1] for i = 1 , . . . , n , and it has zero mean.
Subgaussianity then implies that the following holds for all λ > 0 :
[Equation]
Now, let E ( w, (cid:101) z ) ≡ supp ( P S i | w (cid:101) z ) where P S i | w (cid:101) z is shorthand for P S
i | W = w, (cid:101)
Z = (cid:101) z .
Then,
[Equation]
Through a change of measure from P S i to P S i | w (cid:101) z , we get (after reorganizing terms)
[Equation]
We now have a disintegrated, samplewise exponential inequality.
Next, we use Jensen’s inequality and then minimize over λ to ﬁnd that
[Equation]
We now use that E P Si | w
[Equation]
log d P Si [Equation] w
[Equation]
z P Si (cid:105) = D ( P S i | w (cid:101) z || P S i ) and then take the expectation with respect to P W (cid:101) Z to ﬁnd that
[Equation] [Equation]
The desired bound then follows because
[Equation]
A. P ROOF OF T HEOREM
The proof essentially mimics parts of the derivation of (Steinke & Zakynthinou, 2020, Thm. 2.(3)).
For convenience, we begin by proving an exponential inequality for a binary random variable X satisfying P ( X = a ) =
P ( X = b ) = 1 / 2 where a, b ∈ [0 , 1] .
Also, let ¯
X
= b if X = a and ¯ X
=
a
if X = b .
Finally, let λ, γ > 0 and c = e λ − 1 − λ .
Then,
[Equation]
Here, the ﬁrst inequality follows because e y ≤ 1 + y + cy 2 /λ 2 for all y ≤ λ .
Expanding the squares and removing negative terms, we ﬁnd that
[Equation]
In view of (10), we are interested in values of λ and γ such that λ (1 − γ ) +( e λ − 1 − λ ) · (1+ γ 2 ) ≤ 0 , so that the left-hand side of (26) is no larger than 1 .
Furthermore, it will turn out convenient to select pairs ( λ, γ ) so that λ is as large as possible and γ is as small as possible.
A possible choice is λ = 1 / 3 and γ = 2 .
Thus, we conclude that
[Equation]
Next, we will apply (27) with X = (cid:96) ( w, Z i ( ¯ S i )) and ¯ X = (cid:96) ( w, Z i ( S i )) for ﬁxed w and (cid:101) z .
Note that these random variables satisfy the required assumptions on X and ¯ X , since the loss function is supported on [0 , 1] and
the random variables S
i are Bernoulli distributed.
Let Q W (cid:101)
Z = Q W | (cid:101) Z P (cid:101)
Z .
It then follows that
[Equation]
[Equation] Now let E = supp ( P W (cid:101) ZS ) .
Then
[Equation]
[Equation] The desired result follows after a change of measure to P W (cid:101) ZS (Polyanskiy & Wu, 2019, Prop. 17.1).
A. P ROOF OF C OROLLARY
We begin by applying Jensen’s inequality to (10) to move the expectation inside the exponential.
We then obtain (11) by simply taking the logarithm of both sides and reorganizing terms.
To derive (12), we ﬁrst apply Jensen’s inequality in (10), this time only with respect only P W | (cid:101) ZS , to get
[Equation]
We now use Markov’s inequality in the following form.
Let U ∼ P
U be a nonnegative random variable satisfying E
[ U ] ≤ 1 .
Then,
[Equation]
Applying (31) to (30) we ﬁnd that, with probability at least 1 − δ under P (cid:101) ZS ,
[Equation]
Taking the logarithm and reorganizing terms, we obtain (12).
Finally, to derive (13), we apply (31) to (10) immediately to conclude that, with probability at least 1 − δ under P W (cid:101) ZS ,
[Equation]
The desired bound (13) follows after taking the logarithm and reorganizing terms.
A. P ROOF OF C OROLLARY
Consider a ﬁxed w and (cid:101) z .
As shown in Appendix A.2,
[Equation]
Let E = supp ( P S i | w (cid:101) z ) , where P S i | w (cid:101) z is short for P S
i | W = w, (cid:101)
Z = [Equation] z .
By changing measure we get
E P Si
[Equation]
Moving the expectation inside the exponential through the use of Jensen’s inequality and taking the logarithm [Equation] we get
[Equation]
The desired result now follows by noting that
[Equation]
and applying (37) to each term in the sum in (38) [Equation]
B F AST -R ATE B OUNDS FOR THE I NTERPOLATING C ASE
In this section [Equation] we discuss how to tighten the bound in Corollary 1 under the additional assumption that the training loss L Z ( S ) ( W ) is 0 for all W ∼ P W | Z ( S ) (interpolating assumption), and for the special case Q W | (cid:101)
Z = P W | (cid:101) Z .
We begin by proving the following exponential inequality, the derivation of which is similar to part of the proof of the realizable fast-rate bound from (Steinke & Zakynthinou, 2020).
Proposition 2.
Consider the setting of Theorem 1, with the additional assumption that L Z ( S ) ( W ) = 0 for all W ∼ P W | Z ( S ) and that Q W | (cid:101)
Z = P W | (cid:101) Z .
Then,
[Equation]
Proof.
Let λ, γ > 0 .
Furthermore, let S (cid:48) be independent of W , (cid:101) Z , and S , and distributed as S .
Then,
[Equation]
Let E = supp ( P W (cid:101) ZS )
[Equation]
It now follows from (41) that
[Equation]
We now change measure to P W (cid:101) ZS to conclude that [Equation]
[Equation]
We now use the interpolating assumption, set λ
= log 2 , and let γ → ∞ .
These steps, together with the assumption that (cid:96) ( W, Z i ( ¯ S i )) ∈
[0 , 1] , imply that the right-hand side of (44) is no larger than 1.
From this, the desired result follows.
Using Proposition 2, we can derive bounds that are analogous to those of Corollary 1.
We present these bounds below without proof, since they can be established following steps similar to the ones used to prove Corollary 1.
Corollary 3.
Consider the setting of Proposition 2.
Then, the average population loss is bounded by
[Equation]
Furthermore, with probability at least 1 − δ over P (cid:101) ZS , the PAC-Bayesian population loss is bounded by
[Equation]
Finally, with probability at least 1 − δ over P W (cid:101) ZS , the single-draw population loss is bounded by
[Equation]
Finally [Equation] we present a samplewise bound that tightens Corollary 2 under the interpolating assumption.
Its derivation is inspired by the techniques used to establish Proposition 1 and Proposition 2.
Corollary 4.
Consider the setting of Proposition 2.
Then, the average population loss is bounded by
[Equation]
Proof.
Let λ, γ > 0 and let S (cid:48)
i be independent of W , (cid:101) Z [Equation] and S
i
[Equation]
and distributed as S i .
Then, for all i , E
[Equation]
We now let E = supp ( P W (cid:101) Z S i ) .
It follows from (49)–(50) that
[Equation]
[Equation]
By performing a change of measure from P W (cid:101) Z P S i to P W (cid:101) Z S
i we conclude that
[Equation]
Here, ı ( W, S i | (cid:101) Z )
= log
[Equation]
Z P Si .
We now use the interpolating assumption, set λ
= log 2 , and let γ → ∞ .
These steps, together with the assumption that (cid:96) ( · , · ) ∈ [0 , 1] , imply that the right-hand side of (52) is no larger than 1.
Thus,
[Equation]
Next, we use Jensen’s inequality to move the expectation in (53) inside the exponential.
Taking the logarithm and reorganizing terms, we get
[Equation]
The result now follows because
[Equation]
C E XPERIMENT D ETAILS
Here, we provide a detailed description of the network architectures and training procedures considered in this paper.
We also note that, when evaluating the fast-rate bounds (12) and (13), we use the constants 1 . 975 and 2 . 98 in place of 2 and 3 , respectively.
This choice leads to valid bounds, as pointed out in Appendix A.2.
C. N ETWORK ARCHITECTURES
The LeNet-5 architecture used in the numerical results is described in Table 1.
This is different from most standard implementations of LeNet-5, but coincides with the architecture used by Zhou et al.
(2019) and Dziugaite et al.
It has 431 080 parameters.
The fully connected neural network denoted by 600 2 consists of an input layer with 784 units, 2 fully connected layers with 600 units and
ReLU activations, followed by an output layer with 10 units and softmax activations.
It has 837parameters.
C. T RAINING PROCEDURES
We now provide additional details on the training procedures described in Section 4.
The initial weights of all the networks used for each instance of Z ( S ) were set to the same randomly selected values drawn from a zero-mean normal distribution with standard deviation 0 .
All networks were trained using the cross-entropy loss, optimized using either SGD with momentum and a ﬁxed learning rate or SGD without momentum and a decaying learning rate.
First, we describe the details of SGD with momentum.
For MNIST, we used a learning rate of 0 . 003 , and for Fashion-MNIST, we used 0 . 001 .
In all experiments, the momentum parameter is set to 0 . We used a batch size of 512 . [Table] For SGD without momentum, we used a decaying learning rate schedule, where the learning rate α for a given epoch E is given by
[Equation]
Here, α 0 is the initial learning rate, γ is the decay rate, and E 0 is the number of epochs between each decay.
In all experiments, we used α 0 = 0 .
01 , γ = 2 , and E 0 = 20 .
Again, we used a batch size of 512 .
To choose σ 1 , we pick the largest value with one signiﬁcant digit (i.e., of the form a · 10 − b with a ∈ [1 : 9] and b ∈ Z ) such that the absolute value of the difference between the training loss on Z ( S ) of the deterministic network with weights µ 1 and empirical average of the training loss of 5 NNs with weights drawn independently from N ( W | µ 1 , σ 21 I d ) was no larger than some speciﬁed threshold. When producing the results reported in Figure 2 and Figures 1a–d, we used a threshold of 0 .for
MNIST, while for Fashion-MNIST, we used a threshold of 0 . 10 .
In Appendix D, we perform additional experiments with other thresholds.
Speciﬁcally, for Figure 1e, we use a threshold of 0.05, while we use a threshold of 0.20 for Figure 1f. For the randomized label experiment in Table 2, we use a threshold of 0.10.
To ﬁnd σ 2 , we use as starting point the same procedure as for determining σ 1 , but with µ 2 in place of µ 1 and the training loss evaluated on all of (cid:101) Z .
Let us call the value found by this procedure σ (cid:48) 2 = a (cid:48) · 10 − b (cid:48) .
Then, among the values of the form a · 10 − b with a ∈ [1 : 9] and b ∈ { b (cid:48) − 1 , b (cid:48) , b (cid:48) + 1 } , we choose σ 2 to be the one that minimizes the bound on the test loss.
In all our experiments, this procedure resulted in σ 2 = σ 1 .
To guarantee that the ﬁnal bound holds with a given conﬁdence level, all 27 bounds resulting from all possible choices of a and b need to hold with the same conﬁdence level.
Since we consider both slow-rate and fast-rate bounds, a total of 54 bounds need to hold simultaneously.
We ensure that this is the case via the union bound.
Thus, if each individual bound holds with probability at least 1 − δ , the optimized bounds hold with probability at least 1 − 54 δ .
We compute the bounds with δ = 0 .
05 / 54 , so the optimized bounds hold with 95% conﬁdence.
D A DDITIONAL E XPERIMENTS D. D EPENDENCE ON TRAINING SET SIZE
In this section, we study the dependence of the bounds on the size n of the training set.
We perform experiments for different values of n by restricting (cid:101)
Z to be a 2 n -dimensional randomly chosen subset of the set of 6 · 10 4 training samples available in MNIST and Fashion-MNIST.
The training set Z ( S ) is then formed by selecting n of these samples at random.
We then train a network, either LeNetor the 600 2 FCNN, on this restricted training set, until the training error is lower than some target error.
For MNIST, we use a target training error of 0 .
05 , while we use 0 .
15 for Fashion-MNIST.
The results are shown in Figure 2.
As seen in Figure 2, the bounds on the test loss for large values of n are fairly accurate for both of these architectures, especially so for MNIST.
We also see that, as previously discussed, the relative ordering of the slow-rate bounds and the fast-rate bounds from a quantitative standpoint depends on the details of the learning setup.
In particular, higher values for the training loss and information measures tend to make the slow-rate bounds tighter due to their smaller constant factors.
[Figure]
l a ss i ﬁ ca ti
[Figure]
[Table]
[Figure]
[Figure]
Training error
[Table]
[Table]
[Table]
D. R ANDOMIZED L ABELS
In order to assess the tightness of our bounds in an overﬁtting scenario, we consider data sets with partially randomized labels.
Speciﬁcally, we set the labels of a ﬁxed proportion of both the training and test sets of MNIST uniformly at random, and then perform training using SGD with momentum as described in Appendix C.
In order to simplify training with randomized labels, we consider a binarized version of MNIST where the digits 0 , . .
. , 4 are merged into one class and 5 , . . .
, 9 into another.
The results are shown in Table 2.
The slow-rate bound is computed using (8), while the fast-rate bound is based on (12).
As usual, the quantitative difference between these bounds and the corresponding single-draw bounds in (9) and (13) is negligible.
As shown in Table 2, our bounds become vacuous when randomized labels are used.
The fast-rate bound is signiﬁcantly worse than its slow-rate counterpart, which is to be expected: when the prior and posterior are optimized using randomized labels, a larger discrepancy between them arises.
This increases the value of the KL divergence in (8) and (12), which, as previously discussed, penalizes the fast-rate bound more.
We note, though, that the qualitative behavior of the bounds is in agreement with the empirically evaluated test error: an increased proportion of randomized labels, and thus an increased test error, is accompanied by an increase in the values of our bounds.
Furthermore, the slow-rate bound consistently overestimates the test error by a factor of approximately 25 .
To the best of our knowledge, all bounds available in the literature for overﬁtting situations such as the one considered in this section are vacuous.
The best result can be found in (Dziugaite & Roy, 2017, Tab. 1), where an FCNN with one hidden layer is trained on a binarized version of MNIST with fully randomized labels.
Despite directly optimizing the evaluated PAC-Bayesian bound as part of the training procedure, the obtained test error bound of 1 . 365 is vacuous.