PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization
Abstract
While there has been progress in developing non-vacuous generalization bounds for deep neural networks, these bounds tend to be uninformative about why deep learning works.
In this paper, we develop a compression approach based on quantizing neural network parameters in a linear subspace, profoundly improving on previous results to provide state-of-the-art generalization bounds on a variety of tasks, including transfer learning.
We use these tight bounds to better understand the role of model size, equivariance, and the implicit biases of optimization, for generalization in deep learning.
Notably, we find large models can be compressed to a much greater extent than previously known, encapsulating Occam’s razor
Introduction
Despite many more parameters than the number of training datapoints, deep learning models generalize extremely well and can even fit random labels [72].
These observations are not explained through classical statistical learning theory such as VC-dimension or Rademacher complexity which focus on uniform convergence over the hypothesis class [53].
The PAC-Bayes framework, in contrast, gives a convenient way of constructing generalization bounds where the generalization gap depends on the deep learning model found by training rather than the hypothesis set as a whole.
Using this framework, many different potential explanations have been proposed drawing on properties of a deep learning model that are induced by the training dataset, such as low spectral norm [57], noise stability [2], flat minima [30], derandomization [55], robustness, and compression [2, 73].
In this work, we show that neural networks, when paired with structured training datasets, are substantially more compressible than previously known.
Constructing tighter generalization bounds than have been previously achieved, we show that this compression alone is sufficient to explain much of the generalization of neural networks.
In particular:
We develop a new approach for training compressed neural networks that adapt the compressed size to the difficulty of the problem.
We train in a random linear subspace of the parameters [45] and perform learned quantization.
Consequently, we achieve extremely low compressed sizes for neural networks at a given accuracy level, which is essential for our tight bounds.
(See Section 4).
Using a prior encoding Occam’s razor and our compression scheme, we construct the best generalization bounds to date on image datasets, both with data-dependent and dataindependent priors.
We also show how transfer learning improves compression and thus ∗ Equal contribution.
our generalization bounds, explaining the practical performance benefits of pre-training.
(See Section 5).
PAC-Bayes bounds only constrain the adaptation of the prior to the posterior.
For bounds constructed with data-dependent priors, we show that the prior alone achieves performance comparable to the generalization bound.
Therefore we argue that bounds constructed from data-independent priors are more informative for understanding generalization.
(See Section 5.2).
Through the lens of compressibility, we are able to help explain why deep learning models generalize on structured datasets like CIFAR-10, but not when structure is broken such as by shuffling the pixels or shuffling the labels.
Similarly, we describe the benefits of equivariant models, e.g. why CNNs outperform MLPs.
Finally, we investigate double descent and whether the implicit regularization of SGD is necessary for generalization.
(See Section 6).
We emphasize that while we achieve state-of-the-art in both data-dependent bounds and dataindependent bounds through our newly developed compression approach, an important focus of our work is to leverage tighter bounds to understand generalization in neural networks.
Among others, Figure 1 highlights some of our observations regarding (a) data-dependent bounds, (b) how our method trades-off between data fit and model compression in relation to generalization, and (c) the explanation of several deep learning phenomena through model compressibility using our bounds.
All code to reproduce results is available here.
   
)UDFWLRQRIWKHGDWD      ( UU R U  % RXQG V
    ']LXJDLWHHWDO>
@ 3HUH]2UWL]HWDO>@ 3HUH]2UWL]HWDO>@ ']LXJDLWHHWDO>@ +RHIIGLQJ%RXQGRQ'DWD'HSHQGHQW3ULRUV'DWD'HSHQGHQW3$&%D\HV%RXQGV
[Figure]
Related Work
Optimizing the PAC-Bayes Bound.
Dziugaite and Roy [18] obtained the first non-vacuous generalization bounds for deep stochastic neural networks on binary MNIST.
The authors constructed a relaxation of the Langford and Seeger [41] bound and optimized it to find a posterior distribution that covers a large volume of low-loss solutions around a local minimum obtained using SGD.
Rivasplata et al.
[60] further extended the idea by developing novel relaxations
Model Compression and PAC-Bayes Bounds.
Noting the robustness of neural networks to small perturbations [29, 30, 40, 39, 37, 57, 8], Arora et al.
[2] developed a compression-based approach that uses noise stability.
Additionally they use the ability to reconstruct weight matrices with random
[Table]
projections to study generalization of neural networks.
Subsequently, Zhou et al.
[73] developed a
PAC-Bayes bound that uses the representation of a compressed model in bits
[Table]
and added noise stability through the use of Gaussian posteriors and Gaussian mixture priors.
Furthermore, they achieved even smaller model representations through pruning and quantization [27, 10].
In our work we follow a similar compression framing as Zhou et al.
[73] but with key improvements.
First, we train in a lower dimensional subspace using intrinsic dimensionality [45] and FiLM subspaces [58] which proves to be more effective and adaptable than pruning.
Second, we develop a more aggressive quantization scheme with variable length code and quantization aware training.
Finally, we exploit the increased compression provided by transfer learning and data-dependent priors.
Data-Dependent Priors.
Dziugaite et al.
[19] demonstrate that for linear PAC-Bayes bounds such as Thiemann et al.
[64], a tighter bound can be achieved by choosing the prior distribution to be data-dependent, i.e., the prior is trained to concentrate around low loss regions on held-out data.
More precisely, the authors show that the optimal data-dependent prior is the conditional expectation of the posterior given a subset of the training data.
They approximate this data-dependent prior by solving a variational problem over Gaussian distributions.
They evaluate the bounds for SGDtrained networks on data-dependent priors obtaining tight bounds on MNIST, Fashion MNIST, and
CIFAR-10.
In a similar vein, P ´ erez-Ortiz et al.
[59] combine data-dependent priors [19] with the
PAC-Bayes with Backprop (PBB) [60] to obtain state-of-the-art PAC-Bayes non-vacuous bounds for MNIST and CIFAR-10 using data-dependent priors.
Downstream Transferability.
Ding et al.
[16] investigate different correlates of generalization derived from PAC-Bayes bounds to predict the transferability of various upstream models; however, because of this different aim they do not actually compute the full bounds.
Our focus is to achieve better bounds in order to better understand generalization in deep neural networks.
For example, we investigate the effects of transfer learning, equivariance, and stochastic training on the bounds, and argue for the importance of data-independent bounds in explaining generalization.
We summarize improvements of our bounds relative to prior results in Table 1.
A Primer on PAC-Bayes Bounds
PAC-Bayes bounds are fundamentally an expression of Occam’s razor: simpler descriptions of the data generalize better.
As an illustration, consider the finite hypothesis generalization bound.
Let ˆ
R ( h ) = 1 n (cid:80) ni =1 ℓ ( h ( x i ) , y i ) be the empirical risk of a hypothesis h from a finite hypothesis set
H for the 0 - 1 loss ℓ , and R ( h ) =
E [ ˆ R ( h )] its population risk.
With probability at least 1 − δ , the population risk for a hypothesis h using n data samples satisfies
[Equation]
bounded by the empirical risk and a complexity term log |H| counting the number of bits needed to specify any hypothesis h ∈ H .
But what if we don’t believe that each hypothesis is equally likely?
If we consider that a given prior distribution P over H concentrates around the optimal hypothesis, then we can construct a variable length code that uses fewer bits for the more likely hypothesis.
With the best possible code for P , a given hypothesis h will take log 2 1 P ( h ) bits to represent.
Using this prior may only result in a smaller complexity term even if the hypothesis set is large, so long as the hypotheses that are consistent with the data are also likely under the prior.
The number of bits required can further be reduced from log 2 1 P ( h ) to KL ( Q ∥ P ) by considering a distribution of “good” solutions Q .
If we don’t care which element of Q we arrive at, we can gain some bits back from this insensitivity (which could be used to code a separate message).
The average number of bits to code a sample from Q using the prior P is the cross entropy H ( Q, P ) , and one gets on average the entropy H ( Q ) bits back from being agnostic about which sample h ∼ Q to use, yielding the KL-divergence between Q and P : H ( Q, P ) − H ( Q ) =
KL ( Q ∥ P ) .
With these improvements of the finite hypothesis bound — replacing log |H| with KL ( Q ∥ P ) , and sampling a hypothesis h ∈ H — one arrives (with minor bookkeeping) at the PAC-Bayes bound introduced in McAllester [51].
With probability at least 1 − δ ,
[Equation]
Many refinements of Eq.
(2) have been developed [41, 50, 7, 64] but retain the same character.
The lower the ratio of the KL-divergence to the number of data points n , the tighter the gap between empirical and expected risk.
In this work, we use the tighter Catoni [7] variant of the PAC-Bayes bound (see Appendix J for details).
Universal Prior.
Connecting back to Occam’s razor, one can define a prior that explicitly penalizes the minimum compressed length of the hypothesis, also known as the universal prior [62]:
[Equation]
2 − K ( h ) /Z , where K is the prefix Kolmogorov complexity [33] of h (the length of the shortest program that produces h and also delimits itself), and Z ≤ 1 .
2 Using a point mass posterior on a single hypothesis h ∗ , KL ( Q ∥ P ) from the bound reduces to
[Equation]
where l ( h ) is the length of a given program that reproduces h not including the delimiter, which we can use when upper bounding the KL.
For convenience, we can condition on using the same method for compression and decompression for all elements of the prior.
If we can reduce the compressed length l ( h ) of the hypothesis we find that fits the training data, then we can improve the tightness of the PAC-Bayes bound.
Model Compression.
Model compression aims to find a nearly equivalent model that can be expressed in a fewer bits and is useful for deployments on mobile devices or for improving inference time on specialized hardware [9, 10].
For computing PAC-Bayes generalization bounds, however, only the model size matters.
Therefore, we can employ compression methods which may otherwise be unfavorable in practice due to worse computational requirements.
Pruning and quantization are among the most widely used methods for model compression.
In this work, we rely on quantization (Section 4.2) to achieving tighter generalization bounds.
Tighter Generalization Bounds via Adaptive Subspace Compression
Training a neural network involves taking many gradient steps in a high-dimensional space R D .
Although D may be large, the loss landscape has been found to be simpler than typically believed [14, 26, 23].
Analogous to the notion of intrinsic dimensionality more generally, Li et al.
[45] attempt to find the lowest dimensional subspace in which the network can be trained and still fit the training data.
The weights of a neural network θ ∈ R D are parametrized in terms of an initialization θ 0 and a projection w ∈ R d to a lower dimensional subspace through a fixed matrix
[Equation]
[Equation]
[Figure]
In its original form intrinsic dimensionality (ID) is only used as a scientific tool to measure the complexity of the learning task. Unlike methods like pruning, intrinsic dimension [45] scales with complexity of the task — more complex tasks require a larger intrinsic dimension.
Subsequently, we find that ID combined with quantization can serve as an effective model compression method.
We note that ideas similar to the intrinsic dimensionality of a model have been explored in the context of model compression for estimating bounds [2].
For our work, the ability to find the intrinsic dimension d ≪ D has profound implications for the compressibility of the models, and therefore our ability to construct generalization bounds.
As demonstrated by Zhou et al.
[73], the compressibility of a neural network has a direct connection to generalization and allows us to compute non-vacuous
PAC-Bayes bounds for transfer learning.
Building upon ID, our key building blocks to tight generalization bounds are composed of (i) a new scalable method to train an intrinsic dimensionality neural network parameterized by Eq.
(3) (Section 4.1), and (ii) a new approach to simultaneously train both the quantized neural network weights and the quantization levels for maximum compression (Section 4.2).
Our complete method is summarized in Algorithm 1.
4.1 Finding Better Random Subspaces
To further improve upon the scalability and effectiveness of the projections P used by Li et al.
[45], we introduce three novel methods.
Kronecker Sum Projector :
Using the Kronecker product ⊗ , we construct the matrix
P ⊕ =
( 1 ⊗ R 1 + R 2 ⊗ 1 ) / √ 2 D where R 1 , R 2 ∼ N (0 , 1) √ D × d and 1 is the vector of all ones in R √ D .
From the fact that R 1 ⊥⊥ R 2 and the entries are standard normal, one can verify that
P ⊤⊕ P ⊕ =
I
d × d + O (1 / √ D ) .
Kronecker Product Projector : Alternatively, we can form the matrix P ⊗ = Q 1 ⊗ Q 2 / √ D with the smaller Q 1 , Q 2 ∼ N (0 , 1) √ D ×√ d , and again this matrix is approximately orthogonal: P ⊤⊗ P ⊗ =
( Q ⊤ 1 Q 1 / √ D ) ⊗ ( Q ⊤ 2 Q 2 / √ D )
=
I ⊗ I
+ O ( D − 1 / 4 )
=
I d × d + O ( D − 1 / 4 ) .
The matrix vector multiply w (cid:55)→ Pw for both the above projectors can be performed in time
O ( d √ D ) and O ( √ dD ) respectively, rather than the O ( dD ) that is required by the dense random matrix.
Figure 2 demonstrates the runtime speedup and training performance improvement in com- parison to the methods using by Li et al.
Notably, the Kronecker-structured projections retain the fidelity of the dense random matrix while being orders of magnitude faster than the alternative operators when scaling to larger values of d .
FiLM projector : BatchNorm parameters have an outsized effect on the downstream task performance relative to their size, and this fact has been used in Featurewise independent Linear Modulation (FiLM)
[58, 17] for efficient control of neural networks in many different settings.
Several authors have explored performing fine-tuning for transfer learning solely on these parameters and the final linear layer [36].
Drawing on these observations, we construct a projection matrix P FiLM where only columns corresponding with BatchNorm or head parameters are nonzero and sampled from N (0 , 1) d / √ D , which we also show in Figure 2.
While the FiLM projector is highly effective for transfer learning (shown in Figure 2 left), the performance saturates quickly when training from scratch.
For this reason, when training from scratch we employ the sum
P FiLM+ ⊗ =
( P FiLM + P ⊗ ) / √ 2 , which outperforms the two projectors individually as shown in
Figure 2 right.
4.2 Quantization Scheme and Training
Through quantization, the average number of bits used per parameter can be substantially reduced.
When optimizing purely for model size rather than efficiency on specialized hardware, one can choose non-linearly spaced quantization levels which are learned, and use variable length coding schemes as shown in Han et al.
Additionally, the straight through estimator has been central to learning weights in binary neural networks [31].
We combine these ideas to simultaneously optimize the quantized weights and the quantization levels for maximum compression.
Given the full precision weights w =
[ w 1 , . .
w d ] ∈ R d and a vector c =
[ c 1 , . . .
c L ] ∈ R L of L quantization levels, we construct the quantized vector ˆ w =
[ ˆ w 1 , . .
, ˆ w d ] such that ˆ w i =
c q ( i ) where q ( i ) =
arg min k | w i − c k | .
The quantization levels c are learned alongside w , where the gradients are defined using the straight through estimator [3, 70]:
[Equation]
We initialize c with uniform spacing between the minimum and maximum values in parameter vector w or k-means [11].
To further compress the network, we use a variable length code in the form of arithmetic coding [49], which takes advantage of the fact that certain quantization levels are more likely than others.
Given probabilities p k (empirical fractions) for cluster c k , arithmetic coding of w takes at most ⌈ d × H ( p ) ⌉ + 2 bits, where H ( p ) is the entropy H ( p ) =
− (cid:80) k
p k log 2 p k .
For a small number of quantization levels, we find it makes a small improvement in the compression over more common Huffman coding.
In summary, we use ⌈ d × H ( p ) ⌉ + 2 bits for coding the quantized weights ˆ w , 16 L bits for the codebook c (represented in half precision), and additional L × ⌈ log 2 d ⌉ bits for representing the probabilities p k , arriving at l ( w ) ≤ ⌈ d × H ( p )
+ L × (16+ ⌈ log 2 d ⌉ )+2 .
As we show in Appendix B, we can optimize over the subspace dimension d and the number of quantization levels L and any other hyperparameters, by including them in the compressed description of our model, contributing only a few extra bits.
4.3 Transfer Learning
For transfer learning, we replace θ 0 with a learned initialization θ D that is found using the pretraining task and data D .
With the ID compression, the universal prior P ( h | D ) ∝ 2 − K ( h |D ) will place higher likelihood on solutions θ that are close to the pre-training solution θ D .
Empirical Non-Vacuous Bounds 5.1 Non-Vacuous PAC-Bayes Bounds
To compute our bounds on benchmark datasets, we use the joint approach of projected subspace training, parameter quantization, and optimizing the subspace dimension with respect to the bounds
1: Inputs: Neural network f θ , Training dataset { x i , y i } ni =1 , Clusters L , Intrinsic dimension d ,
Confidence 1 − δ , and Prior distribution P .
2: function COMPUTE BOUND ( f θ , L, d, ( x i , y i ) ni =1 , δ, P ) 3: w ← TRAIN ID ( f θ , d , ( x i , y i ) ni =1 ) ▷ (Section 4.1) 4: ˆ w ← TRAIN QUANTIZE ( w , L , ( x i , y i ) ni =1 ) 5:
Compute quantized train error ˆ R ( ˆ w ) .
KL ( Q, P ) ← GET KL ( ˆ w, P )
▷ (Section 3) 7: return GET CATONI BOUND
( ˆ R ( ˆ w ) , KL ( Q, P ) , δ , n ) ▷ (Section 3) 8: end function 9: function TRAIN QUANTIZE ( w, L, ( x i , y i )
ni =1 )
▷ (Section 4.2) 10:
Initialize c ← GET CLUSTERS ( w, L ) 11: for i = 1 to quant epochs do 12: c ← c − ρ ∇ c L ( w, c ) and w ← w − ρ ∇ w L ( w, c ) 13: end for 14: return ˆ w 15: end function 16: function GET KL ( ( ˆ w, P ) ) 17: c , count ← GET UNIQUE VALS COUNTS ( ˆ w ) 18: message size ← DO ARITHMETIC ENCODING ( ˆ w , c , count ) 19: message size ← message size + hyperparam search ▷ (Appendix B) 20: return message size + 2 × log ( message size ) 21: end function
[Table]
as summarized in Algorithm 1.
We also optimize other bound hyperparameters such as the learning rate of the quantization-aware training.
We offer more details about bound optimization in Appendix
E. Our results for data-independent priors, i.e., priors that are not pre-trained on a subset of the training data, are shown in Table 2.
We derive the first non-vacuous bounds on FashionMNIST, CIFAR-10, and CIFAR-100 without data-dependent priors.
These results are of significance as we argue in Section 5.2 that using data-dependent priors are not explanatory about the learning process.
In particular, we improve over the compression bound results obtained by Zhou et al.
[73] on MNIST from 46% to 11 .
55% and on ImageNet from 96 .
5 to 94 .
In terms of compression, we dramatically improve the rates as we reduce the compressed size for the best MNIST bound by 94% bringing it down from 6 .
KB to 0 .
38 KB with LeNet5 and, on ImageNet, by 87% bringing it down from 358 KB to 46 .
3 KB with MobileViT. Since we perform transfer learning with an ImageNettrained checkpoint, we omit transfer learning experiments on the ImageNet (downstream) dataset.
The tightness of our SOTA subspace compression bounds allows us to improve the understanding of several deep learning phenomena as discussed in Section 6.
See Appendix A for additional results.
5.2 Data-Dependent PAC-Bayes Bounds
So far, we demonstrated the strength of our bounds on data-independent priors, where we considerably improve on the state-of-the-art.
However, a number of recent papers have considered data-dependent priors as a way of achieving tighter bounds [59, 19].
In this setup, the training data
D = { ( x i , y i ) } ni =1 is partitioned into two parts, D a and D b , with length n − m and m .
The first dataset is used to construct a data-dependent prior P ( h | D a ) , and then the bound is formed over the remaining part of the process: the adaptation of the prior P ( h | D a ) to the posterior Q ( h ) using the data D b .
The empirical risk is computed over D b only.
Intuitively, using dataset D a it is possible to construct a much tighter prior over the possible neural network solutions.
In our setting, similar to transfer learning, we use the prior P D a ( θ ) = 2 − K
( θ | θ D a ) /Z
where for compression we use θ = θ D a + Pw , and θ D a is the solution found by training the model (without random projections) on the data D a rather than initializing randomly.
With these data-dependent priors, we achieve the best bounds in Table 2.
However, our adaptive approach exposes a significant downside of data-dependent priors.
To the extent that PAC-Bayes bounds can be used for explanation, data-dependent bounds only yield insights into the procedure used to adapt the prior P D a ( θ ) to the posterior Q using D b : any learning that is done in finding P D a ( θ ) is not constrained or explained by the bound (we have no information on the compressibility of θ D a ).
Given the ability to adapt the size of the KL to the difficulty of the problem, it is possible to squeeze all of the learning into P D a ( θ ) and none in this adaption to Q .
This happens as the KL → 0 , which we find happens empirically (or very nearly so) for many splits of the data where n − m is large.
Setting Q ( θ ) =
1 [ θ = θ D a ] , the KL has only the contribution from the optimization over d : KL ( Q || P D a ) ≤ log D .
We find that the bound is nothing more than a variant of the simple Hoeffding bound where D b is the validation set R ( θ D a ) ≤
ˆ R D b ( θ D a ) + (cid:113) log( Dm/δ )
+2 2 m −.
We can see this phenomenon in Figure 1(a) where we compare existing data-dependent bounds to the simple Hoeffding bound applied directly to the data-dependent prior which was trained on only a small fraction of the data.
We can consider the Hoeffding bound as the simplest data-dependent bound without any fine-tuning so that the prior , a single pre-trained checkpoint, is directly evaluated on held-out validation data with no KL-divergence term.
If another data-dependent bound cannot achieve significantly stronger guarantees than the prior Hoeffding bound, then it only explains that neural networks generalize because the priors already have low validation error which is no explanation for generalization at all.
Indeed, we see in Figure 1 that the strength of existing datadependent bounds relies almost entirely on the a priori properties of the data-dependent prior rather than constraining the actual learning process through compressibility.
Similarly, from a minimum description length (MDL) perspective, data-independent bounds can be used to provide a lossless compression of the training data, whereas data-dependent bounds cannot (see Appendix H).
We also note that with data-dependent priors, optimization over the subspace dimension selects very low dimensionality, even if the data does not have low intrinsic dimension.
Because most of the data fitting is moved into fitting the prior, the bound selects a low complexity solution with respect to the prior without hurting data fit by choosing a low subspace dimensionality (Appendix D).
In contrast, data-independent bounds explain generalization for the entirety of the learning process.
Similarly, our transfer learning bounds meaningfully constrain what happens in the fine-tuning on the downstream task, but they do not constrain the prior determined from the upstream task.
5.3 Non-Vacuous PAC-Bayes Bounds for Transfer Learning
By directly interpreting PAC-Bayes bounds through the lens of compression, we immediately see the benefits of employing an upstream dataset for transfer learning.
Transfer learning allows us to constrain the prior around parameters consistent with the upstream task, and hopefully the downstream task as well, reducing the KL-divergence between the prior and the posterior and leading to even tighter bounds as we show in Table 2.
Our tighter data-independent transfer learning bounds provide a theoretical certification that transfer learning can improve generalization.
Our PAC-Bayes transfer learning approach also indicates that transfer learning can boost generalization whenever codings optimized on a pre-training task are more efficient for encoding a downstream posterior than an a priori guess made before seeing data.
In contrast, downstream tasks which greatly differ from the upstream task may only be consistent with models that are not compressible under the learned prior, a scenario that describes negative transfer.
See Appendix C for more details.
Understanding Generalization through PAC-Bayes Bounds
The classical learning theory viewpoint of uniform convergence focuses on properties of the hypothesis class as a whole, such as its size.
In contrast, the PAC-Bayes perspective shows that the ability to generalize is not merely a result of the hypothesis class but also results from the exact dataset at hand and the individual functions in the hypothesis class which we fit to that dataset.
After all, many elements of our hypothesis class are not compressible, yet in order to guarantee generalization, we need to choose the ones that are.
Real datasets actually contain a tremendous amount of structure, or else we could not learn from them as famously argued by Hume [32] and No Free Lunch theorems [67, 25].
This high degree of structure in real-world datasets is reflected in the compressibility of the functions (i.e. neural networks) we find in our hypothesis class which fit them.
In this section, we examine exactly how dataset structure manifests in compressible models by applying our generalization bounds, and we see what happens when this structure is broken, for example by shuffling pixels or fitting random labels.
Corrupting the dataset degrades both compressibility and generalization.
[Figure]
MLPs vs CNNs: It is well known that convolutional neural networks (CNNs) generalize much better than standard multilayer perceptrons (MLPs) with alternating fully-connected layers and activation functions on image classification problems, even when controlling for the number of parameters.
We find that this fact is reflected in the improved compressibility of CNNs and consequently our generalization bounds, shown in Figure 3 (left) on the image dataset CIFAR-10 using the same number of parameters for the two models.
The CNN is able find a more structured and lower description length explanation of the training data.
Shuffled Pixels:
However, when this image structure is broken by shuffling the pixels, we find that
CNNs are no better at generalizing than MLPs, and this is reflected in their compressed sizes.
The
CNNs become substantially less compressible when trained on this dataset, and hence our bounds show them generalizing worse than MLPs, see Figure 3 (center).
MLPs do not suffer when this structure is broken since they never used it in the first place.
Shuffled Labels: When the structure of the dataset is entirely broken by shuffling the labels, the compressibility of the models (both for CNNs and MLPs) which fit the random labels is lost.
Regardless of the subspace dimension used, our generalization bounds are all at 100% error as shown in Figure 3 (right).
It is not possible to fit the training data using low subspace dimensions, and when using a large enough dimension to fit the data, the compressed size of the model is larger than the training data and hence the generalization bounds are vacuous.
Equivariance: Designing models which are equivariant to certain symmetry transformations has been a guiding principle for the development of data-efficient neural networks in many domains [12, 13, 65, 66, 22, 35].
While intuitively it is clear that respecting dataset symmetries severely improves generalization, relatively little has been proven for neural networks [48, 21, 74, 4, 20].
We compress and evaluate rotationally equivariant ( C 8 ) and non-equivariant Wide ResNets [66, 71] trained on MNIST and a rotated version of MNIST.
As shown in Figure 1 (c), the rotationally equivariant models are more compressible and provably generalize better than their non-equivariant counterparts when paired with a dataset that also has the rotational symmetry.
We expand further on these experiments in Appendix F.
Is Stochasticity Necessary for Generalization?
It is widely hypothesized that the implicit biases of
SGD help find solutions which generalize better.
For example, Arora et al.
[1] argue that there is no regularizer that replicates the benefits of gradient noise.
Wu et al.
[68], Smith et al.
[61], and Li et al.
[47] advocate that gradient noise is necessary to achieve state-of-the-art performance.
In contrast, recent work by Geiping et al.
[24] shows that full-batch gradient descent can match state-of-theart performance, and Izmailov et al.
[34] shows that full-batch Hamiltonian Monte Carlo sampling generalizes significantly better than mini-batch MCMC and stochastic optimization.
We train ResNet-18 and LeNet5 models on CIFAR-10 and MNIST, respectively, using full-batch and stochastic gradient descent with different intrinsic dimensionalities.
We provide the training details in Appendix G, but it is worth noting that, unlike Geiping et al.
[24], we do not use flatness-seeking regularization for full-batch training.
For MNIST with LeNet5, the best generalization bounds that we obtain are 11 .
55% and 11 .
20% using stochastic gradient descent (SGD) and full-batch training respectively.
The best generalization bounds that we obtain for CIFAR-10 with ResNet-18 are 74 .
68% and 75 .
3% using SGD and full-batch training respectively.
We also extend this analysis to
SVHN to MNIST transfer learning with LeNet5 and obtain PAC-Bayes bounds of 9 . 0% and 8 .
7% using SGD and full-batch training respectively.
These close theoretical guarantees on the generalization error for both SGD and full-batch training suggest that while the implicit biases of SGD may be helpful, they are not at all necessary for understanding why neural networks generalize well.
We expand this analysis in Appendix G.
Double Descent: Our bounds are also tight enough to predict the double descent phenomenon noted in Nakkiran et al.
See Appendix I for a depiction of these experiments and a discussion of their significance.
Discussion
In this work, we have constructed a new method for compressing deep learning models which is highly adaptive to the level of structure in the model and the training dataset.
According to the
Occam’s prior, which considers shorter compressed length models to be more likely, we construct state-of-the-art generalization bounds across a variety of settings.
Through our bounds and the viewpoint of Occam’s razor and compressibility, we show how generalization relates to the structure in the dataset and the structure in the model, and we are able to explain aspects of neural network generalization for natural image datasets, shuffled pixels, shuffled labels, equivariant models, and stochastic training.
Limitations.
Despite the power of our compression scheme and our bounds, we are scratching the surface of explaining generalization.
Our compression bounds prefer models with a smaller number of base parameters as shown in Appendix H, instead of larger models which actually tend to generalize better.
While we have achieved better model compression than previous works, it is unlikely that we are close to theoretical limits.
We hope that with an improved compression scheme (possibly using nonlinear parameter projections), we might find that larger deep learning model solutions are compressible to smaller sizes than smaller models.
There is also a question of whether normally trained models can be related and bounded by the performance of the compressed models, perhaps leveraging ideas from Nagarajan and Kolter [52] and others investigating this question from a theoretical lens.
Additionally, while our bounds show that the compressibility of our models implies generalization, we make no claims about the reverse direction.
However, we believe that model compression and Occam’s razor have yet untapped explanatory power in deep learning.
Acknowledgements
We thank Pavel Izmailov and Nate Gruver for helpful discussions.
This research is supported by NSF
CAREER IIS-2145492, NSF I-DISRE 193471, NIH R01DA048764-01A1, NSF IIS-1910266, NSF 1922658 NRT-HDR, Meta Core Data Science, Google AI Research, BigHat Biosciences, Capital
One, and an Amazon Research Award.
This work is also supported in part through the NYU IT
High Performance Computing resources, services, and staff expertise.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We provide evidence for the claims made in the abstract and introduction in Section 4, Section 5.1, and Section 6.
(b) Did you describe the limitations of your work?
[Yes] We describe limitations in Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix J. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix J. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See Appendix K. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Our code is submitted in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplementary Material for
PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization
Appendix Outline
The appendix is organized as follows.
• In Appendix A, we report results for additional bounds for SVHN and ImageNet.
We also report the compression size corresponding to our best bound values and compare it to the compression size obtained through standard pruning.
Furthermore, in Appendix A.1 we prove why models cannot both be compressible and fit random labels.
• In Appendix B, we describe how optimization over hyperparameters like the intrinsic dimension impact the PAC-Bayes bound • In Appendix C, we show how our PAC-Bayes bound benefit from transfer learning.
• In Appendix D, we discuss data-dependent priors and their effect on the subspace dimension optimization.
• In Appendix E, we detail our experimental setup including models, datasets, and hyperparameter settings for training and bound computation.
• In Appendix F, we provide a compression perspective to why equivariant models may be more desirable for generalization.
• In Appendix G, we further discuss how our through our PAC-Bayes compression bounds, we provide evidence that SGD is not necessary for generalization.
• In Appendix H, we ablate the model size and show how it impacts our bounds and compressibility, we identify the best performing size of models for our bounds.
• In Appendix I, we present our observations on double descent and their preditability from our PAC-Bayes bounds.
• In Appendix J, we expand our theoretical discussion and emphasize conceptual differences between our method and previous ones in the literature.
• Lastly, in Appendix K we provide licensing information on the datasets we use.
A Additional Results
In addition to the results reported in Table 2, we report the best bounds for SVHN and ImageNet-1k as well as the corresponding compressed size in Tables 3 and 4.
In Table 3 we show how compressing the model via intrinsic dimension (ID) yields better results than standard pruning.
In this table, we basically run our method but substitute ID with pruning and then proceed by quantizing the remaining weights and encoding them through arithmetic encoding.
When pruning we used the standard iterative procedure following Han et al.
[27], for the MNIST model we pruned 98.8% of the weights, for the FMNIST model 97.0% of the weights, for the SVHN model 98.8% of the weights and for both the CIFAR-10 and CIFAR-100 models we pruned 52.1% of the weights and stopped there as the accuracy dropped significantly if we kept pruning.
Error bars on our bounds: We re-run the bounds computation for 10 times and observe that the values are consistent.
On average, we obtain ± 0 .
5% variation in our bounds for models trained from scratch and ± 0 .
1% variation for transfer learning models.
[Table]
[Table]
A.1 Models that can fit random labels cannot be compressed
Our ability to construct nonvacuous generalization bounds rests on the ability to construct models which both fit the training data and are highly compressible. However, when the structure in the dataset has been completely destroyed by shuffling the labels, then we do not find that our models are compressible (shown in Figure 3 right).
This is not just an empirical fact, but one that can be proven apriori: models which fit random labels cannot be compressed.
While this result is a trivial consequence of complexity theory, we present an argument here for illustration.
Almost all random datasets are incompressible
When sampling labels uniformly at random, almost all datasets are not substantially compressible.
Given a dataset D = { x i , y i } n i =1 (where we are only considering the labels y i , and conditioning on the inputs x i ), and denoting |D| as the length of the string of labels, the probability that a given dataset can be compressed to size |D|− c is less than 2 − c +1 .
To see this, one must consider that there are only (cid:80)
|D|− c
i = 2 i ≤ 2 |D|− c +1 programs of length ≤ |D| − c (fewer still when restricting to self delimiting programs), and there are 2 |D| possible datasets.
Therefore averaging over all randomly labeled datasets the fraction which are compressible to less than or equal to |D| − c bits is at most 2 |D|− c +1 / 2 |D| = 2 − c +1 .
A compressible model which fits the data is a compression of the dataset
Let prior P that includes a specification of the model architecture, and the model h which outputs probabilities for each of the outcomes: p ( y = k
| x i )
=
h ( x i ) k .
We can decompose the (prefix)
Kolmogorov complexity of the dataset (given the prior) as
[Equation]
The term K ( D | h, P ) can be interpreted as a model fit term and upper bounded by the total negative log likelihood simply using the model probabilities as a distribution to encode the labels:
K ( D | h, P ) ≤
[Equation]
(cid:80)
[Table]
[Table]
[Table]
Using the fact that almost all random datasets are incompressible, and choosing c = 1 + log 2 (1 /δ ) , we have that with probability at least 1 − δ over all randomly sampled datasets K ( D | P ) > |D| − log 2 (1 /δ )
− 1 .
Plugging into Eq.
(5) and rearranging, we have with probability 1 − δ ,
[Equation]
In Figure 6 we plot the quantity K ( h | P ) + NLL
( D | h ) which represents the compressed size of the dataset achieved by our model (related to the minimum description length principle).
We see that the value is considerably lower than the size of the dataset |D| , emphasizing that real machine learning datasets such as CIFAR-10 have a very low Kolmogorov complexity and are very unlike those with random labels.
B Subspace Dimension Optimization and Hyperparameters in the Universal Prior
The smaller the chosen intrinsic dimension d , the more similar θ is to the initialization θ 0 in Eq.
Consequently, that value of θ is more likely under the universal prior given the shorter description length.
Note that in this prior, we condition on the random seed used to generate θ 0 and P .
As we optimize over different parameters such as the subspace dimension d = 1 , .., D , and possibly other hyperparameters such as the learning rate, or number of quantization levels L , we must encode these into our prior and thus pay a penalty for optimizing over them.
We can accomodate this very simply by considering the hypothesis h as not just specifying the weights, but also specifying these hyperparameters: h = ( θ, d, L, lr) , and therefore using the universal prior P ( h ) = 2 − K ( h ) /Z
we pay additional bits for each of these quantities: K ( h ) ≤ K ( θ | d, L )
+ K ( d ) + K ( L ) + K
(lr) .
If we optimize over a fixed number H of distinct values known in advance for a given hyperparameter such as L , then we can code L using this information in only log 2 ( H ) bits.
In general, we can also bound the dimensionalities searched over by the maximum D so that K ( d ) ≤ ⌈ log 2 D
in any case.
C Transfer Learning Bounds
We show the expanded results both with and without transfer learning in Table 3.
When finetuning from ImageNet we use the larger EfficientNet-B0 models rather than the small convnet.
Despite the fact that the model is significantly larger than the convnet or resnet models that we use to achieve the best bounds for from scratch training, the difference between the finetuned and pretrained models is highly compressible.
D Data Dependent Priors
We observe that when using data dependent priors, our optimization over the subspace dimension (and the complexity of the model used to fit the data when measured against the prior) favors very low dimensions and low KL values which we show empirically in Figure 4.
Indeed, a large fraction of the data fitting is moved into fitting a good prior, particularly when the dataset fraction used to train the prior is large.
When the prior is already fitted on the data, the final solution can have a very low complexity with respect to that prior without affecting data fit, and is encouraged to do so.
E Experimental Details
In this section we provide experimental details to reproduce our results.
E.1 Model Training Details
We use a standard small convolutional architecture for our experiments, which we find produces better bounds than its ResNet counterparts.
The architecture is detailed in Table 5, and we use k = 16 for experiments, but this value is ablated in Figure 6.
[Figure]
Stochastic training: All models were trained for 500 epochs using the Adam optimizer with learning rate 0 .
001 , except for ImageNet which was trained for 80 epochs with SGD using learning rate of 0 .
05 and weight decay of 0 . 00002 .
The model architectures for each dataset are listed below:
• MNIST [44] ( + SVHN [56] Transfer): LeNet-5 [44].
• FashionMNIST
69] ( + CIFAR-10 [38] Transfer): ResNet20 [28].
• SVHN: ConvNet (Table 5).
• SVHN + ImageNet Transfer: EfficientNet-B0 [63].
• CIFAR-10: ConvNet.
• CIFAR-10 + ImageNet Transfer: EfficientNet-B0 [63].
• CIFAR-100 [38]: ConvNet.
• CIFAR-100 + ImageNet [15] Transfer: EfficientNet-B0.
[Table]
Conv( 3 , k ), BN,
[Table]
Conv(
[Table]
[Table]
[Table]
Conv [Table]
MaxPool2d(2)
Conv( 2 k , 2 k ), BN,
[Table]
Conv( 2 k , 2 k ), BN,
[Table]
Conv( 2 k , 2 k ), BN, ReLU
MaxPool2d(2)
Conv( 2 k , 2 k ), BN, ReLU
Conv( 2 k , 2 k ), BN, ReLU
Conv(
[Table] [Table]
), BN, ReLU
GlobalAveragePool2dLinear
[Equation]
Full-batch training: We train all models for 3000 epochs, use learning rates equal to 0 .
1 (MNIST + LeNet-5 and CIFAR-10 + ResNet-18) and 0 .
5 (CIFAR-10 + ConvNet), and a cosine learning rate scheduler that we warm-up for 10 epochs.
We also clip the full gradient to have an L 2 -norm of at most 0 .
before performing parameter updates in each epoch [24].
Transfer Learning All previous training details remain the same, except that θ 0 from Eq.
(3) is initialized from a pre-trained checkpoint instead of a random initialization.
As typically done in literature, the final classification layer is replaced with a randomly initialized fully-connected layer to account for the number of classes in the downstream task.
E.2 Bound Hyperparameter Optimization
As explained in Appendix B, we optimize the bound hyperparameters by considering that the hypothesis of interest h specifies the hyperparameters in addition to the weights.
Therefore, we pay bits back for the combination of hyperparemeters that we select.
For example, if we are doing a grid search over 2 values of the quantization-aware training learning rate, 2 values of the intrinsic dimensionality values, 2 values of the quantization levels, and use k-means by default, then the number of bits that we pay is log 2 (2 × 2 × 2)
=
3 bits.
Optimizing PAC-Bayes bounds for data-independent priors: Our PAC-Bayesian subspace compression bounds for data-independent priors have 4 hyperparameters that we list here-under alongside the possible values that we consider for each hyperparameter:
• The learning rate for the quantization-aware training, possible values: { 0 . 001 , 0 . 003 , 0 .
• The intrinsic dimensionality, possible values: { 0 , 1000 , 2500 , 3000 , 3500 , 4000 , 5000 , 7500 , 8000 , 10000 , 12000 , 15000 , 20000 , 25000 , 50000 , 100000 , 250000 , 500000 } , except for the ImageNet transfer learning which was conducted on the more limited range: { 500 , 1000 , 2000 , 3000 , 4000 , 6000 , 8000 } •
The number of quantization levels, possible values: { 0 , 7 , 11 , 30 , 50 } .
• The quantization initialization, possible values: { uniform , k-means } .
Note that we only use a subset of these hyperparameter values for some datasets, depending on the dataset size and other considerations.
For all bound computations, we use arithmetic encoding and 30 epochs of quantization-aware training.
In Table 6, we summarize the hyperparameters corresponding to the data-independent bounds that we report in Table 2.
transfer learning with data-independent priors .
All bound results are obtained with 95% confidence,
i
[Table]
e. δ
= .
[Table]
Optimizing PAC-Bayes bounds for data-dependent priors:
In addition to the hyperparameters listed above
[Table]
we also tune the hyperparameter corresponding to the subset of the training dataset that we use to train the prior on.
We consider the following values for the subset of the training dataset:
In Table
[Table]
we summarize the the hyperparameters corresponding to the data-dependent bounds that we report in Table 2.
The best bounds are obtained for intrinsic dimensionality equal to 0 , therefore no quantization is performed.
E.3 Computational Infrastructure & Resources
Our computational hardware involved a mix of NVIDIA GeForce RTX 2080 Ti (12GB), NVIDIA
TITAN RTX (24GB), NVIDIA V100 (32GB)
[Table]
and NVIDIA RTX
(48GB).
The experiments were managed via W&B [5].
The total computational cost of all experiments (including the ones that do not appear in this work) amounts to ≈ 8000 GPU hours.
[Table]
F Equivariance
We conduct a simple experiment to evaluate the extent to which model equivariance has on the compressibility of deep learning models and the tightness of our generalization bounds.
We use the rotationally equivariant C 8 WideResNet model from Weiler and Cesa [66] which has an 8 -fold rotational symmetry, and we also use a non equivariant version of this model.
The equivariant model has a depth of 10 and a widen factor of 4 yielding 1 .
451 M parameters.
We control for the number of parameters by adjusting the widen factor of the non equivariant model to 4 .
67 yielding 1 .
447 M parameters.
[Figure]
data than the non equivariant model, and hence yields a better generalization bound.
(Note the difference in dataset size, RotMNIST has only 12K data points unlike MNIST)
We evaluate these models both on MNIST and the RotMNIST dataset [42] consisting of 12K training examples of rotated MNIST digits.
As shown in Figure 5 (a), when paired with the rotationally symmetric RotMNIST dataset, the rotationally equivariant model achieves better bounds and is more compressible than it’s non equivariant counterpart despite having the same number of parameters.
However, when this symmetry of the dataset is removed by considering standard MNIST, we see that the benefits of equivariance to the generalization bound and compressibility vs the WRN model dissapear.
G Full-Batch vs. Stochastic Training (SGD)
To further expand on the results that we present in Section 6, we study the impact of hyperparameters, namely the weight decay and the architecture choice, on the bounds obtained through full-batch (F-B) training.
Table 8 summarizes these results and we provide the training and bound computation details in Appendix E. Our PAC-Bayes subspace compression bounds provide similar theoretical guarantees for both full-batch and stochastic training, suggesting that the implicit biases of SGD are not necessary to guarantee good generalization.
Moreover, we see that the results are consistent for different configurations, which result in comparable bounds overall.
Transfer learning using full-batch training: We perform full batch training for transfer learning from SVHN to MNIST using LeNet-5 and the same experimental setup described in Appendix E.
Our best PAC-Bayes subspace compression bounds for SVHN to MNIST transfer are 8 .
7% and 9 . 0% for full-batch and SGD training, respectively.
This finding provides further evidence that good generalization of neural networks, and the success of transfer learning in particular, does not necessarily require stochasticity or additional flatness-inducing procedures to be achieved.
Finally, we note that we optimize over the same set of hyperparameters for the bound computation for both full-batch and stochastic training.
[Table]
H Model Size vs. Compressibility
We perform an ablation to determine how the size of the model affects our generalization bounds.
Using the fixed model architecture Table
[Table]
we vary the width k from 4 to 192 .
Using our subspace compression scheme, we find that the compression ratio of the model does increase with model size, however the total compressed size still increases slowly making our bounds less strong for larger models.
For this paper, we find the sweet spot k
= 16 is just above the point with equal number of parameters and data points.
We note that this finding leaves room for an improved compression scheme and generalization bounds which are able to explain why even larger models still generalize better.
Curiously, when plotting to the total compressed dataset size ( K ( h | P ) + NLL ) using the model as a compression scheme, we find that the MDL principle which favors shorter description lengths of the data actually prefers larger models than our PAC-Bayes generalization bounds selects.
I Double Descent
Under select conditions, we are able to reproduce the double descent phenomenon in our generalization bounds.
In Figure 7 (right), we show that our bound exhibits a double descent similar to what we see in terms of the test error Figure 7 (left).
The results we show in Figure 7 (right) are obtained for a fixed intrinsic dimensionality of 35000 , but we observed that this middle descent consistently appears in our bounds plots for a given (fixed) intrinsic dimensionality where we select the best bound for each base width.
However, we expect that extending the plot out to larger model widths the bound gets worse again as explained in Appendix H.
J PAC-Bayes Bounds J.1 Catoni PAC-Bayes Bound
In our case, since neural networks achieve low training error, we focus on a bound like Catoni [7] which becomes tighter when KL ( Q, P ) is large.
This is the same bound used in Zhou et al.
[Figure]
[Figure]
Theorem 1 (Catoni [7]) .
Given a 0 - 1 loss ℓ , a fixed α > 1 and a confidence level δ ∈ (0 , 1) then θ ∼ [Equation]
holds with probability higher than 1 − δ and where
[Equation]
J.2 Variable Length Encoding and Robustness Adjustment
In Zhou et al.
[73], the authors assume a fixed length encoding for the weights.
Given that the distribution over quantization levels is highly nonuniform, using a variable length encoding (such as Huffman encoding or arithmetic encoding) can represent the same information using fewer bits.
While this choice gives significant benefits, it means that we cannot immediately make use of ro- bustness adjustment from Zhou et al.
[73], where the robustness adjustment comes from considering neighboring models that result from perturbing the weights slightly.
Revisiting the prior derivation in Zhou et al.
[73], we show why the method used for bounding the
KL does not transfer over to variable length encodings.
In Zhou et al.
[73], the prior used is
[Equation]
where S denotes the encoding of the position of the pruned weights, C denotes the codebook, Q the codebook value that the weight take, d the number of nonzero weights, L the number of clusters, ˆ w the quantized weight and τ 2 the prior variance.
Note that ˆ w changes depending on S, Q, C and also note that the fixed-length encoding can be seen in how we sum over d ⌈ log L ⌉ options.
This prior is a mixture of Gaussians centered at the quantized values.
With this choice of prior Zhou et al.
[73] and setting the posterior to be also Gaussian centered at a quantized value, one can upper bound the
KL with a computationally tractable term involving the sum over dimensions.
Crucially, for their decomposition they use the fact that the size of the encoding | Q | is d ⌈ log L ⌉ , which is independent of the coding Q and only dependent on the codebook C .
Therefore they are able to upper bound
the
KL.
[Equation]
due to the independence of the fixed length encoding to that of the values that each quantized value takes, see appendix in Zhou et al.
This independence is broken for variable length encoding as the cluster centers and the values that each weight can take are interlinked.
Thus, we cannot express and satisfactorily approximate the first high-dimensional KL term as a sum of one-dimensional elements that can be estimated through quadrature or Monte Carlo.
K Licences
MNIST 4 [44] is made available under the terms of the Creative Commons Attribution-Share Alike 3.0 license.
FashionMNIST 5 [69] is available under the MIT license.
CIFAR-10 and CIFAR-
[38] are made available under the MIT license.
ImageNet 6 [15] is the copyright of Stanford Vision
Lab, Stanford University, Princeton University.
SVHN 7 [56] is released for non-commericial use only.