O N P REDICTING G ENERALIZATION USING GAN S
Microsoft
Research
A BSTRACT
Research on generalization bounds for deep networks seeks to give ways to predict test error using just the training dataset and the network parameters.
While generalization bounds can give many insights about architecture design, training algorithms etc., what they do not currently do is yield good predictions for actual test error.
A recently introduced Predicting Generalization in Deep Learning competition (Jiang et al., 2020) aims to encourage discovery of methods to better predict test error.
The current paper investigates a simple idea: can test error be predicted using synthetic data, produced using a Generative Adversarial Network (GAN) that was trained on the same training dataset?
Upon investigating several GAN models and architectures, we ﬁnd that this turns out to be the case.
In fact, using GANs pre-trained on standard datasets, the test error can be predicted without requiring any additional hyper-parameter tuning.
This result is surprising because GANs have well-known limitations (e g. mode collapse) and are known to not learn the data distribution accurately.
Yet the generated samples are good enough to substitute for test data.
Several additional experiments are presented to explore reasons why GANs do well at this task.
In addition to a new approach for predicting generalization, the counter-intuitive phenomena presented in our work may also call for a better understanding of GANs’ strengths and limitations.
I NTRODUCTION
Why do vastly overparametrized neural networks achieve impressive generalization performance across many domains, with very limited capacity control during training?
Despite some promising initial research, the mechanism behind generalization remains poorly understood.
A host of papers have tried to adapt classical generalization theory to prove upper bounds of the following form on the difference between training and test error:
[Equation]
where S is the training dataset and C is a so-called complexity measure , typically involving some function of the training dataset as well as the trained net parameters (e.g., a geometric norm).
Current upper bounds of this type are loose, and even vacuous.
There is evidence that such classically derived bounds may be too loose (Dziugaite & Roy, 2017) or that they may not correlate well with generalization (Jiang et al., 2019).
This has motivated a more principled empirical study of the effectiveness of generalization bounds.
The general idea is to use machine learning to determine which network characteristics promote good generalization in practice and which do not —in other words, treat various deep net characteristics/norms/margins etc.
as inputs to a machine learning model that uses them to predict the generalization error achieved by the net.
This could help direct theorists to new types of complexity measures and motivate new theories.
A recently started competition of Predicting Generalization in Deep Learning (Jiang et al., 2020) (PGDL) seeks to increase interest in such investigations, in an effort to uncover new and promising network characteristics and/or measures of network complexity that correlate with good generalization.
As required in Jiang et al.
(2020), a complexity measure should depend on the trained model, optimizer, and training set, but not the held out test data.
The ﬁrst PGDL competition in 2020 did uncover quite a few measures that seemed to be predictive of good generalization but had not been identiﬁed by theory work.
In this paper, we explore a very simple baseline for predicting generalization that had hitherto not received attention: train a Generative Adversarial Network (GAN) on the training dataset, and use performance on the synthetic data produced by the GAN to predict generalization.
At ﬁrst sight
GANs may not appear to be an obvious choice for this task, due to their well known limitations.
For instance, while the goal of GANs training is to ﬁnd a generator that fools the discriminator net —in the sense that the discriminator has low probability of spotting a difference between GAN samples and the samples from the true distribution—in practice the discriminator is often able to discriminate very well at the end, demonstrating that it was not fooled.
Also, GANs’ generators are known to exhibit mode collapse i.e., the generated distribution is a tiny subset of the true distribution.
There is theory and experiments suggesting this may be difﬁcult to avoid (Arora et al., 2018; Santurkar et al., 2018;
Bau et al., 2019).
Given all these negative results about GANs, the surprising ﬁnding in the current paper is that GANs do allow for good estimates of test error (and generalization error).
This is veriﬁed for families of well-known GANs and datasets including primarily CIFAR-10/100, Tiny ImageNet and wellknown deep neural classiﬁer architectures.
In particular, in Section 3.1 and 3.2 we evaluate on the PGDL and DEMOGEN benchmarks of predicting generalization and present strong results.
In
Section 4 and 5, we also investigate reasons behind the surprising efﬁcacy of GANs in predicting generalization as well as the effects of using data augmentation during GAN training.
R ELATED W ORK
Generalization Bounds Traditional approaches to predict generalization construct a generalization bound based on some notion of capacity such as parameter count, VC dimension, Rademacher complexity, etc.
Neyshabur et al.
(2018) provide a tighter generalization bound that decreases with increasing number of hidden units.
Dziugaite & Roy (2017) reveal a way to compute nonvacuous PAC-Bayes generalization bounds.
Recently, bounds based on knowledge distillation have also come to light (Hsu et al., 2020).
Despite progress in these approaches, a study conducted by Jiang et al.
with extensive hyper-parameter search showed that current generalization bounds may not be effective, and the root cause of generalization remains elusive.
Given the arduous nature of constructing such bounds, an interest in complexity measures has arisen.
Predicting Generalization in Deep Learning The PGDL competition (Jiang et al., 2020) was held in NeuRIPS 2020 in an effort to encourage the discovery of empirical generalization measures following the seminal work of Jiang et al.
The winner of the PGDL competition Natekar & Sharma (2020) investigated properties of representations in intermediate layers to predict generalization.
Kashyap et al.
(2021), the second place winner, experiment with robustness to ﬂips, random erasing, random saturation, and other such natural augmentations.
Afterwards, Jiang et al.
(2021) interestingly ﬁnd that generalization can be predicted by running SGD on the same architecture multiple times, and measuring the disagreement ratio between the different resulting networks on an unlabeled test set.
There are also some pitfalls in predicting generalization, as highlighted by
Dziugaite et al.
They ﬁnd that distributional robustness over a family of environments is more applicable to neural networks than straight averaging.
Limitations of GANs Arora et al.
(2017) prove the lack of diversity enforcing in GAN’s training objective, and Arora et al.
(2018) introduce the Birthday Paradox test to conclude that many popular GAN models in practice do learn a distribution with relatively small support.
Santurkar et al.
(2018) and Bau et al.
(2019) investigate the extent to which GAN generated samples can match the distributional statistics of the original training data, and ﬁnd that they have signiﬁcant shortcomings.
Ravuri & Vinyals (2019) ﬁnd that GAN data is of limited use in training ResNet models, and ﬁnd that neither inception score nor FID are predictive of generalization performance.
Notably, despite the small support, Arora et al.
(2018) reveal that GANs generate distinct images from their nearest neighbours in the training set.
Later, Webster et al.
(2019) use latent recovery to conclude more carefully that GANs do not memorize the training set.
[Figure]
Theoretical Justiﬁcation for Using GANs
Arora et al.
(2017) construct a generator that passes training against any bounded capacity discriminator but can only generate a small number of distinct data points either from the true data distribution or simply from the training set.
For predicting generalization, it is crucial for the generator not to memorize training data.
While Arora et al.
(2017) do not answer why GANs do not memorize training data, a recent empirical study by Huang et al.
(2021) demonstrates the difﬁculty of recovering input data by inverting gradients.
Their work may cast light on how the generator could learn to generate data distinct from the training set when trained with gradient feedbacks from the discriminator.
However, we are not aware of any theory that fully explains GANs’ strength for predicting generalization despite limitations.
P REDICTING TEST PERFORMANCE USING GAN SAMPLES
We now precisely deﬁne what it means to predict test performance in our setting.
We denote by
S train , S test and S syn the training set, test set and the synthetic dataset generated by GANs.
Given a classiﬁer f trained on the training set S train , we aim to predict its classiﬁcation accuracy g ( f ) :=|
S test | (cid:80) ( x,y ) ∈ S test 1 [ f ( x ) =
y ] on a test set S test .
Our proposal is to train a conditional GAN model on the very same training set S train , and then sample from the generator a synthetic dataset S syn of labeled examples.
In the end, we simply use f ’s accuracy on the synthetic dataset as our prediction for its test accuracy.
Algorithm 1 formally describes this procedure.
Require: target classiﬁer f , training set S train , GAN training algorithm
A
Train a conditional GAN model using S train : G, D
= A ( S train ) where G, D are the generator and discriminator networks.
Generate a synthetic dataset by sampling from the generator G : S syn =
{ (˜ x 1 , ˜ y 1 ) , . . .
, (˜ x N , ˜ y N ) } where ˜ x i , ˜ y i =
G ( z i , ˜ y i ) .
The z i ’s are drawn i.i.d.
from G ’s default input distribution.
N and ˜ y
i ’ are chosen so as to match statistics of the training set
Output: the synthetic accuracy ˆ g ( f ) :=
[Equation]
as the prediction
Remark.
Any N ≥ | S train | is a safe choice to ensure that ˆ g ( f ) concentrates 1 around its mean E z, ˜ y
[ 1 [ f ( G ( z, ˜ y ))
=
˜
y ]] and its deviation has negligible inﬂuence on the performance.
[Table]
We demonstrate the results in Figure 1. The test accuracy g ( f ) consistently resides in a small neighborhood of ˆ g ( f ) for a diverse class of deep neural net classiﬁers trained on different datasets.
For the choice of GAN architecture, we adopt the pre-trained BigGAN+DiffAug models (Brock et al., 2019; Zhao et al., 2020) from the StudioGAN library 2 .
We evaluate on pretrained deep neural net classiﬁers with architectures ranging from VGG-(11, 13, 19), ResNet- (18, 34, 50), DenseNet-(121, 169), ShufﬂeNet, PNASNet and MobileNet trained on CIFAR10 and Tiny ImageNet with hyper-parameter setting (learning rate, weight decay factor, learning rate schedule) uniformly sampled from the grid (cid:8) 10 − 1 , 10 − 2 (cid:9) × (cid:8) 5 × 10 − 4 , 10 − 3 (cid:9) × { CosineAnnealing , ExponentialDecay } .
We use SGD with momentum 0 .
9 and batch size 128 and data augmention of horizontal ﬂips for training all classiﬁers.
3. E VALUATION ON PGDL C OMPETITION
We evaluate our proposed method on the tasks of NeurIPS 2020 Competition on Predicting Generalization of Deep Learning.
The PGDL competition consists of 8 tasks, each containing a set of pre-trained deep net classiﬁers of similar architectures but with different hyper-parameter settings, as well as the training data.
The tasks of PGDL are based on a wide range of datasets inlcuding
CIFAR-10, SVHN, CINIC-10, Oxford Flowers, Oxford Pets and Fashion MNIST.
For every task, the goal is to compute a scalar prediction for each classiﬁer based on its parameters and the training data that correlates as much as possible with the actual generalization errors of the classiﬁers measured on a test set.
The correlation score is measured by the so-called Conditional Mutual Information, which is designed to indicate whether the computed predictions contain all the information about the generalization errors such that knowing the hyper-parameters does not provide additional information, see (Jiang et al., 2020) for details.
Since the goal of PGDL is to predict generalization gap instead of test accuracy, our proposed solution is a slight adaptation of Algorithm 1.
For each task, we ﬁrst train a GAN model on the provided training dataset and sample from it a labeled synthetic dataset.
Then for each classiﬁer within the task, instead of the raw synthetic accuracy, we use as prediction the gap between training and synthetic accuracies.
For all tasks, we use BigGAN+diffAug with the default 3 implementation for CIFAR-10 from the StudioGAN library.
We found that a subset of them (Task 6 with Oxford
Flowers and Task 7 with Oxford Pets) were not well-suited for standard GAN training without much tweaking due to a small training set and highly uneven class sizes.
We focused only on the subset of the tasks where GAN training worked out of the box, speciﬁcally Task 1, 2, 4, 5, 8, and 9.
We report the results in Table 1, where we observe that on tasks involving either CIFAR-10 or VGGlike classiﬁers, our proposed solution out-performs all of the solutions from the top 3 teams of the competition by a large margin.
One potential reason may be that the default hyper-parameters of [Figure] the BigGAN model have been engineered towards CIFAR-10 like datasets and VGG/ResNet-like discriminator networks.
It is worth mentioning that we conducted absolutely zero hyper-parameter search for all tasks here.
Especially for CIFAR-10 tasks, we directly take the pre-trained models from the StudioGAN library.
It is likely that we may achieve even better scores with ﬁne-tuned
GAN hyper-parameters and optimized training procedure for each PGDL task, and we leave it to future work.
3. E VALUATION ON DEMOGEN BENCHMARK
The seminal work of Jiang et al.
(2018) constructed the Deep Model Generalization benchmark (DEMOGEN) that consists of 216 ResNet-32 (He et al., 2016) models and 216 Network-inNetworks (Lin et al., 2013) models trained on CIFAR-10 plus another 324 ResNet-32 models trained on CIFAR-100, along with their training and test performances.
These models are conﬁgured with different normalization techniques (group-norm v.s. batch-norm), network widths, learning rates, weight-decay factors, and notably whether to train with or without data augmentation.
The goal of DEMOGEN, different from PGDL, is to numerically predict generalization gap (or equivalently test accuracy) with the additional knowledge of the test accuracies of a few pretrained classiﬁers.
Following the setup of Jiang et al.
(2018), we use a simple linear predictor g ( f )
= a · ˆ g ( f )
+ b with two scalar variables a, b ∈ R instead of directly using ˆ g ( f ) for the best performance.
Here a, b are estimated using least squares regression on a separate pool of pre-trained classiﬁers with known test accuracies.
Jiang et al.
(2018) proposes to use variants of coefﬁcient of determination ( R 2 ) of the resulted predictor as evaluation metrics.
Speciﬁcally, the adjusted R 2 and k -fold R 2 are adopted (see Jiang et al.
(2018) for concrete deﬁnitions).
NiN+CIFARResNet+CIFARResNet+CIFARadj
R10 -fold Radj
R10 -fold Radj R10 -fold Rqrt+log 0.
0 .qrt+log+unnorm 0.
0 .moment+log 0.
Self-Attention GAN 0.
Spec-Norm GAN 0.
BigGAN 0.
BigGAN+ADA 0.
[Table]
In Table 2, we compare the scores of our method with several state-of-the-art GAN models.
Specifically, we again take the pre-trained GAN models from the StudioGAN library to generate synthetic datasets for the CIFAR-10 tasks.
For the CIFAR-100 task, we use the same hyper-parameters as the
CIFAR-10 models to train a plain BigGAN and a BigGAN+DiffAug model from scratch.
Again, each synthetic dataset is sampled only once.
Overall, predicting using GAN samples achieves remarkably higher scores than all methods proposed in Jiang et al.
We also visualize the resulted linear ﬁt for each task in Figure 2 where we observe the strong linear correlation.
Note that the resulting linear ﬁts are close the ideally desired y
=
x
ﬁt.
In fact, using directly ˆ g ( f ) as the prediction would also lead to extremely competitive R 2 values 0 . 928 , 0 .
995 and 0 .
975 respectively.
GAN SAMPLES ARE CLOSER TO TEST SET THAN TRAINING SET
In this section, we demonstrate a surprising property of the synthetic examples produced by GANs that may provide insights into their observed ability to predict generalization: synthetic datasets generated by GANs are ‘closer’ to test set than training set in the feature space of well-trained deep net classiﬁers, even though GAN models had (indirect) access to the training set but no access to the test set.
Here ‘well-trained’ classiﬁers are the ones that almost perfectly classify the training examples, i.e. its training accuracy is > 97% .
To deﬁne closeness properly, we modify the widely used standard Fr ´ echet Distance proposed by Heusel et al.
(2017) to account for labeled examples.
We name our modiﬁed version as the Class-conditional Fr ´ echet Distance.
4. C LASS - CONDITIONAL F R ´ ECHET D ISTANCE
The goal is to measure how close the labeled synthetic dataset is to the test set in the eyes of a certain pre-trained classiﬁer.
However, FID does not capture any label information by default.
Our proposed class-conditional Fr ´ echet Distance can be viewed as the sum over each class of the Fr ´ echet
Distance between test and synthetic examples in a certain feature space.
The original FID measures a certain distance between the feature embeddings of real examples and synthetic examples from
GANs, in the feature space of an Inception network pre-trained on ImageNet.
For our purposes, we modify the notion of FID to use any feature extractor h .
Speciﬁcally, we deﬁne the distance between two sets of examples S and S (cid:48) w.r.t.
a feature extractor h as
[Equation]
where we have the empirical mean of h ’s output µ S ( h ) := 1 | S | (cid:80) ( x
,y ) ∈
S h
( x ) and the empirical covariance Cov S ( h ) :
= 1 | S | (cid:80) ( x,y ) ∈ S ( h ( x ) − µ S ( h ))
( h ( x ) − µ S ( h ))
(cid:62) over the examples from
S , and µ S (cid:48) ( h ) , Cov S (cid:48)
( h ) are computed analogously.
Furthermore, [ · ] 1 / 2 is the matrix square root and Tr [ · ] is the trace function.
For labeled datasets S and ˜ S , we denote by C the collection of all possible labels.
For each label c ∈ C , we let S c and
˜ S c denote the the subsets of examples with label c in S and ˜ S respectively.
This leads to the class-conditional Fr´echet Distance:
[Equation]
In Figure 3, we evaluate d h on CIFAR-10 among the training set S train , the test set S test and the synthetic set S syn , with h being the feature map immediately before the fully-connected layer of each pre-trained deep net used in the middle sub-ﬁgure of Figure 2.
The synthetic dataset is generated by the same BigGAN+DiffAug model used in Figure 2 as well.
We observe that for all pre-trained nets, d h ( S syn , S test ) <
d h ( S train , S test ) by a big margin, which provides a partial explanation for why GAN synthesized samples may be a good replace for the test set.
Furthermore, for every well-trained net, d h ( S syn , S test ) <
d h ( S syn , S train ) as well.
This is particularly surprising because S syn ends up being ‘closer’ to the unseen S test rather than S train , despite the GAN model being trained to minimize certain distance between S syn and S train as part of the training objective.
4. C LASS - CONDITIONAL F R ´ ECHET D ISTANCE ACROSS CLASSIFIER ARCHITECTURES
It is natural to wonder whether we can attribute the phenomenon of GAN samples being closer to test set to the architectural similarity between the evaluated classiﬁers and the discriminator network
[Figure]
This indicates that in their feature space the labeled synthetic examples behave more similarly to the test set rather than to the training set.
used in the GAN model — they are all derivatives from the ResNet family.
Thus we investigate the values of d h for the collection of classiﬁers with diverse architectures used in Figure 1, which contains network architectures that may be remote from ResNet such as VGG, MobileNet, PNASNet etc.
The result is demonstrated in Figure 4, where we observe the surprising phenomenon that the single copy of the synthetic dataset is regarded closer to the test set by all classiﬁers in the collection.
This suggests that certain GAN models are capable of matching, to a degree, the feature distribution universally for a diverse set of deep neural classiﬁers.
E XPLORATION : E FFECTS OF DATA AUGMENTATION
In this section, we discuss certain intriguing effects of data augmentation techniques used during
GAN training on the performance of the resulted synthetic datasets for predicting generalization.
We use the CIFAR-100 task from DEMOGEN as a demonstration where the effects are the most pronounced.
The task consists of 324 ResNet classiﬁers, only half of which are trained with data augmentation, speciﬁcally random crop and horizontal ﬂipping.
Given the success of complexity measures that verify ‘robustness to augmentation’ in the PGDL competition, one would imagine applying data augmentations to the real examples during GAN training could be beneﬁcial for prediciting generalization, since the generator would be encouraged [Figure] all classifiers [Figure]
all classifiers d(syn, test) / d(syn, train) to match the distribution of the augmented images.
However, show in Figure 5 this is not the case here.
Speciﬁcally, Figure 5a indicates that, when evaluated on a synthetic dataset from a GAN model trained with augmentation only applied to real examples, the ResNets trained without any data augmentation exhibit a test accuracy v.s. synthetic accuracy curve that substantially deviates from y = x , while the ones trained with data augmentations are relatively unaffected.
In comparison, once data augmentation is removed from GAN’s training procedure, this deviation disappears, as captured in Figure 5b.
This could be because the ResNets trained without data augmentation are sensitive to the perturbations learned by the generator.
Overall, the best result is achieved by deploying the differential augmentation technique that applies the same differentiable augmentation to both real and fake images and enables the gradients to be propagated through the augmentation back to the generator so as to regularize the discriminator without manipulating the target distribution [Figure]
D ISCUSSION AND C ONCLUSIONS
Competitions such as PGDL are an exciting way to inject new ideas into generalization theory.
The interaction of GANs and good generalization needs further exploration.
It is conceivable that the quality of the GAN-based prediction is correlated with the niceness of architectures in some way.
It would not be surprising that it works for popular architectures but does not work for all.
We leave it as future work to determine the limits of using GANs for predicting generalization
Our ﬁnding presented here offers a new approach for predicting generalization, and it sheds light on a possibly bigger question — what can GANs learn despite known limitations?
To our best knowledge, we are not aware of any existing theory that completely explains this counter-intuitive empirical phenomena.
Such a theoretical understanding will necessarily require innovative techniques and thus is perceivably non-trivial.
However, we believe it is an important future direction as it will potentially suggest more use cases of GANs and will help improve existing models in a principled way.
We hope our work inspires new endeavors along this line