S EEING IS N OT N ECESSARILY B ELIEVING : L IMITATIONS OF B IG GAN S FOR D ATA A UGMENTATION
A BSTRACT
Recent advances in Generative Adversarial Networks (GANs) – in architectural design, training strategies, and empirical tricks – have led nearly photorealistic samples on large-scale datasets such as ImageNet.
In fact, for one model in particular, BigGAN, metrics such as Inception Score or Frechet Inception Distance nearly match those of the dataset, suggesting that these models are close to matching the distribution of the training set.
Given the quality of these models, it is worth understanding to what extent these samples can be used for data augmentation , a task expressed as a long-term goal of the GAN research project.
To that end, we train ResNet-50 classiﬁers using either purely BigGAN images or mixtures of ImageNet and BigGAN images, and test on the ImageNet validation set.
Our preliminary results suggest both a measured view of state-of-the-art GAN quality and highlight limitations of current metrics.
Using only BigGAN images, we ﬁnd that Top-1 and Top-5 error increased by 120% and 384%, respectively, and furthermore, adding more BigGAN data to the ImageNet training set at best only marginally improves classiﬁer performance.
Finally, we ﬁnd that neither Inception Score, nor FID, nor combinations thereof are predictive of classiﬁcation accuracy.
These results suggest that as GANs are beginning to be deployed in downstream tasks, we should create metrics that better measure downstream task performance.
We propose classiﬁcation performance as one such metric that, in addition to assessing per-class sample quality, is more suited to such downstream tasks.
I NTRODUCTION
Recent years have witnessed a marked improvement in sample quality in Deep Generative Models.
One model class in particular, Generative Adversarial Networks (Goodfellow et al., 2014), has begun to generate nearly photorealistic images.
While applications of adversarial training have found their way into image translation (Zhu et al., 2017) and style transfer (Chan et al., 2018), a typically discussed goal for such models, and in particular conditional ones, is data augmentation.
Such models have enjoyed limited success in these tasks thus far for large-scale datasets such as ImageNet, likely because existing models did not generate sufﬁciently high-quality samples.
Recently, however, BigGANs (Brock et al., 2018) have generated photorealistic images of ImageNet data up to 512 × 512 resolution, and moreover, achieve Inception Scores and Frechet Inception Distances similar to the dataset on which they were trained.
Such results suggest, though do not prove, that
BigGANs are indeed capturing the data distribution.
If this were true, then it seems plausible that these samples can be used in downstream tasks, especially in situations in which limited labelled data are available.
In this work, we test the rather simple hypothesis that BigGANs are indeed useful for data augmentation , or more drastically, data replacement of the original data distribution.
To that end, we use
BigGANs for two simple experiments.
First, we train ImageNet classiﬁers, replacing the original training set with one produced by BigGAN.
Second, we augment the original ImageNet training set with samples from BigGAN.
Our working hypothesis is that if BigGANs were indeed capturing the data distribution, then we could use those samples, instead of or in addition to the original training set, to improve performance on classiﬁcation.
That it does not – on replacement, Top-5 classiﬁcation
[Table]
error increased by 384% compared to the original training set; and on augmentation, classiﬁcation performance improves only marginally while dramatically increasing training time – suggests that naively augmenting the dataset with BigGAN samples is of limited utility and more work is required for BigGANs to be used in downstream tasks
[Table]
Though a negative result
[Table]
a more positive byproduct of the work is the introduction of a new metric that can better identify issues with GAN and other generative models.
In particular, training a classiﬁer allows us to identify, for conditional generative models, which classes are particularly poor, either due to low quality samples or underrepresentation of dataset diversity.
E XPERIMENTS 2. S ETUP
Our experiments are rather simple: we use BigGAN-deep (further denoted as BigGAN) models to either replace or augment the ImageNet training set, train an image classiﬁer, and compare performance on the ImageNet validation set.
In the data replacement experiments, we replace the
ImageNet training set with one from BigGAN-deep, and each example from the original training set is replaced with a model sample from the same class.
In the augmentation experiments, we add to the ImageNet training set, 25%, 50%, or 100% more data from BigGAN.
Moreover, since the truncation trick – which resamples dimensions that are outside the mean of the distribution – seems to trade off quality for diversity, we perform experiments for a sweep of truncation parameters: 0.2, 0.42, 0.5, 0.8, 1.0, 1.5, and 2.0.
In addition, we compare performance on replacement and augmentation to two traditional GAN metrics: Inception Score (Salimans et al., 2016) and Frechet Inception
Distance (FID) (Heusel et al., 2017), as these metrics are the current gold standard for GAN comparison.
Both rely on a feature space from a classiﬁer trained on ImageNet, suggesting that if metrics are useful at predicting performance on a downstream task, it would indeed be this one.
We used ResNet-50 (He et al., 2016) classiﬁer for our models, with single-crop evaluation.
The classiﬁer is trained for 90 epochs using TensorFlow’s momentum optimizer, a learning rate schedule linearly increasing from 0.0 to 0.4 for the ﬁrst 5 epochs, and decreased by a factor of 10 at epochs 30, 60, and 80.
It mirrors the 8,192 batch setup of Goyal et al.
(2017) with gradual warmup.
2. R ESULTS
Table 1 shows the performance of classiﬁers trained on BigGAN datasets compared to the real dataset.
At every truncation level, ResNet-50 classiﬁers trained on BigGAN samples generalize substantially worse to real images than the classiﬁer trained on real data.
To better understand why this has occurred, we broke down the performance by class for the best-performing truncation level: 1.5.
As shown in the left pane of Figure 1, nearly every class suffers a drop in performance compared to the original dataset.
Performance on six classes – partridge, red fox, jaguar/panther, squirrel monkey, African elephant, and strawberry – did improve over the original dataset, though the improvement for those classes was marginal.
The right pane of Figure 1 shows the two best and two worst performing categories, as measured by the difference in classiﬁcation performance.
Notably, for the two worst performing categories and two others – balloon, paddlewheel, pencil sharpener, and spatula – classiﬁcation accuracy was 0% on the validation set.
That said, at the best truncation levels, Top-5 Error is roughly 35%, suggesting that BigGANs are learning nontrivial distributions.
Given the performance of BigGAN in the replacement experiments, one should not necessarily expect improved classiﬁer accuracy by augmenting ImageNet training set with BigGAN samples.
Figure 2 illustrates the performance of the classiﬁers when we increase the amount of BigGAN training data.
Perhaps somewhat surprisingly, BigGAN models that sample from lower truncation values, and have lower sample diversity, are able to perform better on data augmentation compared to those models that performed the best on the data replacement experiment.
In fact, for some of the lowest truncation values, one found modest improvement in classiﬁcation performance: roughly 3% improvement relative on Top-1 Error (but at the cost of 1.5 times the amount of training time).
Finally, Inception Score and FID had very little correlation with performance on either the replacement or augmentation experiments, suggesting that alternative metrics will be needed when we turn our attention to downstream tasks.
For our replacement experiments, the correlation coefﬁcient between Top-1 error and FID is 0.16, and Inception Score 0.86, the latter result incorrectly suggesting that improved Inception Score is highly correlated with increased error.
Moreover, the best-performing methods have rather poor Inception Score and FIDs.
That models that perform poorly on Inception Score and Frechet Inception Distance also perform poorly on classiﬁcation is no surprise; that models that perform well on Inception Scores and FIDs perform poorly on classiﬁcation suggests that alternative metrics are needed.
One can easily diagnose the issue with Inception
Score: as Barratt & Sharma (2018) noted, Inception Score does not account for intra-class diversity, and a training set with little intra-class diversity may make the classiﬁer fail to generalize to a more diverse test set.
FID should better account for this lack of diversity at least grossly, as the metric, calculated as FID ( P x , P y )
= (cid:107) µ x − µ y (cid:107) 2 + tr (Σ x + Σ y − 2(Σ x Σ y ) 1 / 2 ) , compares the covariance matrices of the data and model distribution.
By comparison, per-class classiﬁcation error offers a ﬁner measure of model performance, as it provides us a per-class metric to identify which classes have better or worse performance.
While in theory one could calculate a per-class FID, FID is known to suffer from high bias (Bi´nkowski et al., 2018) for low number of samples, likely making the per-class estimates unreliable.
The results on augmentation highlight different desiderata for samples that are added to the dataset rather than replaced.
Clearly, the samples added should be sufﬁciently different from the data to allow the classiﬁer to better generalize, and yet, poorer sample quality may lead to poorer generalization compared to the original dataset.
This may be the reason why extending the dataset with samples generated from a lower truncation value noise – which are higher-quality, but less diverse – perform better on augmentation than replacement.
Furthermore, this may also explain why Inception Score, Frechet Inception Distance, and data replacement classiﬁcation error are not predictive of data augmentation classiﬁcation performance.
R ELATED W ORK
This work encompasses two lines of work: GANs for data augmentation and improved evaluation metrics for GANs.
For data augmentation, Antoniou et al.
(2017) proposed an image-conditioned model for augmentation, and found improved results on smaller datasets.
Frid-Adar et al.
(2018) used a GAN to generate synthetic training data of size 64 × 64 × 1 of images of liver lesions.
For evaluation metrics, Theis et al.
(2016) noted there is difﬁculty in designing evaluation metrics that will illustrate the general performance in the model.
Despite this ﬁnding, those interested in measuring the quality of implicit generative models have proposed practical metrics to compare sample quality from different models: which have led to introduction of Inception Score and FID.
Lopez-Paz & Oquab (2016) recommends the use classiﬁer two-sample tests to test GAN samples as a metric.
Other measures attempt to determine other properties of generative models.
Lucic et al.
(2018) constructs synthetic datasets for which precision and recall can be computed approximately and compares Inception Score and FID to changes in precision and recall.
Geometry Score (Khrulkov &
Oseledets, 2018) constructs approximate manifolds from data and samples, and uses them for GAN samples to determine whether there was mode collapse.
Arora & Zhang (2017) attempt to determine the support size of GANs by using a Birthday Paradox test, though it requires a human to identify two nearly-identical samples.
C ONCLUSION
In this work, we investigated to what extent BigGAN, the state-of-the-art GAN on ImageNet, captures the data distribution, and to what extent those samples can be used for data augmentation.
Our results demonstrate that despite excellent scores on traditional GAN metrics such as Inception Score and Frechet Inception Distance, current state-of-the-art GAN models do not capture the distribution for large-scale datasets such as ImageNet.
Moreover, we found only a modest improvement in classiﬁer performance when the training set was augmented with BigGAN samples.
Finally, through classiﬁer metrics outlined in the work, we can identify on which classes BigGAN performed well, and on which ones researchers should focus their future efforts.
An open question in this work is how to create metrics predictive of performance on downstream tasks.
Even for the classiﬁer metric, results on data replacement did not necessarily correlate with those on data augmentation.
Better evaluation metrics will help us understand to what extent GANs, or any other Deep Generative Models, can be used for downstream tasks.