Investigating Shifts in GAN Output-Distributions
Abstract
A fundamental and still largely unsolved question in the context of Generative
Adversarial Networks is whether they are truly able to capture the real data distribution and, consequently, to sample from it.
In particular, the multidimensional nature of image distributions leads to a complex evaluation of the diversity of GAN distributions.
Existing approaches provide only a partial understanding of this issue, leaving the question unanswered.
In this work, we introduce a loop-training scheme for the systematic investigation of observable shifts between the distributions of real training data and GAN generated data.
Additionally, we introduce several bounded measures for distribution shifts, which are both easy to compute and to interpret.
Overall, the combination of these methods allows an explorative investigation of innate limitations of current GAN algorithms.
Our experiments on different data-sets and multiple state-of-the-art GAN architectures show large shifts between input and output distributions, showing that existing theoretical guarantees towards the convergence of output distributions appear not to be holding in practice.
Introduction
The efﬁcient and accurate modeling of complex multi-modal data distributions is one of the core problem behind many machine learning tasks.
While most of the recently very successful classiﬁcation algorithms can retreat to the simpler sub-problem of ﬁnding separation-functions in data distributions, the task of generative algorithms is to provide a model able to reproduce such complex multi-modal data distributions to their full extent.
One of the most popular approaches towards generative models are Generative Adversarial Networks (GANs) [1].
GANs have been used for a wide range of use cases and data distributions [2], including numerous applications involving image data generation [3].
In this work we study the distribution shifts in GANs outputs and evaluate to what extent GANs can capture the full diversity of the underlying true distribution.
To accomplish these tasks, we design a loop-training strategy, where the misalignment between real and generated data distributions is ampliﬁed, allowing in this way a painstaking investigation.
On top of that, we introduce shift indicators on sub-distribution of the real image space, which allow an easy evaluation of lower bounds for very complex distribution shifts.
Finally, we evaluate structural differences between inand output-distributions for commonly used GAN architectures on several standard benchmarks.
Our experiments show that the observable distribution shifts in our indicators are so large, that generated samples can easily be detected by the use of binary classiﬁers.
Taken together, these experimental results provide strong evidence that GANs learn a non-trivial but shifted version of the true distribution.
Related Work
The original formulation of Generative Adversarial Networks [1] describes GAN training as an optimization problem, minimizing the Jensen-Shannon divergence between generated and real data distributions.
Even though there are several theoretical discussions of GAN convergence, like [4, 5, 6], they all come with certain constraints or assume some preconditions – leaving very little guarantees for practical real world applications.
Despite producing visual appealing results on images, [7] showed that this gap does exist and GANs do not produce samples from the real data distribution, but from an approximated low dimensional manifold.
These distribution shifts, approximations, lead to limitations in terms of data synthesization and generalization.
Recent work [8, 9] also showed that
GANs systematically fail to reproduce image distributions in the frequency domain.
Methods
Loop-Training.
We propose a loop-training scheme (see Figure 1) to highlight the demotion of generated images due to the distribution shifts in GANs models.
It consists of (1) training the model and (2) generating a new dataset using the trained generator.
Observable Sub-Distributions of Images.
We denote real images x ∈ I m ⊂ (cid:16) R √ m × R √ m × 3 (cid:17) as samples from the space I m of all possible color images of a given size m .
As real distributions p r
( I m ) in this image space are typically extremely complex and hard to compare explicitly, we retreat to the analysis of simpler subdistributions which provide computable but meaningful measures for observable lower bounds of shifts in the full image distributions.
FID: The Standard Measure for GAN Distribution Quality.
Fréchet Inception Distance (FID) [10] is the metric commonly used to assess the distribution of generated images.
It compares the distribution of generated images with the distribution of those real images used to train the model in the feature-space of a pre-trained Inception-v3 classiﬁcation model [11] model.
Color Histograms.
Besides FID, we propose to use color-distributions as an indicator.
Although it neglects all spacial information, its correctness is still a necessary condition for the distribution of generated data p g
( I m ) and thus deﬁnes a suitable bound for the estimation of distribution shifts.
We model these color-distributions in form of n averaged samples of color-histograms H b ( x ) ∈ R 3 b with b bins p
[Equation]
p g is computed respectively with ¯ x i .
Class Histograms.
We also propose to analyse the shift in class distributions.
Here we apply a pre-trained classiﬁer on the generated data and compute the class histograms which are a suitable indicator for partial collapse of output distributions.
Measuring Distribution Similarities.
We employ the Kullback-Leibler divergence (KL) to measure the similarities between p g and p r (or N (0 , 1) respectively) formulated as
[Equation]
where p r ( i ) denotes the i -th element of the discrete representation (see Equation 1).
Empirical Evaluation of GAN Shifted Distributions
For a fair comparison and reproducibility, we conduct the experiments using a standardized GAN library [12].
This allows to train all models under the same conditions using different datasets.
Models.
We employ a set of different GANs: Wasserstein GAN with Gradient Penalty (WGANGP) [13], Spectral Normalization GAN (SNGAN) [14], Self-supervised GAN (SSGAN) [15], and
InfoMax-GAN (InfoGAN) [16].
Datasets.
We evaluate results on seven commonly used datasets: CIFAR-10 [17], CIFAR-100 [17],
STL-10 [18], CelebA [19] at two different resolutions, ImageNet 1 [20], and LSUN-Bedroom 2 [21].
4.1 Experimental Results
Demotion of Distributions in the training loop.
Figure 2 shows the demotion of the FID score over training iterations, using different data-sets and models.
We can observe that the FID undergoes an almost linear decline in GANs performances, independent of the scenario.
This behavior reveals that the feature space distribution of the generated data is “degenerating” with each iteration, increasing the shift towards the distribution of the real data.
We hypothesize that this behavior is caused by larger distributions shifts, which is also supported by the strong degradation of class distributions shown in Figure 3.
As one might expect, those models that undergo several iterations will inevitably start to suffer drastically from mode collapse.
Moreover, we observe empirically, that this misalignment between the real and the synthetic distributions cannot not be prevented by alternative loss formulations such as Wasserstein [13], nor by advanced regularizes such as spectrum normalization [14] nor even by semi-supervised approaches such as
SSGAN [15].
Observation: Strong Shift Towards Gaussian Sub-Distributions.
Our empirical analysis of the color-histogram sub-distributions of the same experiments show another surprising effect: the distributions appear to converge towards a Gaussian over the iterations of the loop-training.
Figureshows a typical example of this observation.
Measuring the KL between color-distributions of generated and real data, we see a strong correlation to the increase in the FID score in Figure 4.
Impact of Distribution Shifts on Individual Samples.
The previous results show strong shifts in the distributions from real input images to GAN generated images.
However, these distributions have been estimated from a larger number of samples.
Hence, it is also interesting to assess if these shifts are systematic enough to be detectable in single samples.
We therefore train simple classiﬁer models on the color-histograms of single samples, labelled as real or generated.
Table 1 shows that it is, in fact possible [Table] to predict generated data with up to 85% accuracy for a wide range of investigated datasets.
[Table]
Discussion and outlook
We present clear experimental results, investigating the distribution shifts from which GANs architectures suffer.
Furthermore, we introduce a sub-distribution based on color-image space which can be utilized to gain valuable insights of the generator capacity.
We also ﬁnd a strong correlation between sub-distribution shifts and the FID scores.
Taken together, these results show that GANs learn a shifted version of the true distribution and immediately raise the question, why discriminators appear to be unable to detect evident differences between input and output distributions of a GAN.
Supplementary Material
This supplementary section contains an extended version of the implementation details that we have presented in the main paper.
Additionally, we provide more results, discussing the direct effect that distribution shifts have on our sub-distribution.
6.1 Details on Loop-Training
Figure 1 shows the training pipeline of the looptraining experiment, introduced in the main paper.
We conduct further evaluations using the loop-training setup.
In particular, we compare FID results between loop-and long-training pipelines.
The difference between these experiments is that despite both trainings train the same amount of epochs, the latter trains with real data as reference during the whole training.
In Figure 6, we can observe that the distribution shifts of the long-training, i.e., the capacity of the GAN model to generate images covering the real distribution, remains stable.
On the other hand, when using generated data to train the next model (the next iteration), the results start to worsen, offering every iteration a more skewed and shifted distribution of the output distributions.
6.2 Details on Observable Sub-Distributions
Figure 7 illustrates how is measured the distribution similarity.
As we explained before, the idea is to compute the best Gaussian ﬁt and the proportion that this ﬁt represents for the whole Gauss curve to weight the ﬁnal KL scores.
In this way, we can penalize those ﬁts that barely have a Gaussian shape but still have a good ﬁtting, e.g., those ﬁts that just cover a small part of a tail.
6.3 Color-Distribution Cluster Analysis
Following the classiﬁcation analysis from the main paper, we propose this time an unsupervised clustering approach for the color-distribution space.
Using this rather simple method, one can easily visualize the mode-differences between in- and output distributions, even allowing to pinpoint single samples from the training data which can not be reconstructed by the GAN.
Showing again the notorious capacity of GANs to model entirely the real distribution.
We run clustering experiments based on k-Means, where we set different sizes of clusters ( k ).
In this way, we can observe which images are plausible to be reproduced by the generative model, and which ones are not.
Figure 8 depicts two different experiments on CIFAR10.
We can see how the real data has a more diverse distribution, while the generated samples reduce to a few clusters.
As a result, it is likely that generative model starts to produce samples very similar to each other and eventually this might lead to mode collapse.
6.4 Gaussian-like Distribution Analysis
Finally, we conduct an experimental study to visualize the Gaussian tendency that output distributions of GANs seem to be shifted to.
To accomplish this task, we approximate the color-distribution of generated data and real data to their best Gaussian ﬁt, and then, we compute the KL distance between these results.
Figure 9 give a good impression of the typical distribution shift that generated data undergoes, resulting in a dominant Gaussian shape of GAN outputs distributions.