A study of quality and diversity in K+1 GANs
Abstract
We study the K+1 GAN paradigm which generalizes the canonical true/fake GAN by training a generator with a K+1-ary classiﬁer instead of a binary discriminator.
We show how the standard formulation of the K+1 GAN does not take advantage of class information fully and show how its learned generative data distribution is no different than the distribution that a traditional binary GAN learns.
We then investigate another GAN loss function that dynamically labels its data during training, and show how this leads to learning a generative distribution that emphasizes the target distribution modes.
We investigate to what degree our theoretical expectations of these GAN training strategies have impact on the quality and diversity of learned generators on real-world data.
Introduction
When GANs were ﬁrst introduced [1] deep learning had already had striking successes with learning discriminative models while high quality deep generative models had yet to appear.
The GAN framework combined these two models in an adversarial setting that allowed the successes of discriminative models to be used to train generative models that produced realistic samples.
The ﬁrst
GAN pitted a two class real/fake discriminator against a generator network and real images [1].
Since then a multitude of innovations in the architectures of these networks, and loss functions for how to combine them, have ﬂourished [2] with the chief goals of improving sample diversity and quality [3].
Many works have found new ways to incorporate class information into the GAN training process to improve image generation [4, 5, 6, 7, 8, 9].
In this work we focus on K+1 GANs which combine the task of real/fake discrimination with class discrimination directly [9].
K+1 GANs proved most successful in improving semi-supervised learning classiﬁcation performance [9, 10, 7], but have more recently been challenged by a different class of methods [11, 12].
They have never been as widely successful as GANs with auxiliary classiﬁers (ACGAN) [4], or GANs that use class embeddings without the classiﬁcation task [6, 13, 14].
Here we investigate why the GAN architecture which most closely uniﬁes real sample classiﬁcation with fake sample classiﬁcation has been overtaken by other methods.
We study a hypothesis testing inspired extension of the binary GAN into the K+1 setting [9], where
K real distributions are classiﬁed against each other, and against a fake distribution.
We show how the optimal discriminator/classiﬁer in this setting is a straightforward generalization of the optimal binary GAN discriminator, and show that the formulation of the log likelihood loss prevents the generator from fully beneﬁting from supervision over the classes of real samples.
Then we then introduce another generator loss criterion that uses dynamic labeling [7], and describe how it has increased class speciﬁcity over the K+1 GAN by emphasizing the modes of the real data.
Finally we demonstrate with experiments on CelebA and CIFAR10 to what degree both GAN formulations have an effect on increasing the quality and diversity of samples over the original binary GAN and more popular ACGAN.
GANs and their multi-class generalizations 2.1 Binary GANs and K+1 GANs
The generalization of the binary GAN [1], known as the K+1 GAN (and also as LabelGAN [7]), was originally developed for semi-supervised learning with GANs [9].
We let M = K + 1 denote the total number of classes, K real classes plus the fake class, which we refer to as the zeroth class.
This generalization echoes the form of the generalization of binary minimax hypothesis testing to be
M-ary [15].
We use the notation that X, H, D, Z are respectively the images, hypotheses (fake, real class 1, real class 2, etc), decisions, and random codes.
Our discriminator d ( x ) outputs the vector d m ( x )
=
P [ D = m | X
= x ] , m = 0 , 1 , . . .
, M − 1 .
The objective function of the binary GAN naturally generalizes to training all outputs of the discriminator:
[Equation]
[Equation] We denote the fake and real densities as f h ( x ) =
P [ X = x | H = h ] , h = 0 , 1 , . . .
, K and p m is the prior on each hypothesis (often taken to be uniform).
For every image x the optimal decision vector d ∗ ( x ) exists and each of its M elements is given by
[-(cid:80) M − 1 i =
 p i f i ( x ) .-][Equation]
This can be proved for discrete densities using the classic optimality criterion for differentiable convex objectives [16].
Plugging in d ∗ ( x ) into Equation (1) we see:
[Equation]
where f avg ( x ) =
(cid:80) M − 1 m =
p m f m ( x ) .
The task of the generator is min f 0 L K+
( d ∗ , g ) , which for discrete PMFs f m is a convex optimization problem, the solution is p 0 f 0 ( x ) → f real ( x ) = (cid:80) M −m =1 p m f m ( x ) .
This is consistent with the result from [1] for M = 2 and p
= p 1 = 1 / 2 .
The typical intuition in training neural networks is that learning from more data and sharing parameters across tasks leads to better performance.
But here, surprisingly the theory states that by using class information in the K + 1 GAN to train a discriminator that can discriminate between classes, and between fake images and each of the classes, is no better than ignoring the class information.
That is, if we have labeled classes for data, but choose to throw it away by merging f 1 , . .
, f K into a single f real , the generator of the binary GAN that we train should have the same distribution f 0 as if we trained the K+1 GAN.
This result is also consistent with examining the gradients of the K+1 GAN which reveal a "overlaid-gradient" problem whereby the overall gradient w.r.t.
a generated example is the same as that in the binary GAN [7].
The K + 1 GAN’s original purpose was however to improve the state of the art in semi-supervised classiﬁcation [9], rather than improve generation of images (though human annotators were said to prefer images from a K+1 GAN).
We note that in Equation (1) we omitted the part of the loss for unlabeled real images.
Nevertheless we have shown that counter-intuitively, training a fully supervised K + 1 discriminator to classify the real examples in parallel with telling fake examples apart from real examples does not change the generator learned.
2.2 Mode Emphasizing GAN via dynamic labeling
One extension that increases the class speciﬁcity of the generator learned by a K+1 GAN is to introduce dynamic labeling [7].
Such a choice leads to f 0 converging to emphasize the modes of f 1 , . . .
, f K , rather than being the average of them f real .
Looking at the last integral in Equation (1), we notice that the g ∗ that solves min g E [log P
[ D = 0 |
X ] | H = 0] also solves min g − E
[log P [ D (cid:54) =
| X ] | H = 0]
=
min g − E [log (cid:80) m> 0 P
[ D = m |
X ] | H = 0] .
This equivalence provides the intuition for why the density of the generator will be the average of all the real distributions.
Let: L Dyn ( d, g ) =
− E [log max l> 0 P [ D = l | X ] | H = 0] .
That is for each fake g ( z ) we look at the "most likely mistake" the discriminator makes d ( g ( z )) , and train g to emphasize it.
For d we keep the K+1 GAN objective unchanged.
If we let d ∗ be optimal in the sense of the previous section, then
[Equation]
[Equation] Comparing this to Equation (2), our intuition expects f 0 ( x ) ≈ max m> 0 f m ( x ) with Dynamic
Labeling instead of f 0 ( x ) = (cid:80) m> 0 f m ( x ) in K+1 GAN for the following reason.
We note that min f 0 KL ( f 0 || ptwise max m> 0 f m ) upper bounds Equation (3), for discrete PMFs it is a convex optimization problem, and the solution is f 0 ( x ) → max m> 0 f m ( x ) .
Dynamic labeling GAN was found to be experimentally preferable to competing non-dynamically labeled GANs in the sense of Inception Score (IS) [7].
Our analysis here, which looks at [-theunconditional K+
-]the unconditional K+1 GAN with dynamic labeling, explains why but also reveals a potential failure mode.
IS is proportional to the KL divergence of P ( Y | X ) and P ( Y ) for images X and labels Y from a pretrained Inception v3 network [9, 17].
Thus a generator that emphasizes all the modes (classes) of the real data distribution should generate easily recognizable images and have low entropy P ( Y | X ) and have a high entropy P ( Y ) by displaying all the classes, giving a high IS.
However emphasizing the modes of the data could also result in lower observed diversity from ﬁnite samplings of g since the density of f 0 is concentrated around the modes.
For this reason we also refer to this GAN as "Mode GAN".
2.3 Other multi-class GANs
In this work we focus on the properties of incorporating class information into GANs via a K +architecture.
Some other notable GAN architectures that utilize class speciﬁc information include Auxiliary Classiﬁer GANs (ACGANs) [4], and GANs that use projection discrimination with spectral normalization (SNGAN) [6]. ACGANs use a standard binary GAN architecture and train an additional fully connected classiﬁcation layer on the penultimate features of the discriminator. ACGANs don’t completely unify the tasks of discrimination and classiﬁcation like K + 1 GANs, but parameters are shared between the two tasks.
In projection discrimination GANs, the one hot class label is embedded with a spectrally normalized matrix, and the penultimate features of a binary discriminator network are projected onto this embedding (dot product of two vectors) and added to the binary discriminator output. Projection discrimination has become the state of the art way to incorporate class information in GANs [14, 18]. Interestingly, these learned embeddings are not naturally accurate classiﬁers by themselves, but adding a training objective to improve their classiﬁcation ability can improve GAN metrics like IS and FID further [19].  Experiments and discussion 3.1 Datasets and methods
We demonstrate our ﬁndings on K+1 GANs using three datasets: synthetic 2D Gaussians, CelebA, and CIFAR10.
Our Gaussian data is created from 10 highly overlapping 2D Gaussians whose means are distributed uniformly on a circle, samples from different Gaussians are different "classes".
Samples from all the classes are shown in Figure 1i.
CelebA is a dataset with 40 binary attribute annotations and 5 landmark locations.
The images we use are 64 × 64 and have the background cropped.
We split this dataset into 5 classes:
Front facing female/male with mouth open/closed are 4, and all faces rotated more than 15 degrees are the 5th.
CIFAR10 is a 32 × 32 dataset with 10 classes.
We use all 50,000 images for training, and the 10 classes are the labels for our supervised models.
[Figure] We train three SN GAN networks [13] (Binary GAN, ACGAN, Mode GAN) with default hyperparameters and only vary the loss function used.
We also train two additional SN GAN networks (Mode
GAN and K+1 GAN) that also used projection discrimination [20].
All networks were trained for 200k iters for CelebA and 500k iters for CIFAR10.
Quantitative evaluation of GANs is itself an active area of research and many information measures for GANs exist. We focus on the two most popular, Inception Score (IS) and Frechet Inception
Distance (FID) [9, 21].
The Inception Score calculates KL( P ( Y | X ) || P ( Y )) , where P is a reference pretrained Inception v3 network.
It evaluates whether generated images can be classiﬁed with conﬁdence by the reference network.
Frechet Inception distance calculates || µ r − µ g || +Tr(Cov r + Cov g − 2 (cid:112) Cov r Cov g ) from the penultimate features of the same reference network. That is µ r and µ g are the mean feature vectors of the real images and generated images respectively, and Cov r and
Cov g are the covariance matrices of the features of real and generated images respectively.
3.2 Discussion
When the form of the true densities f m ( x ) are known, we can observe the learned density of the generator f 0 via the histogram of its samples.
For the Dynamically Labeled GAN we see the PMF that minimizes the convex upper-bound of Equation (3) in Figure 1ii emphasizes the modes of the data.
In Figure 1iii we see that the learned g from minimizing Equation (3) while training a GAN indeed emphasizes the modes of the data: the samples are in high density around the means of each class, and are low density in the overlap between classes.
However, this simple example does not generalize neatly to real data.
We may expect an analogous situation in the domain of faces to be that few samples exist between the "means" of the 5 classes we chose: i.e. no interpolations between males facing left and females facing right.
But we see in Figure 1iv that Dynamically Labeled GAN generates unwanted low quality samples just like the baseline Binary GAN and ACGAN networks.
To measure the quality of our CelebA GAN’s output, we use the conﬁdence output of MTCNN, a publicly available pretrained face detector, to measure the frequency of bad generator samples.
In
Figure 1iv are examples of images that MTCNN assigns 0 conﬁdence output by all 3 GANs.
For images that are far away from the modes of the CelebA data density, for example images that blur together multiple face orientations into 1 image, MTCNN will assign a low conﬁdence.
The right hand side of Figure 2 shows out of 100,000 samples generated by each GAN, the percentage of images with MTCNN conﬁdence greater than c = 0 , 1 − 10 − 1 , . .
, 1 − 10 − 4 is always largest in
Mode GAN, but only by a small margin.
And as shown in the ﬁgure, all the networks produce a score 0 face 4-6 times every 10000 images.
The left hand side of Figure 2 shows that performance was tied for the three networks on CelebA and
CIFAR10.
Adding projection discrimination (PJ) to the networks yeilded a much greater difference in IS performance than the changes in loss function.
We show some samples from the [-CIFAR
-]CIFAR
GANs in Figure 3, which are hard to qualitatively rank. Method FID IS
CelebA
CIFARSN GAN 4.
SN ACGAN 5. SN Mode GAN 3. PJ [Figure]
Mode GAN 8.
PJ
[Figure]
GAN 8.
[Figure]
In iv we plot the mean pairwise distance the outputs the three
That is for in then
To measure diversity within a batch of generated images, we embed the images in a feature space with an auxiliary network, calculate all the pairwise distances, and record the mean.
In Figure 3iv we see Mode GAN always has a higher mean pairwise distance only within a margin of error.
Thus we don’t observe that ﬁtting the modes of real data yields a big impact on quality or diversity of samples.
Conclusion
This work provided a hypothesis testing perspective on the canonical binary GAN, its K+1 GAN generalization, and an extension meant to increase the class speciﬁcity during training, Dynamically
Labeled GAN.
We showed that the GAN’s generator is not beneﬁted by the classiﬁcation of real vs real samples in K+1 GANs.
And we demonstrated that Dynamically Labeled GAN emphasizes the modes of the real data distributions, but this has a tenuous link to increased quality and diversity of generated samples.