MGAN: T RAINING G ENERATIVE A DVERSARIAL N ETS WITH M ULTIPLE G ENERATORS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose in this paper a new approach to train the Generative Adversarial Nets (GANs) with a mixture of generators to overcome the mode collapsing problem.
The main intuition is to employ multiple generators, instead of using a single one as in the original GAN.
The idea is simple, yet proven to be extremely effective at covering diverse data modes, easily overcoming the mode collapsing problem and delivering state-of-the-art results.
A minimax formulation was able to establish among a classiﬁer, a discriminator, and a set of generators in a similar spirit with
GAN.
Generators create samples that are intended to come from the same distribution as the training data, whilst the discriminator determines whether samples are true data or generated by generators, and the classiﬁer speciﬁes which generator a sample comes from.
The distinguishing feature is that internal samples are created from multiple generators, and then one of them will be randomly selected as ﬁnal output similar to the mechanism of a probabilistic mixture model.
We term our method Mixture Generative Adversarial Nets (MGAN).
We develop theoretical analysis to prove that, at the equilibrium, the Jensen-Shannon divergence (JSD) between the mixture of generators’ distributions and the empirical data distribution is minimal, whilst the JSD among generators’ distributions is maximal, hence effectively avoiding the mode collapsing problem.
By utilizing parameter sharing, our proposed model adds minimal computational cost to the standard GAN, and thus can also efﬁciently scale to large-scale datasets.
We conduct extensive experiments on synthetic 2D data and natural image databases (CIFAR-10, STL-10 and
ImageNet) to demonstrate the superior performance of our MGAN in achieving state-of-the-art Inception scores over latest baselines, generating diverse and appealing recognizable objects at different resolutions, and specializing in capturing different types of objects by the generators.
I NTRODUCTION
Generative Adversarial Nets (GANs) (Goodfellow et al., 2014) are a recent novel class of deep generative models that are successfully applied to a large variety of applications such as image, video generation, image inpainting, semantic segmentation, image-to-image translation, and text-to-image synthesis, to name a few (Goodfellow, 2016).
From the game theory metaphor, the model consists of a discriminator and a generator playing a two-player minimax game, wherein the generator aims to generate samples that resemble those in the training data whilst the discriminator tries to distinguish between the two as narrated in (Goodfellow et al., 2014).
Training GAN, however, is challenging as it can be easily trapped into the mode collapsing problem where the generator only concentrates on producing samples lying on a few modes instead of the whole data space (Goodfellow, 2016).
Many GAN variants have been recently proposed to address this problem.
They can be grouped into two main categories: training either a single generator or many generators.
Methods in the former include modifying the discriminator’s objective (Salimans et al., 2016;
Metz et al., 2016), modifying the generator’s objective (Warde-Farley & Bengio, 2016), or employing additional discriminators to yield more useful gradient signals for the generators (Nguyen et al., 2017; Durugkar et al., 2016).
The common theme in these variants is that generators are shown, at equilibrium, to be able to recover the data distribution, but convergence remains elusive in practice.
Most experiments are conducted on toy datasets or on narrow-domain datasets such as LSUN (Yu et al., 2015) or CelebA (Liu et al., 2015).
To our knowledge, only Warde-Farley & Bengio (2016) and Nguyen et al.
(2017) perform quantitative evaluation of models trained on much more diverse datasets such as STL- (Coates et al., 2011) and ImageNet (Russakovsky et al., 2015).
Given current limitations in the training of single-generator GANs, some very recent attempts have been made following the multi-generator approach.
Tolstikhin et al.
(2017) apply boosting techniques to train a mixture of generators by sequentially training and adding new generators to the mixture.
However, sequentially training many generators is computational expensive.
Moreover, this approach is built on the implicit assumption that a single-generator GAN can generate very good images of some modes, so reweighing the training data and incrementally training new generators will result in a mixture that covers the whole data space.
This assumption is not true in practice since current single-generator GANs trained on diverse datasets such as ImageNet tend to generate images of unrecognizable objects.
Arora et al.
(2017) train a mixture of generators and discriminators, and optimize the minimax game with the reward function being the weighted average reward function between any pair of generator and discriminator.
This model is computationally expensive and lacks a mechanism to enforce the divergence among generators.
Ghosh et al.
(2017) train many generators by using a multi-class discriminator that, in addition to detecting whether a data sample is fake, predicts which generator produces the sample.
The objective function in this model punishes generators for generating samples that are detected as fake but does not directly encourage generators to specialize in generating different types of data.
We propose in this paper a novel approach to train a mixture of generators.
Unlike aforementioned multi-generator GANs, our proposed model simultaneously trains a set of generators with the objective that the mixture of their induced distributions would approximate the data distribution, whilst encouraging them to specialize in different data modes.
The result is a novel adversarial architecture formulated as a minimax game among three parties: a classiﬁer, a discriminator, and a set of generators.
Generators create samples that are intended to come from the same distribution as the training data, whilst the discriminator determines whether samples are true data or generated by generators, and the classiﬁer speciﬁes which generator a sample comes from.
We term our proposed model as Mixture Generative Adversarial Nets (MGAN) .
We provide analysis that our model is optimized towards minimizing the Jensen-Shannon Divergence (JSD) between the mixture of distributions induced by the generators and the data distribution while maximizing the JSD among generators.
Empirically, our proposed model can be trained efﬁciently by utilizing parameter sharing among generators, and between the classiﬁer and the discriminator.
In addition, simultaneously training many generators while enforcing JSD among generators helps each of them focus on some modes of the data space and learn better.
Trained on CIFAR-10, each generator learned to specialize in generating samples from a different class such as horse, car, ship, dog, bird or airplane.
Overall, the models trained on the CIFAR-10, STL-10 and ImageNet datasets successfully generated diverse, recognizable objects and achieved state-of-the-art Inception scores (Salimans et al., 2016).
The model trained on the CIFAR-10 even outperformed GANs trained in a semi-supervised fashion (Salimans et al., 2016; Odena et al., 2016).
In short, our main contributions are: (i) a novel adversarial model to efﬁciently train a mixture of generators while enforcing the JSD among the generators; (ii) a theoretical analysis that our objective function is optimized towards minimizing the JSD between the mixture of all generators’ distributions and the real data distribution, while maximizing the JSD among generators; and (iii) a comprehensive evaluation on the performance of our method on both synthetic and real-world large-scale datasets of diverse natural scenes.
G ENERATIVE A DVERSARIAL N ETS
Given the discriminator D and generator G , both parameterized via neural networks, training GAN can be formulated as the following minimax objective function:
[Equation]
where x is drawn from data distribution P data , z is drawn from a prior distribution P z .
The mapping
G ( z ) induces a generator distribution P model in data space.
GAN alternatively optimizes D and G using stochastic gradient-based learning.
As a result, the optimization order in 1 can be reversed, causing the minimax formulation to become maximin.
G is therefore incentivized to map every z to a single x that is most likely to be classiﬁed as true data, leading to mode collapsing problem.
Another
[Table]
[Figure]
(cid:12)(cid:8)(cid:2)(cid:14)(cid:8)(cid:9)(cid:6)(cid:11)(cid:5)(cid:11)(cid:15)(cid:13)(cid:4)(cid:16)(cid:15)(cid:9)(cid:12)(cid:13)(cid:3)(cid:9)(cid:7)(cid:3)(cid:11)(cid:1)(cid:17) commonly asserted cause of generating less diverse samples in GAN is that, at the optimal point of
D , minimizing G is equivalent to minimizing the JSD between the data and model distributions, which has been empirically proven to prefer to generate samples around only a few modes whilst ignoring other modes (Husz ´ ar, 2015; Theis et al., 2015).
P ROPOSED M IXTURE GAN S
We now present our main contribution of a novel approach that can effectively tackle mode collapse in GAN.
Our idea is to use a mixture of many distributions rather than a single one as in the standard
GAN, to approximate the data distribution, and simultaneously we enlarge the divergence of those distributions so that they cover different data modes.
To this end, an analogy to a game among K generators G 1:K , a discriminator D and a classiﬁer C can be formulated.
Each generator G k maps z to x = G k ( z ) , thus inducing a single distribution
P G k ; and K generators altogether induce a mixture over K distributions, namely P model in the data space.
An index u is drawn from a multinomial distribution Mult ( π ) where π
=
[ π 1 , π 2 , ..., π K ] is the coefﬁcients of the mixture; and then the sample G u ( z ) is used as the output.
Here, we use a predeﬁned π and ﬁx it instead of learning.
The discriminator D aims to distinguish between this sample and the training samples.
The classiﬁer C performs multi-class classiﬁcation to classify samples labeled by the indices of their corresponding generators.
We term this whole process and our model the Mixture Generative Adversarial Nets (MGAN).
Fig.
1 illustrates the general architecture of our proposed MGAN, where all components are parameterized by neural networks.
G k (s) tie their parameters together except the input layer, whilst C and
D share parameters except the output layer.
This parameter sharing scheme enables the networks to leverage their common information such as features at low-level layers that are close to the data layer, hence helps to train model effectively.
In addition, it also minimizes the number of parameters and adds minimal complexity to the standard GAN, thus the whole process is still very efﬁcient.
More formally, D , C and G 1:K now play the following multi-player minimax optimization game:
[Equation]
where C k ( x ) is the probability that x is generated by G k and β > 0 is the diversity hyper-parameter.
The ﬁrst two terms show the interaction between generators and the discriminator as in the standard
GAN.
The last term should be recognized as a standard softmax loss for a multi-classiﬁcation setting, which aims to maximize the entropy for the classiﬁer.
This represents the interaction between generators and the classiﬁer, which encourages each generator to produce data separable from those produced by other generators.
The strength of this interaction is controlled by β .
Similar to GAN, our proposed network can be trained by alternatively updating
3. T HEORETICAL A NALYSIS
Assuming all C , D and G 1:K have enough capacity, we show below that at the equilibrium point of the minimax problem in Eq.
(2), the JSD between the mixture induced by G 1:K and the data distribution is minimal, i.e. p data = p model , and the JSD among K generators is maximal, i.e. two arbitrary generators almost never produce the same data.
In what follows we present our mathematical statement and the sketch of their proofs.
We refer to Appendix B for full derivations.
Proposition 1.
For ﬁxed generators G 1 , G 2 , ..., G K and their mixture weights π 1 , π 2 , ..., π K , the optimal solution C ∗ =
C ∗ 1:K and D ∗ for J ( G 1:K , C, D ) in Eq.
(2) are:
[Equation]
Proof.
It can be seen that the solution C ∗ k is a general case of D ∗ when D classiﬁes samples from two distributions with equal weight of 1 / 2 .
We refer the proofs for D ∗ to Prop. 1 in (Goodfellow et al., 2014), and our proof for C ∗ k to Appendix B in this manuscript.
Based on Prop. 1, we further show that at the equilibrium point of the minimax problem in
Eq.
(2), the optimal generator G ∗ =
[ G ∗ 1 , ..., G ∗ K ] induces the generated distribution p ∗ model ( x )
= (cid:80) K k
=
π k p G ∗ k ( x ) which is as closest as possible to the true data distribution p data ( x ) while maintaining the mixture components p G ∗ k
( x ) (s) as furthest as possible to avoid the mode collapse.
Theorem 2.
At the equilibrium point of the minimax problem in Eq.
(2), the optimal G ∗ , D ∗ , and
C ∗ satisfy
[Equation]
Proof.
Substituting C ∗ 1:K and D ∗ into Eq.
[Equation]
we reformulate the objective function for G 1:K as follows:
[Equation]
Since the last two terms in Eq.
(4) are constant
[Equation]
that concludes our proof.
This theorem shows that progressing towards the equilibrium is equivalently to minimizing
JSD ( P data (cid:107) P model ) while maximizing JSD π ( P G 1 , P G 2 , ..., P G K ) .
In the next theorem, we further clarify the equilibrium point for the speciﬁc case wherein the data distribution has the form p data ( x ) =
(cid:80) K k
=
π k q k ( x ) where the mixture components q k
( x ) (s) are well-separated in the sense that E x ∼ Q k
[ q j ( x )] = 0 for j (cid:54)
=
k , i.e., for almost everywhere x , if q k ( x ) > 0 then q j ( x ) = 0 , ∀ j (cid:54) =
k .
Theorem 3.
If the data distribution has the form: p data ( x ) =
(cid:80) K k
=
π k q k ( x ) where the mixture components q k
( x ) (s) are well-separated, the minimax problem in Eq.
(2) or the optimization problem in Eq.
(3) has the following solution:
[Equation]
, and the corresponding objective value of the optimization problem in Eq.
(3) is − β H ( π )
= − β (cid:80) K k
=1 π k log 1 π k , where H ( π ) is the Shannon entropy.
Proof.
Please refer to our proof in Appendix B of this manuscript.
Thm. 3 explicitly offers the optimal solution for the speciﬁc case wherein the real data are generated from a mixture distribution whose components are well-separated.
This further reveals that if the mixture components are well-separated, by setting the number of generators as the number of mixtures in data and maximizing the divergence between the generated components p G k ( x ) (s), we can exactly recover the mixture components q k ( x ) (s) using the generated components p G k ( x ) (s), hence strongly supporting our motivation when developing MGAN.
In practice, C , D , and G 1:K are parameterized by neural networks and are optimized in the parameter space rather than in the function space.
As all generators G 1:K share the same objective function, we can efﬁciently update their weights using the same backpropagation passes.
Empirically, we set the parameter π k = 1K , ∀ k ∈ { 1 , ..., K } , which further minimizes the objective value − β H ( π ) = − β (cid:80) K k =
π k log 1 π k w.r.t π in Thm. 3.
To simplify the computational graph, we assume that each generator is sampled the same number of times in each minibatch.
In addition, we adopt the non-saturating heuristic proposed in (Goodfellow et al., 2014) to train G 1:K by maximizing log D ( G k ( z )) instead of minimizing log D (1 − G k ( z )) .
R ELATED W ORK
Recent attempts to address the mode collapse by modifying the discriminator include minibatch discrimination (Salimans et al., 2016), Unrolled GAN (Metz et al., 2016) and Denoising Feature
Matching (DFM) (Warde-Farley & Bengio, 2016).
The idea of minibatch discrimination is to allow the discriminator to detect samples that are noticeably similar to other generated samples.
Although this method can generate visually appealing samples, it is computationally expensive, thus normally used in the last hidden layer of discriminator.
Unrolled GAN improves the learning by unrolling computational graph to include additional optimization steps of the discriminator.
It could effectively reduce the mode collapsing problem, but the unrolling step is expensive, rendering it unscalable up to large-scale datasets.
DFM augments the objective function of generator with one of a Denoising AutoEncoder (DAE) that minimizes the reconstruction error of activations at the penultimate layer of the discriminator.
The idea is that gradient signals from DAE can guide the generator towards producing samples whose activations are close to the manifold of real data activations.
DFM is surprisingly effective at avoiding mode collapse, but the involvement of a deep DAE adds considerable computational cost to the model.
An alternative approach is to train additional discriminators.
D2GAN (Nguyen et al., 2017) employs two discriminators to minimize both Kullback-Leibler (KL) and reverse KL divergences, thus placing a fair distribution across the data modes.
This method can avoid the mode collapsing problem to a certain extent, but still could not outperform DFM.
Another work uses many discriminators to boost the learning of generator (Durugkar et al., 2016).
The authors state that this method is robust to mode collapse, but did not provide experimental results to support that claim.
Another direction is to train multiple generators.
The so-called MIX+GAN (Arora et al., 2017) is related to our model in the use of mixture but the idea is very different.
Based on min-max theorem (Neumann, 1928), the MIX+GAN trains a mixture of multiple generators and discriminators with different parameters to play mixed strategies in a min-max game.
The total reward of this game is computed by weighted averaging rewards over all pairs of generator and discriminator.
The lack of parameter sharing renders this method computationally expensive to train.
Moreover, there is no mechanism to enforce the divergence among generators as in ours.
Some attempts have been made to train a mixture of GANs in a similar spirit with boosting algorithms.
Wang et al.
(2016) propose an additive procedure to incrementally train new GANs on a subset of the training data that are badly modeled by previous generators.
As the discriminator is expected to classify samples from this subset as real with high conﬁdence, i.e. D ( x ) is high, the subset can be chosen to include x where D ( x ) is larger than a predeﬁned threshold.
Tolstikhin et al.
(2017), however, show that this heuristic fails to address the mode collapsing problem.
Thus they propose AdaGAN to introduce a robust reweighing scheme to prepare training data for the next
GAN.
AdaGAN and boosting-inspired GANs in general are based on the assumption that a singlegenerator GAN can learn to generate impressive images of some modes such as dogs or cats but fails to cover other modes such as giraffe.
Therefore, removing images of dogs or cats from the training data and train a next GAN can create a better mixture.
This assumption is not true in practice as current single-generator GANs trained on diverse data sets such as ImageNet (Russakovsky et al., 2015) tend to generate images of unrecognizable objects.
The most closely related to ours is MAD-GAN (Ghosh et al., 2017) which trains many generators and uses a multi-class classiﬁer as the discriminator.
In this work, two strategies are proposed to address the mode collapse: (i) augmenting generator’s objective function with a user-deﬁned similarity based function to encourage different generators to generate diverse samples, and (ii) modifying discriminator’s objective functions to push different generators towards different identiﬁable modes by separating samples of each generator.
Our approach is different in that, rather than modifying the discriminator, we use an additional classiﬁer that discriminates samples produced by each generator from those by others under multi-class classiﬁcation setting.
This nicely results in an optimization problem that maximizes the JSD among generators, thus naturally enforcing them to generate diverse samples and effectively avoiding mode collapse.
E XPERIMENTS
In this section, we conduct experiments on both synthetic data and real-world large-scale datasets.
The aim of using synthetic data is to visualize, examine and evaluate the learning behaviors of our proposed MGAN, whilst using real-world datasets to quantitatively demonstrate its efﬁcacy and scalability of addressing the mode collapse in a much larger and wider data space.
For fair comparison, we use experimental settings that are identical to previous work, and hence we quote the results from the latest state-of-the-art GAN-based models to compare with ours.
We use TensorFlow (Abadi et al., 2016) to implement our model and will release the code after publication.
For all experiments, we use: (i) shared parameters among generators in all layers except for the weights from the input to the ﬁrst hidden layer; (ii) shared parameters between discriminator and classiﬁer in all layers except for the weights from the penultimate layer to the output; (iii) Adam optimizer (Kingma & Ba, 2014) with learning rate of 0.0002 and the ﬁrst-order momentum of 0.5; (iv) minibatch size of 64 samples for training discriminators; (v) ReLU activations (Nair & Hinton, 2010) for generators; (vi) Leaky ReLU (Maas et al., 2013) with slope of 0.2 for discriminator and classiﬁer; and (vii) weights randomly initialized from Gaussian distribution N (0 , 0 .
02 I ) and zero biases.
We refer to Appendix C for detailed model architectures and additional experimental results.
5. S YNTHETIC DATA
In the ﬁrst experiment, following (Nguyen et al., 2017) we reuse the experimental design proposed in (Metz et al., 2016) to investigate how well our MGAN can explore and capture multiple data modes.
The training data is sampled from a 2D mixture of 8 isotropic Gaussian distributions with a covariance matrix of 0.02 I and means arranged in a circle of zero centroid and radius of 2.0.
Our purpose of using such small variance is to create low density regions and separate the modes.
We employ 8 generators, each with a simple architecture of an input layer with 256 noise units drawn from isotropic multivariate Gaussian distribution N (0 , I ) , and two fully connected hidden layers with 128 ReLU units each.
For the discriminator and classiﬁer, one hidden layer with
ReLU units is used.
The diversity hyperparameter β is set to 0.125.
Fig.
2c shows the evolution of 512 samples generated by our model and baselines through time.
It can be seen that the regular GAN generates data collapsing into a single mode hovering around the valid modes of data distribution, thus reﬂecting the mode collapse in GAN as expected.
At the same time, UnrolledGAN (Metz et al., 2016), D2GAN (Nguyen et al., 2017) and our MGAN distribute data around all 8 mixture components, and hence demonstrating the abilities to successfully learn multimodal data in this case.
Our proposed model, however, converges much faster than the other two since it successfully explores and neatly covers all modes at the early step 15K, whilst two baselines produce samples cycling around till the last steps.
At the end, our MGAN captures data modes more precisely than UnrolledGAN and D2GAN since, in each mode, the UnrolledGAN
. .
. .
. 6WHS     6 \ PP H W U L F  .
/  G L Y *$1 8QUROOHG*$1 '*$1 0*$ (
[Figure]
.
. .
.
. 6WHS     
 : D VV H U V W H L Q  H V W L P D W H *$18QUROOHG*$1'*$10*$
[Figure]
[Figure]
step 5K step 10 K step 15 K [Figure] 20 K
[Figure]
K
[Figure]
generates data that concentrate only on several points around the mode’s centroid, thus seems to produce fewer samples than ours whose samples fairly spread out the entire mode, but not exceed the boundary whilst the D2GAN still generates many points scattered between two adjacent modes.
Next we further quantitatively compare the quality of generated data.
Since we know the true distribution P data in this case, we employ two measures, namely Wasserstein distance and symmetric Kullback-Leibler (KL) divergence, which is the average of KL and reverse KL.
These measures compute the distance between the normalized histograms of 10,000 points generated from the model to true P data .
Figs.
2a and 2b again clearly demonstrate the superiority of our approach over GAN, UnrolledGAN and D2GAN w.r.t both distances (lower is better); notably the Wasserstein distances from ours and D2GAN’s to the true distribution almost reduce to zero, and at the same time, our symmetric KL metric is signiﬁcantly better than that of D2GAN.
These ﬁgures also show the stability of our MGAN (black curves) and D2GAN (red curves) during training as they are much less ﬂuctuating compared with GAN (green curves) and UnrolledGAN (blue curves).
Lastly, we perform experiments with different numbers of generators.
The MGAN models with 2, 3, 4 and 10 generators all successfully explore 8 modes but the models with more generators generate fewer points scattered between adjacent modes.
We also examine the behavior of the diversity coefﬁcient β by training the 4-generator model with different values of β .
Without the JSD force ( β = 0 ), generated samples cluster around one mode.
When β = 0 .
25 , the JSD force is weak and generated data cluster near 4 different modes.
When β = 0 .
75 or 1 . 0 , the JSD force is too strong and causes the generators to collapse, generating 4 increasingly tight clusters.
When β = 0 .
5 , generators successfully cover all of the 8 modes.
Please refer to Appendix C.1 for experimental details.
5. R EAL - WORLD D ATASETS
Next we train our proposed method on real-world databases from natural scenes to investigate its performance and scalability on much more challenging large-scale image data.
Datasets.
We use 3 widely-adopted datasets: CIFAR-10 (Krizhevsky & Hinton, 2009), STL- (Coates et al., 2011) and ImageNet (Russakovsky et al., 2015).
CIFAR-10 contains 50,000 32 ×training images of 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
STL-10, subsampled from ImageNet, is a more diverse dataset than CIFAR-10, containing about 100,000 96 × 96 images.
ImageNet (2012 release) presents the largest and most diverse consisting of over 1.2 million images from 1,000 classes.
In order to facilitate fair comparison with the baselines in (Warde-Farley & Bengio, 2016; Nguyen et al., 2017), we follow the procedure of (Krizhevsky et al., 2012) to resize the STL-10 and ImageNet images down to 48 × 48 and 32 × 32, respectively.
Evaluation protocols.
For quantitative evaluation, we adopt the Inception score proposed in (Salimans et al., 2016), which computes exp ( E x [ KL ( p ( y | x ) (cid:107) p ( y ))]) where p ( y | x ) is the conditional label distribution for the image x estimated by the reference Inception model (Szegedy et al., 2015).
This metric rewards good and varied samples and is found to be well-correlated with human judgment (Salimans et al., 2016).
We use the code provided in (Salimans et al., 2016) to compute the
Inception scores for 10 partitions of 50,000 randomly generated samples.
For qualitative demonstration of image quality obtained by our proposed model, we show samples generated by the mixture as well as samples produced by each generator.
Samples are randomly drawn rather than cherry-picked.
Model architectures.
Our generator and discriminator architectures closely follow the DCGAN’s design (Radford et al., 2015).
The only difference is we apply batch normalization (Ioffe & Szegedy, 2015) to all layers in the networks except for the output layer.
Regarding the classiﬁer, we empirically ﬁnd that our proposed MGAN achieves the best performance (i.e., fast convergence rate and high inception score) when the classiﬁer shares parameters of all layers with the discriminator except for the output layer.
The reason is that this parameter sharing scheme would allow the classiﬁer and discriminator to leverage their common features and representations learned at every layer, thus helps to improve and speed up the training progress.
When the parameters are not tied, the model learns slowly and eventually yields lower performance.
During training we observe that the percentage of active neurons chronically declined (see Appendix C.2).
One possible cause is that the batch normalization center (offset) is gradually shifted to the negative range, thus deactivating up to 45% of ReLU units of the generator networks.
Our ad-hoc solution for this problem is to ﬁx the offset at zero for all layers in the generator networks.
The rationale is that for each feature map, the ReLU gates will open for about 50% highest inputs in a minibatch across all locations and generators, and close for the rest.
We also experiment with other activation functions of generator networks.
First we use Leaky ReLU and obtain similar results with using ReLU.
Then we use MaxOut units (Goodfellow et al., 2013) and achieves good Inception scores but generates unrecognizable samples.
Finally, we try SeLU (Klambauer et al., 2017) but fail to train our model.
Hyperparameters.
Three key hyperparameters of our model are: number of generators K , coefﬁcient β controlling the diversity and the minibatch size.
We use a minibatch size of [ 128 / K ] for each generator, so that the total number of samples for training all generators is about 128.
We train models with 4 generators and 10 generators corresponding with minibatch sizes of 32 andeach, and ﬁnd that models with 10 generators performs better.
For ImageNet, we try an additional setting with 32 generators and a minibatch size of 4 for each.
The batch of 4 samples is too small for updating sufﬁcient statistics of a batch-norm layer, thus we drop batch-norm in the input layer of each generator.
This 32-generator model, however, does not obtain considerably better results than the 10-generator one.
Therefore in what follows we only report the results of models withgenerators.
For the diversity coefﬁcient β , we observe no signiﬁcant difference in Inception scores when varying the value of β but the quality of generated images declines when β is too low or too high.
Generated samples by each generator vary more when β is low, and vary less but become less realistic when β is high.
We ﬁnd a reasonable range for β to be (0 . 01 , 1 . 0) , and ﬁnally set to 0.
for CIFAR-10, 0.1 for ImageNet and 1.0 for STL-10.
Inception results.
We now report the Inception scores obtained by our MGAN and baselines in
Tab. 1.
It is worthy to note that only models trained in a completely unsupervised manner without label information are included for fair comparison; and DCGAN’s and D2GAN’s results on STL10 are available only for the models trained on 32 × 32 resolution.
Overall, our proposed model outperforms the baselines by large margins and achieves state-of-the-art performance on all datasets.
Moreover, we would highlight that our MGAN obtains a score of 8.33 on CIFAR-10 that is even better than those of models trained with labels such as 8.09 of Improved GAN (Salimans et al., 2016) and 8.25 of AC-GAN (Odena et al., 2016).
In addition, we train our model on the original 96 × 96 resolution of STL-10 and achieve a score of 9.79 ± 0.08.
This suggests the MGAN can be successfully trained on higher resolution images and achieve the higher Inception score.
Image generation.
Next we present samples randomly generated by our proposed model trained on the 3 datasets for qualitative assessment.
Fig.
3a shows CIFAR-10 32 × 32 images containing a
[Table]
wide range of objects in such as airplanes, cars, trucks, ships, birds, horses or dogs.
Similarly, STL10 48 × 48 generated images in Fig. 3b include cars, ships, airplanes and many types of animals [Table] but with wider range of different themes such as sky, underwater, mountain and forest.
Images generated for ImageNet 32 × 32 are diverse with some recognizable objects such as lady, old man, birds, human eye, living room, hat, slippers, to name a few.
Fig.
4a shows several cherry-picked STL-10 96 ×images [Table] which demonstrate that the MGAN is capable of generating visually appealing images with complicated details.
However, many samples are still incomplete and unrealistic as shown in Fig.
4b, leaving plenty of room for improvement.
[Figure]
Finally, we investigate samples generated by each generator as well as the evolution of these samples through numbers of training epochs.
Fig.
5 shows images generated by each of the 10 generators in our MGAN trained on CIFAR-10 at epoch 20, 50, and 250 of training.
Samples in each row correspond to a different generator.
Generators start to specialize in generating different types of objects as early as epoch 20 and become more and more consistent: generator 2 and 3 in ﬂying objects (birds and airplanes), generator 4 in full pictures of cats and dogs, generator 5 in portraits of cats and dogs, generator 8 in ships, generator 9 in car and trucks, and generator 10 in horses.
Generator 6 seems to generate images of frog or animals in a bush.
Generator 7, however, collapses in epoch 250.
One possible explanation for this behavior is that images of different object classes tend to have different themes.
Lastly, Wang et al.
(2016) noticed one of the causes for non-convergence in
GANs is that the generators and discriminators constantly vary; the generators at two consecutive epochs of training generate signiﬁcantly different images.
This experiment demonstrates the effect of the JSD force in preventing generators from moving around the data space.
[Figure] [Figure]
C ONCLUSION
We have presented a novel adversarial model to address the mode collapse in GANs.
Our idea is to approximate data distribution using a mixture of multiple distributions wherein each distribution captures a subset of data modes separately from those of others.
To achieve this goal, we propose a minimax game of one discriminator, one classiﬁer and many generators to formulate an optimization problem that minimizes the JSD between P data and P model , i.e., a mixture of distributions induced by the generators, whilst maximizes JSD among such generator distributions.
This helps our model generate diverse images to better cover data modes, thus effectively avoids mode collapse.
We term our proposed model Mixture Generative Adversarial Network (MGAN).
The MGAN can be efﬁciently trained by sharing parameters between its discriminator and classiﬁer, and among its generators, thus our model is scalable to be evaluated on real-world largescale datasets.
Comprehensive experiments on synthetic 2D data, CIFAR-10, STL-10 and ImageNet databases demonstrate the following capabilities of our model: (i) achieving state-of-the-art Inception scores; (ii) generating diverse and appealing recognizable objects at different resolutions; and (iv) specializing in capturing different types of objects by the generators.
A A PPENDIX : F RAMEWORK
In our proposed method generators G 1 , G 2 , ... G K are deep convolutional neural networks parameterized by θ G .
These networks share parameters in all layers except for the input layers.
The input layer for generator G k is parameterized by the mapping f θ G ,k ( z ) that maps the sampled noise z to the ﬁrst hidden layer activation h .
The shared layers are parameterized by the mapping g θ G ( h ) that maps the ﬁrst hidden layer to the generated data.
The pseudo-code of sampling from the mixture is described in Alg. 1.
Classiﬁer C and classiﬁer D are also deep convolutional neural networks that are both parameterized by θ CD .
They share parameters in all layers except for the last layer.
The pseudo-code of alternatively learning θ G and θ CD using stochastic gradient descend is described in
Alg. 2.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Update classiﬁer C and discriminator D by descending along their gradient:
[Table] [Table]
L
[Table] [Table]
[Table] [Table]
[Equation]
, ..., x
[Equation]
(N) (cid:17) and N indices ( u 1 , u 2 , ..., u N ) from the current mixture.
[Equation]
[Equation]
Update the mixture of generators G by ascending along its gradient: ∇ θ G L G .
10: end for
B A PPENDIX : P ROOFS FOR S ECTION 3.
Proposition 1 (Prop. 1 restated).
For ﬁxed generators G 1 , G 2 , ..., G K and mixture weights π 1 , π 2 , ..., π K , the optimal classiﬁer C ∗ =
C ∗ 1:K and discriminator D ∗ for J ( G, C, D ) are:
[Equation]
Proof.
The optimal D ∗ was proved in Prop. 1 in (Goodfellow, 2016).
This section shows a similar proof for the optimal C ∗ .
Assuming that C ∗ can be optimized in the functional space, we can calculate the functional derivatives of J ( G, C, D ) with respect to each C k ( x ) for k ∈ { 2 , ..., K } and set them equal to zero:
[Equation]
Setting δ J ( G,C,D ) δC k ( x ) to 0 for k ∈ { 2 , ..., K } , we get:
[Equation] [Equation]
) results from Eq.
(6) due to the fact that (cid:80) K k
=1 C ∗ k ( x )
= 1 .
Reformulation of L ( G 1:K ) .
Replacing the optimal C ∗ and D ∗ into Eq.
(2), we can reformulate the objective function for the generator as follows:
[Equation]
The sum of the ﬁrst two terms in Eq.
(7) was shown in (Goodfellow et al., 2014) to be 2 ·
JSD ( P data (cid:107) P model ) − log 4 .
The last term β {∗} of Eq.
(7) is related to the JSD for the K distributions:
[Equation]
where H ( P ) is the Shannon entropy for distribution P .
Thus, L ( G 1:K ) can be rewritten as:
[Equation]
Theorem 3 (Thm. 3 restated).
If the data distribution has the form: p data ( x ) =
(cid:80)
Kk =
π k q k ( x ) where the mixture components q k
( x ) (s) are well-separated, the minimax problem in Eq.
(2) or the optimization problem in Eq.
(3) has the following solution:
[Equation] [Equation]
and p model
[Equation]
, and the corresponding objective value of the optimization problem in Eq.
(3) is − β H ( π )
= − β (cid:80)
Kk =
[Equation]
π k log
[Equation]
k .
Proof.
We ﬁrst recap the optimization problem for ﬁnding the optimal G ∗ :
[Equation]
The JSD in Eq.
(8) is given by:
[Equation]
The i -th expectation in Eq.
(9) can be derived as follows:
[Equation]
and the equality occurs if
[Equation]
almost everywhere or equivalently for almost every x except for those in a zero measure set, we have:
[Equation]
Therefore, we obtain the following inequality:
[Equation]
and the equality occurs if for almost every x except for those in a zero measure set, we have:
[Equation]
It follows that
[Equation]
and we peak the minimum if p G k = q k , ∀ k since this solution satisﬁes both
[Equation]
and the conditions depicted in Eq.
That concludes our proof.
C A PPENDIX : A DDITIONAL E XPERIMENTS C. S YNTHETIC 2D G AUSSIAN DATA
The true data is sampled from a 2D mixture of 8 Gaussian distributions with a covariance matrix 0.02 I and means arranged in a circle of zero centroid and radius 2.0.
We use a simple architecture of 8 generators with two fully connected hidden layers and a classiﬁer and a discriminator with one shared hidden layer.
All hidden layers contain the same number of 128 ReLU units.
The input layer of generators contains 256 noise units sampled from isotropic multivariate Gaussian distribution
N (0 , I ) .
We do not use batch normalization in any layer.
We refer to Tab. 2 for more speciﬁcations of the network and hyperparameters.
“Shared” is short for parameter sharing among generators or between the classiﬁer and the discriminator.
Feature maps of 8/1 in the last layer for C and D means that two separate fully connected layers are applied to the penultimate layer, one for C that outputs 8 logits and another for D that outputs 1 logit.
[Table]
N
[Table]
×
[Table]
√
C
[Table]
D (
[Table] [Table]
√
[Table]
real
[Table] [Table]
β =
[Table]
( β 1 = 0 .
5 , β 2 =
[Table] [Table]
[Table]
N ( µ = 0 , σ = 0 .
02 I ) ,
The effect of the number of generators on generated samples.
Fig.
6 shows samples produced by MGANs with different numbers of generators trained on synthetic data for 25,000 epochs.
The model with 1 generator behaves similarly to the standard GAN as expected.
The models with 2, 3 and 4 generators all successfully cover 8 modes, but the ones with more generators draw fewer points scattered between adjacent modes.
Finally, the model with 10 generators also covers 8 modes wherein 2 generators share one mode and one generator hovering around another mode.
[Figure]
The effect of β on generated samples.
To examine the behavior of the diversity coefﬁcient β ,
Fig.
7 compares samples produced by our MGAN with 4 generators after 25,000 epochs of training with different values of β .
Without the JSD force ( β = 0 ), generated samples cluster around one mode.
When β = 0 .
25 , generated data clusters near 4 different modes.
When β = 0 .
75 or 1 . 0 , the JSD force is too strong and causes the generators to collapse, generating 4 increasingly tight clusters.
When β = 0 .
5 , generators successfully cover all of the 8 modes.
[Figure]
C. R EAL - WORLD DATASETS
Fixing batch normalization center.
During training we observe that the percentage of active neurons, which we deﬁne as ReLU units with positive activation for at least 10% of samples in the minibatch, chronically declined.
Fig.
8a shows the percentage of active neurons in generators trained on CIFAR-10 declined consistently to 55% in layer 2 and 60% in layer 3.
Therefore, the quality of generated images, after reaching the peak level, started declining.
One possible cause is that the batch normalization center (offset) is gradually shifted to the negative range as shown in the histogram in Fig.
8b.
We also observe the same problem in DCGAN.
Our ad-hoc solution for this problem, i.e., we ﬁx the offset at zero for all layers in the generator networks.
The rationale is that for each feature map, the ReLU gates will open for about 50% highest inputs in a minibatch across all locations and generators, and close for the rest.
Therefore, batch normalization can keep ReLU units alive even when most of their inputs are otherwise negative, and introduces a form of competition that encourages generators to “specialize” in different features.
This measure signiﬁcantly improves performance but does not totally solve the dying ReLUs problem.
We ﬁnd that late in the training, the input to generators’ ReLU units became more and more right-skewed, causing the ReLU gates to open less and less often. [Figure]
Experiment settings.
For the experiments on three large-scale natural scene datasets (CIFAR10, STL-10, ImageNet), we closely followed the network architecture and training procedure of
DCGAN.
The speciﬁcations of our models trained on CIFAR-10, STL-10 48 × 48, STL-10 96 ×and ImageNet datasets are described in Tabs.
(3, 4, 5, 6), respectively.
“BN” is short for batch normalization and “BN center” is short for whether to learn batch normalization’s center or set it at zero.
“Shared” is short for parameter sharing among generators or between the classiﬁer and the discriminator.
Feature maps of 10/1 in the last layer for C and D means that two separate fully connected layers are applied to the penultimate layer, one for C that outputs 10 logits and another for D that outputs 1 logit.
Finally, Figs.
(9, 10, 11, 12, 13) respectively are the enlarged version of
Figs. (3a, 3b, 3c, 4a, 4b) in the main manuscript.
[Table]
[Table]
[Table]
[Table]