I MPROVING THE I MPROVED T RAINING OF W ASSERSTEIN GAN S : A C ONSISTENCY T ERM AND I TS D UAL E FFECT
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite being impactful on a variety of problems and applications, the generative adversarial nets (GANs) are remarkably difﬁcult to train.
This issue is formally analyzed by Arjovsky & Bottou (2017), who also propose an alternative direction to avoid the caveats in the minmax two-player training of GANs.
The corresponding algorithm, called Wasserstein GAN (WGAN), hinges on the 1-Lipschitz continuity of the discriminator.
In this paper, we propose a novel approach to enforcing the Lipschitz continuity in the training procedure of WGANs.
Our approach seamlessly connects WGAN with one of the recent semi-supervised learning methods.
As a result, it gives rise to not only better photo-realistic samples than the previous methods but also state-of-the-art semi-supervised learning results.
In particular, our approach gives rise to the inception score of more than 5.0 with only 1,000 CIFAR-10 images and is the ﬁrst that exceeds the accuracy of 90% on the
CIFAR-10 dataset using only 4,000 labeled images, to the best of our knowledge.
I NTRODUCTION
We have witnessed a great surge of interests in deep generative networks in recent years (Kingma &
Welling, 2013; Goodfellow et al., 2014; Li et al., 2015).
The central idea therein is to feed a random vector to a ( e.g. , feedforward) neural network and then take the output as the desired sample.
This sampling procedure is very efﬁcient without the need of any Markov chains.
In order to train such a deep generative network, two broad categories of methods are proposed.
The ﬁrst is to use stochastic variational inference (Kingma & Welling, 2013; Rezende et al., 2014;
Kingma et al., 2014) to optimize the lower bound of the data likelihood.
The other is to use the samples as a proxy to minimize the distribution divergence between the model and the real through a two-player game (Goodfellow et al., 2014; Salimans et al., 2016), maximum mean discrepancy (Li et al., 2015; Dziugaite et al., 2015; Li et al., 2017b), f-divergence (Nowozin et al., 2016;
Nock et al., 2017), and the most recent Wasserstein distance (Arjovsky et al., 2017; Gulrajani et al., 2017).
With no doubt, the generative adversarial networks (GANs) among them (Goodfellow et al., 2014) have the biggest impact thus far on a variety of problems and applications (Radford et al., 2015;
Denton et al., 2015; Im et al., 2016; Isola et al., 2016; Springenberg, 2015; Sutskever et al., 2015;
Odena, 2016; Zhu et al., 2017).
GANs learn the generative network (generator) by playing a twoplayer game between the generator and an auxiliary discriminator network.
While the generator has no difference from other deep generative models in the sense that it translates a random vector into a desired sample, it is impossible to calculate the sample likelihood from it.
Instead, the discriminator serves to evaluate the quality of the generated samples by checking how difﬁcult it is to differentiate them from real data points.
However, it is remarkably difﬁcult to train GANs without good heuristics (Goodfellow et al., 2014;
Salimans et al., 2016;
Radford et al., 2015) which may not generalize across different network architectures or application domains.
The training dynamics are often unstable and the generated samples could collapse to limited modes.
These issues are formally analyzed by Arjovsky & Bottou (2017), who also propose an alternative direction (Arjovsky et al., 2017) to avoid the caveats in the minmax two-player training of GANs.
The corresponding algorithm, namely, Wasserstein GAN
(WGAN), shows not only superior performance over GANs but also a nice correlation between the sample quality and the value function that GANs lack.
1. B ACKGROUND : WGAN AND THE I MPROVED T RAINING OF WGAN
WGAN (Arjovsky et al., 2017) aims to learn the generator network G ( z ) , for any random vector z ∼ P z , such that the Wasserstein distance is minimized between the resulting distribution P G of the generated samples { G ( z ) } and the real distribution P r underlying the observed data points { x } ; in other words, min G W ( P r , P G ) .
The Wasserstein distance W ( P r , P G ) is shown a more sensible cost function for learning the distributions supported by low-dimensional manifolds than the other popular distribution divergences and distances — for example, the Jensen-Shannon (JS) divergence implicitly employed in GANs (Goodfellow et al., 2014).
Due to the Kantorovich-Rubinstein duality (Villani, 2008) for calculating the Wasserstein distance, the value function of WGAN is then written as
[Equation]
where D the set of 1-Lipschitz functions.
Analogous to GANs, we still call D the “discriminator” although it is actually a real-valued function and not a classiﬁer at all.
Arjovsky et al.
(2017) specify this family of functions D by neural networks and then use weight clipping to enforce the Lipschitz continuity.
However, as the authors note, the networks’ capacities become limited due to the weight clipping
and there could be gradient vanishing problems in the training.
Improved training of WGAN.
Gulrajani et al.
(2017) give more concrete examples to illustrate the perils of the weight clipping and propose an alternative way of imposing the Lipschitz continuity.
In particular, they introduce a gradient penalty term by noting that the differentiable discriminator
D ( · ) is 1-Lipschitz if and only if the norm of its gradients is at most 1 everywhere ,
[Equation]
where (cid:98) x is uniformly sampled from the straight line between a pair of data points sampled from the model P G and the real P r , respectively.
A similar regularization is used by Kodali et al.
Potential caveats.
Unlike the weight clipping, however, by no means one can penalize everywhere using this term through a ﬁnite number of training iterations.
As a result, the gradient penalty term
GP takes effect only upon the sampled points (cid:98) x , leaving signiﬁcant parts of the support domain not examined at all.
In particular, consider the observed data points and their underlying manifold that supports the real distribution P r .
At the beginning of the training stage, the generated sample G ( z ) and hence (cid:98) x could be distant from the manifold.
The Lipschitz continuity over the manifold is not enforced until the generative model P G becomes close enough to the real one P r , if it can.
1. O UR A PPROACH AND C ONTRIBUTIONS
[Equation]
In light of the above pros and cons, we propose to improve the improved training of WGAN by additionally laying the Lipschitz continuity condition over the manifold of the real data x ∼ P r .
Moreover, instead of focusing on one particular data point at a time, we devise a regularization over a pair of data points drawn near the manifold following the most basic deﬁnition of the 1-Lipschitz continuity.
In particular, we perturb each real data point x twice and use a Lipschitz constant to bound the difference between the discriminator’s responses to the perturbed data points x (cid:48) , x
[Table]
Figure 1 illustrates our main idea.
The gradient penalty
GP | (cid:98) x often fails to check the continuity of region near the real data x , around which the discriminator function can freely violate the 1-Lipschitz continuity.
We alleviate this issue by explicitly checking the continuity condition using the two perturbed version x (cid:48) , x (cid:48)(cid:48) near any observed real data point x .
In this paper, we make the following contributions.
(1) We propose an alternative mechanism for enforcing the Lipschitz continuity over the family of discriminators by resorting to the basic definition of the Lipschitz continuity.
It effectively improves the gradient penalty method (Gulrajani et al., 2017) and gives rise to generators with more photo-realistic samples and higher inception scores (Salimans et al., 2016).
(2) Our approach is very data efﬁcient in terms of being less prone to overﬁtting even for very small training sets.
We do not observe obvious overﬁtting phenomena even when the model is trained on only 1000 images of CIFAR-10 (Krizhevsky & Hinton, 2009).
Our approach can be seamlessly integrated with GANs to be a competitive semi-supervised training technique (Chapelle et al., 2009) thanks to that both inject noise to the real data points.
As results, we are able to report the state-of-the-art results on the generative model with a inception score of 8 .
81 ± 0 .
13 on CIFAR-10, and the semi-supervised learning results of 9 .
98 ± 0 .
21 on CIFAR10 using only 4,000 labeled images — especially, they are signiﬁcantly better than the existing
GAN-based semi-supervised learning results.
A PPROACH
We ﬁrstly review the deﬁnition of the Lipschitz continuity and then discuss how to use it to regularize the training of WGAN.
We then arrive at an approach that can be seamlessly integrated with the semi-supervised learning method (Chapelle et al., 2009).
By bringing the best of the two worlds, we report better semi-supervised learning results than both (Laine & Aila, 2016) and previously
GAN-based methods.
2. I MPROVING THE IMPROVED TRAINING OF WGAN
Let d denote the (cid:96) 2 metric on an input space used in this paper.
A discriminator D : X (cid:55)→ Y is
Lipschitz continuous if there exists a real constant M (cid:62) 0 such that, for all x 1 , x 2 ∈ X ,
[Equation]
Immediately, we can add the following soft consistency term ( CT ) to the value function of WGAN in order to penalize the violations to the inequality in eq.
[Equation]
Remarks.
Here we face the same snag as in (Gulrajani et al., 2017), i.e. , it is impractical to substitute all the possibilities of ( x 1 , x 2 ) pairs into eq.
What pairs and which regions of the input set
X should we check for eq.
Arguably, it is fairly safe to limit our scope to the manifold that supports the real data distribution P r and its surrounding regions mainly for two reasons.
First, we keep the gradient penalty term and improve it by the proposed consistency term in our overall approach.
While the former enforces the continuity over the points sampled between the real and generated points, the latter complement the former by focusing on the region around the real data manifold instead.
Second, the distribution of the generative model P G is virtually desired to be as close as possible to P r .
We use the notation M in eq.
(3) and a different M (cid:48) in eq.
(4) to reﬂect the fact that the continuity will be checked only sparsely at some data points in practice.
Perturbing the real data.
To this end, the very ﬁrst version we tried was to directly add Gaussian noise δ to each real data point, resulting in a pair of x + δ 1 , x + δ 2 , where x ∼ P r .
However, as noted by Arjovsky et al.
(2017) and Wu et al.
, we found that the samples from the generator become blurry due to the Gaussian noise used in the training.
We have also tested the dropout noise that gives rise to generated MNIST samples, which are cut off here and there.
The success comes after we perturb the hidden layers of the discriminator using dropout, as opposed to the input x .
When the dropout rate is small, the perturbed discriminator’s output can be considered as the output of the clean discriminator in response to a “virtual” data point x (cid:48) that is not far from x .
Thus, we denote by D ( x (cid:48) )
the discriminator output after applying dropout to its hidden layers.
In
Require: b size.
1: for iter of training iterations do 2: for N of iterations do 3: for i = 1 , ..., b do 4:
Sample real data x ∼ P r ,random vector z ∼ P z , a random number (cid:15) ∼ U [0 , 1] .
5: ˆ x ← (cid:15) x + (1 − (cid:15) )
G ( z ) 6:
L ( i ) ← D ( G ( z ))
− D ( x ) + λ 1 GP | (cid:98) x + λ 2 CT | x (cid:48) , x (cid:48)(cid:48) 7: end for 8: θ d ← Adam ( ∇ θ d 1 b (cid:80) bi =1 L ( i ) , θ d , γ ) 9: end for 10:
Sample b random vectors (cid:8) z ( i ) (cid:9) 11: θ g ← Adam ( ∇ θ g 1 b (cid:80) bi =1 − D ( G ( z ( i ) )) , θ g , γ ) 12: end for the same manner, we ﬁnd the second virtual point x (cid:48)(cid:48) around x by applying the (stochastic) dropout again to the hidden layers of the discriminator, and denote by D ( x (cid:48)(cid:48) )
the corresponding output.
Note that, however, it becomes impossible to compute the distance d ( x (cid:48) , x (cid:48)(cid:48) ) between the two virtual data points.
In this work, we assume it is bounded by a constant and absorb the constant to M (cid:48) .
Accordingly, we tune M (cid:48) in our experiments to take account of this unknown constant; the best results are obtained between M (cid:48)
=
and M (cid:48)
= 0 .
For consistency, we use M (cid:48)
=
to report all the results in this paper.
A consistency regularization.
Our ﬁnal consistency regularization takes the following form,
[Equation]
5) where D ( x (cid:48) ) is the output of the discriminator given the input x and after we apply dropout to the hidden layers of the discriminator.
We envision this is equivalent to passing a “virtual” data point x (cid:48) through the clean discriminator.
We ﬁnd that it slightly improves the performance by further controlling the second-to-last layer D − ( · ) of the discriminator, i.e. , the d ( D − ( x (cid:48) ) , D − ( x (cid:48)(cid:48) )) term above.
This new consistent regularization CT | x (cid:48) , x (cid:48)(cid:48) enforces the Lipschitz continuity over the data manifold and its surrounding regions, effectively complementing and improving the gradient penalty GP | (cid:98) x used in the improved training of WGAN.
Putting them together, our new objective function for updating the weigts of the discriminator is
[Equation]
Algorithm 1 shows the complete algorithm for learning a WGAN in this paper.
For the hyperparameters, we borrow λ 1 = 10 from Gulrajani et al.
and use λ 2 = 2 for all our experiments no matter on which dataset.
Another hyper-parameter is M (cid:48) in eq.
As stated previously, M (cid:48)
=in our experiments.
2. A SEAMLESS CONNECTION WITH A SEMI - SUPERVISED LEARNING METHOD
In this section, we extend the WGAN to a semi-supervised learning approach by drawing insights from two related works.
• Following (Salimans et al., 2016), we modify the output layer of the discriminator such that it has K + 1 output neurons, where K is the number of classes of interest and the additional neuron is reserved for contrasting the generated samples with the real data using the Wasserstein distance in the WGAN.
We use a ( K + 1) -way softmax as the activation function of the last layer.
• Following (Laine & Aila, 2016), we add our consistency regularization CT | x (cid:48) , x (cid:48)(cid:48) to the objective function of the semi-supervised training, in order to take advantage of the effect of temporal ensembling.
[Figure]
Figure 2 shows the framework for training the discriminator, with the objective function detailed below,
[Equation]
where the ﬁrst three terms are the same as in (Salimans et al., 2016), while the last consistency regularization is calculated after we apply dropout to the discriminator.
The last term essentially leads to a temporal self-ensembling scheme to beneﬁt the semi-supervised learning.
Please see (Laine &
Aila, 2016) for more insightful discussions about it.
For the Generator loss, we follow (Salimans et al., 2016) and the objective function is detailed below,
[Equation]
E XPERIMENTAL RESULTS
We conduct experiments on the prevalent MNIST (LeCun et al., 1998), and CIFAR-10 (Krizhevsky & Hinton, 2009) datasets.
The code will be released for reproducibility of our results.
3. MNIST
The MNIST dataset provides 70,000 handwritten digits in total, and 10,000 of them are often left out for the testing purpose.
Following (Gulrajani et al., 2017), we use only 1,000 of them to train the
WGAN for a fair comparison with it.
We use all the 60,000 training examples in the semi-supervised learning experiments and reveal the labels of 100 of them (10 per class).
This setup is the same as in (Rasmus et al., 2015), and no data augmentation is used.
Please see Appendix A for the network architectures of the generator and discriminator, respectively.
Qualitative results.
Figure 3 shows the generated samples with improved training of WGAN by the gradient penalty (GP-WGAN), and ours with the consistency regularization (CT-GAN), respectively, after 50,000 generator iterations.
It is clear that our approach gives rise to more realistic samples than GP-WGAN.
The contrasts of our samples between the foreground and the background are in general sharper than those of GP-WGAN.
Overﬁtting.
We ﬁnd that our approach is less prone to overﬁtting.
To demonstrate this point, we show the convergence curves of the discriminator’s value functions by GP-WGAN and our CT-GAN in Figure 4.
The red curves are evaluated on the training set and the blue ones are on the test set.
We can see that the results on the test set become saturated pretty early in GP-WGAN, while ours can consistently decrease the costs on both the training and the test sets.
This observation also holds for the CIFAR-10 dataset (cf.
Appendix E).
Semi-supervised learning results.
We compare our semi-supervised learning results with those of several competitive methods in Table 1.
Our approach is among the best on this MNIST dataset.
[Figure] [Figure]
3. CIFAR-
CIFAR-10 (Krizhevsky & Hinton, 2009) contains 50,000 natural images of size 32 × 32.
We use it to test two networks for the generative model: a small CNN and a ResNet (cf.
Appendix A for the network structures).
For the former, we use only 1,000 images to train the model
[Figure]
and we use the whole training set to learn the ResNet.
Qualitative results.
Figure 5 contrasts the samples we generated to those by GP-WGAN when the generator is the small CNN.
Figure 7 shows the results by a larger-scale ResNet.
Our results are more photo-realistic.
Additionally, we also draw the histograms of the discriminator’s weights in Figure 6 after we train it using GP-WGAN and CT-GAN, respectively.
It is interesting to see that ours controls the weights within a smaller and more symmetric range [ − 0 .
67 , 0 . 96] than the [ − 2 . 00 , 10 . 12] by GP-WGAN, partially explaining why our approach is less prone to overﬁtting.
[Table]
[Figure] [Figure]
Comparison of the inception scores.
Finally, we compare our approach with GP-WGAN on the whole training set for both unsupervised and supervised generative-purpose task using ResNet.
For model selection, we use the ﬁrst 50,000 samples to compute the inception scores (Salimans et al., 2016), then choose the best model, and ﬁnally report the “test” score on another 50,000 samples.
The experiment follows the previous setup in (Odena et al., 2016).
From the comparison results in
Tables 2, we conclude that our proposed model achieves the highest inception score in the c dataset, to the best of our knowledge.
Correspondingly, the generated samples are shown in Figure 7.
For the small CNN based generator, the inception scores of GP-WGAN and our CT-GAN are 2 .
98 ± 0 .
11 and 5 .
13 ± 0 .
12 , respectively.
Supervised IS
Unsupervised IS
Accuracy(%)
SteinGANs (Wang & Liu, 2016)
[Table]
DCGANs (Radford et al., 2015) 6.
[Table]
16 ±
[Table]
Improved GANs (Salimans et al.
[Table] [Table]
[Figure]
Semi-supervised learning.
For the semi-supervised learning approach, we follow the standard training/test split of the dataset but use only 4,000 labels in the training.
A regular data augmentation with ﬂipping the images horizontally and randomly translating the images within [-2,2] pixels is used in our paper (No ZCA whitening).
We report the semi-supervised learning results in Table 3.
The mean and standard errors are are obtained by running the experiments 5 rounds.
Comparing to several very competitive methods, ours is able to achieve state-of-the-art results.
Notably, our
CT-GAN outperfroms all the GAN based methods by a large margin.
Please see Appendix A for the network architectures and Appendix C for the ablation study of our algorithm.
C ONCLUSION
In this paper, we present a consistency term derived from Lipschitz inequality, which boosts the performance of GANs model.
The proposed term has been demonstrated to be an efﬁcient manner to ease the over-ﬁtting problem when data amount is limited.
Experiments show that our model obtains the state-of-the-art accuracy and Inception score on CIFAR-10 dataset for both semi-supervised learning task and generative model.
Appendices A PPENDIX A N ETWORK ARCHITECTURES
Table 4 (MNIST) and Table 5 (CIFAR-10) detail the network architectures used in our classiﬁcationpurpose CT-GAN, where the classiﬁers are the same as the widely used ones in most semi-supervised networks (Laine & Aila, 2016) except that we apply weight-norm rather than batch-norm.
For the generators, we follow the network structures in GP-WGAN (Salimans et al., 2016), but we use lower dimensional noise (50D) as the input to the generator for CIFAR-10 in order not to reproduce the complicated images and instead shift the focus of the training to the classiﬁer.
[Table] [Table]
A PPENDIX B H YPER - PARAMETERS AND OTHER TRAINING DETAILS
For the semi-supervised learning experiments, we set λ = 1 . 0 in Eq.(7) in all our experiments.
For
CIFAR-10, the number of training epochs is set to 1,000 with a constant learning rate of 0.0003.
For
MNIST, the number of training epochs is set to 300 with a constant learning rate of 0.003.
The other hyper-parameters are exactly the same as in the improved GAN (Salimans et al., 2016).
For the experiments on the generative models
[Table]
to have a fair comparison for our results with the existing ones
[Table]
we keep the network structure and hyper-parameters the same as in the improved training of WGAN (Gulrajani et al., 2017) except that we add three dropout layers to some of the hidden layers as shown in tables 6 to 8.
[Table] [Table]
[Table]
A PPENDIX C A BLATION STUDY OF OUR APPROACH TO SSL
We run an ablation study of our approach to the semi-supervised learning (SSL).
Thanks to the dual effect of the proposed consistency (CT) term, we are able to connect GAN with the temporal ensembling (TE) Laine & Aila (2016) method for SSL.
Our superior results thus beneﬁt from both of them, veriﬁed by the ablation study detailed in Table 9.
If we remove the CT term, the test error goes up to 14 . 98 , signifying the effectiveness of the CT regularization.
If we remove GAN from our approach, it almost reduces to TE; in fact, all the settings here are the same as TE except that we use an extra regularization ( D ( ., . ) in CT) over the second-to-last layer.
We can see that the error is still signiﬁcantly larger than our overall method.
We use the weight normalization as in Salimans et al.
(2016), which becomes a core constituent of our approach.
The batch normalization would actually invalidate the feature matching in Salimans et al.
Finally, we also test the version without the regularization to the second-to-last layer and observe a little drop in the performance.
[Table]
A PPENDIX D E XAMINING THE 1-L IPSCHITZ CONTINUITY
Norm of gradient.
In our experiments, we ﬁnd that although the GP-WGAN (Gulrajani et al., 2017) has applied a Lipschitz constraint in the form of the gradient penalty over the input sampled between a real data point and a generated one, the actual effect on the (cid:96) 2 norm of the gradient is not as good as our CT-GAN model in the real data points, as illustrated in Figure 1.
We empirically verify this fact by Figure 8, which shows the (cid:96) 2 norms of the gradients of the discriminator with respect to the real data points.
The closer to 1 the norms are, the better the 1-Lipschitz continuity is preserved.
Figure 8 further demonstrates that our consistency (CT) regularization is able to improve
GP-WGAN (Gulrajani et al., 2017).
[Figure]
Deﬁnition of the Lipschitz continuity.
Additionally, we check how much the 1-Lipschitz continuity is satisﬁed according to its vanilla deﬁnition (cf.
Eq.
Figures 9 and 10 plot the CTs of Eq.
(4) and Eq.
(5), respectively, over different iterations of the training using 1000 CIFARimages.
Figure 10 is the actual CTs used to train the generative model. Figure 9 is drawn as follows. For every 100 iterations, we randomly pick up 64 real examples and split them into two subsets of the same size.
We compute d(D(x1)-D(x2))/d(x1-x2) for all the (x1,x2) pairs, where x1 is from the ﬁrst subset and x2 is from the second.
The maximum of d(D(x1)-D(x2))/d(x1-x2) is plotted in Figure 9.
We can see that the CT-GAN curve converges under a certain value much faster than GP-WGAN. The 1-Lipschitz continuity is better maintained by CT-GAN than GP-WGAN over the whole course of the training procedure on the manifold of the real data.
[Figure] [Figure]
A PPENDIX E GP-WGAN WITH DROPOUT
We run another experiment of GP-WGAN by adding the same dropout layers used in our method to the networks of GP-WGAN, denoted by GP-WGAN+dropout.
This may help understand the contribution of our approach in preventing the overﬁtting problem — is it due to the CT regularization, or merely due to the dropout?
This experiment is done by removing the CT term out of our value function for training WGAN using 1,000 CIFAR-10 images and yet keep the dropout layers.
The Inception score of GP-WGAN+dropout is 4 .
29 ± 0 .
12 , and the generated samples are shown in Figure 11(a).
We also plot the curve of CT in the training procedure of GP-WGAN+dropout and contrast it to the curve of our CT-GAN in Figure 11(b).
It is clear that GP-WGAN+dropout outperforms GP-WGAN, and our method outperforms GP-WGAN with a large margin.
In other words, the dropout itself actually weakens GP-WGAN, while the CT regularization we proposed improves GP-WGAN.
The improvement is because CT enforces the 1-Lipschitz continuity of the discriminator, and not because of the dropout used by CT.
[Figure]
Finally, we plot the convergence curves of the discriminators’ negative cost function learned by GPWGAN, GP-WGAN+Dropout
[Figure]
and our CT-GAN in Figure 12.
We can see that dropout is able to reduce the overﬁtting of GP-WGAN, but it is not as effective as our CT-GAN.
[Figure]