S AMPLE WEIGHTING AS AN EXPLANATION FOR MODE COLLAPSE IN GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative adversarial networks were introduced with a logistic M ini M ax cost formulation, which normally fails to train due to saturation, and a N on- S aturating reformulation.
While addressing the saturation problem, NS-GAN also inverts the generator’s sample weighting, implicitly shifting emphasis from higher-scoring to lower-scoring samples when updating parameters.
We present both theory and empirical results suggesting that this makes NS-GAN prone to mode dropping.
We design MM-nsat, which preserves MM-GAN sample weighting while avoiding saturation by rescaling the MM-GAN minibatch gradient such that its magnitude approximates NS-GAN’s gradient magnitude.
MM-nsat has qualitatively different training dynamics and on MNIST and CIFAR-10 it is stronger in terms of mode coverage, stability and FID.
While the empirical results for MM-nsat are promising and favorable also in comparison with the LS-GAN and Hinge-GAN formulations, our main contribution is to show how and why NS-GAN’s sample weighting causes mode dropping and training collapse.
I NTRODUCTION
Generative adversarial networks have come a long way since their introduction (Goodfellow et al., 2014) and are currently state of the art for some tasks, such as generating images.
A combination of deep learning developments, GAN speciﬁc advances and vast improvements in data sets and computational resources have enabled GANs to generate high resolution images that require some effort to distinguish from real photos (Zhang et al., 2018;
Brock et al., 2018; Karras et al
GANs use two competing networks: a generator G that maps input noise to samples mimicking real data, and a discriminator D that outputs estimated probabilities of samples being real rather than generated by G .
We summarize their cost functions, J D and J G , for the minimax and nonsaturating formulations introduced in Goodfellow et al.
We denote samples from real data and noise distributions by x and z and omit the proper expectation value formalism:
[Equation]
For clarity, we use subscripts to distinguish between the discriminator’s pre-activation logit output
D l and the probability representation D p :
[Equation]
Both formulations have the same cost function for D , representing the cross entropy between probability estimates and ground truth.
In the minimax formulation (MM-GAN), G is simply trained to maximize D ’s cost.
Ideally, G matches its outputs to the real data distribution while also achieving meaningful generalization, but many failure modes are observed in practice.
NS-GAN uses a modiﬁed cost for G that is non-saturating when D distinguishes real and generated data with very high conﬁdence, such that G ’s gradients do not vanish.
(Supplementary: C)
Various publications establish what the different cost functions optimize in terms of the JensenShannon and reverse Kullback-Leibler divergences between real and generated data:
[Equation]
Husz ´ ar (2015) and Arjovsky & Bottou (2017) have suggested NS-GAN’s divergence as an explanation for the ubiquitous mode dropping and mode collapsing problems with GANs (Metz et al., 2016; Salimans et al., 2016; Srivastava et al., 2017).
While MM-GAN seems promising in terms of its Jensen-Shannon divergence, the formulation has largely been ignored because the saturating cost causes training to break down.
A variety of other GAN formulations have been introduced, such as WGAN-GP (Arjovsky et al., 2017; Gulrajani et al., 2017), LS-GAN (Mao et al., 2016) and Hinge-GAN (Miyato et al., 2018).
Lucic et al.
(2018) ﬁnds that different cost formulations tend to get similar results given sufﬁcient parameter tuning, including various forms of regularization.
Despite the questionable form of NSGAN in terms of divergences, it is widely used and can produce very impressive results, such as in the improved StyleGAN (Karras et al., 2019).
T HEORY 2. MM-GAN SATURATION
causes G ’s gradient to vanish when
D p ( G ( z )) → 0 , which corresponds to the maximum value of its cost.
The parameters of a network are typically trained with some form of gradient descent on a cost function.
We ﬁnd the expressions for D ’s and G ’s gradients with respect to their parameters, φ and θ : (Supplementary: F) [Equation] [Equation]
[Equation]
We emphasize the two kinds of scaling factors for these gradients in red and blue: they are plotted in ﬁgure 2.
The discriminator’s scaling factors decrease as it minimizes its cost, approaching 0 towards the optima for both the real and the generated data term.
The minimax formulation J G = − J D is suitable for adversarial training in terms of the generator’s optimum, but the unchanged scaling factor means that G ’s gradients increase towards and decrease away from its optimum.
The saturation effect described in Goodfellow et al.
(2014) is that lim
D p ( G ( z ))
→ 0 ∇ θ J G MM = 0 , such that G stops training when D is highly conﬁdent that its samples are fake.
More generally, the scaling factor makes J G concave with respect to D l , which interacts poorly with common optimization methods (see section 2.4).
As ∇ θ J G MM and ∇ θ J G NS are the same aside from their scaling factors, the different behaviors of the two formulations must follow from these.
NS-GAN’s scaling factor avoids saturation, but gives rise to a different, more subtle mode dropping tendency (see section 2.3).
2. N ON - SATURATION AND SAMPLE WEIGHTING
As can be seen from eq 4, the NS-GAN and MM-GAN gradients are parallel for a single sample , but with different magnitudes.
Stochastic gradient descent estimates the gradient of the cost over the entire input distribution by using a number of samples (a minibatch).
We can express the NS-GAN minibatch gradient in terms of the MM-GAN gradient:
[Equation]
[Equation] Due to the bracketed factor, NS-GAN rescales the contribution from each sample relative to MMGAN, implicitly emphasizing samples with smaller values of D p .
Seeing as saturation is caused by the gradient’s vanishing magnitude, this additional effect on the gradient’s direction is questionable.
The exact ratio of the minibatch gradient magnitudes for NS-GAN and MM-GAN depends on ∂/∂θ ( D l ( G ( z ))) for each sample and has no convenient expression.
We can approximate it by replacing D p ( G ( z i )) in eq 5 with its mean over the minibatch, D p = 1 N (cid:80) N − 1 i =
D p ( G ( z i )) .
This allows us to formulate a form of non-saturation for MM-GAN that mimicks NS-GAN: [Equation]
[Equation]
We refer to the formulation with this generator gradient as MM-nsat .
The relative weights of samples in each batch are as for MM-GAN, while the gradient magnitude approximates that of NS-GAN.
Note, however, that the relative weights of samples may be disturbed across batches, such as when the minibatch size is small and D p ﬂuctuates.
Despite different theoretical motivation, MM-nsat is very closely related to importance weighted NS-GAN (Hu et al., 2017).
(Supplementary: D)
2. S AMPLE WEIGHTING AND MODE DROPPING
If we use r ( x ) and g ( x ) to denote the density of real and generated samples at a point x in data space, the function that minimizes J D MM,NS is given by:
[Equation]
This expression for the optimial discriminator assumes idealized conditions: that D is optimized for a ﬁxed G , with unlimited capacity and without having to estimate the underlying real and generated data distributions by ﬁnite sampling. While D opt is not realized in practice, we use it to form some intuitions about D ’s behaviors.
Suppose we have real data with two disjunct, equiprobable modes, and that one of these modes, O , is overrepresented in generated data.
For convergence, G would need to shift probability mass from O to the underrepresented mode, U .
However, the minibatches used to update G will have more samples from O , simply because they are generated more often.
For a strong discriminator, these samples will also tend towards smaller values of D p ( G ( z )) , due to eq 7. This in turn causes them to be weighted differently by NS-GAN and MM-GAN, because D p ( G ( z )) appears in their scaling factors (equation 4). We refer to the ﬁrst effect as over- and underrepresentation (of generated samples relative to real samples) and the second effect as up- and downweighting (depending on the scaling factor).
The fundamental problem with NS-GAN can be seen by looking at how these two effects interact. The NS-GAN generator has a scaling factor 1 − D p ( G ( z )) , which combines overepresentation with upweighting and underrepresentation with downweighting, allowing an overrpresented mode O to dominate U ’s contributions to the parameter updates.
If parameter updates overwhelmingly based on gradients from O has an adverse effect on G ’s ability to generate samples from U , this mode may become increasingly underrepresented, making O yet more dominant.
MM-GAN’s scaling factor has the [-same argument, NS-GAN struggles to discover new modes: g ( x ) ≈ 
-]reverse behavior, pairing overrepresentation with downweighting and underrepresentation with upweighting, such that the two effects combine in a stabilizing rather than destabilizing way.
For the extreme example of a nearly dropped or newly discovered mode, r ( x ) (cid:29) g ( x ) such that we expect D p ( x ) ≈ 1 for a very strong discriminator.
NS-GAN’s sample weighting disregards gradients from such samples, whereas MM-GAN’s sample weighting emphasizes them.
This difference between MM-GAN and NS-GAN in terms of scaling factors reﬂects the different divergences they have been shown to optimize (eq 3).
Husz´ar (2015) and Arjovsky et al.
(2017) both connect the mode dropping tendencies of NS-GAN to its reverse Kullback-Leibler divergence, which strongly penalizes G for generating samples outside of the real data modes, but not for dropping modes.
A variety of attempts to address mode dropping, mode collapse and mode hopping in GANs seem unaware that this is reasonable behavior for NS-GAN given its divergence: mode dropping minimizes D RKL and maximizes D JS .
2. MM-GAN INTERACTION WITH A
[Figure] GANs are generally trained with the Adam optimizer (Kingma & Ba, 2014), as was recommended by Radford et al.
(2015) while introducing the DCGAN architecture.
The parameter update step for Adam is given by:
[Equation]
Here, g t is the gradient at timestep t , and ˆ m t
and ˆ v t are the ﬁrst and second order exponential moving averages of the parameterwise gradients, bias-corrected to account for zero-initialization.
There are four hyperparamters: (cid:15) is a small constant primarily for numerical stability, α is the learning rate and β 1 and β 2 determine the effective memories of the moving averages.
The fraction ˆ m t √ ˆ v t resembles unit normalization of a vector, and for a constant gradient g t =
g (such that the moving averages are trivial) update steps depend only on the sign of g , if | g | (cid:29) (cid:15) .
[Equation]
However, this normalization does not necessarily address MM-GAN’s saturation problem, due to the training dynamics.
D opt l = ±∞ where the real and generated data do not overlap (eq 7), such that if D can cleanly separate real from generated samples, we expect it to further decrease its loss by inﬂating its outputs.
Supposing that D approaches this optimum linearly, i.e. D tl ( G ( z )) = at , we get:
[Equation]
D p also appears as the MM-GAN scaling factor (eq 4), such that G will be optimized with gradients of the form g t = g 0 exp( at ) .
For reasonable values of a , β 1 and β 2 , the update step size for each of G ’s parameters can be approximated: (Supplementary: G)
[Equation]
Given the commonly used β 2 = 0 .
999 , parameter updates will vanish exponentially if a < − 0 . 0005 , given that | D l | increases fast enough and conforms reasonably well to our simpliﬁed model.
If D learns to distinguish the real and generated data manifolds before they meaningfully intersect, this interaction between D , G and Adam threatens to freeze parameter updates altogether.
(Supplementary: L)
M ETHOD 3. C OST FUNCTION MODIFICATIONS
In addition to our novel non-saturating version of MM-GAN (MM-nsat: eq 6), we normalize the gradient magnitudes of the original cost functions (MM-unit, NS-unit).
Motivated by our model for
MM-GAN saturation in section 2.4, we also test if modifying the Adam β 2 parameter for G gives the expected results (MM β 2 = 0 . 99 ).
(Supplementary: H)
MM-nsat is the only of these cost functions that is interesting in its own right: the others are used to demonstrate various behaviors and highlight the roles of sample weighting as opposed to gradient magnitude.
Note that the preﬁxes MM and NS always correspond to the sample weighting used by the cost function.
Furthermore, gradient magnitudes are matched for MM-unit & NS-unit and approximately matched for MM-nsat & NS-GAN. 3. E VALUATION
Evaluating a generator is generally difﬁcult.
In addition to visual inspection, we use Frech ´ et Inception Distance (Heusel et al., 2017), which we ﬁnd informative also on MNIST, in spite of the feature extraction network being trained on natural images.
Due to our focus on the mode dropping problem, we also use an unusual metric for datasets with class labels: we compute the Jensen-Shannon divergence between class distributions in real and generated data.
For G , we estimate its class distribution by drawing samples and using a pre-trained classiﬁcation network to label them.
Note that this metric only sees class distrubtions and is not sensitivite to the degree of mode coverage or collapse within any given class.
(Supplementary: I)
[Equation]
3. E XPERIMENTS
For the majority of our experiments we use very simple networks without tuning their hyperparameters: either fully connected networks (FC) with a ﬁxed number of hidden units, or strided convolutional networks with kernel size 3 and doubling the number of ﬁlters when the width and height is halved (Conv).
We use ReLU activations, except for the ﬁnal layer, where G uses tanh activation with real data normalized to [ − 1 , 1] , while D uses sigmoid activation to map D l to D p .
We make use of how fully connected networks are harder to train than convolutional ones (ThanhTung et al., 2018) and how training grows increasingly fragile for deeper networks and higher resolution datasets to ﬁnd illustrative test cases.
We make use of batch normalization (Ioffe & Szegedy, 2015) (bn), zero centred real data gradient penalty (Mescheder, 2018) (sgp) and spectral normalization (Miyato et al., 2018) (sn) only where explicitly mentioned.
In addition to the simpler networks, we test the full DCGAN (Radford et al., 2015) and StyleGAN
(Karras et al., 2018) architectures.
We primarily use the MNIST (LeCun & Cortes, 2010) and CIFAR-10 (Krizhevsky et al., 2009) datasets to investigate the differences between cost functions.
We run additional experiments using the CAT (Zhang et al., 2008; Jolicoeur-Martineau, 2018) and FFHQ (Karras et al., 2018) datasets to study behaviors for higher resolution images.
R ESULTS AND DISCUSSION 4. Q UALITATIVE PRELIMINARIES
Training GANs on a ring of Gaussians has been used to study both the mode dropping tendency of GANs (Metz et al., 2016;
Srivastava et al., 2017) and how to address it.
MM-GAN tends to do very well on such problems, as shown in ﬁgure 4.
For this problem, it is easy to generate samples indistinguishable from real ones, limiting problems with saturation.
MM-GAN is more mode-covering in practice as suggested by its divergence (Husz´ar, 2015; Arjovsky et al., 2017).
(Supplementary: Q)
Using MNIST and weak, fully-connected networks, we replicate the well-known failure mode of
MM-GAN that motivates the NS-GAN reformulation.
In ﬁgure 5, we show the gradient magnitudes early in training, in particular a super-exponentially vanishing gradient for MM-GAN after roughly 3 epochs that halts training altogether.
Additionally, we show that reducing β 2 only for G ’s optimizer (effectively giving it a shorter memory for the second order momentum) stabilizes the training process, as suggested by our theory on the interaction of MM-GAN and Adam.
(Supplementary: L)
For the same setup, we show samples at the end of prolonged training for NS and MM-nsat cost functions in ﬁgure 6.
We see that our version of minimax non-saturation trains well.
The difference in terms of mode coverage is visually striking and corresponds well with our numerical evaluations for the same generators in table 1.
For MNIST, NS-GAN’s mode collapse can mostly be addressed by early stopping or regularization.
In ﬁgure 7 we show samples from training on the more challenging CAT 128x128 dataset.
We ﬁnd that MM-nsat trains well and generates fairly realistic samples, whereas for NS-GAN, catastrophical mode collapse occurs before G learns to produce reasonable samples.
More quantitative results are presented in the supplementary material, particularly S and T.
4. Q UANTITATIVE EVALUATION
Figure 8 shows more comprehensive results, empasizing the training dynamics for MM-GAN and
NS-GAN plus variants of the same cost functions.
There are two key results: that simple stabilization techniques allow us to train GANs with MM-GAN sample weighting, without the saturation issues [Table] [Table] that plague unmodiﬁed MM-GAN; and that their behavior is qualitatively different from those using
NS-GAN sample weighting.
Gradient magnitudes mostly inﬂuence training stability.
Additionally, the plots on the right hand side show that the performance gain for minimax variants is reﬂected by more correct class distributions in generated data.
In some cases, such as for 3-layer fully connected networks on MNIST, there is a notable gap between D JS CD values, even while FID is similar.
The importance of sample weighting is best demonstrated by comparing MM-unit and NS-unit.
By construction, these variants have the same gradient magnitude, such that the only difference between the formulations is whether high- or low-scoring samples are emphasized when ﬁnding the total gradient for a minibatch of samples.
The results show that this difference is crucial for the training dynamics.
Aside from stability, there are no clear differences between the variants with minimax sample weighting.
While MM-nsat and MM-unit avoid saturation in very different ways, their results are highly similar, to each other and to the results for unmodiﬁed MM-GAN when it does not saturate.
This suggests that the approximation we make in eq 6 when introducing D p is of limited importance, and that MM-nsat is more faithful to the original, logistic minimax formulation than NS-GAN.
The main issue with NS-GAN is its strong tendency to deteriorate in terms of both FID and D JS CD as training progresses.
In some of the cases, there is an early stopping point where NS is competitive with MM-nsat.
The results for CIFAR-10 are the most extreme: NS and NS-unit consistently undergo catastrophical mode collapse.
(Supplementary results: M through U)
C ONCLUSION AND FURTHER WORK
Based on a more thorough, theoretical analysis of the differences between the formulations introduced in Goodfellow et al.
(2014), we have designed a form of non-saturation for the minimax cost function for GANs that rescales the minibatch gradient.
This corrects the training difﬁculties for the original minimax GAN, without the side-effects inherent to the NS-GAN reformulation.
Running experiments, we have shown that our new stabilization has qualitatively different behavior, in particular better mode coverage as indicated both by our theory and by previous works showing which divergences are optimized by NS-GAN and MM-GAN.
We have shown promising results with MM-nsat (our gradient rescaled version of MM-GAN), but it is primarily designed for demonstration purposes.
Results with simpler networks on MNIST and
CIFAR-10 are much stronger for MM-nsat compared to NS-GAN, Hinge-GAN and LS-GAN, and experiments on higher resolution images from the CAT and FFHQ datasets show that MM-nsat has less issues with catastrophic mode collapse during the early stages of training, greatly reducing the stability issues that GANs tend to suffer from.
Interactions with discriminator regularization are unclear and important for more advanced applications (see O).
Performance when combining MM-nsat with various designs that directly address mode dropping remains to be determined, for instance the class conditioning (Mirza & Osindero, 2014) used in BigGAN (Brock et al., 2018) and the minibatch standard deviation used in StyleGAN (Karras et al., 2018).
Unlike the normalizations and regularizations that mask the underlying issues with NS-GAN and often introduce extra hyperpameters, MM-nsat only modiﬁes the cost function.
While we have focused on variants of MM-GAN and NS-GAN, with the convenience that these all work with the same discriminator cost function, we expect analysis in terms of sample weighting to apply much more generally, explaining training dynamics and tradeoffs between individual sample quality and overall sample diversity also for other GAN formulations.
S UPPLEMENTARY MATERIAL A T ABLE OF COST FUNCTIONS
Refer to table 2 for an overview of the cost functions used in this paper.
Note that sample emphasis is not directly comparable between formulations with different costs for the discriminator.
B I NFORMAL AND EXTENDED SUMMARY
We include an extended summary of the most central points from our work.
This presentation is intended to be less technical and somewhat more accessible to a wider audience.
When training a classiﬁer, the standard cross entropy loss has stronger gradients for samples with greater error.
This means that correcting misclassiﬁcations is emphasized over further increasing the conﬁdence level of correctly classiﬁed samples.
Training usually runs for multiple epochs, giving the classiﬁer repeated opportunities to readjust if its cost increases for any particular sample.
Like for classiﬁers, every term in the NS-GAN cost functions for the discriminator and generator emphasizes the samples which are furthest from their optima.
While this seems reasonable, it does not necessarily interact well with adversarial training.
To minimize its cost when training on equal amounts of real and generated data, the discriminator
D p ( x ) approximates P ( y is real | y = x ) .
This means that D p ≈ 1 for real data that G does not generate and D p ≈ 0 for generated data outside of the real data manifold.
More generally, D p ( x ) approximates the density of real samples at x relative to the sum of densities of real and generated samples.
The generator cost function includes a real data term which is often omitted: without any gradient with respect to G ’s parameters it does not inﬂuence the training.
This makes it difﬁcult for G to act on dropped modes, because its updates only see D ’s gradients at the points in data space where G generates samples.
The degree to which NS-GAN’s generator cost emphasizes samples further from their optimum is quantiﬁed by the scaling factor given in equation 4.
NS-GAN weights contributions from each generated sample by a factor 1 − D p , such that low-scoring (“unrealistic”)
samples contribute the most to the gradient.
MM-GAN’s generator instead uses the scaling factor D p , emphasizing high-scoring (“realistic”) samples.
MM-GAN saturation follows directly from this scaling factor: for highly unrealistic generated samples, D p → 0 , and this weighting causes G ’s gradient vanish.
Adam does not work around this problem with the usual hyperparameter settings.
In either case, G ’s parameters are updated by sampling the gradient of D with respect to G ’s parameters at points x where samples happen to be generated.
This means that the updates see most contributions from regions where the density of generated samples is high.
The main problem with NS-GAN is how this sampling frequency interacts with its scaling factor.
Regions on the real data manifold where the density of generated samples is too low tend towards small values of 1 − D p : these generated samples are considered realistic by D and thus have small error for G and are weighted down by the scaling factor.
Since this region in data space is also sampled rarely, due to the G ’s low density, its contribution to G ’s parameter updates will be small.
At the same time, regions with too high density of generated samples have large scaling factors and are sampled frequently.
In a situation where different modes disagree on the best conﬁguration of
G ’s parameters, undersampled modes will be massively outvoted.
This would not be a problem if D ’s gradients were able to push the density of generated samples away from oversampled modes and discover undersampled or entirely dropped modes.
However, we have no guarantee that this sort of action at a distance will work in practice: the literature instead shows that mode collapse is a recurring problem, likely because D ’s gradients fail to be informative across longer distances in data space.
Sample weighting also explains why MM-GAN behaves much better in terms of mode coverage.
Since MM-GAN uses the opposite scaling factor, D p , it boosts the contributions to parameter updates from undersampled modes.
The MM-GAN scaling factor partially cancels against the sampling bias effect, making the parameter update less dependent on a single, dominant mode.
While a generated sample in an undersampled mode is not itself an error , the very high value of D p for this generated sample reﬂects a large discrepancy between generated and real densities at this point.
MM-GAN’s sample weighting makes it better able to address errors of this sort.
The main problem with MM-GAN is that its gradient decreases when the cost is increases.
This makes it very difﬁcult to optimize, also with Adam.
NS-GAN, the established solution to this problem, changes both sample weighting and gradient magnitudes.
The problem can instead be addressed with a very simple rescaling of the minibatch gradient, as shown in eq 10.
We call the resulting formulation MM-nsat as it combines minimax sample weighting with the same form of non-saturation as used in Goodfellow’s NS-GAN.
The key difference is that the NS-GAN non-saturation is implicitly applied to each sample, disturbing sample weights, while the MM-nsat rescaling is explicitly applied to the minibatch gradient as a whole.
Our experimental results validate that this minibatch gradient rescaling corrects the saturation problem, and comparing NS-GAN and MM-nsat allows us to show that sample weighting is highly important for training dynamics.
Mode coverage and general stability is greatly improved for MM-nsat relative to NS-GAN, which sometimes suffers from gradual mode dropping throughout training, and sometimes from catastrophical mode collapse early in training.
Note, however, that strong models already tend to include various designs to counteract the problems caused by NS-GAN’s unfortunate sample weighting, limiting the beniﬁts of simply replacing NS-GAN with MM-nsat.
C G OODFELLOW ’ S MOTIVATION OF NS-GAN
We include Goodfellow et al.
(2014)’s motivation for NS-GAN for easy reference: “In practice, equation 1 [minimax cost] may not provide sufﬁcient gradient for G to learn well.
Early in learning, when G is poor, D can reject samples with high conﬁdence because they are clearly different from the training data.
In this case, log(1 − D ( G ( z ))) saturates.
Rather than training G to minimize log(1 − D ( G ( z )))
we can train G to maximize log D ( G ( z )) .
This objective function results in the same ﬁxed point of the dynamics of G and D but provides much stronger gradients early in learning.”
D I MPORTANCE WEIGHTING
Hu et al.
(2017) studies the parallels between generative adversarial networks and variational autoencoders.
They transfer the idea of importance weighting from VAE to GANs, arriving at an update rule for the generator (Hu et al.
(2017): equation 22) which reweights G ’s NS-GAN gradients for each sample in the minibatch with a normalized weighting factor.
Closely related ideas are seen in
Hjelm et al.
(2018) and Che et al.
The expression for the unnormalized weighting factor is given as:
[Equation]
Here, q φ 0 and q rφ 0 represent the discriminator and its reverse respectively, i.e. q rφ 0 ( y | x ) =
q φ 0 (1 − y | x ) , where y represents the true label of a sample x .
In our notation, this is corresponds to the following, as given explicitly in Hjelm et al.
[Equation]
Comparing this weighting factor to the bracketed factor in our eq 5, it is clear that it perfectly cancels against the reweighting of NS-GAN relative to MM-GAN.
In other words, importance weighted NSGAN actually recovers the same original MM-GAN sample weighting that is used by MM-nsat, such that our approach of avoiding the NS-GAN sample reweighting leads us to rediscover this update rule.
The remaining difference is that of the gradient magnitude.
When reweighting the NS-GAN gradients, Hu et al.
(2017) and Hjelm et al.
(2018) choose to normalize the weights over the minibatch: omitting to do this would make their importance weighting also recover the MM-GAN gradient magnitude, and thus reintroduce the MM-GAN saturation problem.
They normalize by calculating the actual sum of the reweighting factors, which is slightly different from our approximation where we replace D p with its mean over the minibatch (eq 6).
Note that neither method is exact unless the samplewise gradients in the minibatch have the same magnitude.
As seen by for instance comparing
MM-nsat and MM-unit, which have highly similar performance despite massively different gradient magnitudes, these diferences should have little effect on training dynamics.
The primary difference between importance weighthing for NS-GAN and our design of MM-nsat is the logical steps and the theoretical motivation.
In our work, the direct relationship to the original
MM-GAN update rule is made explicit, whereas importance weighting goes the cirucuitous route of ﬁrst taking the heuristic NS-GAN formulation in place the original MM-GAN formulation, then reweighting it to arrive at an importance weighted NS-GAN formulation, which our work shows to actually be just a non-saturating version of MM-GAN.
E C ONCURRENT WORK
A concurrent work recently made available as a pre-print, Sinha et al.
(2020), makes use of a simple design which is directly comparable to our method.
When updating G , they ignore the gradients arising from the lowest scoring generated samples, where the fraction of the samples ignored increases throughout training.
This effectively shifts weighting from low-scoring to high-scoring samples, which is the same overall effect as is achieved by using our MM-nsat gradient.
Their method is much simpler and less principled as to exactly how low- and high-scoring samples are weighted, but it is also more easily applied to other GAN cost formulations.
Similar to our ﬁndings, Sinha et al.
(2020) observe an increase in mode coverage and overall quality.
By investigating the cosine similarity between low- and high-scoring samples, they observe that optimization conﬂicts between these, as we suggest in section 2.3.
Their analysis in terms of pushing samples towards and away from modes is somewhat at odds with ours, where we note that lowscoring samples can be high-quality, but lie in modes that are signiﬁcantly oversampled by G .
The discriminator score is not a direct measure of the sample’s quality, but rather of relative densities of real and generated data, such that we should not expect D ’s gradients to necessarily point towards the centres of the real data modes.
F G RADIENTS DERIVED FROM COST FUNCTIONS
Finding the MM-GAN gradient for a single sample is a matter of straightforward calculus.
The expression for NS-GAN’s gradient can be found using the same approach.
First note that the relationship between D p and D l is given by eq 2, such that:
[Equation] [Equation]
We take J G = − J D (eq 1) as the starting point, as deﬁnes the minimax formulation.
Note that the ﬁrst term does not depend on G or θ , such that it can be omitted in the G cost function:
[Equation]
It might seem counterintuitive to mix D l and D p when expressing the gradients, instead of using for instance the expression on line 2 of equation 17.
However, the non-linear behavior of ∂D p /∂θ makes interpretation difﬁcult.
As D p approaches either 0 or 1, increasingly small changes in D p reﬂect the same change in odds ratios.
It is less clear from the gradient in terms of only D p that the
MM-GAN gradient saturates as D p approaches 0, or that it does not diverge as D p approaches 1.
However, for the scaling factor itself, we ﬁnd the D p formulation most intuitive.
The interpretation suggested by eq 7 is particularly convenient.
G MM-GAN INTERACTION WITH A DAM
We include additional details for the approximation of the Adam update step.
We combine equation 8 and g t = g 0 exp( at ) . For simplicity, we assume that sgn ( g t ) does not depend on t : if the sign of the gradient changes, m t is reduced and v t unaffected. This can contribute to making ∆ θ t vanish, but cannot prevent it. This allows us to express the momentum as a sum of contributions from each previous time step i .
[Equation]
[Equation] For reasonably large values of t and assuming a (cid:54) = log( β 1 ) , we can approximate this sum by a deﬁnite integral:
[Equation]
Using this approximation (similarly also for v t ), assuming (cid:15) to be negligible and packing assorted time independent constants including − α into C , we get the update step:
[Equation]
Having already assumed t to be large, we omit the ﬁrst fraction (bias corrections) for simplicity:
[Equation]
For sufﬁciently large values of t , behavior is determined by which of the terms in the numerator and denominator have the largest constant in the exponent (i.e. slowest decay).
We enumerate all the four possible cases:
[Equation]
Using hyperparameters such that β 1 > √ β 2 is highly unusual.
It is required to satisfy the inequalities for the last case and means that the base is greater than one in the second case.
In both cases, we get exploding update steps, which is generally undesirable.
Assuming β 1 < √ β 2 , the ﬁrst three cases show intended behavior: update steps are normalized for constant or even exploding gradients, while they diminish for vanishing gradients, enabling convergence.
The third case, a < 12 log( β 2 ) , is discussed in the main text.
In the second case, where a < log( β 1 ) , we get a different kind of vanishing behavior governed by the relationship between βand β 2 .
Additional results and discussion for this problem is given in L.
H I MPLEMENTATION DETAILS
For consistency with the theoretical framework and to allow testing of magnitude normalized gradients, we modify GANs by rescaling the generator gradient.
This amounts to calculating the minibatch gradient explicitly and multiplying it by a factor R calculated as given in algorithms 1 andbefore passing it on to the optimizer.
(Code repository: J)
Algorithm
Require: G, D l : generator and logit valued discriminator functions
Require: N : the minibatch size
Require:
z 0 , . .
z N − 1 : N generator inputs
Require: (cid:15) R : a small constant, prevents zero-division overﬂow such that R max =
(cid:15) − 1 R 1: D
[Table] = [Table]
N (cid:80) i =
[Table]
− 1 i = (
[Table]
[Table] [Table] [Table]
))) −
[Table]
Require: ∇ θ : the gradient of the cost function with respect to G ’s parameters
Require: N θ : the number of parameters of G
Require: (cid:15) R : a small constant, prevents zero-division overﬂow such that R max =
N θ (cid:15) − 1 R 1: |∇ θ | = ( (cid:80) i = N θ − 1 i = ∇ 2 θ i )2: R ←
N θ (cid:15) R + |∇ θ | However, for proper behavior with penalty terms as well as convenience, we strongly recommend implementing MM-nsat by multiplying the minimax cost by a factor R which depends on D ( G ( z )) for the entire batch and disabling back-propagation through this scaling factor.
In TensorFlow, a straightforward implementation is:
[Equation]
The rescaling factor is constant across the minibatch and can be applied either samplewise or to the total cost.
For MM-unit and NS-unit, the rescaling factors unfortunately cannot be calculated without ﬁrst computing the gradient for the unmodiﬁed cost functions.
Applying gradient rescaling directly does not interact well with penalty terms.
When the MM-GAN cost saturates, the gradient will typically be dominated by the penalties, causing both MM-nsat and
MM-unit rescaling to inﬂate their effect.
Using the approach above, rescale the adversarial minimax cost and leave penalty terms unchanged.
Parallelization may raise some issues when calculating the rescaling factor R , because proper minimax behavior requires R to be calculated across all parallels and a naive approach will introduce some overhead.
Ideally, D ( G ( z )) and gradients from parallels should be aggregated in the same step, in order to rescale gradients before running them through the optimizer.
Correctly implemented, the overhead should be unnoticeable for multi-GPU training.
We note that for very small batch sizes, or where R for other reasons vary signiﬁcantly between subsequent batches, MM-nsat will have some NS-like behaviors, because weighting between samples in different batches is disturbed by the difference between their values of R .
However, not even unmodiﬁed MM-GAN weights properly across batches, due to updates to D and optimizer corrections.
It is possible to smooth R across batches using for instance an exponential running mean, at the risk of reintroducing saturation issues.
In algorithm 2, we make the unintuitive choice of normalizing the gradient not to unity, but to N θ , the number of network parameters.
Due to cancellation effects in Adam’s parameter update step, this time-independent scaling factor is only relevant when g t is small and (cid:15) non-negligible.
Because
Adam’s ∆ θ t is calculated for each parameter independently, this choice of normalization prevents introducing an inadvertent relationship between the number of parameters of a network and whether updates vanish due to (cid:15) as shown in eq 9.
I C LASS DISTRIBUTION DIVERGENCE
Implementing a class distribution divergence requires us to have labels for the real data and some way to determine which class generated samples belong to.
This limits its applicability to simpler problems where we can train strong classiﬁers.
An additional complication is that we are applying this classiﬁer to generated data, where it may not generalize well.
The classiﬁer is calibrated for real data samples and its behavior is not well deﬁned outside of this manifold.
In particular, a standard classiﬁer is compelled by design to assign a class to all samples, even when they are so poor that they cannot meaningfully be said to belong to any class at all.
It is conceivable that a generator producing only noise-like samples will be assigned functionally random classes and achieve a very good D JS CD.
Furthermore, D JS CD is not sensitive to the degree of coverage or collapse within any given class.
In practice, we ﬁnd that a poor generator invariably scores badly in terms of D JS CD, with nearly all its samples being assigned to the same class, and that FID and D JS CD are strongly correlated.
Nonetheless, combining D JS CD with a standard generator evaluation method such as FID is recommended, and for poor generators, D JS CD may be misleading.
Similarly, the class divergence serves as a control for our use of FID for MNIST, which is signiﬁcantly removed from the natural images
FID’s Inception network in trained on.
For interpretation, note that we use a normalized Jensen-Shannon divergence, such that D JS ∈ [0 , 1] .
The lower bound corresponds to perfectly matched distributions and increasingly mode dropping generators will approach the upper bound.
When the real data distribution is non-zero for all classes, ( D JS CD ) max < 1 .
For the case where there are ten evenly distributed classes in real data, which includes MNIST and CIFAR-10, we get ( D JS CD ) max
≈ 0 .
For classiﬁcation, we train a simple, convolutional network for MNIST and a ResNet (He et al., 2015) based convolution model with batch normalization (Ioffe & Szegedy, 2015) for CIFARwith approximately 99% and 89% validation accuracy (see section J).
For MNIST-1K, we apply the MNIST classiﬁer to each channel separately, treating each ordered triplet of classiﬁcations as its own class.
In line with the established standard for FID, we use 50000 samples for estimating
D JS CD for a generator.
J C ODE AND NETWORKS
We have made code for Python 2.7 and TensorFlow available in a GitHub-repository: to maintain anonymity, see uploaded supplementary material instead.
Qualitatively reproducing the MNIST and toy experiments in the main paper is straightforward, but other datasets as well as running the evaluation metrics involves additional setup (see repository for details).
The repository includes code for training classiﬁers for D JS CD for MNIST and CIFAR-10.
Step by step setup for the CAT and FFHQ datasets is not included, nor is the forked version of StyleGAN for training it with MM-nsat.
K G RADIENT NORM AND DIRECTION FOR MM- NSAT
Figure 9 shows how the gradients obtained by the NS-GAN and MM-nsat generator costs differ during training, both in terms of norm and direction.
These results illustrate the theory from section 2.2 and give an impression of the quality of the approximation used for MM-nsat (eq 6), in order to renormalize its gradient magnitude of MM-GAN to match with that of NS-GAN.
For 4-layer convolutional networks, FID improves for both formulations with very similar quantitative results for roughly 200 epochs.
During this period, gradients are also the most similar in terms of norm and direction.
Afterwards, cosine similarity drops, indicating that gradients are becoming increasingly different for the purpose of optimization, and NS-GAN and MM-nsat diverge in terms of FID due to NS-GAN increasingly dropping modes.
For 4-layer fully connected networks, cosine similarity drops much earlier, and FID for NS-GAN falls behind very early in training, reﬂefcting mode dropping (see D JS CD for FC-4 in ﬁgure 8).
During the later stages of training, gradient norms diverge, particularly for the case where we train with the NS-GAN gradient.
Note, however, that this is a stage of training where NS-GAN shows very pathological behavior, with more than 34 of generated samples representing the digit 1.
For interpretation of the cosine similarity, note that gradients are high dimensional vectors, such that near-orthogonality is unexceptional.
Gradient magnitudes are not perfectly matched for NS-GAN and MM-nsat, particularly not later in training where values of D p become more extreme and NSGAN starts falling off in terms of FID.
When training with the MM-nsat gradient, its norm relative to the NS-GAN gradient are matched to within an order of magnitude and allows MM-nsat to train without any apparent issues. {+Figure 10 shows the same kind of plots as ﬁgure 9, but comparing the original MM-GAN with MMnsat instead. As the gradients obtained by these formulations are parallel by construction, the cosine similarity is almost exactly 1 . 
 at all times.+}
We use differently scaled y-axes for the relative gradient magnitudes, as these approach 10 16 when MM-GAN falls into its saturating failure mode. The primary difference in gradient magnitude between MM-GAN and MM-nsat is a relatively brief spike very early in training, corresponding well with the the observation by Goodfellow et al. (2014) cited in section C. This spike is both taller and wider in the FC-4 setting, likely reﬂecting G having more difﬁculty learning to generate somewhat realistic samples.
During the rest of the training, the magnitude of MM-nsat relative to MM-GAN increases slowly and consistently.
[Figure]
’s
As suggested by in section
, MM-GAN saturation depends on β 2 values causing gradients to vanish.
In particular, a lower value of β 2 for G
D disrupts the feedback mechanism that otherwise brings training to a
When D learns its task faster than G , D p ( G ( z )) will begin to approach 0 , such that both D ’s and
G ’s gradients diminish.
The effect on updates after Adam’s normalization step depends on whether the ﬁrst or second order momentum diminishes faster.
The usual case is that β 2 > β 1 , such that the second order momentum has a longer effective memory.
This partially disables normalization of updates.
For D , this works as intended: as the cost approaches its minimum, the optimizer sees the gradients becoming smaller and reduces its step size, allowing proper convergence.
The problem when optimizing MM-GAN with Adam is that this same behavior triggers for the ill-behaved G cost function when it is failing at its task.
While the update steps go in the right direction, the diminishing step size effectively causes parameters to get stuck in a poor conﬁguration.
Note that this problem does not really lie with Adam: it is the MM-GAN cost function that has a shortcoming that Adam cannot address since it only sees the decreasing gradients and has no way of knowing that this actually means that the MM-GAN cost is increasing .
The role of the gradient rescaling in equation 6 is to make it so that G ’s gradients and cost increase and decrease in parallel, such that the optimizer behaves properly. L. T HE A DAM β 2 HYPERPARAMETER
Here, we provide additional empirical results (ﬁgure 11).
In particular, we investigate the importance of how quickly D and G ’s updates vanish, relative to each other , by adjusting the β 2 parameters of
D and G (our default is β 2 = 0 . 999 ).
After all, the theory in 2.4 assumes a gradient that is simply exponential in time and does not model the interactions between D and G when the generated data gradients begin to vanish.
A fairly straightforward pattern emerges: for the standard case, where D and G have the same value of β 2 , there are multiple U -shaped dips, where gradients alternate between diminishing and recovering, until the gradient ﬁnally vanishes for good.
This still holds when when we reduce β 2 for both optimizers, effectively maintaining their balance.
However, both setups where β 2 is lower for
G than for D avoids vanishing updates entirely, whereas when we reduce the value of β 2 only for
D , training breaks down earlier and more decisively.
Finally, we show that we can avoid vanishing gradients by simply reinitializing G ’s optimizer at regular intervals: this resets both the ﬁrst and second order momenta.
This stabilizes the training process, which is somewhat surprising seeing as we are effectively crippling the optimizer’s memory.
However, this effect is well explained by our hypothesis of large, lingering contributions to the second order momentum disabling normalization.
L. L INEAR MODEL FOR D l ( G ( z ))
Figure 12 shows more detailed diagnostics for two runs where MM-GAN with standard Adam settings falls into its saturating failure mode.
In addition to the gradients, we also plot the following:
D l , the mean of the discriminator’s logit outputs, both for real and generated data; and | ∆ θ | and | ∆ φ | , the norm of the parameter update for G and D at each timestep.
In section 2.4, we assume that D l ( G ( z )) behaves as a linear function of training steps. Note that this simple model is not intended to fully capture the training dynamics for MM-GAN, but to explain why Adam does not reliably address MM-GAN saturation despite its built in normalization. The actual behavior of D l ( G ( z )) in ﬁgure 12 is more complicated than we assume in our model. Particularly, | D l | grows much faster early on than it does towards the end, where it eventually tapers off.
However, for D l ( G ( z ))
= − 20 , D p ( G ( z ))
≈ 2 ∗ 10 − 9 , whereas G only produces a total of 6 ∗ 10 7 samples during the 1000 epochs we use to train the networks.
The behavior of D l past this point is of limited interest.
We ﬁnd an early, critical period most relevant for understanding the interaction between Adam and the saturating MM-GAN cost, characterized by D l becoming monotonic for each of real and generated data.
In ﬁgure 12, these critical periods start around 0 .
4 and 2 .
9 epochs for Conv-4 and
FC-4 respectively, without any clearly deﬁned endpoint.
Within a single epoch, the step size for G ’s updates drops by 3 to 4 orders of magnitude, and G ( z ) ceases to change during training, as shown in ﬁgure 13.
The exception to the above is the sudden spike after 100 epochs for FC-4.
Ideally, this spike would allow the training process to recover, as the size of G ’s gradients and updates increases.
However, while gradients are larger after this spike than before, the update step size plummets by an additional 4 orders of magnitude, due an unusually large contribution to the second order momentum effectively disabling Adam normalization for a long while, a very illustrative example of the problematic interaction between the saturating MM-GAN cost and Adam.
L. A DDITIONAL COMMENTS
Stabilizing MM-GAN by use of the β 2 parameter is not recommended: we intended these experiments to demonstrate the effects discussed in section 2.4.
The setting where only G ’s β 2 is reduced to 0 .
99 suffers the least from noisy updates and performs well in some settings, but all these variants are less stable than the more principled MM-nsat version.
However, it should be noted that instead of changing the MM-GAN cost to work with Adam, as we do for MM-nsat, it is possible to modiﬁy the optimization procedure to account for MM-GAN’s peculiarly shaped cost function.
NS-GAN owes some of its success to how D and G have different scaling factors.
Only D ’s gradients vanish as D p ( G ( z ))
→ 0 , and only G ’s gradients vanish as D p ( G ( z ))
→ 1 .
This gives the training process a limited form of self-stabilization: update step sizes (roughly equivalent to learning rates) increase for whichever of the networks is doing worse.
Since MM-nsat has the same gradient magnitudes as NS-GAN, this behavior carries over. [Figure]
M H INGE -GAN AND LS-GAN RESULTS
Figure 14 shows results as in ﬁg 8 in the main text, but with a different set of cost functions.
Fig 1 shows a subset of these plots.
For the bottom ﬁgure, note the use of spectral normalization and a much smaller interval of FID values than other plots.
LS-GAN (Mao et al., 2016) and Hinge-GAN (Miyato et al., 2018) are used in the comparisons because they only require modiﬁed cost functions and can be directly compared to NS-GAN and
MM-nsat.
The widely used WGAN (Arjovsky et al., 2017) formulation requires additional changes to prevent D ’s outputs from diverging, most commonly a 1-centred gradient penalty on interpolations between real and generated data (Gulrajani et al., 2017).
Hinge-GAN is very similar to WGAN, but simply clips D ’s cost outside of the (0 , 1) interval and is known to produce strong results in for instance
BigGAN
(Brock et al., 2018).
Generally speaking, Hinge-GAN performs better than NS-GAN but worse than MM-nsat and suffers from some of the same gradual mode collapse issues as NS-GAN.
Where Hinge-GAN and MM-nsat are most similar in terms of FID, MM-nsat tends towards better class balance.
LS-GAN performs remarkably well for MNIST Conv-4, but is otherwise unimpressive and suffers the most from stability issues.
Interestingly, the results suggest that the mode behavior is not speciﬁc to NS-GAN, but also found in other cost formulations.
MM-GAN is very unusual in that it emphasizes high-scoring samples, which is both somewhat counterintuitive and gives rise to the saturation problem which we work around with rescaling the minibatch gradient.
N
MM-GAN AND NS-GAN LINEAR COMBINATIONS
Motivated by theory connecting GAN divergences and mode dropping (section 2.3) and Husz´ar (2016) who suggests adding together the NS-GAN and MM-GAN cost functions in order to get rid of the subtracted Jensen-Shannon divergence for NS-GAN (eq 3), we run experiments using linear combinations of the cost function.
Husz´ar’s suggestion changes the scaling factor (1 − D p ) → 1 , which is an improvement in terms of the effect discussed in section 2.3, since it reduces the emphasis on overrepresented modes.
However, if D p ( G ( z )) is close to zero during training, as often happens with GANs since D has an easier task than G , there will be little difference in practice.
We use the following linear interpolations of the MM-GAN and NS-GAN cost functions, parametrized by a which determines the weight of each term:
[Equation]
Due to the MM-GAN and NS-GAN scaling factors (see ﬁg 2), we expect linear combinations of
MM-GAN and NS-GAN to be dominated by the non-saturating term, since D p ( G ( z ))
tends to be small when training GANs unless D is strongly regularized.
On the other hand, using our modiﬁed
MM-nsat approximately balances the gradient norm for each term, such that the behavior of linear combinations should correspond to the weighting of the NS-GAN and MM-nsat terms.
Figure 15 shows results using linear combinations of NS-GAN, MM-GAN and MM-nsat for G ’s cost.
Results for NS-GAN and MM-nsat are as shown in the main text in ﬁgure 8.
The behavior for NS-GAN and MM-nsat interpolations is most straightforward, with numerical results falling inbetween the pure NS-GAN and MM-nsat variants with roughly the degree of separation suggested by their weighting constants.
Since NS-GAN and MM-nsat have approximately equal gradient magnitudes for all values of D p ( G ( z )) , the contributions from each term are predictable.
For NS-GAN and MM-GAN interpolations, behavior is less consistent and more similar to NSGAN.
This is reasonable, seeing as NS-GAN and MM-GAN have scaling factors 1 − D p and D p respectively, such that the two terms are effectively weighted both explicitly by the weighting constant and implicitly by whatever the values of D p ( G ( z )) happen to be during training.
Note that if D completely fails to discriminate between real and generated samples, it can minimize its cost by making D p ( x ) =
D p ( G ( z ))
= 0 .
5 , such that NS-GAN’s scaling factor will almost always be larger than that of MM-GAN.
For the FC-4 results, note that even a 910 weighting for MM-GAN still gives behavior qualitatively similar to NS-GAN: the early stopping metrics are much better than for pure NS-GAN and the mode collapsing behavior is delayed, but the ﬁnal metrics are worse both in terms of FID and D JS CD even than for the 13 -weighted MM-nsat.
Generally, Husz ´ ar’s suggestion of adding together the NS-GAN and MM-GAN leads to somewhat improved results, but does not come close to stability and mode coverage achieved by MM-nsat.
Compare also to the divergences shown in eq 3.
[Figure]
Figure 16 shows results for MM-nsat and NS cost functions when combined with zero-centred real data gradient penalty (Mescheder, 2018).
See also results for spectral normalized networks at the bottom in ﬁgures 8 and 14, where results are much more similar than in the unregularized case: this still holds true with different cost functions and a different form of regularization.
This is essentially a special case of an effect shown in Qin et al.
(2018), that all strongly regularized cost functions degenerate into the same behavior.
The key result is how strong regularization makes the performance of MM-nsat and NS (and other cost functions) much more similar.
Some form of regularization is often used to obtain strong results, since the introduction of various forms of gradient penalties such as the one used in WGAN-GP (Gulrajani et al., 2017) and spectral normalization (Miyato et al., 2018).
Discriminator regularization has many effects, such as reducing the ability of D to learn quickly and smoothing the landscape of D ’s outputs.
It is difﬁcult to fully understand how it affects the training process for GANs.
Generally speaking, limiting the Lipschitz value of D (as is done explicitly by spectral normalization and partially by zero-centred real data gradient penalty) ﬂattens the shape of
D and ultimately limits the difference between the outputs for real and generated data.
It follows that assuming that D ≈ D opt as given by eq 7 is less reasonable for a regularized discriminator, such that the effect discussed in section 2.3 will be less pronounced.
As discussed in section 2.1, the difference between MM-GAN and NS-GAN is only in the scaling factors, D p and 1 − D p respectively.
Strong regularization effectively squeezes D p towards 0 .
5 : this makes D p and 1 − D p more similar, and thus also MM-nsat and NS, as seen in ﬁg 16.
For the runs we have plotted, regularization vastly improves the results for NS, but the beneﬁts for MM-nsat are much more limited.
The common default value of the weighting for the gradient penalty, λ = 10 1 , is actually somewhat harmful for MM-nsat compared to the unregularized case, and looking only at values for this regularization gives the misleading impression that NS is stronger than MM-nsat.
We consider a more thorough discussion of this problem out of scope for this work, but suggest that it might explain the fairly similar results for MM-nsat and NS when applied to the StyleGAN model (see U).
Absent a better understanding of the interactions between discriminator regularization and cost functions, hyperparameters for regularization terms should be tested extensively when applying
MM-nsat.
P S CALING FACTOR EXPERIMENTS
In section 2.1, we show that the gradients for the MM-GAN and NS-GAN generators have different scaling factors:
[Equation]
In section 2.3, we suggest that the different emphasis this places on under- and oversampled modes is the reason for the more zero-avoiding and mode-covering behaviors of NS-GAN and MM-GAN.
To test this hypothesis, we can modify the scaling factors and see whether we get the expected change in behavior.
The linear interpolations used in section N probe the same effect in a less direct manner.
While we expect the importance of the scaling factors to depend on the relative weights they assign to different scoring samples, it is not clear which relationships are most crucial.
The difference between MM-GAN and NS-GAN is most pronounced for low- and high-scoring samples, but the important difference might very well be the relative weights of two samples with different, low scores.
A simple way to modify the NS-GAN gradient is to add a constant a to the scaling factor:
[Equation]
Which is obtained by the following single-sample cost function:
[Equation]
Renormalizing the total gradient magnitude in the same way as for MM-nsat, we get the following cost function:
[Equation]
And similarly for MM-GAN:
[Equation] [Equation]
These cost functions are only reasonable for a ≥ 0 , such that we avoid negative scaling factors for samples.
For increasingly large values of a , sample weights effectively become uniform across the whole range of possible scores.
Since these modiﬁcations only allow us to make MM and NS less extreme and more similar to each other, we also try a different approach, introducing a exponentiation parameter c for the scaling factor:
[Equation]
The general solution of this differential equation can be expressed in terms of the hypergeometric function, which does not lend itself to efﬁcient computation.
However, speciﬁc values of c give rise to useful cost functions.
As before, we consistently renormalize the gradient magnitude to that of
NS-GAN:
[Equation] [Equation]
[Equation]
Renormalizing the overall gradient magnitude goes a long way towards stabilizing the adversarial training dynamics for these cost functions.
However, overemphasis on oversampled modes tends to accelerate catastrophical mode collapse.
Furthermode, expressions such as log( √ e D l )) are numerically unstable for large values of | D l | .
Results using these variant cost functions are shown in ﬁgure 17.
The add -variants have very clean behavior, all falling in between NS and MM-nsat in terms of FID and D JS CD, in the same order as suggested by their more uniform scaling factors.
The exp -variants are more erratic, with NS-exphaving major stability issues and MM-nsat-exp-2 falling off later in training for CIFAR-10 Conv-4.
Aside from these points, results correspond with our theoretical expectations.
Perhaps the most striking result is the strong performance of the MM-nsat-exp-2 cost function.
This strange variant is designed simply to have a more extreme version of the minimax scaling factor, which we expect to further temper the the mode-dropping mechanism described in section 2.3.
Indeed, it generally improves performance relative MM-nsat.
Finally, we note that all the MM-nsat-add variants tend towards stronger mode collapse than the
MM-nsat-exp variants, regardless of the choice of parameter.
We suggest the following explanation for this behavior: Consider two generated samples, both such that D p ( G ( z )) is close to 0.
With NS sample weighting or with MM-nsat-add variants, the relative weights of these samples will be close to 1, simply because a small D p is negligible relative to the additive constants.
With MM sample weighting and MM-nsat-exp variants, on the other hand, relative weights have a strong dependence the exact values of D p for each sample and
In particular, MM-nsat-exp-2, using a scaling factor with even more emphasis on underrepresented modes, achieves better values of FID and D JS CD than the more standard cost functions.
Q R ING OF G AUSSIANS
The ring of Gaussians toy problem has a number of degrees of freedom, both for the dataset (number of modes, number of standard deviations of separation) and the model (architecture, batch size, training iterations).
There are settings where both MM-GAN and NS-GAN cover or drop modes and signiﬁcant variation between individual runs.
We do not present a thorough analysis of this problem: in the main text, we use settings chosen to obtain the same qualitative NS-GAN behavior as shown in Metz et al.
; Srivastava et al.
To give some impression of the diversity of possible behaviors and the usual differences between
NS-GAN and MM-GAN, we include results for a more challenging toy problem in ﬁgure 18 and table 3.
In this case, where NS-GAN drops only some modes, we again ﬁnd that MM-GAN has qualitatively better mode coverage.
Furthermore, frequencies of samples from each mode is much better aligned with real data for MM-GAN than for NS-GAN.
Interestingly, while NS-GAN places particular emphasis on avoiding generated samples outside of the real data manifold, more samples fall outside of the real data modes for NS-GAN than for MM-GAN.
MM-GAN trains well on this toy problem despite its saturating cost function and shows decent coverage of all modes.
Cross-reference with table
[Table]
R MNIST-1K
We run tests on the Stacked MNIST dataset (Metz et al., 2016)
(also known as MNIST-1K), in order to get a more demanding and multi-modal training task where we can still classify generated samples.
The samples in this dataset are obtained by combining three samples from MNIST to represent each of the three color channels for an RGB image.
We ﬁnd a variety of network architectures difﬁcult to train for any of our cost functions and resort to regularizing D .
We show results in ﬁgure 19: in both cases, differences between linear combinations are minor.
For the most heavily regularized model, there is no clear trend.
Removing spectral normalization and relaxing the zero centred gradient penalty improves the models and makes them slightly more different, recovering the usual ordering in terms of D JS .
S CAT 128 X
[Figure]
We run tests with CAT 128x
(Zhang et al., 2008), chosen as a reasonably difﬁcult dataset which is still amenable to simple network architectures.
FID during training for two architectures is shown in ﬁgure 20, emphasizing training stability.
Figures 21a and 21b show samples for DCGAN with spectral normalization for D and self attention: these are shown cropped in the main text in ﬁgure 7.
Figures 21c, 21d and 21e show samples for other network architectures.
Note that samples are
JPEG compressed.
We ﬁnd that MM-unit and MM-nsat perform much better than NS in this setting.
In particular, we ﬁnd that the failure mode of NS cannot be addressed by early stopping in these experiments, unlike for MNIST, where NS tends to produce better results early on and mode dropping is mostly due to extended training (see ﬁgure 8).
For CAT 128x128, catastrophical mode collapse tends to happen before G has learned to produce reasonable samples.
MM-nsat is much less susceptible to this failure mode than NS.
As usual, results are most similar with regularized discriminator.
T FFHQ AT VARIOUS RESOLUTIONS
Figure 22 shows results for training simple convolutional GANs on the FFHQ dataset downsampled to various resolutions, comparing NS-GAN and MM-nsat.
The differences between the two cost functions are generally as seen in other experiments.
Aside from the decent early stopping results for
NS-GAN even at high resolutions the and relatively strong performance of MM-nsat even at 512resolution with very unsophisticated network architectures, these results only replicate previously discussed effects.
Figure 23 shows samples from training at 512 2 , showcasing the mode collapsing behavior that causes FID to increase.
Note that samples are JPEG compressed
[Figure]
[Table]
U S TYLE GAN
We run experiments training StyleGAN
(Karras et al., 2018), replacing the traditional NS loss used in the original implementation with our MM-nsat.
Due to available resources, we train each model using only 4 GPUs.
We make no other adjustments.
We show results in table 4 and ﬁgures 24a and 24b.
The primary result from these tests is that MM-nsat performs reasonably well, albeit worse than
NS-GAN, as a drop-in replacement in a sophisticated, state of the art implementation.
In the full resolution case, which is carefully optimized by the original authors (for instance learning rate adjustments throughout training and a speciﬁc value of truncation for the latent input for G ), NS achieves much better FID and MM-nsat samples seem to have more artifacts.
For the lower resolution case, which is a somewhat more fair comparison, FID values are closely matched: NS gets the best ﬁnal value, whereas MM-nsat has the best early stopping value.
Notably, StyleGAN uses zero centred gradient penalty and minibatch standard deviation features in D (see O).
Both of these can serve to limit the failure modes we have shown for NS-GAN.
For
StyleGAN, coverage is limited, particularly for the latent truncation settings that achieve the best
FID (Kynk ¨ a ¨ anniemi et al., 2019), and visual quality of samples is the ﬁrst priority.
Implementing MM-nsat for StyleGAN requires some minor modiﬁcations, which essentially consist of accumulating values of D p from parallels in the same way the original implementation does for gradients and rescaling the total gradient before passing it back to the parallels to update parameters for each network clone.
[Figure] [Figure]
For the CIFAR-10 Conv-4-sn results at the bottom of ﬁgure 8, we use exceedingly simple networks as described in section 3.3.
In order to compare MM-nsat against a benchmark from the literature, we the copy the settings from Miyato et al.
(2018), i.e. the network architectures from table 3 and the hyperparameter settings C from table 1 .
The main differences between this setup and our Conv-4-sn settings are:
• Batch normalization in G • Additional 1-stride convolutions in D • Spectral normalization also for the ﬁnal layer of D • Leaky ReLU in D •
Approximately 10 times as many network parameters • Doubled learning rate
Additionally, for comparing FID directly against values tabulated in Miyato et al.
(2018), note the following:
• Likely different batch size and training epochs •
We use all real and 50k generated images to estimate FID, instead of 10k and 5k
Across ﬁve runs, we obtain median FID values of 27 .
4 for NS-GAN and 25 .
5 for MM-nsat, using the same set of hyperparameters as Miyato et al.
(2018) optimized for their NS-GAN results, reporting a value of 29 . 3 .
As in other experiments, we again see that D JS CD favors MM-nsat, even at the points during training where FID values are closely matched.
Unlike for our networks, we see no clear tendency for the NS-GAN FID to increase towards the end of training, possibly due to higher quality of generated samples and spectral normalization also in the ﬁnal layer of D combining to pull D p values closer to 0 .
5 throughout training.
While there is signiﬁcant overlap between the cost functions, MM slightly better performance overall.