T HE
U NUSUAL E FFECTIVENESS OF A VERAGING IN GAN T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
We examine two different techniques for parameter averaging in GAN training.
Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum.
Whilst MA is known to lead to convergence in bilinear settings, we provide the ﬁrst to our knowledge theoretical arguments in support of EMA.
We show that
EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and moreover enhance the stability of general GAN training.
We establish experimentally that both techniques are strikingly effective in the non convex-concave GAN setting as well.
Both improve inception and FID scores on different architectures and for different GAN objectives.
We provide comprehensive experimental results across a range of datasets – mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet – to demonstrate its effectiveness.
We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.
I NTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are two player zero-sum games.
They are known to be notoriously hard to train, unstable, and often don’t converge.
There has been a lot of recent work to improve the stability of GANs by addressing various underlying issues.
Searching for a more stable function family (Radford et al., 2015) and utilizing better objectives (Arjovsky et al., 2017; Gulrajani et al., 2017) have been explored for training and instability issues.
For tackling non-convergence, regularizing objective functions (Mescheder et al., 2017; Salimans et al., 2016), regularizing optimizers (Daskalakis et al., 2018; Heusel et al., 2017b) have been studied.
One reason for non-convergence is cycling, where the players move around an optimal solution but do not converge to it (Mertikopoulos et al., 2018; Daskalakis et al., 2018; Bailey & Piliouras, 2018).
There might be multiple reasons for this cycling behavior: (i) eigenvalues of the Jacobian of the gradient vector ﬁeld have zero real part and big imaginary part (Mescheder et al., 2017).
(ii) Gradients of the discriminator may not move the optimization in the right direction and may need to be updated many times to do so (Arjovsky et al., 2017).
(iii) Sampling noise causes the discriminator to move in different directions at each iteration, as a result of which, the generator ﬂuctuates.
(iv) Even if the gradients of the discriminator point in the right direction, the learning rate of the generator might be too large.
When this overshooting happens, the gradients of the discriminator lead in the wrong direction due to over-conﬁdence (Mescheder et al., 2018).
In this work, we explore in detail simple strategies for tackling the cycling behavior without inﬂuencing the adversarial game.
Our strategies average generator parameters over time, outside the training loop.
Averaging generator and discriminator parameters is known to be an optimal solution for convex-concave min-max games (Freund & Schapire, 1999).
However, no such guarantees are known (even for bilinear games) if we apply exponential discounting.
Our contribution are: (i) We show theoretically that although Exponentially Moving Average does not converge to equilibrium even in simple bilinear games, it nevertheless helps to stabilize cyclic behavior by shrinking its amplitude.
(ii) We demonstrate that both averaging techniques consistently improve results for several different datasets, network architectures, and GAN objectives.
(iii)
We compare this approach with several other methods which try to alleviate the cycling or non-convergence problem, and demonstrate its unusual effectiveness.
R ELATED W ORK
There have been attempts to improve stability and alleviate cycling issues arising in GANs.
Salimans et al.
(2016) uses historical averaging of both generator and discriminator parameters as a regularization term in the objective function: each model deviating from its time average is penalized to improve stability.
Ge et al.
(2018) updates the discriminator by using a mixture of historical generators.
Heusel et al.
(2017b) uses two time scale update rules to converge to a local equilibrium.
Mescheder et al.
(2017) states that a reason for non-convergence is due to presence of zero real part and big imaginary part of eigenvalues of the Jacobian of the gradient vector.
They address this issue by including squared gradients of the players to the objective.
Daskalakis et al.
(2018) includes a regret mechanism in the optimization to regularize non-convergence.
Some of the above stated methods such as Salimans et al.
; Mescheder et al.
(2017) inﬂuence the min-max game by introducing regularization terms into the objective function.
Regularizing the adversarial objective can potentially change the optimal solution of the original objective.
Our method on the other hand does not change the game dynamics, by keeping an average over iterates outside of the training loop.
Averaging at maximum likelihood objective: Uniform Average and Exponential Moving Average
over model parameters has been used for a long time (Polyak & Juditsky, 1992; van den Oord et al., 2017; Athiwaratkun et al., 2018; Izmailov et al., 2018).
However we would like to stress on the difference between convex/non-convex optimization and saddle point optimization.
GANs are an instance of the latter approach.
There is no formal connection between the two approaches.
So, we can’t extrapolate our understanding of averaging at maximum likelihood objectives to saddle point objectives.
Averaging at minimax objective: (Gidel et al., 2018) has used uniform averaging similar to our work, however we have showed the shortcomings of it and how exponential decay averaging performs better.
While this strategy has been used in (Karras et al., 2017) recently (see Appendix of their paper), their paper lacks any insights or a detailed analysis of this approach.
M ETHOD
In case of convex-concave min-max games, it is known that the average of generator/discriminator parameters is an optimal solution (Freund & Schapire, 1999), but there is no such guarantee for a non-convex/concave setting.
Moving Average (MA) over parameters θ is an efﬁcient implementation of uniform averaging without saving iterates at each time point:
[Equation]
When the generator iterates reaches a stationary distribution, Eq.1 approximates its mean.
However there are two practical difﬁculties with this equation.
First we do not know at which time point the iterates reach a stationary distribution.
Second, and more importantly, iterates may not stay in the same distribution after a while.
Averaging over samples coming from different distributions can produce worse results, which we have also observed in our own experiments.
Because of the issues stated above, we use Exponential Moving Average (EMA):
[Equation]
) where θ (0) EMA =
θ (0) .
EMA has an effective time window over iterates, where early iterates fade at a rate depending on the β value.
As β approaches to 1 . 0 , averaging effectively computes longer time windows.
Both EMA and MA offer performance gains with EMA typically being more robust (for more details see section 5).
Averaging methods are simple to implement and have minimal computation overhead.
As they operate outside of the training loop, they do not inﬂuence optimal points of the game.
Hyperparameter search for EMA can be done with a single training run by simultaneously keeping track of multiple averages with different β parameters.
W HY DOES AVERAGING WORK ? 4. G LOBAL ANALYSIS FOR SIMPLE BILINEAR GAMES
As we will show experimentally parameter averaging provides signiﬁcant beneﬁts in terms of GAN training across a wide range of different setups.
Although one cannot hope to completely justify this experimental success theoretically in general non-convex-concave settings, it is at least tempting to venture some informed conjectures about the root causes of this phenomenon.
To do so we will focus in the simplest possible class of saddle point problems, the class of planar bilinear problems (i.e. zero-sum games such as Matching Pennies).
This is a toy, albeit classic, pedagogical example used to explain phenomena arising in GANs.
Speciﬁcally, similar to Goodfellow (2017) we will also move to a continuous time model to simplify the mathematical analysis.
In terms of the moving average method, it is well known that time-averaging sufﬁces to lead to convergence to equilibria (Freund & Schapire, 1999).
In fact, it has recently been established that the smooth (continuous-time) analogues of ﬁrst order methods such as online gradient descent (follow-the-regularized leader) in bilinear zero-sum games are recurrent (i.e. effectively periodic) with trajectories cycling back into themselves (Mertikopoulos et al., 2018).
As a result, the timeaverage of the system converges fast, at a O (1 /T ) rate, to the solution of the saddle point problem (Mertikopoulos et al.
In contrast, not much is known about the behavior of EMA methods even in the simplest case of zero-sum games.
As we will show by analyzing the simplest case of a bilinear saddle problem max x min y xy even in this toy case EMA does not lead to convergent behavior but instead reduces the size of the oscillations leading to more stable behavior whilst enabling exploration in a close neighborhood around the equilibrium.
In this case the gradient descent dynamics are as follows:
[Equation]
This corresponds to a standard Hamiltonian system whose energy function is equal to H = x 2 + y 2.
All trajectories are cycles centered at 0 .
In fact, the solution to the system has the form of a periodic orbit ( x ( t ) , y ( t ))
= ( a cos( t ) , a sin( t )) .
These connections have been recently exploited by Balduzzi et al.
(2018) to design training algorithms for GANs
but here we will be taking a different approach.
For simplicity let’s assume a = 1 .
The Exponential Moving Average method now corresponds to an integral of the form C · (cid:82) T
β T − t θ ( t ) dt where θ ( t ) =
(cos( t ) , sin( t )) and C a normalizing term such that the EMA of a constant function is equal to that constant (i.e. C = 1 / (cid:82) T 0 β T − t dt ).
Some simple algebra implies that C · [Equation] t
[Equation] As the terms β T vanish exponentially fast, this function reduces to [Equation] ) (cos( T ) + ln( β ) sin( T )) .
This is a periodic function, similarly to the components of θ ( t ) but its amplitude is signiﬁcantly reduced and as a result it is more concentrated around its mean value (which is the same as its corresponding θ ( t ) component, i.e. zero).
For β some constant that is very close to one, e.g. β = 0 .
999 this periodic function can be further approximated by ln( β ) cos( T ) .
In other words, no matter how close to one we choose the discounting factor β , even in the toy case of a planar bilinear saddle problem (i.e. the Matching Pennies game)
EMA has a residual non-vanishing periodic component, alas of very small amplitude around the solution of the saddle point problem. 4. L OCAL S TABILITY A NALYSIS FOR GAN S
The model of the previous section introduced two simpliﬁcations so as to be theoretically tractable: i) it focused on a (small) bilinear game and ii) it was a continuous time model.
Despite these simpliﬁcations the global behavior was still be rather complex indicating the hardness of analyzing completely a more realistic model.
Nevertheless, in this section we examine such a model (discretetime non-convex loss functions) and show that EMA preserves the stability of locally stable ﬁxed points (Nash equilibria) providing further theoretical evidence in support of averaging. Due to the complexity of GANs, local stability analysis is often used a method for obtaining theoretical insights, e.g., Nagarajan & Kolter (2017); Mescheder et al. Suppose we have a generator G parameterized by θ ∈ R p , and a discriminator D parameterized by φ ∈ R q ; let x = ( θ, φ ) ∈ R n , n = p + q . Then, during training, G and D minimize loss functions
L ( θ ) ( θ, φ ) and L ( φ ) ( θ, φ ) respectively.
Following the same formulation as in (Gidel et al., 2018):
[Equation]
For speciﬁc possible instantiations of these loss functions see Goodfellow et al.
Arjovsky et al.
We deﬁne the gradient vector ﬁeld V ( θ, φ ) ∈ R n and its Jacobian J V ( θ, φ ) ∈ R n × n as follows:
[Equation]
In this analysis, we assume that G and D are trained with some method that can be expressed as the iteration
[Equation]
for some function F , and where x t =
( θ t , φ t ) denotes the parameters obtained after t training steps.
This framework covers all training algorithms that rely only on a single past iterate to compute the next iterate, and includes simultaneous gradient descent and alternating gradient descent methods.
More concretely, simultaneous gradient descent with a learning rate η can be written in this form as follows
[Equation]
EMA then averages the iterates x t to produce a new sequence of iterates ˆ x t such that
[Equation]
(by deﬁnition (2))
[Equation]
(Replace x t using Eq.
[Equation]
(Replace x t − 1 using deﬁnition (2))
We have thus expressed the EMA algorithm purely in terms of a new set of iterates ˆ x t .
Note that computing the next iterate now requires the past two iterates.
Flammarion & Bach (2015) observed that the standard moving average can similarly be expressed as a second-order difference equation.
We can then deﬁne the EMA operator ˆ F β ( x t , x t − 1 ) such that [ x t x t − 1 ]
= ˆ F β ( x t −
[Equation]
x t −
[Equation]
The Jacobian of ˆ F β , J ˆ F β is therefore
[Equation]
We can now proceed to explore the behavior of the EMA algorithm near a local Nash equilibrium by examining the eigenvalues of its Jacobian J ˆ F β , which are described by Theorem 1.
Theorem 1.
The eigenvalues of the Jacobian J ˆ F β ( x ∗ , x ∗ ) at a local Nash equilibrium x ∗ are β (with multiplicity n ) and the eigenvalues of J F ( x ∗ ) , the Jacobian of the training operator at the equilibirum.
Proof.
We solve det ( γ I 2 n − J ˆ F β )
{+= 
+}
to obtain eigenvalues γ .
Some algebra reveals that
[Equation]
where we used the following equality (e.g. see Lemma 1 in Gidel et al.
(2018)): | A B C D | = |
AD − BC | if C and D commute.
{+Thus, det ( γ I 2 n − J ˆ F β ) = 
+}
if γ = β or det ( γ I n − J F ( x ∗ ))
{+= 
+}
so γ is an eigenvalue of J F ( x ∗ ) .
Given that 0 < β < 1 in the EMA algorithm, we have the following corollary that formalizes the intuition that the dynamics of EMA near a local Nash equilibrium are tightly linked to those of the underlying training algorithm.
Corollary 1.1.
Assuming that 0 < β < 1 , the eigenvalues of J F lie within the unit ball if and only if the eigenvalues of J ˆ F β lie within the unit ball.
Also, we see that EMA could potentially slow down the rate of convergence compared to the underlying training method if β is larger than the magnitude of all other eigenvectors of J F β .
We use both illustrative examples (i.e., mixtures of Gaussians) as well as four commonly used real-world datasets, namely CIFAR-10 (Krizhevsky et al.)
, STL-10 (Coates et al., 2011), CelebA (Liu et al., 2015), and ImageNet (Russakovsky et al., 2015) to show the effectiveness of averaging.
For mixtures of Gaussians, the architecture, hyperparameters and other settings are deﬁned in its own sections.
For CelebA, 64x64 pixel resolution is used, while all other experiments are conducted on 32x32 pixel images.
All dataset samples are scaled to [ − 1 , 1] range.
No labels are used in any of the experiments.
For the optimizer, we use ADAM (Kingma & Ba, 2014) with α = 0 . 0002 , β 1 = 0 . 0 and β 2 = 0 .
For the min-max objective, we use the original GAN (Goodfellow et al., 2014) objective and the
Wasserstein-1 (Arjovsky et al., 2017) objective, with Lipschitz constraint satisﬁed by gradient penalty (Gulrajani et al., 2017).
Spectral normalization (Miyato et al., 2018) is only used with the original objective.
In all experiments, the updates are alternating gradients.
The network architecture is similar to a full version of a progressively growing GAN (Karras et al., 2017), with details provided in the Appendix.
We call this architecture conventional.
ResNet from (Gulrajani et al., 2017) is used as a secondary architecture.
Unless stated otherwise, the objective is the original one, the architecture is conventional, discriminator to generator update ratio is 1, β value is 0 .
9999 and MA starting point is 100k.
Inception (Salimans et al., 2016) and FID (Heusel et al., 2017a)
scores are used as quantitative measures.
Higher inception scores and lower FID scores are better.
We have used online Chainer implementation 1 for both scores.
For Inception score evaluation, the guideline from Salimans et al.
(2016) is followed.
FID score is evaluated on 10k generated images and statistics of data are calculated at the same scale of generation e.g. 32x32 data statistics for 32x32 generation.
The maximum number of iterations for any experiment is 500k.
For each experiment, we state how long it has been trained.
At every 10k iterations, the Inception and FID scores are evaluated, and we pick the best point.
All experiments are repeated at least 3 times with random initialization to show that the results are consistent across different runs.
In addition to these experiments, we also compare the averaging methods, with Consensus Optimization (Mescheder et al., 2017), Optimistic Adam (Daskalakis et al., 2018) and Zero Centered
Gradient Penalty (Mescheder et al., 2018) on the Mixture of Gaussians data set and CIFAR-10.
For
CIFAR-10 comparison, we use DCGAN Radford et al.
(2015) like architecture and follow the same hyperparameter setting as in Mixture of Gaussians except WGAN-GP’s λ which is 10 . 0 .
To ensure a fair comparison, we use the same samples from the prior distribution for both averaged and non-averaged generators.
In this way, the effect of averaging can be directly observed, as the generator generates similar images.
We did not cherry-pick any images: all images generated are from randomly selected samples from the prior.
For extended results refer to the Appendix.
5. M IXTURE OF G AUSSIANS
This illustrative example is a two-dimensional mixture of 16 Gaussians where the mean of each
Gaussian lies on the intersection points of a 4x4 grid.
Each Gaussian is isotropic with σ = 0 . 2 .
Original GAN (Goodfellow et al., 2014) is used as the objective function with Gradient Penalty from
Gulrajani et al.
(2017), with 1:1 discriminator/generator update and alternative update rule.
We run the experiment for 40k iterations, and measure Wasserstein-1 2 distance with 2048 samples from both distributions at every 2k iteration.
We take the average of the last 10 measurements from the same experiment and repeat it 5 times.
We compare this baseline with Optimistic Adam (OMD), Consensus
Optimization (CO), and Zero Centered Gradient Penalty (Zero-GP) by using the same architecture, and following the original implementations as closely as possible.
We have also combined EMA and
MA with OMD, CO and Zero-GP to see whether they can improve over these methods.
Detailed settings are listed in the Appendix.
Table 1 shows the distance for various methods.
In all cases, EMA outperforms other methods, and interestingly, it also improves OMD, CO and Zero-GP.
This indicates that OMD, CO and Zero-GP do not necessarily converge in non-convex/concave settings, but still cycle.
MA also improves the results in certain cases but not as strong as EMA.
Our observation is uniform averaging over long time iterates in the non-convex/concave case, hurts the performance (More on this in later experiments).
Because of this, we have started uniform averaging at later stages in training and treat it as hyper-parameter.
For this experiment, start interval selected as 20k.
[Table]
Figure 5 is illustration of the results above.
The blue regions indicate a lack of support, while the red ones indicate a region of support.
Without averaging, the generator omits certain modes at different time points, and covers them later while omitting previously covered modes.
We observe a clear ﬂuctuation in this case, while the EMA consistently shows more balanced and stable supports across different iterations.
OMD, CO and Zero-GP perform better than the baseline, however they seem to be less stable across different iterations than EMA results
5. Q UALITATIVE S CORES
Table 2 tabulates Inception and FID scores with no averaging, EMA and MA.
In all datasets, signiﬁcant improvements are seen in both scores for EMA.
EMA achieves better results more consistently on [Table] different datasets when compared to MA.
The main reason why MA gets worse results is that it averages over longer iterates with equal weights (Refer to Figure 6 and other ﬁgures in Appendix).
We have seen improvements when the starting point of MA is pushed later in training, however for convenience and ease of experimentation our main results are based on the EMA method.
5. CIFAR-
Figure 2 shows images generated with and without EMA for CIFAR-10 after 300k iteration.
Objects exhibit fewer artifacts and look visually better with EMA.
It is interesting to note that the averaged and non-averaged versions converge to the same object, but can have different color and texture.
Figure 6 shows Inception and FID scores during training.
EMA model outperform its non-averaged counter-part consistently by a large margin, which is more or less stable throughout the training.
However MA model’s FID performance reduces gradually as it considers longer time windows for averaging.
This observation also holds in STL10 and ImageNet experiments (See Appendix).
Our intuition is that this phenomenon occurs because generator iterates does not stay in the same parameter region and averaging over parameters of different regions produces worse results.
Besides, we have seen clear disagreement between FID and IS in case of MA.
We think this happens because IS is not a proper metric to measure difference between two distributions and has many known ﬂaws such as it does not measure intra-class diversity (Barratt & Sharma, 2018;
Zhou et al., 2018; Heusel et al., 2017a).
In Figure 7, we compare EMA, CO and OMD.
EMA clearly improves baseline and OMD in both scores, while its improvement on CO is smaller but still consistent.
C ELEB A
Figure 3 shows the same 10 samples from the prior fed to the non-averaged, EMA generator with various β values and MA generator at 250k iteration of the training.
The non-averaged images show strong artifacts, but as we average over longer time windows, image quality improves noticeably.
Interestingly, signiﬁcant attribute changes occurs as β value gets closer to 1 . 0 .
There is a shift in gender, appearance, hair color, background etc.
although images are still similar in pose and composition.
Also EMA results with β = 0 .
999 and β = 0 .
9999 look better than MA results, which are averaged over the last 200k iterates.
This shows there is an optimal β /window size to tune for best results.
Figure 4 compares the stability of a non-averaged generator and EMA by using the same 2 samples from the prior.
Starting from 50k iteration images are generated from both models with 10k interval until 200k iterations.
Images from the non-averaged generator change attributes frequently, while only preserving generic image composition.
Meanwhile, the averaged generator produces smoother changes with attributes changing more slowly.
Figure 9 and 10 show images generated with and w/o EMA for STL-10 and ImageNet, respectively.
Even though quantitative scores are better for EMA, we don’t see as clear visual improvements as in previous results but rather small changes.
Both models produce images that are unrecognizable to a large degree.
This observation strengthens our intuition that averaging brings the cycling generator closer to the local optimal point, but it does not necessarily ﬁnd a good solution as the local optimum may not be good enough.
C ONCLUSION
We have explored the effect of two different techniques for averaging parameters outside of the GAN training loop, moving average (MA) and exponential moving average (EMA).
We have shown that both techniques signiﬁcantly improve the quality of generated images on various datasets, network architectures and GAN objectives.
In the case of the EMA technique we have provided the ﬁrst to our knowledge theoretical analysis of its implications, showing that even in simple bilinear settings it converges to stable limit cycles of small amplitude around the solution of the saddle problem.
Averaging methods are easy to implement and have minimal computation overhead thus these techniques are readily applicable in a wide range of settings.
In the future, we would like to explore A A DDITIONAL E XPERIMENTS A. P ARAMETERIZED O NLINE A VERAGING As an additional method we have used parameterized online averaging: [Equation]
where α regulates the importance of the terms. When α = 1 , the equation corresponds to moving average (Eq. 1). As t − αt might be negative, this algorithm should be restricted to the case of t ≥ α . We have started to apply it when t ≥ α .
We have experimented (Table 3) with this method on
CIFAR-10 with the same setting of the ﬁrst row of Table 2.
α = 1 has failed as it started averaging from the ﬁrst iteration which aligns with out earlier observations.
α = 10 , 100 , 1000 improves the scores at varying degree, however none is as good as EMA.
[Table]
A. Q UANTITATIVE EFFECT OF DIFFERENT β FOR EMA
In the main body of this paper, we have used β = 0 .
9999 in all experiments but CelebA as it performs the best.
In this section we would like to show the effect of β to quantitative scores.
We have experimented different β values ranging from 0 . 9 to 0 . 9999 on CIFAR-10 with the same setting of the ﬁrst row of Table 2. We have run the experiment for four times and provide the mean and std in Table 4.
Each experiments runs for 500k.
IS, FID score are collected with 20k iteration interval and best scores are published.
From Table 4, we can see clear and progressive improvement when β increases from 0 .
9 to 0 .
9999 in both IS and FID.
[Table]
B F URTHER RESULTS
[Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
C N ETWORK A RCHITECTURES
When spectral normalization is used in ResNet
[Table]
the feature number in each layer doubled by 2.
Prior distribution for the generator is a 512-dimensional isotropic Gaussian distribution for conventional architecture and 128 for ResNet.
Samples from the distribution are normalized to make them lie on a unit hypersphere before passing them into the generator.
[Table]
[Table]
D H YPERPARAMETERS AND O THER S ETTINGS D. M IXTURE OF G AUSSIAN
We have done 4 experiments for comparison.
Our baseline, Optimistic Adam , Consensus Optimization and Zero-GP settings are listed in Table 7, Table 8, Table 9 and Table 10 respectively.
Generator has 4 layers with 256 units in each layer and an additional layer that projects into the data space.
The discriminator also has 4 layers with 256 units in each layer and a classiﬁer layer on top.
ReLU activation function is used after each afﬁne transformation.
[Table]
[Table]
D.
CIFAR-10, STL-10, C ELEB A, I MAGE N ET
[Table] [Table]
[Table] [Table]