A DAM IS NO BETTER THAN NORMALIZED SGD: D ISSECTING HOW ADAPTIVE METHODS IMPROVE GAN S PERFORMANCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Adaptive methods are widely used for training generative adversarial networks (GAN).
While there has been some work to pinpoint the marginal value of adaptive methods in minimization problems, it remains unclear why it is still the method of choice for GAN training.
This paper formally studies how adaptive methods help performance in GANs.
First, we dissect Adam—the most popular adaptive method for GAN training—by comparing with SGDA the direction and the norm of its update vector.
We empirically show that SGDA with the same vector norm as Adam reaches similar or even better performance than the latter.
This empirical study encourages us to consider normalized stochastic gradient descent ascent (nSGDA) as a simpler alternative to Adam.
We then propose a synthetic theoretical framework to understand why nSGDA yields better performance than SGDA for GANs.
In that situation, we prove that a GAN trained with nSGDA provably recovers all the modes of the true distribution.
In contrast, the same networks trained with SGDA (and any learning rate conﬁguration) suffers from mode collapsing.
The critical insight in our analysis is that normalizing the gradients forces the discriminator and generator to update at the same pace .
We empirically show the competitive performance of nSGDA on real-world datasets.
I NTRODUCTION
It is commonly accepted that adaptive algorithms are required to train modern neural network architectures in various deep learning tasks.
This includes minimization problems that arise in natural language processing (Vaswani et al., 2017) and fMRI (Zbontar et al., 2018) or min-max problems such as generative adversarial network (GAN) training (Goodfellow et al., 2014).
Indeed, it has been empirically observed that Adam (Kingma & Ba, 2014) yields a solution with better generalization than stochastic gradient descent (SGD) in these problems (Choi et al., 2019).
Several works have attempted to explain this phenomenon in the minimization case.
Common explanations are that adaptive methods train faster (Zhou et al., 2018), escape faster very ﬂat saddle-point like plateaus (Orvieto et al., 2021) or deal better with heavy-tailed stochastic gradients (Zhang et al., 2019b).
However, much less is known regarding min-max problems such as GANs.
In this paper, we investigate why GANs trained with adaptive methods outperform those trained using stochastic gradient descent ascent with momentum (SGDA).
Some prior works attribute this outperformance to the superior convergence speed of adaptive methods.
For instance, Liu et al.
(2019) show that a variant of Optimistic Gradient Descent (Daskalakis et al., 2017) converges faster than SGDA for a class of non-convex non-concave min-max problems.
However, contrary to the minimization setting, convergence to a stationary point is not guaranteed and not even a requirement to ensure a satisfactory GAN performance.
Indeed, Mescheder et al.
(2018) empirically shows that popular architectures such as Wasserstein GANs (WGANs)
(Arjovsky et al., 2017) do not always converge, and yet produce realistic images.
We support this observation through the following experiment.
We train a DCGAN (Radford et al., 2015) using Adam –the most popular adaptive method– and set up the generator ( G ) and discriminator ( D ) step-sizes respectively as η D , η G .
Note that D is usually trained faster than G i.e. η D ≥ η G .
Figure 1(a) displays the
GAN convergence measured by the ratio of gradient norms, and the GAN’s performance measured in FID score (Heusel et al., 2017).
We observe that when η D /η G is close to 1 , the algorithm does not converge and yet, the model produces high-quality solutions.
On the other hand, when η D /η G (cid:29) 1 , the model converges to an equilibrium –a similar statement has been proved by Jin et al.
(2020) and
Fiez & Ratliff (2020) in the case of SGDA.
However, the GAN produces low-quality solutions at
[Figure]
this equilibrium.
Thus, simply comparing the convergence speed of adaptive methods and SGDA cannot explain the GAN’s performance obtained with adaptive methods.
This observation motivates the central question in this paper: What factors explain that Adam produces better quality solutions than SGDA when training GANs?
To address this question we dissect Adam following the approach by Agarwal et al.
They frame a generic optimizer’s update as W ( t +1) =
W ( t ) − ηa ( t ) G ( t ) , where W ( t ) ∈
R
d is the iterate,
G ( t ) ∈
R
d such that (cid:107)G ( t ) (cid:107) 2 = 1 is the optimizer’s direction and a ( t ) ≥ 0 is the optimizer’s magnitude .
Therefore, a ﬁrst step in our paper is to understand whether Adam outperforms SGDA maintly due to its direction or to its magnitude.
As detailed in Section 2, we train a GAN using i) AdaLR, an algorithm that updates in the direction of SGDA but with the magnitude of Adam ii) AdaDir which uses the direction of Adam but the magnitude of SGDA.
We empirically show that not only, AdaLR signiﬁcantly outperforms AdaDir, SGDA, and Adam itself.
This observation encourages us to conclude that:
Adam produces higher quality solutions relative to SGDA in GANs mainly due to its adaptive magnitude and not to its adaptive direction.
In Section 2, we empirically analyze the adaptive magnitude of AdaLR and observe that it stays approximately constant throughout training.
This observation eventually encourages the study of
AdaLR with a constant step-size.
Such algorithm actually corresponds to normalized SGDA (nSGDA).Compared to SGDA, nSGDA has the same direction but differs in magnitude since we divide the gradient by its norm.
Intuitively, this normalization forces D and G to be updated by vectors with constant magnitudes no matter how different the norms of D ’s and G ’s gradients are.
Motivated by the aforementioned observations, this paper studies the performance of GANs trained with nSGDA.
We believe that this is a ﬁrst step to formally understand the role of adaptive methods in GANs.
Our contributions are divided as follows:
– In Section 3, we experimentally conﬁrm that nSDGA consistently competes with Adam and outperforms SGDA when using different GAN architectures on a wide range of datasets.
– In Section 4, we provide a theoretical explanation on why GANs trained with nSGDA outperform those trained with SGDA.
More precisely, we devise a data generation problem where the target distribution D is made of multiple modes.
The model trained with nSGDA provably recovers all the modes in the target distribution while the SGDA’s one fails to do it under any step-size conﬁguration : We prove that even when SGDA converges to a locally optimal min-max equilibrium, the model still suffers from mode collapsing and fails
to learn recover the modes separately.
The key insight of our theoretical analysis is that no matter how the step-sizes are prescribed, D
and
G necessarily update at very different speeds when we use SGDA.
Therefore, i) either D updates its weights too fast, thus learns a weighted average of the modes of D .
This makes G learn this weighted average of modes ii) or D does not update its weights fast enough and thus, G aligns its weights with those of D .
This forces D to converge to a locally optimal min-max equilibrium that classiﬁes any instance as ”fake”.
On the other hand, by normalizing the gradients as done in SGDA, we force D and G to update at the same speed throughout training .
Thus, whenever D learns a mode of the distribution, G learns it right after, which makes both of them learn all the modes of the distribution separately.
Our paper advocates for the use of balanced updates in GAN training i.e. the ratio of D vs G updates should remain close to constant.
To our knowledge, we are the ﬁrst to theoretically show the importance of these balanced updates.
This insight contrasts with the related work that analyzes
GANs, and more generally zero-sum differentiable games, in the regime where D is updated much faster than G i.e. η D /η G (cid:29) 1 (Fiez & Ratliff, 2020; Jin et al., 2020; Fiez et al., 2020).
R ELATED WORK
Adaptive methods in games optimization.
Several works designed adaptive algorithms and analyzed their convergence to show their beneﬁts relative to SGDA.
For variational inequality problems,
Gasnikov et al.
Antonakopoulos et al.
(2019); Bach & Levy (2019);
Antonakopoulos et al.
(2020) propose adaptive algorithms that reach optimal convergence rates under regularity assumptions.
For a class of non-convex non-concave min-max problems, Liu et al.
; Barazandeh et al.
(2021) design algorithms that converge faster than SGDA.
On the other hand, Heusel et al.
(2017) show that Adam locally converge to a Nash equilibrium in the regime where the step-size of the discriminator is much larger than the one of the generator.
Our work differs from these papers as we analyze Adam and do not focus on the convergence properties but rather on the ﬁt of the trained model on the true (and not empirical) data distribution.
Besides, contrary to some of the aforementioned papers, our work is not in the two-time scale learning-rates regime which do not correspond to what is mostly done in practice.
Importance of balanced updates in GANs.
The importance of balanced updates for GANs has been noticed in the literature.
The most popular GAN architectures (Radford et al., 2015; Arjovsky et al., 2017; Brock et al., 2018) set the step-sizes such that η D /η G is constant.
On the other hand,
Berthelot et al.
(2017) introduced a control variable to ensure that the updates between G and D are at the same speed.
In this work, we do not modify the GAN objective loss to enforce balancedness.
Instead, we empirically and theoretically investigates how Adam and nSGDA enforce balanced updates.
Statistical results in GANs.
Early works investigate whether GANs memorize the training data or actually learn the distribution (Arora et al., 2017; 2018; Dumoulin et al., 2016).
Zhang et al.
Bai et al.
(2018) then show that for speciﬁc GANs, the model learn some distributions with nonexponential sample complexity (Liang, 2017; Feizi et al., 2017).
Recently, Li & Dou (2020); AllenZhu & Li (2021) further characterized the distributions learned by the generator.
On the other hand, some works attempted to explain GAN performance through the optimization lens.
Lei et al.
Balaji et al.
(2021) show that GAN models trained with SGDA converge to a global saddle point when the generator is one-layer neural network and the discriminator is a speciﬁc quadratic/linear function.
Our contribution signiﬁcantly differs from these two works as i) we construct a setting where SGDA converges to a locally optimal min-max equilibrium and yet suffer from mode collapse.
Conversely, nSGDA does not necessarily converge and yet, recovers the true distribution ii) our setting is more challenging since we need at least a degree-3 discriminator to learn the distribution – see Section 4 for a justiﬁcation.
Normalized gradient descent.
Introduced by Nesterov (1984), normalized gradient descent has been widely used in the minimization setting.
Indeed, it has been observed that normalizing out the gradient improves the ’slow crawling’ problem of gradient descent and avoids the iterates to be stuck in ﬂat regions – such as spurious local minima or saddle points – (Hazan et al., 2015; Levy, 2016;
Murray et al., 2019).
Normalized gradient descent or its variants outperform the non-normalized counterparts in multi-agent coordination (Cort´es, 2006) and deep learning tasks (You et al., 2017; 2019; Cutkosky & Mehta, 2020; Liu et al., 2021).
Our work rather considers the min-max setting and shows that nSGDA performs better than SGDA as it forces the discriminator and generator to update at the same rate.
F ROM A DAM TO N SGDA
Generative adversarial networks.
Given a training set sampled from some target distribution D , a GAN learns to generate new data from this distribution.
The architecture is constituted of two do
Receive
=
Compute gradient oracles
Update networks desired samples
D .
More formally, the generator is a mapping G V : R k
→ R d where V is some parameter set.
Generally, the latent variables are sampled from the normal distribution.
On the other hand, the discriminator is a mapping D W : R d → R where W is some parameter set.
In this section, one can think of these parameter sets as made of matrices and vectors.
To train the model, we consider the WGAN-GP problem formulation (Gulrajani et al., 2017) (
where (cid:98) D =
(cid:15) D
+ (1 − (cid:15) )
D z for (cid:15)
[Equation]
Adaptive methods.
In this paper, we particularly focus on second-moment-based adaptive optimizers to solve (1).
Figure 1 captures the usual formulations of Adam (Kingma & Ba, 2014), Adagrad (Duchi et al., 2011) and
RMSprop (Tieleman et al., 2012) up to some scaling conventions that can be absorbed into { ( η ( k ) G , η ( k ) D ) } .
Note that all operations on vectors in Figure 1 are entry-wise and g 2 denotes the entry-wise power on vector g .
The exponential window parameters β 1 , β 2 ∈ [0 , 1) respectively denote the ﬁrst- and second-order momentum parameters.
A rule of thumb is to set them as β 1 = 0 .
5 and β 2 = 0 .
As we aim to understand the role of adaptivity, an important baseline is stochastic gradient descent ascent with momentum (SGDA) which consists in the update of Figure 1when the denominator (cid:112) M Y , 2 + (cid:15) is omitted.
In what follows, we refer to SGDA as the SGDA with momentum update .
Optimizers as autorefalg:adaptive are adaptive in that they keep updating step-sizes while training the model.
Therefore, two types of schedules are involved: the explicit step size schedules { ( η ( t ) G , η ( t ) D ) } that the practitioner manually sets up and the implicit step size schedules induced by the optimizers.
The contributions of these two schedules overlap and it remains unclear how they contribute to the superior performance of adaptive methods relative to SGDA in GANs.
Method dissection using grafting.
To decouple the explicit and implicit schedules, we adopt the step size grafting approach proposed by Agarwal et al.
and described as follows.
At each iteration, we compute stochastic gradients, pass them to two optimizers
A 1 , A 2 and make a grafted step which combines the magnitude of A 1 ’s step and direction of A 2 ’s step.
Since our goal is to understand the beneﬁts of adaptivity, it is natural to consider the grafting approach with Adam and
SGDA.
We deﬁne AdaLR , an optimizer that updates in the SGDA direction with Adam magnitude:
[Equation]
and AdaDir which updates in the Adam direction with SGDA magnitude
[Equation]
Note that two implementations are possible for AdaLR and AdaDir.
In the layer-wise version, Y ( t ) is a single parameter group (typically a layer in a neural network) and therefore, the update is applied to each group.
In the global version, Y ( t ) contains all of the model’s weights.
Note that in Figure 2, we set up AdaLR and AdaDir with the layer-wise version –the global AdaLR and AdaDir perform approximately as well as their layer-wise counterparts.
[Figure]
We train a WGAN-GP (Arjovsky et al., 2017) on CIFAR-10 with AdaLR, AdaDir and Adam.
Figure 2(a) shows the GAN performance measured in FID score (Heusel et al., 2017) obtained by the three trained models.
We observe that AdaDir does not generate samples from the desired distribution as its FID score is high.
On the other hand, AdaLR performs slightly better than Adam.
Therefore, we deduce that the success of adaptive methods in GANs may be explained by the implicit step-size schedule induced by the algorithm.
From AdaLR to normalized SGDA.
The previous experiment hints that AdaLR which combines
Adam’s magnitude and SGDA direction performs better than Adam.
We here take a closer look at this algorithm.
In particular, we investigate the ﬂuctuations of Adam’s magnitude during training.
Figures 2(b),2(c) show that this magnitude barely varies for both generator and discriminator.
Therefore, an update with SGDA direction and constant step-size seems to be a valid approximation of Adam in GAN training.
Such an algorithm corresponds to normalized SGDA (nSGDA),
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
Similarly to AdaLR and AdaDir we also deﬁne two versions of nSGDA: layer-wise nSGDA (lnSGDA) and global nSGDA (g-nSGDA).
We use both of these in the numerical experiments.
As a comparison to (4) the SGDA (with momentum) update is:
[Equation]
N UMERICAL PERFORMANCE OF N SGDA
Section 2 indicates that nSGDA may work as well as
Adam in GAN training.
In this section, we empirically verify this hypothesis through an extensive numerical study.
To evaluate the proposed algorithm, we conducted extensive experiments on CIFAR-
(Krizhevsky et al., 2009), LSUN Churches (Yu et al., 2016), STL-10 (Coates et al., 2011) and Celeba-HQ (Liu et al., 2015).
Similarly to above, we choose the Fr´echet Inception distance (FID) (Heusel et al., 2017) to quantitatively assess the performance of the model.
In all our experiments, 50k samples are randomly generated for each model to compute the
FID.
As for the architectures, we choose Resnets (He et al., 2016) from Gidel et al.
and set up the
WGAN-GP loss (Gulrajani et al., 2017).
Note that for each optimizer, we grid-search over stepsizes to ﬁnd the best one in terms of FID.
Due to limited computational resources, we trained the models for 100 epochs in the case of CIFAR-10 and Celeba-HQ and for 50 epochs for LSUN and STL-10.
We apply a linear decay learning rate scheduling during training.
All the results are averaged over 10 seeds.
nSGDA competes with Adam.
Figure 4 shows the performance of l-nSGDA, g-nSGDA, Adam, and SGD on different datasets.
We ﬁrst observe that l-nSGDA and g-nSGDA compete with Adam
[Figure]
in all these datasets.
On the other hand, SGD struggles to produce good quality images compared to the other methods as expected.
It is worth reminding that the nSGDA methods and SGD have the same direction but differ in their magnitude .
Therefore, this experiment conﬁrms our hypothesis that adaptive methods outperform SGDA thanks to their implicit step-size schedule.
Lastly, we highlight that l-nSGDA and g-nSGDA perform almost as well.
This suggests that a global step-size adaptivity is enough to perform well in GANs.
Similar observations hold for DCGAN (see Appendix A).
Inﬂuence of batch size.
While the nSGDA methods compete with Adam in GAN, it remains unclear whether such performance is proper to the algorithm or tied to the stochastic noise of the gradients.
Figure 3 shows the performance of the model trained with l-nSGDA, g-nSGDA, Adam, and SGD and using a wide range of batch sizes.
We remark that the nSGDA methods perform better than Adam when the batch size is small.
However, as the batch size increases, their performance seriously deteriorates.
On the other hand, the performance of Adam seems to be less sensitive to the change of batch size.
As expected, SGD overall performs poorly.
nSGDA performs well in a relevant batchsize regime, as many GAN architectures such as DCGAN, WGAN-GP, SNGAN (Miyato et al., 2018), SAGAN (Zhang et al., 2019a) requires small batch sizes for optimal performances.
W HY DOES N SGDA PERFORM BETTER THAN SGDA IN GAN S ?
In Section 2, we numerically observed that Adam outperforms SGDA mainly thanks to its adaptive magnitude (implicit step-size schedule).
In Section 3, we observed that nSGDA methods – which differs from SGDA only by adaptive magnitude – compete with Adam and signiﬁcantly surpass
SGDA.
To the best of our knowledge, there is no theoretical result that demonstrates the importance of adaptive magnitude in GANs performance.
Thus, we set the following question:
From a theoretical perspective, for what distribution learning problem using GANs does nSGDA perform better comparing to SGDA
We devise a simple data generation problem where the target distribution consists of two modes, described as vectors u 1 , u 2 , that are slightly correlated (See Assumption 1).
We show that using standard GANs’ training objective, with high probability, SGDA – with any reasonable 1 step-size conﬁguration – only learn distributions that suffer from mode collapse i.e. u 1 , u 2 always show up in the learned distribution simultaneously.
Conversely, nSGDA learns the two modes separately .
Notations.
We set the GAN 1-sample loss L ( t ) V , W ( X, z )
=
log( D ( t ) W ( X ))
+ log(1 − D ( t ) W (
G ( t ) V ( z ))) .
g ( t ) Y = ∇ Y L ( t ) V , W ( X, z ) is the 1-sample stochastic gradient (without momentum).
4. S ETTINGS
In this section, we consider the classical GAN problem formulation.
[Equation]
In the following paragraphs, we describe each element of our formulation.
Data distribution D .
The data generation problem consists in having a training dataset with points sampled from a target distribution D .
Using this dataset, our goal is to train a model that generates samples with the same statistics as D .
The latter is deﬁned as follows.
Let u 1 and u 2 two vectors in R d
such that (cid:107) u
i (cid:107) 2 = 1 , for i ∈ { 1 , 2 } .
Each sample from D consists of an input data X generated as:
Sample s = ( s 1 , s 2 ) from the distribution S deﬁned as s i ∈ [0 , 1] , P [ s i = 1] ≥ 1 / 2 for i ∈ { 1 , 2 } , such that (cid:107) s (cid:107) 0 ≥ 1 .
Note that s 1 and s 2 are not necessarily independent.
Deﬁne data-point X =
s 1 u 1 + s 2 u 2 ∈ R d .
The distribution D generates points that are a linear combination of two modes u 1 and u 2 .
In what follows, we make the following assumptions on the coefﬁcients ( s i ) and modes ( u i ) .
Assumption 1 (Data structure) .
Let γ = 1 polylog( d ) .
The coefﬁcients s 1 , s 2 and modes u 1 , u 2 of the distribution D respect one of the following conditions:
Correlated Modes: (cid:104) u 1 , u 2 (cid:105) =
γ and the generated data point is either X =
u 1 or X =
u 2 . 2.
Correlated Coefﬁcients: P [ s
= s 2 = 1]
= γ and the modes are orthogonal, i.e., (cid:104) u 1 , u 2 (cid:105) = 0 .
Assumption 1 captures some of the realistic structure of images.
Case 1 models the setting where we have two pure modes (e.g., two different types of cats) that are correlated.
Case 2 corresponds to two (roughly) orthogonal modes (e.g., vertical and horizontal edges), that may sometimes be mixed together (images containing object corners in the example above).
Learner models.
To learn the true distribution D , we use a linear generator G V deﬁned as
[Equation]
where V =
[ v (cid:62) 1 , v (cid:62) 2 , · · · , v (cid:62) m G ] ∈ R m G ×
d is the weight matrix and z ∈ { 0 , 1 } m G .
We set V = { V } .
Intuitively, G V outputs linear combinations of modes ( v i ) .
We assume that G V samples from the latent distribution D z deﬁned for z ∈ { 0 , 1 } m G , (cid:107) z (cid:107) 0 ≥ 1 as:
[Equation]
First, remark that vectors sampled from D z are binary-valued.
Although usual latent distributions in GANs are either Gaussian or uniform, the distribution D z should be rather seen as modelling the weights’ distribution of a hidden layer of a deep generator.
Indeed, Allen-Zhu & Li (2021) argue that the distributions of these hidden layers are sparse, non-negative, and non-positively correlated.
Besides, in (8), Pr[ z i = z
j = 1] = 1 m 2 G polylog( d ) ensures that that the output of the generator is only made of one mode with probability 1 − o (1) and thus avoid creating weighted averages of the two or more v i ’s (which might cause mode collapsing).
To assess the distribution learned by G V , we also train a non-linear neural network as discriminator D W
D W ( X
[Equation]
= sigmoid
[Equation]
otherwise
[Equation] [Equation]
where W = [ w (cid:62) 1 , . . .
, w (cid:62) m D ] ∈ R m D ×
d and a, b ∈ R are the trainable parameters of D W , λ > 0 is a ﬁxed scaling factor (speciﬁed below) and Λ = d 0 . 2 .
For simplicity, we set W =
{ W, a, b } .
Besides, σ ( · ) is the truncated degree-3 activation function—it is thus made Lipschitz, which is only needed in the proof to deal with the case where the generator is trained much faster than the discriminator.
Note that this latter case is uncommon in practice.
Lastly, we choose a cubic activation as it is the smallest polynomial degree for the discriminator that is sufﬁcient: as pointed out in Li & Dou (2020), with linear or quadratic activations, the generator can fool the discriminator by only matching the ﬁrst and second moments of D .
By doing so, the generator cannot recover the modes u 1 , u 2 but only weighted averages of them even at the global optimal solution.
Algorithms.
We solve the training problem (6) using SGDA and nSGDA with momentum = 0 .
For simplicity, we also set the batch-size to 1.
At each step t, we sample X ∼ D and z ∼ D z and deﬁne
SGDA’s update as
[Equation]
[Figure]
V 1 V 2 V
[Figure]
V 4 V
[Figure]
1102030404142434445 It e r a t i on s u 1 u 2 u 1 u 2 u 1 u 2 u 1 u 2 u 1 u0.
(d) where η D , η G > 0 are constant step-sizes.
On the other hand, nSGDA is deﬁned by the update rule
[Equation]
Compared to (4), (nSGDA) is a global nSGDA update (without momentum)
(i.e. we do not consider layer-wise normalization).
Indeed, (cid:107) g ( t ) Y (cid:107) 2 in the update refers to the sum of norms of the gradients with respect to a, b, W .
We now detail how to set parameters in (SGDA) and (nSGDA).
Parametrization 4.1.
When running SGDA and nSGDA on (1), we set the parameters as
– Initialization : b (0) = 0 , a (0) ∼ N (cid:16) 0 ,
[Figure]
[Equation]
for i ∈ [ m D ] , j ∈ [ m G ] .
– Number of iterations : we run SGDA for t ≤ T 0 iterations where T 0 is the ﬁrst iteration such that (cid:107)∇ E
[ L V ( T 0) , W ( T 0)
( X, z )] (cid:107) 2 ≤ 1 / poly( d ) . .
For nSGDA, we run for T 1 = ˜Θ (cid:16) 1 η D (cid:17) iterations.
– Step-sizes : For SGDA, η D , η G ∈ (0 , 1 poly( d ) ) .
For nSGDA, η D ∈ (0 , 1 poly( d ) ] , η G = η D polylog( d ) .
– Over-parametrization : For SGDA, m D , m G = polylog( d ) are arbitrarily chosen i.e. m D may be larger than m G or the opposite.
For nSGDA, we set m D = log( d ) and m G = log( d ) log log d .
Parametrization 4.1 corresponds to usual initialization and optimization hyper-parameters.
Regarding initialization, the discriminator’s weights are sampled from a standard normal and its bias is set to zero.
The weights of the generator are initialized from a normal with variance 1 /d 2 (instead of the 1 /d in standard normal).
Such a choice is explained as follows.
In practice, the target X ∼ D is a 1D image, thus has entries in [0 , 1] d and norm O ( √ d ) .
Yet, we sample the initial generator’s weights from N (0 , I d /d ) in this case.
In our case, since (cid:107) u i (cid:107) 2 = 1 , the target X = s 1 u 1 + s 2 uhas norm O (1) .
Therefore, we scale down the variance in the normal distribution by a factor of 1 /d to match the conﬁguration encountered in practice.
Therefore, we also set λ = 1 √ d polylog( d ) in (9) to ensure that the weights and the bias in the discriminator learn at the same speed.
Regarding the number of iterations our theorem holds when running SGDA for any (polynomially) possible number of iterations –after T 0 steps, the gradient becomes inverse polynomially small and
SGDA essentially stops updating the parameters.
Lastly, we allow any step-size conﬁguration for
SGDA i.e. larger, smaller, or equal step-size for D compared to G .
To our knowledge, this setting has never been studied in GAN performance.
Note that our choice of step-sizes for nSGDA matches with the one used in practice i.e. η D slightly larger than η G .
4. M AIN RESULTS
We state our main results on the performance of models trained using (SGDA) and (nSGDA).
We show that nSGDA is able to learn the two modes of the distribution D while SGDA is not.
Theorem 4.1 (SGDA suffers from mode collapse) .
Let T 0 , η G , η D and the initialization as deﬁned in Parametrization 4.1.
Let t be such that t ≤ T 0 .
Run SGDA for t iterations with step-sizes η G , η D .
Then, with probability at least 1 − o (1) , for all z ∈ { 0 , 1 } m G , we have:
[Equation]
, where α ( t ) ( z ) ∈ R and ξ ( t ) ( z ) ∈ R d , such that for all (cid:96) ∈ [2] , |(cid:104) ξ ( t ) ( z ) , u (cid:96) (cid:105)| =
o (1) (cid:107) ξ ( t ) ( z ) (cid:107) 2 for every z ∈ { 0 , 1 } m G .
In the speciﬁc case where η G = √ dη D polylog( d ) , the model mode collapses i.e. (cid:107) ξ
( T 0 ) ( z ) (cid:107)
= o ( α ( T 0 ) ( z )) .
Theorem 4.1 indicates that when using SGDA with any step-size conﬁguration, the generator either does not learn the modes at all – when α ( t ) ( z ) = 0 , G ( t ) V ( z )
=
ξ ( t ) ( z ) – or learns an average of the modes – when α ( t ) ( z ) (cid:54) = 0 , G ( t ) V ( z )
≈ α ( t ) ( z )( u 1 + u 2 ) .
We emphasize that the theorem holds for any time t ≤ T 0 which is the iteration where SGDA converges to an approximate ﬁrst-order locally optimal min-max equilibrium.
Conversely, nSGDA succeeds to learn the two modes separately.
Theorem 4.2 (nSGDA recovers modes separately) .
Let T 1 , η G , η D and the initialization as deﬁned in Parametrization 4.1.
Run nSGDA for T 1 iterations with step-sizes η G , η D .
Then, the generator learns both modes u 1 , u 2 i.e.,
[Equation]
Theorem 4.2 indicates that when we train a GAN with nSGDA in the regime where the discriminator updates slightly faster than the generator (as done in practice), the generator successfully learns the distribution containing the direction of both modes.
4. W HY DOES SGDA SUFFER FROM MODE COLLAPSE ?
We now sketch the reason why SGDA suffers from mode collapse using Figure 5.
Figure 5(a) displays the relative update speed
[Equation]
2 , Figure 5(b) the correlation (cid:104)
[Equation]
i
[Equation]
2 between D ’s neuron and mode u (cid:96) and Figure 5(c) the correlation (cid:104) v [Equation]
We focus on the case where the discriminator’s step-size is set so it updates slightly faster than the generator at the beginning – as displayed in iterations 1–5 in Figure 5(a).
The key observation is that in the early stages, the discriminator’s update is approximately
[Equation]
With high probability, there exists at least a neuron i such that (cid:104) w ( t ) i , X (cid:105) > 0 .
Thus, (11) implies that (cid:104) w ( t ) i , X (cid:105) is an increasing sequence.
As t increases, w ( t ) i gradually grows its correlation with one of the modes u (cid:96) (iterations 1-20 in Figure 5(b)) and D ’s gradient norm thus increases.
Therefore, after a ﬁrst phase where D updates slightly faster than G , D ’s update speed becomes signiﬁcantly larger than G ’s one (iterations 5–20 in Figure 5(a)).
Thus, D learns the ﬁrst mode after 20 iterations in Figure 5(b).
However, one of the goal of D (from the training objective in (6)) is to maximize its average correlation with the target distribution D .
Since G does not catch up and D is made of two modes showing up with equal probability, the optimal solution for D is to have each of its w i =
α i ( u 1 + u 2 ) to maximize the average correlation.
Therefore, after iteration 20, D learns the second mode and eventually gets this optimal solution (which is a mode collapse) at iteration 40.
Then,
D ’s update speed starts dropping as we see in Figure 5(a) which helps G to catch up and grow its update speed.
However, since D already learnt a weighted average of the modes, it can only teach G to learn this average and thus mode collapses as we see in Figure 5(c).
On the other hand, nSGDA ensures that G and D always learn at the same speed , so that G can learn one mode immediately when D learns (such as at iteration 25 in (b) in Figure (5)), which avoids mode collapse.
C ONCLUSION
Our work offers a complementary view to several works in the min-max optimization literature where the discriminator is much faster than the generator to converge to an equilibrium.
Here, instead, we advocate the use of balanced updates to ensure that the GAN performs well.
Our work is a ﬁrst step towards understanding how adaptive methods improve the GAN performance.
Our empirical observations and theorems heavily rely on the fact that the batch size is small.
However, nSGDA methods seem to not work well for large batch sizes and may not be suitable to some large-scale GANs such as BigGAN (Brock et al., 2018).
It would be interesting to understand why adaptive methods are crucial in this case.
Another interesting direction would be to improve the training of nSGDA in the batch setting.