Original: D ISSECTING ADAPTIVE METHODS IN GAN S
Revised: D ISSECTING ADAPTIVE METHODS IN GAN S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Adaptive methods are a crucial component widely used for training generative adversarial networks (GANs).
Revised: Adaptive methods are a crucial component widely used for training generative adversarial networks (GANs).

Original: While there has been some work to pinpoint the “marginal value of adaptive methods” in standard tasks, it remains unclear why they are still critical for GAN training.
Revised: While there has been some work to pinpoint the “marginal value of adaptive methods” in standard tasks, it remains unclear why they are still critical for GAN training.

Original: In this paper, we formally study how adaptive methods help train GANs; inspired by the grafting method proposed in
Revised: In this paper, we formally study how adaptive methods help train GANs; inspired by the grafting method proposed in

Original: Agarwal et al.
Revised: Agarwal et al.

Original: (2020), we separate the magnitude and direction components of the
Revised: (2020), we separate the magnitude and direction components of the

Original: Adam updates, and graft them to the direction and magnitude of SGDA updates respectively.
Revised: Adam updates, and graft them to the direction and magnitude of SGDA updates respectively.

Original: By considering an update rule with the magnitude of the Adam update and the normalized direction of SGD, we empirically show that the adaptive magnitude of Adam is key for GAN training.
Revised: By considering an update rule with the magnitude of the Adam update and the normalized direction of SGD, we empirically show that the adaptive magnitude of Adam is key for GAN training.

Original: This motivates us to have a closer look at the class of normalized stochastic gradient descent ascent (nSGDA) methods in the context of GAN training.
Revised: This motivates us to have a closer look at the class of normalized stochastic gradient descent ascent (nSGDA) methods in the context of GAN training.

Original: We propose a synthetic theoretical framework to compare the performance of nSGDA and SGDA for GAN training with neural networks.
Revised: We propose a synthetic theoretical framework to compare the performance of nSGDA and SGDA for GAN training with neural networks.

Original: We prove that in that setting, GANs trained with nSGDA recover all the modes of the true distribution, whereas the same networks trained with SGDA (and any learning rate configuration) suffer from mode collapse.
Revised: We prove that in that setting, GANs trained with nSGDA recover all the modes of the true distribution, whereas the same networks trained with SGDA (and any learning rate configuration) suffer from mode collapse.

Original: The critical insight in our analysis is that normalizing the gradients forces the discriminator and generator to be updated at the same pace.
Revised: The critical insight in our analysis is that normalizing the gradients forces the discriminator and generator to be updated at the same pace.

Original: We also experimentally show that for several datasets, Adam’s performance can be recovered with nSGDA methods.
Revised: We also experimentally show that for several datasets, Adam’s performance can be recovered with nSGDA methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Adaptive algorithms have become a key component in training modern neural network architectures in various deep learning tasks.
Revised: Adaptive algorithms have become a key component in training modern neural network architectures in various deep learning tasks.

Original: Minimization problems that arise in natural language processing (Vaswani et al., 2017), fMRI (Zbontar et al., 2018), or min-max problems such as generative adversarial networks (GANs) (Goodfellow et al., 2014) almost exclusively use adaptive methods, and it has been empirically observed that Adam (Kingma & Ba, 2014) yields a solution with better generalization than stochastic gradient descent (SGD) in such problems (Choi et al., 2019).
Revised: Minimization problems that arise in natural language processing (Vaswani et al., 2017), fMRI (Zbontar et al., 2018), or min-max problems such as generative adversarial networks (GANs) (Goodfellow et al., 2014) almost exclusively use adaptive methods, and it has been empirically observed that Adam (Kingma & Ba, 2014) yields a solution with better generalization than stochastic gradient descent (SGD) in such problems (Choi et al., 2019).

Original: Several works have attempted to explain this phenomenon in the minimization setting.
Revised: Several works have attempted to explain this phenomenon in the minimization setting.

Original: Common explanations are that adaptive methods train faster (Zhou et al., 2018), escape flat “saddle-point”–like plateaus faster (Orvieto et al., 2021), or handle heavy-tailed stochastic gradients better (Zhang et al., 2019;
Revised: Common explanations are that adaptive methods train faster (Zhou et al., 2018), escape flat “saddle-point”–like plateaus faster (Orvieto et al., 2021), or handle heavy-tailed stochastic gradients better (Zhang et al., 2019;

Original: Gorbunov et al., 2022).
Revised: Gorbunov et al., 2022).

Original: However, much less is known about why adaptive methods are so critical for solving min-max problems such as GANs.
Revised: However, much less is known about why adaptive methods are so critical for solving min-max problems such as GANs.

Original: Several previous works attribute this performance to the superior convergence speed of adaptive methods.
Revised: Several previous works attribute this performance to the superior convergence speed of adaptive methods.

Original: For instance, Liu et al.
Revised: For instance, Liu et al.

Original: (2019) show that an adaptive variant of Optimistic Gradient Descent (Daskalakis et al., 2017) converges faster than SGDA for a class of non-convex, non-concave min-max problems.
Revised: (2019) show that an adaptive variant of Optimistic Gradient Descent (Daskalakis et al., 2017) converges faster than stochastic gradient descent ascent (SGDA) for a class of non-convex, non-concave min-max problems.

Original: However, contrary to the minimization setting, convergence to a stationary point is not guaranteed, nor is it even a requirement to ensure a satisfactory GAN performance.
Revised: However, contrary to the minimization setting, convergence to a stationary point is not required to obtain satisfactory GAN performance.

Original: Mescheder et al.
Revised: Mescheder et al.

Original: (2018) empirically shows that popular architectures such as Wasserstein GANs (WGANs)
Revised: (2018) empirically shows that popular architectures such as Wasserstein GANs (WGANs)

Original: (Arjovsky et al., 2017) do not always converge, and yet they produce realistic images.
Revised: (Arjovsky et al., 2017) do not always converge, and yet they produce realistic images.

Original: We support this observation with our own experiments in Section 2 (see Fig. 1b.)
Revised: We support this observation with our own experiments in Section 2.

Original: This finding motivates the central question in this paper: what factors of Adam contribute to better quality solutions than SGDA when training GANs?
Revised: Our findings motivate the central question in this paper: what factors of Adam contribute to better quality solutions over SGDA when training GANs?

Original: In this paper, we investigate why GANs trained with adaptive methods outperform those trained using stochastic gradient descent ascent (SGDA).
Revised: In this paper, we investigate why GANs trained with adaptive methods outperform those trained using SGDA.

Original: Directly analyzing Adam is challenging due to the highly non-linear nature of its gradient oracle and its path-dependent update rule.
Revised: Directly analyzing Adam is challenging due to the highly non-linear nature of its gradient oracle and its path-dependent update rule.

Original: Inspired by the grafting approach in (Agarwal et al., 2020), we disentangle the adaptive magnitude and direction of Adam and show evidence that an algorithm made of the adaptive magnitude of Adam and the direction of SGDA (Ada-nSGDA) recovers the performance of Adam in GANs.The adaptive magnitude in Adam is thus key for the performance, and the standard nSGDA direction .
Revised: Inspired by the grafting approach in (Agarwal et al., 2020), we disentangle the adaptive magnitude and direction of Adam and show evidence that an algorithm using the adaptive magnitude of Adam and the direction of SGDA (which we call Ada-nSGDA) recovers the performance of Adam in GANs.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • In Section 2, we present the Ada-nSGDA algorithm and the standard normalized SGDA (nSGDA). We further show that for some architectures and datasets, nSGDA can be used to model the dynamics of the performance of Adam in GAN training.
Revised: • In Section 2, we present the Ada-nSGDA algorithm. We emprically show that the adaptive magnitude in Ada-nSGDA stays within a constant range and does not heavily fluctuate which motivates the focus on normalized SGDA (nSGDA) which only contains the direction of SGDA.

Original: • In Section 3, we prove that for a synthetic dataset consisting of two modes, a model trained with SGDA suffers from mode collapse (producing only a single type of output), while a model trained with nSGDA does not.
Revised: • In Section 3, we prove that for a synthetic dataset consisting of two modes, a model trained with SGDA suffers from mode collapse (producing only a single type of output), while a model trained with nSGDA does not.

Original: This provides an explanation for why GANs trained with nSGDA outperform those trained with SGDA.
Revised: This provides an explanation for why GANs trained with nSGDA outperform those trained with SGDA.

Original: • In Section 4, we empirically confirm that Ada-nSDGA recovers the performance of Adam when using different GAN architectures on a wide range of datasets.
Revised: • In Section 4, we empirically confirm that nSGDA mostly recovers the performance of Ada-nSGDA when using different GAN architectures on a wide range of datasets.

Original: Our key theoretical insight is that when using SGDA and any step-size configuration, either the generator G or discriminator D updates much faster than its counterpart.
Revised: Our key theoretical insight is that when using SGDA and any step-size configuration, either the generator G or discriminator D updates much faster than the other.

Original: By normalizing the gradients as done in nSGDA, D and G are forced to update at the same speed throughout training.
Revised: By normalizing the gradients as done in nSGDA, D and G are forced to update at the same speed throughout training.

Original: The consequence is that whenever D learns a mode of the distribution, G learns it right after, which makes both of them learn all the modes of the distribution separately.
Revised: The consequence is that whenever D learns a mode of the distribution, G learns it right after, which makes both of them learn all the modes of the distribution separately at the same pace.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Adaptive methods in games optimization.
Revised: Adaptive methods in games optimization.

Original: Several works designed adaptive algorithms and analyzed their convergence to show their benefits relative to SGDA e.g. in variational inequality problems,
Revised: Several works designed adaptive algorithms and analyzed their convergence to show their benefits relative to SGDA e.g. in variational inequality problems,

Original: Gasnikov et al.
Revised: Gasnikov et al.

Original: Antonakopoulos et al.
Revised: Antonakopoulos et al.

Original: (2019); Bach & Levy (2019);
Revised: (2019); Bach & Levy (2019);

Original: Antonakopoulos et al.
Revised: Antonakopoulos et al.

Original: Liu et al.
Revised: Liu et al.

Original: ; Barazandeh et al.
Revised: ; Barazandeh et al.

Original: Heusel et al.
Revised: Heusel et al.

Original: (2017) show that Adam locally converges to a Nash equilibrium in the regime where the step-size of the discriminator is much larger than the one of the generator.
Revised: (2017) show that Adam locally converges to a Nash equilibrium in the regime where the step-size of the discriminator is much larger than the one of the generator.

Original: Our work differs as we do not focus on the convergence properties of
Revised: Our work differs as we do not focus on the convergence properties of

Original: Adam, but rather on the fit of the trained model to the true (and not empirical) data distribution.
Revised: Adam, but rather on the fit of the trained model to the true (and not empirical) data distribution.

Original: Statistical results in GANs.
Revised: Statistical results in GANs.

Original: Early works studied whether GANs memorize the training data or actually learn the distribution (Arora et al., 2017; Liang, 2017; Feizi et al., 2017; Zhang et al., 2017;
Revised: Early works studied whether GANs memorize the training data or actually learn the distribution (Arora et al., 2017; Liang, 2017; Feizi et al., 2017; Zhang et al., 2017;

Original: Arora et al., 2018; Bai et al., 2018;
Revised: Arora et al., 2018; Bai et al., 2018;

Original: Dumoulin et al., 2016).
Revised: Dumoulin et al., 2016).

Original: Some works explained GAN performance through the lens of optimization.
Revised: Some works explained GAN performance through the lens of optimization.

Original: Lei et al.
Revised: Lei et al.

Original: Balaji et al.
Revised: Balaji et al.

Original: (2021) show that GANs trained with
Revised: (2021) show that GANs trained with

Original: SGDA converge to a global saddle point when the generator is one-layer neural network and the discriminator is a specific quadratic/linear function.
Revised: SGDA converge to a global saddle point when the generator is one-layer neural network and the discriminator is a specific quadratic/linear function.

Original: Our contribution differs as i) we construct a setting where SGDA converges to a locally optimal min-max equilibrium but still suffers from mode collapse, and ii) we have a more challenging setting since we need at least a degree-3 discriminator to learn the distribution, which is discussed in Section 3.
Revised: Our contribution differs as i) we construct a setting where SGDA converges to a locally optimal min-max equilibrium but still suffers from mode collapse, and ii) we have a more challenging setting since we need at least a degree-3 discriminator to learn the distribution, which is discussed in Section 3.

Original: Normalized gradient descent.
Revised: Normalized gradient descent.

Original: Introduced by Nesterov (1984), normalized gradient descent has been widely used in minimization problems.
Revised: Introduced by Nesterov (1984), normalized gradient descent has been widely used in minimization problems.

Original: Normalizing the gradient remedies the issue of iterates being stuck in flat regions such as spurious local minima or saddle points (Hazan et al., 2015; Levy, 2016).
Revised: Normalizing the gradient remedies the issue of iterates being stuck in flat regions such as spurious local minima or saddle points (Hazan et al., 2015; Levy, 2016).

Original: Normalized gradient descent methods outperforms their non-normalized counterparts in multiagent coordination (Cort ´ es, 2006) and deep learning tasks (Cutkosky & Mehta, 2020).
Revised: Normalized gradient descent methods outperforms their non-normalized counterparts in multiagent coordination (Cort ´ es, 2006) and deep learning tasks (Cutkosky & Mehta, 2020).

Original: Our work considers the min-max setting and shows that nSGDA outperforms SGDA as it forces discriminator and generator to update at same rate.
Revised: Our work considers the min-max setting and shows that nSGDA outperforms SGDA as it forces discriminator and generator to update at same rate.

Original: 1. B ACKGROUND
Revised: 1. B ACKGROUND

Original: Generative adversarial networks.
Revised: Generative adversarial networks.

Original: Given a training set sampled from some target distribution D , a
Revised: Given a training set sampled from some target distribution D , a

Original: GAN learns to generate new data from this distribution.
Revised: GAN learns to generate new data from this distribution.

Original: The architecture is comprised of two networks: a generator that maps points in the latent space D z to samples of the desired distribution, and a discriminator which evaluates these samples by comparing them to samples from D .
Revised: The architecture is comprised of two networks: a generator that maps points in the latent space D z to samples of the desired distribution, and a discriminator which evaluates these samples by comparing them to samples from D .

Original: More formally, the generator is a mapping G V : R k
Revised: More formally, the generator is a mapping G V : R k

Original: → R d and the discriminator is a mapping D W : R d → R , where V and W are their corresponding parameter sets.
Revised: → R d and the discriminator is a mapping D W : R d → R , where V and W are their corresponding parameter sets.

Original: To find the optimal parameters of these two networks, one must solve a min-max optimization problem of the form
Revised: To find the optimal parameters of these two networks, one must solve a min-max optimization problem of the form

Original: [Equation]
Revised: [Equation]

Original: where p data is the distribution of the training set, p z the latent distribution, G V the generator and
Revised: where p data is the distribution of the training set, p z the latent distribution, G V the generator and

Original: D W the discriminator.
Revised: D W the discriminator.

Original: Contrary to minimization problems where convergence to a local minimum is required for high generalization, we empirically verify that most of the well-performing GANs do not converge to a stationary point.
Revised: Contrary to minimization problems where convergence to a local minimum is required for high generalization, we empirically verify that most of the well-performing GANs do not converge to a stationary point.

Original: [Figure]
Revised: [Figure]

Original: Convergence and performance are decorrelated in GANs.
Revised: Convergence and performance are decorrelated in GANs.

Original: We support this observation through the following experiment.
Revised: We support this observation through the following experiment.

Original: We train a DCGAN (Radford et al., 2015) using Adam and set up the step-sizes for G and D as η D , η G , respectively.
Revised: We train a DCGAN (Radford et al., 2015) using Adam and set up the step-sizes for G and D as η D , η G , respectively.

Original: Note that D is usually trained faster than G i e. η D ≥ η G .
Revised: Note that D is usually trained faster than G i e. η D ≥ η G .

Original: Figure 1a displays the GAN convergence measured by the ratio of gradient norms and the
Revised: Figure 1a displays the GAN convergence measured by the ratio of gradient norms and the

Original: GAN’s performance measured in FID score (Heusel et al., 2017).
Revised: GAN’s performance measured in FID score (Heusel et al., 2017).

Original: We observe that when η D /η G is close to 1 , the algorithm does not converge, and yet, the model produces high-quality solutions.
Revised: We observe that when η D /η G is close to 1 , the algorithm does not converge, but the model produces high-quality samples.

Original: On the other hand, when η D /η G ≫ 1 , the model converges to an equilibrium; a similar statement has been proved by Jin et al.
Revised: On the other hand, when η D /η G ≫ 1 , the model converges to an equilibrium; a similar statement has been proved by Jin et al.

Original: and
Revised: and

Original: Fiez & Ratliff (2020) in the case of SGDA.
Revised: Fiez & Ratliff (2020) in the case of SGDA.

Original: However, the trained GAN produces low-quality solutions at this equilibrium, so simply comparing the convergence speed of adaptive methods and SGDA cannot explain the performance obtained with adaptive methods.
Revised: However, the trained GAN produces low-quality solutions at this equilibrium, so simply comparing the convergence speed of adaptive methods and SGDA cannot explain the performance obtained with adaptive methods.

Original: SGDA and adaptive methods.
Revised: SGDA and adaptive methods.

Original: The most simple algorithm to solve the min-max (GAN) is SGDA, which is defined as follows:
Revised: The most simple algorithm to solve the min-max (GAN) is SGDA, which is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where M ( t ) W , 1 , M ( t ) V , 1 are the first-order momentum gradients as defined in Algorithm 1.
Revised: where M ( t ) W , 1 , M ( t ) V , 1 are the first-order momentum gradients as defined in Algorithm 1.

Original: While this method has been used in the first GANs (Radford et al., 2015), most modern GANs are trained with adaptive methods such as Adam (Kingma & Ba, 2014).
Revised: While this method has been used in the first GANs (Radford et al., 2015), most modern GANs are trained with adaptive methods such as Adam (Kingma & Ba, 2014).

Original: The definition of this algorithm for game optimizations is given in Algorithm 1.
Revised: The definition of this algorithm for game optimizations is given in Algorithm 1.

Original: The hyperparameters β 1 , β 2 ∈ [0 , 1) control the weighting of the exponential moving average of the first and second-order moments.
Revised: The hyperparameters β 1 , β 2 ∈ [0 , 1) control the weighting of the exponential moving average of the first and second-order moments.

Original: In many deep-learning tasks, practitioners have found that setting β 2 = 0 .
Revised: In many deep-learning tasks, practitioners have found that setting β 2 = 0 .

Original: 9 works for most problem settings.
Revised: 9 works for most problem settings.

Original: It has been empirically observed that having no momentum (i.e., β 1 ≈ 0 ) is optimal for many popular  architectures (Karras et al., 2020; Brock et al., 2018).
Revised: Additionally, it has been empirically observed that having no momentum (i.e., β 1 ≈ 0 ) is optimal for many popular GAN architectures (Karras et al., 2020; Brock et al., 2018).

Original: Thus, in what follows, we only consider the case where β 1 = 0 .
Revised: Therefore, we only consider the case where β 1 = 0 .

Original: Optimizers such as Adam (Algorithm 1) are adaptive because they keep updating step-sizes while training the model. 
Revised: Optimizers such as Adam (Algorithm 1) are adaptive since they use a step-size for each parameter that is different than the magnitude of the gradient g ( t ) Y for that parameter up to some constant (such as the global learning rate), and this step-size updates while training the model. There are three components that makes the adaptive update differ from the standard SGDA update: 1) the adaptive normalization by ∥ g ( t ) Y ∥ 2 , 2) the change of direction from g ( t ) Y / ∥ g ( t ) Y ∥ 2 to A ( t ) Y / ∥

Original: There are two components that contribute to the update step: the adaptive magnitude ∥ A ( t ) Y ∥ 2 and the adaptive direction
Revised: A ( t ) Y ∥ 2 and 3) adaptive scaling by ∥ A ( t ) Y ∥ 2 . In summary, the steps from the standard to the adaptive update, are:

Original: A ( t ) Y / ∥
Revised: [Equation] [Table]

Original: A ( t ) Y ∥ 2 .
Revised: [Equation] [Table]

Original:  The two components are entangled and it remains unclear how they contribute to the superior performance of adaptive methods relative to 
Revised: [Equation] The three components are entangled and it remains unclear how they contribute to the superior performance of adaptive methods relative to SGDA in GANs. [Table] [Table]

Original: SGDA in GANs.
Revised: [Table] [Table] [Table] [Table]

Original: N SGDA AS A MODEL TO ANALYZE A DAM IN GAN S
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: In this section, we show that normalized stochastic gradient descent-ascent (nSGDA) is a suitable proxy to study the learning dynamics of Adam.
Revised: [Table] [Table] [Table] [Table]

Original: Input : initial points W ( size Initialize to do Receive stochastic gradients g ( V evaluated at W ( t ) and V ( t ) .
Revised: [Table]

Original: Update for Y ∈ {W , V} : M ( t +1) Y , 1 = β 1 M ( t ) Y , 1 + g ( t ) Y and M ( t +1) Y , 2 = β 2 M ( t ) Y , 2 + g ( t ) Y 2 . Compute gradient oracles for Y ∈ { V, W } : A (
Revised: [Table]

Original: +1) Y = M ( t +1) Y , 1 / (cid:114) M ( t +1)
Revised: [Table]

Original: Y , 2 + ε .
Revised: [Table]

Original:  Update: W ( t +1) = W ( t ) + η ( t ) D A ( t +1) W ,
Revised:  N SGDA AS A MODEL TO ANALYZE A DAM IN GAN S

Original: V ( t +1) = V ( t ) − η ( t ) G A ( t +1) V
Revised: In this section, we show that normalized stochastic gradient descent-ascent (nSGDA) is a suitable proxy to study the learning dynamics of Adam.

Original: [Figure] To decouple the adaptive magnitude and direction in Adam, we adopt thestep-size grafting approach proposed by Agarwal et al.
Revised: To decouple the normalization, change of direction, and adaptive scaling in Adam, we adopt the grafting approach proposed by Agarwal et al.

Original: At each iteration, we compute stochastic gradients, pass them to two optimizers
Revised: At each iteration, we compute stochastic gradients, pass them to two optimizers

Original: A 1 , A 2 and make a grafted step that combines the magnitude of A 1 ’s step and direction of A 2 ’s step.
Revised: A 1 , A 2 and make a grafted step that combines the magnitude of A 1 ’s step and direction of A 2 ’s step.

Original: We focus on the optimizer defined by grafting the Adam magnitude onto the SGDA direction, i.e:
Revised: We focus on the optimizer defined by grafting the Adam magnitude onto the SGDA direction, which corresponds to omitting the change of direction in (2):

Original: [Equation]
Revised: [Equation]

Original: where A ( t ) V , A ( t ) W are the Adam gradient oracles as in Algorithm 1 and g ( t ) V , g
Revised: where A ( t ) V , A ( t ) W are the Adam gradient oracles as in Algorithm 1 and g ( t ) V , g

Original: ( t ) W the stochastic gradients.
Revised: ( t ) W the stochastic gradients.

Original: We refer to this algorithm as Ada-nSGDA (combining the Adam magnitude and SGDA direction).
Revised: We refer to this algorithm as Ada-nSGDA (combining the Adam magnitude and SGDA direction).

Original: There are two natural implementations for nSDGA.
Revised: There are two natural implementations for nSDGA.

Original: In the layer-wise version, Y ( t ) is a single parameter group (typically a layer in a neural network), and the updates are applied to each group.
Revised: In the layer-wise version, Y ( t ) is a single parameter group (typically a layer in a neural network), and the updates are applied to each group.

Original: In the global version, Y ( t ) contains all of the model’s weights.
Revised: In the global version, Y ( t ) contains all of the model’s weights.

Original: In Fig.
Revised: In Fig.

Original: 2, we show that Ada-nSGDA and Adam appear to have similar learning dynamics in terms of the FID score.
Revised: 2a, we see that Ada-nSGDA and Adam appear to have similar learning dynamics in terms of the FID score.

Original: Both Adam and Ada-nSGDA significantly outperform SGDA as well as AdaDir, which is the alternate case of (2) where we instead graft the magnitude of the SGDA update to the direction of the Adam update.
Revised: Both Adam and Ada-nSGDA significantly outperform SGDA as well as AdaDir, which is the alternate case of (3) where we instead graft the magnitude of the SGDA update to the direction of the Adam update.

Original: AdaDir diverged after a single step so we omit it in Fig.
Revised: AdaDir diverged after a single step so we omit it in Fig.

Original: This confirms that thecritical component of Adam is the adaptive magnitude , and that the standard update direction recovered by SGDA is sufficient to recover  a good solution.
Revised: These results show that the adaptive scaling and normalization components are sufficient to recover the performance of Adam, suggesting that Ada-nSGDA is a valid proxy for Adam

Original: However, from a theoretical perspective, directly analyzing Ada-nSGDA is difficult due to the  adaptive magnitudes ∥
Revised: A natural question that arises is how the total adaptive magnitude varies during training. We empirically investigate this by tracking the layer-wise adaptive magnitudes of the Adam gradient oracles when training a GAN with Ada-nSGDA, and summarize our key findings here (see Sectionfor complete experimental details).

Original: A ( t ) V ∥ 2 , ∥ A ( t ) W ∥ 2 .
Revised: We first train a WGAN-GP (Arjovsky et al., 2017) model, and find that the adaptive magnitude is bound within a constant range, and that all the layers have approximately the same adaptive magnitude (Fig. 2 (b,c)).

Original: Therefore, in Section 3, we analyze normalized SGDA (nSGDA) which is AdanSGDA (2) where we omit the  adaptive magnitudes.
Revised: This suggests that the adaptive scaling component is constant (in expectation) and motivates the use of nSGDA , corresponding to AdanSGDA with a constant adaptive scaling factor.

Original: Although nSGDA does not consider the adaptive magnitude, it still recovers the performance of Adam for some architectures such as WGAN-GP (Arjovsky et al., 2017) as we show in Fig.
Revised: We then train a WGAN-GP model with nSGDA and we find that nSGDA mostly recovers the FID score obtained by Ada-nSGDA (Fig. 2a). We also validate this observation for more complicated GAN architectures by repeating this study on StyleGAN2 (Karras et al., 2019).

Original: This may come from the fact that the adaptive magnitudes stay within a constant range and do not fluctuate across time, as shown inFigs.
Revised: We find that the adaptive magnitudes also vary within a constant range, but each layer has its own constant scaling factor. Thus, training StyleGAN2 with nSGDA and a global normalization fails, but training with nSGDA with a different constant step-size for each layer yields a performance that mostly recovers that of Ada-nSGDA (Fig 5). These results suggest that the schedule of the adaptive scaling is not central in the success of Ada-nSGDA in

Original: 2b and 2c.
Revised: GANs. Instead, adaptive methods are successful because they normalize the gradients for each layer, which allows for more balanced updates between G and D as we will show in Section 3. We conduct more experiments in Section 4 and in Appendix A. [Figure]

Original:  W HY DOES N SGDA PERFORM BETTER THAN SGDA IN GAN S ?
Revised:  W HY DOES N SGDA PERFORM BETTER THAN SGDA IN GAN S ?

Original: In Section 2, we numerically showed thatnSGDA recovers the performance Adam. Similar to other optimization works in minimization, we use nSGDA as a model to understand Adam in GANs.
Revised: In Section 2, we empirically showed that the most important component of the adaptive magnitude is the normalization , and that nSGDA (an algorithm consisting of this component alone) is sufficient to recover most of the performance of Ada-nSGDA (and by extension, Adam).

Original: Our goal is to construct a dataset and model where we can prove that a model trained with nSGDA generates samples from the true training distribution while SGDA fails.
Revised: Our goal is to construct a dataset and model where we can prove that a model trained with nSGDA generates samples from the true training distribution while SGDA fails.

Original: To this end, we consider a dataset where the underlying distribution consists of two modes, defined as vectors u 1 , u 2 , that are slightly correlated (See Assumption 1) and consider the standard GANs’ training objective.
Revised: To this end, we consider a dataset where the underlying distribution consists of two modes, defined as vectors u 1 , u 2 , that are slightly correlated (see Assumption 1) and consider the standard GANs’ training objective.

Original: We show that a GAN trained with SGDA using any reasonable 1 step-size configuration suffers from mode collapse (Theorem 3.1); it only outputs samples from a single mode which is a weighted average of u 1 and u 2 .
Revised: We show that a GAN trained with SGDA using any reasonable 1 step-size configuration suffers from mode collapse (Theorem 3.1); it only outputs samples from a single mode which is a weighted average of u 1 and u 2 .

Original: Conversely, nSGDA-trained GANs learn the two modes separately (Theorem 3.2).
Revised: Conversely, nSGDA-trained GANs learn the two modes separately (Theorem 3.2).

Original: Notation We set the GAN 1-sample loss L ( t ) V , W ( X, z )
Revised: Notation We set the GAN 1-sample loss L ( t ) V , W ( X, z )

Original: =
Revised: =

Original: log( D ( t ) W ( X ))
Revised: log( D ( t ) W ( X ))

Original: + log(1 − D ( t ) W (
Revised: + log(1 − D ( t ) W (

Original: G ( t ) V ( z ))) .
Revised: G ( t ) V ( z ))) .

Original: We denote g ( t ) Y = ∇ Y L ( t ) V , W ( X, z ) as the 1-sample stochastic gradient.
Revised: We denote g ( t ) Y = ∇ Y L ( t ) V , W ( X, z ) as the 1-sample stochastic gradient.

Original: We use the asymptotic complexity notations when defining the different constants e.g. poly( d ) refers to any polynomial in the dimension d , polylog( d ) to any polynomial in log( d ) , and o (1) to a constant ≪ d .
Revised: We use the asymptotic complexity notations when defining the different constants e.g. poly( d ) refers to any polynomial in the dimension d , polylog( d ) to any polynomial in log( d ) , and o (1) to a constant ≪ d .

Original: We denote a ∝ b for vectors a and b in R d if there is a positive scaling factor c > 

Revised: We denote a ∝ b for vectors a and b in R d if there is a positive scaling factor c >

Original: such that ∥ a − cb ∥ 2 =
Revised: such that ∥ a − cb ∥ 2 =

Original: o ( ∥ b
Revised: o ( ∥ b

Original: ∥ 2 ) .
Revised: ∥ 2 ) .

Original: 3. S ETTING
Revised: 3. S ETTING

Original: In this section, we present the setting to sketch our main results in Theorem 3.1 and Theorem 3.2.
Revised: In this section, we present the setting to sketch our main results in Theorem 3.1 and Theorem 3.2.

Original: We first define the distributions for the training set and latent samples, and specify our GAN model and the algorithms we analyze to solve (GAN).
Revised: We first define the distributions for the training set and latent samples, and specify our GAN model and the algorithms we analyze to solve (GAN).

Original: Note that for many assumptions and theorems below, we present informal statements which are sufficient to capture the main insights.
Revised: Note that for many assumptions and theorems below, we present informal statements which are sufficient to capture the main insights.

Original: The precise statements can be found in Appendix B.
Revised: The precise statements can be found in Appendix B.

Original: Our synthetic theoretical framework considers a bimodal data distribution with two correlated modes:
Revised: Our synthetic theoretical framework considers a bimodal data distribution with two correlated modes:

Original: Assumption 1 ( p data structure) .
Revised: Assumption 1 ( p data structure) .

Original: Let γ = 1 polylog( d ) .
Revised: Let γ = 1 polylog( d ) .

Original: We assume that the modes are correlated.
Revised: We assume that the modes are correlated.

Original: This means that ⟨ u 1 , u 2 ⟩ = γ > 0 and the generated data point X is either X = u 1 or X =
Revised: This means that ⟨ u 1 , u 2 ⟩ = γ > 0 and the generated data point X is either X = u 1 or X =

Original: u 2 .
Revised: u 2 .

Original: Next, we define the latent distribution p z that G V samples from and maps to p data .
Revised: Next, we define the latent distribution p z that G V samples from and maps to p data .

Original: Each sample from p z consists of a data-point z that is a binary-valued vector z ∈ { 0 , 1 } m G , where m G is the number of neurons in G V , and has non-zero support, i.e. ∥ z ∥ 0 ≥ 1 .
Revised: Each sample from p z consists of a data-point z that is a binary-valued vector z ∈ { 0 , 1 } m G , where m G is the number of neurons in G V , and has non-zero support, i.e. ∥ z ∥ 0 ≥ 1 .

Original: Although the typical choice of a latent distributions in GANs is either Gaussian or uniform, we choose p z to be a binary distribution because it models the weights’ distribution of a hidden layer of a deep generator; Allen-Zhu & Li (2021) argue that the distributions of these hidden layers are sparse, non-negative, and non-positively correlated.
Revised: Although the typical choice of a latent distributions in GANs is either Gaussian or uniform, we choose p z to be a binary distribution because it models the weights’ distribution of a hidden layer of a deep generator; Allen-Zhu & Li (2021) argue that the distributions of these hidden layers are sparse, non-negative, and non-positively correlated.

Original: We now make the following assumptions on the coefficients of z :
Revised: We now make the following assumptions on the coefficients of z :

Original: Assumption 2 ( p z structure) .
Revised: Assumption 2 ( p z structure) .

Original: Let z ∼ p z .
Revised: Let z ∼ p z .

Original: We assume that with probability 1 − o (1) , there is only one non-zero entry in z .
Revised: We assume that with probability 1 − o (1) , there is only one non-zero entry in z .

Original: The probability that the entry i ∈ [ m G ] is non-zero is Pr[ z i = 1] =
Revised: The probability that the entry i ∈ [ m G ] is non-zero is Pr[ z i = 1] =

Original: Θ(1 /m G ) .
Revised: Θ(1 /m G ) .

Original: In Assumption 2, the output of G V is only made of one mode with probability 1 − o (1) .
Revised: In Assumption 2, the output of G V is only made of one mode with probability 1 − o (1) .

Original: This avoids summing two of the generator’s neurons, which may cause mode collapse.
Revised: This avoids summing two of the generator’s neurons, which may cause mode collapse.

Original: To learn the target distribution p data , we use a linear generator G V with m G neurons and a non-linear neural network with m D neurons:
Revised: To learn the target distribution p data , we use a linear generator G V with m G neurons and a non-linear neural network with m D neurons:

Original: [Equation]
Revised: [Equation]

Original: where V =
Revised: where V =

Original: [ v ⊤ 1 , v ⊤ 2 , · · · , v ⊤ m G ] ∈
Revised: [ v ⊤ 1 , v ⊤ 2 , · · · , v ⊤ m G ] ∈

Original: R m G × d , z ∈ { 0 , 1 } m G , W = [ w ⊤ 1 , . .
Revised: R m G × d , z ∈ { 0 , 1 } m G , W = [ w ⊤ 1 , . .

Original: . , w ⊤ m D ] ∈ R m D × d , and a, b ∈ R .
Revised: . , w ⊤ m D ] ∈ R m D × d , and a, b ∈ R .

Original: Intuitively, G V outputs linear combinations of the modes v i .
Revised: Intuitively, G V outputs linear combinations of the modes v i .

Original: We choose a cubic activation as it is the smallest monomial degree for the discriminator’s non-linearity that is sufficient for the generator to recover the modes u 1 , u 2 .
Revised: We choose a cubic activation as it is the smallest monomial degree for the discriminator’s non-linearity that is sufficient for the generator to recover the modes u 1 , u 2 .

Original: We now state the SGDA and nSGDA algorithms used to solve the GAN training problem (GAN).
Revised: We now state the SGDA and nSGDA algorithms used to solve the GAN training problem (GAN).

Original: For simplicity, we set the batch-size to 1.
Revised: For simplicity, we set the batch-size to 1.

Original: The resultant update rules for SGDA and nSGDA are:
Revised: The resultant update rules for SGDA and nSGDA are:

Original: SGDA: at each step t > 0 , sample X ∼ p data and z ∼ p z and update as
Revised: SGDA: at each step t > 0 , sample X ∼ p data and z ∼ p z and update as

Original: [Equation]
Revised: [Equation]

Original: nSGDA: at each step t > 0 , sample X ∼ p data and z ∼ p z and update as
Revised: nSGDA : at each step t > 0 , sample X ∼ p data and z ∼ p z and update as

Original: [Equation]
Revised: [Equation]

Original: Compared to the versions of SGDA and Ada-nSGDA that we introduced in Section 2, we have the same algorithms except that we set β 1 = 0 and omit ε in (4) and (5).
Revised: Compared to the versions of SGDA and Ada-nSGDA that we introduced in Section 2, we have the same algorithms except that we set β 1 = 0 and omit ε in (5) and (6).

Revised: Note that since there is only one layer in the neural networks we study in this paper, the global-wise and layer-wise versions of nSGDA are actually the same.

Original: Lastly, we detail how to set the optimization parameters for SGDA and nSGDA in (4) and (5).
Revised: Lastly, we detail how to set the optimization parameters for SGDA and nSGDA in (5) and (6).

Original: Parametrization 3.1 (Informal) .
Revised: Parametrization 3.1 (Informal) .

Original: When running SGDA and nSGDA on (GAN), we set:
Revised: When running SGDA and nSGDA on (GAN), we set:

Original: – Initialization : b (0) = 0 , and a (0) , w (0) i ( i ∈ [ m D ]) , v (0) j ( j ∈ [ m G ]) are initialized with a Gaussian with small variance.
Revised: – Initialization : b (0) = 0 , and a (0) , w (0) i ( i ∈ [ m D ]) , v (0) j ( j ∈ [ m G ]) are initialized with a Gaussian with small variance.

Original: – Number of iterations : we run SGDA for t ≤ T 0 iterations where T 0 is the first iteration such that the algorithm converges to an approximate first order local minimum.
Revised: – Number of iterations : we run SGDA for t ≤ T 0 iterations where T 0 is the first iteration such that the algorithm converges to an approximate first order local minimum.

Original: For nSGDA, we run for T 1 = ˜Θ(

Revised: For nSGDA, we run for T 1 = ˜Θ(

Original: /η D ) iterations.
Revised: /η D ) iterations.

Original: – Step-sizes : For SGDA, η D , η G ∈ (0 , 1 poly( d ) ) can be arbitrary.
Revised: – Step-sizes : For SGDA, η D , η G ∈ (0 , 1 poly( d ) ) can be arbitrary.

Original: For nSGDA, η D ∈ (0 , [Equation] d ) ] , and η G is slightly smaller than η D .
Revised: For nSGDA, η D ∈ (0 , 1 poly( d ) ] , and η G is slightly smaller than η D .

Original: – Over-parametrization : For SGDA, m D , m G = polylog( d ) are arbitrarily chosen i.e. m D may be larger than m G or the opposite.
Revised: – Over-parametrization : For SGDA, m D , m G = polylog( d ) are arbitrarily chosen i e. m D may be larger than m G or the opposite.

Original: For nSGDA, we set m D = log( d ) and m G = 2 log( d ) .
Revised: For nSGDA, we set m D = log( d ) and m G = 2 log( d ) .

Original: Our theorem holds when running SGDA for any (polynomially) possible number of iterations; after
Revised: Our theorem holds when running SGDA for any (polynomially) possible number of iterations; after

Original: T 0 steps, the gradient becomes inverse polynomially small and SGDA essentially stops updating the parameters.
Revised: T 0 steps, the gradient becomes inverse polynomially small and SGDA essentially stops updating the parameters.

Original: Additionally, our setting allows any step-size configuration for SGDA i.e. larger, smaller, or equal step-size for D compared to G .
Revised: Additionally, our setting allows any step-size configuration for SGDA i.e. larger, smaller, or equal step-size for D compared to G .

Original: Note that our choice of step-sizes for nSGDA is the one used in practice, i.e. η D slightly larger than η G .
Revised: Note that our choice of step-sizes for nSGDA is the one used in practice, i.e. η D slightly larger than η G .

Original: 3. M AIN RESULTS
Revised: 3. M AIN RESULTS

Original: We state our main results on the performance of models trained using SGDA (4) and nSGDA (5).
Revised: We state our main results on the performance of models trained using SGDA (5) and nSGDA (6).

Original: We show that nSGDA learns the modes of the distribution p data while SGDA does not.
Revised: We show that nSGDA learns the modes of the distribution p data while SGDA does not.

Original: Theorem 3.1 (Informal) .
Revised: Theorem 3.1 (Informal) .

Original: Consider a training dataset and a latent distribution as described above and let Assumption 1 and Assumption 2 hold.
Revised: Consider a training dataset and a latent distribution as described above and let Assumption 1 and Assumption 2 hold.

Original: Let T 0 , η G , η D and the initialization be as defined in
Revised: Let T 0 , η G , η D and the initialization be as defined in

Revised: [Figure]

Original: {+V 1 V 2 V 3 V 4 V 5 Generator's neurons 
+}
Revised: {+V 1 V 2 V 3 V 4 V 5 Generator's neurons 
+}

Revised: It e r a t i on s u 1 u 2 u 1 u 2 u 1 u 2 u 1 u 2 u 1 u0.

Revised: [Figure]

Original: Parametrization 3.1.
Revised: Parametrization 3.1.

Original: Let t be such that t ≤ T 0 .
Revised: Let t be such that t ≤ T 0 .

Original: Run SGDA onthe GAN problem defined in (GAN) for t iterations with step-sizes η G , η D .
Revised: Run SGDA on (GAN) for t iterations with step-sizes η G , η D .

Original: Then, with probability at least 1 − o (1) , the generator outputs for all z ∈ { 0 , 1 } m G : 
Revised: Then, with probability at least 1 − o (1) , the generator outputs for all z ∈ { 0 , 1 } m G : [Equation] otherwise

Original: [Equation]
Revised: [Equation]

Original: (6) where ξ ( t ) ( z
Revised: (7) where ξ ( t ) ( z

Original: ) ∈ R d is some vector that is not correlated to any of the modes.
Revised: ) ∈ R d is some vector that is not correlated to any of the modes.

Original: Formally, ∀ ℓ ∈ [2] , cos( ξ ( t ) ( z ) , u ℓ ) =
Revised: Formally, ∀ ℓ ∈ [2] , cos( ξ ( t ) ( z ) , u ℓ ) =

Original: o (1) for all z ∈ { 0 , 1 } m G .
Revised: o (1) for all z ∈ { 0 , 1 } m G .

Original: A formal proof can be found in Appendix G. Theorem 3.1 indicates that when training with SGDA and any step-size configuration, the generator either does not learn the modes at all ( G ( t ) V ( z )
Revised: A formal proof can be found in Appendix G. Theorem 3.1 indicates that when training with SGDA and any step-size configuration, the generator either does not learn the modes at all ( G ( t ) V ( z )

Original: = ξ ( t ) ( z ) ) or learns an average of the modes (
Revised: = ξ ( t ) ( z ) ) or learns an average of the modes (

Original: G ( t ) V ( z ) ∝ u 1 + u 2 ).
Revised: G ( t ) V ( z ) ∝ u 1 + u 2 ).

Original: The theorem holds for any time t ≤ Twhich is the iteration where SGDA converges to an approximate first-order locally optimal min-max equilibrium.
Revised: The theorem holds for any time t ≤ Twhich is the iteration where SGDA converges to an approximate first-order locally optimal min-max equilibrium.

Original: Conversely, nSGDA succeeds in learning the two modes separately:
Revised: Conversely, nSGDA succeeds in learning the two modes separately:

Original: [Figure]

Original: [-V 1 V 2 V 3 V 4 V 5 Generator's neurons 
-]
Revised: [-V 1 V 2 V 3 V 4 V 5 Generator's neurons 
-]

Original: It e r a t i on s u 1 u 2 u 1 u 2 u 1 u 2 u 1 u 2 u 1 u0.

Original: [Figure]

Original: Theorem 3.2 (Informal) .
Revised: Theorem 3.2 (Informal) .

Original: Consider a training dataset and a latent distribution as described above and let Assumption 1 and Assumption 2 hold.
Revised: Consider a training dataset and a latent distribution as described above and let Assumption 1 and Assumption 2 hold.

Original: Let T 1 , η G , η D and the initialization as defined in
Revised: Let T 1 , η G , η D and the initialization as defined in

Original: Parametrization 3.1.
Revised: Parametrization 3.1.

Original: Run nSGDA on (GAN) for T 1 iterations with step-sizes η G , η D .
Revised: Run nSGDA on (GAN) for T 1 iterations with step-sizes η G , η D .

Original: Then, the generator learns both modes u 1 , u 2 i.e., for ℓ ∈ { 1 , 2 } ,
Revised: Then, the generator learns both modes u 1 , u 2 i.e., for ℓ ∈ { 1 , 2 } ,

Original: [Equation]
Revised: [Equation]

Original: A formal proof can be found in Appendix I. Theorem 3.2 indicates that when we train a GAN with nSGDA in the regime where the discriminator updates slightly faster than the generator (as done in practice), the generator successfully learns the distribution containing the direction of both modes.
Revised: A formal proof can be found in Appendix I. Theorem 3.2 indicates that when we train a GAN with nSGDA in the regime where the discriminator updates slightly faster than the generator (as done in practice), the generator successfully learns the distribution containing the direction of both modes.

Original: We implement the setting introduced in Subsection 3.1 and validate Theorem 3.1 and Theorem 3.
Revised: We implement the setting introduced in Subsection 3.1 and validate Theorem 3.1 and Theorem 3.

Original: in Fig.
Revised: in Fig.

Original: Fig.
Revised: Fig.

Original: 3a displays the relative update speed η ∥ g ( t ) Y ∥ 2 / ∥Y ( t ) ∥ 2 , where Y corresponds to the parameters of either D or G .
Revised: 3a displays the relative update speed η ∥ g ( t ) Y ∥ 2 / ∥Y ( t ) ∥ 2 , where Y corresponds to the parameters of either D or G .

Original: Fig.
Revised: Fig.

Original: 3b shows the correlation ⟨ w ( t ) i , u ℓ ⟩ / ∥ w ( t ) i ∥ 2 between one of D ’s neurons and a mode u ℓ and Fig.
Revised: 3b shows the correlation ⟨ w ( t ) i , u ℓ ⟩ / ∥ w ( t ) i ∥ 2 between one of D ’s neurons and a mode u ℓ and Fig.

Original: 3c the correlation ⟨ v ( t ) j , u ℓ ⟩ / ∥ v ( t ) j ∥ 2 between G ’s neurons and u ℓ .
Revised: 3c the correlation ⟨ v ( t ) j , u ℓ ⟩ / ∥ v ( t ) j ∥ 2 between G ’s neurons and u ℓ .

Original: We discuss the interpretation of these plots to the next section.
Revised: We discuss the interpretation of these plots to the next section.

Original: W HY DOES SGDA SUFFER FROM MODE COLLAPSE AND N SGDA LEARN THE MODES ?
Revised: W HY DOES SGDA SUFFER FROM MODE COLLAPSE AND N SGDA LEARN THE MODES ?

Original: We now explain why SGDA suffers from mode collapse, which corresponds to the case where η D ≥ η G .
Revised: We now explain why SGDA suffers from mode collapse, which corresponds to the case where η D ≥ η G .

Original: Our explanation relies on the interpretation of Figs.
Revised: Our explanation relies on the interpretation of Figs.

Original: 3a, 3b, and 3c, and on the updates around initialization that are defined as followed.
Revised: 3a, 3b, and 3c, and on the updates around initialization that are defined as followed.

Original: There exists i ∈
Revised: There exists i ∈

Original: [ m D ] such that D ’s update iswith high probability
Revised: [ m D ] such that D ’s update is

Original: [Equation]
Revised: [Equation]

Original: Thus, the weights of D receive gradients directed by u 1 and u 2 .
Revised: Thus, the weights of D receive gradients directed by u 1 and u 2 .

Original: On the other hand, the weights of G at early stages receive gradients directed by w ( t ) j :
Revised: On the other hand, the weights of G at early stages receive gradients directed by w ( t ) j :

Original: [Equation]
Revised: [Equation]

Original: We observe that the learning process in Figs.
Revised: We observe that the learning process in Figs.

Original: 3a & 3b has three distinct phases.
Revised: 3a & 3b has three distinct phases.

Original: In the first phase (iterations 1-20), D learns one of the modes ( u 1 or u 2 ) of p data (Fig. 3b) and G barely updates its weights (Fig. 3a).
Revised: In the first phase (iterations 1-20), D learns one of the modes ( u 1 or u 2 ) of p data (Fig. 3b) and G barely updates its weights (Fig. 3a).

Original: In the second phase (iterations 20-40), D learns the weighted average u 1 + u(Fig.
Revised: In the second phase (iterations 20-40), D learns the weighted average u 1 + u(Fig.

Original: 3b) while G starts moving its weights (Fig. 3a).
Revised: 3b) while G starts moving its weights (Fig. 3a).

Original: In the final phase (iterations 40+), G learns u 1 + u 2 (Fig. 3c) from D .
Revised: In the final phase (iterations 40+), G learns u 1 + u 2 (Fig. 3c) from D .

Original: In more detail, the learning process is described as follows:
Revised: In more detail, the learning process is described as follows:

Original: Phase 1 : At initialization, w (0) j and v (0)
Revised: Phase 1 : At initialization, w (0) j and v (0)

Original: j are small.
Revised: j are small.

Original: Assume w.l.o.g. that ⟨ w (0) i , u 2 ⟩ > ⟨ w (0) i , u 1 ⟩ .
Revised: Assume w.l.o.g. that ⟨ w (0) i , u 2 ⟩ > ⟨ w (0) i , u 1 ⟩ .

Original: Because of the ⟨ w ( t ) i , u l ⟩ 2 in front of u 2 in (8), the parameter w ( t ) i gradually grows its correlation with u 2 (Fig. 3b) and D ’s gradient norm thus increases (Fig. 3a).
Revised: Because of the ⟨ w ( t ) i , u l ⟩ 2 in front of u 2 in (9), the parameter w ( t ) i gradually grows its correlation with u 2 (Fig. 3b) and D ’s gradient norm thus increases (Fig. 3a).

Original: While ∥ w ( t ) j ∥ ≪ 1 ∀ j , we have that v ( t ) i ≈ v (0) i (Fig. 3a).
Revised: While ∥ w ( t ) j ∥ ≪ 1 ∀ j , we have that v ( t ) i ≈ v (0) i (Fig. 3a).

Original: Phase 2 : D has learned u 2 .
Revised: Phase 2 : D has learned u 2 .

Original: Because of the sigmoid in the gradient of w ( t ) i (that was negligible during
Revised: Because of the sigmoid in the gradient of w ( t ) i (that was negligible during

Original: Phase 1) and ⟨ u 1 , u 2 ⟩ = γ > 0 , w ( t ) i now mainly receives updates with direction u 2 .
Revised: Phase 1) and ⟨ u 1 , u 2 ⟩ = γ > 0 , w ( t ) i now mainly receives updates with direction u 2 .

Original: Since G did not update its weights yet, the min-max problem (GAN) is approximately just a minimization problem with respect to D ’s parameters.
Revised: Since G did not update its weights yet, the min-max problem (GAN) is approximately just a minimization problem with respect to D ’s parameters.

Original: Since the optimum of such a problem is the weighted average u 1 + u 2 , w ( t ) j slowly converges to this optimum.
Revised: Since the optimum of such a problem is the weighted average u 1 + u 2 , w ( t ) j slowly converges to this optimum.

Original: Meanwhile, v ( t ) i start to receive some significant signal (Fig. 3a) but mainly learn the direction u 1 + u 2 (Fig. 3c), because w ( t ) j is aligning with this direction.
Revised: Meanwhile, v ( t ) i start to receive some significant signal (Fig. 3a) but mainly learn the direction u 1 + u 2 (Fig. 3c), because w ( t ) j is aligning with this direction.

Original: Phase 3: The parameters of G only receive gradient directed by u 1 + u 2 .
Revised: Phase 3: The parameters of G only receive gradient directed by u 1 + u 2 .

Original: The norm of its relative updates stay large and D only changes its last layer terms (slope a and bias b ).
Revised: The norm of its relative updates stay large and D only changes its last layer terms (slope a and bias b ).

Original: In contrast to SGDA, nSGDA ensures that G and D always learn at the same speed with the updates:w ( t +1) i
Revised: In contrast to SGDA, nSGDA ensures that G and D always learn at the same speed with the updates:

Original: [Equation]
Revised: [Equation]

Original: , andv
Revised: and

Original: [Equation]
Revised: [Equation]

Original: No matter how large ⟨ w ( t ) i , X ⟩ is, G still learns at the same speed with D .
Revised: No matter how large ⟨ w ( t ) i , X ⟩ is, G still learns at the same speed with D .

Original: There is a tight window (iteration 25 , Fig. 3b) where only one neuron of D is aligned with u 1 .
Revised: There is a tight window (iteration 25 , Fig. 3b) where only one neuron of D is aligned with u 1 .

Original: This is when G can also learn to generate u 1 by “catching up” to D at that point, which avoids mode collapse.
Revised: This is when G can also learn to generate u 1 by “catching up” to D at that point, which avoids mode collapse.

Original:  N UMERICAL PERFORMANCE OF N SGDA
Revised:  N UMERICAL PERFORMANCE OF N SGDA

Original: In Section 2, we present the Ada-nSGDA algorithm (2) which corresponds to “grafting” the Adam magnitude onto the SGDA direction.
Revised: In Section 2, we presented the Ada-nSGDA algorithm (3) which corresponds to “grafting” the Adam magnitude onto the SGDA direction.

Original: In Section 3, we construct a dataset and GAN model where we prove that a GAN trained with nSGDA can generate examples from the true training distribution, while a GAN trained with SGDA fails due to mode collapse.
Revised: In Section 3, we construct a dataset and GAN model where we prove that a GAN trained with nSGDA can generate examples from the true training distribution, while a GAN trained with SGDA fails due to mode collapse.

Original: We now provide more experiments comparing nSGDA and Ada-nSGDA with Adam on real GANs and datasets.
Revised: We now provide more experiments comparing nSGDA and Ada-nSGDA with Adam on real GANs and datasets.

Original: We train a ResNet WGAN with gradient penalty on CIFAR-10 (Krizhevsky et al., 2009) and STL- (Coates et al., 2011) with Adam, Ada-nSDGA, SGDA, as well as nSGDA with a fixed learning rate as done in Section 3.
Revised: We train a ResNet WGAN with gradient penalty on CIFAR-10 (Krizhevsky et al., 2009) and STL- (Coates et al., 2011) with Adam, Ada-nSDGA, SGDA, and nSGDA with a fixed learning rate as done in Section 3.

Original: We use the default architectures and training parameters specified in Gulrajani et al.
Revised: We use the default architectures and training parameters specified in Gulrajani et al.

Original: (2017) ( λ GP = 10 , n dis = 5 , learning rate decayed linearly to 0 over 100k steps).
Revised: (2017) ( λ GP = 10 , n dis = 5 , learning rate decayed linearly to 0 over 100k steps).

Original: We also train a StyleGAN2 model (Karras et al., 2020) on FFHQ (Karras et al., 2019) and LSUN Churches (Yu et al., 2016) (both resized to 128 × 128 pixels) with Adam, Ada-nSGDA,  and SGDA.
Revised: We also train a StyleGAN2 model (Karras et al., 2020) on FFHQ (Karras et al., 2019) and LSUN Churches (Yu et al., 2016) (both resized to 128 × 128 pixels) with Adam, Ada-nSGDA, SGDA, and nSGDA.

Original: We use the recommended StyleGAN2 hyperparameter configuration for this resolution (batch size = 32, γ = 0 .
Revised: We use the recommended StyleGAN2 hyperparameter configuration for this resolution (batch size = 32, γ = 0 .

Original: 1024 , map depth = 2, channel multiplier = 16384).
Revised: 1024 , map depth = 2, channel multiplier = 16384).

Original: We use the Fr ´ echet Inception distance (FID) (Heusel et al., 2017) to quantitatively assess the performance of the model.
Revised: We use the Fr´echet Inception distance (FID) (Heusel et al., 2017) to quantitatively assess the performance of the model.

Original: For each optimizer, we conduct a coarse log-space sweep over step sizes and optimize for FID.
Revised: For each optimizer, we conduct a coarse log-space sweep over step sizes and optimize for FID.

Original: We train the WGAN-GP models for 2880 thousand images (kimgs) on CIFAR-10 and STL-10 (45k steps with a batch size of 64), and the StyleGAN2 models for 2600 kimgs on FFHQ and LSUN Churches.
Revised: We train the WGAN-GP models for 2880 thousand images (kimgs) on CIFAR-10 and STL-10 (45k steps with a batch size of 64), and the StyleGAN2 models for 2600 kimgs on FFHQ and LSUN Churches.

Original: We average our results over 5 seeds for the WGAN-GP ResNets, and over 3 seeds for the StyleGAN2 models due to the computational cost associated with training GANs.
Revised: We average our results over 5 seeds for the WGAN-GP ResNets, and over 3 seeds for the StyleGAN2 models due to the computational cost associated with training GANs.

Original: WGAN-GP Figures 4a and 4b show the FID curves during training for the  WGAN-GP model. We test Adam, Ada-nSGDA, AdaDir (that contains the adaptive direction), nSGDA and SGDA. The first remark is that Ada-nSGDA recovers the performance of Adam while AdaDir diverges ( NaN loss) for these experiments and is hence omitted. This means that the adaptive magnitude is key to the performance of adaptive methods.
Revised: WGAN-GP Figures 4a and 4b validates the conclusions on WGAN-GP from Section 2.

Original: Additionally, nSGDA not only recovers the performance of Adam but obtains a final FID of ∼ 2-3 points lower than Ada-nSGDA.
Revised: We find that both Ada-nSGDA and nSGDA mostly recover the performance of Adam, with nSGDA obtaining a final FID of ∼ 2-3 points lower than Ada-nSGDA.

Original: Such performance is possible because the adaptive magnitude∥ A ( t ) Y ∥ 2 stays within a constant range and does not fluctuate much (Figs. 2b, 2c) and a constant learning rate as in nSGDA is enough to recover Adam’s performance. Thus, nSGDA is a valid model for Adam in the case of WGAN-GP.
Revised: As discussed in Section 2, such performance is possible because the adaptive magnitude stays within a constant range.

Original: In contrast, models trained with SGDA consistently perform significantly worse, with final FID scores 4 × larger than Adam. 
Revised: In contrast, models trained with SGDA consistently perform significantly worse, with final FID scores 4 × larger than Adam. [Figure]

Original:  StyleGAN2 Figures 4c and 4d show the  FID curves during training for StyleGAN2.Similar to
Revised: [Figure] StyleGAN2 Figures 4c and 4d show the final FID scores when training a StyleGAN2.

Original: WGAN-GP, we observe that Ada-nSGDA recovers  theAdam performance while SGDA performs much worse. However, nSGDA was not able to recover the performance of Adam in this setting. There are several hypotheses to explain this: [Figure]
Revised: We find that Ada-nSGDA recovers most of the performance of Adam, but one difference with WGAN-GP is that nSGDA does not work if we use the same global learning rate for each layer.

Original: [Figure] • We observe in Figs. 5b, 5c that the adaptive magnitude the ratio ∥ A ( t ) Y ∥ 2 stays within a constant range but varies significantly across time.
Revised: As discussed in Section 2, nSGDA with a different (but constant) step-size for each layer does work, and is able to mostly recover Ada-nSGDA’s performance (Fig. 5a).

Original: This violates our theoretical setting of assuming a constant adaptive magnitude. In contrast, ∥ A ( t ) Y ∥ 2 stays within a constant range and does not fluctuate across time in the WGAN-GP experiment (Figs. 2b, 2c). • The GAN problem formulation in the case of StyleGAN2 is very different from the one in WGANGP. Specifically, StyleGAN2 has a drastically different generator architecture than the ResNet generator used in the WGAN-GP experiments (utilizing weight demodulation), as well as using adaptive data augmentation and additional regularizers (such as a path-length regularization).
Revised: To choose the scaling for each layer, we train StyleGAN2 with Ada-nSGDA on FFHQ-128, track the layer-wise adaptive magnitudes and take the mean of these magnitudes over the training run (for each layer).

Original: Because our theoretical setting does not capture these observations, we do not expect nSGDA to work in this setting.

Original: However in spite of these differences, Ada-nSGDA performed similarly to Adam indicating that the nSGDA direction that we theoretically study is still valid in modern real-world GAN architectures. We further validate our theory with additional experiments on DCGAN (Radford et al., 2015) which more closely matches our theoretical setting in Appendix A, and find that nSGDA recovers the performance of Adam as in WGAN-GP.
Revised: Figure 5b shows that the fluctuations for each layer are bound to a constant range, validating our assumption of constant step-sizes. Additionally, the same scaling obtained from training FFHQ seems to work for different datasets; we used it to train StyleGAN2 with nSGDA on LSUN Churches-128 and recovered similar performance to training on this dataset with Ada-nSGDA (Fig. 4d).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our work addresses the question of how adaptive methods improve the GAN performance.
Revised: Our work addresses the question of which mechanisms in adaptive methods are critical for training

Original:  We empirically showed that  the adaptive magnitude of Adam is responsible for thehigh performance of  adaptive methods in GANs.
Revised: GANs, and why they outperform non-adaptive methods. We empirically show that Ada-nSGDA, an algorithm composed of the adaptive magnitude of Adam and the direction of SGD, recovers most of the performance of Adam. We further decompose the adaptive magnitude into two components: normalization, and adaptive step-size.

Original: We further show that the adaptive magnitude is almost constant in some settings (for instance in the case of WGAN-GP), and nSGDA isable to recover the learning dynamics of Adam.
Revised: We then show that the adaptive step size is roughly constant (bounded fluctuations) for multiple architectures and datasets. This empirically indicates that the normalization component of the adaptive magnitude is the key mechanism of Ada-nSGDA, and motivates the study of nSGDA; we verify that it too recovers the performance of Ada-nSGDA.

Original: We constructed a setting where we proved that nSGDA –thanks to its balanced updates– recovers the modes of the true distribution while SGDA fails to do it.
Revised: Having shown that nSGDA is a good proxy for a key mechanism for adaptive methods, we then construct a setting where we proved that nSGDA –thanks to its balanced updates– recovers the modes of the true distribution while SGDA fails to do it.

Original: Our theory provides insights on the effectiveness of adaptive methods in architectures such as DCGAN or WGAN-GP, however the limitation of our work is that we do not fully capture all of the components in modern architectures such as StyleGAN2. An exciting direction would be  to  findan optimization model that recovers the same perform for Adam on these modern GAN architectures.
Revised: The key insight from our theoretical analysis is that the ratio of the update of D and G must be close to 1 during training in order to recover the modes of the distribution. This matches the experimental setting with nSGDA, as we find that global norm of the parameter updates for both D and G are almost equal for optimal choices of learning rates.