Original: LOGAN: L ATENT O PTIMISATION FOR G ENERATIVE A DVERSARIAL N ETWORKS
Revised: LOGAN: L ATENT O PTIMISATION FOR G ENERATIVE A DVERSARIAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training generative adversarial networks requires balancing of delicate adversarial dynamics.
Revised: Training generative adversarial networks requires balancing of delicate adversarial dynamics.

Original: Even with careful tuning, training may diverge or end up in a bad equilibrium with dropped modes.
Revised: Even with careful tuning, training may diverge or end up in a bad equilibrium with dropped modes.

Original: In this work, we introduce a new form of latent optimisation inspired by the CS-GAN and show that it improves adversarial dynamics by enhancing interactions between the discriminator and the generator.
Revised: In this work, we introduce a new form of latent optimisation inspired by the CS-GAN and show that it improves adversarial dynamics by enhancing interactions between the discriminator and the generator.

Original: We develop supporting theoretical analysis from the perspectives of differentiable games and stochastic approximation.
Revised: We develop supporting theoretical analysis from the perspectives of differentiable games and stochastic approximation.

Original: Our experiments demonstrate that latent optimisation can signiﬁcantly improve GAN training, obtaining state-of-the-art performance for the ImageNet ( 128 × 128 ) dataset.
Revised: Our experiments demonstrate that latent optimisation can signiﬁcantly improve GAN training, obtaining state-of-the-art performance for the ImageNet ( 128 × 128 ) dataset.

Original: Our model achieves an Inception
Revised: Our model achieves an Inception

Original: Score (IS) of 148 and an Fr ´ echet Inception Distance (FID) of 3 . 4 , an improvement of 17% and 32% in IS and FID respectively, compared with the baseline
Revised: Score (IS) of 148 and an Fr´echet Inception Distance (FID) of 3 . 4 , an improvement of 17% and 32% in IS and FID respectively, compared with the baseline

Original: BigGAN-deep model with the same architecture and number of parameters.
Revised: BigGAN-deep model with the same architecture and number of parameters.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative Adversarial Nets (GANs) are implicit generative models that can be trained to match a given data distribution.
Revised: Generative Adversarial Nets (GANs) are implicit generative models that can be trained to match a given data distribution.

Original: GANs were originally proposed and demonstrated for images by Goodfellow et al.
Revised: GANs were originally proposed and demonstrated for images by Goodfellow et al.

Original: As the ﬁeld of generative modelling has advanced, GANs have remained at the frontier, generating high-ﬁdelity images at large scale (Brock et al., 2018).
Revised: As the ﬁeld of generative modelling has advanced, GANs have remained at the frontier, generating high-ﬁdelity images at large scale (Brock et al., 2018).

Original: However, despite growing insights into the dynamics of GAN training, most recent advances in large-scale image generation come from architectural improvements (Radford et al., 2015; Zhang et al., 2019), or
Revised: However, despite growing insights into the dynamics of GAN training, most recent advances in large-scale image generation come from architectural improvements (Radford et al., 2015; Zhang et al., 2019), or

Original: regularisation focusing on particular parts of the model (Miyato et al., 2018; Miyato & Koyama, 2018).
Revised: regularisation focusing on particular parts of the model (Miyato et al., 2018; Miyato & Koyama, 2018).

Original: Inspired by the compressed sensing GAN (CS-GAN; Wu et al., 2019), we further exploit the beneﬁt of latent optimisation in adversarial games using natural gradient descent to optimise the latent variable z at each step of training, presenting a scalable and easy to implement approach to improve the dynamical interaction between the discriminator and the generator.
Revised: Inspired by the compressed sensing GAN (CS-GAN; Wu et al., 2019), we further exploit the beneﬁt of latent optimisation in adversarial games using natural gradient descent to optimise the latent variable z at each step of training, presenting a scalable and easy to implement approach to improve the dynamical interaction between the discriminator and the generator.

Original: For clarity, we unify these approaches as latent optimised GANs (LOGAN).
Revised: For clarity, we unify these approaches as latent optimised GANs (LOGAN).

Original: To summarise our contributions:
Revised: To summarise our contributions:

Original: We present a novel analysis of latent optimisation in GANs from the perspective of differentiable games and stochastic approximation (Balduzzi et al., 2018; Heusel et al., 2017), arguing that latent optimisation can improve the dynamics of adversarial training.
Revised: We present a novel analysis of latent optimisation in GANs from the perspective of differentiable games and stochastic approximation (Balduzzi et al., 2018; Heusel et al., 2017), arguing that latent optimisation can improve the dynamics of adversarial training.

Original: Motivated by this analysis, we improve latent optimisation by taking advantage of efﬁcient second-order updates.
Revised: Motivated by this analysis, we improve latent optimisation by taking advantage of efﬁcient second-order updates.

Original: Our algorithm improves the state-of-the-art BigGAN-deep model (Brock et al., 2018) by a signiﬁcant margin, without introducing any architectural change or additional parameters, resulting in higher quality images and more diverse samples (Figure 1 and 2).
Revised: Our algorithm improves the state-of-the-art BigGAN-deep model (Brock et al., 2018) by a signiﬁcant margin, without introducing any architectural change or additional parameters, resulting in higher quality images and more diverse samples (Figure 1 and 2).

Original:  B ACKGROUND 2. N OTATION
Revised:  B ACKGROUND 2. N OTATION

Original: We use θ D and θ G to denote the vectors representing parameters of the generator and discriminator.
Revised: We use θ D and θ G to denote the vectors representing parameters of the generator and discriminator.

Original: We use x for images, and z for the latent source generating an image.
Revised: We use x for images, and z for the latent source generating an image.

Original: The prime (cid:48) is used to denote  a variable afterone update step, e.g., θ(cid:48)
Revised: The prime (cid:48) is used to denote [Figure] [Figure] a variable after update step, g θ

Original: D= θ D − α ∂f ( z ; θ D,θ G)
Revised: D θ D − α ∂f ( z ; θ D G

Original: ∂θ D .
Revised: ∂θ D .

Original: p ( x ) and p ( z ) denote the data 
Revised: p ( x ) and p ( z ) denote the data distribution and source distribution respectively.

Original: [Figure] [Figure]
Revised: E p ( x ) [ f ( x )] indicates taking the expectation of function f ( x ) over the distribution p ( x ) .

Original: 2. G ENERATIVE A DVERSARIAL N ETS
Revised: 2. G ENERATIVE A DVERSARIAL N ETS

Original: A GAN consists of a generator that generates image x = G ( z ; θ G ) from a latent source z ∼ p
Revised: A GAN consists of a generator that generates image x = G ( z ; θ G ) from a latent source z ∼ p

Original: ( z ) , and a discriminator that scores the generated images as D ( x ; θ D )
Revised: ( z ) , and a discriminator that scores the generated images as D ( x ; θ D )

Original: (Goodfellow et al., 2014).
Revised: (Goodfellow et al., 2014).

Original: Training
Revised: Training

Original: GANs involves an adversarial game: while the discriminator tries to distinguish generated samples x = G ( z ; θ G ) from data x ∼ p ( x ) , the generator tries to fool the discriminator.
Revised: GANs involves an adversarial game: while the discriminator tries to distinguish generated samples x = G ( z ; θ G ) from data x ∼ p ( x ) , the generator tries to fool the discriminator.

Original: This procedure can be summarised as the following min-max game:
Revised: This procedure can be summarised as the following min-max game:

Original: [Equation]
Revised: [Equation]

Original:  The exact form of h ( · ) depends on the choice of loss function (Goodfellow et al., 2014; Arjovsky et al., 2017;
Revised: [Equation] [Table] The exact form of h ( · ) depends on the choice of loss function (Goodfellow et al., 2014; Arjovsky et al., 2017;

Original: Nowozin et al., 2016).
Revised: Nowozin et al., 2016).

Original: To simplify our presentation and analysis, we use the Wasserstein[Table] loss (Arjovsky et al., 2017), so that h D ( t ) = − t and h G ( t ) =
Revised: To simplify our presentation and analysis, we use the Wasserstein loss (Arjovsky et al., 2017), so that h D ( t ) = − t and h G ( t ) =

Original: t .
Revised: t .

Original: Our experiments with BigGANdeep uses the hinge loss (Lim & Ye, 2017; Tran et al., 2017), which is identical to this form in its linear regime.
Revised: Our experiments with BigGANdeep uses the hinge loss (Lim & Ye, 2017; Tran et al., 2017), which is identical to this form in its linear regime.

Original: Our analysis can be generalised to other losses as in previous theoretical work (e.g.,
Revised: Our analysis can be generalised to other losses as in previous theoretical work (e.g.,

Original: Arora et al. 2017).
Revised: Arora et al. 2017).

Original: To simplify notation, we abbreviate f ( z ; θ D , θ G )
Revised: To simplify notation, we abbreviate f ( z ; θ D , θ G )

Original: =
Revised: =

Original: D ( G ( z ; θ G ) ; θ D ) , which may be further simpliﬁed as f ( z ) when the explicit dependency on θ D and θ G can be omitted.
Revised: D ( G ( z ; θ G ) ; θ D ) , which may be further simpliﬁed as f ( z ) when the explicit dependency on θ D and θ G can be omitted.

Original: Training GANs requires carefully balancing updates to D and G , and is sensitive to both architecture and algorithm choices (Salimans et al., 2016;
Revised: Training GANs requires carefully balancing updates to D and G , and is sensitive to both architecture and algorithm choices (Salimans et al., 2016;

Original: Radford et al., 2015).
Revised: Radford et al., 2015).

Original: A recent milestone is BigGAN (and BigGAN-deep, Brock et al. 2018), which pushed the boundary of high ﬁdelity image generation by scaling up GANs to an unprecedented level.
Revised: A recent milestone is BigGAN (and BigGAN-deep, Brock et al. 2018), which pushed the boundary of high ﬁdelity image generation by scaling up GANs to an unprecedented level.

Original: BigGANs use an architecture based on residual blocks (He et al., 2016), in combination with regularisation mechanisms and self-attention (Saxe et al., 2014; Miyato et al., 2018; Zhang et al., 2019).
Revised: BigGANs use an architecture based on residual blocks (He et al., 2016), in combination with regularisation mechanisms and self-attention (Saxe et al., 2014; Miyato et al., 2018; Zhang et al., 2019).

Original: Here we aim to improve the adversarial dynamics during training.
Revised: Here we aim to improve the adversarial dynamics during training.

Original: We focus on the second term in eq. 1 which is at the heart of the min-max game, with adversarial losses for D and G , which can be written as
Revised: We focus on the second term in eq. 1 which is at the heart of the min-max game, with adversarial losses for D and G , which can be written as

Original: [Equation]
Revised: [Equation]

Original: Computing the gradients with respect to θ D and θ G obtains the following gradient, which cannot be expressed as the gradient of any single function (Balduzzi et al., 2018):
Revised: Computing the gradients with respect to θ D and θ G obtains the following gradient, which cannot be expressed as the gradient of any single function (Balduzzi et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: The fact that g is not the gradient of a function implies that gradient updates in GANs can exhibit cycling behaviour which can slow down or prevent convergence.
Revised: The fact that g is not the gradient of a function implies that gradient updates in GANs can exhibit cycling behaviour which can slow down or prevent convergence.

Original: In Balduzzi et al.
Revised: In Balduzzi et al.

Original: (2018), vector ﬁelds of this form are referred to as the simultaneous gradient .
Revised: (2018), vector ﬁelds of this form are referred to as the simultaneous gradient .

Original: Although many GAN models use alternating update rules (e.g., Goodfellow et al. 2014;
Revised: Although many GAN models use alternating update rules (e.g., Goodfellow et al. 2014;

Original: Brock et al. 2018), following the gradient with respect to θ D and θ G alternatively in each step, they can still suffer from cycling, so we use the simpler simultaneous gradient (eq. 3) for our analysis.
Revised: Brock et al. 2018), following the gradient with respect to θ D and θ G alternatively in each step, they can still suffer from cycling, so we use the simpler simultaneous gradient (eq. 3) for our analysis.

Original: 2.  SYMPLECTIC G RADIENT A DJUSTMENT
Revised: 2. L ATENT O PTIMISED GAN S

Original: Balduzzi et al. (2018) proposed Symplectic Gradient Adjustment (SGA) to correct the gradient for 
Revised: Inspired by compressed sensing (Candes et al., 2006; Donoho, 2006), Wu et al. introduced latent optimisation for GANs.

Original: Hamiltonian games.
Revised: We call this type of model latent-optimised GANs (LOGAN). Latent optimization has been shown to improve the stability of training as well as the ﬁnal performance for medium-sized models such as DCGANs and Spectral Normalised GANs (Radford et al., 2015; Miyato et al., 2018). Latent optimisation exploits knowledge from D to guide updates of z .

Original: For a game with gradient g (eq. 3), the Hessian is the second order derivatives with respect to the parameters, H = ∇ θ g .
Revised: Intuitively, the gradient ∇ z f ( z ) = ∂f ( z ) ∂z points in the direction that satisﬁes the discriminator D , which implies better samples. Therefore, instead of using the randomly sampled z ∼ p ( z ) , Wu et al.

Original: SGA uses the adjusted gradient
Revised: (2019) uses the optimised latent

Original: [Equation]
Revised: [Equation]

Original: where λ is a positive constant (4) and A = 12 ( H − H T ) is the anti-symmetric component that can cause the updates to rotate or cycle.
Revised: in eq. 1 for training 1 .

Original: Applying SGA to GANs results in the adjusted gradient (see Appendix B.1 for details):
Revised: The general algorithm is summarised in Algorithm 1 and illustrated in Figurea

Original: [Equation] Compared with g in eq. 3, the adjusted gradient g ∗ has second-order terms reﬂecting the interactions between D and G . SGA is expensive to scale due to need to compute second-order derivatives with respect to all parameters, but it has been shown to signiﬁcantly improve GAN training (Balduzzi et al., 2018), allowing faster and more robust convergence to stable ﬁxed points (local Nash equilibria).
Revised: We develop the natural gradient descent form of latent update in Section

Original: [Figure]
Revised: [Figure]

Original: Input: data distribution p ( x ) , latent distribution p ( z ) , D ( · ; θ D ) , G ( · ; θ G ) , learning rate α , batch size N repeat
Revised: Input: data distribution p ( x ) , latent distribution p ( z ) , D ( · ; θ D ) , G ( · ; θ G ) , learning rate α , batch size N repeat

Original: Initialise discriminator and generator parameters θ D , θ G for i = 1 to N do
Revised: Initialise discriminator and generator parameters θ D , θ G for i = 1 to N do

Original: Sample z ∼ p ( z ) , x ∼ p ( x )
Revised: Sample z ∼ p ( z ) , x ∼ p ( x )

Original: Compute the gradient ∂D ( G ( z ))
Revised: Compute the gradient ∂D ( G ( z ))

Original: ∂z and use it to obtain ∆ z from eq.
Revised: ∂z and use it to obtain ∆ z from eq.

Original: 6 (GD) or eq.
Revised: 4 (GD) or eq.

Original: 12 (NGD)
Revised: 12 (NGD)

Original: Optimise the latent z (cid:48) ←  z + [∆ z ] , [ · ] indicates clipping the value between − 1 and
Revised: Optimise the latent z (cid:48) ← [ z + ∆ z ] , [ · ] indicates clipping the value between − 1 and

Original: Compute generator loss L ( i ) G = − D ( G ( z (cid:48) ))
Revised: Compute generator loss L ( i ) G = − D ( G ( z (cid:48) ))

Original: Compute discriminator loss L ( i ) D
Revised: Compute discriminator loss L ( i ) D

Original: = D ( G ( z (cid:48) ))
Revised: = D ( G ( z (cid:48) ))

Original: − D ( x ) end for
Revised: − D ( x ) end for

Original: Compute batch losses L G = 1 N (cid:80) N i =1{+Ni =
+} L ( i ) G and L D = 1 N (cid:80) N i =1 L ( i ) D
Revised: Compute batch losses L G = 1 N (cid:80) {+Ni =
+} L ( i ) G and L D = 1 N (cid:80) Ni =

Original:  Update θ D and θ G with the gradients ∂L D ∂θ D , ∂L G ∂θ G until reaches the maximum training steps
Revised: 1 L ( i ) D Update θ D and θ G with the gradients ∂L D ∂θ D , ∂L G ∂θ G until reaches the maximum training steps

Original:  L ATENT O PTIMISED GAN S
Revised:  A NALYSIS OF THE A LGORITHM

Original: Inspired by compressed sensing (Candes et al., 2006; Donoho, 2006), Wu et al. introduced latent optimisation for GANs. We call this type of model latent-optimised GANs (LOGAN).
Revised: To understand how latent optimisation improves GAN training, we analyse LOGAN as a 2-player differentiable game following Balduzzi et al.

Original: Latent optimization has been shown to improve the stability of training as well as the ﬁnal performance for medium-sized models such as DCGANs and Spectral Normalised GANs (Radford et al., 2015; Miyato et al., 2018). Latent optimisation exploits knowledge from D to guide updates of z . Intuitively, the gradient ∇ z f ( z ) =
Revised: ; Gemp & Mahadevan (2018); Letcher et al.

Original: ∂f ( z ) ∂z points in the direction that satisﬁes the discriminator D , which implies better samples. Therefore, instead of using the randomly sampled z ∼ p ( z ) , Wu et al.
Revised: The appendix provides a complementary analysis that relates LOGAN to unrolled GANs (Metz et al., 2016) and stochastic approximation (Heusel et al., 2017; Borkar, 1997).

Original: (2019) uses the optimised latent [Equation]
Revised: 3. T HE S YMPLECTIC G RADIENT A DJUSTMENT (SGA)

Original:  in eq. 1 for training 1 . The general algorithm is summarised in Algorithm 1 and illustrated in Figurea . We develop the natural gradient descent form of latent update in Section 4.
Revised: An important problem with gradient-based optimization in GANs is that the vector-ﬁeld generated by the losses of the discriminator and generator is not a gradient vector ﬁeld.

Original: To understand why latent optimisation improves GAN training, we analyse LOGAN from the perspectives of 2-player differentiable games (Balduzzi et al., 2018;
Revised: It follows that gradient descent is not guaranteed to ﬁnd a local optimum and can cycle, which can slow down convergence or lead to phenomena like mode collapse and mode hopping.

Original: Letcher et al., 2019) and stochastic approximation (Heusel et al., 2017; Borkar, 1997).
Revised: Balduzzi et al. ; Gemp &

Original: We ﬁrst write the gradients for D and G at z (cid:48) , after one step of latent optimisation (eq. 6):
Revised: Mahadevan (2018) proposed Symplectic Gradient Adjustment (SGA) to improve the dynamics of gradient-based methods in adversarial games. For a game with gradient g (eq. 3), deﬁne the Hessian as the second order derivatives with respect to the parameters, H = ∇ θ g . SGA uses the adjusted gradient [Equation]

Revised: where λ is a positive constant

Revised: [Equation]

Revised: 5) and A = 12 ( H − H T ) is the anti-symmetric component of the Hessian.

Revised: Applying SGA to GANs yields the adjusted updates (see Appendix B.1 for details):

Revised: [Equation]

Revised: Compared with g in eq. 3, the adjusted gradient g ∗ has second-order terms reﬂecting the interactions between D and G .

Revised: SGA has been shown to signiﬁcantly improve GAN training in basic examples (Balduzzi et al., 2018), allowing faster and more robust convergence to stable ﬁxed points (local

Revised: Nash equilibria).

Revised: Unfortunately, SGA is expensive to scale because computing the second-order derivatives with respect to all parameters is expensive.

Revised: Explicitly computing the gradients for the discriminator and generator at z (cid:48) after one step of latent optimisation (eq. 4) obtains

Original: [Equation]
Revised: [Equation]

Original: In both equations, the ﬁrst terms represent how f ( z (cid:48) ) depends on the parameters directly and the second terms represent how f ( z (cid:48) ) depends on the parameters via the optimised latent source.
Revised: In both equations, the ﬁrst terms represent how f ( z (cid:48) ) depends on the parameters directly and the second terms represent how f ( z (cid:48) ) depends on the parameters via the optimised latent source.

Original: For the last equality, we substitute ∆ z = α ∂f ( z ) ∂z as the gradient-based update of z .
Revised: For the second equality, we substitute ∆ z = α ∂f ( z ) ∂z as the gradient-based update of z and use ∂f ( z (cid:48) ) ∂ ∆ z = ∂f ( z (cid:48) )

Original:  The original GAN’s gradient (eq. 3) does not include any second-order term, since ∆ z = 0 without latent optimisation.
Revised: ∂z (cid:48) . The original GAN’s gradient (eq. 3) does not include any second-order term, since ∆ z =without latent optimisation.

Original: In LOGAN, these extra terms are computed by automatic differentiation when back-propagating through the latent optimisation process (see Algorithm 1).
Revised: In LOGAN, these extra terms are computed by automatic differentiation when back-propagating through the latent optimisation process (see Algorithm 1).

Original: We focus on how these extra terms affect training dynamics, andoffer new insights into the closely related work of Unrolled GANs (Metz et al., 2016).
Revised: The SGA updates in eq. 6 and the LOGAN updates in eq. 8 are strikingly similar, suggesting that the latent step used by LOGAN reduces the negative effects of cycling by introducing a symplectic gradient adjustment into the optimization procedure.

Original: Detailed derivations are presented in Appendix B. As an overview, our analysis shows that the second-order terms in eq.
Revised: The role of the latent step can be formalized in terms of a third player, whose goal is to help the generator, see appendix B for details.

Original: 7 and 8 approximate the more expensive second-order terms used in SGA, recall eq. 5. Our main ﬁndings are: Latent optimisation approximates SGA using only second-order derivatives with respect to the latent z and parameters.
Revised: Crucially, latent optimisation approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately . The second order terms involving parameters of both the discriminator and the generator – which are extremely expensive to compute – are not used.

Original: For  z ’s with dimensions typically used in GANs (e g., 128–256, orders of magnitude less than the number of parameters), these can be computed efﬁciently.
Revised: For latent z ’s with dimensions typically used in GANs (e.g., 128–256, orders of magnitude less than the number of parameters), these can be computed efﬁciently.

Original: Latent optimisation accelerates the speed of updating D relative to that of G , facilitating convergence according to Heusel et al.
Revised: In short, latent optimisation efﬁciently couples the gradients of the discriminator and generator, as prescribed by SGA, but using the much lower-dimensional latent source z which makes the adjustment scalable.

Original: (2017) (see also Figure 4 b ).
Revised: An important consequence of reducing the rotational aspect of GAN dynamics is that it is possible to use larger step sizes during training which suggests using stronger optimisers to fully take advantage of latent optimisation.

Original: Thus, latent optimisation can improve GAN training dynamics further by taking a larger single step ∆ z towards the direction of ∂f ( z ) ∂z without overshooting .
Revised: Latent optimisation can improve GAN training dynamics further by allowing larger single steps ∆ z towards the direction of ∂f ( z ) ∂z without overshooting .

Original: The intuition behind 1 is that latent optimisation efﬁciently couples the gradients of  D and G , as presribed by SGA but using the much lower-dimensional latent source z which makes the adjustment scalable.
Revised: Appendix B further explains how LOGAN relates to unrolled GANs (Metz et al., 2016) and stochastic approximation. Our main ﬁnding is that latent optimisation accelerates the speed of updating D relative to that of G , facilitating convergence according to Heusel et al. (2017) (see also Figure 4 b ).

Original: Point 2 can be understood as G requires less update compared with D to achieve the same reduction of loss, because latent optimisation “helps” G .
Revised: In particular, the generator requires less update compared with D to achieve the same reduction of loss, because latent optimisation “helps” G .

Original: Finally, point 3 suggests using stronger optimisers to fully take advantage of latent optimisation.
Revised: [Figure]

Original:  LOGAN WITH N ATURAL G RADIENT D ESCENT
Revised:  LOGAN WITH N ATURAL G RADIENT D ESCENT

Original: Although our analysis suggests using strong optimisers for optimising z , Wu et al.
Revised: Although our analysis suggests using strong optimisers for optimising z , Wu et al.

Original: (2019) only used basic gradient descent (GD) with a ﬁxed step-size.
Revised: (2019) only used basic gradient descent (GD) with a ﬁxed step-size.

Original: This choice limits the size ∆ z can take: in order not to overshoot when the curvature is large, the step size would be too conservative when the curvature is small.
Revised: This choice limits the size ∆ z can take: in order not to overshoot when the curvature is large, the step size would be too conservative when the curvature is small.

Original: We hypothesis that GD is more detrimental for larger models, which have complex loss surfaces with highly varying curvatures.
Revised: We hypothesis that GD is more detrimental for larger models, which have complex loss surfaces with highly varying curvatures.

Original: Consistent with this hypothesis, we observed only marginal improvement over the baseline using GD (section 5.3, Table 1, Figure 3 b ).
Revised: Consistent with this hypothesis, we observed only marginal improvement over the baseline using GD (section 5.3, Table 1, Figure 3 b ).

Original: In this work, we instead use natural gradient descent (NGD, Amari 1998) for latent optimisation.
Revised: In this work, we instead use natural gradient descent (NGD, Amari 1998) for latent optimisation.

Original: NGD can be seen as an approximate second-order optimisation method (Pascanu & Bengio, 2013;
Revised: NGD can be seen as an approximate second-order optimisation method (Pascanu & Bengio, 2013;

Original: Martens, 2014), and has been applied successfully in many domains.
Revised: Martens, 2014), and has been applied successfully in many domains.

Original: By using the positive semideﬁnite (PSD) Gauss-Newton matrix to approximate the (possibly negative deﬁnite) Hessian, NGD often works even better than exact second-order methods.
Revised: By using the positive semideﬁnite (PSD) Gauss-Newton matrix to approximate the (possibly negative deﬁnite) Hessian, NGD often works even better than exact second-order methods.

Original: NGD is expensive in high dimensional parameter spaces, even with approximations (Martens, 2014).
Revised: NGD is expensive in high dimensional parameter spaces, even with approximations (Martens, 2014).

Original: However, we demonstrate it is efﬁcient for latent optimisation even in very large models.[Figure]
Revised: However, we demonstrate it is efﬁcient for latent optimisation, even in very large models.

Original: Given the gradient of z , g = ∂f ( z ) ∂z , NGD computes the update as
Revised: Given the gradient of z , g = ∂f ( z ) ∂z , NGD computes the update as

Original: [Equation]
Revised: [Equation]

Original: where the Fisher information matrix F is deﬁned as
Revised: where the Fisher information matrix F is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The log-likelihood function ln p ( t | z ) typically corresponds to commonly used error functions such as cross entropy loss.
Revised: The log-likelihood function ln p ( t | z ) typically corresponds to commonly used error functions such as cross entropy loss.

Original: Although this correspondence is less obvious for the hinge loss commonly used in GANs (Lim & Ye, 2017; Tran et al., 2017), Appendix C shows that it can be interpreted as a Poisson log-likelihood function.
Revised: This correspondence is not necessary when NGD is interpreted as an approximate second-order method, as has long been used in practice (Martens, 2014). Nevertheless, Appendix C provides a Poisson log-likelihood interpretation for the hinge loss commonly used in GANs (Lim & Ye, 2017; Tran et al., 2017).

Original: An important difference between latent optimisation and commonly seen senarios using NGD is that the expectation over the condition ( z ) is absent.
Revised: An important difference between latent optimisation and commonly seen senarios using NGD is that the expectation over the condition ( z ) is absent.

Original: Since each z is only responsible for generating one image, it only minimises the loss L G ( z ) for this particular instance.
Revised: Since each z is only responsible for generating one image, it only minimises the loss L G ( z ) for this particular instance.

Original: Computing per-sample Fisher this way is necessary for approximating SGA (see
Revised: Computing per-sample Fisher this way is necessary to approximate SGA (see

Original: Appendix B.1 for details).
Revised: Appendix B.1 for details).

Original: More speciﬁcally, we use the empirical Fisher F (cid:48) with Tikhonov damping, as in TONGA (Roux et al., 2008)
Revised: More speciﬁcally, we use the empirical Fisher F (cid:48) with Tikhonov damping, as in TONGA (Roux et al., 2008)

Original: [Equation]
Revised: [Equation]

Original: F (cid:48) is cheaper to compute compared with the full Fisher, since g is already available.
Revised: F (cid:48) is cheaper to compute compared with the full Fisher, since g is already available.

Original: The damping factor β regularises the step size, which is important when F (cid:48) only poorly approximates the Hessian  or when the Hessian changes too much across the step.
Revised: The damping factor β regularises the step size, which is important when F (cid:48) only poorly approximates the Hessian [Figure] instances with stop or when the Hessian changes too much across the step.

Original: Using the Sherman-Morrison formula, the
Revised: Using the Sherman-Morrison formula, the

Original: NGD update can be simpliﬁed into the following closed form:
Revised: NGD update can be simpliﬁed into the following closed form:

Original: [Equation]
Revised: [Equation]

Original: which does not involve any matrix inversion.
Revised: which does not involve any matrix inversion.

Original: Here NGD adapts the step size according to the curvature estimate c = 
Revised: Thus, NGD adapts the step size according to the curvature estimate c = 1 β (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:16)
Revised: − (cid:107) g

Original: [Equation]
Revised: [Equation]

Original:  Figure 4 a illustrates the scaling for different values of β .Here NGD automatically smooths the scale of updates by down-scaling the gradients as their norm grow, which also contributes to the smoothed norms of updates (Figure 4 b ).
Revised: g (cid:107) 2 (cid:17) . Figure 4 a illustrates the scaling for different values of β . NGD automatically smooths the scale of updates by down-scaling the gradients as their norm grows, which also contributes to the smoothed norms of updates (Figure 4 b ).

Original: Since the NGD update remains proportional to g , our analysis based on gradient descent in section 3 still holds.
Revised: Since the NGD update remains proportional to g , our analysis based on gradient descent in section 3 still holds.

Original:  E XPERIMENTS AND A NALYSIS
Revised:  E XPERIMENTS AND A NALYSIS

Original: We focus on large scale GANs based on BigGAN-deep (Brock et al., 2018) trained on 128 ×size images from the ImageNet dataset (Deng et al., 2009).
Revised: We focus on large scale GANs based on BigGAN-deep (Brock et al., 2018) trained on 128 ×size images from the ImageNet dataset (Deng et al., 2009).

Original: In Appendix E, we present results from applying our algorithm on Spectral Normalised GANs trained with CIFAR dataset (Krizhevsky et al., 2009), which obtains state-of-the-art scores on this model.
Revised: In Appendix E, we present results from applying our algorithm on Spectral Normalised GANs trained with CIFAR dataset (Krizhevsky et al., 2009), which obtains state-of-the-art scores on this model.

Original: [Figure]
Revised: [Equation]

Original: All instances with stop gradient collapsed (FID went up) early training.

Original: 5. M ODEL C ONFIGURATION

Original: We used the standard BigGAN-deep architecture with three minor modiﬁcations: 1.
Revised: We used the standard BigGAN-deep architecture with three minor modiﬁcations: 1.

Original: We increased the size of the latent source from 128 to 256 , to compensate the randomness of the source lost when optimising z . 2.
Revised: We increased the size of the latent source from 128 to 256 , to compensate the randomness of the source lost when optimising z . 2.

Original: We use the uniform distribution U ( − 1 , 1) instead of the standard normal distribution N (0 , 1) for p ( z ) , to be consistent with the clipping operation (Algorithm 1).
Revised: We use the uniform distribution U ( − 1 , 1) instead of the standard normal distribution N (0 , 1) for p ( z ) , to be consistent with the clipping operation (Algorithm 1).

Original: We use leaky ReLU instead of ReLU as the non-linearity for smoother gradient ﬂow for ∂f ( z ) ∂z .
Revised: We use leaky ReLU instead of ReLU as the non-linearity for smoother gradient ﬂow for ∂f ( z ) ∂z .

Original: Consistent with detailed ﬁndings in Brock et al.
Revised: Consistent with detailed ﬁndings in Brock et al.

Original: (2018) that these changes have limited effect, our experiment with this baseline model obtains only slightly better scores compared with those in Brock et al.
Revised: (2018) that these changes have limited effect, our experiment with this baseline model obtains only slightly better scores compared with those in Brock et al.

Original: (Table 1, see also Figure 8).
Revised: (Table 1, see also Figure 8).

Original: The FID and IS are computed as in Brock et al.
Revised: The FID and IS are computed as in Brock et al.

Original: (2018), and
Revised: (2018), and

Original: IS values are computed from checkpoints with the lowest FIDs.
Revised: IS values are computed from checkpoints with the lowest FIDs.

Original: The means and standard deviations are computed from 5 models with different random seeds.
Revised: The means and standard deviations are computed from 5 models with different random seeds.

Original: To apply latent optimisation, we use a damping factor β = 5 . 

Revised: To apply latent optimisation, we use a damping factor β = 5 .

Original: combined with a large step size of α = 0 .
Revised: combined with a large step size of α = 0 .

Original: As an additional way of damping, we only optimise 50% of z ’s dimensions.
Revised: As an additional way of damping, we only optimise 50% of z ’s dimensions.

Original: Optimising the entire population of z was unstable in our experiments.
Revised: Optimising the entire population of z was unstable in our experiments.

Original: Similar to Wu et al.
Revised: Similar to Wu et al.

Original: (2019), we found it was helpful to regularise the Euclidean norm of weight-change ∆ z , with a regulariser weight of300 . 0 .
Revised: (2019), we found it was helpful to regularise the Euclidean norm of weight-change ∆ z , with a regulariser weight of

Original: All other hyper-parameters, including learning rates and a large batch size of 2048, remain the same as in BigGAN-deep; we did not optimise these hyper-parameters.
Revised: All other hyper-parameters, including learning rates and a large batch size of 2048, remain the same as in BigGAN-deep; we did not optimise these hyper-parameters.

Original: We call this model
Revised: We call this model

Original: LOGAN (NGD).
Revised: LOGAN (NGD).

Original: 5. B ASIC R ESULTS
Revised: 5. B ASIC R ESULTS

Original: Employing the same architecture and number of parameters as the BigGAN-deep baseline, LOGAN (NGD) achieved better FID and IS (Table 1).
Revised: Employing the same architecture and number of parameters as the BigGAN-deep baseline, LOGAN (NGD) achieved better FID and IS (Table 1).

Original: As observed by Brock et al.
Revised: As observed by Brock et al.

Original: (2018), BigGAN training always eventually collapsed.
Revised: (2018), BigGAN training always eventually collapsed.

Original: Training with LOGAN also collapsed, perhaps due to higher-order dynamics beyond the scope we have analysed, but took signiﬁcantly longer (600k steps versus 300k steps with BigGAN-deep).
Revised: Training with LOGAN also collapsed, perhaps due to higher-order dynamics beyond the scope we have analysed, but took signiﬁcantly longer (600k steps versus 300k steps with BigGAN-deep).

Original: During training, LOGAN was 2 − 3 times slower per step compared with BigGAN-deep because of the additional forward and backward pass.
Revised: During training, LOGAN was 2 − 3 times slower per step compared with BigGAN-deep because of the additional forward and backward pass.

Original: We found that optimising z during evaluation did not improve sample scores (even up to 10 steps), so we do not optimise z for evaluation.
Revised: We found that optimising z during evaluation did not improve sample scores (even up to 10 steps), so we do not optimise z for evaluation.

Original: Therefore, LOGAN has the same evaluation cost as original BigGAN-deep.
Revised: Therefore, LOGAN has the same evaluation cost as original BigGAN-deep.

Original: To help understand this behaviour, we plot the change from ∆ z during training in Figure 5 a .
Revised: To help understand this behaviour, we plot the change from ∆ z during training in Figure 5 a .

Original: Although the movement in
Revised: Although the movement in

Original: Euclidean space (cid:107) ∆ z (cid:107) grew until training collapsed, the movement in D ’s output space, measured as (cid:107)
Revised: Euclidean space (cid:107) ∆ z (cid:107) grew until training collapsed, the movement in D ’s output space, measured as (cid:107)

Original: f
Revised: f

Original: ( z + ∆ z ) − f ( z ) (cid:107) , remained unchanged (see Appendix D for details).
Revised: ( z + ∆ z ) − f ( z ) (cid:107) , remained unchanged (see Appendix D for details).

Original: As shown in our analysis, optimising z improves the training dynamics, so LOGANs work well after training without requiring latent optimisation.
Revised: As shown in our analysis, optimising z improves the training dynamics, so LOGANs work well after training without requiring latent optimisation.

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: We verify our theoretical analysis in section 3 by examining key components of Algorithm 1 via ablation studies.
Revised: We verify our theoretical analysis in section 3 by examining key components of Algorithm 1 via ablation studies.

Original: First, we experimented with using basic GD to optimising z , as in Wu et al.
Revised: First, we experimented with using basic GD to optimising z , as in Wu et al.

Original: (2019), and call this model LOGAN (GD).
Revised: (2019), and call this model LOGAN (GD).

Original: A smaller step size of α = 0 . 0001 was required; larger values were unstable and led to premature collapse of training.
Revised: A smaller step size of α = 0 . 0001 was required; larger values were unstable and led to premature collapse of training.

Original: As shown in Table 1, the scores from
Revised: As shown in Table 1, the scores from

Original: LOGAN (GD) were worse than LOGAN (NGD) and similar to the baseline model.
Revised: LOGAN (GD) were worse than LOGAN (NGD) and similar to the baseline model.

Original: We then evaluate the effects of removing those terms depending on ∂f ( z ) ∂z in eq.7 and 8, which are not in the ordinary gradient (eq. 3).
Revised: We then evaluate the effects of removing those terms depending on ∂f ( z ) ∂z in eq. 8, which are not in the ordinary gradient (eq. 3).

Original: Since these terms were computed when backpropagating through the latent optimisation procedure, we removed them by selectively blocking back-propagation with “ stop gradient ” operations (e.g., in TensorFlow Abadi et al. 2016).
Revised: Since these terms were computed when back-propagating through the latent optimisation procedure, we removed them by selectively blocking back-propagation with “ stop gradient ” operations (e.g., in TensorFlow Abadi et al. 2016).

Original: Figure 5 b shows the change of FIDs for the three models corresponding to removing (cid:16) [Table]
Revised: Figure 5 b shows the change of FIDs for the three models corresponding to removing (cid:16) ∂ ∆ z ∂θ G (cid:17) T ∂f ( z (cid:48) )

Original:  , removing (cid:16) ∂ ∆ z ∂θ D (cid:17) T ∂f ( z (cid:48) ) ∂z (cid:48) and removing both terms.
Revised: ∂z (cid:48) , removing (cid:16) ∂ ∆ z ∂θ D (cid:17) T ∂f ( z (cid:48) ) ∂z (cid:48) and removing both terms.

Original: As predicted by our analysis (section 3), both terms help stabilise training; training diverged early for all three ablations.
Revised: As predicted by our analysis (section 3), both terms help stabilise training; training diverged early for all three ablations.

Original: 5. T RUNCATION AND S AMPLES
Revised: 5. T RUNCATION AND S AMPLES

Original: Truncation is a technique introduced by Brock et al.
Revised: Truncation is a technique introduced by Brock et al.

Original: (2018) to illustrate the trade-off between the
Revised: (2018) to illustrate the trade-off between the

Original: FID and IS in a trained model.
Revised: FID and IS in a trained model.

Original: For a model trained with z ∼ p ( z ) from a source distribution symmetric around 0 , such as the standard normal distribution N (0 , 1) and the uniform distribution
Revised: For a model trained with z ∼ p ( z ) from a source distribution symmetric around 0 , such as the standard normal distribution N (0 , 1) and the uniform distribution

Original: U ( − 1 , 1) , down-scaling (truncating) the source ¯ z = s · z with 0 ≤ s ≤ 1 gives samples with higher visual quality but reduced diversity.
Revised: U ( − 1 , 1) , down-scaling (truncating) the source ¯ z = s · z with 0 ≤ s ≤ 1 gives samples with higher visual quality but reduced diversity.

Original: This observation is quantiﬁed as higher IS and lower FID when evaluating samples from truncated distributions.
Revised: This observation is quantiﬁed as higher IS and lower FID when evaluating samples from truncated distributions.

Original: Figure 3 b plots the truncation curves for the baseline BigGAN-deep model, LOGAN (GD) and
Revised: Figure 3 b plots the truncation curves for the baseline BigGAN-deep model, LOGAN (GD) and

Original: LOGAN (NGD), obtained by varying the truncation (value of s ) from 1 . 

Revised: LOGAN (NGD), obtained by varying the truncation (value of s ) from 1 .

Original: (no truncation, upper-left ends of the curves) to 0 .
Revised: (no truncation, upper-left ends of the curves) to 0 .

Original: 02 (extreme truncation, bottom-right ends).
Revised: 02 (extreme truncation, bottom-right ends).

Original: Each curve shows the trade-off between FID and IS for an individual model; curves towards the upper-right corner indicate better overall sample quality.
Revised: Each curve shows the trade-off between FID and IS for an individual model; curves towards the upper-right corner indicate better overall sample quality.

Original: The relative positions of curves in Figure 3 ( b ) shows LOGAN (NGD) has the best sample quality.
Revised: The relative positions of curves in Figure 3 ( b ) shows LOGAN (NGD) has the best sample quality.

Original: Interestingly, although LOGAN (GD) and the baseline model have similar scores without truncation (upper-left ends of the curves, see also Table 1), LOGAN (GD) was better behaved with increasing truncation, suggesting LOGAN (GD) still converged to a better equilibrium.
Revised: Interestingly, although LOGAN (GD) and the baseline model have similar scores without truncation (upper-left ends of the curves, see also Table 1), LOGAN (GD) was better behaved with increasing truncation, suggesting LOGAN (GD) still converged to a better equilibrium.

Original: For further reference, we plot truncation curves from additional baseline models in Figure 8.
Revised: For further reference, we plot truncation curves from additional baseline models in Figure 8.

Original: Figure 1 and Figure 2 show samples from chosen points on the truncation curves.
Revised: Figure 1 and Figure 2 show samples from chosen points on the truncation curves.

Original: In the high IS domain, C and D on the truncation curves both have similarly high IS of near 260.
Revised: In the high IS domain, C and D on the truncation curves both have similarly high IS of near 260.

Original: Samples from batches with such high IS have almost photo-realistic image quality.
Revised: Samples from batches with such high IS have almost photo-realistic image quality.

Original: Figure 1 show that while the baseline model produced nearly uniform samples, LOGAN (NGD) could still generate highly diverse samples.
Revised: Figure 1 show that while the baseline model produced nearly uniform samples, LOGAN (NGD) could still generate highly diverse samples.

Original: On the other hand, A and B from Figure 3 b have similarly low FID of near 5, indicating high sample diversity.
Revised: On the other hand, A and B from Figure 3 b have similarly low FID of near 5, indicating high sample diversity.

Original: Samples in Figure 2 b show higher quality compared with those in a (e.g., the interfaces between the elephants and ground, the contours around the pandas).
Revised: Samples in Figure 2 b show higher quality compared with those in a (e.g., the interfaces between the elephants and ground, the contours around the pandas).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we present the LOGAN model which signiﬁcantly improves the state-of-the-art on large scale GAN training for image generation by online optimising the latent source z .
Revised: In this work we present the LOGAN model which signiﬁcantly improves the state-of-the-art on large scale GAN training for image generation by online optimising the latent source z .

Original: Our results illustrate improvements in quantitative evaluation and samples with higher quality and diversity.
Revised: Our results illustrate improvements in quantitative evaluation and samples with higher quality and diversity.

Original: Moreover, our analysis suggests that LOGAN fundamentally improves adversarial training dynamics.
Revised: Moreover, our analysis suggests that LOGAN fundamentally improves adversarial training dynamics.

Original: We therefore expect our method to be useful in other tasks that involve adversarial training, including representation learning and inference (Donahue et al., 2017;
Revised: We therefore expect our method to be useful in other tasks that involve adversarial training, including representation learning and inference (Donahue et al., 2017;

Original: Dumoulin et al., 2017), text generation (Zhang et al., 2019), style learning (Zhu et al., 2017; Karras et al., 2019), audio generation (Donahue et al., 2018) and video generation (Vondrick et al., 2016; Clark et al., 2019).
Revised: Dumoulin et al., 2017), text generation (Zhang et al., 2019), style learning (Zhu et al., 2017; Karras et al., 2019), audio generation (Donahue et al., 2018) and video generation (Vondrick et al., 2016; Clark et al., 2019).

Original: A A DDITIONAL S AMPLES AND R ESULTS
Revised: A A DDITIONAL S AMPLES AND R ESULTS

Original: Figure 6 and 7 provide additional samples, organised similarly as in Figure 1 and 2.
Revised: Figure 6 and 7 provide additional samples, organised similarly as in Figure 1 and 2.

Original: Figure 8 shows additional truncation curves.
Revised: Figure 8 shows additional truncation curves.

Original: B D ETAILED A NALYSIS OF L ATENT O PTIMISATION
Revised: B D ETAILED A NALYSIS OF L ATENT O PTIMISATION

Original: In this section we present three complementary analyses of LOGAN.
Revised: In this section we present three complementary analyses of LOGAN.

Original: In particular, we show how the algorithm brings together ideas from symplectic gradient adjustment, unrolled GANs and stochastic approximation with two time scales.
Revised: In particular, we show how the algorithm brings together ideas from symplectic gradient adjustment, unrolled GANs and stochastic approximation with two time scales

Original:  B. A PPROXIMATE S YMPLECTIC G RADIENT A DJUSTMENT
Revised: . B. A PPROXIMATE S YMPLECTIC G RADIENT A DJUSTMENT

Original: It is useful for our analysis to introduce ∆ z as  a third player in the adversarial game, whose parameter,simply ∆ z , is optimised online for each z ∼ p ( z ) .
Revised: To analyse LOGAN as a differentiable game we treat the latent step ∆ z as adding a third player to the original game played by the discriminator and generator. The third player’s parameter, ∆ z , is optimised online for each z ∼ p ( z ) .

Original: Together the three players ( ∆ z , discriminator,  generator) have the loss averaged over a batch of samples :
Revised: Together the three players (latent player, discriminator, and generator) have losses averaged over a batch of samples :

Original: [Equation]
Revised: [Equation]

Original: [Equation] where η = 1 N ( N is the batch size) reﬂects the fact that each ∆ z is only optimised for a single sample z , so its contribution to the total loss across a batch is small compared with θ D and θ G which are directly optimised for batch losses.
Revised: where η = 1 N ( N is the batch size) reﬂects the fact that each ∆ z is only optimised for a single sample z , so its contribution to the total loss across a batch is small compared with θ D and θ G which are directly optimised for batch losses.

Original: This choice of η is essential for the following derivation, and has important practical implication.
Revised: This choice of η is essential for the following derivation, and has important practical implication.

Original: It means that the per-sample loss L G ( z (cid:48) ) , instead of the loss summed over a batch (cid:80)
Revised: It means that the per-sample loss L G ( z (cid:48) ) , instead of the loss summed over a batch (cid:80)

Original: Nn =
Revised: Nn =

Original: 1 L G ( z (cid:48) n ) , should be the only loss function guiding latent optimisation.
Revised: 1 L G ( z (cid:48) n ) , should be the only loss function guiding latent optimisation.

Original: Therefore, when using natural gradient descent (Section 4), the Fisher information matrix should only be computed using the current sample z .
Revised: Therefore, when using natural gradient descent (Section 4), the Fisher information matrix should only be computed using the current sample z .

Original: The resulting  gradient is
Revised: The resulting simultaneous gradient is

Original: [Equation]
Revised: [Equation]

Original: Following Balduzzi et al.
Revised: Following Balduzzi et al.

Revised: (2018), we can write the Hessian of the game as:

Original: [Equation]
Revised: [Equation]

Original: we can write the Hessian of the game as:
Revised: 

Original: [Equation]
Revised: [Equation]

Original: The presence of a non-zero anti-symmetric component in the Hessian
Revised: The presence of a non-zero anti-symmetric component in the Hessian

Original: [Equation]
Revised: [Equation]

Original:  (16) suggests the dynamics have a rotational component which can cause cycling.
Revised: implies the dynamics have a rotational component which can cause cycling or slow down convergence.

Original: Since η (cid:28) 1 for typical batch sizes (e [Equation] g., 164 for DCGAN and 1 2048 for BigGAN-deep), we abbreviate γ = 1+ η 2η ≈ 1 − η 2η to simplify notations.
Revised: Since η (cid:28) 1 for typical batch sizes (e.g., 164 for DCGAN and 1 2048 for BigGAN-deep), we abbreviate γ = 1+ η 2 ≈ 1 − η 2 to simplify notations.

Original: Symplectic gradient adjustment (SGA) counteracts the cyclic by adding an adjustment term to the gradient to obtain g ∗ ← g
Revised: Symplectic gradient adjustment (SGA) counteracts the rotational force by adding an adjustment term to the gradient to obtain g ∗ ← g

Original: + λ A T g , which for the discriminator and generator has the form:
Revised: + λ A T g , which for the discriminator and generator has the form:

Original: [Equation]
Revised: [Equation]

Original: The gradient with respect to ∆ z is ignored since the convergence of training only depends on θ D and θ G .
Revised: The gradient with respect to ∆ z is ignored since the convergence of training only depends on θ D and θ G .

Original: If we drop the last terms in eq.17 and 18, which are expensive to compute for large models with high-dimensional θ D and θ G , and use ∂f ( z (cid:48)) ∂ ∆ z = ∂f
Revised: If we drop the last terms in eq.17 and 18, which are expensive to compute for large models with high-dimensional θ D and θ G , and use ∂f ( z (cid:48)

Original: [Equation]
Revised: [Equation]

Original:  z [Equation]
Revised: ∆ z = ∂f ( z (cid:48) )

Original: ∂z (cid:48) , the adjusted updates can be rewritten as
Revised: ∂z (cid:48) , the adjusted updates can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Because of the third player, there are still the terms depend on ∂f ( z (cid:48) )
Revised: Because of the third player, there are still the terms depend on ∂f ( z (cid:48) )

Original: ∂z (cid:48) to adjust the gradients.
Revised: ∂z (cid:48) to adjust the gradients.

Original: Efﬁciently computing ∂ 2 f ( z (cid:48) ) 
Revised: Efﬁciently computing ∂ 2 f ( z (cid:48) ) ∂z

Original: ∂z (cid:48)
Revised: [Equation]

Original: ∂θ D and ∂ 2 f ( z(cid:48) )
Revised: ∂θ D and ∂ 2 f [Equation] z

Revised: [Equation]

Original: ∂z (cid:48)
Revised: ∂z (cid:48)

Original: ∂θ D is non-trivial (e.g., Pearlmutter 1994).
Revised: ∂θ D is non-trivial (e.g., Pearlmutter 1994).

Original: However, if we introduce the local approximation
Revised: However, if we introduce the local approximation

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: then the adjusted gradient becomes identical toeq.7 and 8 from latent optimisation.
Revised: then the adjusted gradient becomes identical to 8 from latent optimisation.

Original: In other words, automatic differentiation using commonly used machine learning packages can compute the adjusted gradient for θ D and θ G when back-propagating through the latent optimisation process.
Revised: In other words, automatic differentiation by commonly used machine learning packages can compute the adjusted gradient for θ D and θ G when back-propagating through the latent optimisation process.

Original: Despite the approximation involved in this analysis, both our experiments in section 5 and the results from Wu et al.
Revised: Despite the approximation involved in this analysis, both our experiments in section 5 and the results from Wu et al.

Original: (2019) veriﬁed that latent optimisation can signiﬁcantly improve GAN training.
Revised: (2019) veriﬁed that latent optimisation can signiﬁcantly improve GAN training.

Original: Notice that the direction of ∆ z can be arbitrary merely from the perspective of SGA; however, section B.3 shows that the direction used in latent optimisation, i.e., the same direction as ∂ 2 f ( z ) ∂z∂θ D , helps convergence from the perspective of stochastic approximation.

Original: B. R ELATION WITH U NROLLED GAN S
Revised: B. R ELATION WITH U NROLLED GAN S

Original: Latent optimisation can be seen as unrolling GANs (Metz et al., 2016) in the space of the latent, rather than the parameters.
Revised: Latent optimisation can be seen as unrolling GANs (Metz et al., 2016) in the space of the latent, rather than the parameters.

Original: Unrolling in the latent space has the advantages that:
Revised: Unrolling in the latent space has the advantages that:

Original: LOGAN is more scalable than Unrolled GANs because it avoids second order derivatives over a potentially very large number of parameters.
Revised: LOGAN is more scalable than Unrolled GANs because it avoids second-order derivatives over a potentially very large number of parameters.

Original: While unrolling the update of D only affects the parameters of G (as in Metz et al. 2016), latent optimisation effectson both D and G as shown in eq.7 and 8.
Revised: While unrolling the update of D only affects the parameters of G (as in Metz et al. 2016), latent optimisation effects both D and G as shown in eq.

Original: We next formally present this connection by showing that SGA can be seen as approximating Unrolled GANs (Metz et al., 2016).
Revised: We next formally present this connection by showing that SGA can be seen as approximating Unrolled GANs (Metz et al., 2016).

Original: For the update θ (cid:48) D
Revised: For the update θ (cid:48) D

Revised: [Equation]

Original: = θ D + ∆ θ D , we have the Taylor expansion approximation at θ D :
Revised: θ D [Equation] θ D , we have the Taylor expansion approximation at θ D :

Original: [Equation]
Revised: [Equation]

Revised: Substitute ∆ θ D = − α ∂f ( z ; θ D ,

Original: [Equation]
Revised: [Equation]

Original: We then substitute ∆ θ D = − α ∂f ( z ; θ D ,θ G ) 
Revised: G ) ∂θ [Equation]

Original: ∂θ D , and take the derivatives with respect to θ G on both sides:
Revised: , and take the derivatives with respect to θ G on both sides:

Original: [Equation]
Revised: [Equation]

Original: (23) which is the same as eq.
Revised: which is the same as eq.

Original: 18 (taking the negative sign).
Revised: 18 (taking the negative sign).

Original: Compared with the exact gradient from the unroll:
Revised: Compared with the exact gradient from the unroll:

Original: [Equation]
Revised: [Equation]

Original: The approximation in eq.
Revised: The approximation in eq.

Original: 23 comes from using ∂f ( 
Revised: 23 comes from using ∂f ( z ; θ D ,θ G ) ∂θ D [Equation] and

Original: [Equation]
Revised: [Equation]

Original: as a result of the linear approximation.
Revised: as a result of the linear approximation.

Original: At this point, unrolling D update only affects θ D .
Revised: At this point, unrolling D update only affects θ D .

Original: Although it is expensive to unroll both D and
Revised: Although it is expensive to unroll both D and

Original: G , in principle, we can unroll G update and compute the gradient of θ D similarly using ∆ θ G = [Equation]
Revised: G , in principle, we can unroll G update and compute the gradient of θ D similarly using ∆ θ G = α ∂f ( z ; θ D ,θ G ) ∂θ G :

Original: [Equation]
Revised: [Equation]

Original: which gives us the same update rule as SGA (eq. 17)
Revised: which gives us the same update rule as SGA (eq. 17)

Original: This correspondence based on ﬁrst order
Revised: This correspondence based on ﬁrst order

Original: Taylor expansion is unsurprising, as SGA is based on linearising the adversarial dynamics (Balduzzi et al., 2018).
Revised: Taylor expansion is unsurprising, as SGA is based on linearising the adversarial dynamics (Balduzzi et al., 2018).

Original: B. S TOCHASTIC A PPROXIMATION WITH T WO T IME S CALES
Revised: B. S TOCHASTIC A PPROXIMATION WITH T WO T IME S CALES

Original: Heusel et al.
Revised: Heusel et al.

Original: (2017) used the theory of stochastic approximation to analyse GAN training.
Revised: (2017) used the theory of stochastic approximation to analyse GAN training.

Original: Viewing the training process as stochastic approximation with two time scales (Borkar, 1997; Konda &
Revised: Viewing the training process as stochastic approximation with two time scales (Borkar, 1997; Konda &

Original: Borkar, 1999), they suggest that the update of D should be fast enough compared with that of G .
Revised: Borkar, 1999), they suggest that the update of D should be fast enough compared with that of G .

Original: Under mild assumptions, Heusel et al.
Revised: Under mild assumptions, Heusel et al.

Original: (2017) proved that such two time-scale update converges to local Nash equilibrium.
Revised: (2017) proved that such two time-scale update converges to local Nash equilibrium.

Original: Their analysis follows the idea of ( τ, δ ) perturbation (Hirsch, 1989), where the slow updates ( G ) is interpreted as a small perturbation over the ODE describing the fast update ( D ).
Revised: Their analysis follows the idea of ( τ, δ ) perturbation (Hirsch, 1989), where the slow updates ( G ) is interpreted as a small perturbation over the ODE describing the fast update ( D ).

Original: Importantly, the size of perturbation δ is measured in the magnitude of parameter change, which is affected by both the learning rate and gradients.
Revised: Importantly, the size of perturbation δ is measured in the magnitude of parameter change, which is affected by both the learning rate and gradients.

Original: Here we show that LOGAN accelerates D update and slows down G update, thus helping the convergence of discriminator according to Heusel et al.
Revised: Here we show that LOGAN accelerates discriminator updates and slows down generator updates, thus helping the convergence of discriminator according to Heusel et al.

Original: We start from analysing the change of θ G .
Revised: We start from analysing the change of θ G .

Original: We assume that, without LO, it takes ∆ θ G = θ (cid:48)
Revised: We assume that, without LO, it takes ∆ θ G = θ (cid:48)

Original: G − θ G to make a small constant amount of reduction in loss L G :
Revised: G − θ G to make a small constant amount of reduction in loss L G :

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Now using the optimised z (cid:48)
Revised: Now using the optimised z (cid:48)

Original: =
Revised: =

Original: z + ∆ z , we assess the change δθ G required to achieve the same amount of reduction: 
Revised: z + ∆ z , we assess the change δθ G required to achieve the same amount of reduction: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Intuitively, when z “helps” θ G to achieve the same goal of increasing f
Revised: Intuitively, when z “helps” θ G to achieve the same goal of increasing f

Original: ( z ; θ D , θ G ) by ρ , the responsible of θ G becomes smaller, so it does not need to change as much as ∆ θ G , thus (cid:107) δθ G (cid:107) < (cid:107) ∆ θ G (cid:107) .
Revised: ( z ; θ D , θ G ) by ρ , the responsible of θ G becomes smaller, so it does not need to change as much as ∆ θ G , thus (cid:107) δθ G (cid:107) < (cid:107) ∆ θ G (cid:107) .

Original: Formally, f ( z ; θ D , θ G ) and f ( z + ∆; θ D , θ G + δθ G ) have the following Taylor expansions around z and θ G :
Revised: Formally, f ( z ; θ D , θ G ) and f ( z + ∆; θ D , θ G + δθ G ) have the following Taylor expansions around z and θ G :

Original: [Equation]
Revised: [Equation]

Original: Where (cid:15) ( · ) ’s are higher order terms of the increments.
Revised: Where (cid:15) ( · ) ’s are higher order terms of the increments.

Original: Using the assumption of eq.
Revised: Using the assumption of eq.

Original: 26 and [Equation] we can combine eq.
Revised: 26 and 27, we can combine eq.

Original: 28 and 29:
Revised: 28 and 29:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) =
Revised: where (cid:15) =

Original: (cid:15) (∆ z, δθ G ) − (cid:15) (∆ θ G ) .
Revised: (cid:15) (∆ z, δθ G ) − (cid:15) (∆ θ G ) .

Original: Since ∆ z ∝ ∂f ( z ; θ D ,θ G ) ∂z in gradient descent (eq. 3),
Revised: Since ∆ z ∝ ∂f ( z ; θ D ,θ G ) ∂z in gradient descent (eq. 3),

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have the inequality
Revised: Therefore, we have the inequality

Original: [Equation]
Revised: [Equation]

Original: If we further assume ∆ θ G and δθ G are obtained from stochastic gradient descent with identical learning rate,
Revised: If we further assume ∆ θ G and δθ G are obtained from stochastic gradient descent with identical learning rate,

Original: [Equation]
Revised: [Equation]

Revised: substituting eq.

Revised: 33 into eq.

Revised: 32 gives

Original: [Equation]
Revised: [Equation]

Original: The same analysis applies to the discriminator.
Revised: The same analysis applies to the discriminator.

Original: The similar intuition is that it takes the discriminator additional effort to compensate the exploitation from the optimised z (cid:48) .
Revised: The similar intuition is that it takes the discriminator additional effort to compensate the exploitation from the optimised z (cid:48) .

Original: Wecan then obtain (cid:18)
Revised: We then obtain [Equation] D

Original: [Equation]
Revised: [Equation]

Original: However, since the adversarial loss L D = − L G , we have ∆ θ D = − α ∂f ( z ; θ D ,θ G )
Revised: However, since the adversarial loss L D = − L G , we have ∆ θ D = − α ∂f ( z ; θ D ,θ G )

Original: ∂θ D and δθ D
Revised: ∂θ D and δθ D

Original: = − α ∂f ( z ; θ D ,θ G )
Revised: = − α ∂f ( z ; θ D ,θ G )

Original: ∂θ D taking the opposite signs of eq.33.
Revised: ∂θ D taking the opposite signs of eq.33.

Original: For sufﬁciently small ∆ z , ∆ θ G and δθ G , (cid:15) is close to zero, so (cid:107) ∆ θ D (cid:107) < (cid:107) δθ D (cid:107) under our assumptions of small ∆ z , ∆ θ G and δθ G .
Revised: For sufﬁciently small ∆ z , ∆ θ G and δθ G , (cid:15) is close to zero, so (cid:107) ∆ θ D (cid:107) < (cid:107) δθ D (cid:107) under our assumptions of small ∆ z , ∆ θ G and δθ G .

Original: Importantly, the bigger the product ∂f ( z ) ∂z ∆ z is, the more robust the inequality is to the error from (cid:15) .Moreover, bigger step increases the speed gap between updating D and G, further facilitating convergence according to Heusel et al.
Revised: Importantly, the bigger the product ∂f ( z ) ∂z ∆ z is, the more robust the inequality is to the error from (cid:15) . Moreover, bigger step increases the speed gap between updating D and G, further facilitating convergence according to Heusel et al.

Original: Overall, our analysis suggests:
Revised: Overall, our analysis suggests:

Original: More than 1 gradient descent steps may not be helpful, since ∆ z from multiple GD steps may deviate from the direction of ∂f ( z ) ∂z .
Revised: More than one gradient descent step may not be helpful, since ∆ z from multiple GD steps may deviate from the direction of ∂f ( z ) ∂z .

Original: Large step of ∆ z is more helpful in facilitating convergence by widening the gap between D and G updates (Heusel et al., 2017).
Revised: Large step of ∆ z is more helpful in facilitating convergence by widening the gap between D and G updates (Heusel et al., 2017).

Original: However, the step of ∆ z cannot be too large.
Revised: However, the step of ∆ z cannot be too large.

Original: In addition to the linear approximation we used throughout our analysis, the approximate SGA breaks down when eq.21 is strongly violated when “overshoot” brings the gradients at ∂f ( z (cid:48) ) ∂z (cid:48) to the opposite sign of ∂f ( z ) ∂z .
Revised: In addition to the linear approximation we used throughout our analysis, the approximate SGA breaks down when eq.21 is strongly violated when “overshoot” brings the gradients at ∂f ( z (cid:48) ) ∂z (cid:48) to the opposite sign of ∂f ( z ) ∂z .

Original: C P OISSON L IKELIHOOD FROM H INGE LOSS
Revised: C P OISSON L IKELIHOOD FROM H INGE LOSS

Revised: Here we provide a probabilistic interpretation of the hinge loss for the generator, which leads naturally to the scenario of a family of discriminators.

Revised: Although this interpretation is not necessary for our current algorithm, it may provides useful guidance for incorporating multiple discriminators.

Revised: We introduce the label t = 1 for real data and t = 0 fake samples.

Original: This section shows that the generator hinge loss
Revised: This section shows that the generator hinge loss

Original: [Equation]
Revised: [Equation]

Original: ) can be interpreted as a negative log-likelihood function given an ideal generator.
Revised: can be interpreted as a negative log-likelihood function:

Original: [Equation]
Revised: [Equation]

Original: It further justiﬁes our use of natural gradient descent (Amari, 1998).
Revised: Here p ( t = 1; z, D, G ) is the probability that the generated image G ( z ) can fool the discriminator D .

Original: The original GAN’s discriminator can be interpreted as outputting a Bernoulli distributions p ( t ; p g )
Revised: The original GAN’s discriminator can be interpreted as outputting a Bernoulli distribution p ( t ; β G )

Original: = p tg · (1 − p g ) 1 − t, from which a sample of t = 1 means the input image is true and vice versa . An ideal generator can perfectly fool the generator, giving the sample with label t = 1 .
Revised: = β tG · (1 − β G ) 1 − t .

Original: In this case, if we parametrise p g = D ( g ( z )) , the generator loss is the negative log-likelihood
Revised: In this case, if we parameterise β G = D ( G ( z )) , the generator loss is the negative log-likelihood

Original: [Equation]
Revised: [Equation]

Original: Bernoulli, however, is not the only valid choice as the discriminator’s output distribution.
Revised: Bernoulli, however, is not the only valid choice as the discriminator’s output distribution.

Original: Instead of sampling “1” or “0”, we assume  there are many identical discriminators that can independently vote to reject the input samples as fake.
Revised: Instead of sampling “1” or “0”, we assume that there are many identical discriminators that can independently vote to reject an input sample as fake.

Original: The number of votes k in a given interval can be described by a Poisson distribution with parameter λ with the following PMF:
Revised: The number of votes k in a given interval can be described by a Poisson distribution with parameter λ with the following PMF:

Original: [Equation]
Revised: [Equation]

Original: An ideal generator should receive no vote for rejection, so we have the following negative loglikelihood as thetraining objective for the generator  if we parametrise λ = − D ( G ( z )) :
Revised: The probability that a generated image can fool all the discriminators is the probability of G ( z ) receiving no vote for rejection [Equation] Therefore, we have the following negative log-likelihood as the generator loss if we parameterise λ = − D ( G ( z )) :

Original: [Equation]
Revised: [Equation]

Original: This interpretation has a caveat that when D ( G ( z )) > 

Revised: This interpretation has a caveat that when D ( G ( z )) >

Original: the Poisson distribution is not well deﬁned.
Revised: the Poisson distribution is not well deﬁned.

Original: However, in general the discriminator’s hinge loss
Revised: However, in general the discriminator’s hinge loss

Original: [Equation]
Revised: [Equation]

Original: pushes D ( g ( z )) < 0 via training.
Revised: pushes D ( G ( z )) < 0 via training.

Original: D D ETAILS IN C OMPUTING D ISTANCES IN F IGURE 5 A
Revised: D D ETAILS IN C OMPUTING D ISTANCES IN F IGURE 5 A

Original: For a temporal sequence x 1 , x 2 , . . .
Revised: For a temporal sequence x 1 , x 2 , . . .

Original: , x T (changes of z or f ( z ) at each training step in this paper), to normalise its variance while accounting for the non-stationarity, we process it as follows.
Revised: , x T (changes of z or f ( z ) at each training step in this paper), to normalise its variance while accounting for the non-stationarity, we process it as follows.

Original: We ﬁrst compute the moving average and standard deviation over a window of size N :
Revised: We ﬁrst compute the moving average and standard deviation over a window of size N :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Then normalise the sequence as:
Revised: Then normalise the sequence as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The result in Figure 5 a is robust to the choice of window size.
Revised: The result in Figure 5 a is robust to the choice of window size.

Original: Our experiments with N from 10 to 50 yielded visually similar plots.
Revised: Our experiments with N from 10 to 50 yielded visually similar plots.

Original: E E XPERIMENTS WITH DCGAN AND CIFAR
Revised: E E XPERIMENTS WITH DCGAN AND CIFAR

Original: To test if latent optimisation works with models at more moderate scales, we applied it on SN-GANs (Miyato et al., 2018).
Revised: To test if latent optimisation works with models at more moderate scales, we applied it on SN-GANs (Miyato et al., 2018).

Original: Although our experiments on this model are less thorough than in the main paper with BigGAN-deep, we hope to provide basic guidelines for researchers interested in applying latent optimisation on smaller models.
Revised: Although our experiments on this model are less thorough than in the main paper with BigGAN-deep, we hope to provide basic guidelines for researchers interested in applying latent optimisation on smaller models.

Original: The experiments follows the same basic setup and hyper-parameter settings as the CS-GAN in Wu et al.
Revised: The experiments follows the same basic setup and hyper-parameter settings as the CS-GAN in Wu et al.

Original: There is no class conditioning in this model.
Revised: There is no class conditioning in this model.

Original: For NGD, we found a smaller damping factor β = 0 .
Revised: For NGD, we found a smaller damping factor β = 0 .

Original: 1 , a (cid:107) z (cid:107) regulariser weight of 3 . 

Revised: 1 , a (cid:107) z (cid:107) regulariser weight of 3 .

Original: (the same as in Wu et al. 2019), combined with optimising 70% of the latent source (instead of 50% for BigGAN-deep) worked best for SN-GANs.
Revised: (the same as in Wu et al. 2019), combined with optimising 70% of the latent source (instead of 50% for BigGAN-deep) worked best for SN-GANs.

Original: In addition, we found running extra latent optimisation steps beneﬁted evaluation, so we use ten steps of latent optimisation in evaluation for results in this section, although the models were still trained with a single optimisation step.
Revised: In addition, we found running extra latent optimisation steps beneﬁted evaluation, so we use ten steps of latent optimisation in evaluation for results in this section, although the models were still trained with a single optimisation step.

Original: We reckon that smaller models might not be “over-parametrised” enough to fully amortise the computation from optimising z , which can then further exploit the architecture in evaluation time.
Revised: We reckon that smaller models might not be “over-parametrised” enough to fully amortise the computation from optimising z , which can then further exploit the architecture in evaluation time.

Original: On the other hand, the overhead from running multiple iterations of latent optimisation is relatively small at this scale.
Revised: On the other hand, the overhead from running multiple iterations of latent optimisation is relatively small at this scale.

Original: We aim to further investigate this difference in future studies.
Revised: We aim to further investigate this difference in future studies.

Original: Table 2 shows the FID and IS alongside SN-GAN and CS-CAN which used the same architecture.
Revised: Table 2 shows the FID and IS alongside SN-GAN and CS-CAN which used the same architecture.

Original: Here we observe similarly signiﬁcant improvement over the baseline SN-GAN model, with an improvement of 16 .
Revised: Here we observe similarly signiﬁcant improvement over the baseline SN-GAN model, with an improvement of 16 .

Original: 8% in IS and 39 .
Revised: 8% in IS and 39 .

Original: 6% in FID.
Revised: 6% in FID.

Original: Figure 9 shows random samples from these two models.
Revised: Figure 9 shows random samples from these two models.

Original: Overall, samples from LOGAN (NGD) have higher contrasts and sharper contours.
Revised: Overall, samples from LOGAN (NGD) have higher contrasts and sharper contours.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Revised: ( a b

Original: [Figure]
Revised: [Figure]