Original: L ARGE S CALE GAN T RAINING FOR H IGH F IDELITY N ATURAL I MAGE S YNTHESIS
Revised: L ARGE S CALE GAN T RAINING FOR H IGH F IDELITY N ATURAL I MAGE S YNTHESIS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal.
Revised: Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal.

Original: To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities speciﬁc to such scale.
Revised: To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities speciﬁc to such scale.

Original: We ﬁnd that applying orthogonal regularization to the generator renders it amenable to a simple “truncation trick,” allowing ﬁne control over the trade-off between sample ﬁdelity and variety by reducing the variance of the Generator’s input.
Revised: We ﬁnd that applying orthogonal regularization to the generator renders it amenable to a simple “truncation trick,” allowing ﬁne control over the trade-off between sample ﬁdelity and variety by reducing the variance of the Generator’s input.

Original: Our modiﬁcations lead to models which set the new state of the art in class-conditional image synthesis.
Revised: Our modiﬁcations lead to models which set the new state of the art in class-conditional image synthesis.

Original: When trained on ImageNet at 128 × 128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.3 and Fr ´ echet Inception Distance (FID) of 9.6, improving over the previous best IS of 52.52 and FID of 18.65.
Revised: When trained on ImageNet at 128 × 128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.5 and Fr ´ echet Inception Distance (FID) of 7.4, improving over the previous best IS of 52.52 and FID of 18.65.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The state of generative image modeling has advanced dramatically in recent years, with Generative
Revised: The state of generative image modeling has advanced dramatically in recent years, with Generative

Original: Adversarial Networks (GANs, Goodfellow et al.
Revised: Adversarial Networks (GANs, Goodfellow et al.

Original: at the forefront of efforts to generate highﬁdelity, diverse images with models learned directly from data.
Revised: at the forefront of efforts to generate highﬁdelity, diverse images with models learned directly from data.

Original: GAN training is dynamic, and sensitive to nearly every aspect of its setup (from optimization parameters to model architecture), but a torrent of research has yielded empirical and theoretical insights enabling stable training in a variety of settings.
Revised: GAN training is dynamic, and sensitive to nearly every aspect of its setup (from optimization parameters to model architecture), but a torrent of research has yielded empirical and theoretical insights enabling stable training in a variety of settings.

Original: Despite this progress, the current state of the art in conditional ImageNet modeling (Zhang et al., 2018) achieves an Inception Score (Salimans et al., 2016) of 52.5, compared to 233 for real data.
Revised: Despite this progress, the current state of the art in conditional ImageNet modeling (Zhang et al., 2018) achieves an Inception Score (Salimans et al., 2016) of 52.5, compared to 233 for real data.

Original: In this work, we set out to close the gap in ﬁdelity and variety between images generated by GANs and real-world images from the ImageNet dataset.
Revised: In this work, we set out to close the gap in ﬁdelity and variety between images generated by GANs and real-world images from the ImageNet dataset.

Original: We make the following three contributions towards this goal:
Revised: We make the following three contributions towards this goal:

Original: • We demonstrate that GANs beneﬁt dramatically from scaling, and train models with two to four times as many parameters and eight times the batch size compared to prior art.
Revised: • We demonstrate that GANs beneﬁt dramatically from scaling, and train models with two to four times as many parameters and eight times the batch size compared to prior art.

Original: We introduce two simple, general architectural changes that improve scalability, and modify a regularization scheme to improve conditioning, demonstrably boosting performance.
Revised: We introduce two simple, general architectural changes that improve scalability, and modify a regularization scheme to improve conditioning, demonstrably boosting performance.

Original: • As a side effect of our modiﬁcations, our models become amenable to the “truncation trick,” a simple sampling technique that allows explicit, ﬁne-grained control of the tradeoff between sample variety and ﬁdelity.
Revised: • As a side effect of our modiﬁcations, our models become amenable to the “truncation trick,” a simple sampling technique that allows explicit, ﬁne-grained control of the tradeoff between sample variety and ﬁdelity.

Original: • We discover instabilities speciﬁc to large scale GANs, and characterize them empirically.
Revised: • We discover instabilities speciﬁc to large scale GANs, and characterize them empirically.

Original: Leveraging insights from this analysis, we demonstrate that a combination of novel and existing techniques can reduce these instabilities, but complete training stability can only be achieved at a dramatic cost to performance
Revised: Leveraging insights from this analysis, we demonstrate that a combination of novel and existing techniques can reduce these instabilities, but complete training stability can only be achieved at a dramatic cost to performance.

Original: Our modiﬁcations substantially improve class-conditional GANs.
Revised: Our modiﬁcations substantially improve class-conditional GANs.

Original: When trained on ImageNet at 128 × 128 resolution, our models (BigGANs) improve the state-of-the-art Inception Score (IS) and
Revised: When trained on ImageNet at 128 × 128 resolution, our models (BigGANs) improve the state-of-the-art Inception Score (IS) and

Original: Fr´echet Inception Distance (FID) from 52.52 and 18.65 to 166.3 and 9.6 respectively.
Revised: Fr´echet Inception Distance (FID) from 52.52 and 18.65 to 166.5 and 7.4 respectively.

Original: We also successfully train BigGANs on ImageNet at 256 × 256 and 512 × 512 resolution, and achieve IS and
Revised: We also successfully train BigGANs on ImageNet at 256 × 256 and 512 × 512 resolution, and achieve IS and

Original: FID of 233.0 and 9.3 at 256 × 256 and IS and FID of 241.4 and 10.9 at 512 × 512.
Revised: FID of 232.5 and 8.1 at 256 × 256 and IS and FID of 241.5 and 11.5 at 512 × 512.

Original: Finally, we train our models on an even larger internal dataset, and demonstrate that our design choices transfer well from ImageNet.
Revised: Finally, we train our models on an even larger dataset – JFT-300M – and demonstrate that our design choices transfer well from ImageNet.

Revised: Code and weights for our pretrained generators are publicly available 1 .

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: A Generative Adversarial Network (GAN) involves Generator ( G ) and Discriminator ( D ) networks whose purpose, respectively, is to map random noise to samples and discriminate real and generated samples.
Revised: A Generative Adversarial Network (GAN) involves Generator ( G ) and Discriminator ( D ) networks whose purpose, respectively, is to map random noise to samples and discriminate real and generated samples.

Original: Formally, the GAN objective, in its original form (Goodfellow et al., 2014) involves ﬁnding a Nash equilibrium to the following two player min-max problem:
Revised: Formally, the GAN objective, in its original form (Goodfellow et al., 2014) involves ﬁnding a Nash equilibrium to the following two player min-max problem:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: z ∈ R
Revised: z ∈ R

Original: d
Revised: d

Original: z is a latent variable drawn from distribution p ( z ) such as N (0 , I ) or U [ − 1 , 1] .
Revised: z is a latent variable drawn from distribution p ( z ) such as N (0 , I ) or U [ − 1 , 1] .

Original: When applied to images, G and D are usually convolutional neural networks (Radford et al., 2016).
Revised: When applied to images, G and D are usually convolutional neural networks (Radford et al., 2016).

Original: Without auxiliary stabilization techniques, this training procedure is notoriously brittle, requiring ﬁnely-tuned hyperparameters and architectural choices to work at all.
Revised: Without auxiliary stabilization techniques, this training procedure is notoriously brittle, requiring ﬁnely-tuned hyperparameters and architectural choices to work at all.

Original: Much recent research has accordingly focused on modiﬁcations to the vanilla GAN procedure to impart stability, drawing on a growing body of empirical and theoretical insights (Nowozin et al., 2016;
Revised: Much recent research has accordingly focused on modiﬁcations to the vanilla GAN procedure to impart stability, drawing on a growing body of empirical and theoretical insights (Nowozin et al., 2016;

Original: Sønderby et al., 2017; Fedus et al., 2018).
Revised: Sønderby et al., 2017; Fedus et al., 2018).

Original: One line of work is focused on changing the objective function (Arjovsky et al., 2017; Mao et al., 2016; Lim & Ye, 2017; Bellemare et al., 2017; Salimans et al., 2018) to encourage convergence.
Revised: One line of work is focused on changing the objective function (Arjovsky et al., 2017; Mao et al., 2016; Lim & Ye, 2017; Bellemare et al., 2017; Salimans et al., 2018) to encourage convergence.

Original: Another line is focused on constraining
Revised: Another line is focused on constraining

Original: D through gradient penalties (Gulrajani et al., 2017; Kodali et al., 2017; Mescheder et al., 2018) or normalization (Miyato et al., 2018), both to counteract the use of unbounded loss functions and ensure D provides gradients everywhere to G .
Revised: D through gradient penalties (Gulrajani et al., 2017; Kodali et al., 2017; Mescheder et al., 2018) or normalization (Miyato et al., 2018), both to counteract the use of unbounded loss functions and ensure D provides gradients everywhere to G .

Original: Of particular relevance to our work is Spectral Normalization (Miyato et al., 2018), which enforces
Revised: Of particular relevance to our work is Spectral Normalization (Miyato et al., 2018), which enforces

Original: Lipschitz continuity on D by normalizing its parameters with running estimates of their ﬁrst singular values, inducing backwards dynamics that adaptively regularize the top singular direction.
Revised: Lipschitz continuity on D by normalizing its parameters with running estimates of their ﬁrst singular values, inducing backwards dynamics that adaptively regularize the top singular direction.

Original: Relatedly
Revised: Relatedly

Original: Odena et al.
Revised: Odena et al.

Original: (2018) analyze the condition number of the Jacobian of G and ﬁnd that performance is dependent on G ’s conditioning.
Revised: (2018) analyze the condition number of the Jacobian of G and ﬁnd that performance is dependent on G ’s conditioning.

Original: Zhang et al.
Revised: Zhang et al.

Original: ﬁnd that employing Spectral Normalization in
Revised: ﬁnd that employing Spectral Normalization in

Original: G improves stability, allowing for fewer D steps per iteration.
Revised: G improves stability, allowing for fewer D steps per iteration.

Original: We extend on these analyses to gain further insight into the pathology of GAN training.
Revised: We extend on these analyses to gain further insight into the pathology of GAN training.

Original: Other works focus on the choice of architecture, such as SA-GAN (Zhang et al., 2018) which adds the self-attention block from (Wang et al., 2018) to improve the ability of both G and D to model global structure.
Revised: Other works focus on the choice of architecture, such as SA-GAN (Zhang et al., 2018) which adds the self-attention block from (Wang et al., 2018) to improve the ability of both G and D to model global structure.

Original: ProGAN (Karras et al., 2018) trains high-resolution GANs in the single-class setting by training a single model across a sequence of increasing resolutions.
Revised: ProGAN (Karras et al., 2018) trains high-resolution GANs in the single-class setting by training a single model across a sequence of increasing resolutions.

Original: In conditional GANs (Mirza & Osindero, 2014) class information can be fed into the model in various ways.
Revised: In conditional GANs (Mirza & Osindero, 2014) class information can be fed into the model in various ways.

Original: In (Odena et al., 2017) it is provided to G by concatenating a 1-hot class vector to the noise vector, and the objective is modiﬁed to encourage conditional samples to maximize the corresponding class probability predicted by an auxiliary classiﬁer.
Revised: In (Odena et al., 2017) it is provided to G by concatenating a 1-hot class vector to the noise vector, and the objective is modiﬁed to encourage conditional samples to maximize the corresponding class probability predicted by an auxiliary classiﬁer.

Original: de Vries et al.
Revised: de Vries et al.

Original: (2017) and
Revised: (2017) and

Original: Dumoulin et al.

Original: (2017) modify the way class conditioning is passed to G by supplying it with classconditional gains and biases in BatchNorm (Ioffe & Szegedy, 2015) layers.

Original: In Miyato & Koyama (2018), D is conditioned by using the cosine similarity between its features and a set of learned class embeddings as additional evidence for distinguishing real and generated samples, effectively encouraging generation of samples whose features match a learned class prototype.

Original: [Table]
Revised: [Table]

Original: is total number of parameters, Ch. is the channel multiplier representing the number of units in each layer, Shared is using shared embeddings,Hier.
Revised: Ch. is the channel multiplier representing the number of units in each layer, Shared is using shared embeddings,

Original:  is using a hierarchical latent space, Ortho.
Revised: Skip- z is using skip connections from the latent to multiple layers, Ortho.

Original: is Orthogonal Regularization, and Itreither indicates that the setting is stable to 10 6 iterations, orthat it collapses at the given iteration.
Revised: is Orthogonal Regularization, and Itr indicates if the setting is stable to 10 6 iterations, or it collapses at the given iteration.

Original: Other than rows 1-4, results are computed across 8different random initializations. 
Revised: Other than rows 1-4, results are computed across 8 random initializations. Dumoulin et al.

Revised: (2017) modify the way class conditioning is passed to G by supplying it with classconditional gains and biases in BatchNorm (Ioffe & Szegedy, 2015) layers.

Revised: In Miyato & Koyama (2018), D is conditioned by using the cosine similarity between its features and a set of learned class embeddings as additional evidence for distinguishing real and generated samples, effectively encouraging generation of samples whose features match a learned class prototype.

Original: Objectively evaluating implicit generative models is difﬁcult (Theis et al., 2015).
Revised: Objectively evaluating implicit generative models is difﬁcult (Theis et al., 2015).

Original: A variety of works have proposed heuristics for measuring the sample quality of models without tractable likelihoods (Salimans et al., 2016; Heusel et al., 2017; Bi ´ nkowski et al., 2018; Wu et al., 2017).
Revised: A variety of works have proposed heuristics for measuring the sample quality of models without tractable likelihoods (Salimans et al., 2016; Heusel et al., 2017; Bi ´ nkowski et al., 2018; Wu et al., 2017).

Original: Of these, the Inception Score (IS, Salimans et al.
Revised: Of these, the Inception Score (IS, Salimans et al.

Original: and Fr ´ echet Inception Distance (FID, Heusel et al.
Revised: and Fr ´ echet Inception Distance (FID, Heusel et al.

Original: (2017)) have become popular despite their notable ﬂaws (Barratt & Sharma, 2018).
Revised: (2017)) have become popular despite their notable ﬂaws (Barratt & Sharma, 2018).

Original: We employ them as approximate measures of sample quality, and to enable comparison against previous work.
Revised: We employ them as approximate measures of sample quality, and to enable comparison against previous work.

Original:  S CALING U P GAN S
Revised:  S CALING U P GAN S

Original: In this section, we explore methods for scaling up GAN training to reap the performance beneﬁts of larger models and larger batches.
Revised: In this section, we explore methods for scaling up GAN training to reap the performance beneﬁts of larger models and larger batches.

Original: As a baseline, we employ the SA-GAN architecture of Zhang et al.
Revised: As a baseline, we employ the SA-GAN architecture of Zhang et al.

Original: (2018), which uses the hinge loss (Lim & Ye, 2017; Tran et al., 2017) GAN objective.
Revised: (2018), which uses the hinge loss (Lim & Ye, 2017; Tran et al., 2017) GAN objective.

Original: We provide class information to G with class-conditional BatchNorm (Dumoulin et al., 2017; de Vries et al., 2017) and to D with projection (Miyato & Koyama, 2018).
Revised: We provide class information to G with class-conditional BatchNorm (Dumoulin et al., 2017; de Vries et al., 2017) and to D with projection (Miyato & Koyama, 2018).

Original: The optimization settings follow Zhang et al.
Revised: The optimization settings follow Zhang et al.

Original: (2018) (notably employing Spectral Norm in G ) with the modiﬁcation that we halve the learning rates and take two D steps per G step.
Revised: (2018) (notably employing Spectral Norm in G ) with the modiﬁcation that we halve the learning rates and take two D steps per G step.

Original: For evaluation, we employ moving averages of G ’s weights following Karras et al.
Revised: For evaluation, we employ moving averages of G ’s weights following Karras et al.

Original: ; Mescheder et al.
Revised: ; Mescheder et al.

Original: ; Yazc et al.
Revised: ; Yazc et al.

Original: (2018), with a decay of 0 . 9999 .
Revised: (2018), with a decay of 0 . 9999 .

Original: We use Orthogonal Initialization (Saxe et al., 2014), whereas previous works used N (0 , 0 .
Revised: We use Orthogonal Initialization (Saxe et al., 2014), whereas previous works used N (0 , 0 .

Original: 02 I )
Revised: 02 I )

Original: (Radford et al., 2016) or Xavier initialization
Revised: (Radford et al., 2016) or Xavier initialization

Original: Glorot & Bengio, 2010).
Revised: Glorot & Bengio, 2010).

Original: Each model is trained on 128 to 512 cores of a Google TPU Pod (Google, 2018), and computes BatchNorm statistics in G across all devices, rather than per-device as is typical.
Revised: Each model is trained on 128 to 512 cores of a Google TPUv3 Pod (Google, 2018), and computes BatchNorm statistics in G across all devices, rather than per-device as is typical.

Original: We ﬁnd progressive growing (Karras et al., 2018) unnecessary even for our 512 × 512 models.
Revised: We ﬁnd progressive growing (Karras et al., 2018) unnecessary even for our 512 × 512 models.

Original: Additional details are in Appendix C.
Revised: Additional details are in Appendix C.

Original: We begin by increasing the batch size for the baseline model, and immediately ﬁnd tremendous beneﬁts in doing so.
Revised: We begin by increasing the batch size for the baseline model, and immediately ﬁnd tremendous beneﬁts in doing so.

Original: Rows 1-4 of Table 1 show that simply increasing the batch size by a factor of 8 improves the state-of-the-art IS by 46%.
Revised: Rows 1-4 of Table 1 show that simply increasing the batch size by a factor of 8 improves the state-of-the-art IS by 46%.

Original: We conjecture that this is a result of each batch covering more modes, providing better gradients for both networks.
Revised: We conjecture that this is a result of each batch covering more modes, providing better gradients for both networks.

Original: One notable side effect of this scaling is that our models reach better ﬁnal performance in fewer iterations, but become unstable and undergo complete training collapse.
Revised: One notable side effect of this scaling is that our models reach better ﬁnal performance in fewer iterations, but become unstable and undergo complete training collapse.

Original: We discuss the causes and ramiﬁcations of this in Section 4.
Revised: We discuss the causes and ramiﬁcations of this in Section 4.

Original: For these experiments, we report scores from checkpoints saved just before collapse.
Revised: For these experiments, we report scores from checkpoints saved just before collapse.

Original: We then increase the width (number of channels) in each layer by 50%, approximately doubling the number of parameters in both models.
Revised: We then increase the width (number of channels) in each layer by 50%, approximately doubling the number of parameters in both models.

Original: This leads to a further IS improvement of 21%, which we posit is due to the increased capacity of the model relative to the complexity of the dataset.
Revised: This leads to a further IS improvement of 21%, which we posit is due to the increased capacity of the model relative to the complexity of the dataset.

Original: Doubling  the depth does not appear to have the same effect on ImageNet models, instead degrading performance.
Revised: Doubling [Figure] the depth did not initially lead to improvement – we addressed this later in the BigGAN-deep model, which uses a different residual block structure.

Original: We note that class embeddings c used for the conditional BatchNorm layers in G contain a large number of weights.
Revised: We note that class embeddings c used for the conditional BatchNorm layers in G contain a large number of weights.

Original: Instead of having a separate layer for each embedding (Miyato et al., 2018;
Revised: Instead of having a separate layer for each embedding (Miyato et al., 2018;

Original: Zhang et al., 2018), we opt to use a shared embedding, which is linearly projected to each layer’s gains and biases (Perez et al., 2018).
Revised: Zhang et al., 2018), we opt to use a shared embedding, which is linearly projected to each layer’s gains and biases (Perez et al., 2018).

Original: This reduces computation and memory costs, and improves[Figure] training speed (in number of iterations required to reach a given performance) by 37%.
Revised: This reduces computation and memory costs, and improves training speed (in number of iterations required to reach a given performance) by 37%.

Original: Next, we employ a variant of hierarchical latent spaces, where the noise vector z is fed into multiple layers of G rather than just the initial layer.
Revised: Next, we add direct skip connections (skip- z ) from the noise vector z to multiple layers of G rather than just the initial layer.

Original: The intuition behind this design is to allow G to use the latent space to directly inﬂuence features at different resolutions and levels of hierarchy.
Revised: The intuition behind this design is to allow G to use the latent space to directly inﬂuence features at different resolutions and levels of hierarchy.

Original: For our architecture, this iseasily accomplished by splitting z into one chunk per resolution, and concatenating each chunk to the conditional vector c which gets projected to the BatchNorm gains and biases. 
Revised: In BigGAN, this is accomplished by splitting z into one chunk per resolution, and concatenating each chunk to the conditional vector c which gets projected to the BatchNorm gains and biases. In BigGAN-deep, we use an even simpler design, concatenating the entire z with the conditional vector without splitting it into chunks.

Original: Previous works (Goodfellow et al., 2014; Denton et al., 2015) have considered variants of this concept; our contribution is a minor modiﬁcation of this design.
Revised: Previous works (Goodfellow et al., 2014; Denton et al., 2015) have considered variants of this concept; our implementation is a minor modiﬁcation of this design.

Original: Hierarchical latents improve memory and compute costs (primarily by reducing the parametric budget of the ﬁrst linear layer), provide a modest performance improvement of around 4%, and improve training speed by a further 18%.
Revised: Skip- z provides a modest performance improvement of around 4%, and improves training speed by a further 18%.

Original: 3. T RADING OFF VARIETY AND FIDELITY WITH THE T RUNCATION T RICK
Revised: 3. T RADING OFF VARIETY AND FIDELITY WITH THE T RUNCATION T RICK

Original: Unlike models which need to backpropagate through their latents, GANs can employ an arbitrary prior p ( z ) , yet the vast majority of previous works have chosen to draw z from either N (0 , I ) or
Revised: Unlike models which need to backpropagate through their latents, GANs can employ an arbitrary prior p ( z ) , yet the vast majority of previous works have chosen to draw z from either N (0 , I ) or

Original: U [ − 1 , 1] .
Revised: U [ − 1 , 1] .

Original: We question the optimality of this choice and explore alternatives in Appendix E.
Revised: We question the optimality of this choice and explore alternatives in Appendix E.

Original: Remarkably, our best results come from using a different latent distribution for sampling than was used in training.
Revised: Remarkably, our best results come from using a different latent distribution for sampling than was used in training.

Original: Taking a model trained with z ∼ N (0 , I ) and sampling z from a truncated normal (where values which fall outside a range are resampled to fall inside that range) immediately provides a boost to IS and FID.
Revised: Taking a model trained with z ∼ N (0 , I ) and sampling z from a truncated normal (where values which fall outside a range are resampled to fall inside that range) immediately provides a boost to IS and FID.

Original: We call this the Truncation Trick : truncating a z vector by resampling the values with magnitude above a chosen threshold leads to improvement in individual sample quality at the cost of reduction in overall sample variety.
Revised: We call this the Truncation Trick : truncating a z vector by resampling the values with magnitude above a chosen threshold leads to improvement in individual sample quality at the cost of reduction in overall sample variety.

Original: Figure 2(a) demonstrates this: as the threshold is reduced, and elements of z are truncated towards zero (the mode of the latent distribution), individual samples approach the mode of G ’s output distribution.
Revised: Figure 2(a) demonstrates this: as the threshold is reduced, and elements of z are truncated towards zero (the mode of the latent distribution), individual samples approach the mode of G ’s output distribution.

Original: Related observations about this trade-off were made in (Marchesi, 2016;
Revised: Related observations about this trade-off were made in (Marchesi, 2016;

Original: Pieters & Wiering, 2014).
Revised: Pieters & Wiering, 2014).

Original: This technique allows ﬁne-grained, post-hoc selection of the trade-off between sample quality and variety for a given G .
Revised: This technique allows ﬁne-grained, post-hoc selection of the trade-off between sample quality and variety for a given G .

Original: Notably, we can compute FID and IS for a range of thresholds, obtaining the variety-ﬁdelity curve reminiscent of the precision-recall curve (Figure 16).
Revised: Notably, we can compute FID and IS for a range of thresholds, obtaining the variety-ﬁdelity curve reminiscent of the precision-recall curve (Figure 17).

Original: As IS does not penalize lack of variety in class-conditional models, reducing the truncation threshold leads to a direct increase in IS (analogous to precision).
Revised: As IS does not penalize lack of variety in class-conditional models, reducing the truncation threshold leads to a direct increase in IS (analogous to precision).

Original: FID penalizes lack of variety (analogous to recall) but also rewards precision, so we initially see a moderate improvement in FID, but as truncation approaches zero and variety diminishes, the FID sharply drops.
Revised: FID penalizes lack of variety (analogous to recall) but also rewards precision, so we initially see a moderate improvement in FID, but as truncation approaches zero and variety diminishes, the FID sharply drops.

Original: The distribution shift caused by sampling with different latents than those seen in training is problematic for many models.
Revised: The distribution shift caused by sampling with different latents than those seen in training is problematic for many models.

Original: Some of our larger models are not amenable to truncation, producing saturation artifacts (Figure 2(b)) when fed truncated noise.
Revised: Some of our larger models are not amenable to truncation, producing saturation artifacts (Figure 2(b)) when fed truncated noise.

Original: To counteract this, we seek to enforce amenability to truncation by conditioning G to be smooth, so that the full space of z will map to good output samples.
Revised: To counteract this, we seek to enforce amenability to truncation by conditioning G to be smooth, so that the full space of z will map to good output samples.

Original: For this, we turn to Orthogonal
Revised: For this, we turn to Orthogonal

Original: Regularization (Brock et al., 2017), which directly enforces the orthogonality condition:
Revised: Regularization (Brock et al., 2017), which directly enforces the orthogonality condition:

Original: [Equation]
Revised: [Equation]

Original: ) where W is a weight matrix and β a hyperparameter.
Revised: where W is a weight matrix and β a hyperparameter.

Original: This regularization is known to often be too limiting (Miyato et al., 2018), so we explore several variants designed to relax the constraint while still imparting the desired smoothness to our models.
Revised: This regularization is known to often be too limiting (Miyato et al., 2018), so we explore several variants designed to relax the constraint while still imparting the desired smoothness to our models.

Original: The version we ﬁnd to work best removes the diagonal terms from the regularization, and aims to minimize the pairwise cosine similarity between ﬁlters but does not constrain their norm:
Revised: The version we ﬁnd to work best removes the diagonal terms from the regularization, and aims to minimize the pairwise cosine similarity between ﬁlters but does not constrain their norm:

Original: [Equation]
Revised: [Equation]

Original: where 1 denotes a matrix with all elements set to 1 .
Revised: where 1 denotes a matrix with all elements set to 1 .

Original: We sweep β values and select 10 − 4 , ﬁnding this small added penalty sufﬁcient to improve the likelihood that our models will be amenable to truncation.
Revised: We sweep β values and select 10 − 4 , ﬁnding this small added penalty sufﬁcient to improve the likelihood that our models will be amenable to truncation.

Original: Across runs in Table 1, we observe that without Orthogonal Regularization, only 16% of models are amenable to truncation, compared to 60% when trained with Orthogonal Regularization.
Revised: Across runs in Table 1, we observe that without Orthogonal Regularization, only 16% of models are amenable to truncation, compared to 60% when trained with Orthogonal Regularization.

Original: 3. S UMMARY
Revised: 3. S UMMARY

Original: We ﬁnd that current GAN techniques are sufﬁcient to enable scaling to large models and distributed, large-batch training.
Revised: We ﬁnd that current GAN techniques are sufﬁcient to enable scaling to large models and distributed, large-batch training.

Original: We ﬁnd that we can dramatically improve the state of the art and train models up to 512 × 512 resolution without need for explicit multiscale methods like Karras et al.
Revised: We ﬁnd that we can dramatically improve the state of the art and train models up to 512 × 512 resolution without need for explicit multiscale methods like Karras et al.

Original: Despite these improvements, our models undergo training collapse, necessitating early stopping in practice.
Revised: Despite these improvements, our models undergo training collapse, necessitating early stopping in practice.

Original: In the next two sections we investigate why settings which were stable in previous works become unstable when applied at scale.
Revised: In the next two sections we investigate why settings which were stable in previous works become unstable when applied at scale.

Original:  A NALYSIS
Revised:  A NALYSIS

Original: [Figure]
Revised: [Figure]

Original: 4. C HARACTERIZING I NSTABILITY : T HE G ENERATOR
Revised: 4. C HARACTERIZING I NSTABILITY : T HE G ENERATOR

Original: Much previous work has investigated GAN stability from a variety of analytical angles and on toy problems, but the instabilities we observe occur for settings which are stable at small scale, necessitating direct analysis at large scale.
Revised: Much previous work has investigated GAN stability from a variety of analytical angles and on toy problems, but the instabilities we observe occur for settings which are stable at small scale, necessitating direct analysis at large scale.

Original: We monitor a range of weight, gradient, and loss statistics during training, in search of a metric which might presage the onset of training collapse, similar to (Odena et al., 2018).
Revised: We monitor a range of weight, gradient, and loss statistics during training, in search of a metric which might presage the onset of training collapse, similar to (Odena et al., 2018).

Original: We found the top three singular values σ 0 , σ 1 , σ 2 of each weight matrix to be the most informative.
Revised: We found the top three singular values σ 0 , σ 1 , σ 2 of each weight matrix to be the most informative.

Original: They can be efﬁciently computed using the Alrnoldi iteration method (Golub & der Vorst, 2000), which extends the power iteration method, used in Miyato et al.
Revised: They can be efﬁciently computed using the Alrnoldi iteration method (Golub & der Vorst, 2000), which extends the power iteration method, used in Miyato et al.

Original: (2018), to estimation of additional singular vectors and values.
Revised: (2018), to estimation of additional singular vectors and values.

Original: A clear pattern emerges, as can be seen in
Revised: A clear pattern emerges, as can be seen in

Original: Figure 3(a) and Appendix F: most G layers have well-behaved spectral norms, but some layers (typically the ﬁrst layer in G , which is over-complete and not convolutional) are ill-behaved, with spectral norms that grow throughout training and explode at collapse.
Revised: Figure 3(a) and Appendix F: most G layers have well-behaved spectral norms, but some layers (typically the ﬁrst layer in G , which is over-complete and not convolutional) are ill-behaved, with spectral norms that grow throughout training and explode at collapse.

Original: To ascertain if this pathology is a cause of collapse or merely a symptom, we study the effects of imposing additional conditioning on G to explicitly counteract spectral explosion.
Revised: To ascertain if this pathology is a cause of collapse or merely a symptom, we study the effects of imposing additional conditioning on G to explicitly counteract spectral explosion.

Original: First, we directly regularize the top singular values σ 0 of each weight, either towards a ﬁxed value σ reg or towards some ratio r of the second singular value, r · sg ( σ 1 ) (with sg the stop-gradient operation to prevent the regularization from increasing σ 1 ).
Revised: First, we directly regularize the top singular values σ 0 of each weight, either towards a ﬁxed value σ reg or towards some ratio r of the second singular value, r · sg ( σ 1 ) (with sg the stop-gradient operation to prevent the regularization from increasing σ 1 ).

Original: Alternatively, we employ a partial singular value decomposition to instead clamp σ 0 .
Revised: Alternatively, we employ a partial singular value decomposition to instead clamp σ 0 .

Original: Given a weight W , its ﬁrst singular vectors u 0 and v 0 , and σ clamp the value to which the σ 0 will be clamped, our weights become:
Revised: Given a weight W , its ﬁrst singular vectors u 0 and v 0 , and σ clamp the value to which the σ 0 will be clamped, our weights become:

Original: [Equation]
Revised: [Equation]

Original: where σ clamp is set to either σ reg or r · sg ( σ 1 ) .
Revised: where σ clamp is set to either σ reg or r · sg ( σ 1 ) .

Original: We observe that both with and without Spectral
Revised: We observe that both with and without Spectral

Original: Normalization these techniques have the effect of preventing the gradual increase and explosion of either σ 0 or σ 0 σ 1 , but even though in some cases they mildly improve performance, no combination prevents training collapse.
Revised: Normalization these techniques have the effect of preventing the gradual increase and explosion of either σ 0 or σ 0 σ 1 , but even though in some cases they mildly improve performance, no combination prevents training collapse.

Original: This evidence suggests that while conditioning G might improve stability, it is insufﬁcient to ensure stability.
Revised: This evidence suggests that while conditioning G might improve stability, it is insufﬁcient to ensure stability.

Original: We accordingly turn our attention to D .
Revised: We accordingly turn our attention to D .

Original: 4. C HARACTERIZING I NSTABILITY : T HE D ISCRIMINATOR
Revised: 4. C HARACTERIZING I NSTABILITY : T HE D ISCRIMINATOR

Original: As with G , we analyze the spectra of D ’s weights to gain insight into its behavior, then seek to stabilize training by imposing additional constraints.
Revised: As with G , we analyze the spectra of D ’s weights to gain insight into its behavior, then seek to stabilize training by imposing additional constraints.

Original: Figure 3(b) displays a typical plot of σ 0 for D (with further plots in Appendix F).
Revised: Figure 3(b) displays a typical plot of σ 0 for D (with further plots in Appendix F).

Original: Unlike G , we see that the spectra are noisy, σ 0 σ 1 is well-behaved, and the singular values grow throughout training but only jump at collapse, instead of exploding.
Revised: Unlike G , we see that the spectra are noisy, σ 0 σ 1 is well-behaved, and the singular values grow throughout training but only jump at collapse, instead of exploding.

Original: The spikes in D ’s spectra might suggest that it periodically receives very large gradients, but we observe that the Frobenius norms are smooth (Appendix F), suggesting that this effect is primarily concentrated on the top few singular directions.
Revised: The spikes in D ’s spectra might suggest that it periodically receives very large gradients, but we observe that the Frobenius norms are smooth (Appendix F), suggesting that this effect is primarily concentrated on the top few singular directions.

Original: We posit that this noise is a result of optimization through the adversarial training process, where G periodically produces batches which strongly perturb D .
Revised: We posit that this noise is a result of optimization through the adversarial training process, where G periodically produces batches which strongly perturb D .

Original: If this spectral noise is causally related to instability, a natural counter is to employ gradient penalties, which explicitly regularize changes in D ’s Jacobian.
Revised: If this spectral noise is causally related to instability, a natural counter is to employ gradient penalties, which explicitly regularize changes in D ’s Jacobian.

Original: We explore the R 1 zero-centered gradient penalty from Mescheder et al.
Revised: We explore the R 1 zero-centered gradient penalty from Mescheder et al.

Original: [Equation]
Revised: [Equation]

Original: With the default suggested γ strength of 10, training becomes stable and improves the smoothness and boundedness of spectra in both G and D , but performance severely degrades, resulting in a 45% reduction in IS.
Revised: With the default suggested γ strength of 10, training becomes stable and improves the smoothness and boundedness of spectra in both G and D , but performance severely degrades, resulting in a 45% reduction in IS.

Original: Reducing the penalty partially alleviates this degradation, but results in increasingly ill-behaved spectra; even with the penalty strength reduced to 1 (the lowest strength for which sudden collapse does not occur) the IS is reduced by 20%.
Revised: Reducing the penalty partially alleviates this degradation, but results in increasingly ill-behaved spectra; even with the penalty strength reduced to 1 (the lowest strength for which sudden collapse does not occur) the IS is reduced by 20%.

Original: Repeating this experiment with various strengths of Orthogonal Regularization, DropOut (Srivastava et al., 2014), and L2 (See Appendix I for details), reveals similar behaviors for these regularization strategies: with high enough penalties on D , training stability can be achieved, but at a substantial cost to performance.
Revised: Repeating this experiment with various strengths of Orthogonal Regularization, DropOut (Srivastava et al., 2014), and L2 (See Appendix I for details), reveals similar behaviors for these regularization strategies: with high enough penalties on D , training stability can be achieved, but at a substantial cost to performance.

Original: We also observe that D ’s loss approaches zero during training, but undergoes a sharp upward jump at collapse (Appendix F).
Revised: We also observe that D ’s loss approaches zero during training, but undergoes a sharp upward jump at collapse (Appendix F).

Original: One possible explanation for this behavior is that D is overﬁtting to the training set, memorizing training examples rather than learning some meaningful boundary between real and generated images.
Revised: One possible explanation for this behavior is that D is overﬁtting to the training set, memorizing training examples rather than learning some meaningful boundary between real and generated images.

Original: As a simple test for D ’s memorization (related to Gulrajani et al.
Revised: As a simple test for D ’s memorization (related to Gulrajani et al.

Original: , we evaluate uncollapsed discriminators on the ImageNet training and validation sets, and measure what percentage of samples are classiﬁed as real or generated.
Revised: , we evaluate uncollapsed discriminators on the ImageNet training and validation sets, and measure what percentage of samples are classiﬁed as real or generated.

Original: While the training accuracy is consistently above 98%, the validation accuracy falls in the range of 50-55%, no better than random guessing (regardless of regularization strategy).
Revised: While the training accuracy is consistently above 98%, the validation accuracy falls in the range of 50-55%, no better than random guessing (regardless of regularization strategy).

Original: This conﬁrms that D is indeed memorizing the training set; we deem this in line with D ’s role, which is not explicitly to generalize, but to distill the training data and provide a useful learning signal for G .
Revised: This conﬁrms that D is indeed memorizing the training set; we deem this in line with D ’s role, which is not explicitly to generalize, but to distill the training data and provide a useful learning signal for G .

Original: Additional experiments and discussion are provided in Appendix G.
Revised: Additional experiments and discussion are provided in Appendix G.

Original: 4. S UMMARY
Revised: 4. S UMMARY

Original: We ﬁnd that stability does not come solely from G or D , but from their interaction through the adversarial training process.
Revised: We ﬁnd that stability does not come solely from G or D , but from their interaction through the adversarial training process.

Original: While the symptoms of their poor conditioning can be used to track and  identify instability, ensuring reasonable conditioning proves necessary for training but insufﬁcient to prevent eventual training collapse.
Revised: While the symptoms of their poor conditioning can be used to track and [Table] identify instability, ensuring reasonable conditioning proves necessary for training but insufﬁcient to prevent eventual training collapse.

Original: It is possible to enforce stability by strongly constraining D , but doing so incurs a dramatic cost in performance.
Revised: It is possible to enforce stability by strongly constraining D , but doing so incurs a dramatic cost in performance.

Original: With current techniques, better ﬁnal performance can be achieved by relaxing this conditioning and allowing collapse to occur at the later stages of training, by which time a model is sufﬁciently trained to achieve good results.[Figure] Model Res.
Revised: With current techniques, better ﬁnal performance can be achieved by relaxing this conditioning and allowing collapse to occur at the later stages of training, by which time a model is sufﬁciently trained to achieve good results.

Original:  FID/IS (min FID) / IS FID / (valid IS) FID / (max IS) [Table]
Revised:  E XPERIMENTS

Original: [Table]

Original: N/A

Original: N/A

Original: N/A

Original: [Table]

Original: [Table]

Original: N/A

Original: N/A

Original: N/A

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Figure]

Original: E XPERIMENTS 5. E VALUATION ON I MAGE N ET
Revised: 5. E VALUATION ON I MAGE N ET

Original: We evaluate our models on ImageNet ILSVRC 2012 (Russakovsky et al., 2015) at 128 × 128, 256 × 256, and 512 × 512 resolutions, employing the settings from Table 1, row 8.
Revised: We evaluate our models on ImageNet ILSVRC 2012 (Russakovsky et al., 2015) at 128 × 128, 256 × 256, and 512 × 512 resolutions, employing the settings from Table 1, row 8.

Original: Architectural details for each resolution are available in Appendix B. Samples are presented in Figure 4, with additional samples in Appendix A, and we report IS and FID in Table 2.
Revised: The samples generated by our models are presented in Figure 4, with additional samples in Appendix A, and online 2 . We report IS and FID in Table 2.

Original: As our models are able to trade sample variety for quality, it is unclear how best to compare against prior art; we accordingly report values at three settings, with detailed curves in Appendix D. First, we report the FID/IS values at the truncation setting which attains the best FID.
Revised: As our models are able to trade sample variety for quality, it is unclear how best to compare against prior art; we accordingly report values at three settings, with complete curves in Appendix D. First, we report the FID/IS values at the truncation setting which attains the best FID.

Original: Second, we report the FID at the truncation setting for which our model’s IS is the same as that attained by the real validation data, reasoning that this is a passable measure of maximum sample variety achieved while still achieving a good level of “objectness.”
Revised: Second, we report the FID at the truncation setting for which our model’s IS is the same as that attained by the real validation data, reasoning that this is a passable measure of maximum sample variety achieved while still achieving a good level of “objectness.”

Original: Third, we report FID at the maximum IS achieved by each model, to demonstrate how much variety must be traded off to maximize quality.
Revised: Third, we report FID at the maximum IS achieved by each model, to demonstrate how much variety must be traded off to maximize quality.

Original: In all three cases, our models outperform the previous state-of-the-art IS and FID scores achieved by Miyato et al.
Revised: In all three cases, our models outperform the previous state-of-the-art IS and FID scores achieved by Miyato et al.

Original: (2018) and Zhang et al.
Revised: (2018) and Zhang et al.

Revised: In addition to the BigGAN model introduced in the ﬁrst version of the paper and used in the majority of experiments (unless otherwise stated), we also present a 4x deeper model (BigGAN-deep) which uses a different conﬁguration of residual blocks.

Revised: As can be seen from Table 2, BigGAN-deep substantially outperforms BigGAN across all resolutions and metrics.

Revised: This conﬁrms that our ﬁndings

Revised: [Table]

Revised: extend to other architectures, and that increased depth leads to improvement in sample quality.

Original:  Our observation that D overﬁts to the training set, coupled with our model’s sample quality, raises the obvious question of whether or not G simply memorizes training points.
Revised: Both BigGAN and BigGAN-deep architectures are described in Appendix B Our observation that D overﬁts to the training set, coupled with our model’s sample quality, raises the obvious question of whether or not G simply memorizes training points.

Original: To test this, we perform class-wise nearest neighbors analysis in pixel space and the feature space of pre-trained classiﬁer networks (Appendix A).
Revised: To test this, we perform class-wise nearest neighbors analysis in pixel space and the feature space of pre-trained classiﬁer networks (Appendix A).

Original: In addition, we present both interpolations between samples and class-wise interpolations (where z is held constant) in Figures 8 and 9.
Revised: In addition, we present both interpolations between samples and class-wise interpolations (where z is held constant) in Figures 8 and 9.

Original: Our model convincingly interpolates between disparate samples, and the nearest neighbors for its samples are visually distinct, suggesting that our model does not simply memorize training data.
Revised: Our model convincingly interpolates between disparate samples, and the nearest neighbors for its samples are visually distinct, suggesting that our model does not simply memorize training data.

Original: We note that some failure modes of our partially-trained models are distinct from those previously observed.
Revised: We note that some failure modes of our partially-trained models are distinct from those previously observed.

Original: Most previous failures involve local artifacts (Odena et al., 2016), images consisting of texture blobs instead of objects (Salimans et al., 2016), or the canonical mode collapse.
Revised: Most previous failures involve local artifacts (Odena et al., 2016), images consisting of texture blobs instead of objects (Salimans et al., 2016), or the canonical mode collapse.

Original: We observe class leakage , where images from one class contain properties of another, as exempliﬁed by Figure 4(d).
Revised: We observe class leakage , where images from one class contain properties of another, as exempliﬁed by Figure 4(d).

Original: We also ﬁnd that many classes on ImageNet are more difﬁcult than others for our model; our model is more successful at generating dogs (which make up a large portion of the dataset, and are mostly distinguished by their texture) than crowds (which comprise a small portion of the dataset and have more large-scale structure).
Revised: We also ﬁnd that many classes on ImageNet are more difﬁcult than others for our model; our model is more successful at generating dogs (which make up a large portion of the dataset, and are mostly distinguished by their texture) than crowds (which comprise a small portion of the dataset and have more large-scale structure).

Original: Further discussion is available in Appendix A.
Revised: Further discussion is available in Appendix A.

Original: [Table] A DDITIONAL EVALUATION ON AN INTERNAL DATASET
Revised: 5. A DDITIONAL EVALUATION ON JFT-300M

Original: To conﬁrm that our design choices are effective for even larger and more complex and diverse datasets, we also present results of our system on an internal dataset.
Revised: To conﬁrm that our design choices are effective for even larger and more complex and diverse datasets, we also present results of our system on a subset of JFT-300M (Sun et al., 2017).

Revised: The full JFT-300M dataset contains 300M real-world images labeled with 18K categories.

Revised: Since the category distribution is heavily long-tailed, we subsample the dataset to keep only images with the 8.5K most common labels.

Original: The  dataset contains 292M imageslabeled with 8.5K categories – two orders of magnitude larger than ImageNet.
Revised: The resulting dataset contains 292M images – two orders of magnitude larger than ImageNet.

Original: For images with multiple labels, we sample a single label randomly and independently whenever an image is sampled.
Revised: For images with multiple labels, we sample a single label randomly and independently whenever an image is sampled.

Original: To compute IS and FID for the GANs trained on this dataset, we use an Inception v2 classiﬁer (Szegedy et al., 2016) trained on this dataset.
Revised: To compute IS and FID for the GANs trained on this dataset, we use an Inception v2 classiﬁer (Szegedy et al., 2016) trained on this dataset.

Original: Quantitative results are presented in Table 3.
Revised: Quantitative results are presented in Table 3.

Original: All models are trained with batch size 2048.
Revised: All models are trained with batch size 2048.

Original: We compare an ablated version of our model – comparable to SA-GAN (Zhang et al., 2018) but with the larger batch size – against a “full” version that makes uses of all of the techniques applied to obtain the best results on ImageNet (shared embedding, hierarchical latents, and orthogonal regularization).
Revised: We compare an ablated version of our model – comparable to SA-GAN (Zhang et al., 2018) but with the larger batch size – against a “full” BigGAN model that makes uses of all of the techniques applied to obtain the best results on ImageNet (shared embedding, skip- z , and orthogonal regularization).

Original: Our results show that these techniques substantially improve performance even in the setting of this much larger dataset at the same model capacity ( base channels).
Revised: Our results show that these techniques substantially improve performance even in the setting of this much larger dataset at the same model capacity (64 base channels).

Original: We further show that for a dataset of this scale, we see signiﬁcant additional improvements from expanding the capacity of our models to 128 base channels, while for ImageNet GANs that additional capacity was not beneﬁcial.
Revised: We further show that for a dataset of this scale, we see signiﬁcant additional improvements from expanding the capacity of our models tobase channels, while for ImageNet GANs that additional capacity was not beneﬁcial.

Original: In Figure 18 (Appendix D), we present truncation plots for models trained on this dataset.
Revised: In Figure 19 (Appendix D), we present truncation plots for models trained on this dataset.

Original: Unlike for ImageNet, where truncation limits of σ ≈ 0 tend to produce the highest ﬁdelity scores,here IS is typically maximized  when the truncation value σ ranges from 0.5 to 1.
Revised: Unlike for ImageNet, where truncation limits of σ ≈ 0 tend to produce the highest ﬁdelity scores, IS is typically maximized for our JFT-300M models when the truncation value σ ranges from 0.5 to 1.

Original: We suspect that this is at least partially due to thedataset’s intra-class variability, as well as the relative complexity of the image distribution, which includes images with multiple objects at a variety of scales.
Revised: We suspect that this is at least partially due to the intra-class variability of JFT-300M labels, as well as the relative complexity of the image distribution, which includes images with multiple objects at a variety of scales.

Original: Interestingly, unlike models trained on ImageNet, where training tends to collapse without heavy regularization (Section 4), the models trained on the larger dataset remain stable over many hundreds of thousands of iterations.
Revised: Interestingly, unlike models trained on ImageNet, where training tends to collapse without heavy regularization (Section 4), the models trained on JFT-300M remain stable over many hundreds of thousands of iterations.

Original: This suggests that moving beyond ImageNet to larger datasets may partially alleviate GAN stability issues.
Revised: This suggests that moving beyond ImageNet to larger datasets may partially alleviate GAN stability issues.

Original: The improvement over the baseline GAN model that we achieve on this dataset without changes to the underlying models or training and regularization techniques (beyond expanded capacity) demonstrates that our ﬁndings extend from ImageNet to datasets with scale and complexity thus far unprecedented for generative models of images.
Revised: The improvement over the baseline GAN model that we achieve on this dataset without changes to the underlying models or training and regularization techniques (beyond expanded capacity) demonstrates that our ﬁndings extend from ImageNet to datasets with scale and complexity thus far unprecedented for generative models of images.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have demonstrated that Generative Adversarial Networks trained to model natural images of multiple categories highly beneﬁt from scaling up, both in terms of ﬁdelity and variety of the generated samples.
Revised: We have demonstrated that Generative Adversarial Networks trained to model natural images of multiple categories highly beneﬁt from scaling up, both in terms of ﬁdelity and variety of the generated samples.

Original: As a result, our models set a new level of performance among ImageNet GAN models, improving on the state of the art by a large margin.
Revised: As a result, our models set a new level of performance among ImageNet GAN models, improving on the state of the art by a large margin.

Original: We have also presented an analysis of the training behavior of large scale GANs, characterized their stability in terms of the singular values of their weights, and discussed the interplay between stability and performance. 
Revised: We have also presented an analysis of the training behavior of large scale GANs, characterized their stability in terms of the singular values of their weights, and discussed the interplay between stability and performance. A CKNOWLEDGMENTS

Revised: We would like to thank Kai Arulkumaran, Matthias Bauer, Peter Buchlovsky, Jeffrey Defauw,

Revised: Sander Dieleman, Ian Goodfellow, Ariel Gordon, Karol Gregor, Dominik Grewe, Chris Jones, Jacob

Revised: Menick, Augustus Odena, Suman Ravuri, Ali Razavi, Mihaela Rosca, and Jeff Stanway.

Original: A PPENDIX A A DDITIONAL S AMPLES , I NTERPOLATIONS , AND N EAREST N EIGHBORS FROM I MAGE N ET MODELS
Revised: A PPENDIX A A DDITIONAL S AMPLES , I NTERPOLATIONS , AND N EAREST N EIGHBORS FROM I MAGE N ET MODELS

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX B A RCHITECTURAL DETAILS
Revised: A PPENDIX B A RCHITECTURAL DETAILS

Original: We use the ResNet (He et al., 2016)GAN architecture of (Zhang et al., 2018).
Revised: In the BigGAN model (Figure 15), we use the ResNet (He et al., 2016)

Original: This architecture  is identical to that used by (Miyato et al., 2018), but with the channel pattern in D modiﬁed so that the number of ﬁlters in the ﬁrst convolutional layer of each block is equal to the number of output ﬁlters (rather than the number of input ﬁlters, as in Miyato et al.
Revised: GAN architecture of (Zhang et al., 2018), which is identical to that used by (Miyato et al., 2018) , but with the channel pattern in D modiﬁed so that the number of ﬁlters in the ﬁrst convolutional layer of each block is equal to the number of output ﬁlters (rather than the number of input ﬁlters, as in Miyato et al.

Original: ; Gulrajani et al.
Revised: Gulrajani et al.

Revised: We use a single shared class embedding in G , and skip connections for the latent vector z (skip- z ).

Revised: In particular, we employ hierarchical latent spaces, so that the latent vector z is split along its channel dimension into chunks of equal size ( 20 -D in our case), and each chunk is concatenated to the shared class embedding and passed to a corresponding residual block as a conditioning vector.

Original: We use a single shared class embedding in G , which is linearly projected to produce per-sample gains and biases for the BatchNorm layers.
Revised: The conditioning of each block is linearly projected to produce per-sample gains and biases for the BatchNorm layers of the block.

Original: The bias projections are zero-centered, while the gain projections are one-centered.
Revised: The bias projections are zero-centered, while the gain projections are centered at 1 .

Original: When employing hierarchical latent spaces, the latent vector z is split along its channel dimension into equal sized chunks, and each chunk is separately concatenated to the copy of the class embedding passed into a given block. For compatibility with the number of blocks, the  dimensionality of z  for 128 × 128 pixels is 120 instead of 128, and for 256 × 256 pixels it is 140 instead of 128 (denoted with ”or” in Tables 4 and 5).
Revised: Since the number of residual blocks depends on the image resolution, the full dimensionality of z is 120 for 128 × 128 , 140 for 256 × 256 , and 160 for 512 × 512 images. The BigGAN-deep model (Figure 16) differs from BigGAN in several aspects. It uses a simpler variant of skip- z conditioning: instead of ﬁrst splitting z into chunks, we concatenate the entire z with the class embedding, and pass the resulting vector to each residual block through skip connections.

Original: [Figure]
Revised: BigGAN-deep is based on residual blocks with bottlenecks (He et al., 2016), which incorporate two additional 1 × 1 convolutions: the ﬁrst reduces the number of channels by a factor of 4 before the more expensive 3 × 3 convolutions; the second produces the required number of output channels. While BigGAN relies on 1 × 1 convolutions in the skip connections whenever the number of channels needs to change, in BigGAN-deep we use a different strategy aimed at preserving identity throughout the skip connections. In G , where the number of channels needs to be reduced, we simply retain the ﬁrst group of channels and drop the rest to produce the required number of channels.

Original: z ∈ R 120 ∼ N (0 , I ) dense, 4 × 4 × 16 · ch ResBlock up 16 · ch ResBlock up 8 · ch ResBlock up 4 · ch ResBlock up 2 · ch Non-Local Block (64 × 64)
Revised: In D , where the number of channels should be increased, we pass the input channels unperturbed, and concatenate them with the remaining channels produced by a 1 × 1 convolution.

Original: ResBlock up 1 · ch
Revised: As far as the network conﬁguration is concerned, the discriminator is an exact reﬂection of the generator. There are two blocks at each resolution (BigGAN uses one), and as a result BigGAN-deep is four times deeper than BigGAN. Despite their increased depth, the BigGAN-deep models have signiﬁcantly fewer parameters mainly due to the bottleneck structure of their residual blocks. For example, the 128 × 128 BigGAN-deep G and D have 50.4M and 34.6M parameters respectively, while the corresponding original BigGAN models have 70.4M and 88.0M parameters.

Original: BN, ReLU, 3 × 3 conv Tanh (
Revised: All BigGAN-deep models use attention at 64 × 64 resolution, channel width multiplier ch = 128 , and z ∈ R 128 .

Original: [Table]
Revised: [Figure]

Original: Generator

Original: RGB image
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: ResBlock down 1 · ch Non-Local Block (64 × 64) ResBlock down 2 · ch ResBlock down 4 · ch
Revised: [Table]

Original: ResBlock down 8 · ch ResBlock down 16 · ch ResBlock 16· ch
Revised: 16 ch

Original: ReLU Global sum pooling
Revised: ReLU, Global average pooling

Original: Embed( y ) · h + (dense → 1) 
Revised: Embed( y ) · h + (linear → 1) (b) Discriminator

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: down 1 · ch
Revised: → 2 ch

Original: ResBlock down 2 · ch

Original: Non-Local Block (64 × 64)
Revised: Non-Local Block ( 64 × 64 )

Original: ResBlock down  4· ch ResBlock down 8 · ch
Revised: ResBlock down 2 ch → 4 ch ResBlock 4 ch → 4 ch

Original: ResBlock down  8 · ch
Revised: ResBlock down 4 ch → 8 ch ResBlock 8 ch → 8 ch ResBlock down 8 ch → 16 ch ResBlock 16 ch → 16 ch

Original: ResBlock down 16 · ch ResBlock 16 · ch
Revised: ResBlock down 16 ch → 16 ch ResBlock 16 ch → 16 ch

Original: ReLU Global sum pooling
Revised: ReLU, Global average pooling

Original: Embed( y ) · h + (dense → 1) (b) Discriminator 
Revised: Embed( y ) · h + (linear → 1) (b) Discriminator [Table] [Table]

Original: A PPENDIX C E XPERIMENTAL D ETAILS
Revised: A PPENDIX C E XPERIMENTAL D ETAILS

Original: Our basic setup follows SA-GAN (Zhang et al., 2018), and is implemented in TensorFlow (Abadi et al.).
Revised: Our basic setup follows SA-GAN (Zhang et al., 2018), and is implemented in TensorFlow (Abadi et al., 2016).

Original: We employ the architectures detailed in Appendix B, with non-local blocks inserted at a single stage in each network.
Revised: We employ the architectures detailed in Appendix B, with non-local blocks inserted at a single stage in each network.

Original: Both G and D networks are initialized with Orthogonal Initialization (Saxe et al., 2014).
Revised: Both G and D networks are initialized with Orthogonal Initialization (Saxe et al., 2014).

Original: We usethe Adam optimizer (Kingma & Ba, 2014), with  a constant learning rate of 2 · 10 − 4 in D and 5 · 10 − 5 in G; in both networks, β 1 = 0 and β 2 = 0 . 999 .
Revised: We use Adam optimizer (Kingma & Ba, 2014) with β 1 = 0 and β 2 = 0 . 999 and a constant learning rate. For BigGAN models at all resolutions, we use 2 · 10 − 4 in D and 5 · 10 −in G .

Revised: For BigGAN-deep, we use the learning rate of 2 · 10 − 4 in D and 5 · 10 − 5 in G for 128 ×models, and 2 . 5 · 10 − 5 in both D and G for 256 × 256 and 512 × 512 models.

Original: We experimented with the number of D steps per G step (varying it from 1 to 6 ) and found that two D steps per G step gave the best results.
Revised: We experimented with the number of D steps per G step (varying it from 1 to 6 ) and found that two D steps per G step gave the best results.

Original: We use an exponential moving average of the weights of G at sampling time, with a decay rate set to 0.9999.
Revised: We use an exponential moving average of the weights of G at sampling time, with a decay rate set to 0.9999.

Original: We employ cross-replica BatchNorm (Ioffe & Szegedy, 2015) in G , where batch statistics are aggregated across all devices, rather than a single device as in standard implementations.
Revised: We employ cross-replica BatchNorm (Ioffe & Szegedy, 2015) in G , where batch statistics are aggregated across all devices, rather than a single device as in standard implementations.

Original: Spectral
Revised: Spectral

Original: Normalization (Miyato et al., 2018) is used in both G and D , following SA-GAN (Zhang et al., 2018).
Revised: Normalization (Miyato et al., 2018) is used in both G and D , following SA-GAN (Zhang et al., 2018).

Original: We train on a Google TPU  Pod, with the number of cores proportional to the resolution: 128 for 128 × 128, 256 for 256 × 256, and 512 for 512 × 512.
Revised: We train on a Google TPU v3 Pod, with the number of cores proportional to the resolution: 128 for 128 × 128, 256 for 256 × 256, and 512 for 512 × 512.

Original: Training takes between 24 and 48 hours for most models.
Revised: Training takes between 24 and 48 hours for most models.

Original: We increase (cid:15) from the default 10 − 8 to 10 − 4 in BatchNorm and Spectral Norm to mollify low-precision numerical issues.
Revised: We increase (cid:15) from the default 10 − 8 to 10 − 4 in BatchNorm and Spectral Norm to mollify low-precision numerical issues.

Original: We preprocess data by cropping along the long edge and rescaling to a given resolution with area resampling.
Revised: We preprocess data by cropping along the long edge and rescaling to a given resolution with area resampling.

Original: As the ImageNet dataset has many low-resolution images, directly training at 512 ×produces aliased results, so we ﬁlter out all images with a short edge length less than 400 pixels.

Original: Similar to the CelebA-HQ dataset employed by Karras et al.

Original: (2018), this reduces the dataset size to around 200,000 instances.

Original: C. B ATCH N ORM S TATISTICS AND S AMPLING
Revised: C. B ATCH N ORM S TATISTICS AND S AMPLING

Original: The default behavior with batch normalized classiﬁer networks is to use a running average of the activation moments at test time.
Revised: The default behavior with batch normalized classiﬁer networks is to use a running average of the activation moments at test time.

Original: Previous works (Radford et al., 2016) have instead used batch statistics when sampling images.
Revised: Previous works (Radford et al., 2016) have instead used batch statistics when sampling images.

Original: While this is not technically an invalid way to sample, it means that results are dependent on the test batch size (and how many devices it is split across), and further complicates reproducibility.
Revised: While this is not technically an invalid way to sample, it means that results are dependent on the test batch size (and how many devices it is split across), and further complicates reproducibility.

Original: We ﬁnd that this detail is extremely important, with changes in test batch size producing drastic changes in performance.
Revised: We ﬁnd that this detail is extremely important, with changes in test batch size producing drastic changes in performance.

Original: This is further exacerbated when one uses exponential moving averages of G ’s weights for sampling, as the BatchNorm running averages are computed with non-averaged weights and are poor estimates of the activation statistics for the averaged weights.
Revised: This is further exacerbated when one uses exponential moving averages of G ’s weights for sampling, as the BatchNorm running averages are computed with non-averaged weights and are poor estimates of the activation statistics for the averaged weights.

Original: To counteract both these issues, we employ “standing statistics,” where we compute activation statistics at sampling time by running the G through multiple forward passes (typically 100) each with different batches of random noise, and storing means and variances aggregated across all forward passes.
Revised: To counteract both these issues, we employ “standing statistics,” where we compute activation statistics at sampling time by running the G through multiple forward passes (typically 100) each with different batches of random noise, and storing means and variances aggregated across all forward passes.

Original: Analogous to using running statistics, this results in G ’s outputs becoming invariant to batch size and the number of devices, even when producing a single sample.
Revised: Analogous to using running statistics, this results in G ’s outputs becoming invariant to batch size and the number of devices, even when producing a single sample.

Original: C. CIFAR- 
Revised: C. CIFAR-

Original: We run our networks on CIFAR-10 (Krizhevsky & Hinton, 2009) using the settings from Table 1, row 8, and achieve an IS of 9.22 and an FID of 14.73 without truncation.
Revised: We run our networks on CIFAR-10 (Krizhevsky & Hinton, 2009) using the settings from Table 1, row 8, and achieve an IS of 9.22 and an FID of 14.73 without truncation.

Original: C. I NCEPTION S CORES OF I MAGE N ET I MAGES
Revised: C. I NCEPTION S CORES OF I MAGE N ET I MAGES

Original: We compute the IS for both the training and validation sets of ImageNet.
Revised: We compute the IS for both the training and validation sets of ImageNet.

Original: At 128 × 128 the training data has an IS of 233, and the validation data has an IS of 166.
Revised: At 128 × 128 the training data has an IS of 233, and the validation data has an IS of 166.

Original: At 256 × 256 the training data has an
Revised: At 256 × 256 the training data has an

Original: IS of 377, and the validation data has an IS of 234.
Revised: IS of 377, and the validation data has an IS of 234.

Original: At 512 × 512 the training data has an IS of 348, and the validation data has an IS of 241.
Revised: At 512 × 512 the training data has an IS of 348, and the validation data has an IS of 241.

Original: The discrepancy between training and validation scores is due to the Inception classiﬁer having been trained on the training data, resulting in high-conﬁdence outputs that are preferred by the Inception Score.A PPENDIX D A DDITIONAL P LOTS [Figure] [Figure]
Revised: The discrepancy between training and validation scores is due to the Inception classiﬁer having been trained on the training data, resulting in high-conﬁdence outputs that are preferred by the Inception Score.

Original: I n t e r n a l D a t a s e t F I

Original: [Figure]

Original: =

Original: [Figure]

Original: Ch=64Ch=

Original: [Figure]

Original: I

Original: [Figure]

Original: t

Original: [Figure]

Original: r n a l D a t a s

Original: [Figure]

Original: t F I D

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX E C HOOSING L ATENT S PACES
Revised: A PPENDIX E C HOOSING L ATENT S PACES

Original: While most previous work has employed N (0 , I ) or U [ − 1 , 1] as the prior for z (the noise input to
Revised: While most previous work has employed N (0 , I ) or U [ − 1 , 1] as the prior for z (the noise input to

Original: G ), we are free to choose any latent distribution from which we can sample.
Revised: G ), we are free to choose any latent distribution from which we can sample.

Original: We explore the choice of latents by considering an array of possible designs, described below.
Revised: We explore the choice of latents by considering an array of possible designs, described below.

Original: For each latent, we provide the intuition behind its design and brieﬂy describe how it performs when used as a drop-in replacement for z ∼ N (0 , I ) in an SA-GAN baseline.
Revised: For each latent, we provide the intuition behind its design and brieﬂy describe how it performs when used as a drop-in replacement for z ∼ N (0 , I ) in an SA-GAN baseline.

Original: As the Truncation Trick proved more beneﬁcial than switching to any of these latents, we do not perform a full ablation study, and employ z ∼ N (0 , I ) for our main results to take full advantage of truncation.
Revised: As the Truncation Trick proved more beneﬁcial than switching to any of these latents, we do not perform a full ablation study, and employ z ∼ N (0 , I ) for our main results to take full advantage of truncation.

Original: The two latents which we ﬁnd to work best without truncation are Bernoulli { 0 , 1 } and Censored Normal max ( N (0 , I ) , 0) , both of which improve speed of training and lightly improve ﬁnal performance, but are less amenable to truncation.
Revised: The two latents which we ﬁnd to work best without truncation are Bernoulli { 0 , 1 } and Censored Normal max ( N (0 , I ) , 0) , both of which improve speed of training and lightly improve ﬁnal performance, but are less amenable to truncation.

Original: We also ablate the choice of latent space dimensonality (which by default is z ∈ R 128 ), ﬁnding that we are able to successfully train with latent dimensions as low as z ∈ R 8 , and that with z ∈ R 32 we see a minimal drop in performance.
Revised: We also ablate the choice of latent space dimensonality (which by default is z ∈ R 128 ), ﬁnding that we are able to successfully train with latent dimensions as low as z ∈ R 8 , and that with z ∈ R 32 we see a minimal drop in performance.

Original: While this is substantially smaller than many previous works, direct comparison to single-class networks (such as those in Karras et al.
Revised: While this is substantially smaller than many previous works, direct comparison to single-class networks (such as those in Karras et al.

Original: (2018), which employ a z ∈ R 512 latent space on a highly constrained dataset with 30,000 images) is improper, as our networks have additional class information provided as input.
Revised: (2018), which employ a z ∈ R 512 latent space on a highly constrained dataset with 30,000 images) is improper, as our networks have additional class information provided as input.

Original: L ATENTS
Revised: L ATENTS

Original: • N (0 , I ) .
Revised: • N (0 , I ) .

Original: A standard choice of the latent space which we use in the main experiments.
Revised: A standard choice of the latent space which we use in the main experiments.

Original: • U [ − 1 , 1] .
Revised: • U [ − 1 , 1] .

Original: Another standard choice; we ﬁnd that it performs similarly to N (0 , I ) .
Revised: Another standard choice; we ﬁnd that it performs similarly to N (0 , I ) .

Original: • Bernoulli { 0 , 1 } .
Revised: • Bernoulli { 0 , 1 } .

Original: A discrete latent might reﬂect our prior that underlying factors of variation in natural images are not continuous, but discrete (one feature is present, another is not).
Revised: A discrete latent might reﬂect our prior that underlying factors of variation in natural images are not continuous, but discrete (one feature is present, another is not).

Original: This latent outperforms N (0 , I ) (in terms of IS) by 8% and requires 60% fewer iterations.
Revised: This latent outperforms N (0 , I ) (in terms of IS) by 8% and requires 60% fewer iterations.

Original: • max ( N (0 , I ) , 0) , also called Censored Normal.
Revised: • max ( N (0 , I ) , 0) , also called Censored Normal.

Original: This latent is designed to introduce sparsity in the latent space (reﬂecting our prior that certain latent features are sometimes present and sometimes not), but also allow those latents to vary continuously, expressing different degrees of intensity for latents which are active.
Revised: This latent is designed to introduce sparsity in the latent space (reﬂecting our prior that certain latent features are sometimes present and sometimes not), but also allow those latents to vary continuously, expressing different degrees of intensity for latents which are active.

Original: This latent outperforms N (0 , I ) (in terms of IS) by 15-20% and tends to require fewer iterations.
Revised: This latent outperforms N (0 , I ) (in terms of IS) by 15-20% and tends to require fewer iterations.

Original: • Bernoulli {− 1 , 1 } .
Revised: • Bernoulli {− 1 , 1 } .

Original: This latent is designed to be discrete, but not sparse (as the network can learn to activate in response to negative inputs).
Revised: This latent is designed to be discrete, but not sparse (as the network can learn to activate in response to negative inputs).

Original: This latent performs near-identically to N (0 , I ) .
Revised: This latent performs near-identically to N (0 , I ) .

Original: • Independent Categorical in {− 1 , 0 , 1 } , with equal probability.
Revised: • Independent Categorical in {− 1 , 0 , 1 } , with equal probability.

Original: This distribution is chosen to be discrete and have sparsity, but also to allow latents to take on both positive and negative values.
Revised: This distribution is chosen to be discrete and have sparsity, but also to allow latents to take on both positive and negative values.

Original: This latent performs near-identically to N (0 , I ) .
Revised: This latent performs near-identically to N (0 , I ) .

Original: • N (0 , I ) multiplied by Bernoulli { 0 , 1 } .
Revised: • N (0 , I ) multiplied by Bernoulli { 0 , 1 } .

Original: This distribution is chosen to have continuous latent factors which are also sparse (with a peak at zero), similar to Censored Normal but not constrained to be positive.
Revised: This distribution is chosen to have continuous latent factors which are also sparse (with a peak at zero), similar to Censored Normal but not constrained to be positive.

Original: This latent performs near-identically to N (0 , I ) .
Revised: This latent performs near-identically to N (0 , I ) .

Original: • Concatenating N (0 , I ) and Bernoulli { 0 , 1 } , each taking half of the latent dimensions.
Revised: • Concatenating N (0 , I ) and Bernoulli { 0 , 1 } , each taking half of the latent dimensions.

Original: This is inspired by Chen et al.
Revised: This is inspired by Chen et al.

Original: and is chosen to allow some factors of variation to be discrete, while others are continuous.
Revised: and is chosen to allow some factors of variation to be discrete, while others are continuous.

Original: This latent outperforms N (0 , I ) by around 5%.
Revised: This latent outperforms N (0 , I ) by around 5%.

Original: • Variance annealing: we sample from N (0 , σI ) , where σ is allowed to vary over training.
Revised: • Variance annealing: we sample from N (0 , σI ) , where σ is allowed to vary over training.

Original: We compared a variety of piecewise schedules and found that starting with σ = 2 and annealing towards σ = 1 over the course of training mildly improved performance.
Revised: We compared a variety of piecewise schedules and found that starting with σ = 2 and annealing towards σ = 1 over the course of training mildly improved performance.

Original: The space of possible variance schedules is large, and we did not explore it in depth – we suspect that a more principled or better-tuned schedule could more strongly impact performance.
Revised: The space of possible variance schedules is large, and we did not explore it in depth – we suspect that a more principled or better-tuned schedule could more strongly impact performance.

Original: • Per-sample variable variance: N (0 , σ i I ) , where σ i ∼ U [ σ l , σ h ] independently for each sample i in a batch, and ( σ l , σ h ) are hyperparameters.
Revised: • Per-sample variable variance: N ( 0 , σ i I ) , where σ i ∼ U [ σ l , σ h ] independently for each sample i in a batch, and ( σ l , σ h ) are hyperparameters.

Original: This distribution was chosen to try and improve amenability to the Truncation Trick by feeding the network noise samples with non-constant variance.
Revised: This distribution was chosen to try and improve amenability to the Truncation Trick by feeding the network noise samples with non-constant variance.

Original: This did not appear to affect performance, but we did not explore it in depth.
Revised: This did not appear to affect performance, but we did not explore it in depth.

Original: One might also consider scheduling ( σ l , σ h ) , similar to variance annealing.
Revised: One might also consider scheduling ( σ l , σ h ) , similar to variance annealing.

Original: A PPENDIX F M ONITORED T

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A PPENDIX G A DDITIONAL D ISCUSSION : S TABILITY AND C OLLAPSE
Revised: A PPENDIX G A DDITIONAL D ISCUSSION : S TABILITY AND C OLLAPSE

Original: In this section, we present and discuss additional investigations into the stability of our models, expanding upon the discussion in Section 4.
Revised: In this section, we present and discuss additional investigations into the stability of our models, expanding upon the discussion in Section 4.

Original: G. I NTERVENING B EFORE C OLLAPSE
Revised: G. I NTERVENING B EFORE C OLLAPSE

Original: The symptoms of collapse are sharp and sudden, with sample quality dropping from its peak to its lowest value over the course of a few hundred iterations.
Revised: The symptoms of collapse are sharp and sudden, with sample quality dropping from its peak to its lowest value over the course of a few hundred iterations.

Original: We can detect this collapse when the singular values in G explode, but while the (unnormalized) singular values grow throughout training, there is no consistent threshold at which collapse occurs.
Revised: We can detect this collapse when the singular values in G explode, but while the (unnormalized) singular values grow throughout training, there is no consistent threshold at which collapse occurs.

Original: This raises the question of whether it is possible to prevent or delay collapse by taking a model checkpoint several thousand iterations before collapse, and continuing training with some hyperparameters modiﬁed (e.g., the learning rate).
Revised: This raises the question of whether it is possible to prevent or delay collapse by taking a model checkpoint several thousand iterations before collapse, and continuing training with some hyperparameters modiﬁed (e.g., the learning rate).

Original: We conducted a range of intervention experiments wherein we took checkpoints of a collapsed model ten or twenty thousand iterations before collapse, changed some aspect of the training setup, then observed whether collapse occurred, when it occurred relative to the original collapse, and the ﬁnal performance attained at collapse.
Revised: We conducted a range of intervention experiments wherein we took checkpoints of a collapsed model ten or twenty thousand iterations before collapse, changed some aspect of the training setup, then observed whether collapse occurred, when it occurred relative to the original collapse, and the ﬁnal performance attained at collapse.

Original: We found that increasing the learning rates (relative to their initial values) in either G or D , or both G and D , led to immediate collapse.
Revised: We found that increasing the learning rates (relative to their initial values) in either G or D , or both G and D , led to immediate collapse.

Original: This occurred even when doubling the learning rates from 2 · 10 −in D and 5 · 10 − 5 in G , to 4 · 10 − 4 in D and 1 · 10 − 4 in G , a setting which is not normally unstable when used as the initial learning rates.
Revised: This occurred even when doubling the learning rates from 2 · 10 −in D and 5 · 10 − 5 in G , to 4 · 10 − 4 in D and 1 · 10 − 4 in G , a setting which is not normally unstable when used as the initial learning rates.

Original: We also tried changing the momentum terms (Adam’s βand β 2 ), or resetting the momentum vectors to zero, but this tended to either make no difference or, when increasing the momentum, cause immediate collapse.
Revised: We also tried changing the momentum terms (Adam’s βand β 2 ), or resetting the momentum vectors to zero, but this tended to either make no difference or, when increasing the momentum, cause immediate collapse.

Original: We found that decreasing the learning rate in G , but keeping the learning rate in D unchanged could delay collapse (in some cases by over one hundred thousand iterations), but also crippled training— once the learning rate in G was decayed, performance either stayed constant or slowly decayed.
Revised: We found that decreasing the learning rate in G , but keeping the learning rate in D unchanged could delay collapse (in some cases by over one hundred thousand iterations), but also crippled training— once the learning rate in G was decayed, performance either stayed constant or slowly decayed.

Original: Conversely, reducing the learning rate in D while keeping G ’s learning rate led to immediate collapse.
Revised: Conversely, reducing the learning rate in D while keeping G ’s learning rate led to immediate collapse.

Original: We hypothesize that this is because of the need for D to remain optimal throughout training—if its learning rate is reduced, it can no longer “keep up” with G , and training collapses.
Revised: We hypothesize that this is because of the need for D to remain optimal throughout training—if its learning rate is reduced, it can no longer “keep up” with G , and training collapses.

Original: With this in mind, we also tried increasing the number of D steps per G step, but this either had no effect, or delayed collapse at the cost of crippling training (similar to decaying G ’s learning rate).
Revised: With this in mind, we also tried increasing the number of D steps per G step, but this either had no effect, or delayed collapse at the cost of crippling training (similar to decaying G ’s learning rate).

Original: To further illuminate these dynamics, we construct two additional intervention experiments, one where we freeze G before collapse (by ceasing all parameter updates) and observe whether D remains stable, and the reverse, where we freeze D before collapse and observe whether G remains stable.
Revised: To further illuminate these dynamics, we construct two additional intervention experiments, one where we freeze G before collapse (by ceasing all parameter updates) and observe whether D remains stable, and the reverse, where we freeze D before collapse and observe whether G remains stable.

Original: We ﬁnd that when G is frozen, D remains stable, and slowly reduces both components of its loss towards zero.
Revised: We ﬁnd that when G is frozen, D remains stable, and slowly reduces both components of its loss towards zero.

Original: However, when D is frozen, G immediately and dramatically collapses, maxing out
Revised: However, when D is frozen, G immediately and dramatically collapses, maxing out

Original: D ’s loss to values upwards of 300, compared to the normal range of 0 to 3.
Revised: D ’s loss to values upwards of 300, compared to the normal range of 0 to 3.

Original: This leads to two conclusions: ﬁrst, as has been noted in previous works (Miyato et al., 2018;
Revised: This leads to two conclusions: ﬁrst, as has been noted in previous works (Miyato et al., 2018;

Original: Gulrajani et al., 2017; Zhang et al., 2018), D must remain optimal with respect to G both for stability and to provide useful gradient information.
Revised: Gulrajani et al., 2017; Zhang et al., 2018), D must remain optimal with respect to G both for stability and to provide useful gradient information.

Original: The consequence of G being allowed to win the game is a complete breakdown of the training process, regardless of G ’s conditioning or optimization settings.
Revised: The consequence of G being allowed to win the game is a complete breakdown of the training process, regardless of G ’s conditioning or optimization settings.

Original: Second, favoring D over G (either by training it with a larger learning rate, or for more steps) is insufﬁcient to ensure stability even if D is well-conditioned.
Revised: Second, favoring D over G (either by training it with a larger learning rate, or for more steps) is insufﬁcient to ensure stability even if D is well-conditioned.

Original: This suggests either that in practice, an optimal D is necessary but insufﬁcient for training stability, or that some aspect of the system results in D not being trained towards optimality.
Revised: This suggests either that in practice, an optimal D is necessary but insufﬁcient for training stability, or that some aspect of the system results in D not being trained towards optimality.

Original: With the latter possibility in mind, we take a closer look at the noise in D ’s spectra in the following section.
Revised: With the latter possibility in mind, we take a closer look at the noise in D ’s spectra in the following section.

Original: [Figure]
Revised: [Figure]

Original: If some element of D ’s training process results in undesirable dynamics, it follows that the behavior of D ’s spectra may hold clues as to what that element is.
Revised: If some element of D ’s training process results in undesirable dynamics, it follows that the behavior of D ’s spectra may hold clues as to what that element is.

Original: The top three singular values of D differ from G ’s in that they have a large noise component, tend to grow throughout training but only show a small response to collapse, and the ratio of the ﬁrst two singular values tends to be centered around one, suggesting that the spectra of D have a slow decay.
Revised: The top three singular values of D differ from G ’s in that they have a large noise component, tend to grow throughout training but only show a small response to collapse, and the ratio of the ﬁrst two singular values tends to be centered around one, suggesting that the spectra of D have a slow decay.

Original: When viewed up close (Figure 28), the noise spikes resemble an impulse response: at each spike, the spectra jump upwards, then slowly decrease, with some oscillation.
Revised: When viewed up close (Figure 29), the noise spikes resemble an impulse response: at each spike, the spectra jump upwards, then slowly decrease, with some oscillation.

Original: One possible explanation is that this behavior is a consequence of D memorizing the training data, as suggested by experiments in Section 4.2.
Revised: One possible explanation is that this behavior is a consequence of D memorizing the training data, as suggested by experiments in Section 4.2.

Original: As it approaches perfect memorization, it receives less and less signal from real data, as both the original GAN loss and the hinge loss provide zero gradients when D outputs a conﬁdent and correct prediction for a given example.
Revised: As it approaches perfect memorization, it receives less and less signal from real data, as both the original GAN loss and the hinge loss provide zero gradients when D outputs a conﬁdent and correct prediction for a given example.

Original: If the gradient signal from real data attenuates to zero, this can result in D eventually becoming biased due to exclusively received gradients that encourage its outputs to be negative.
Revised: If the gradient signal from real data attenuates to zero, this can result in D eventually becoming biased due to exclusively received gradients that encourage its outputs to be negative.

Original: If this bias passes a certain threshold, D will eventually misclassify a large number of real examples and receive a large gradient encouraging positive outputs, resulting in the observed impulse responses.
Revised: If this bias passes a certain threshold, D will eventually misclassify a large number of real examples and receive a large gradient encouraging positive outputs, resulting in the observed impulse responses.

Original: This argument suggests several ﬁxes.
Revised: This argument suggests several ﬁxes.

Original: First, one might consider an unbounded loss (such as the
Revised: First, one might consider an unbounded loss (such as the

Original: Wasserstein loss (Arjovsky et al., 2017)) which would not suffer this gradient attentuation.
Revised: Wasserstein loss (Arjovsky et al., 2017)) which would not suffer this gradient attentuation.

Original: We found that even with gradient penalties and brief re-tuning of optimizer hyperparameters, our models did not stably train for more than a few thousand iterations with this loss.
Revised: We found that even with gradient penalties and brief re-tuning of optimizer hyperparameters, our models did not stably train for more than a few thousand iterations with this loss.

Original: We instead explored changing the margin of the hinge loss as a partial compromise: for a given model and minibatch of data, increasing the margin will result in more examples falling within the margin, and thus contributing to the loss.
Revised: We instead explored changing the margin of the hinge loss as a partial compromise: for a given model and minibatch of data, increasing the margin will result in more examples falling within the margin, and thus contributing to the loss.

Original: Training with a smaller margin (by a factor of 2) measurably reduces performance, but training with a larger margin (by up to a factor of 3) does not prevent collapse or reduce the noise in D ’s spectra.
Revised: Training with a smaller margin (by a factor of 2) measurably reduces performance, but training with a larger margin (by up to a factor of 3) does not prevent collapse or reduce the noise in D ’s spectra.

Original: Increasing the margin beyond 3 results in unstable training similar to using the Wasserstein loss.
Revised: Increasing the margin beyond 3 results in unstable training similar to using the Wasserstein loss.

Original: Finally, the memorization argument might suggest that using a smaller D or using dropout in D would improve training by reducing its capacity to memorize, but in practice this degrades training.
Revised: Finally, the memorization argument might suggest that using a smaller D or using dropout in D would improve training by reducing its capacity to memorize, but in practice this degrades training.

Original: A PPENDIX H N EGATIVE R ESULTS
Revised: A PPENDIX H N EGATIVE R ESULTS

Original: We explored a range of novel and existing techniques which ended up degrading or otherwise not affecting performance in our setting.
Revised: We explored a range of novel and existing techniques which ended up degrading or otherwise not affecting performance in our setting.

Original: We report them here; our evaluations for this section are not as thorough as those for the main architectural choices.
Revised: We report them here; our evaluations for this section are not as thorough as those for the main architectural choices.

Original: Our intention in reporting these results is to save time for future work, and to give a more complete picture of our attempts to improve performance or stability.
Revised: Our intention in reporting these results is to save time for future work, and to give a more complete picture of our attempts to improve performance or stability.

Original: We note, however, that these results must be understood to be speciﬁc to the particular setup we used.
Revised: We note, however, that these results must be understood to be speciﬁc to the particular setup we used.

Original: A pitfall of reporting negative results is that one might report that a particular technique doesn’t work, when the reality is that this technique did not have the desired effect when applied in a particular way to a particular problem.
Revised: A pitfall of reporting negative results is that one might report that a particular technique doesn’t work, when the reality is that this technique did not have the desired effect when applied in a particular way to a particular problem.

Original: Drawing overly general conclusions might close off potentially fruitful avenues of research.
Revised: Drawing overly general conclusions might close off potentially fruitful avenues of research.

Original: • We found that doubling the depth (by inserting an additional Residual block after every upor down-sampling block) hampered performance.
Revised: • We found that doubling the depth (by inserting an additional Residual block after every upor down-sampling block) hampered performance.

Original: • We experimented with sharing class embeddings between both G and D (as opposed to just within G ).
Revised: • We experimented with sharing class embeddings between both G and D (as opposed to just within G ).

Original: This is accomplished by replacing D ’s class embedding with a projection from G ’s embeddings, as is done in G ’s BatchNorm layers.
Revised: This is accomplished by replacing D ’s class embedding with a projection from G ’s embeddings, as is done in G ’s BatchNorm layers.

Original: In our initial experiments this seemed to help and accelerate training, but we found this trick scaled poorly and was sensitive to optimization hyperparameters, particularly the choice of number of D steps per G step.
Revised: In our initial experiments this seemed to help and accelerate training, but we found this trick scaled poorly and was sensitive to optimization hyperparameters, particularly the choice of number of D steps per G step.

Original: • We tried replacing BatchNorm in G with WeightNorm (Salimans & Kingma, 2016), but this crippled training.
Revised: • We tried replacing BatchNorm in G with WeightNorm (Salimans & Kingma, 2016), but this crippled training.

Original: We also tried removing BatchNorm and only having Spectral Normalization, but this also crippled training.
Revised: We also tried removing BatchNorm and only having Spectral Normalization, but this also crippled training.

Original: • We tried adding BatchNorm to D (both class-conditional and unconditional) in addition to Spectral Normalization, but this crippled training.
Revised: • We tried adding BatchNorm to D (both class-conditional and unconditional) in addition to Spectral Normalization, but this crippled training.

Original: • We tried varying the choice of location of the attention block in G and D (and inserting multiple attention blocks at different resolutions) but found that at 128 × 128 there was no noticeable beneﬁt to doing so, and compute and memory costs increased substantially.
Revised: • We tried varying the choice of location of the attention block in G and D (and inserting multiple attention blocks at different resolutions) but found that at 128 × 128 there was no noticeable beneﬁt to doing so, and compute and memory costs increased substantially.

Original: We found a beneﬁt to moving the attention block up one stage when moving to 256 × 256, which is in line with our expectations given the increased resolution.
Revised: We found a beneﬁt to moving the attention block up one stage when moving to 256 × 256, which is in line with our expectations given the increased resolution.

Original: • We tried using ﬁlter sizes of 5 or 7 instead of 3 in either G or D or both.
Revised: • We tried using ﬁlter sizes of 5 or 7 instead of 3 in either G or D or both.

Original: We found that having a ﬁlter size of 5 in G only provided a small improvement over the baseline but came at an unjustiﬁable compute cost.
Revised: We found that having a ﬁlter size of 5 in G only provided a small improvement over the baseline but came at an unjustiﬁable compute cost.

Original: All other settings degraded performance.
Revised: All other settings degraded performance.

Original: • We tried varying the dilation for convolutional ﬁlters in both G and D at 128 × 128, but found that even a small amount of dilation in either network degraded performance.
Revised: • We tried varying the dilation for convolutional ﬁlters in both G and D at 128 × 128, but found that even a small amount of dilation in either network degraded performance.

Original: • We tried bilinear upsampling in G in place of nearest-neighbors upsampling, but this degraded performance.
Revised: • We tried bilinear upsampling in G in place of nearest-neighbors upsampling, but this degraded performance.

Original: • In some of our models, we observed class-conditional mode collapse, where the model would only output one or two samples for a subset of classes but was still able to generate samples for all other classes.
Revised: • In some of our models, we observed class-conditional mode collapse, where the model would only output one or two samples for a subset of classes but was still able to generate samples for all other classes.

Original: We noticed that the collapsed classes had embedings which had become very large relative to the other embeddings, and attempted to ameliorate this issue by applying weight decay to the shared embedding only.
Revised: We noticed that the collapsed classes had embedings which had become very large relative to the other embeddings, and attempted to ameliorate this issue by applying weight decay to the shared embedding only.

Original: We found that small amounts of weight decay ( 10 − 6 ) instead degraded performance, and that only even smaller values ( 10 − 8 ) did not degrade performance, but these values were also too small to prevent the class vectors from exploding.
Revised: We found that small amounts of weight decay ( 10 − 6 ) instead degraded performance, and that only even smaller values ( 10 − 8 ) did not degrade performance, but these values were also too small to prevent the class vectors from exploding.

Original: Higher-resolution models appear to be more resilient to this problem, and none of our ﬁnal models appear to suffer from this type of collapse.
Revised: Higher-resolution models appear to be more resilient to this problem, and none of our ﬁnal models appear to suffer from this type of collapse.

Original: • We experimented with using MLPs instead of linear projections from G ’s class embeddings to its BatchNorm gains and biases, but did not ﬁnd any beneﬁt to doing so.
Revised: • We experimented with using MLPs instead of linear projections from G ’s class embeddings to its BatchNorm gains and biases, but did not ﬁnd any beneﬁt to doing so.

Original: We also experimented with Spectrally Normalizing these MLPs, and with providing these (and the linear projections) with a bias at their output, but did not notice any beneﬁt.
Revised: We also experimented with Spectrally Normalizing these MLPs, and with providing these (and the linear projections) with a bias at their output, but did not notice any beneﬁt.

Original: • We tried gradient norm clipping (both the global variant typically used in recurrent networks, and a local version where the clipping value is determined on a per-parameter basis) but found this did not alleviate instability.
Revised: • We tried gradient norm clipping (both the global variant typically used in recurrent networks, and a local version where the clipping value is determined on a per-parameter basis) but found this did not alleviate instability.

Original: A PPENDIX I H YPERPARAMETERS
Revised: A PPENDIX I H YPERPARAMETERS

Original: We performed various hyperparameter sweeps in this work:
Revised: We performed various hyperparameter sweeps in this work:

Original: • We swept the Cartesian product of the learning rates for each network through [ 10 − 5 , 5 · 10 − 5 , 10 − 4 , 2 · 10 − 4 , 4 · 10 − 4 , 8 · 10 − 4 , 10 − 3 ], and initially found that the SA-GAN settings ( G ’s learning rate 10 − 4 , D ’s learning rate 4 · 10 − 4 ) were optimal at lower batch sizes; we did not repeat this sweep at higher batch sizes but did try halving and doubling the learning rate, arriving at the halved settings used for our experiments.
Revised: • We swept the Cartesian product of the learning rates for each network through [ 10 − 5 , 5 · 10 − 5 , 10 − 4 , 2 · 10 − 4 , 4 · 10 − 4 , 8 · 10 − 4 , 10 − 3 ], and initially found that the SA-GAN settings ( G ’s learning rate 10 − 4 , D ’s learning rate 4 · 10 − 4 ) were optimal at lower batch sizes; we did not repeat this sweep at higher batch sizes but did try halving and doubling the learning rate, arriving at the halved settings used for our experiments.

Original: • We swept the R1 gradient penalty strength through [ 10 − 3 , 10 − 2 , 10 − 1 , 0 . 5 , 1 , 2 , 3 , 5 , 10 ].
Revised: • We swept the R1 gradient penalty strength through [ 10 − 3 , 10 − 2 , 10 − 1 , 0 . 5 , 1 , 2 , 3 , 5 , 10 ].

Original: We ﬁnd that the strength of the penalty correlates negatively with performance, but that settings above 0 .
Revised: We ﬁnd that the strength of the penalty correlates negatively with performance, but that settings above 0 .

Original: 5 impart training stability.
Revised: 5 impart training stability.

Original: • We swept the keep probabilities for DropOut in the ﬁnal layer of D through [ 0 . 5 , 0 . 6 , 0 .
Revised: • We swept the keep probabilities for DropOut in the ﬁnal layer of D through [ 0 . 5 , 0 . 6 , 0 .

Original: We ﬁnd that DropOut has a similar stabilizing effect to R1 but also degrades performance.
Revised: We ﬁnd that DropOut has a similar stabilizing effect to R1 but also degrades performance.

Original: • We swept D ’s Adam β 1 parameter through [ 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 ] and found it to have a light regularization effect similar to DropOut, but not to signiﬁcantly improve results.
Revised: • We swept D ’s Adam β 1 parameter through [ 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 ] and found it to have a light regularization effect similar to DropOut, but not to signiﬁcantly improve results.

Original: [Table]
Revised: [Table]

Original: • We swept the strength of the modiﬁed Orthogonal Regularization penalty in G through [ 10 − 5 , 5 · 10 − 5 , 10 − 4 , 5 · 10 − 4 , 10 − 3 , 10 − 2 ], and selected 10 − 4 .
Revised: • We swept the strength of the modiﬁed Orthogonal Regularization penalty in G through [ 10 − 5 , 5 · 10 − 5 , 10 − 4 , 5 · 10 − 4 , 10 − 3 , 10 − 2 ], and selected 10 − 4 .