Original: D IFFUSION -GAN: T RAINING GAN S WITH D IFFUSION
Revised: D IFFUSION -GAN: T RAINING GAN S WITH D IFFUSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative adversarial networks (GANs) are challenging to train stably, and a promising remedy of injecting instance noise into the discriminator input has not been very effective in practice.
Revised: Generative adversarial networks (GANs) are challenging to train stably, and a promising remedy of injecting instance noise into the discriminator input has not been very effective in practice.

Original: In this paper, we propose Diffusion-GAN, a novel
Revised: In this paper, we propose Diffusion-GAN, a novel

Original: GAN framework that leverages a forward diffusion chain to generate Gaussianmixture distributed instance noise.
Revised: GAN framework that leverages a forward diffusion chain to generate Gaussianmixture distributed instance noise.

Original: Diffusion-GAN consists of three components, including an adaptive diffusion process, a diffusion timestep-dependent discriminator, and a generator.
Revised: Diffusion-GAN consists of three components, including an adaptive diffusion process, a diffusion timestep-dependent discriminator, and a generator.

Original: Both the observed and generated data are diffused by the same adaptive diffusion process.
Revised: Both the observed and generated data are diffused by the same adaptive diffusion process.

Original: At each diffusion timestep, there is a different noise-to-data ratio and the timestep-dependent discriminator learns to distinguish the diffused real data from the diffused generated data.
Revised: At each diffusion timestep, there is a different noise-to-data ratio and the timestep-dependent discriminator learns to distinguish the diffused real data from the diffused generated data.

Original: The generator learns from the discriminator’s feedback by backpropagating through the forward diffusion chain, whose length is adaptively adjusted to balance the noise and data levels.
Revised: The generator learns from the discriminator’s feedback by backpropagating through the forward diffusion chain, whose length is adaptively adjusted to balance the noise and data levels.

Original: We theoretically show that the discriminator’s timestep-dependent strategy gives consistent and helpful guidance to the generator, enabling it to match the true data distribution.
Revised: We theoretically show that the discriminator’s timestep-dependent strategy gives consistent and helpful guidance to the generator, enabling it to match the true data distribution.

Original: We demonstrate the advantages of Diffusion-GAN over strong GAN baselines on various datasets, showing that it can produce more realistic images with higher stability and data efficiency than state-of-the-art GANs.
Revised: We demonstrate the advantages of Diffusion-GAN over strong GAN baselines on various datasets, showing that it can produce more realistic images with higher stability and data efficiency than state-of-the-art GANs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative adversarial networks (GANs) (Goodfellow et al., 2014) and their variants (Brock et al., 2018; Karras et al., 2019; 2020a; Zhao et al., 2020) have achieved great success in synthesizing photo-realistic high-resolution images.
Revised: Generative adversarial networks (GANs) (Goodfellow et al., 2014) and their variants (Brock et al., 2018; Karras et al., 2019; 2020a; Zhao et al., 2020) have achieved great success in synthesizing photo-realistic high-resolution images.

Original: GANs in practice, however, are known to suffer from a variety of issues ranging from non-convergence and training instability to mode collapse (Arjovsky and Bottou, 2017; Mescheder et al., 2018).
Revised: GANs in practice, however, are known to suffer from a variety of issues ranging from non-convergence and training instability to mode collapse (Arjovsky and Bottou, 2017; Mescheder et al., 2018).

Original: As a result, a wide array of analyses and modifications has been proposed for GANs, including improving the network architectures (Karras et al., 2019;
Revised: As a result, a wide array of analyses and modifications has been proposed for GANs, including improving the network architectures (Karras et al., 2019;

Original: Radford et al., 2016; Sauer et al., 2021; Zhang et al., 2019), gaining theoretical understanding of
Revised: Radford et al., 2016; Sauer et al., 2021; Zhang et al., 2019), gaining theoretical understanding of

Original: GAN training (Arjovsky and Bottou, 2017; Heusel et al., 2017; Mescheder et al., 2017; 2018), changing the objective functions (Arjovsky et al., 2017; Bellemare et al., 2017; Deshpande et al., 2018;
Revised: GAN training (Arjovsky and Bottou, 2017; Heusel et al., 2017; Mescheder et al., 2017; 2018), changing the objective functions (Arjovsky et al., 2017; Bellemare et al., 2017; Deshpande et al., 2018;

Original: Li et al., 2017a;
Revised: Li et al., 2017a;

Original: Nowozin et al., 2016; Zheng and Zhou, 2021; Yang et al., 2021), regularizing the weights and/or gradients (Arjovsky et al., 2017; Fedus et al., 2018; Mescheder et al., 2018;
Revised: Nowozin et al., 2016; Zheng and Zhou, 2021; Yang et al., 2021), regularizing the weights and/or gradients (Arjovsky et al., 2017; Fedus et al., 2018; Mescheder et al., 2018;

Original: Miyato et al., 2018a; Roth et al., 2017; Salimans et al., 2016), utilizing side information (Wang et al., 2018; Zhang et al., 2017; 2020b), adding a mapping from the data to latent representation (Donahue et al., 2016;
Revised: Miyato et al., 2018a; Roth et al., 2017; Salimans et al., 2016), utilizing side information (Wang et al., 2018; Zhang et al., 2017; 2020b), adding a mapping from the data to latent representation (Donahue et al., 2016;

Original: Dumoulin et al., 2016; Li et al., 2017b), and applying differentiable data augmentation (Karras et al., 2020a; Zhang et al., 2020a; Zhao et al., 2020).
Revised: Dumoulin et al., 2016; Li et al., 2017b), and applying differentiable data augmentation (Karras et al., 2020a; Zhang et al., 2020a; Zhao et al., 2020).

Original: A simple technique to stabilize GAN training is to inject instance noise, i.e. , to add noise to the discriminator input, which can widen the support of both the generator and discriminator distributions and prevent the discriminator from overfitting (Arjovsky and Bottou, 2017;
Revised: A simple technique to stabilize GAN training is to inject instance noise, i.e. , to add noise to the discriminator input, which can widen the support of both the generator and discriminator distributions and prevent the discriminator from overfitting (Arjovsky and Bottou, 2017;

Original: Sønderby et al., 2017).
Revised: Sønderby et al., 2017).

Original: However, this technique is hard to implement in practice, as finding a suitable noise distribution is challenging (Arjovsky and Bottou, 2017).
Revised: However, this technique is hard to implement in practice, as finding a suitable noise distribution is challenging (Arjovsky and Bottou, 2017).

Original: Roth et al.
Revised: Roth et al.

Original: (2017) show that adding instance noise to the high-dimensional discriminator input does not work well, and propose to approximate it by adding a zero-centered gradient penalty on the discriminator.
Revised: (2017) show that adding instance noise to the high-dimensional discriminator input does not work well, and propose to approximate it by adding a zero-centered gradient penalty on the discriminator.

Original: This approach is theoretically and empirically shown to converge in Mescheder et al.
Revised: This approach is theoretically and empirically shown to converge in Mescheder et al.

Original: (2018), who also demonstrate that adding zero-centered gradient penalties to non-saturating GANs can result in stable training and better or comparable generation quality compared to WGAN-GP (Arjovsky et al., 2017).
Revised: (2018), who also demonstrate that adding zero-centered gradient penalties to non-saturating GANs can result in stable training and better or comparable generation quality compared to WGAN-GP (Arjovsky et al., 2017).

Original: However, Brock et al.
Revised: However, Brock et al.

Original: (2018) caution that zero-centered gradient penalties and other similar regularization methods may stabilize training at the cost of generation performance.
Revised: (2018) caution that zero-centered gradient penalties and other similar regularization methods may stabilize training at the cost of generation performance.

Original: To the best of our knowledge, there has
Revised: To the best of our knowledge, there has

Original: [Figure]
Revised: [Figure]

Original: been no existing work that is able to empirically demonstrate the success of using instance noise in
Revised: been no existing work that is able to empirically demonstrate the success of using instance noise in

Original: GAN training on high-dimensional image data.
Revised: GAN training on high-dimensional image data.

Original: To inject proper instance noise that can facilitate GAN training, we introduce Diffusion-GAN, which uses a diffusion process to generate Gaussian-mixture distributed instance noise.
Revised: To inject proper instance noise that can facilitate GAN training, we introduce Diffusion-GAN, which uses a diffusion process to generate Gaussian-mixture distributed instance noise.

Original: We show a graphical representation of Diffusion-GAN in Figure 1.
Revised: We show a graphical representation of Diffusion-GAN in Figure 1.

Original: In Diffusion-GAN, the input to the diffusion process is either a real or a generated image, and the diffusion process consists of a series of steps that gradually add noise to the image.
Revised: In Diffusion-GAN, the input to the diffusion process is either a real or a generated image, and the diffusion process consists of a series of steps that gradually add noise to the image.

Original: The number of diffusion steps is not fixed, but depends on the data and the generator.
Revised: The number of diffusion steps is not fixed, but depends on the data and the generator.

Original: We also design the diffusion process to be differentiable, which means that we can compute the derivative of the output with respect to the input.
Revised: We also design the diffusion process to be differentiable, which means that we can compute the derivative of the output with respect to the input.

Original: This allows us to propagate the gradient from the discriminator to the generator through the diffusion process, and update the generator accordingly.
Revised: This allows us to propagate the gradient from the discriminator to the generator through the diffusion process, and update the generator accordingly.

Original: Unlike vanilla GANs, which compare the real and generated images directly,
Revised: Unlike vanilla GANs, which compare the real and generated images directly,

Original: Diffusion-GAN compares the noisy versions of them, which are obtained by sampling from the
Revised: Diffusion-GAN compares the noisy versions of them, which are obtained by sampling from the

Original: Gaussian mixture distribution over the diffusion steps, with the help of our timestep-dependent discriminator.
Revised: Gaussian mixture distribution over the diffusion steps, with the help of our timestep-dependent discriminator.

Original: This distribution has the property that its components have different noise-to-data ratios, which means that some components add more noise than others.
Revised: This distribution has the property that its components have different noise-to-data ratios, which means that some components add more noise than others.

Original: By sampling from this distribution, we can achieve two benefits: first, we can stabilize the training by easing the problem of vanishing gradient, which occurs when the data and generator distributions are too different; second, we can augment the data by creating different noisy versions of the same image, which can improve the data efficiency and the diversity of the generator.
Revised: By sampling from this distribution, we can achieve two benefits: first, we can stabilize the training by easing the problem of vanishing gradient, which occurs when the data and generator distributions are too different; second, we can augment the data by creating different noisy versions of the same image, which can improve the data efficiency and the diversity of the generator.

Original: We provide a theoretical analysis to support our method, and show that the min-max objective function of Diffusion-GAN, which measures the difference between the data and generator distributions, is continuous and differentiable everywhere.
Revised: We provide a theoretical analysis to support our method, and show that the min-max objective function of Diffusion-GAN, which measures the difference between the data and generator distributions, is continuous and differentiable everywhere.

Original: This means that the generator in theory can always receive a useful gradient from the discriminator, and improve its performance.
Revised: This means that the generator in theory can always receive a useful gradient from the discriminator, and improve its performance.

Original: Our main contributions include: 1) We show both theoretically and empirically how the diffusion process can be utilized to provide a model- and domain-agnostic differentiable augmentation, enabling data-efficient and leaking-free stable GAN training.
Revised: Our main contributions include: 1) We show both theoretically and empirically how the diffusion process can be utilized to provide a model- and domain-agnostic differentiable augmentation, enabling data-efficient and leaking-free stable GAN training.

Original: 2) Extensive experiments show that
Revised: 2) Extensive experiments show that

Original: Diffusion-GAN boosts the stability and generation performance of strong baselines, including StyleGAN2 (Karras et al., 2020b), Projected GAN (Sauer et al., 2021), and InsGen (Yang et al., 2021), achieving state-of-the-art results in synthesizing photo-realistic images, as measured by both the
Revised: Diffusion-GAN boosts the stability and generation performance of strong baselines, including StyleGAN2 (Karras et al., 2020b), Projected GAN (Sauer et al., 2021), and InsGen (Yang et al., 2021), achieving state-of-the-art results in synthesizing photo-realistic images, as measured by both the

Original: Fr´echet Inception Distance (FID) (Heusel et al., 2017) and Recall score (Kynk¨a¨anniemi et al., 2019).
Revised: Fr´echet Inception Distance (FID) (Heusel et al., 2017) and Recall score (Kynk¨a¨anniemi et al., 2019).

Original:  P RELIMINARIES : GAN S AND DIFFUSION - BASED GENERATIVE MODELS
Revised:  P RELIMINARIES : GAN S AND DIFFUSION - BASED GENERATIVE MODELS

Original: GANs (Goodfellow et al., 2014) are a class of generative models that aim to learn the data distribution p ( x ) of a target dataset by setting up a min-max game between two neural networks: a generator and a discriminator.
Revised: GANs (Goodfellow et al., 2014) are a class of generative models that aim to learn the data distribution p ( x ) of a target dataset by setting up a min-max game between two neural networks: a generator and a discriminator.

Original: The generator G takes as input a random noise vector z sampled from a simple prior distribution p ( z ) , such as a standard normal or uniform distribution, and tries to produce realistic-looking samples
Revised: The generator G takes as input a random noise vector z sampled from a simple prior distribution p ( z ) , such as a standard normal or uniform distribution, and tries to produce realistic-looking samples

Original: G ( z ) that resemble the data.
Revised: G ( z ) that resemble the data.

Original: The discriminator D receives either a real data sample x drawn from p ( x ) or a fake sample G ( z ) generated by G , and tries to correctly classify them as real or fake.
Revised: The discriminator D receives either a real data sample x drawn from p ( x ) or a fake sample G ( z ) generated by G , and tries to correctly classify them as real or fake.

Original: The goal of G is to fool D into making mistakes, while the goal of D is to accurately distinguish G ( z ) from x .
Revised: The goal of G is to fool D into making mistakes, while the goal of D is to accurately distinguish G ( z ) from x .

Original: The min-max objective function of GANs is given by
Revised: The min-max objective function of GANs is given by

Original: [Equation]
Revised: [Equation]

Original: In practice, this vanilla objective function is often modified to improve the stability and performance of GANs(Goodfellow et al., 2014; Miyato et al., 2018a; Fedus et al., 2018), but the general idea of adversarial learning between G and D remains the same.
Revised: In practice, this vanilla objective function is often modified to improve the stability and performance of GANs(Goodfellow et al., 2014; Miyato et al., 2018a; Fedus et al., 2018), but the general idea of adversarial learning between G and D remains the same.

Original: Diffusion-based generative models (Ho et al., 2020; Sohl-Dickstein et al., 2015; Song and Ermon, 2019) assume p θ ( x 0 ) :
Revised: Diffusion-based generative models (Ho et al., 2020b; Sohl-Dickstein et al., 2015; Song and Ermon, 2019) assume p θ ( x 0 ) :

Original: = (cid:82) p θ ( x 0: T ) d x 1: T , where x 1 , . .
Revised: = (cid:82) p θ ( x 0: T ) d x 1: T , where x 1 , . .

Original: , x T are latent variables of the same dimensionality as the data x 0 ∼ p ( x 0 ) .
Revised: , x T are latent variables of the same dimensionality as the data x 0 ∼ p ( x 0 ) .

Original: There is a forward diffusion chain that gradually adds noise to the data x 0 ∼ q ( x 0 ) in T steps with pre-defined variance schedule β t and variance σ 2 : q ( x 1: T | x 0 ) :
Revised: There is a forward diffusion chain that gradually adds noise to the data x 0 ∼ q ( x 0 ) in T steps with pre-defined variance schedule β t and variance σ 2 : q ( x 1: T | x 0 ) :

Original: = (cid:81)
Revised: = (cid:81)

Original: Tt =
Revised: Tt =

Original: 1 q ( x t | x t − 1 ) , q ( x t |x t − 1 ) :
Revised: 1 q ( x t | x t − 1 ) , q [Equation] t |

Original: =
Revised: [Equation] 1 ) := [Equation] [Equation]

Original: N ( x t ; √ 1 − β t x t − 1 , β t σ 2 I ) .
Revised: t [Equation] [Equation] t σ 2 I ) .

Original: A notable property is that x t at an arbitrary time-step t can be sampled in closed form as q ( x t | x 0 ) =
Revised: A notable property is that x t at an arbitrary time-step t can be sampled in closed form as q ( x t | x 0 ) =

Original: N ( x t ; √ ¯ α t x 0 , (1 − ¯ α t ) σ 2 I ) , where α t := 1 − β t , ¯ α t :=
Revised: N ( x t ; √ ¯ α t x 0 , (1 − ¯ α t ) σ 2 I ) , where α t := 1 − β t , ¯ α t :=

Original: (cid:81) ts =1 α s .
Revised: (cid:81) ts =1 α s .

Original: [Equation]
Revised: [Equation]

Original: A variational lower bound (Blei et al., 2017) is then used to optimize the reverse diffusion chain as p θ ( x 0: T ) :
Revised: A variational lower bound (Blei et al., 2017) is then used to optimize the reverse diffusion chain as p θ ( x 0: T ) :

Original: = N ( x T ; 0 , σ 2 I ) (cid:81)
Revised: = N ( x T ; 0 , σ 2 I ) (cid:81)

Original: Tt =1 p θ ( x t − 1 | x t ) .
Revised: Tt =1 p θ ( x t − 1 | x t ) .

Original:  D IFFUSION -GAN: M ETHOD AND T HEORETICAL A NALYSIS
Revised:  D IFFUSION -GAN: M ETHOD AND T HEORETICAL A NALYSIS

Original: To construct Diffusion-GAN, we describe how to inject instance noise via diffusion, how to train the generator by backpropagating through the forward diffusion process, and how to adaptively adjust the diffusion intensity.
Revised: To construct Diffusion-GAN, we describe how to inject instance noise via diffusion, how to train the generator by backpropagating through the forward diffusion process, and how to adaptively adjust the diffusion intensity.

Original: We further provide theoretical analysis illustrated with a toy example.
Revised: We further provide theoretical analysis illustrated with a toy example.

Original: 3. I NSTANCE NOISE INJECTION VIA DIFFUSION
Revised: 3. I NSTANCE NOISE INJECTION VIA DIFFUSION

Original: We aim to generate realistic samples x g from a generator network G that maps a latent variable z sampled from a simple prior distribution p ( z ) to a high-dimensional data space, such as images.
Revised: We aim to generate realistic samples x g from a generator network G that maps a latent variable z sampled from a simple prior distribution p ( z ) to a high-dimensional data space, such as images.

Original: The distribution of generator samples x g = G ( z ) , z ∼ p ( z ) is denoted by p g ( x ) =
Revised: The distribution of generator samples x g = G ( z ) , z ∼ p ( z ) is denoted by p g ( x ) =

Original: (cid:82) p ( x g | z ) p ( z )
Revised: (cid:82) p ( x g | z ) p ( z )

Original: d z .
Revised: d z .

Original: To make the generator more robust and diverse, we inject instance noise into the generated samples x g by applying a diffusion process that adds Gaussian noise at each step.
Revised: To make the generator more robust and diverse, we inject instance noise into the generated samples x g by applying a diffusion process that adds Gaussian noise at each step.

Original: The diffusion process can be seen as a Markov chain that starts from the original sample x and gradually erases its information until reaching a noise level σ 2 after T steps.
Revised: The diffusion process can be seen as a Markov chain that starts from the original sample x and gradually erases its information until reaching a noise level σ 2 after T steps.

Original: We define a mixture distribution q ( y | x ) that models the noisy samples y obtained at any step of the diffusion process, with a mixture weight π t for each step t .
Revised: We define a mixture distribution q ( y | x ) that models the noisy samples y obtained at any step of the diffusion process, with a mixture weight π t for each step t .

Original: The mixture components q ( y | x , t ) are Gaussian distributions with mean proportional to x and variance depending on the noise level at step t .
Revised: The mixture components q ( y | x , t ) are Gaussian distributions with mean proportional to x and variance depending on the noise level at step t .

Original: We use the same diffusion process and mixture distribution for both the real samples x ∼ p ( x ) and the generated samples x g ∼ p
Revised: We use the same diffusion process and mixture distribution for both the real samples x ∼ p ( x ) and the generated samples x g ∼ p

Original: g ( x ) .
Revised: g ( x ) .

Original: More specifically, the diffusion-induced mixture distributions are expressed as
Revised: More specifically, the diffusion-induced mixture distributions are expressed as

Original: [Equation]
Revised: [Equation]

Original: where q ( y | x ) is a T -component mixture distribution, the mixture weights π t are non-negative and sum to one, and the mixture components q ( y | x , t ) are obtained via diffusion as in Equation (1), expressed as
Revised: where q ( y | x ) is a T -component mixture distribution, the mixture weights π t are non-negative and sum to one, and the mixture components q ( y | x , t ) are obtained via diffusion as in Equation (1), expressed as

Original: [Equation]
Revised: [Equation]

Original: Samples from this mixture can be drawn as t ∼ p π :=
Revised: Samples from this mixture can be drawn as t ∼ p π :=

Original: Discrete ( π 1 , . .
Revised: Discrete ( π 1 , . .

Original: . , π T ) , y ∼ q ( y | x , t ) .
Revised: . , π T ) , y ∼ q ( y | x , t ) .

Original: By sampling y from this mixture distribution, we can obtain noisy versions of both real and generated samples with varying degrees of noise.
Revised: By sampling y from this mixture distribution, we can obtain noisy versions of both real and generated samples with varying degrees of noise.

Original: The more steps we take in the diffusion process, the more noise we add to y and the less information we preserve from x .
Revised: The more steps we take in the diffusion process, the more noise we add to y and the less information we preserve from x .

Original: We can then use this diffusion-induced mixture distribution to train a timestep-dependent discriminator D that distinguishes between real and generated noisy samples, and a generator G that matches the distribution of generated noisy samples to the distribution of real noisy samples.
Revised: We can then use this diffusion-induced mixture distribution to train a timestep-dependent discriminator D that distinguishes between real and generated noisy samples, and a generator G that matches the distribution of generated noisy samples to the distribution of real noisy samples.

Original: Next we introduce Diffusion-GAN that trains its discriminator and generator with the help of the diffusion-induced mixture distribution.
Revised: Next we introduce Diffusion-GAN that trains its discriminator and generator with the help of the diffusion-induced mixture distribution.

Original: 3. A DVERSARIAL T RAINING
Revised: 3. A DVERSARIAL T RAINING

Original: The Diffusion-GAN trains its generator and discriminator by solving a min-max game objective as
Revised: The Diffusion-GAN trains its generator and discriminator by solving a min-max game objective as

Original: V
Revised: V

Original: [Equation]
Revised: [Equation]

Original: Here, p ( x ) is the true data distribution, p π is a discrete distribution that assigns different weights π t to each diffusion step t ∈ { 1 , . .
Revised: Here, p ( x ) is the true data distribution, p π is a discrete distribution that assigns different weights π t to each diffusion step t ∈ { 1 , . .

Original: , T } , and q ( y | x , t ) is the conditional distribution of the perturbed sample y given the original data x and the diffusion step t .
Revised: , T } , and q ( y | x , t ) is the conditional distribution of the perturbed sample y given the original data x and the diffusion step t .

Original: By Equation (2), with Gaussian reparameterization, the perturbation function could be written as y = √
Revised: By Equation (2), with Gaussian reparameterization, the perturbation function could be written as y = √

Original: ¯ α t x + √ 1 − ¯ α t σ ϵ , where 1 − ¯ α t = 1 − (cid:81) ts =

Revised: ¯ α t x + √ 1 − ¯ α t σ ϵ , where 1 − ¯ α t = 1 − (cid:81) ts =

Original: α s is the cumulative noise level at step t , σ is a scale factor, and ϵ ∼ N (0 , I ) is a Gaussian noise.
Revised: α s is the cumulative noise level at step t , σ is a scale factor, and ϵ ∼ N (0 , I ) is a Gaussian noise.

Original: The objective function in Equation (3) encourages the discriminator to assign high probabilities to the perturbed real data and low probabilities to the perturbed generated data, for any diffusion step t .
Revised: The objective function in Equation (3) encourages the discriminator to assign high probabilities to the perturbed real data and low probabilities to the perturbed generated data, for any diffusion step t .

Original: The generator, on the other hand, tries to produce samples that can deceive the discriminator at any diffusion step t .
Revised: The generator, on the other hand, tries to produce samples that can deceive the discriminator at any diffusion step t .

Original: Note that the perturbed generated sample y g ∼ q ( y | G θ ( z ) , t ) can be rewritten as y g =
Revised: Note that the perturbed generated sample y g ∼ q ( y | G θ ( z ) , t ) can be rewritten as y g =

Original: √ ¯
Revised: √ ¯

Original: α t G θ ( z ) + (cid:112) (1 − ¯ α t ) σ ϵ , ϵ ∼ N (0 , I ) .
Revised: α t G θ ( z ) + (cid:112) (1 − ¯ α t ) σ ϵ , ϵ ∼ N (0 , I ) .

Original: This means that the objective function in
Revised: This means that the objective function in

Original: Equation (3) is differentiable with respect to the generator parameters, and we can use gradient descent to optimize it with back-propagation.
Revised: Equation (3) is differentiable with respect to the generator parameters, and we can use gradient descent to optimize it with back-propagation.

Original: The objective function Equation (3) is similar to the one used by the original GAN (Goodfellow et al., 2014), except that it involves the diffusion steps and the perturbation functions.
Revised: The objective function Equation (3) is similar to the one used by the original GAN (Goodfellow et al., 2014), except that it involves the diffusion steps and the perturbation functions.

Original: We can show that this objective function also minimizes an approximation of the Jensen–Shannon (JS) divergence between the true and the generated distributions, but with respect to the perturbed samples and the diffusion steps, as follows:
Revised: We can show that this objective function also minimizes an approximation of the Jensen–Shannon (JS) divergence between the true and the generated distributions, but with respect to the perturbed samples and the diffusion steps, as follows:

Original: [Equation]
Revised: [Equation]

Original: The JS divergence measures the dissimilarity between two probability distributions, and it reaches its minimum value of zero when the two distributions are identical.
Revised: The JS divergence measures the dissimilarity between two probability distributions, and it reaches its minimum value of zero when the two distributions are identical.

Original: The proof of the equality in
Revised: The proof of the equality in

Original: Equation (4) is given in Appendix C. A natural question that arises from this result is whether minimizing the JS divergence between the perturbed distributions implies minimizing the JS divergence between the original distributions, i.e. , whether the optimal generator for Equation (3) is also the optimal generator for D JS ( p ( x )
Revised: Equation (4) is given in Appendix C. A natural question that arises from this result is whether minimizing the JS divergence between the perturbed distributions implies minimizing the JS divergence between the original distributions, i.e. , whether the optimal generator for Equation (3) is also the optimal generator for D JS ( p ( x )

Original: || p g ( x )) .
Revised: || p g ( x )) .

Original: We will answer this question affirmatively and provide a theoretical justification in Section 3.4.
Revised: We will answer this question affirmatively and provide a theoretical justification in Section 3.4.

Original: 3. A DAPTIVE DIFFUSION
Revised: 3. A DAPTIVE DIFFUSION

Original: With the help of the perturbation function and timestep dependency, we have a new strategy to optimize the discriminator.
Revised: With the help of the perturbation function and timestep dependency, we have a new strategy to optimize the discriminator.

Original: We want the discriminator D to have a challenging task, neither too easy to allow overfitting the data (Karras et al., 2020a;
Revised: We want the discriminator D to have a challenging task, neither too easy to allow overfitting the data (Karras et al., 2020a;

Original: Zhao et al., 2020) nor too hard to impede learning.
Revised: Zhao et al., 2020) nor too hard to impede learning.

Original: Therefore, we adjust the intensity of the diffusion process, which adds noise to both y and y g , depending on how much D can distinguish them.
Revised: Therefore, we adjust the intensity of the diffusion process, which adds noise to both y and y g , depending on how much D can distinguish them.

Original: When the diffusion step t is larger, the noise-to-data ratios are higher and the task is harder.
Revised: When the diffusion step t is larger, the noise-to-data ratios are higher and the task is harder.

Original: We use 1 − ¯ α t to measure the intensity of the diffusion, which increases as t grows.
Revised: We use 1 − ¯ α t to measure the intensity of the diffusion, which increases as t grows.

Original: To control the diffusion intensity, we adaptively modify the maximum number of steps T .
Revised: To control the diffusion intensity, we adaptively modify the maximum number of steps T .

Original: Our strategy is to make the discriminator learn from the easiest samples first, which are the original data samples, and then gradually increase the difficulty by feeding it samples from larger t .
Revised: Our strategy is to make the discriminator learn from the easiest samples first, which are the original data samples, and then gradually increase the difficulty by feeding it samples from larger t .

Original: To do this, we use a self-paced schedule for T , which depends on a metric r d that estimates how much the discriminator overfits to the data:
Revised: To do this, we use a self-paced schedule for T , which depends on a metric r d that estimates how much the discriminator overfits to the data:

Original: [Equation]
Revised: [Equation]

Original: where r d is the same as in Karras et al.
Revised: where r d is the same as in Karras et al.

Original: (2020a)
Revised: (2020a)

Original: and C is a constant.
Revised: and C is a constant.

Original: We calculate r d and update T every four minibatches.
Revised: We calculate r d and update T every four minibatches.

Original: We have two options for the distribution p π that we use to sample t for the diffusion process:
Revised: We have two options for the distribution p π that we use to sample t for the diffusion process:

Original: [Equation]
Revised: [Equation]

Original: The ‘priority’ option gives more weight to larger t , which means the discriminator will see more new samples from the new steps when T increases.
Revised: The ‘priority’ option gives more weight to larger t , which means the discriminator will see more new samples from the new steps when T increases.

Original: This is because we want the discriminator to focus on the new and harder samples that it has not seen before, as this indicates that it is confident about the easier ones.
Revised: This is because we want the discriminator to focus on the new and harder samples that it has not seen before, as this indicates that it is confident about the easier ones.

Original: Note that even with the ‘priority’ option, the discriminator can still see samples from smaller t , because q ( y | x ) is a mixture of Gaussians that covers all steps of the diffusion chain.
Revised: Note that even with the ‘priority’ option, the discriminator can still see samples from smaller t , because q ( y | x ) is a mixture of Gaussians that covers all steps of the diffusion chain.

Original: To avoid sudden changes in T during training, we use an exploration list t epl that contains t values sampled from p π .
Revised: To avoid sudden changes in T during training, we use an exploration list t epl that contains t values sampled from p π .

Original: We keep t epl fixed until we update T , and we sample t from t epl to generate noisy samples for the discriminator.
Revised: We keep t epl fixed until we update T , and we sample t from t epl to generate noisy samples for the discriminator.

Original: This way, the model can explore each t sufficiently before moving to a higher T .
Revised: This way, the model can explore each t sufficiently before moving to a higher T .

Original: We give the details of training Diffusion-GAN in Algorithm 1 in Appendix F.
Revised: We give the details of training Diffusion-GAN in Algorithm 1 in Appendix F.

Original: 3. T HEORETICAL ANALYSIS WITH E XAMPLES
Revised: 3. T HEORETICAL ANALYSIS WITH E XAMPLES

Original: To better understand the theoretical properties of our proposed method, we present two theorems that address two important questions about the use of diffusion-based instance noise injection for training
Revised: To better understand the theoretical properties of our proposed method, we present two theorems that address two important questions about the use of diffusion-based instance noise injection for training

Original: GANs.
Revised: GANs.

Original: The proofs of these theorems are deferred to Appendix B.
Revised: The proofs of these theorems are deferred to Appendix B.

Original: The first question, denoted as (a) , is whether adding noise to the real and generated samples in a diffusion process can facilitate the learning.
Revised: The first question, denoted as (a) , is whether adding noise to the real and generated samples in a diffusion process can facilitate the learning.

Original: The second question, denoted as (b) , is whether minimizing the JS divergence between the joint distributions of the noisy samples and the noise levels, p ( y , t ) and p g ( y , t ) , can lead to the same optimal generator as minimizing the JS divergence between the original distributions of the real and generated samples, p ( x ) and p g ( x ) .
Revised: The second question, denoted as (b) , is whether minimizing the JS divergence between the joint distributions of the noisy samples and the noise levels, p ( y , t ) and p g ( y , t ) , can lead to the same optimal generator as minimizing the JS divergence between the original distributions of the real and generated samples, p ( x ) and p g ( x ) .

Original: To answer (a) , we prove that for any choice of noise level t and any choice of convex function f , the f -divergence (Nowozin et al., 2016) between the marginal distributions of the noisy real and generated samples, q ( y | t ) and q ( y g | t ) , is a smooth function that can be computed and optimized by the discriminator.
Revised: To answer (a) , we prove that for any choice of noise level t and any choice of convex function f , the f -divergence (Nowozin et al., 2016) between the marginal distributions of the noisy real and generated samples, q ( y | t ) and q ( y g | t ) , is a smooth function that can be computed and optimized by the discriminator.

Original: This implies that the diffusion-based noise injection does not introduce any singularity or discontinuity in the objective function of the GAN.
Revised: This implies that the diffusion-based noise injection does not introduce any singularity or discontinuity in the objective function of the GAN.

Original: The JS divergence is a special case of f -divergence, where f ( u ) = − log(2 u ) − log(2 − 2 u ) .
Revised: The JS divergence is a special case of f -divergence, where f ( u ) = − log(2 u ) − log(2 − 2 u ) .

Original: Theorem 1 (Valid gradients anywhere for GANs training) .
Revised: Theorem 1 (Valid gradients anywhere for GANs training) .

Original: Let p ( x ) be a fixed distribution over X and z be a random noise over another space Z .
Revised: Let p ( x ) be a fixed distribution over X and z be a random noise over another space Z .

Original: Denote G θ : Z → X as a function with parameter θ and input z and p g ( x ) as the distribution of G θ ( z ) .
Revised: Denote G θ : Z → X as a function with parameter θ and input z and p g ( x ) as the distribution of G θ ( z ) .

Original: Let q ( y | x , t ) =
Revised: Let q ( y | x , t ) =

Original: N ( y ; √ ¯ α t x , (1 − ¯ α t ) σ 2 I ) , where ¯ α t ∈ (0 , 1) and σ > 0 .
Revised: N ( y ; √ ¯ α t x , (1 − ¯ α t ) σ 2 I ) , where ¯ α t ∈ (0 , 1) and σ > 0 .

Original: Let q ( y | t ) =
Revised: Let q ( y | t ) =

Original: (cid:82) p ( x ) q ( y | x , t ) d x and q g ( y | t ) =
Revised: (cid:82) p ( x ) q ( y | x , t ) d x and q g ( y | t ) =

Original: (cid:82) p g ( x ) q ( y | x , t ) d x .
Revised: (cid:82) p g ( x ) q ( y | x , t ) d x .

Original: Then, ∀ t , if function G θ is continuous and differentiable, the f-divergence D f ( q ( y | t )
Revised: Then, ∀ t , if function G θ is continuous and differentiable, the f-divergence D f ( q ( y | t )

Original: ||
Revised: ||

Original: q g ( y | t )) is continuous and differentiable with respect to θ .
Revised: q g ( y | t )) is continuous and differentiable with respect to θ .

Original: Theorem 1 shows that with the help of diffusion noise injection by q ( y | x , t ) , ∀ t , y and y g are defined on the same support space, the whole X , and D f ( q ( y | t )
Revised: Theorem 1 shows that with the help of diffusion noise injection by q ( y | x , t ) , ∀ t , y and y g are defined on the same support space, the whole X , and D f ( q ( y | t )

Original: ||
Revised: ||

Original: q g ( y | t )) is continuous and differentiable everywhere.
Revised: q g ( y | t )) is continuous and differentiable everywhere.

Original: Then, one natural question is what if D f ( q ( y | t )
Revised: Then, one natural question is what if D f ( q ( y | t )

Original: ||
Revised: ||

Original: q g ( y | t )) keeps a near constant value and hence provides little useful gradient.
Revised: q g ( y | t )) keeps a near constant value and hence provides little useful gradient.

Original: Hence, we empirically show that by injecting noise through a mixture defined over all steps of the diffusion chain, there is always a good chance that a sufficiently large t is sampled to provide a useful gradient, via the toy example below.
Revised: Hence, we empirically show that by injecting noise through a mixture defined over all steps of the diffusion chain, there is always a good chance that a sufficiently large t is sampled to provide a useful gradient, via the toy example below.

Original: Toy example.
Revised: Toy example.

Original: We use the same simple example from Arjovsky et al.
Revised: We use the same simple example from Arjovsky et al.

Original: (2017) to illustrate our method.
Revised: (2017) to illustrate our method.

Original: Let x = (0 , z ) be the real data and x g =
Revised: Let x = (0 , z ) be the real data and x g =

Original: ( θ, z ) be the data generated by a one-parameter generator, where z is a uniform random variable in [0 , 1] .
Revised: ( θ, z ) be the data generated by a one-parameter generator, where z is a uniform random variable in [0 , 1] .

Original: The JS divergence between the real and the generated distributions, D JS ( p ( x ) || p
Revised: The JS divergence between the real and the generated distributions, D JS ( p ( x ) || p

Original: ( x g )) , is discontinuous: it is log 2 when θ = 0 and zero otherwise, so it does not provide a useful gradient to guide θ towards zero.
Revised: ( x g )) , is discontinuous: it is log 2 when θ = 0 and zero otherwise, so it does not provide a useful gradient to guide θ towards zero.

Original: We introduce diffusion-based noise to both the real and the generated data, as shown in the first row of Figure 2.
Revised: We introduce diffusion-based noise to both the real and the generated data, as shown in the first row of Figure 2.

Original: The noisy data, y and y g , have supports that cover the whole space R 2 and their densities overlap more or less depending on the diffusion step t .
Revised: The noisy data, y and y g , have supports that cover the whole space R 2 and their densities overlap more or less depending on the diffusion step t .

Original: In the second row, left, of Figure 2, we plot how the JS divergence between the noisy distributions, D JS ( q ( y | t )
Revised: In the second row, left, of Figure 2, we plot how the JS divergence between the noisy distributions, D JS ( q ( y | t )

Original: ||
Revised: ||

Original: q g ( y | t )) , varies with θ for different t values.
Revised: q g ( y | t )) , varies with θ for different t values.

Original: The black line with t = 0 is the original JS divergence, which has a discontinuity at θ = 0 .
Revised: The black line with t = 0 is the original JS divergence, which has a discontinuity at θ = 0 .

Original: As t increases, the JS divergence curves become smoother and have nonzero gradients for a larger range of θ .
Revised: As t increases, the JS divergence curves become smoother and have nonzero gradients for a larger range of θ .

Original: However, some values of t , such as t = 200 in this example, still have flat regions where the JS divergence is nearly constant.
Revised: However, some values of t , such as t = 200 in this example, still have flat regions where the JS divergence is nearly constant.

Original: To avoid this, we use a mixture of all steps to ensure that there is always a high chance of getting informative gradients.
Revised: To avoid this, we use a mixture of all steps to ensure that there is always a high chance of getting informative gradients.

Original: For the discriminator optimization, as shown in the second row, right, of Figure 2, the optimal discriminator under the original JS divergence is discontinuous and unattainable.
Revised: For the discriminator optimization, as shown in the second row, right, of Figure 2, the optimal discriminator under the original JS divergence is discontinuous and unattainable.

Original: With diffusionbased noise, the optimal discriminator changes with t : a smaller t makes it more confident and a larger t makes it more cautious.
Revised: With diffusionbased noise, the optimal discriminator changes with t : a smaller t makes it more confident and a larger t makes it more cautious.

Original: Thus the diffusion acts like a scale to balance the power of the
Revised: Thus the diffusion acts like a scale to balance the power of the

Original: 4 x 0.
Revised: 4 x 0.

Original: y t=   4 x 1 y t=  2 x  1 y t=  2 x  1 y t= 1 2 x   y t=   1 x   y t= 10.
Revised: y t=   4 x 1 y t=  2 x  1 y t=  2 x  1 y t= 1 2 x   y t=   1 x   y t= 10.

Original: D J S ( q ( y | t ) | | q g ( y | t )) t
Revised: D J S ( q ( y | t ) | | q g ( y | t )) t

Original: = t = t = t =
Revised: = t = t = t =

Original: t =
Revised: t =

Original: t =     6 x 0.
Revised: t =     6 x 0.

Original: O p t i m a l
Revised: O p t i m a l

Original: D i s c
Revised: D i s c

Original: r
Revised: r

Original: i m i n a t
Revised: i m i n a t

Original: o r V a l u e D
Revised: o r V a l u e D

Original: * ( x ) t = t =
Revised: * ( x ) t = t =

Original: t = t =
Revised: t = t =

Original: t = t = discriminator.
Revised: t = t = discriminator.

Original: This suggests the use of a differentiable forward diffusion chain that can provide various levels of gradient smoothness to help the generator training.
Revised: This suggests the use of a differentiable forward diffusion chain that can provide various levels of gradient smoothness to help the generator training.

Original: Theorem 2 (Non-leaking noise injection) .
Revised: Theorem 2 (Non-leaking noise injection) .

Original: Let x ∼ p ( x ) , y ∼ q ( y | x ) and x g ∼ p
Revised: Let x ∼ p ( x ) , y ∼ q ( y | x ) and x g ∼ p

Original: g
Revised: g

Original: ( x ) , y g ∼ q ( y g | x g ) , where q ( y | x ) is the transition density.
Revised: ( x ) , y g ∼ q ( y g | x g ) , where q ( y | x ) is the transition density.

Original: Given certain q ( y | x ) , if y could be reparameterized into y = f
Revised: Given certain q ( y | x ) , if y could be reparameterized into y = f

Original: ( x ) + h ( ϵ ) , ϵ ∼ p
Revised: ( x ) + h ( ϵ ) , ϵ ∼ p

Original: ( ϵ ) , where p ( ϵ ) is a known distribution, and both f and h are one-to-one mapping functions
Revised: ( ϵ ) , where p ( ϵ ) is a known distribution, and both f and h are one-to-one mapping functions

Original: , then we could have p ( y ) =
Revised: , then we could have p ( y ) =

Original: p g ( y ) ⇔ p ( x ) =
Revised: p g ( y ) ⇔ p ( x ) =

Original: p g ( x ) .
Revised: p g ( x ) .

Original: To answer question (b) , we present Theorem 2, which shows a sufficient condition for the equality of the original and the augmented data distributions.
Revised: To answer question (b) , we present Theorem 2, which shows a sufficient condition for the equality of the original and the augmented data distributions.

Original: By Theorem 2, the function f maps each x to a unique y , the function h maps each ϵ to a unique noise term, and the distribution of ϵ is known and independent of x .
Revised: By Theorem 2, the function f maps each x to a unique y , the function h maps each ϵ to a unique noise term, and the distribution of ϵ is known and independent of x .

Original: Under these assumptions, the theorem proves that the distribution of y is the same as the distribution of y g , if and only if the distribution of x is the same as the distribution of x g .
Revised: Under these assumptions, the theorem proves that the distribution of y is the same as the distribution of y g , if and only if the distribution of x is the same as the distribution of x g .

Original: If we take y | t as the y introduced in the theorem, then for ∀ t , Equation (2) fits the assumption made.
Revised: If we take y | t as the y introduced in the theorem, then for ∀ t , Equation (2) fits the assumption made.

Original: This means that, by minimizing the divergence between q ( y | t ) and q g ( y | t ) , which is the same as minimizing the divergence between p ( x ) | t and p g ( x ) | t , we are also minimizing the divergence between p ( x ) and p g ( x ) .
Revised: This means that, by minimizing the divergence between q ( y | t ) and q g ( y | t ) , which is the same as minimizing the divergence between p ( x ) | t and p g ( x ) | t , we are also minimizing the divergence between p ( x ) and p g ( x ) .

Original: This implies that the noise injection does not affect the quality of the generated samples, and we can safely use our noise injection to improve the training of the generative model.
Revised: This implies that the noise injection does not affect the quality of the generated samples, and we can safely use our noise injection to improve the training of the generative model.

Original: 3. R ELATED WORK
Revised: 3. R ELATED WORK

Original: The proposed Diffusion-GAN can be related to previous works on stabilizing the GAN training, building diffusion-based generative models, and constructing differential augmentation for dataefficient GAN training.
Revised: The proposed Diffusion-GAN can be related to previous works on stabilizing the GAN training, building diffusion-based generative models, and constructing differential augmentation for dataefficient GAN training.

Original: A detailed discussion on these related works is deferred to Appendix A.
Revised: A detailed discussion on these related works is deferred to Appendix A.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct extensive experiments to answer the following questions: (a) Will Diffusion-GAN outperform state-of-the-art GAN baselines on benchmark datasets?
Revised: We conduct extensive experiments to answer the following questions: (a) Will Diffusion-GAN outperform state-of-the-art GAN baselines on benchmark datasets?

Original: (b) Will the diffusion-based noise injection help the learning of GANs in domain-agnostic tasks?
Revised: (b) Will the diffusion-based noise injection help the learning of GANs in domain-agnostic tasks?

Original: (c) Will our method improve the performance of data-efficient GANs trained with a very limited amount of data?
Revised: (c) Will our method improve the performance of data-efficient GANs trained with a very limited amount of data?

Original: Datasets.
Revised: Datasets.

Original: We conduct experiments on image datasets ranging from low-resolution ( e.g. , 32 × 32 ) to high-resolution ( e.g. , 1024 × 1024 ) and from low-diversity to high-diversity: CIFAR-
Revised: We conduct experiments on image datasets ranging from low-resolution ( e.g. , 32 × 32 ) to high-resolution ( e.g. , 1024 × 1024 ) and from low-diversity to high-diversity: CIFAR-

Original: (Krizhevsky, 2009), STL-10 (Coates et al., 2011), LSUN-Bedroom (Yu et al., 2015), LSUN-Church (Yu et al., 2015), AFHQ(Cat/Dog/Wild) (Choi et al., 2020), and FFHQ (Karras et al., 2019).
Revised: (Krizhevsky, 2009), STL-10 (Coates et al., 2011), LSUN-Bedroom (Yu et al., 2015), LSUN-Church (Yu et al., 2015), AFHQ(Cat/Dog/Wild) (Choi et al., 2020), and FFHQ (Karras et al., 2019).

Original: More details on these benchmark datasets are provided in Appendix E.
Revised: More details on these benchmark datasets are provided in Appendix E.

Original: Evaluation protocol.
Revised: Evaluation protocol.

Original: We measure image quality using FID (Heusel et al., 2017).
Revised: We measure image quality using FID (Heusel et al., 2017).

Original: Following Karras et al.
Revised: Following Karras et al.

Original: (2019; 2020b), we measure FID using 50k generated samples, with the full training set used
Revised: (2019; 2020b), we measure FID using 50k generated samples, with the full training set used

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: as reference.
Revised: as reference.

Original: We use the number of real images shown to the discriminator to evaluate convergence (Karras et al., 2020a; Sauer et al., 2021).
Revised: We use the number of real images shown to the discriminator to evaluate convergence (Karras et al., 2020a; Sauer et al., 2021).

Original: Unless specified otherwise, all models are trained withmillion images to ensure convergence (these trained with more or fewer images are specified in table captions).
Revised: Unless specified otherwise, all models are trained withmillion images to ensure convergence (these trained with more or fewer images are specified in table captions).

Original: We further report the improved Recall score introduced by Kynk ¨ a ¨ anniemi et al.
Revised: We further report the improved Recall score introduced by Kynk ¨ a ¨ anniemi et al.

Original: (2019) to measure the sample diversity of generative models.
Revised: (2019) to measure the sample diversity of generative models.

Original: Implementations
Revised: Implementations

Original: and
Revised: and

Original: resources
Revised: resources

Original: We build Diffusion-GANs based on the code of StyleGAN (Karras et al., 2020b), ProjectedGAN (Sauer et al., 2021), and InsGen (Yang et al., 2021) to answer questions (a) , (b) , and (c) , respectively.
Revised: We build Diffusion-GANs based on the code of StyleGAN (Karras et al., 2020b), ProjectedGAN (Sauer et al., 2021), and InsGen (Yang et al., 2021) to answer questions (a) , (b) , and (c) , respectively.

Original: Diffusion GANs inherit from their corresponding base
Revised: Diffusion GANs inherit from their corresponding base

Original: GANs all their network architectures and training hyperparamters, whose details are provided in
Revised: GANs all their network architectures and training hyperparamters, whose details are provided in

Original: Appendix G. Specifically for StyleGAN2 and InsGen, we construct the discriminator as D ϕ
Revised: Appendix G. Specifically for StyleGAN2 and InsGen, we construct the discriminator as D ϕ

Original: ( y , t ) , where t is injected via their mapping network.
Revised: ( y , t ) , where t is injected via their mapping network.

Original: For ProjectedGAN, we empirically find t in the discriminator could be ignored to simplify the implementation and minimize the modifications to ProjectedGAN.
Revised: For ProjectedGAN, we empirically find t in the discriminator could be ignored to simplify the implementation and minimize the modifications to ProjectedGAN.

Original: More implementation details are provided in Appendix H. By applying our diffusionbased noise injection, we denote our models as Diffusion StyleGAN2/ProjectedGAN/InsGen.
Revised: More implementation details are provided in Appendix H. By applying our diffusionbased noise injection, we denote our models as Diffusion StyleGAN2/ProjectedGAN/InsGen.

Original: In the following experiments, we train related models with their official code if the results are unavailable, while others are all reported from references and marked with ∗ .
Revised: In the following experiments, we train related models with their official code if the results are unavailable, while others are all reported from references and marked with ∗ .

Original: We run all our experiments with either 4 or 8 NVIDIA V100 GPUs depending on the demands of the inherited training configurations.
Revised: We run all our experiments with either 4 or 8 NVIDIA V100 GPUs depending on the demands of the inherited training configurations.

Original: 4. C OMPARISON TO STATE - OF - THE - ART GAN S
Revised: 4. C OMPARISON TO STATE - OF - THE - ART GAN S

Original: We compare Diffusion-GAN with its state-of-the-art GAN backbone, StyleGAN2 (Karras et al., 2020a), and to evaluate its effectiveness from the data augmentation perspective, we compare it with both StyleGAN2 + DiffAug (Zhao et al., 2020) and StyleGAN2 + ADA (Karras et al., 2020a), in terms of both sample fidelity (FID) and sample diversity (Recall) over extensive benchmark datasets.
Revised: We compare Diffusion-GAN with its state-of-the-art GAN backbone, StyleGAN2 (Karras et al., 2020a), and to evaluate its effectiveness from the data augmentation perspective, we compare it with both StyleGAN2 + DiffAug (Zhao et al., 2020) and StyleGAN2 + ADA (Karras et al., 2020a), in terms of both sample fidelity (FID) and sample diversity (Recall) over extensive benchmark datasets.

Original: We present the quantitative and qualitative results in Table 1 and Figure 3.
Revised: We present the quantitative and qualitative results in Table 1 and Figure 3.

Original: Qualitatively, these generated images from Diffusion StyleGAN2 are all photo-realistic and have good diversity, ranging from low-resolution (32 × 32) to high-resolution (1024 × 1024).
Revised: Qualitatively, these generated images from Diffusion StyleGAN2 are all photo-realistic and have good diversity, ranging from low-resolution (32 × 32) to high-resolution (1024 × 1024).

Original: Additional randomly generated
Revised: Additional randomly generated

Original: [Figure]
Revised: [Figure]

Original: images are can be found in Appendix K. Quantitatively, Diffusion StyleGAN2 outperforms all the
Revised: images are can be found in Appendix L. Quantitatively, Diffusion StyleGAN2 outperforms all the

Original: GAN baselines in generation diversity, as measured by Recall, on all 6 benchmark datasets and outperforms them in FID by a clear margin on 5 out of the 6 benchmark datasets.
Revised: GAN baselines in generation diversity, as measured by Recall, on all 6 benchmark datasets and outperforms them in FID by a clear margin on 5 out of the 6 benchmark datasets.

Original: From the data augmentation perspective, we observe that Diffusion StyleGAN2 always clearly outperforms the backbone model StyleGAN2 across various datasets, which empirically validates our
Revised: From the data augmentation perspective, we observe that Diffusion StyleGAN2 always clearly outperforms the backbone model StyleGAN2 across various datasets, which empirically validates our

Original: Theorem 2.
Revised: Theorem 2.

Original: By contrast, both the ADA (Karras et al., 2020b) and Diffaug (Zhao et al., 2020) techniques could sometimes impair the generation performance on sufficiently large datasets, e.g. ,
Revised: By contrast, both the ADA (Karras et al., 2020b) and Diffaug (Zhao et al., 2020) techniques could sometimes impair the generation performance on sufficiently large datasets, e.g. ,

Original: LSUN-Bedroom and LSUN-Church, which is also observed by Yang et al.
Revised: LSUN-Bedroom and LSUN-Church, which is also observed by Yang et al.

Original: (2021) on FFHQ.
Revised: (2021) on FFHQ.

Original: This is possibly because their risk of leaking augmentation overshadows the benefits of data augmentation.
Revised: This is possibly because their risk of leaking augmentation overshadows the benefits of data augmentation.

Original: To investigate how the adaptive diffusion process works during training, we illustrate in Figure 4 the convergence of the maximum timestep T in our adaptive diffusion and discriminator outputs.
Revised: To investigate how the adaptive diffusion process works during training, we illustrate in Figure 4 the convergence of the maximum timestep T in our adaptive diffusion and discriminator outputs.

Original: We see that T is adaptively adjusted: The T for Diffusion StyleGAN2 increases as the training goes while the T for Diffusion ProjectedGAN first goes up and then goes down.
Revised: We see that T is adaptively adjusted: The T for Diffusion StyleGAN2 increases as the training goes while the T for Diffusion ProjectedGAN first goes up and then goes down.

Original: Note that the T is adjusted according to the overfitting status of the discriminator.
Revised: Note that the T is adjusted according to the overfitting status of the discriminator.

Original: The second panel shows that trained with the diffusion-based mixture distribution, the discriminator is always well-behaved and provides useful learning signals for the generator, which validates our analysis in Section 3.4 and Theorem 1.
Revised: The second panel shows that trained with the diffusion-based mixture distribution, the discriminator is always well-behaved and provides useful learning signals for the generator, which validates our analysis in Section 3.4 and Theorem 1.

Original: Memory and time costs.
Revised: Memory and time costs.

Original: Generally speaking, the memory and time costs of a Diffusion-GAN are comparable to those of the corresponding GAN baseline.
Revised: Generally speaking, the memory and time costs of a Diffusion-GAN are comparable to those of the corresponding GAN baseline.

Original: More specifically, switching from ADA (Karras et al., 2020a) to our diffusion-based augmentation, the added memory cost is negative, the added training time cost is negative, and the added inference time cost is zero.
Revised: More specifically, switching from ADA (Karras et al., 2020a) to our diffusion-based augmentation, the added memory cost is negative, the added training time cost is negative, and the added inference time cost is zero.

Original: For example, for
Revised: For example, for

Original: CIFAR-10, with four NVIDIA V100 GPUs, the training time for each 4k images is around 8.0s for
Revised: CIFAR-10, with four NVIDIA V100 GPUs, the training time for each 4k images is around 8.0s for

Original: StyleGAN2, 9.8s for StyleGAN2-ADA, and 9.5s for Diffusion-StyleGAN2.
Revised: StyleGAN2, 9.8s for StyleGAN2-ADA, and 9.5s for Diffusion-StyleGAN2.

Original: 4. E FFECTIVENESS OF D IFFUSION -GAN FOR DOMAIN - AGNOSTIC AUGMENTATION
Revised: 4. E FFECTIVENESS OF D IFFUSION -GAN FOR DOMAIN - AGNOSTIC AUGMENTATION

Original: To verify whether our method is domain-agnostic, we apply Diffusion-GAN onto the input feature vectors of GANs.
Revised: To verify whether our method is domain-agnostic, we apply Diffusion-GAN onto the input feature vectors of GANs.

Original: We conduct experiments on both low-dimensional and high-dimensional feature vectors, for which commonly used image augmentation methods are no longer applicable.
Revised: We conduct experiments on both low-dimensional and high-dimensional feature vectors, for which commonly used image augmentation methods are no longer applicable.

Original: -Gaussians
Revised: -Gaussians

Original: Example.
Revised: Example.

Original: We conduct experiments on the popular 25-Gaussians generation task.
Revised: We conduct experiments on the popular 25-Gaussians generation task.

Original: The 25-Gaussians dataset is a 2-D toy data, generated by a mixture of 25 two-dimensional Gaussian distributions.
Revised: The 25-Gaussians dataset is a 2-D toy data, generated by a mixture of 25 two-dimensional Gaussian distributions.

Original: Each data point is a 2-dimensional feature vector.
Revised: Each data point is a 2-dimensional feature vector.

Original: We train a small GAN model, whose generator and discriminator are both parameterized by multilayer perceptrons (MLPs), with two 128-unit hidden layers and LeakyReLu nonlinearities.
Revised: We train a small GAN model, whose generator and discriminator are both parameterized by multilayer perceptrons (MLPs), with two 128-unit hidden layers and LeakyReLu nonlinearities.

Original: The training results are shown in Figure 5.
Revised: The training results are shown in Figure 5.

Original: We observe that the vanilla GAN exhibits severe mode collapsing, capturing only a few modes.
Revised: We observe that the vanilla GAN exhibits severe mode collapsing, capturing only a few modes.

Original: Its discriminator outputs of real and fake samples depart from each other very quickly.
Revised: Its discriminator outputs of real and fake samples depart from each other very quickly.

Original: This implies a strong overfitting of the discriminator happened so that the discriminator stops providing useful learning signals for the generator.
Revised: This implies a strong overfitting of the discriminator happened so that the discriminator stops providing useful learning signals for the generator.

Original: However, DiffusionGAN successfully captures all the 25 Gaussian modes and the discriminator is under control to continuously provide useful learning signals.
Revised: However, DiffusionGAN successfully captures all the 25 Gaussian modes and the discriminator is under control to continuously provide useful learning signals.

Original: We interpret the improvement from two perspectives:
Revised: We interpret the improvement from two perspectives:

Original: First, non-leaking augmentation helps provide more information about the data space; Second, the discriminator is well behaved given the adaptively adjusted diffusion-based noise injection.
Revised: First, non-leaking augmentation helps provide more information about the data space; Second, the discriminator is well behaved given the adaptively adjusted diffusion-based noise injection.

Original: ProjectedGAN.
Revised: ProjectedGAN.

Original: To verify that our adaptive diffusion-based noise injection could benefit the learning of GANs on high-dimensional feature vectors, we directly apply it onto the discriminator fea-
Revised: To verify that our adaptive diffusion-based noise injection could benefit the learning of GANs on high-dimensional feature vectors, we directly apply it onto the discriminator fea-

Original: [Figure]
Revised: [Figure]

Original: ture space of ProjectedGAN (Sauer et al., 2021).
Revised: ture space of ProjectedGAN (Sauer et al., 2021).

Original: ProjectedGANs generally leverage pre-trained neural networks to extract meaningful features for the adversarial learning of the discriminator and generator.
Revised: ProjectedGANs generally leverage pre-trained neural networks to extract meaningful features for the adversarial learning of the discriminator and generator.

Original: Following Sauer et al.
Revised: Following Sauer et al.

Original: [Figure]
Revised: [Figure]

Original: we adaptively diffuse the feature vectors extracted by
Revised: we adaptively diffuse the feature vectors extracted by

Original: EfficientNet-v0 and keep all the other training part unchanged.
Revised: EfficientNet-v0 and keep all the other training part unchanged.

Original: We report the performance of Diffusion ProjectedGAN on several benchmark datasets in Table
Revised: We report the performance of Diffusion ProjectedGAN on several benchmark datasets in Table

Original: [Figure]
Revised: [Figure]

Original: which verifies that our augmentation method is domain-agnostic
Revised: which verifies that our augmentation method is domain-agnostic

Original: Under the ProjectedGAN framework, we see that with noise properly injected onto the high-dimensional feature space, Diffusion ProjectedGAN shows clear improvement in terms of both FID and Recall.
Revised: Under the ProjectedGAN framework, we see that with noise properly injected onto the high-dimensional feature space, Diffusion ProjectedGAN shows clear improvement in terms of both FID and Recall.

Original: We reach state-of-the-art FID results with Diffusion ProjectedGAN on STL-10 and LSUN-Bedroom/Church datasets.
Revised: We reach state-of-the-art FID results with Diffusion ProjectedGAN on STL-10 and LSUN-Bedroom/Church datasets.

Original: [Table]
Revised: [Table]

Original: 4. E FFECTIVENESS OF D IFFUSION -GAN FOR LIMITED DATA
Revised: 4. E FFECTIVENESS OF D IFFUSION -GAN FOR LIMITED DATA

Original: We evaluate whether Diffusion-GAN can provide data-efficient GAN training.
Revised: We evaluate whether Diffusion-GAN can provide data-efficient GAN training.

Original: We first generate five FFHQ ( 1024 × 1024 ) dataset splits, consisting of 200, 500, 1k, 2k
Revised: We first generate five FFHQ ( 1024 × 1024 ) dataset splits, consisting of 200, 500, 1k, 2k

Original: [Table]
Revised: [Table]

Original: and 5k images respectively, where 200 and 500 images are considered to be extremely limited data cases.
Revised: and 5k images respectively, where 200 and 500 images are considered to be extremely limited data cases.

Original: We also consider
Revised: We also consider

Original: AFHQ-Cat, -Dog
Revised: AFHQ-Cat, -Dog

Original: [Table]
Revised: [Table]

Original: and -Wild (512 × 512), each with as few as around 5k images.
Revised: and -Wild (512 × 512), each with as few as around 5k images.

Original: Motivated by the success of InsGen (Yang et al., 2021) on small datasets, we build our Diffusion-GAN upon it.
Revised: Motivated by the success of InsGen (Yang et al., 2021) on small datasets, we build our Diffusion-GAN upon it.

Original: We note on limited data, InsGen convincingly outperforms both StyleGAN2+ADA and +DiffAug, and currently holds the state-of-the-art performance for data-efficient GAN training.
Revised: We note on limited data, InsGen convincingly outperforms both StyleGAN2+ADA and +DiffAug, and currently holds the state-of-the-art performance for data-efficient GAN training.

Original: The results in
Revised: The results in

Original: Table 3 show that our Diffusion-GAN method can help further boost the performance of InsGen in limited data settings.
Revised: Table 3 show that our Diffusion-GAN method can help further boost the performance of InsGen in limited data settings.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present Diffusion-GAN, a novel GAN framework that uses a variable-length forward diffusion chain with a Gaussian mixture distribution to generate instance noise for GAN training.
Revised: We present Diffusion-GAN, a novel GAN framework that uses a variable-length forward diffusion chain with a Gaussian mixture distribution to generate instance noise for GAN training.

Original: This approach enables model- and domain-agnostic differentiable augmentation that leverages the advantages of diffusion without requiring a costly reverse diffusion chain.
Revised: This approach enables model- and domain-agnostic differentiable augmentation that leverages the advantages of diffusion without requiring a costly reverse diffusion chain.

Original: We prove theoretically and demonstrate empirically that Diffusion-GAN can prevent discriminator overfitting and provide non-leaking augmentation.
Revised: We prove theoretically and demonstrate empirically that Diffusion-GAN can prevent discriminator overfitting and provide non-leaking augmentation.

Original: We also demonstrate that Diffusion-GAN can produce high-resolution photo-realistic images with high fidelity and diversity, outperforming its corresponding state-of-theart GAN baselines on standard benchmark datasets according to both FID and Recall.
Revised: We also demonstrate that Diffusion-GAN can produce high-resolution photo-realistic images with high fidelity and diversity, outperforming its corresponding state-of-theart GAN baselines on standard benchmark datasets according to both FID and Recall.

Original: Appendix A R ELATED WORK
Revised: Appendix A R ELATED WORK

Original: Stabilizing GAN training.
Revised: Stabilizing GAN training.

Original: A root cause of training difficulties in GANs is often attributed to the JS divergence that GANs intend to minimize.
Revised: A root cause of training difficulties in GANs is often attributed to the JS divergence that GANs intend to minimize.

Original: This is because when the data and generator distributions have non-overlapping supports, which are often the case for high-dimensional data supported by low-dimensional manifolds, the gradient of the JS divergence may provide no useful guidance to optimize the generator (Arjovsky and Bottou, 2017; Arjovsky et al., 2017; Mescheder et al., 2018;
Revised: This is because when the data and generator distributions have non-overlapping supports, which are often the case for high-dimensional data supported by low-dimensional manifolds, the gradient of the JS divergence may provide no useful guidance to optimize the generator (Arjovsky and Bottou, 2017; Arjovsky et al., 2017; Mescheder et al., 2018;

Original: Roth et al., 2017).
Revised: Roth et al., 2017).

Original: For this reason, Arjovsky et al.
Revised: For this reason, Arjovsky et al.

Original: (2017) propose to instead use the Wassersteindistance, which in theory can provide useful gradient for the generator even if the two distributions have disjoint supports.
Revised: (2017) propose to instead use the Wassersteindistance, which in theory can provide useful gradient for the generator even if the two distributions have disjoint supports.

Original: However, Wasserstein GANs often require the use of a critic function under the 1-Lipschitz constraint, which is difficult to satisfy in practice and hence realized with heuristics such as weight clipping Arjovsky et al.
Revised: However, Wasserstein GANs often require the use of a critic function under the 1-Lipschitz constraint, which is difficult to satisfy in practice and hence realized with heuristics such as weight clipping Arjovsky et al.

Original: (2017), gradient penalty (Gulrajani et al., 2017), and spectral normalization (Miyato et al., 2018a).
Revised: (2017), gradient penalty (Gulrajani et al., 2017), and spectral normalization (Miyato et al., 2018a).

Original: While the divergence minimization perspective has played an important role in motivating the construction of Wasserstein GANs and gradient penalty-based regularizations, cautions should be made on purely relying on it to understand GAN training, due to not only the discrepancy between the divergence in theory and the actual min-max objective function used in practice, but also the potential confounding between different divergences and different training and regularization strategies (Fedus et al., 2018; Mescheder et al., 2018).
Revised: While the divergence minimization perspective has played an important role in motivating the construction of Wasserstein GANs and gradient penalty-based regularizations, cautions should be made on purely relying on it to understand GAN training, due to not only the discrepancy between the divergence in theory and the actual min-max objective function used in practice, but also the potential confounding between different divergences and different training and regularization strategies (Fedus et al., 2018; Mescheder et al., 2018).

Original: E.g., Mescheder et al.
Revised: E.g., Mescheder et al.

Original: (2018) have provided a simple example where in theory the Wasserstein GAN is predicted to succeed while the vanilla GAN is predicted to fail, but in practice the Wasserstein GAN with a finite number of discriminator updates per generator update fails to converge while the vanilla GAN with the non-saturating loss can slowly converge.
Revised: (2018) have provided a simple example where in theory the Wasserstein GAN is predicted to succeed while the vanilla GAN is predicted to fail, but in practice the Wasserstein GAN with a finite number of discriminator updates per generator update fails to converge while the vanilla GAN with the non-saturating loss can slowly converge.

Original: Fedus et al.
Revised: Fedus et al.

Original: (2018) provide a rich set of empirical evidence to discourage viewing GANs purely from the perspective of minimizing a specific divergence at each training step and emphasize the important role played by gradient penalties on stabilizing GAN training.
Revised: (2018) provide a rich set of empirical evidence to discourage viewing GANs purely from the perspective of minimizing a specific divergence at each training step and emphasize the important role played by gradient penalties on stabilizing GAN training.

Original: Diffusion models.
Revised: Diffusion models.

Original: Due to the use of a forward diffusion chain, the proposed Diffusion-GAN can be related to diffusion-based (or score-based) deep generative models (Ho et al., 2020; SohlDickstein et al., 2015; Song and Ermon, 2019) that employ both a forward (inference) and a reverse (generative) diffusion chain.
Revised: Due to the use of a forward diffusion chain, the proposed Diffusion-GAN can be related to diffusion-based (or score-based) deep generative models (Ho et al., 2020b; SohlDickstein et al., 2015; Song and Ermon, 2019) that employ both a forward (inference) and a reverse (generative) diffusion chain.

Original: These diffusion-based generative models are stable to train and can generate high-fidelity photo-realistic images (Dhariwal and Nichol, 2021;
Revised: These diffusion-based generative models are stable to train and can generate high-fidelity photo-realistic images (Dhariwal and Nichol, 2021;

Original: Ho et al., 2020; Nichol et al., 2021; Ramesh et al., 2022; Song and Ermon, 2019; Song et al., 2021b).
Revised: Ho et al., 2020b; Nichol et al., 2021; Ramesh et al., 2022; Song and Ermon, 2019; Song et al., 2021b).

Original: However, they are notoriously slow in generation due to the need to traverse the reverse diffusion chain, which involves going through the same U-Net-based generator network hundreds or even thousands of times (Song et al., 2021a).
Revised: However, they are notoriously slow in generation due to the need to traverse the reverse diffusion chain, which involves going through the same U-Net-based generator network hundreds or even thousands of times (Song et al., 2021a).

Original: For this reason, a variety of methods have been proposed to reduce the generation cost of diffusion-based generative models (Kong and Ping, 2021; Luhman and Luhman, 2021; Pandey et al., 2022; San-Roman et al., 2021;
Revised: For this reason, a variety of methods have been proposed to reduce the generation cost of diffusion-based generative models (Kong and Ping, 2021; Luhman and Luhman, 2021; Pandey et al., 2022; San-Roman et al., 2021;

Original: Song et al., 2021a; Xiao et al., 2022; Zheng et al., 2022).
Revised: Song et al., 2021a; Xiao et al., 2021; Zheng et al., 2022).

Original: A key distinction is that Diffusion-GAN needs a reverse diffusion chain during neither training nor generation.
Revised: A key distinction is that Diffusion-GAN needs a reverse diffusion chain during neither training nor generation.

Original: More specifically, its generator maps the noise to a generated sample in a single step.
Revised: More specifically, its generator maps the noise to a generated sample in a single step.

Original: Diffusion-GAN can train and generate as quickly as a vanilla GAN does with the same generator size.
Revised: Diffusion-GAN can train and generate as quickly as a vanilla GAN does with the same generator size.

Original: For example, it takes around 20 hours to sample 50k images of size 32 × 32 from a DDPM (Ho et al., 2020) on an Nvidia 2080 Ti GPU, but would take less than a minute to do so from
Revised: For example, it takes around 20 hours to sample 50k images of size 32 × 32 from a DDPM (Ho et al., 2020b) on an Nvidia 2080 Ti GPU, but would take less than a minute to do so from

Original: Diffusion-GAN.
Revised: Diffusion-GAN.

Original: Differentiable augmentation.
Revised: Differentiable augmentation.

Original: As Diffusion-GAN transforms both the data and generated samples before sending them to the discriminator, we can also relate it to differentiable augmentation (Karras et al., 2020a; Zhao et al., 2020) proposed for data-efficient GAN training.
Revised: As Diffusion-GAN transforms both the data and generated samples before sending them to the discriminator, we can also relate it to differentiable augmentation (Karras et al., 2020a; Zhao et al., 2020) proposed for data-efficient GAN training.

Original: Karras et al.
Revised: Karras et al.

Original: (2020a) introduce a stochastic augmentation pipeline with 18 transformations and develop an adaptive mechanism for controlling the augmentation probability.
Revised: (2020a) introduce a stochastic augmentation pipeline with 18 transformations and develop an adaptive mechanism for controlling the augmentation probability.

Original: Zhao et al.
Revised: Zhao et al.

Original: (2020) propose to use Color + Translation + Cutout as differentiable augmentations for both generated and real images.
Revised: (2020) propose to use Color + Translation + Cutout as differentiable augmentations for both generated and real images.

Original: While providing good empirical results on some datasets, these augmentation methods are developed with domain-specific knowledge and have the risk of leaking augmentation into generation (Karras et al., 2020a).
Revised: While providing good empirical results on some datasets, these augmentation methods are developed with domain-specific knowledge and have the risk of leaking augmentation into generation (Karras et al., 2020a).

Original: As observed in our experiments, they sometime worsen the results when applied to a new dataset, likely because the risk of augmentation leakage overpowers the benefits of enlarging the training set, which could happen especially if the training set size is already sufficiently large.
Revised: As observed in our experiments, they sometime worsen the results when applied to a new dataset, likely because the risk of augmentation leakage overpowers the benefits of enlarging the training set, which could happen especially if the training set size is already sufficiently large.

Original: By contrast, Diffusion-GAN uses a differentiable forward diffusion process to stochastically transform the data and can be considered as both a domain-agnostic and a model-agnostic augmentation method.
Revised: By contrast, Diffusion-GAN uses a differentiable forward diffusion process to stochastically transform the data and can be considered as both a domain-agnostic and a model-agnostic augmentation method.

Original: In other words, Diffusion-GAN can be applied to non-image data or even latent features, for which appropriate data augmentation is difficult to be defined, and easily plugged into an existing
Revised: In other words, Diffusion-GAN can be applied to non-image data or even latent features, for which appropriate data augmentation is difficult to be defined, and easily plugged into an existing

Original: GAN to improve its generation performance.
Revised: GAN to improve its generation performance.

Original: Moreover, we prove in theory and show in experiments that augmentation leakage is not a concern for Diffusion-GAN.
Revised: Moreover, we prove in theory and show in experiments that augmentation leakage is not a concern for Diffusion-GAN.

Original: Tran et al.
Revised: Tran et al.

Original: (2021) provide a theoretical analysis for deterministic non-leaking transformation with differentiable and invertible mapping functions.
Revised: (2021) provide a theoretical analysis for deterministic non-leaking transformation with differentiable and invertible mapping functions.

Original: Bora et al.
Revised: Bora et al.

Original: (2018) show similar theorems to us for specific stochastic transformations, such as Gaussian Projection, Convolve+Noise, and stochastic Block-Pixels, while our Theoremincludes more satisfying possibilities as discussed in Appendix B.
Revised: (2018) show similar theorems to us for specific stochastic transformations, such as Gaussian Projection, Convolve+Noise, and stochastic Block-Pixels, while our Theoremincludes more satisfying possibilities as discussed in Appendix B.

Original: B P ROOF
Revised: B P ROOF

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: For simplicity, let x ∼ P r , x g ∼ P g , y ∼ P r ′ ,t , y g ∼ P g ′ ,t , a t = √ ¯ α t and b t =
Revised: For simplicity, let x ∼ P r , x g ∼ P g , y ∼ P r ′ ,t , y g ∼ P g ′ ,t , a t = √ ¯ α t and b t =

Original: (1 − ¯ α t ) σ 2 .
Revised: (1 − ¯ α t ) σ 2 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Since N ( y ; a t x , b t I ) is assumed to be an isotropic Gaussian distribution, for simplicity, in what follows we show the proof in uni-variate Gaussian, which could be easily extended to multi-variate
Revised: Since N ( y ; a t x , b t I ) is assumed to be an isotropic Gaussian distribution, for simplicity, in what follows we show the proof in uni-variate Gaussian, which could be easily extended to multi-variate

Original: Gaussian by the production rule.
Revised: Gaussian by the production rule.

Original: We first show that under mild conditions, the p r ′ ,t ( y ) and p g ′ ,t ( y ) are continuous functions over y .
Revised: We first show that under mild conditions, the p r ′ ,t ( y ) and p g ′ ,t ( y ) are continuous functions over y .

Original: [Equation]
Revised: [Equation]

Original: where C 1 and C 2 are constants.
Revised: where C 1 and C 2 are constants.

Original: Hence, p r ′ ,t ( y ) is a continuous function defined on y .
Revised: Hence, p r ′ ,t ( y ) is a continuous function defined on y .

Original: The proof of continuity for p g ′ ,t ( y ) is exactly the same proof.
Revised: The proof of continuity for p g ′ ,t ( y ) is exactly the same proof.

Original: Then, given g θ is also a continuous function, it is clear to see that D f ( p r ′ ,t ( y )
Revised: Then, given g θ is also a continuous function, it is clear to see that D f ( p r ′ ,t ( y )

Original: ||
Revised: ||

Original: p g ′ ,t ( y )) is a continuous function over θ .
Revised: p g ′ ,t ( y )) is a continuous function over θ .

Original: Next, we show that D f ( p r ′ ,t ( y )
Revised: Next, we show that D f ( p r ′ ,t ( y )

Original: ||
Revised: ||

Original: p g ′ ,t ( y )) is differentiable.
Revised: p g ′ ,t ( y )) is differentiable.

Original: By the chain rule, showing
Revised: By the chain rule, showing

Original: D f ( p r ′ ,t ( y ) ||
Revised: D f ( p r ′ ,t ( y ) ||

Original: p g ′ ,t ( y )) to be differentiable is equivalent to show p r ′ ,t ( y ) , p r ′ ,t ( y ) and f are differentiable.
Revised: p g ′ ,t ( y )) to be differentiable is equivalent to show p r ′ ,t ( y ) , p r ′ ,t ( y ) and f are differentiable.

Original: Usually, f is defined with differentiability (Nowozin et al., 2016).
Revised: Usually, f is defined with differentiability (Nowozin et al., 2016).

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where C 1 and C 2 are constants.
Revised: where C 1 and C 2 are constants.

Original: Hence, p r ′ ,t ( y ) and p r ′ ,t ( y ) are differentiable, which concludes the proof.
Revised: Hence, p r ′ ,t ( y ) and p r ′ ,t ( y ) are differentiable, which concludes the proof.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: We have p ( y ) =
Revised: We have p ( y ) =

Original: (cid:82) p ( x ) q ( y | x ) d x and p g ( y ) =
Revised: (cid:82) p ( x ) q ( y | x ) d x and p g ( y ) =

Original: (cid:82) p g ( x ) q ( y | x ) d x .
Revised: (cid:82) p g ( x ) q ( y | x ) d x .

Original: ⇐ If p ( x ) = p g ( x ) , then p ( y )
Revised: ⇐ If p ( x ) = p g ( x ) , then p ( y )

Original: =
Revised: =

Original: p g ( y ) ⇒ Let y ∼ p ( y ) and
Revised: p g ( y ) ⇒ Let y ∼ p ( y ) and

Original: y g ∼ p
Revised: y g ∼ p

Original: g
Revised: g

Original: ( y ) .
Revised: ( y ) .

Original: Given the assumption on q ( y | x ) , we have
Revised: Given the assumption on q ( y | x ) , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since f and g are one-to-one mapping functions, f ( x ) and g ( ϵ ) are identifiable, which indicates f ( x ) D
Revised: Since f and g are one-to-one mapping functions, f ( x ) and g ( ϵ ) are identifiable, which indicates f ( x ) D

Original: = f
Revised: = f

Original: ( x g ) ⇒ x D = x g .
Revised: ( x g ) ⇒ x D = x g .

Original: By the property of moment-generating functions (MGF), given f ( x ) is independent with g ( ϵ ) , we have for ∀ s
Revised: By the property of moment-generating functions (MGF), given f ( x ) is independent with g ( ϵ ) , we have for ∀ s

Original: [Equation]
Revised: [Equation]

Original: x g ) ( s ) · M g ( ϵ g ) ( s ) .
Revised: x g ) ( s ) · M g ( ϵ g ) ( s ) .

Original: where M y ( s ) =
Revised: where M y ( s ) =

Original: E
Revised: E

Original: y ∼ p
Revised: y ∼ p

Original: ( y )
Revised: ( y )

Original: [ e s T y ] denotes the MGF of random variable y and the others follow the same form.
Revised: [ e s T y ] denotes the MGF of random variable y and the others follow the same form.

Original: By the moment-generating function uniqueness theorem, given y D =
Revised: By the moment-generating function uniqueness theorem, given y D =

Original: y g
Revised: y g

Original: and g ( ϵ ) D =
Revised: and g ( ϵ ) D =

Original: g
Revised: g

Original: ( ϵ g ) , we have M y ( s ) =
Revised: ( ϵ g ) , we have M y ( s ) =

Original: M y g ( s ) and M g ( ϵ ) ( s ) =
Revised: M y g ( s ) and M g ( ϵ ) ( s ) =

Original: M g ( ϵ g ) ( s ) for ∀ s .
Revised: M g ( ϵ g ) ( s ) for ∀ s .

Original: Then, we could obtain M f ( x ) =
Revised: Then, we could obtain M f ( x ) =

Original: M f ( x g ) for ∀ s .
Revised: M f ( x g ) for ∀ s .

Original: Thus, M f ( x ) =
Revised: Thus, M f ( x ) =

Original: M f ( x g ) ⇒ f
Revised: M f ( x g ) ⇒ f

Original: ( x ) D = f
Revised: ( x ) D = f

Original: ( x g ) ⇒ p
Revised: ( x g ) ⇒ p

Original: ( x ) = p ( x g ) , which concludes the proof.
Revised: ( x ) = p ( x g ) , which concludes the proof.

Original: Discussion.
Revised: Discussion.

Original: Next, we discuss which q ( y | x ) fits the assumption we made on it.
Revised: Next, we discuss which q ( y | x ) fits the assumption we made on it.

Original: We follow the discussion of reparameterization of distributions as used in Kingma and Welling (2014).
Revised: We follow the discussion of reparameterization of distributions as used in Kingma and Welling (2014).

Original: Three basic approaches are:
Revised: Three basic approaches are:

Original: Tractable inverse CDF.
Revised: Tractable inverse CDF.

Original: In this case, let ϵ ∼ U
Revised: In this case, let ϵ ∼ U

Original: ( 0 , I ) , and ψ ( ϵ , y , x ) be the inverse CDF of q ( y | x ) .
Revised: ( 0 , I ) , and ψ ( ϵ , y , x ) be the inverse CDF of q ( y | x ) .

Original: From ψ ( ϵ , y , x ) , if y = f ( x ) + g ( ϵ ) , for example, y ∼ Cauchy ( x, γ ) and y ∼ Logistic ( x, s ) , then Theorem 2 holds.
Revised: From ψ ( ϵ , y , x ) , if y = f ( x ) + g ( ϵ ) , for example, y ∼ Cauchy ( x, γ ) and y ∼ Logistic ( x, s ) , then Theorem 2 holds.

Original: Analogous to the Gaussian example, y ∼ N ( x , σ 2 I )
Revised: Analogous to the Gaussian example, y ∼ N ( x , σ 2 I )

Original: ⇒ y
Revised: ⇒ y

Original: = x + σ · ϵ , ϵ ∼ N ( 0 , I ) .
Revised: = x + σ · ϵ , ϵ ∼ N ( 0 , I ) .

Original: For any “location-scale” family of distributions we can choose the standard distribution (with location = 0, scale = 1) as the auxiliary variable ϵ , and let g ( . )
Revised: For any “location-scale” family of distributions we can choose the standard distribution (with location = 0, scale = 1) as the auxiliary variable ϵ , and let g ( . )

Original: =
Revised: =

Original: location + scale · ϵ .
Revised: location + scale · ϵ .

Original: Examples: Laplace, Elliptical, Student’s t, Logistic, Uniform, Triangular, and Gaussian distributions.
Revised: Examples: Laplace, Elliptical, Student’s t, Logistic, Uniform, Triangular, and Gaussian distributions.

Original: Implicit distributions.
Revised: Implicit distributions.

Original: q ( y | x ) could be modeled by neural networks, which implies y = f
Revised: q ( y | x ) could be modeled by neural networks, which implies y = f

Original: ( x ) + g ( ϵ ) , ϵ ∼ p
Revised: ( x ) + g ( ϵ ) , ϵ ∼ p

Original: ( ϵ ) , where f and g are one-to-one nonlinear transformations.
Revised: ( ϵ ) , where f and g are one-to-one nonlinear transformations.

Original: C D ERIVATIONS Derivation of equality in JSD
Revised: C D ERIVATIONS Derivation of equality in JSD

Original: JSD ( p ( y , t ) , p g ( y , t ))
Revised: JSD ( p ( y , t ) , p g ( y , t ))

Original: =
Revised: =

Original: 2 D KL (cid:20) p ( y , t ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p ( y , t )
Revised: 2 D KL (cid:20) p ( y , t ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p ( y , t )

Original: + p g ( y , t ) (cid:21) +2 D KL (cid:20) p g ( y , t ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p ( y , t )
Revised: + p g ( y , t ) (cid:21) +2 D KL (cid:20) p g ( y , t ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p ( y , t )

Original: + p g ( y , t ) (cid:21) =
Revised: + p g ( y , t ) (cid:21) =

Original: 2 E y ,t ∼ p ( y ,t ) (cid:20) log 2 · p ( y , t ) p ( y , t ) + p g ( y , t ) (cid:21) +

Revised: 2 E y ,t ∼ p ( y ,t ) (cid:20) log 2 · p ( y , t ) p ( y , t ) + p g ( y , t ) (cid:21) +

Original: E y ,t ∼ p
Revised: E y ,t ∼ p

Original: g
Revised: g

Original: ( y ,t ) (cid:20) log 2 · p g ( y , t ) p ( y , t )
Revised: ( y ,t ) (cid:20) log 2 · p g ( y , t ) p ( y , t )

Original: + p g ( y , t ) (cid:21) =
Revised: + p g ( y , t ) (cid:21) =

Original: 2 E t ∼ p
Revised: 2 E t ∼ p

Original: π ( t ) , y ∼ p ( y | t ) (cid:20) log 2 · p ( y | t ) p π ( t ) p ( y | t ) p π ( t ) + p g ( y | t ) p π ( t ) (cid:21)
Revised: π ( t ) , y ∼ p ( y | t ) (cid:20) log 2 · p ( y | t ) p π ( t ) p ( y | t ) p π ( t ) + p g ( y | t ) p π ( t ) (cid:21)

Original: +

Revised: +

Original: E t ∼ p
Revised: E t ∼ p

Original: π ( t ) , y ∼ p g ( y | t ) (cid:20) log 2 · p g ( y | t ) p π ( t ) p ( y | t ) p π ( t ) + p g ( y | t ) p π ( t ) (cid:21) =
Revised: π ( t ) , y ∼ p g ( y | t ) (cid:20) log 2 · p g ( y | t ) p π ( t ) p ( y | t ) p π ( t ) + p g ( y | t ) p π ( t ) (cid:21) =

Original: E t ∼ p
Revised: E t ∼ p

Original: π ( t ) (cid:20)2 E y ∼ p
Revised: π ( t ) (cid:20)2 E y ∼ p

Original: ( y | t ) (cid:20) log 2 · p
Revised: ( y | t ) (cid:20) log 2 · p

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: t ) p
Revised: t ) p

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:21) +

Revised: (cid:21) +

Original: E
Revised: E

Original: y ∼ p
Revised: y ∼ p

Original: g
Revised: g

Original: ( y | t ) (cid:20) log 2 · p g ( y | t ) p ( y | t )
Revised: ( y | t ) (cid:20) log 2 · p g ( y | t ) p ( y | t )

Original: + p g ( y | t ) (cid:21)(cid:21) =
Revised: + p g ( y | t ) (cid:21)(cid:21) =

Original: E t ∼ p
Revised: E t ∼ p

Original: π ( t ) [ JSD ( p ( y | t ) , p g ( y | t ))] .
Revised: π ( t ) [ JSD ( p ( y | t ) , p g ( y | t ))] .

Original: D D ETAILS OF TOY EXAMPLE
Revised: D D ETAILS OF TOY EXAMPLE

Original: Here, we provide the detailed analysis of the JS divergence toy example.
Revised: Here, we provide the detailed analysis of the JS divergence toy example.

Original: Notation.
Revised: Notation.

Original: Let X be a compact metric set (such as the space of images [0 , 1] d ) and Prob ( X ) denote the space of probability measures defined on X .
Revised: Let X be a compact metric set (such as the space of images [0 , 1] d ) and Prob ( X ) denote the space of probability measures defined on X .

Original: Let P r be the target data distribution and P g 1 be the generator distribution.
Revised: Let P r be the target data distribution and P g 1 be the generator distribution.

Original: The JSD between the two distributions P r , P g ∈ Prob ( X ) is defined as:
Revised: The JSD between the two distributions P r , P g ∈ Prob ( X ) is defined as:

Original: [Equation]
Revised: [Equation]

Original: where P m is the mixture ( P r + P g ) /
Revised: where P m is the mixture ( P r + P g ) /

Original: 2 and D KL denotes the Kullback-Leibler divergence, i.e. ,
Revised: 2 and D KL denotes the Kullback-Leibler divergence, i.e. ,

Original: D KL
Revised: D KL

Original: [Equation]
Revised: [Equation]

Original: X p
Revised: X p

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: x ) p θ ( x ) ) dx .
Revised: x ) p θ ( x ) ) dx .

Original: More generally, the f -divergence (Nowozin et al., 2016) between P r and P g is defined as:
Revised: More generally, the f -divergence (Nowozin et al., 2016) between P r and P g is defined as:

Original: [Equation]
Revised: [Equation]

Original: where the generator function f : R + → R is a convex and lower-semicontinuous function satisfying f (1) = 0 .
Revised: where the generator function f : R + → R is a convex and lower-semicontinuous function satisfying f (1) = 0 .

Original: We refer to Nowozin et al.
Revised: We refer to Nowozin et al.

Original: (2016) for more details.
Revised: (2016) for more details.

Original: We recall the typical example introduced in Arjovsky and Bottou (2017) and follow the notations.
Revised: We recall the typical example introduced in Arjovsky and Bottou (2017) and follow the notations.

Original: Example.
Revised: Example.

Original: Let Z ∼ U
Revised: Let Z ∼ U

Original: [0 , 1] be the uniform distribution on the unit interval.
Revised: [0 , 1] be the uniform distribution on the unit interval.

Original: Let X ∼ P r be the distribution of (0 , Z ) ∈ R 2 , which contains a 0 on the x-axis and a random variable Z on the y-axis.
Revised: Let X ∼ P r be the distribution of (0 , Z ) ∈ R 2 , which contains a 0 on the x-axis and a random variable Z on the y-axis.

Original: Let X g ∼ P g be the distribution of ( θ, Z ) ∈ R 2 , where θ is a single real parameter.
Revised: Let X g ∼ P g be the distribution of ( θ, Z ) ∈ R 2 , where θ is a single real parameter.

Original: In this case, the
Revised: In this case, the

Original: D JS ( P r || P g ) is not continuous,
Revised: D JS ( P r || P g ) is not continuous,

Original: [Equation]
Revised: [Equation]

Original: 4 x 0.
Revised: 4 x 0.

Original: y t=   4 x 1 y t=  2 x  1 y t=  2 x  1 y t= 1 2 x   y t=   1 x   y t= 10.
Revised: y t=   4 x 1 y t=  2 x  1 y t=  2 x  1 y t= 1 2 x   y t=   1 x   y t= 10.

Original: D J S ( q ( y | t ) | | q g ( y | t ))
Revised: D J S ( q ( y | t ) | | q g ( y | t ))

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t =     6 x 0.
Revised: t =     6 x 0.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: O p t i m a l
Revised: O p t i m a l

Original: D i s c
Revised: D i s c

Original: r
Revised: r

Original: i m i n a t
Revised: i m i n a t

Original: o r V a l
Revised: o r V a l

Original: [Figure]
Revised: [Figure]

Original: e D * ( x ) t = t =
Revised: e D * ( x ) t = t =

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t =
Revised: t =

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: which can not provide a usable gradient for training.
Revised: which can not provide a usable gradient for training.

Original: The derivation is as follows:
Revised: The derivation is as follows:

Original: [Equation]
Revised: [Equation]

Original: Although this simple example features distributions with disjoint supports, the same conclusion holds when the supports have a non empty intersection contained in a set of measure zero (Arjovsky and Bottou, 2017).
Revised: Although this simple example features distributions with disjoint supports, the same conclusion holds when the supports have a non empty intersection contained in a set of measure zero (Arjovsky and Bottou, 2017).

Original: This happens to be the case when two low dimensional manifolds intersect in general position (Arjovsky and Bottou, 2017).
Revised: This happens to be the case when two low dimensional manifolds intersect in general position (Arjovsky and Bottou, 2017).

Original: To avoid the potential issue caused by having non-overlapping distribution supports, a common remedy is to use Wasserstein-1 distance which in theory can still provide usable gradient (Arjovsky and Bottou, 2017; Arjovsky et al., 2017).
Revised: To avoid the potential issue caused by having non-overlapping distribution supports, a common remedy is to use Wasserstein-1 distance which in theory can still provide usable gradient (Arjovsky and Bottou, 2017; Arjovsky et al., 2017).

Original: In this case, the Wasserstein-1 distance is | θ | .
Revised: In this case, the Wasserstein-1 distance is | θ | .

Original: Diffusion-based noise injection In general, with our diffusion noise injected, we could have,
Revised: Diffusion-based noise injection In general, with our diffusion noise injected, we could have,

Original: [Equation]
Revised: [Equation]

Original: For the previous example, we have Y ′ t and Y ′ g,t such that,
Revised: For the previous example, we have Y ′ t and Y ′ g,t such that,

Original: [Equation]
Revised: [Equation]

Original: where f ( · ) = (cid:82) 1 0 N ( · | a t Z, b t ) U ( Z ) dZ , a t and b t are abbreviations for √ ¯ α t and (1 − ¯ α t ) σ 2 .
Revised: where f ( · ) = (cid:82) 1 0 N ( · | a t Z, b t ) U ( Z ) dZ , a t and b t are abbreviations for √ ¯ α t and (1 − ¯ α t ) σ 2 .

Original: The supports of Y ′ t and Y ′ g,t are both the whole metric space R 2 and they overlap with each other depending on t , as shown in Figure 2.
Revised: The supports of Y ′ t and Y ′ g,t are both the whole metric space R 2 and they overlap with each other depending on t , as shown in Figure 2.

Original: As t increases, the high density region of Y ′ t and Y ′ g,t get closer since the weight a t is decreasing towards 0.
Revised: As t increases, the high density region of Y ′ t and Y ′ g,t get closer since the weight a t is decreasing towards 0.

Original: Then, we derive the JS divergence,
Revised: Then, we derive the JS divergence,

Original: [Equation]
Revised: [Equation]

Original: which is clearly continuous and differentiable.
Revised: which is clearly continuous and differentiable.

Original: We show this D JS ( p r ′ ,
Revised: We show this D JS ( p r ′ ,

Original: t || p g ′ ,t ) with respect to increasing t values and a θ grid in the second row of Figure 2.
Revised: t || p g ′ ,t ) with respect to increasing t values and a θ grid in the second row of Figure 2.

Original: As shown in the left panel, the black line with t = 0 shows the origianl JSD, which is not even continuous, while as the diffusion level t increments, the lines become smoother and flatter.
Revised: As shown in the left panel, the black line with t = 0 shows the origianl JSD, which is not even continuous, while as the diffusion level t increments, the lines become smoother and flatter.

Original: It is clear to see that these smooth curves provide good learning signals for θ .
Revised: It is clear to see that these smooth curves provide good learning signals for θ .

Original: Recall that the Wasserstein-1 distance is | θ | in this case.
Revised: Recall that the Wasserstein-1 distance is | θ | in this case.

Original: Meanwhile, we could observe with an intense diffusion, e.g. , t = 800 , the curve becomes flatter, which indicates smaller gradients and a much slower learning process.
Revised: Meanwhile, we could observe with an intense diffusion, e.g. , t = 800 , the curve becomes flatter, which indicates smaller gradients and a much slower learning process.

Original: This motivates us that an adaptive diffusion could provide different level of gradient smoothness and is possibly better for training.
Revised: This motivates us that an adaptive diffusion could provide different level of gradient smoothness and is possibly better for training.

Original: The right panel shows the optimal discriminator outputs over the space X .
Revised: The right panel shows the optimal discriminator outputs over the space X .

Original: With diffusion, the optimal discriminator is well defined over the space and the gradient is smooth, while without diffusion the optimal discriminator is only valid on two star points.
Revised: With diffusion, the optimal discriminator is well defined over the space and the gradient is smooth, while without diffusion the optimal discriminator is only valid on two star points.

Original: Interestingly, we find that smaller t drives the optimal discriminator to become more assertive while larger t makes discriminator become more neutral.
Revised: Interestingly, we find that smaller t drives the optimal discriminator to become more assertive while larger t makes discriminator become more neutral.

Original: The diffusion here works like a scale to balance the power of the discriminator.
Revised: The diffusion here works like a scale to balance the power of the discriminator.

Original: E D ATASET DESCRIPTIONS
Revised: E D ATASET DESCRIPTIONS

Original: The CIFAR-10 dataset consists of 50k 32 × 32 training images in 10 categories.
Revised: The CIFAR-10 dataset consists of 50k 32 × 32 training images in 10 categories.

Original: The STL-10 dataset originated from ImageNet (Deng et al., 2009) consists of 100k unlabeled images in 10 categories, and we resize them to 64 × 64 resolution.
Revised: The STL-10 dataset originated from ImageNet (Deng et al., 2009) consists of 100k unlabeled images in 10 categories, and we resize them to 64 × 64 resolution.

Original: For LSUN datasets, we sample 200k images from LSUNBedroom, use the whole 125k images from LSUN-Church, and resize them to 256 × 256 resolution for training.
Revised: For LSUN datasets, we sample 200k images from LSUNBedroom, use the whole 125k images from LSUN-Church, and resize them to 256 × 256 resolution for training.

Original: The AFHQ datasets includes around 5k 512 × 512 images per category for dogs, cats, and wild life; we train a separate network for each of them.
Revised: The AFHQ datasets includes around 5k 512 × 512 images per category for dogs, cats, and wild life; we train a separate network for each of them.

Original: The FFHQ contains 70k images crawled from Flickr at 1024 × 1024 resolution and we use all of them for training.
Revised: The FFHQ contains 70k images crawled from Flickr at 1024 × 1024 resolution and we use all of them for training.

Original: F A LGORITHM
Revised: F A LGORITHM

Original: We provide the Diffusion-GAN algorithm in Algorithm 1.
Revised: We provide the Diffusion-GAN algorithm in Algorithm 1.

Original: G H YPERPARAMETERS
Revised: G H YPERPARAMETERS

Original: Comparing to GANs, Diffusion-GAN introduces a new hyperparameter d target , which is a threshold to identify whether the current discriminator is overfitting.
Revised: Comparing to GANs, Diffusion-GAN introduces a new hyperparameter d target , which is a threshold to identify whether the current discriminator is overfitting.

Original: We find StyleGAN2-based models are not sensitive to the values of d target , so we set d target = 0 .
Revised: We find StyleGAN2-based models are not sensitive to the values of d target , so we set d target = 0 .

Original: 6 for them across all dataset, only except that we set d target = 0 .
Revised: 6 for them across all dataset, only except that we set d target = 0 .

Original: 8 for FFHQ ( d target = 0 .
Revised: 8 for FFHQ ( d target = 0 .

Original: 8 for FFHQ is slightly better than 0 .
Revised: 8 for FFHQ is slightly better than 0 .

Original: 6 in FID).
Revised: 6 in FID).

Original: We report d target of Diffusion ProjectedGAN for our experiments in Table 4.
Revised: We report d target of Diffusion ProjectedGAN for our experiments in Table 4.

Original: We also evaluated two t sampling distribution p π , [‘priority’, ‘uniform’], defined in Equation (6).
Revised: We also evaluated two t sampling distribution p π , [‘priority’, ‘uniform’], defined in Equation (6).

Original: In most cases, ‘priority’ works slightly better, while in some cases, such as FFHQ, ‘uniform’ is better.
Revised: In most cases, ‘priority’ works slightly better, while in some cases, such as FFHQ, ‘uniform’ is better.

Original: Overall, we didn’t modify anything in the model architectures and training hyperparameters, such as learning rate and batch size.
Revised: Overall, we didn’t modify anything in the model architectures and training hyperparameters, such as learning rate and batch size.

Original: The forward diffusion configuration and model training configurations are as follows.
Revised: The forward diffusion configuration and model training configurations are as follows.

Original: Diffusion config.
Revised: Diffusion config.

Original: For our diffusion-based noise injection, we set up a linearly increasing schedule for β t , where t ∈ { 1 , 2 , . . .
Revised: For our diffusion-based noise injection, we set up a linearly increasing schedule for β t , where t ∈ { 1 , 2 , . . .

Original: , T } .
Revised: , T } .

Original: For pixel level injection in StyleGAN2, we follows (Ho et al.
Revised: For pixel level injection in StyleGAN2, we follows (Ho et al.

Original: , while i ≤ number of training iterations do
Revised: , while i ≤ number of training iterations do

Original: • Sample minibatch of m noise samples { z 1 , z 2 , . . .
Revised: • Sample minibatch of m noise samples { z 1 , z 2 , . . .

Original: , z m } ∼ p
Revised: , z m } ∼ p

Original: z ( z ) .
Revised: z ( z ) .

Original: • Obtain generated samples { x g, 1 , x g, 2 , . . .
Revised: • Obtain generated samples { x g, 1 , x g, 2 , . . .

Original: , x g,m } by x g = G ( z ) .
Revised: , x g,m } by x g = G ( z ) .

Original: • Sample minibatch of m data examples { x 1 , x 2 , . . .
Revised: • Sample minibatch of m data examples { x 1 , x 2 , . . .

Original: , x m } ∼ p ( x ) .
Revised: , x m } ∼ p ( x ) .

Original: • Sample { t 1 , t 2 , . . .
Revised: • Sample { t 1 , t 2 , . . .

Original: , t m } from t epl list uniformly with replacement.
Revised: , t m } from t epl list uniformly with replacement.

Original: • For j ∈ { 1 , 2 , . . .
Revised: • For j ∈ { 1 , 2 , . . .

Original: , m } , sample y j ∼ q ( y j | x j , t j ) and y g,j ∼ q ( y g,j | x g,j , t j ) • Update discriminator by maximizing Equation (3).
Revised: , m } , sample y j ∼ q ( y j | x j , t j ) and y g,j ∼ q ( y g,j | x g,j , t j ) • Update discriminator by maximizing Equation (3).

Original: • Sample minibatch of m noise samples { z 1 , z 2 , . . .
Revised: • Sample minibatch of m noise samples { z 1 , z 2 , . . .

Original: , z m } ∼ p
Revised: , z m } ∼ p

Original: z ( z ) • Obtain generated samples { x g, 1 , x g, 2 , . . .
Revised: z ( z ) • Obtain generated samples { x g, 1 , x g, 2 , . . .

Original: , x g,m } by x g = G ( z ) .
Revised: , x g,m } by x g = G ( z ) .

Original: • Sample { t 1 , t 2 , . . .
Revised: • Sample { t 1 , t 2 , . . .

Original: , t m } from t epl list with replacement.
Revised: , t m } from t epl list with replacement.

Original: • For j ∈ { 1 , 2 , . . .
Revised: • For j ∈ { 1 , 2 , . . .

Original: , m } , sample y g,j ∼ q ( y g,j | x g,j , t j ) • Update generator by minimizing Equation (3).
Revised: , m } , sample y g,j ∼ q ( y g,j | x g,j , t j ) • Update generator by minimizing Equation (3).

Original: if i mod 4 == 0 then
Revised: if i mod 4 == 0 then

Original: Update T by Equation (5)
Revised: Update T by Equation (5)

Original: Sample t epl =
Revised: Sample t epl =

Original: , 0 , t 1 , . . .
Revised: , 0 , t 1 , . . .

Original: , t 32 ] , where t k ∼ p π for k ∈ { 1 , . .
Revised: , t 32 ] , where t k ∼ p π for k ∈ { 1 , . .

Original: p π is in Equation (6).
Revised: p π is in Equation (6).

Original: { t epl has 64 dimensions.
Revised: { t epl has 64 dimensions.

Original: } end if end while
Revised: } end if end while

Original: Datasets
Revised: Datasets

Original: [Table]
Revised: [Table]

Revised: 2020b)

Original: and set β 0 = 0 . 0001 and β T = 0 .
Revised: and set β 0 = 0 . 0001 and β T = 0 .

Original: We adaptively modify T ranging from T min = 5 to
Revised: We adaptively modify T ranging from T min = 5 to

Original: T max = 1000 .
Revised: T max = 1000 .

Original: The image pixels are usually rescaled to [ − 1 , 1] so we set the Guassian noise standard deviation σ = 0 .
Revised: The image pixels are usually rescaled to [ − 1 , 1] so we set the Guassian noise standard deviation σ = 0 .

Original: For feature level injection in Diffusion ProjectedGAN, we set β 0 = 0 .
Revised: For feature level injection in Diffusion ProjectedGAN, we set β 0 = 0 .

Original: 0001 , β T = 0 .
Revised: 0001 , β T = 0 .

Original: 01 , T min = 5 , T max = 500 and σ = 0 .
Revised: 01 , T min = 5 , T max = 500 and σ = 0 .

Original: We list all these values in Table
Revised: We list all these values in Table

Original: Model config.
Revised: Model config.

Original: For StyleGAN2-based models, we borrow the config settings provided by Karras et al.
Revised: For StyleGAN2-based models, we borrow the config settings provided by Karras et al.

Original: (2020a), which include [‘auto’, ‘stylegan2’, ‘cifar’, ‘paper256’, ‘paper512’, ‘stylegan2’].
Revised: (2020a), which include [‘auto’, ‘stylegan2’, ‘cifar’, ‘paper256’, ‘paper512’, ‘stylegan2’].

Original: We create the ‘stl’ config based on ‘cifar’ with a small modification that we change the gamma term to be 0.01.
Revised: We create the ‘stl’ config based on ‘cifar’ with a small modification that we change the gamma term to be 0.01.

Original: For ProjectedGAN models, we use the recommended default config (Sauer et al., 2021), which is based on FastGAN (Liu et al., 2020).
Revised: For ProjectedGAN models, we use the recommended default config (Sauer et al., 2021), which is based on FastGAN (Liu et al., 2020).

Original: We report the config settings used for our experiments in Table 6.
Revised: We report the config settings used for our experiments in Table 6.

Original: Diffusion config for pixel, priority
Revised: Diffusion config for pixel, priority

Original: [Table]
Revised: [Table]

Original: for

Original: [Table]

Original: , uniform

Original: [Table]

Original: Diffusion config for featureβ
Revised: Diffusion config for feature

Original: [Table] [Table]
Revised: [Table] [Table]

Original: that we change the gamma term to be 0.01.
Revised: that we change the gamma term to be 0.01.

Original: For ProjectedGAN models, we use the recommended default config (Sauer et al., 2021), which is based on FastGAN.
Revised: For ProjectedGAN models, we use the recommended default config (Sauer et al., 2021), which is based on FastGAN.

Original: H I MPLEMENTATION DETAILS
Revised: H I MPLEMENTATION DETAILS

Original: We implement an additional diffusion sampling pipeline, where the diffusion configurations are set in Appendix G. The T in the forward diffusion process is adaptively adjusted and clipped to [ T min , T max ] .
Revised: We implement an additional diffusion sampling pipeline, where the diffusion configurations are set in Appendix G. The T in the forward diffusion process is adaptively adjusted and clipped to [ T min , T max ] .

Original: As illustrated in Algorithm 1, at each update step, we sample t from t epl for each data point x , and then use the analytic Gaussian distribution at diffusion step t to sample y .
Revised: As illustrated in Algorithm 1, at each update step, we sample t from t epl for each data point x , and then use the analytic Gaussian distribution at diffusion step t to sample y .

Original: Next, we use y and t instead of x for optimization.
Revised: Next, we use y and t instead of x for optimization.

Original: Diffusion StyleGAN2.
Revised: Diffusion StyleGAN2.

Original: We inherit all the network architectures from StyleGAN2 implemented by Karras et al.
Revised: We inherit all the network architectures from StyleGAN2 implemented by Karras et al.

Original: (2020a).
Revised: (2020a).

Original: We modify the original mapping network, which is there for label conditioning and unused for unconditional image generation tasks, inside the discriminator to inject t .
Revised: We modify the original mapping network, which is there for label conditioning and unused for unconditional image generation tasks, inside the discriminator to inject t .

Original: Specifically, we change the original input of mapping network, the class label c , to our discrete value timestep t .
Revised: Specifically, we change the original input of mapping network, the class label c , to our discrete value timestep t .

Original: Then, we train the generator and discriminator with diffused samples y and t .
Revised: Then, we train the generator and discriminator with diffused samples y and t .

Original: Diffuson ProjectedGAN.
Revised: Diffuson ProjectedGAN.

Original: To simplify the implementation and minimize the modifications to
Revised: To simplify the implementation and minimize the modifications to

Original: ProjectedGAN, we construct the discriminator as D ϕ ( y ) , where t is ignored.
Revised: ProjectedGAN, we construct the discriminator as D ϕ ( y ) , where t is ignored.

Original: Our method is plugged in as a data augmentation method.
Revised: Our method is plugged in as a data augmentation method.

Original: The only change in the optimization stage is that the discriminator is fed with diffused images y instead of original images x .
Revised: The only change in the optimization stage is that the discriminator is fed with diffused images y instead of original images x .

Original: Diffuson InsGen.
Revised: Diffuson InsGen.

Original: To simplify the implementation and minimize the modifications to InsGen, we keep their contrastive learning part untouched.
Revised: To simplify the implementation and minimize the modifications to InsGen, we keep their contrastive learning part untouched.

Original: We modify the original discriminator network to inject t similarly to Diffusion StyleGAN2.
Revised: We modify the original discriminator network to inject t similarly to Diffusion StyleGAN2.

Original: Then, we train the generator and discriminator with diffused samples y and t .
Revised: Then, we train the generator and discriminator with diffused samples y and t .

Original: I A BLATION ON THE MIXING PROCEDURE 
Revised: I A BLATION ON THE MIXING PROCEDURE AND T ADAPTIVENESS

Original: Note the mixing procedure described in Equation (6), referred to as “priority mixing” in what follows, is designed based on our intuition.
Revised: Note the mixing procedure described in Equation (6), referred to as “priority mixing” in what follows, is designed based on our intuition.

Original: Here we conduct an ablation study on the mixing procedure  by comparing the priority mixing with uniform mixing on three representative datasets.
Revised: Here we conduct an ablation study on the mixing procedure [Figure] by comparing the priority mixing with uniform mixing on three representative datasets.

Original: We report in
Revised: We report in

Original: Table 7 the FID results, which suggest that uniform mixing could work better than priority mixing in some dataset, and hence Diffusion-GAN may be further improved by optimizing its mixing procedure according to the training data.
Revised: Table 7 the FID results, which suggest that uniform mixing could work better than priority mixing in some dataset, and hence Diffusion-GAN may be further improved by optimizing its mixing procedure according to the training data.

Original: While optimizing the mixing procedure is beyond the focus of this paper, it is worth further investigation in future studies.
Revised: While optimizing the mixing procedure is beyond the focus of this paper, it is worth further investigation in future studies.

Original: [Table]
Revised: [Table]

Revised: We further conduct ablation study on whether the T needs to be adaptively adjusted.

Revised: As shown in

Revised: Figure 7, we observe with adaptive diffusion strategy, the training curves of FIDs converge faster and reach lower final FIDs.

Original: J M ORE GAN VARIANTS
Revised: J M ORE GAN VARIANTS

Original: To further validate our noise injection via diffusion-based mixtures, we add our diffusion-based training into two more representative GAN variants: DCGAN (Radford et al., 2015) and SNGAN (Miyato et al., 2018b), which have quite different GAN architectures compared to StyleGAN2.
Revised: To further validate our noise injection via diffusion-based mixtures, we add our diffusion-based training into two more representative GAN variants: DCGAN (Radford et al., 2015) and SNGAN (Miyato et al., 2018b), which have quite different GAN architectures compared to StyleGAN2.

Original: We provide the FIDs for CIFAR-10 in Table 8.
Revised: We provide the FIDs for CIFAR-10 in Table 8.

Original: We observe that both Diffusion-DCGAN and DiffusionSNGAN clearly outperform their corresponding baseline GANs.
Revised: We observe that both Diffusion-DCGAN and DiffusionSNGAN clearly outperform their corresponding baseline GANs.

Original: [Table]
Revised: [Table]

Revised: K I NCEPTION S CORE FOR CIFAR-

Revised: We report the Inception Score (IS) (Salimans et al., 2016) of Diffusion StyleGAN2 for CIFARdataset in Table 9 and also include other state-of-the-art GANs and diffusion models as baselines.

Revised: Note CIFAR-10 is a well-known dataset and tested by almost all baselines, so we pick CIFARhere and we reference the reported IS values from their original papers for a fair comparison.

Revised: [Table]

Original: K M ORE GENERATED IMAGES
Revised: L M ORE GENERATED IMAGES

Original: We provide more randomly generated images for LSUN-Bedroom, LSUN-Church, AFHQ, and
Revised: We provide more randomly generated images for LSUN-Bedroom, LSUN-Church, AFHQ

Original:  FFHQ datasets in Figure7, Figure 8,  and Figure 9. [Figure]
Revised: [Table] and FFHQ datasets in Figure 8, Figure 9, and Figure 10.