Original: PA-GAN: I MPROVING GAN T RAINING BY P ROGRESSIVE A UGMENTATION
Revised: PA-GAN: I MPROVING GAN T RAINING BY P ROGRESSIVE A UGMENTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite recent progress, Generative Adversarial Networks (GANs) still suffer from training instability, requiring careful consideration of architecture design choices and hyper-parameter tuning.
Revised: Despite recent progress, Generative Adversarial Networks (GANs) still suffer from training instability, requiring careful consideration of architecture design choices and hyper-parameter tuning.

Original: The reason for this fragile training behaviour is partially due to the discriminator performing well very quickly; its loss converges to zero, providing no reliable backpropagation signal to the generator.
Revised: The reason for this fragile training behaviour is partially due to the discriminator performing well very quickly; its loss converges to zero, providing no reliable backpropagation signal to the generator.

Original: In this work we introduce a new technique - progressive augmentation of GANs (PAGAN) - that helps to overcome this fundamental limitation and improve the overall stability of GAN training.
Revised: In this work we introduce a new technique - progressive augmentation of GANs (PAGAN) - that helps to overcome this fundamental limitation and improve the overall stability of GAN training.

Original: The key idea is to gradually increase the task difﬁculty of the discriminator by progressively augmenting its input space, thus enabling continuous learning of the generator.
Revised: The key idea is to gradually increase the task difﬁculty of the discriminator by progressively augmenting its input space, thus enabling continuous learning of the generator.

Original: We show that the proposed progressive augmentation preserves the original GAN objective, does not bias the optimality of the discriminator and encourages the healthy competition between the generator and discriminator, leading to a better-performing generator.
Revised: We show that the proposed progressive augmentation preserves the original GAN objective, does not bias the optimality of the discriminator and encourages the healthy competition between the generator and discriminator, leading to a better-performing generator.

Original: We experimentally demonstrate the effectiveness of the proposed approach on multiple benchmarks (MNIST, Fashion-MNIST, CIFAR10, CELEBA) for the image generation task.
Revised: We experimentally demonstrate the effectiveness of the proposed approach on multiple benchmarks (MNIST, Fashion-MNIST, CIFAR10, CELEBA) for the image generation task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are a recent development in the ﬁeld of deep learning, that have attracted a lot of attention in the research community (Radford et al., 2016; Salimans et al., 2016; Arjovsky et al., 2017; Karras et al., 2018).
Revised: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are a recent development in the ﬁeld of deep learning, that have attracted a lot of attention in the research community (Radford et al., 2016; Salimans et al., 2016; Arjovsky et al., 2017; Karras et al., 2018).

Original: GANs fall into the category of generative models, i.e. models that allow sampling of new data points from encoded high-dimensional data distributions, such as images.
Revised: GANs fall into the category of generative models, i.e. models that allow sampling of new data points from encoded high-dimensional data distributions, such as images.

Original: The GAN framework can be formulated as a competing game between the generator and the discriminator.
Revised: The GAN framework can be formulated as a competing game between the generator and the discriminator.

Original: Mathematically, training GANs requires solving a min-max problem.
Revised: Mathematically, training GANs requires solving a min-max problem.

Original: Since both the generator and the discriminator are typically parameterized as deep convolutional neural networks with millions of parameters, optimization is notoriously difﬁcult in practice (Arjovsky et al., 2017;
Revised: Since both the generator and the discriminator are typically parameterized as deep convolutional neural networks with millions of parameters, optimization is notoriously difﬁcult in practice (Arjovsky et al., 2017;

Original: Gulrajani et al., 2017; Miyato et al., 2018).
Revised: Gulrajani et al., 2017; Miyato et al., 2018).

Original: The difﬁculty lies in maintaining the healthy competition between the generator and discriminator.
Revised: The difﬁculty lies in maintaining the healthy competition between the generator and discriminator.

Original: A commonly occurring problem arises when the discriminator overshoots, leading to escalated gradients and oscillatory GAN behaviour (Mescheder et al., 2018).
Revised: A commonly occurring problem arises when the discriminator overshoots, leading to escalated gradients and oscillatory GAN behaviour (Mescheder et al., 2018).

Original: As a result the generator fails to learn the multimodal structure of the true distribution.
Revised: As a result the generator fails to learn the multimodal structure of the true distribution.

Original: Moreover, the supports of the data and model distributions typically lie on low dimensional manifolds and are often disjoint (Arjovsky & Bottou, 2017).
Revised: Moreover, the supports of the data and model distributions typically lie on low dimensional manifolds and are often disjoint (Arjovsky & Bottou, 2017).

Original: Consequently, there exists a nearly trivial discriminator that can perfectly distinguish real data samples from synthetic ones.
Revised: Consequently, there exists a nearly trivial discriminator that can perfectly distinguish real data samples from synthetic ones.

Original: Once such a discriminator is produced, its loss quickly converges to zero and the gradients used for updating parameters of the generator become useless.
Revised: Once such a discriminator is produced, its loss quickly converges to zero and the gradients used for updating parameters of the generator become useless.

Original: In this work we introduce a new technique to overcome this problem - progressive augmentation of GANs (PA-GAN) - that helps to control the behaviour of the discriminator and thus improve the overall training stability.
Revised: In this work we introduce a new technique to overcome this problem - progressive augmentation of GANs (PA-GAN) - that helps to control the behaviour of the discriminator and thus improve the overall training stability.

Original: The key idea is to progressively augment the input of the discriminator network with auxiliary random variables, enlarging the sample space dimensionality, in order to gradually increase the discrimination task difﬁculty (see Figure 1).
Revised: The key idea is to progressively augment the input of the discriminator network with auxiliary random variables, enlarging the sample space dimensionality, in order to gradually increase the discrimination task difﬁculty (see Figure 1).

Original: In doing so, the discriminator can be prevented from becoming over-conﬁdent, enabling continuous learning of the generator.
Revised: In doing so, the discriminator can be prevented from becoming over-conﬁdent, enabling continuous learning of the generator.

Original: As opposed to standard data augmentation techniques (e.g. rotation, cropping, resizing), the proposed progressive augmentation does not directly modify the data samples, but rather is structurally appended to them.
Revised: As opposed to standard data augmentation techniques (e.g. rotation, cropping, resizing), the proposed progressive augmentation does not directly modify the data samples, but rather is structurally appended to them.

Original: In particular, for the single level augmentation along with the data sample x the discriminator takes also as input the binary random variable s ∈ { 0 , 1 } .
Revised: In particular, for the single level augmentation along with the data sample x the discriminator takes also as input the binary random variable s ∈ { 0 , 1 } .

Original: The class of the augmented sample ( x , s ) is then set based on the combination x with s , resulting in real and synthetic samples contained in both true and fake classes.
Revised: The class of the augmented sample ( x , s ) is then set based on the combination x with s , resulting in real and synthetic samples contained in both true and fake classes.

Original: This presents a more challenging task for the discriminator, as it needs to tell the real and synthetic samples apart and additionally understand the association rule.
Revised: This presents a more challenging task for the discriminator, as it needs to tell the real and synthetic samples apart and additionally understand the association rule.

Original: We can further increase the task difﬁculty of the discriminator by progressively augmenting its input space and enlarging the dimensionality of
Revised: We can further increase the task difﬁculty of the discriminator by progressively augmenting its input space and enlarging the dimensionality of

Original: [Figure]
Revised: [Figure]

Original: We show that the proposed progressive augmentation preserves the original GAN objective and is an outcome of a systematic derivation.
Revised: We show that the proposed PA-GAN preserves the original GAN objective and is an outcome of a systematic derivation.

Original: In contrast to prior work (Salimans et al., 2016; Sønderby et al., 2017; Arjovsky & Bottou, 2017), it does not bias the optimality of the discriminator (see Sec. 4)
Revised: In contrast to prior work (Salimans et al., 2016; Sønderby et al., 2017; Arjovsky & Bottou, 2017), it does not bias the optimality of the discriminator (see Sec. 4).

Original: Structurally augmenting the input sample space and mapping it to higher dimensions not only makes the discriminator task challenging, but, in addition, encourages the generator to explore various paths  thetrue data distribution it intelligent the set
Revised: Structurally augmenting the input sample space and mapping it to higher dimensions not only challenges the discrimination task, but, in addition, encourages the generator to explore various paths towards the data distribution, leading to improved variation of the generated samples (see Sec. 5).

Original: Our technique is orthogonal to existing work, it can be successfully employed with other regularizations strategies (Miyato et al., 2018; Gulrajani et al., 2017) and different network architectures (Chen et al., 2016; Radford et al., 2016), which we demonstrate in Sec.
Revised: Our technique is orthogonal to existing work, it can be successfully employed with other regularizations strategies (Miyato et al., 2018; Gulrajani et al., 2017) and different network architectures (Chen et al., 2016; Radford et al., 2016), which we demonstrate in Sec.

Original: We experimentally show the effectiveness of the progressive augmentation of GANs for image generation tasks on multiple benchmarks (MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), CIFAR (Krizhevsky, 2009), CELEBA (Liu et al., 2015)) across different evaluation metrics (IS (Theis et al., 2016), FID (Husz ´ ar, 2015), KID (Bi ´ nkowski et al., 2018)).
Revised: We experimentally show the effectiveness of the progressive augmentation of GANs for image generation tasks on multiple benchmarks (MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), CIFAR (Krizhevsky, 2009), CELEBA (Liu et al., 2015)) across different evaluation metrics (IS (Theis et al., 2016), FID (Husz ´ ar, 2015), KID (Bi ´ nkowski et al., 2018)).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Many recent works have focused on improving the stability of GAN training and the overall visual quality of generated samples (Radford et al., 2016;
Revised: Many recent works have focused on improving the stability of GAN training and the overall visual quality of generated samples (Radford et al., 2016;

Original: Roth et al., 2017; Karras et al., 2018; Gulrajani et al., 2017; Miyato et al., 2018).
Revised: Roth et al., 2017; Karras et al., 2018; Gulrajani et al., 2017; Miyato et al., 2018).

Original: As reported by Arjovsky & Bottou (2017), the reason for the unstable behaviour of GANs is partly due to a dimensional mismatch or non-overlapping support between the real data and the generative model distributions, resulting in an almost trivial task for the discriminator.
Revised: As reported by Arjovsky & Bottou (2017), the reason for the unstable behaviour of GANs is partly due to a dimensional mismatch or non-overlapping support between the real data and the generative model distributions, resulting in an almost trivial task for the discriminator.

Original: Once the performance of the discriminator is maxed out, it provides a non-informative signal to train the generator.
Revised: Once the performance of the discriminator is maxed out, it provides a non-informative signal to train the generator.

Original: To avoid vanishing gradients, the original GAN paper (Goodfellow et al., 2014) proposed to modify the min-max based GAN objective (MM GAN) to a non-saturating loss (NS GAN).
Revised: To avoid vanishing gradients, the original GAN paper (Goodfellow et al., 2014) proposed to modify the min-max based GAN objective (MM GAN) to a non-saturating loss (NS GAN).

Original: However, even with such a re-formulation the generator updates tend to get worse over the course of training and optimization becomes massively unstable (Arjovsky & Bottou, 2017).
Revised: However, even with such a re-formulation the generator updates tend to get worse over the course of training and optimization becomes massively unstable (Arjovsky & Bottou, 2017).

Original: Prior approaches tried to mitigate this issue bymaking the discriminator weaker using heuristics, such as decreasing its learning rate, adding label or input noise.
Revised: Prior approaches tried to mitigate this issue by using heuristics to weaken the discriminator, such as decreasing its learning rate, adding label or input noise.

Original: Salimans et al.
Revised: Salimans et al.

Original: (2016) proposed a one-sided label smoothing technique to smoothen the classiﬁcation boundary of the discriminator, thereby preventing it from being overly conﬁdent, but at the same time biasing its optimality.
Revised: (2016) proposed a one-sided label smoothing technique to smoothen the classiﬁcation boundary of the discriminator, thereby preventing it from being overly conﬁdent, but at the same time biasing its optimality.

Original: The works of Arjovsky & Bottou (2017) and Sønderby et al.
Revised: The works of Arjovsky & Bottou (2017) and Sønderby et al.

Original: (2017) made the job of the discriminator harder by adding Gaussian noise to both generated and real samples.
Revised: (2017) made the job of the discriminator harder by adding Gaussian noise to both generated and real samples.

Original: Moving the manifolds of the data and model distributions closer to each other by adding the input noise ensures a meaningful overlap between their supports, which is desired in order for the generator to eventually approach the data distribution.
Revised: Moving the manifolds of the data and model distributions closer to each other by adding the input noise ensures a meaningful overlap between their supports, which is desired in order for the generator to eventually approach the data distribution.

Original: However, adding high-dimensional noise introduces signiﬁcant variance in the parameter estimation, which slows down the training and requires multiple samples for counteraction.
Revised: However, adding high-dimensional noise introduces signiﬁcant variance in the parameter estimation, which slows down the training and requires multiple samples for counteraction (Roth et al., 2017). Similarly, Sajjadi et al.

Revised: (2018) proposed to blur the input samples and gradually remove the blurring effect during the course of training.

Original: Instead of adding noise to the input, Zhang et al.
Revised: Instead of adding noise to the input, Zhang et al.

Original: (2018b) created mixup samples by interpolating between synthetic and real ones, which leads to a more stable behaviour of GANs.
Revised: (2018b) created mixup samples by interpolating between synthetic and real ones, which leads to a more stable behaviour of GANs.

Original: Both techniques, i.e., additive noise and sample mixup, perform direct modiﬁcations on the data samples.
Revised: These techniques, i.e., additive noise, blurring and sample mixup, perform direct modiﬁcations on the data samples.

Original: Another line of work resorts to cost function reformulation to improve the stability of GAN training, e.g. by using the Pearson χ 2 divergence for least square GANs (LS GANs) (Mao et al., 2016), kernel maximum mean discrepancy (MMD) for MMD-GANs (Li et al., 2017; Dziugaite et al., 2015), or f-divergence for f-GANs (Nowozin et al., 2016).
Revised: Another line of work resorts to cost function reformulation to improve the stability of GAN training, e.g. by using the Pearson χ 2 divergence for least square GANs (LS GANs) (Mao et al., 2016), kernel maximum mean discrepancy (MMD) for MMD-GANs (Li et al., 2017; Dziugaite et al., 2015), or f-divergence for f-GANs (Nowozin et al., 2016).

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2017) proposed the Wasserstein GAN (WGAN) with the training objective derived from the Wasserstein distance, aiming to mitigate the vanishing gradient problem.
Revised: (2017) proposed the Wasserstein GAN (WGAN) with the training objective derived from the Wasserstein distance, aiming to mitigate the vanishing gradient problem.

Original: The drawback of this approach is the weight clipping of the discriminator employed to enforce smoothness, which adversely reduces the capacity of the discriminator.
Revised: The drawback of this approach is the weight clipping of the discriminator employed to enforce smoothness, which adversely reduces the capacity of the discriminator.

Original: Alternative to weight clipping, Gulrajani et al.
Revised: Alternative to weight clipping, Gulrajani et al.

Original: (2017) added a soft penalty on the gradient norm which ensures a 1-Lipschitz discriminator.
Revised: (2017) added a soft penalty on the gradient norm which ensures a 1-Lipschitz discriminator.

Original: The gradient norm penalty can be seen as a weight regularization technique for the discriminator and was shown to improve the performance with other losses as well (Fedus et al., 2018).
Revised: The gradient norm penalty can be seen as a weight regularization technique for the discriminator and was shown to improve the performance with other losses as well (Fedus et al., 2018).

Original: Similarly, Roth et al.
Revised: Similarly, Roth et al.

Original: (2017) proposed to add a penalty on the weighted gradient-norm of the discriminator in the context of f-divergences, showing its equivalence to adding input noise.
Revised: (2017) proposed to add a penalty on the weighted gradient-norm of the discriminator in the context of f-divergences, showing its equivalence to adding input noise.

Original: On the downside, regularizing the discriminator with the gradient penalty depends on the model distribution, which changes during training andthus results in increased runtime.
Revised: On the downside, regularizing the discriminator with the gradient penalty depends on the model distribution which changes during training and results in increased runtime due to additional gradient norm computation.

Original: Miyato et al.
Revised: Miyato et al.

Original: (2018) proposed another way to stabilize the discriminator by normalizing its weights and limiting the spectral norm of each layer to constrain the Lipschitz constant.
Revised: (2018) proposed another way to stabilize the discriminator by normalizing its weights and limiting the spectral norm of each layer to constrain the Lipschitz constant.

Original: This normalization technique does not require intensive tuning of hyper-parameters and is computationally light.
Revised: This normalization technique does not require intensive tuning of hyper-parameters and is computationally light.

Original: Most recently, Zhang et al.
Revised: Most recently, Zhang et al.

Original: (2018a) showed that spectral normalization is also beneﬁcial for the generator by preventing the escalation of parameter magnitudes and avoiding unusual gradients.
Revised: (2018a) showed that spectral normalization is also beneﬁcial for the generator by preventing the escalation of parameter magnitudes and avoiding unusual gradients.

Original: Several methods have proposed to modify the training methodology of GANs in order to further improve stability, e.g. by considering multiple discriminators with different roles (Durugkar et al., 2017) or growing both the generator and discriminator networks progressively (Karras et al., 2018).
Revised: Several methods have proposed to modify the training methodology of GANs in order to further improve stability, e.g. by considering multiple discriminators with different roles (Durugkar et al., 2017) or growing both the generator and discriminator networks progressively (Karras et al., 2018).

Original: In this work we introduce an orthogonal way to stabilize the GAN training by progressively increasing the discrimination task difﬁculty.
Revised: In this work we introduce an orthogonal way to stabilize the GAN training by progressively increasing the discrimination task difﬁculty.

Original: To this end, a novel and structured way of augmenting the discriminator input space is proposed.
Revised: To this end, a novel and structured way of augmenting the discriminator input space is proposed.

Original: In contrast to other techniques, our method does not bias the optimality of the discriminator or alter the training samples.
Revised: In contrast to other techniques, our method does not bias the optimality of the discriminator or alter the training samples.

Original: Furthermore, the proposed augmentation is complementary to prior work.
Revised: Furthermore, the proposed augmentation is complementary to prior work.

Original: It can be employed with different GAN architectures, adapted to various divergence measures and combined with other regularization techniques (see Sec. 5).
Revised: It can be employed with different GAN architectures, adapted to various divergence measures and combined with other regularization techniques (see Sec. 5).

Original:  T HEORETICAL B ACKGROUND
Revised:  T HEORETICAL B ACKGROUND

Original: For generative modeling, one common approach is to adopt divergence measures as loss functions for the generator.
Revised: For generative modeling, one common approach is to adopt divergence measures as loss functions for the generator.

Original: Our method belongs to this line of work.
Revised: Our method belongs to this line of work.

Original: In contrast to prior work, our primary focus is not on explicitly minimizing the divergence between the data and model distributions deﬁned on the sample space X .
Revised: In contrast to prior work, our primary focus is not on explicitly minimizing the divergence between the data and model distributions deﬁned on the sample space X .

Original: Alternatively, we ﬁrst structurally augment the training samples (both real and synthetic ones) and then minimize the divergence between distributions deﬁned on the augmented sample space.
Revised: Alternatively, we ﬁrst structurally augment the training samples (both real and synthetic ones) and then minimize the divergence between distributions deﬁned on the augmented sample space.

Original: For computing the divergence, we adopt the adversary process introduced by (Goodfellow et al., 2014)
Revised: For computing the divergence, we adopt the adversary process introduced by (Goodfellow et al., 2014)

Original: (Sec. 3.1), while the proposed augmentation is inspired by the information theory view of Jensen-Shannon (JS) divergence (Sec. 3.2).
Revised: (Sec. 3.1), while the proposed augmentation is inspired by the information theory view of Jensen-Shannon (JS) divergence (Sec. 3.2).

Original: Both of them are brieﬂy reviewed in this section to lay the theoretical groundwork for our method, which we then discuss in Sec. 4.
Revised: Both of them are brieﬂy reviewed in this section to lay the theoretical groundwork for our method, which we then discuss in Sec. 4.

Original: 3. A DVERSARY P ROCESS OF GAN S
Revised: 3. A DVERSARY P ROCESS OF GAN S

Original: Let X denote a compact metric space such as the image space [0 , 1] d of dimension d .
Revised: Let X denote a compact metric space such as the image space [0 , 1] d of dimension d .

Original: The data distribution P d and the model distribution P g are both probability measures deﬁned on X .
Revised: The data distribution P d and the model distribution P g are both probability measures deﬁned on X .

Original: In the context of GANs, P g is commonly induced by a function G that maps a random noise vector z , following a given prior distribution P z , to a synthetic data sample, i.e. x g = G ( z ) ∈ X .
Revised: In the context of GANs, P g is commonly induced by a function G that maps a random noise vector z , following a given prior distribution P z , to a synthetic data sample, i.e. x g = G ( z ) ∈ X .

Original: The core idea behind GAN training is to set up a competing game between two players, commonly termed the discriminator D and generator G .
Revised: The core idea behind GAN training is to set up a competing game between two players, commonly termed the discriminator D and generator G .

Original: Mathematically, their objective can be formulated as
Revised: Mathematically, their objective can be formulated as

Original: [Equation] 
Revised: [Equation] [Equation]

Original:  with D : X (cid:55)→ [0 , 1] .
Revised: ) with D : X (cid:55)→ [0 , 1] .

Original: The optimal D ∗ respectively classiﬁes x ∼ P d and x ∼ P g as TRUE and
Revised: The optimal D ∗ respectively classiﬁes x ∼ P d and x ∼ P g as TRUE and

Original: FAKE, i.e. binary classiﬁcation.
Revised: FAKE, i.e. binary classiﬁcation.

Original: Its achieved maximum equals the JS divergence between P d and P g , which is then used as the loss function by the generator to optimize G (Goodfellow et al., 2014).
Revised: Its achieved maximum equals the JS divergence between P d and P g , which is then used as the loss function by the generator to optimize G (Goodfellow et al., 2014).

Original: 3. I NFORMATION T HEORY V IEWPOINT
Revised: 3. I NFORMATION T HEORY V IEWPOINT

Original: Apart from measuring the similarity of a distribution pair, the JS divergence has an information theory interpretation that inspires our approach presented in Sec. 4.
Revised: Apart from quantifying distributions’ similarity, the JS divergence has an information theory interpretation that inspires our approach presented in Sec. 4.

Original: In accordance with the binary classiﬁcation task of the discriminator, we introduce a binary random variable s with a uniform distribution P s .
Revised: In accordance with the binary classiﬁcation task of the discriminator, we introduce a binary random variable s with a uniform distribution P s .

Original: Associating s = 0 and s = 1 respectively with x ∼ P d and x ∼ P g , we obtain a joint distribution

Original: [Equation]

Original: checksum
Revised: checksum

Original: True
Revised: True

Original: Fake yes no
Revised: Fake yes no

Original: Even?
Revised: Even?

Original: N
Revised: N

Original: Level L=
Revised: Level L=

Original: L=
Revised: L=

Original: L=N
Revised: L=N

Original: Data
Revised: Data

Original: Progression Levels ...
Revised: Progression Levels ...

Original: The marginal distribution withrespect to x (a.k.a.
Revised: Associating s = 0 and s = 1 respectively with x ∼ P d and x ∼ P g , we obtain a joint distribution P [Equation] x

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: P s

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: P

Revised: [Equation]

Revised: x | s ) with P

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: (cid:26)

Revised: [Equation]

Revised: The marginal distribution with respect to x (a.k.a.

Original: the mixture distribution) is equal to
Revised: the mixture distribution) is equal to

Original: [Equation]
Revised: [Equation]

Original: Computing the mutual information of the two random variables s and x based on P ( x , s ) , we have
Revised: Computing the mutual information of the two random variables s and x based on P ( x , s ) , we have

Original: [Equation]
Revised: [Equation]

Original: where the equality ( a ) is the outcome of computing the two entropy terms based on the reference measure P m .
Revised: where the equality ( a ) is the outcome of computing the two entropy terms based on the reference measure P m .

Original: The minimum of the JS divergence D JS ( P d (cid:107) P g ) equal to zero is attainable iff P d
Revised: The minimum of the JS divergence D JS ( P d (cid:107) P g ) equal to zero is attainable iff P d

Original: = P g .
Revised: = P g .

Original: This condition makes the joint distribution function P ( x , s ) factorizable, indicating the independence between x and s , and thereby yielding zero mutual information.
Revised: This condition makes the joint distribution function P ( x , s ) factorizable, indicating the independence between x and s , and thereby yielding zero mutual information.

Original:  P ROGRESSIVELY A UGMENTED GAN T RAINING
Revised:  P ROGRESSIVELY A UGMENTED GAN T RAINING

Original: Relying on the information theory view of the JS divergence given in the previous section, we can cast the optimization objective of the generator as mutual information minimization
Revised: Relying on the information theory view of the JS divergence given in the previous section, we can cast the optimization objective of the generator as mutual information minimization

Original: [Equation]
Revised: [Equation]

Original: Based on (5), in Sec.
Revised: Based on (5), in Sec.

Original: 4.1 we will ﬁrst derive a series of equivalent problems, particularly showing how the auxiliary random bit s leads us to a novel and structured way to augment the sample x ∈ X for training the discriminator.
Revised: 4.1 we will ﬁrst present an equivalent problem to (5), particularly showing how the auxiliary random bit s leads us to a novel and structured way to augment the sample x ∈ X for training the discriminator.

Original: By further identifying a common principle behind theseries of equivalent problems, in Sec. 4.2 we extend the single level augmentation based on one bit s to progressive multi-level augmentation with an arbitrarily long random bit sequence.
Revised: By further identifying a common principle behind the equivalent problems, in Sec. 4.2 we extend the single level augmentation based on one bit s to progressive multi-level augmentation with an arbitrarily long random bit sequence.

Original: Progressively increasing the number of augmentation levels equips us with a new mechanism to balance the twoplayer competing game.
Revised: Progressively increasing the number of augmentation levels equips us with a new mechanism to balance the two-player competing game.

Original: In Sec.
Revised: In Sec.

Original: 4.3 we describe the integration of the proposed augmentation into neural networks and present how to schedule the augmentation progression during training.
Revised: 4.3 we describe the integration of the proposed augmentation into neural networks and present how to schedule the augmentation progression during training.

Original: 4. S INGLE L EVEL A UGMENTATION
Revised: 4. S INGLE L EVEL A UGMENTATION

Original: Starting from the case of single level augmentation with one bit s , we ﬁrst prove the equivalence of the following problems:
Revised: Starting from the case of single level augmentation with one bit s , we ﬁrst note that the following two minimization problems are equivalent

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst ≡ , i.e. ( a ) in (6), is a convenient copy of (5) followed by ( b ) due to the relation of mutual information with the Kullback-Leibler (KL) divergence.

Original: Namely, the mutual information I ( x ; s ) can also be expressed as the KL divergence of the product of the marginal distributions, P ( x ) P s ( s ) , of the two random variables x and s from their joint distribution P ( x , s ) .

Original: Since the KL divergence as a loss function aims to reduce the difference between two distributions, it can be replaced by the

Original: JS divergence as they are minimized under the same condition.

Original: This yields the equivalence ( c ) in (7).

Original: Proceeding to ( d ) in (7), we note that the product of the two marginal distributions equals

Original: [Equation]

Original: where P ( x , s ) has been deﬁned in (2) and Q ( x , s ) is given asfollows
Revised: where P ( x , s ) has been deﬁned in (2) and Q ( x , s ) is constructed as

Original: [Equation]
Revised: [Equation]

Original: Comparing (9) with (2), Q ( x , s ) is simply an outcome of ﬂipping the associationrule of  s ∈ { 0 , 1 } with x d and  x g . With the relation given in (8), minimizing the JS divergence between P ( x ) P s ( s ) and  P  ( x, s ) requires P ( x , s ) and Q ( x , s ) being mutually identical, i.e. D JS ( P ( x , s ) (cid:107) Q ( x , s ))
Revised: The two joint distribution functions P ( x , s ) and Q ( x , s ) differ from each other by the association of the bit s ∈ { 0 , 1 } with the data and synthetic samples, respectively. Their marginals with respect to x are neither the data nor the generative model distribution, but by construction are identical and equal to P m ( x ) as given in (3). It is worth noting that the equivalence holds even if the feasible solution set of P g determined by G does not include the data distribution P d .

Original: = 0 .
Revised: This is of practical interest as it is often difﬁcult to guarantee the fulﬁllment of such premise when modelling G .

Original: With that, we close the  proof of the equivalences given in (6)-(7).
Revised: For the detailed proof of the equivalence in 6 we refer the reader to App. A.1.

Original: Optimizing G based on (7) instead of (5), the objective of the discriminator Dsubsequently converges to estimation of the JS divergence between P ( x , s ) and Q ( x , s )
Revised: Next, we attempt to solve the equivalent problem given in (6), following the adversary learning process behind GANs. Speciﬁcally, the objective of the discriminator D converges to estimation of the JS divergence between P ( x , s ) and Q ( x , s )

Original: [Equation]
Revised: [Equation]

Original: Comparing with the original discrimination task in GANs, i.e. (1), two differences are worth noting.
Revised: Comparing with the original discrimination task in GANs, i.e. (1), two differences are worth noting.

Original: First, the above discriminator takes s in addition to the sample x ∈ X as the input.
Revised: First, the above discriminator takes s in addition to the sample x ∈ X as the input.

Original: We, therefore, view s as a single level of augmentation to the sample x .
Revised: We, therefore, view s as a single level of augmentation to the sample x .

Original: Second, the distributions that form the two classes (i.e. TRUE vs. FAKE) become P ( x , s ) and Q ( x , s ) , instead of the original data and model distributions.
Revised: Second, the distributions that form the two classes (i.e. TRUE vs. FAKE) become P ( x , s ) and Q ( x , s ) , instead of the original data and model distributions.

Original: Based on the deﬁnitions in (2) and (9), we identify ( x d , s = 0) and ( x g , s = 1) as belonging to the TRUE class, whereas ( x d , s = 1) and ( x g , s = 0) to the FAKE class.
Revised: Based on the deﬁnitions in (2) and (7), we identify ( x d , s = 0) and ( x g , s = 1) as belonging to the TRUE class, whereas ( x d , s = 1) and ( x g , s = 0) to the FAKE class.

Original: Thus, the real samples are no longer always in the TRUE class, and the synthetic samples are no longer always in the FAKE class.
Revised: Thus, the real samples are no longer always in the TRUE class, and the synthetic samples are no longer always in the FAKE class.

Original: TRUE and FAKE now depend on the combination of x with s (see Figure 1).
Revised: TRUE and FAKE now depend on the combination of x with s (see Figure 1).

Original: Here, we introduce a simple trick to easily detect the class of a given pair.
Revised: Here, we introduce a simple trick to easily detect the class of a given pair.

Original: Namely, let the data and synthetic samples respectively convey one bit of information, with x d encoding bit zero and x g encoding bit one from now on.
Revised: Namely, let the data and synthetic samples respectively convey one bit of information, with x d encoding bit zero and x g encoding bit one from now on.

Original: Then, the checksum of the pair ( x , s ) determines the respective class, i.e. checksum zero for TRUE and one for FAKE.
Revised: Then, the checksum of the pair ( x , s ) determines the respective class, i.e. checksum zero for TRUE and one for FAKE.

Original: The checksum computation poses a more challenging task for the discriminator, as it needs to tell the real and synthetic samples apart and additionally understand the checksum rule.
Revised: The checksum computation poses a more challenging task for the discriminator, as it needs to tell the real and synthetic samples apart and additionally understand the checksum rule.

Original: Therefore, such augmentation is usable for preventing early maxing-out of the discriminator.
Revised: Therefore, such augmentation is usable for preventing early maxing-out of the discriminator.

Original: More importantly, it does not compromise the core role of the discriminator in GAN training: informing the generator about the difference between the data and generative model distribution.
Revised: More importantly, it does not compromise the core role of the discriminator in GAN training: informing the generator about the difference between the data and generative model distribution.

Original: This statement is conﬁrmed by the equivalence at (7).
Revised: This statement is conﬁrmed by the equivalence at (6).

Original: 4. P ROGRESSIVE M ULTI -L EVEL A UGMENTATION
Revised: 4. P ROGRESSIVE M ULTI -L EVEL A UGMENTATION

Original: Next, we extend the single level augmentation with one bit s to multi-level augmentation with an arbitrarily long random bit sequence s .
Revised: We further extend the single level augmentation with one bit s to multi-level augmentation with an arbitrarily long random bit sequence s .

Original: Note that the ﬁrst minimization problem in (6) and the last one in (7) share the same structure except for the different distribution pair.
Revised: Note that the two optimization problems on both sides of (6) rely on the JS divergence to quantify the difference of two distributions.

Original: Let us replace the data and model distributions, i.e. P d and P g , respectively with P ( x , s ) and Q ( x , s ) .
Revised: Let us replace the data and generative distributions, i.e. P d and P g , respectively with P ( x , s ) and Q ( x , s ) .

Original: This implies that the last minimization problem takes the starting position in our previous derivation.

Original: Following the same line of argumentation, we can systematically add a new bit.
Revised: Following the same line of argumentation, we can systematically add a new bit.

Original: Repeating this procedure L times will give us a bit sequence s with length L plus a series of equivalent problems with the same structure
Revised: Repeating this procedure L times will give us a bit sequence s with length L plus a series of equivalent problems with the same structure

Original: [Equation] 
Revised: [Equation] [Equation]

Original: To estimate the JS divergence in (11), the augmented discriminator as deﬁned in (10) now takes a bit sequence s instead of the single bit s , in addition to x .
Revised: To estimate the JS divergence in (9), the augmented discriminator as deﬁned in (8) now takes a bit sequence s instead of the single bit s , in addition to x .

Original: The combination of the sample x and s yields a multi-level augmentation.
Revised: The combination of the sample x and s yields a multi-level augmentation.

Original: Following the analysis of the single bit case, it is not difﬁcult to notice that the checksum mechanism remains.
Revised: Following the analysis of the single bit case, it is not difﬁcult to notice that the checksum mechanism remains.

Original: Namely, the discriminator needs to retrieve the one bit information carried by x and then perform a checksum together with the bit sequence s .
Revised: Namely, the discriminator needs to retrieve the one bit information carried by x and then perform a checksum together with the bit sequence s .

Original: The task difﬁculty increases as the length of s grows (see Figure 1).
Revised: The task difﬁculty increases as the length of s grows (see Figure 1).

Original: Therefore it makes sense to increase the augmentation level by adding more bits, whenever the discriminator becomes too powerful.
Revised: Therefore it makes sense to increase the augmentation level by adding more bits, whenever the discriminator becomes too powerful.

Original: More importantly, the consistency of the checksum mechanism across different augmentation levels permits progressive augmentation.
Revised: More importantly, the consistency of the checksum mechanism across different augmentation levels permits progressive augmentation.

Original: The same discriminator can be trained from a lower augmentation level and gradually take more bits into consideration (see Figure 2).
Revised: The same discriminator can be trained from a lower augmentation level and gradually take more bits into consideration (see Figure 2).

Original: 4. I MPLEMENTATION
Revised: 4. I MPLEMENTATION

Original: Network architecture.
Revised: Network architecture.

Original: In this work, we aim to maximally reuse existing neural network architectures tailored for GANs, such as DCGAN with spectral normalization (SN DCGAN)
Revised: In this work, we aim to maximally reuse existing neural network architectures tailored for GANs, such as DCGAN with spectral normalization (SN DCGAN)

Original: (Miyato et al., 2018) and InfoGAN (Chen et al., 2016).
Revised: (Miyato et al., 2018) and InfoGAN (Chen et al., 2016).

Original: According to the above-introduced approach (and Figure 2), the generator architecture can remain unchanged, while the discriminator network requires an alteration to incorporate the augmentation s .
Revised: According to the above-introduced approach (and Figure 2), the generator architecture can remain unchanged, while the discriminator network requires an alteration to incorporate the augmentation s .

Original: To this end, we only modify the input layer of the discriminator network, yielding minimal changes.
Revised: To this end, we only modify the input layer of the discriminator network, yielding minimal changes.

Original: First, the bit sequence s is preprocessed into a form compatible with x . Consider an image sample x with three coordinates, i.e. height, width and color channel (RGB).
Revised: First, the bit sequence s is preprocessed into a form compatible with x . Consider an image sample x with three coordinates, i.e. height, width and color channel (RGB).

Original: Each entry of s creates one extra augmentation channel, whereas the bit value is replicated to match the height and width of x .
Revised: Each entry of s creates one extra augmentation channel, whereas the bit value is replicated to match the height and width of x .

Original: It is worth noting that we let each bit take on values { 0 , 1 } as input to the network.
Revised: It is worth noting that we let each bit take on values { 0 , 1 } as input to the network.

Original: This choice of values is mainly due to the progressive augmentation during the course of training.
Revised: This choice of values is mainly due to the progressive augmentation during the course of training.

Original: When increasing the augmentation level, the additional bit 0 does not change the checksum and thus the output of
Revised: When increasing the augmentation level, the additional bit 0 does not change the checksum and thus the output of

Original: D .
Revised: D .

Original: On the contrary, the additional bit 1 ﬂips the even(odd) checksum to odd(even).
Revised: On the contrary, the additional bit 1 ﬂips the even(odd) checksum to odd(even).

Original: An effective change at the discriminator output is necessary to match the discrimination goal, thereby requiring a non-zero input in the ﬁrst place.
Revised: An effective change at the discriminator output is necessary to match the discrimination goal, thereby requiring a non-zero input in the ﬁrst place.

Original: Using { 0 , 1 } rather than other pairs of values, e.g. {− 1 , 1 } , helps the discriminator to timely catch the change when progression takes place.
Revised: Using { 0 , 1 } rather than other pairs of values, e.g. {− 1 , 1 } , helps the discriminator to timely catch the change when progression takes place.

Original: Second, we keep the input layer in the network to process x and copy its conﬁguration for processing the reformed s .
Revised: Second, we keep the input layer in the network to process x and copy its conﬁguration for processing the reformed s .

Original: Its kernel size, stride and padding type remain, but the input channel size is changed to L to process each entry of s .
Revised: Its kernel size, stride and padding type remain, but the input channel size is changed to L to process each entry of s .

Original: When a new augmentation level is reached, one extra input channel is instantiated to process the bit L + 1 .
Revised: When a new augmentation level is reached, one extra input channel is instantiated to process the bit L + 1 .

Original: All the following layers of the discriminator remain unchanged.
Revised: All the following layers of the discriminator remain unchanged.

Original: Minibatch discrimination.
Revised: Minibatch discrimination.

Original: The gradients for updating D are computed from the loss function given in (10), where the augmentation bit s is replaced by the bit sequence s of length L depending on the current augmentation level.
Revised: The gradients for updating D are computed from the loss function given in (8), where the augmentation bit s is replaced by the bit sequence s of length L depending on the current augmentation level.

Original: The two expectations are approximated by using minibatches.
Revised: The two expectations are approximated by using minibatches.

Original: Each minibatch is constructed with the same number of real data samples, synthetic samples and bit sequences.
Revised: Each minibatch is constructed with the same number of real data samples, synthetic samples and bit sequences.

Original: Each bit sequence is randomly sampled and associated with one real and one synthetic sample.
Revised: Each bit sequence is randomly sampled and associated with one real and one synthetic sample.

Original: By computing the checksums of the formed pairs, we can decide the correct class of each pair ( x , s ) in the minibatch and feed it into the discriminator to compute the cross-entropy loss.
Revised: By computing the checksums of the formed pairs, we can decide the correct class of each pair ( x , s ) in the minibatch and feed it into the discriminator to compute the cross-entropy loss.

Original: This way of generating ( x , s ) guarantees a balanced number of TRUE/FAKE samples.
Revised: This way of generating ( x , s ) guarantees a balanced number of TRUE/FAKE samples.

Original: Non-saturating loss (NS).
Revised: Non-saturating loss (NS).

Original: When employing non-saturating loss for G in the experiments, we follow the reformulation introduced by (Goodfellow et al., 2014).
Revised: When employing non-saturating loss for G in the experiments, we follow the reformulation introduced by (Goodfellow et al., 2014).

Original: Since the two expectation terms in (10) depend on G , this reformulation is applied for both of them, namely
Revised: Since the two expectation terms in (8) depend on G , this reformulation is applied for both of them, namely

Original: [Equation]
Revised: [Equation]

Original: Progression scheduling.
Revised: Progression scheduling.

Original: Bi ´ nkowski et al.
Revised: Bi´nkowski et al.

Original: (2018) introduced the kernel inception distance (KID) to quantify the quality of the synthetic samples and proposed to reduce the learning rate by tracking the reduction of KID over iterations.
Revised: (2018) introduced the kernel inception distance (KID) to quantify the quality of the synthetic samples and proposed to reduce the learning rate by tracking the reduction of KID over iterations.

Original: Here we use KID to decide if the performance of G at the current augmentation level saturates or even starts degrading (typically happens when D starts overﬁtting or becomes too powerful).
Revised: Here we use KID to decide if the performance of G at the current augmentation level saturates or even starts degrading (typically happens when D starts overﬁtting or becomes too powerful).

Original: Speciﬁcally, after t discriminator iterations 2 , we compute the
Revised: Speciﬁcally, after t discriminator iterations 2 , we compute the

Original: KID between synthetic samples and data samples drawn from the training set.
Revised: KID between synthetic samples and data samples drawn from the training set.

Original: If the current KID score is less than 5% of the average of the two previous ones attained at the same augmentation level, the augmentation is leveled up, i.e. L is increased by one.
Revised: If the current KID score is less than 5% of the average of the two previous ones attained at the same augmentation level, the augmentation is leveled up, i.e. L is increased by one.

Original: Once reaching a new augmentation level, we introduce the following mechanisms to assist the discriminator in quickly picking up the change in the input space.
Revised: Once reaching a new augmentation level, we introduce the following mechanisms to assist the discriminator in quickly picking up the change in the input space.

Original: First, the new augmentation bit is generated from a non-uniform distribution, i.e. P ( s = 1) =
Revised: First, the new augmentation bit is generated from a non-uniform distribution, i.e. P ( s = 1) =

Original: p and P ( s = 0)
Revised: p and P ( s = 0)

Original: = 1 − p with p < 0 .
Revised: = 1 − p with p < 0 .

Original: As mentioned before, critical changes on the discriminator side are required for bit 1 .
Revised: As mentioned before, critical changes on the discriminator side are required for bit 1 .

Original: For it to gradually comprehend the new bit, we on purpose create more 0 s than 1 s and gradually increase p up to 0 .
Revised: For it to gradually comprehend the new bit, we on purpose create more 0 s than 1 s and gradually increase p up to 0 .

Original: 5 (the uniform case) after a certain number of iterations.
Revised: 5 (the uniform case) after a certain number of iterations.

Original: A simple linear model is adopted
Revised: A simple linear model is adopted

Original: [Equation] 
Revised: [Equation] [Equation]

Original: (13) where t and t st are the current iteration index and the iteration index when the current augmentation level is started, respectively; and t r controls the rate of increase.
Revised: ) where t and t st are the current iteration index and the iteration index when the current augmentation level is started, respectively; and t r controls the rate of increase.

Original: It is important to note that p (cid:54) = 0 .
Revised: It is important to note that p (cid:54) = 0 .

Original: 5 does not cause unbalanced TRUE and FAKE classes in the constructed minibatches.
Revised: 5 does not cause unbalanced TRUE and FAKE classes in the constructed minibatches.

Original: It only introduces some bias in the generation of the new augmentation bit.
Revised: It only introduces some bias in the generation of the new augmentation bit.

Original: Finally, it is advisable to slow down the learning rate of G when a new augmentation level is reached.
Revised: Finally, it is advisable to slow down the learning rate of G when a new augmentation level is reached.

Original: When D is not properly adjusted to the new level, its feedback to G can be noisy.
Revised: When D is not properly adjusted to the new level, its feedback to G can be noisy.

Original: For instance, when using the Adam optimizer (Kingma & Ba, 2015), we reset the time step recorded by the G optimizer.
Revised: For instance, when using the Adam optimizer (Kingma & Ba, 2015), we reset the time step recorded by the G optimizer.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically evaluate the effectiveness of PA-GAN and compare its performance with other GAN-type algorithms across multiple datasets and different network architectures.

Original: [Figure]

Original: [Figure]

Original: Datasets.
Revised: Datasets.

Original: In our experiments we consider MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky, 2009) and CELEBA (Liu et al., 2015) datasets, with the training set sizes equal to 60 k, 60 k, 50 k and 162 k respectively. 
Revised: In our experiments we consider MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky, 2009) and CELEBA (Liu et al., 2015) datasets, with the training set sizes equal to 60 k, 60 k, 50 k and 162 k respectively. [Figure] [Figure]

Original: Network architectures.
Revised: Network architectures.

Original: We employ two well established deep convolutional GAN architectures,
Revised: We employ two well established deep convolutional GAN architectures,

Original: SN-DCGAN (Miyato et al., 2018) and InfoGAN (Chen et al., 2016)
Revised: SN-DCGAN (Miyato et al., 2018) and InfoGAN (Chen et al., 2016)

Original: (see Appendix A.1 for detailed conﬁgurations).
Revised: (see Appendix A.2 for detailed conﬁgurations).

Original: As they respectively employ spectral normalization (SN) and batch normalization (BN), our aim is to explore the compatibility of PA-GAN with these normalization techniques.
Revised: As they respectively employ spectral normalization (SN) and batch normalization (BN), our aim is to explore the compatibility of PA-GAN with these normalization techniques.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We use Fr ´ echet inception distance (FID) (Husz ´ ar, 2015) as the primary performance evaluation metric.
Revised: We use Fr´echet inception distance (FID) (Husz´ar, 2015) as the primary performance evaluation metric.

Original: Additionally, we also report inception score (IS) (Theis et al., 2016) and kernel inception distance (KID) (Bi ´ nkowski et al., 2018) in Appendix A.2.
Revised: Additionally, we also report inception score (IS) (Theis et al., 2016) and kernel inception distance (KID) (Bi´nkowski et al., 2018) in Appendix A.4.

Original: For quantifying the quality of synthetic samples, all measures are computed based on 10 k test data and 10 k synthetic samples, following the evaluation framework of Lu ˘ ci ´ c et al.
Revised: For quantifying the quality of synthetic samples, all measures are computed based on 10 k test data and 10 k synthetic samples, following the evaluation framework of Lu ˘ ci ´ c et al.

Original: (2018) and Kurach et al.
Revised: (2018) and Kurach et al.

Original: Training details.
Revised: Training details.

Original: We use the minibatch size of 64 and the Adam optimizer (Kingma & Ba, 2015) with the default setting: β 1 = 0 .
Revised: We use the minibatch size of 64 and the Adam optimizer (Kingma & Ba, 2015) with the default setting: β 1 = 0 .

Original: 5 , β 2 = 0 .
Revised: 5 , β 2 = 0 .

Original: 999 and learning rate 2 × 10 − 4 for both the generator and discriminator, which have an equal update rate.
Revised: 999 and learning rate 2 × 10 − 4 for both the generator and discriminator, which have an equal update rate.

Original: The dimension of z is set to 64 and 128 respectively for InfoGAN and SN-DCGAN.
Revised: The dimension of z is set to 64 and 128 respectively for InfoGAN and SN-DCGAN.

Original: The prior distribution P z is uniform.
Revised: The prior distribution P z is uniform.

Original: For scheduling progressive augmentation KID is evaluated every 10 4 discriminator iterations, using 10 k generated samples and 10 k samples randomly drawn from the training set, and t r in (13) is set to 5 × 10 3 .
Revised: For scheduling progressive augmentation KID is evaluated every 10 4 discriminator iterations, using 10 k generated samples and 10 k samples randomly drawn from the training set, and t r in (11) is set to 5 × 10 3 .

Original: 5. CIFAR10 WITH SN-DCGAN
Revised: 5. CIFAR10 WITH SN-DCGAN

Original: In this experiment, we evaluate the progressive augmentation (PA) with NS GAN (GAN with the non-saturating loss) using the SN-DCGAN architecture on CIFAR10.
Revised: In this experiment, we evaluate the progressive augmentation (PA) with NS GAN (GAN with the non-saturating loss) using the SN-DCGAN architecture on CIFAR10.

Original: We analyze the beneﬁts of applying PA on top of NS GAN , experiment with starting PA from different augmentation levels and investigate the complementarity of using both PA and the gradient penalty regularization (GP) (Gulrajani et al., 2017).
Revised: We analyze the beneﬁts of applying PA on top of NS GAN , experiment with starting PA from different augmentation levels and investigate the complementarity of using both PA and the gradient penalty regularization (GP) (Gulrajani et al., 2017).

Original: For fair comparison, we follow the experimental setup of (Kurach et al., 2018).
Revised: For fair comparison, we follow the experimental setup of (Kurach et al., 2018).

Original: NS GAN with PA.
Revised: NS GAN with PA.

Original: Figure 3 and Table 1 compare NS GAN results with and without applying PA.
Revised: Figure 3 and Table 1 compare NS GAN results with and without applying PA.

Original: We are able to closely reproduce the NS GAN results reported in (Kurach et al., 2018, Table 6), after 200 k iterations we obtain the median FID value of 26 . 3 vs. original 26 . 7 .
Revised: We are able to closely reproduce the NS GAN results reported in (Kurach et al., 2018, Table 6), after 200 k iterations we obtain the median FID value of 26 . 3 vs. original 26 . 7 .

Original: By applying PA on top of NS GAN and starting from the augmentation level 0 ( PA L = 0 - NS GAN ), we achieve superior performance, with the median FID of 25 . 7 vs. 26 .
Revised: By applying PA on top of NS GAN and starting from the augmentation level 0 ( PA L = 0 - NS GAN ), we achieve superior performance, with the median FID of 25 . 7 vs. 26 .

Original: 3 of NS GAN .
Revised: 3 of NS GAN .

Original: Training for extra 100 k iterations boosts the performance of PA L = 0 - NS GAN ( 25 . 7 vs. 24 . 6 ).
Revised: Training for extra 100 k iterations boosts the performance of PA L = 0 - NS GAN ( 25 . 7 vs. 24 . 6 ).

Original: Starting PA from the level 2 ( PA L = 2 - NS GAN ) further improves the median FID ( 23 . 8 vs. 25 . 7 ); as CIFAR10 contains diverse images a start from a higher augmentation level is recommended.
Revised: Starting PA from the level 2 ( PA L = 2 - NS GAN ) further improves the median FID ( 23 . 8 vs. 25 . 7 ); as CIFAR10 contains diverse images a start from a higher augmentation level is recommended.

Original: It is worth noting that at early iterations (up to 50 k)
Revised: It is worth noting that at early iterations (up to 50 k)

Original: PA L = 2 - NS GAN has worse performance than NS GAN .
Revised: PA L = 2 - NS GAN has worse performance than NS GAN .

Original: Starting at the augmentation level 2 imposes a more challenging task for the discriminator, thereby showing slower improvement at initial iterations but being beneﬁcial in the longer term.
Revised: Starting at the augmentation level 2 imposes a more challenging task for the discriminator, thereby showing slower improvement at initial iterations but being beneﬁcial in the longer term.

Original: The FID value of PA L = 2 - NS GAN saturates at a slower pace than NS GAN leading to better overall results.
Revised: The FID value of PA L = 2 - NS GAN saturates at a slower pace than NS GAN leading to better overall results.

Original: NS GAN with PA and GP.
Revised: NS GAN with PA and GP.

Original: For GP the interpolates are created analogously to (Gulrajani et al., 2017), i.e. [˜ x , ˜ s ]
Revised: For GP the interpolates are created analogously to (Gulrajani et al., 2017), i.e. [˜ x , ˜ s ]

Original: = α [ x , s ] TRUE
Revised: = α [ x , s ] TRUE

Original: + (1 − α )
Revised: + (1 − α )

Original: [ x , s ] FAKE with α ∼ U(0 , 1) .
Revised: [ x , s ] FAKE with α ∼ U(0 , 1) .

Original: Note that interpolation takes place in the augmented space [ x , s ] , yielding ˜ s ∈ [0 , 1] L .[Table]
Revised: Note that interpolation takes place in the augmented space [ x , s ] , yielding ˜ s ∈ [0 , 1] L .

Original: for the PA results marked with (*).

Original: For (*) the training time is not constrained by the previously speciﬁed number of epochs.

Original: See A.3 for details. In (Kurach et al., 2018), when GP is applied on top of NS GAN a marginal improvement is observed ( 26 . 2 vs. 26 . 7 ).
Revised: In (Kurach et al., 2018), when GP is applied on top of NS GAN a marginal improvement is observed ( 26 . 2 vs. 26 . 7 ).

Original: However, employing PA results in a more noticeable gain for GP ( 22 . 5 vs. 23 . 8 ).
Revised: However, employing PA results in a more noticeable gain for GP ( 22 . 5 vs. 23 . 8 ).

Original: Furthermore, GP helps to accelerate the learning speed of PA L = 2 - NS GAN at the initial iterations. 
Revised: Furthermore, GP helps to accelerate the learning speed of PA L = 2 - NS GAN at the initial iterations. [Table]

Revised: datasets (from left to right), the results are attained after 20 , 20 , 100 and 40 epochs, respectively, except for the PA results marked with (*).

Revised: For (*) the training time is not constrained by the previously speciﬁed number of epochs, see A.5 for details.

Original: As indicated in Figure 1, in the augmented space (e.g., L ∈ { 1 , 2 } ), more paths are created for the generative model to approach the data distribution.
Revised: As indicated in Figure 1, in the augmented space (e.g., L ∈ { 1 , 2 } ), more paths are created for the generative model to approach the data distribution.

Original: GP can help to smoothen the decision boundaries along these paths.
Revised: GP can help to smoothen the decision boundaries along these paths.

Original: As a result, GP and PA jointly improve the performance of NS GAN , being mutually beneﬁcial ( 22 . 5 vs. 26 . 3 ).
Revised: As a result, GP and PA jointly improve the performance of NS GAN , being mutually beneﬁcial ( 22 . 5 vs. 26 . 3 ).

Revised: In A.3, we additionally report results on CELEBA-HQ (Karras et al., 2018).

Revised: The enlarged performance gain ( 18 . 1 vs. 28 . 5 ) reveals a great potential of employing PA for high resolution datasets.

Original: 5. C OMPARISONS AMONG D ATASETS AND GAN- TYPE A LGORITHMS
Revised: 5. C OMPARISONS AMONG D ATASETS AND GAN- TYPE A LGORITHMS

Original: Lu ˘ ci ´ c et al.
Revised: Lu ˘ ci ´ c et al.

Original: compared various GAN-type algorithms under the InfoGAN architecture and reported their FID scores after a wide range of hyper-parameter searching.
Revised: compared various GAN-type algorithms under the InfoGAN architecture and reported their FID scores after a wide range of hyper-parameter searching.

Original: Following their experimental setup, we select hyper-parameters within the candidate set considered by Lu ˘ ci ´ c et al.
Revised: Following their experimental setup, we select hyper-parameters within the candidate set considered by Lu ˘ ci ´ c et al.

Original: (see Appendix A.3 for details) and evaluate PA with NS GAN as well as Wasserstein GAN with GP ( WGAN - GP ).
Revised: (see Appendix A.5 for details) and evaluate PA with NS GAN as well as Wasserstein GAN with GP ( WGAN - GP ).

Original: The maximal number of augmentation levels achieved by PA corresponds to seven.
Revised: The maximal number of augmentation levels achieved by PA corresponds to seven.

Original: The results are reported in Table 2.
Revised: The results are reported in Table 2.

Original: Overall, we observe signiﬁcant gains of employing PA with NS GAN and WGAN - GP across different datasets. 
Revised: Overall, we observe signiﬁcant gains of employing PA with NS GAN and WGAN - GP across different datasets. Note that the numbers provided by Lu ˘ ci ´ c et al.

Revised: (2018) are outcomes of removing outliers up to 20% among the ﬁfty independent runs.

Revised: Since the outlier removing ratio is not speciﬁed for individual cases in (Lu˘ci´c et al., 2018), in Table 2 we report the PA results without removing outliers, see A.7 for further discussion.

Original: As shown in Table 2, the NS GAN performance is quite stable across the datasets.
Revised: As shown in Table 2, the NS GAN performance is quite stable across the datasets.

Original: By applying PA with
Revised: By applying PA with

Original: NS GAN , we achieve a pronounced improvement, particularly when the dataset (and hence the image generation task) becomes more complicated.
Revised: NS GAN , we achieve a pronounced improvement, particularly when the dataset (and hence the image generation task) becomes more complicated.

Original: Note that the PA - NS GAN performance on MNIST is worse than NS GAN .
Revised: Note that the PA - NS GAN performance on MNIST is worse than NS GAN .

Original: This is mainly because NS GAN already performs very well on such a simple dataset and PA requires additional iterations to reach similar results or further improve them as in
Revised: This is mainly because NS GAN already performs very well on such a simple dataset and PA requires additional iterations to reach similar results or further improve them as in

Original: PA - NS GAN (*).
Revised: PA - NS GAN (*).

Original: In contrast to NS GAN , WGAN - GP is sensitive to the dataset as shown in Tableand reported by Mescheder et al.
Revised: In contrast to NS GAN , WGAN - GP is sensitive to the dataset as shown in Tableand reported by Mescheder et al.

Original: However, applying PA helps to stabilize the WGAN - GP performance across different benchmarks.
Revised: However, applying PA helps to stabilize the WGAN - GP performance across different benchmarks.

Original: Similarly to NS GAN , longer training leads to better results for PA with WGAN - GP .
Revised: Similarly to NS GAN , longer training leads to better results for PA with WGAN - GP .

Original: The results in Table 2 indicate that PA generalizes well across different distance measures and is not limited to the JS divergence. 
Revised: The results in Table 2 indicate that PA generalizes well across different distance measures and is not limited to the JS divergence. Besides the experiments in Sec. 5.1 and 5.2, we provide a careful investigation of PA itself in Appendix.

Revised: Ablation studies on the effect of the progression scheduling and the linear model 11 are given in A.6, followed by the analysis of PA training stability in A.7.

Revised: The increase of the task difﬁculty of the discriminator with PA is examined in A.8.

Revised: In addition to GP, we also compare PA with the dropout regularization in A.9.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we have proposed a novel method - progressive augmentation (PA) - to improve the stability of GAN training, and showed a way to integrate it into existing GAN architectures with minimal changes.
Revised: In this work we have proposed a novel method - progressive augmentation (PA) - to improve the stability of GAN training, and showed a way to integrate it into existing GAN architectures with minimal changes.

Original: Different to standard data augmentation our approach does not modify the training samples, instead it progressively increases the dimension of the discriminator input space by augmenting it with auxiliary random variables.
Revised: Different to standard data augmentation our approach does not modify the training samples, instead it progressively increases the dimension of the discriminator input space by augmenting it with auxiliary random variables.

Original: Higher sample space dimensionality helps to entangle the discriminator and thus to avoid its early performance saturation.
Revised: Higher sample space dimensionality helps to entangle the discriminator and thus to avoid its early performance saturation.

Original: Moreover, in the augmented space the generator can explore more paths to approach the data distribution, improving variation of the generated samples.
Revised: Moreover, in the augmented space the generator can explore more paths to approach the data distribution, improving variation of the generated samples.

Original: We experimentally have shown pronounced performance improvements of employing the proposed PA with state-of-the-art GAN methods across multiple benchmarks.
Revised: We experimentally have shown pronounced performance improvements of employing the proposed PA with state-of-the-art GAN methods across multiple benchmarks.

Original: We demonstrated that PA generalizes well across different network architectures and loss functions and is complementary to other regularization techniques.
Revised: We demonstrated that PA generalizes well across different network architectures and loss functions and is complementary to other regularization techniques.

Original: For future work, we ﬁnd a joint optimization of PA with neural architectures an interesting direction, for instance, combining it with progressive growing of GANs (Karras et al., 2018).
Revised: For future work, we ﬁnd a joint optimization of PA with neural architectures an interesting direction, for instance, combining it with progressive growing of GANs (Karras et al., 2018).

Original: Apart from generative modeling, our approach can also be exploited for semi-supervised learning, generative latent modeling and transfer learning.
Revised: Apart from generative modeling, our approach can also be exploited for semi-supervised learning, generative latent modeling and transfer learning.

Original: A  PPENDIX A. N EURAL N ETWORK A RCHITECTURES
Revised: A A PPENDIX A. P ROOF OF THE E QUIVALENCE IN (6)

Revised: In Sec. 3.2, we have provided the connection of the JS divergence to the mutual information between the auxiliary random bit s and the sample x following the joint distribution P ( x , s ) as given in (2)

Revised: [Equation]

Revised: Interchanging the position of P d and P g in constructing P ( x , s ) , we obtain the joint distribution Q ( x ; s ) as given in (7).

Revised: With respect to Q ( x ; s ) , we further compute and denote the mutual information between x and s as ˜I( x ; s ) .

Revised: Analogous to 4 in Sec.

Revised: 3.2, we can show

Revised: [Equation]

Revised: [Equation]

Revised: The equality (12) and (13) jointly yield

Revised: [Equation]

Revised: 14) followed by rewriting mutual information as KL divergence

Revised: [Equation]

Revised: [Equation]

Revised: It is noted that the marginals of P ( x , s ) and Q ( x , s ) with respect to x are identical and equal to P m ( x ) as given in (3), whereas P s ( s ) is their marginal with respect to s .

Revised: With the identiﬁcation of

Revised: [Equation]

Revised: [Equation]

Revised: we reach to

Revised: [Equation]

Revised: The right-hand side term of the above equality is simply the JS divergence between P ( x , s ) and Q ( x , s ) .

Revised: Since the two JS divergences are completely identical, we can use them interchangeably as the objective function while optimizing P g .

Revised: With that we conclude the equivalence proof for the two optimization problems in (6).

Revised: A. N EURAL N ETWORK A RCHITECTURES

Original: Following Kurach et al.
Revised: Following Kurach et al.

Original: (2018) for SN-DCGAN we employed the same architecture as in (Miyato et al., 2018), which we present in Table A1.
Revised: (2018) for SN-DCGAN we employed the same architecture as in (Miyato et al., 2018), which we present in Table A1.

Original: For the InfoGAN architecture we followed Lu˘ci´c et al.
Revised: For the InfoGAN architecture we followed Lu ˘ ci ´ c et al.

Original: and used the network structure of (Chen et al., 2016), which is described in Table A2.
Revised: and used the network structure of (Chen et al., 2016), which is described in Table A2.

Original: For both experiments with SN-DCGAN and InfoGAN we exploited the implementation provided in https://github.com/google/compare_gan . 
Revised: For both experiments with SN-DCGAN and InfoGAN we exploited the implementation provided in https://github.com/google/compare_gan . [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: Linear −

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: 4 , 2] h 4 × w 4 ×

Original: {+DeConv, BN, ReLU [4 , 4 , 2] h 
+}
Revised: {+DeConv, BN, ReLU [4 , 4 , 2] h 
+}

Revised: × w 2 ×

Revised: DeConv, BN, ReLU [4 , 4 , 2] h × w ×

Revised: DeConv, Tanh [3 , 3 , 1] h × w ×

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Figure] [Table]

Revised: A. CELEBA-HQ WITH SN-DCGAN

Revised: Figure A1 and Table A3 report additional results (FID scores) on CELEBA-HQ ( 128 × 128 ) (Karras et al., 2018) using the SN-DCGAN network architecture, following the same experimental setup as in Sec. 5.1.

Revised: The results on CELEBA-HQ ( 128 × 128 ) are consistent with our observations in Sec. 5.1.

Revised: By applying PA on top of NS GAN - GP and starting from the augmentation level( PA L = 2 - NS GAN - GP ), we achieve superior performance, with the median FID of 18 . 8 vs. 27 .

Revised: 3 of

Revised: NS GAN - GP .

Revised: Without PA, NS GAN and NS GAN - GP have almost no FID reduction (or very minor) in the last 100 k iterations, whereas PA enables further improvement.

Revised: The achieved best FID value by

Revised: PA L = 2 - NS GAN - GP is even 36% smaller than the best FID score achieved by NS GAN with ResNet reported by (Kurach et al., 2018)

Revised: ( 18 . 1 vs. 28 . 5 ).

Original: A. CIFAR10 WITH SN-DCGAN
Revised: A. CIFAR10 WITH SN-DCGAN

Original: FigureA1 and A2  together with Table A3 and Table A4 report the inception scores and KID values that are attained following the same experimental setup as in Sec.
Revised: Figure A2 and A3 together with Table A4 and Table A5 report the inception scores and KID values that are attained following the same experimental setup as in Sec.

Original: 5.1 on CIFAR10.
Revised: 5.1 on CIFAR10.

Original: The results on both measures are consistent with our observation drawn from the FID values.
Revised: The results on both measures are consistent with our observation drawn from the FID values.

Original: In the recent work (Arbel et al., 2018), the reported best KID value for CIFAR10 with SN-DCGAN is 0 . 015 , whereas PA L = 2 - NS GAN - GP reduces it to 0 . 
Revised: In the recent work (Arbel et al., 2018), the reported best KID value for CIFAR10 with SN-DCGAN is 0 . 015 , whereas PA L = 2 - NS GAN - GP reduces it to 0 . [Figure] [Figure] [Figure]

Original: Next we present some further remarks on the implementation side.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  First, to explicitly investigate the combination of GP and PA, we add the GP as an extra regularizer to the discriminator loss function only when the augmentation takes place, and GP is computed with respect to the augmented sample after interpolation, i.e. [˜ x , ˜ s ] .
Revised: Next we present some further remarks on the implementation side. First, to explicitly investigate the combination of GP and PA, we add the GP as an extra regularizer to the discriminator loss function only when the augmentation takes place [Figure] and GP is computed with respect to the augmented sample after interpolation, i.e. [˜ x , ˜ s ] .

Original: Figures A4 - A6 together with Tables A5 - A7 report results evaluated with the FID, IS and KID metrics.
Revised: Figures A4 - A6 together with Tables A6 - A8 report results evaluated with the FID, IS and KID metrics.

Original: With the use of GP, the starting level of the augmentation has negligible inﬂuence on the performance after a sufﬁcient number of iterations.
Revised: With the use of GP, the starting level of the augmentation has negligible inﬂuence on the performance after a sufﬁcient number of iterations.

Original: Second, the chosen hyper-parameters are not optimized for PA.
Revised: Second, the chosen hyper-parameters are not optimized for PA.

Original: In fact, they are dedicatedly chosen by (Kurach et al., 2018) for NS GAN , i.e. β 1 = 0 .
Revised: In fact, they are dedicatedly chosen by (Kurach et al., 2018) for NS GAN , i.e. β 1 = 0 .

Original: 5 , β 2 = 0 .
Revised: 5 , β 2 = 0 .

Original: 999 , λ = 1 (weighting factor for GP) and the learning rate 2 × 10 − 4 .
Revised: 999 , λ = 1 (weighting factor for GP) and the learning rate 2 × 10 − 4 .

Original: We adopt them for PA for the purpose of fair comparison.
Revised: We adopt them for PA for the purpose of fair comparison.

Original: Therefore, further optimization on the hyper-parameters for PA may potentially yield better results than the reported ones.
Revised: Therefore, further optimization on the hyper-parameters for PA may potentially yield better results than the reported ones.

Original: Third, in our implementation all of progressively added augmentation channels take on the values { 0 , 1 } to ease the progression, as described in Sec. 4.3.For those augmentation channels that are [Figure] [Figure]
Revised: Third, in our implementation all of progressively added augmentation channels take on the values { 0 , 1 } to ease the progression, as described in Sec. 4.3.

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] [Figure] present from the start of training and if the pixel values of the image are normalized to [ − 1 , 1] , we accordingly experimented with the values {± 1 } .
Revised: For those augmentation channels that are present from the start of training and if the pixel values of the image are normalized to [ − 1 , 1] , we accordingly experimented with the values {± 1 } .

Original: As a result, in the current experiment (i [Figure] e., CIFAR with SN-DCGAN), we used {± 1 } and switch to { 0 , 1 } for subsequent InfoGAN experiment.
Revised: As a result, in the current experiment (i.e., CIFAR with SN-DCGAN), we used {± 1 } and switch to { 0 , 1 } for subsequent InfoGAN experiment.

Original: In general, it is not a critical choice to the performance from our observation.
Revised: In general, it is not a critical choice to the performance from our observation.

Original: At last, a set of synthetic images generated by PA L = 2 - NS GAN - GP with the FID value of 22 . 5 on
Revised: At last, a set of synthetic images generated by PA L = 2 - NS GAN - GP with the FID value of 22 . 5 on

Original: CIFAR10 is shown in Figure A3.
Revised: CIFAR10 is shown in Figure A7.

Original: A. C OMPARISONS AMONG D ATASETS AND GAN- TYPE A LGORITHMS
Revised: A. C OMPARISONS AMONG D ATASETS AND GAN- TYPE A LGORITHMS

Original: In Sec.
Revised: In Sec.

Original: 5.2, we apply PA with NS GAN and WGAN - GP and report the achieved FID values in Table 2.
Revised: 5.2, we apply PA with NS GAN and WGAN - GP and report the achieved FID values in Table 2.

Original: For PA - NS GAN and PA - NS GAN (*) across all datasets we start augmentation with level two, i.e.
Revised: For PA - NS GAN and PA - NS GAN (*) across all datasets we start augmentation with level two, i.e.

Original: L = 2 .
Revised: L = 2 .

Original: The other adopted hyper-parameters for PA - NS GAN are listed in Table A8.
Revised: The other adopted hyper-parameters for PA - NS GAN are listed in Table A9.

Original: It is important to note that they are selected considering the limit on the training epochs speciﬁed by Lu˘ci´c et al.
Revised: It is important to note that they are selected considering the limit on the training epochs speciﬁed by Lu˘ci´c et al.

Original: Allowing more iterations, the performance can be further improved.
Revised: Allowing more iterations, the performance can be further improved.

Original: For instance,
Revised: For instance,

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Original: PA - NS GAN (*) in Table 2 shows the achieved performance if we run 40 k iterations for MNIST and Fashion-MNIST, and run 140 k iterations for CIFAR10 and CELEBA.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: The gains can be further enlarged if the hyper-parameters are adjusted towards the longer training time.
Revised: [Figure] [Table] [Figure] [Table]

Original: For instance, by reducing the learning rate from 10 − 3 to 2 × 10 − 4 and performing 140 k iterations for MNIST, we can achieve the FID of 4 . 5 ± 0 .
Revised: [Figure]

Original: [Table] [Figure] 
Revised: [Figure] [Table]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: PA - NS GAN (*) in Table 2 shows the achieved performance if we run 40 k iterations for MNIST and Fashion-MNIST, and run 140 k iterations for CIFAR10 and CELEBA. The gains can be further enlarged if the hyper-parameters are adjusted towards the longer training time.

Original: [Table]
Revised: For instance, by reducing the learning rate from 10 − 3 to 2 × 10 − 4 and performing 140 k iterations for MNIST, we can achieve the FID of 4 . 5 ± 0 .

Original: Proceeding to the case PA - WGAN - GP , the hyper-parameters are listed in Table A9, the batch normalization for the discriminator is disabled.
Revised: Proceeding to the case PA - WGAN - GP , the hyper-parameters are listed in Table A10, the batch normalization for the discriminator is disabled.

Original: For both MNIST and Fashion-MNIST, the augmentation level starts from one to ensure that it is in place within the 20 epoch training time.
Revised: For both MNIST and Fashion-MNIST, the augmentation level starts from one to ensure that it is in place within the 20 epoch training time.

Original: For CELEBA, we start from the augmentation level zero and will reach the level one augmentation within thetraining epochs.
Revised: For CELEBA, we start from the augmentation level zero and will reach the level one augmentation within thetraining epochs.

Original: Analogous to the previous case, better performance is achievable by PA with the number of iterations increased beyond the original limit, i.e. 40 k for MNIST and Fashion-MNIST and 140 k for CELEBA.
Revised: Analogous to the previous case, better performance is achievable by PA with the number of iterations increased beyond the original limit, i.e. 40 k for MNIST and Fashion-MNIST and 140 k for CELEBA.

Original: Finally, we note that to stabilize progression in WGAN - GP the weighting factor λ for GP requires a careful adjustment.
Revised: Finally, we note that to stabilize progression in WGAN - GP the weighting factor λ for GP requires a careful adjustment.

Original: In this experiment, we relax λ following the idea behind (13), namely by linearly increasing λ from zero to its original value within t r iterations (in this case t r = 5 × 10 3 ).
Revised: In this experiment, we relax λ following the idea behind (11), namely by linearly increasing λ from zero to its original value within t r iterations (in this case t r = 5 × 10 3 ).

Original: Interestingly, we did not ﬁnd such adaptation necessary when using GP with NS GAN .
Revised: Interestingly [Table] we did not ﬁnd such adaptation necessary when using GP with NS GAN .

Original: One possible reason is that GP is critical to WGAN due to the Lipschitz constraint, but optional to NS GAN .
Revised: One possible reason is that GP is critical to WGAN due to the Lipschitz constraint, but optional to NS GAN .

Original: Augmentation changes the input space of the discriminator, and the number of terms involved in the GP also increases with the number of augmentation channels.
Revised: Augmentation changes the input space of the discriminator, and the number of terms involved in the GP also increases with the number of augmentation channels.

Original: Further investigation on the weighting factor adjustment to fully exploit the beneﬁt of combining PA with WGAN - GP is a part of our future work.
Revised: Further investigation on the weighting factor adjustment to fully exploit the beneﬁt of combining PA with WGAN - GP is a part of our future work.

Original: The results reported in this work have conﬁrmed that PA is not limited to the JS divergence. 
Revised: The results reported in this work have conﬁrmed that PA is not limited to the JS divergence. [Table] [Table] A. A BLATION S TUDIES

Original: [Figure]
Revised: Effect of the Progression Scheduling. Here we experiment with InfoGAN on CELEBA, as one of the most challenging cases considered in Table 2, to study the effect of the progression scheduling described in Sec. 4.2 and 4.3. Originally, Lu˘ci´c et al.

Revised: (2018) suggested 40 epoches (approximately 100 k iterations) for the NS GAN training.

Revised: As we can observe from Figure A8, further training can only degrade the performance.

Revised: One important reason for such behaviour is that the discriminator architecture design of InfoGAN (see Table A2) might be suboptimal for CELEBA (e.g., a fully

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: connected rather than a convolutional layer at the end plus batch normalization conducting minibatch discrimination).

Revised: By applying the proposed augmentation strategy, even with one augmentation level and no progression, we can achieve much better FID values at much earlier iterations.

Revised: Moreover, with the progression scheduling of augmentation (from level 2 to 7 ) we can dramatically slow down the degradation process for this challenging case.

Revised: We additionally observe that the standard deviation of FID values across 50 runs is much smaller with PA than without it and the gain of the PA remains consistent, independent of the number of outliers removed, see Figure A9.

Revised: This showcases the positive effect of the proposed PA on the training stability of GANs.

Revised: Nevertheless, it remains difﬁcult to fully counteract the degradation process caused by the suboptimal network architecture design for the task at hand.

Revised: Role of the Linear Model (11) in PA.

Revised: Here we analyse the role of the linear model 11 in progression scheduling described in Sec. 4.3.

Revised: Figure A10 depicts D Loss and G Loss over training iterations on CIFAR10 using SN-DCGAN.

Revised: Performing PA with a newly augmented bit may lead to the confusion of the discriminator, as EVEN inputs can immediately become ODD ones and vice versa.

Revised: Empirically, we observe that the loss of the discriminator (i.e., D Loss ) may stuck at values close to 1 .

Revised: 38 for a considerable number of iterations, because the discriminator is confused about the abrupt change.

Revised: This is particularly true for regularized networks.

Revised: Whenever such situation takes place, it is harmful for the generator as well.

Revised: A confused discriminator can no longer guide it towards the data distribution.

Revised: Figure A10 depicts the above mentioned confusions after a new augmentation level is introduced.

Revised: In this case, there are two plateaus of D Loss for brown curves which do not employ the linear model 11 in PA, occurring between 70 k and 120 k iterations.

Revised: Employing the linear model 11 in PA helps to avoid such long lasting plateaus.

Revised: The blue curves (with 11 in PA) exhibit smooth transitions when

Original: [Figure] 
Revised: [Figure] [Figure] NS GAN NS GAN

Revised: GP the augmentation level increases.

Revised: Overall, the linear model 11 in PA helps to speed up the learning process.

Revised: The main motivation of having it is to take precautions against potential ill adaptation to new augmentation levels.

Revised: A. E FFECT OF PA ON THE T RAINING S TABILITY

Revised: Figure A11 compares D Loss and G Loss of NS GAN and NS GAN - GP with and without PA during the training on CIFAR10 using the SN-DCGAN architecture.

Revised: Without PA the training becomes unstable over time independent of using GP (red curves), while employing PA helps to maintain a healthy competition between the discriminator and the generator.

Revised: Observing the behaviour of D Loss and

Revised: G Loss , we conclude that PA can effectively prevent the discriminator from overﬁtting, alleviating the vanishing gradient issue and thus enabling continuous learning of the generator.

Revised: In Table A11 we present mean and standard deviation values of FID across 50 independent runs on four datasets.

Revised: We follow the work of (Lu˘ci´c et al., 2018) reporting the numbers while excluding signiﬁcant outlier runs.

Revised: Excluding outlier runs mostly inﬂuences the standard deviation of the FID values.

Revised: Table A11 shows that in the same setting as (Lu˘ci´c et al., 2018) PA not only improves the

Revised: FID values, but also reduces their standard deviations across multiple runs.

Revised: This highlights that the training becomes more stable with PA.

Revised: The improvement of using PA is consistent across four datasets, although the value of the mean FID heavily ﬂuctuates dependent on the dataset.

Revised: A. A NALYSIS OF THE T ASK D IFFICULTY OF THE D ISCRIMINATOR WITH PA

Revised: With PA we have casted the discrimination task into a checksum operation, involving two steps.

Revised: The data and synthetic samples are combined with the augmentation bit sequence, resulting in data and

Revised: [Table]

Revised: synthetic samples contained in both true and fake classes.

Revised: So, ﬁrst the discriminator needs to recover the original sample and the augmentation bit sequence, and to decide if the original sample is the data or synthetic sample (this task is essentially identical to the original discrimination task).

Revised: Second, the discriminator needs to learn that the recovered sample and the augmentation bit sequence jointly follow the checksum principle.

Revised: The difﬁculties of these two problems strongly depend on the way of feeding the augmentation bit sequence into the discriminator network.

Revised: Providing the augmentation bit sequence to the input layer or lower layers generally makes the checksum operation more difﬁcult than providing it to the upper layers.

Revised: As the former has more difﬁculties decoupling the two tasks and trying to solve them jointly, while the latter can solve them sequentially.

Revised: One naive design is to combine the bit sequence with the original output of the discriminator.

Revised: In this case the checksum operation becomes nearly trivial and does not have a major inﬂuence on the training.

Revised: The other way is to concatenate the augmentation bit sequence directly with the data and synthetic samples, ensuring that the task remains non-trivial for the discriminator.

Revised: Toy Example: Binary Classiﬁcation with PA.

Revised: For an illustration purpose, here we design a simple experiment.

Revised: We employ discriminator for the classiﬁcation task, omitting the use of the generator.

Revised: Namely, two classes of CIFAR10 (cats and dogs) are extracted, conveying the bit 0 and 1 .

Revised: Together with the augmentation bit sequences, they are provided as input to the discriminator, which task is to perform binary classiﬁcation.

Revised: Using the SN-DCGAN architecture, we directly concatenate the augmentation bit sequences with the cat and dog images.

Revised: Figure A12 depicts the variation of the discriminator loss ( D Loss ) over the periodic progressive augmentation (every 2k iterations).

Revised: It is evident that whenever a new augmentation level is reached an abrupt increase of D Loss is observed, showing that the task of the discriminator becomes harder with each level of PA.

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: Employing Residual/Skip Connections with PA.

Revised: With the presence of skip connections the augmentation bit sequence that is concatenated with the inputs will also be directly wired to the

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Table]

Revised: network output.

Revised: In the next two examples, we examine the task difﬁculty under such situation.

Original: {+First, we switch to using ResNet CIFAR
+}
Revised: {+First, we switch to using ResNet CIFAR
+}

Revised: (Kurach et al., 2018) for the toy example of binary classiﬁcation of cats and dogs, while concatenating the augmentation bits with the inputs of the last residual block.

Revised: By doing so, the skip connection will directly deliver this information to the network ﬁnal layer.

Revised: As we can see from Figure A13, the task remains difﬁcult and PA persistently increases the task difﬁculty.

Revised: Next, we switch to the GAN setting and repeat the experiment of SN-DCGAN CIFAR10 in Sec. 5.1.

Revised: Instead of only concatenating the augmentation bits with the input images, we also insert a skip connection, additionally concatenating them with the input to the last convolutional layer.

Revised: 7 Depicting the D Loss and G Loss during the course of training, Figure A14 conﬁrms that PA remains effective with the skip connection.

Revised: As a result, a similar performance is reported in Figure A and Table A12.

Revised: This showcases that the proposed PA generalizes well across different GAN design choices (including networks with residual or skip connections).

Revised: A. C OMPARISON WITH THE DROPOUT REGULARIZATION

Revised: In this section we compare the proposed PA with another regularization technique which also aims to weaken the discriminator, such as adding dropout at the ﬁrst layer of the discriminator.

Original:  [Figure] 
Revised: Our [Figure] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Revised: proposed augmentation scheme is orthogonal to it and thus can be applied along with the dropout regularization.

Revised: We adopt dropout at the output of the ﬁrst convolutional layer of SN-DCGAN and experiment with two conﬁgurations.

Revised: In the ﬁrst conﬁguration the dropout rate is linearly increased from zero to 0 .

Revised: 7 , we call it an ascending mode ( A ).

Revised: In the second the dropout rate starts from 0 .

Revised: 7 and linearly descends to zero during the training, we call it a descending mode ( D ).

Revised: From Figure A16 and Table A we observe that neither of the two dropout modes improves the performance of NS GAN , whereas

Revised: PA shows to be more effective and preserves the performance gain even in combination with the dropout.