Original: I MPROVING G ENERALIZATION AND S TABILITY OF G ENERATIVE A DVERSARIAL N ETWORKS
Revised: I MPROVING G ENERALIZATION AND S TABILITY OF G ENERATIVE A DVERSARIAL N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative Adversarial Networks (GANs) are one of the most popular tools for learning complex high dimensional distributions.
Revised: Generative Adversarial Networks (GANs) are one of the most popular tools for learning complex high dimensional distributions.

Original: However, generalization properties of GANs have not been well understood.
Revised: However, generalization properties of GANs have not been well understood.

Original: In this paper, we analyze the generalization of GANs in practical settings.
Revised: In this paper, we analyze the generalization of GANs in practical settings.

Original: We show that discriminators trained on discrete datasets with the original GAN loss have poor generalization capability and do not approximate the theoretically optimal discriminator.
Revised: We show that discriminators trained on discrete datasets with the original GAN loss have poor generalization capability and do not approximate the theoretically optimal discriminator.

Original: We propose a zero-centered gradient penalty for improving the generalization of the discriminator by pushing it toward the optimal discriminator.
Revised: We propose a zero-centered gradient penalty for improving the generalization of the discriminator by pushing it toward the optimal discriminator.

Original: The penalty guarantees the generalization and convergence of GANs.
Revised: The penalty guarantees the generalization and convergence of GANs.

Original: Experiments on synthetic and large scale datasets verify our theoretical analysis
Revised: Experiments on synthetic and large scale datasets verify our theoretical analysis.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: GANs (Goodfellow et al., 2014) are one of the most popular tools for modeling high dimensional data.
Revised: GANs (Goodfellow et al., 2014) are one of the most popular tools for modeling high dimensional data.

Original: The original GAN is, however, highly unstable and often suffers from mode collapse.
Revised: The original GAN is, however, highly unstable and often suffers from mode collapse.

Original: Much of recent researches has focused on improving the stability of GANs (Radford et al., 2015; Arjovsky et al., 2017;
Revised: Much of recent researches has focused on improving the stability of GANs (Radford et al., 2015; Arjovsky et al., 2017;

Original: Heusel et al., 2017; Miyato et al., 2018; Karras et al., 2018).
Revised: Heusel et al., 2017; Miyato et al., 2018; Karras et al., 2018).

Original: On the theoretical aspect,
Revised: On the theoretical aspect,

Original: Nagarajan & Kolter (2017) proved that gradient based training of the original GAN is locally stable.
Revised: Nagarajan & Kolter (2017) proved that gradient based training of the original GAN is locally stable.

Original: Heusel et al.
Revised: Heusel et al.

Original: (2017) further proved that GANs trained with Two Timescale Update Rule (TTUR) converge to local equilibria.
Revised: (2017) further proved that GANs trained with Two Timescale Update Rule (TTUR) converge to local equilibria.

Original: However, the generalization of GANs at local equilibria is not discussed in depth in these papers.
Revised: However, the generalization of GANs at local equilibria is not discussed in depth in these papers.

Original: Arora et al.
Revised: Arora et al.

Original: (2017) showed that the generator can win by remembering a polynomial number of training examples.
Revised: (2017) showed that the generator can win by remembering a polynomial number of training examples.

Original: The result implies that a low capacity discriminator cannot detect the lack of diversity.
Revised: The result implies that a low capacity discriminator cannot detect the lack of diversity.

Original: Therefore, it cannot teach the generator to approximate the target distribution.
Revised: Therefore, it cannot teach the generator to approximate the target distribution.

Original: In section 4, we discuss the generalization capability of high capacity discriminators.
Revised: In section 4, we discuss the generalization capability of high capacity discriminators.

Original: We show that high capacity discriminators trained with the original GAN loss tends to overﬁt to the mislabeled samples in training dataset, guiding the generator toward collapsed equilibria (i.e. equilibria where the generator has mode collapse).
Revised: We show that high capacity discriminators trained with the original GAN loss tends to overﬁt to the mislabeled samples in training dataset, guiding the generator toward collapsed equilibria (i.e. equilibria where the generator has mode collapse).

Original: Arora et al.
Revised: Arora et al.

Original: (2018) proposed to measure the generalization capability of GAN by estimating the number of modes in the model distribution using the birthday paradox.
Revised: (2018) proposed to measure the generalization capability of GAN by estimating the number of modes in the model distribution using the birthday paradox.

Original: Experiments on several datasets showed that the number of modes in the model distribution is several times greater than the number of training examples.
Revised: Experiments on several datasets showed that the number of modes in the model distribution is several times greater than the number of training examples.

Original: The author concluded that although GANs might not be able to learn distributions, they do exhibit some level of generalization.
Revised: The author concluded that although GANs might not be able to learn distributions, they do exhibit some level of generalization.

Original: Our analysis shows that poor generalization comes from the mismatch between discriminators trained on discrete ﬁnite datasets and the theoretically optimal discriminator.
Revised: Our analysis shows that poor generalization comes from the mismatch between discriminators trained on discrete ﬁnite datasets and the theoretically optimal discriminator.

Original: We propose a zero-centered gradient penalty for improving the generalization capability of (high capacity) discriminators.
Revised: We propose a zero-centered gradient penalty for improving the generalization capability of (high capacity) discriminators.

Original: Our zero-centered gradient penalty pushes the discriminator toward the optimal one, making GAN to converge to equilibrium with good generalization capability.
Revised: Our zero-centered gradient penalty pushes the discriminator toward the optimal one, making GAN to converge to equilibrium with good generalization capability.

Original: Our contributions are as follow:
Revised: Our contributions are as follow:

Original: We show that discriminators trained with the original GAN loss have poor generalization capability.
Revised: We show that discriminators trained with the original GAN loss have poor generalization capability.

Original: Poor generalization in the discriminator prevents the generator from learning the target distribution.
Revised: Poor generalization in the discriminator prevents the generator from learning the target distribution.

Original: We show that the original GAN objective encourages gradient exploding in the discriminator.
Revised: We show that the original GAN objective encourages gradient exploding in the discriminator.

Original: Gradient exploding in the discriminator can lead to mode collapse in the generator.
Revised: Gradient exploding in the discriminator can lead to mode collapse in the generator.

Original: We propose a zero-centered gradient penalty (0-GP) for improving the generalization capability of the discriminator.
Revised: We propose a zero-centered gradient penalty (0-GP) for improving the generalization capability of the discriminator.

Original: We show that non-zero centered GP and the zero-centered GP proposed in Mescheder et al.
Revised: We show that non-zero centered GP and the zero-centered GP proposed in Mescheder et al.

Original: (2018) cannot make the discriminator generalize.
Revised: (2018) cannot make the discriminator generalize.

Original: Our 0-GP helps GANs to converge to generalizable equilibria.
Revised: Our 0-GP helps GANs to converge to generalizable equilibria.

Original: Theoretical results are veriﬁed on real world datasets.
Revised: Theoretical results are veriﬁed on real world datasets.

Original: We show that 0-GP helps the discriminator to distribute its capacity more equally between regions of the space, effectively preventing mode collapse.
Revised: We show that 0-GP helps the discriminator to distribute its capacity more equally between regions of the space, effectively preventing mode collapse.

Original: Experiments on synthetic and real world datasets verify that 0-GP can prevent mode collapse.
Revised: Experiments on synthetic and real world datasets verify that 0-GP can prevent mode collapse.

Original: GANs with 0-GP is much more robust to changes in hyper parameters, optimizers, and network architectures than the original GAN and GANs with other gradient penalties.
Revised: GANs with 0-GP is much more robust to changes in hyper parameters, optimizers, and network architectures than the original GAN and GANs with other gradient penalties.

Original: Table 1 compares the key properties of our 0-GP with one centered GP (1-GP) (Gulrajani et al., 2017) and zero centered GP on real/fake samples only (0-GP-sample) (Mescheder et al., 2018).
Revised: Table 1 compares the key properties of our 0-GP with one centered GP (1-GP) (Gulrajani et al., 2017) and zero centered GP on real/fake samples only (0-GP-sample) (Mescheder et al., 2018).

Original: N OTATIONS
Revised: N OTATIONS

Original: p r
Revised: p r

Original: the target distribution
Revised: the target distribution

Original: p g
Revised: p g

Original: the model distribution
Revised: the model distribution

Original: p z
Revised: p z

Original: the noise distribution
Revised: the noise distribution

Original: d x
Revised: d x

Original: the dimensionality of a data sample (real or fake)
Revised: the dimensionality of a data sample (real or fake)

Original: d z
Revised: d z

Original: the dimensionality of a noise sample
Revised: the dimensionality of a noise sample

Original: supp ( p )
Revised: supp ( p )

Revised: the

Original: the support of distribution p x ∼ p r a
Revised: support of distribution p x ∼ p r a

Original: real sample
Revised: real sample

Original: z ∼ p z 
Revised: z ∼ p z a

Original: a noise vector drawn from the noise distribution
Revised: noise vector drawn from the noise distribution

Original: p z y = G ( z ) a generated sample
Revised: p z y = G ( z ) a generated sample

Original: D [Equation] the set of n real samples 
Revised: D r = { x 1 , ..., x n } the set of n real samples D ( t ) g

Original: [Equation]
Revised: = (cid:110) y

Original:  the set of m generated samples at step t
Revised: ( t ) 1 , ..., y ( t ) m (cid:111) the set of m generated samples at step t

Original: [Equation]
Revised: D ( t ) =

Original:  the training dataset at step t
Revised: D r ∪ D ( t ) g the training dataset at step t

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Gradient penalties are widely used in GANs literature.
Revised: Gradient penalties are widely used in GANs literature.

Original: There are a plethora of works on using gradient penalty to improve the stability of GANs (Mescheder et al., 2018; Gulrajani et al., 2017;
Revised: There are a plethora of works on using gradient penalty to improve the stability of GANs (Mescheder et al., 2018; Gulrajani et al., 2017;

Original: Petzka et al., 2018; Roth et al., 2017; Qi, 2017).
Revised: Petzka et al., 2018; Roth et al., 2017; Qi, 2017).

Original: However, these works mostly focused on making the training of GANs stable and convergent.
Revised: However, these works mostly focused on making the training of GANs stable and convergent.

Original: Our work aims to improve the generalization capability of GANs via gradient regularization.
Revised: Our work aims to improve the generalization capability of GANs via gradient regularization.

Original: Arora et al.
Revised: Arora et al.

Original: (2018) showed that the number of modes in the model distribution grows linearly with the size of the discriminator.
Revised: (2018) showed that the number of modes in the model distribution grows linearly with the size of the discriminator.

Original: The result implies that higher capacity discriminators are needed for better approximation of the target distribution.
Revised: The result implies that higher capacity discriminators are needed for better approximation of the target distribution.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018) studied the tradeoff between generalization and discrimination in GANs.
Revised: (2018) studied the tradeoff between generalization and discrimination in GANs.

Original: The authors showed that generalization is guaranteed if the discriminator set is small enough.
Revised: The authors showed that generalization is guaranteed if the discriminator set is small enough.

Original: In practice, rich discriminators are usually used for better discriminative power.
Revised: In practice, rich discriminators are usually used for better discriminative power.

Original: Our GP makes rich discriminators generalizable while remaining discriminative.
Revised: Our GP makes rich discriminators generalizable while remaining discriminative.

Original: Although less mode collapse is not exactly the same as generalization, the ability to produce more diverse samples implies better generalization.
Revised: Although less mode collapse is not exactly the same as generalization, the ability to produce more diverse samples implies better generalization.

Original: There are a large number of papers on preventing mode collapse in GANs.
Revised: There are a large number of papers on preventing mode collapse in GANs.

Original: Radford et al.
Revised: Radford et al.

Original: ; Salimans et al.
Revised: ; Salimans et al.

Original: (2016) introduced a number of empirical tricks to help stabilizing GANs.
Revised: (2016) introduced a number of empirical tricks to help stabilizing GANs.

Original: Arjovsky & Bottou (2017) showed the importance of divergences in GAN training, leading to the introduction of Wasserstein GAN (Arjovsky et al., 2017).
Revised: Arjovsky & Bottou (2017) showed the importance of divergences in GAN training, leading to the introduction of Wasserstein GAN (Arjovsky et al., 2017).

Original: The use of weak divergence is further explored by Mroueh & Sercu (2017);
Revised: The use of weak divergence is further explored by Mroueh & Sercu (2017);

Original: Mroueh et al.
Revised: Mroueh et al.

Original: Lucas et al.
Revised: Lucas et al.

Original: (2018) advocated the use of mixed-batches, mini-batches of real and fake data,to smooth out the loss surface.
Revised: (2018) advocated the use of mixed-batches, mini-batches of real and fake data,

Original: The method exploits the distributional information in a mini-batch to prevent mode collapse.

Original: VEEGAN (Srivastava et al., 2017) uses an inverse of the generator to map the data to the prior distribution.

Original: The mismatch between the inverse mapping and the prior is used to detect mode collapse.

Original: If the generator can remember the entire training set, then the inverse mapping can be arbitrarily close the the prior distribution.

Original: It suggests that VEEGAN might not be

Original: GP
Revised: GP

Original: Formula
Revised: Formula

Original: Improve generalization
Revised: Improve generalization

Original: Prevent grad expoding
Revised: Prevent grad expoding

Original: Convergenceguarantee
Revised: Convergenceguarantee

Original: Our 0-GP λ E v ∈C [ (cid:107) ( ∇ D ) v (cid:107) 2 ] , C from y to x (cid:51) (cid:51) (cid:51) 1-GP λ E ˜ x
Revised: Our 0-GP λ E v ∈C [ (cid:107) ( ∇ D ) v (cid:107) 2 ] , C from y to x (cid:51) (cid:51) (cid:51) 1-GP λ E ˜ x

Original: [( (cid:107) ( ∇ D )
Revised: [( (cid:107) ( ∇ D )

Original: ˜ x (cid:107) − 1) 2 ] , where
Revised: ˜ x (cid:107) − 1) 2 ] , where

Original: ˜ x
Revised: ˜ x

Original: = α x + (1 − α ) y
Revised: = α x + (1 − α ) y

Original: (cid:55) (cid:51) (cid:55) 0-GP-sample λ
Revised: (cid:55) (cid:51) (cid:55) 0-GP-sample λ

Original: E v ∈D [ (cid:107) ( ∇ D ) v (cid:107) 2 ] (cid:55) (cid:55) (cid:51)able to help GAN to generalize outside of the training dataset.
Revised: E v ∈D [ (cid:107) ( ∇ D ) v (cid:107) 2 ] (cid:55) (cid:55) (cid:51) to smooth out the loss surface. The method exploits the distributional information in a mini-batch to prevent mode collapse. VEEGAN (Srivastava et al., 2017) uses an inverse of the generator to map the data to the prior distribution. The mismatch between the inverse mapping and the prior is used to detect mode collapse.

Revised: If the generator can remember the entire training set, then the inverse mapping can be arbitrarily close the the prior distribution.

Revised: It suggests that VEEGAN might not be able to help GAN to generalize outside of the training dataset.

Original: Our method helps GANs to discover unseen regions of the target distribution, signiﬁcantly improve the diversity of generated samples.
Revised: Our method helps GANs to discover unseen regions of the target distribution, signiﬁcantly improve the diversity of generated samples.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In the original GAN, the discriminator D maximizes the following objective
Revised: In the original GAN, the discriminator D maximizes the following objective

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2014) showed that if the density functions p g and p r are known, then for a ﬁxed generator G the optimal discriminator is[Equation]
Revised: (2014) showed that if the density functions p g and p r are known, then for a ﬁxed generator G the optimal discriminator is

Original: [Equation]
Revised: [Equation]

Original: In the beginning of the training, p g is very different from p r
Revised: In the beginning of the training, p g is very different from p r

Original: so we have p r ( x ) (cid:29) p g ( x ) , for x ∈ D r and p g ( y )
Revised: so we have p r ( x ) (cid:29) p g ( x ) , for x ∈ D r and p g ( y )

Original: (cid:29) p r
Revised: (cid:29) p r

Original: ( y ) , for y ∈ D g .
Revised: ( y ) , for y ∈ D g .

Original: Therefore, in the beginning of the training D ∗ ( x ) ≈ 1 , for x ∈
Revised: Therefore, in the beginning of the training D ∗ ( x ) ≈ 1 , for x ∈

Original: D r and D ∗ ( y ) ≈ 0 , for y ∈ D g .
Revised: D r and D ∗ ( y ) ≈ 0 , for y ∈ D g .

Original: As the training progresses, the generator will bring p g closer to p r .
Revised: As the training progresses, the generator will bring p g closer to p r .

Original: The game reaches the global equilibrium when p r = p g .
Revised: The game reaches the global equilibrium when p r = p g .

Original: At the global equilibrium, D ∗ ( v ) = 12 , ∀ v ∈ supp ( p r ) ∪ supp ( p g ) .
Revised: At the global equilibrium, D ∗ ( v ) = 12 , ∀ v ∈ supp ( p r ) ∪ supp ( p g ) .

Original: One important result of the original paper is that, if the discriminator is optimal at every step of the GAN algorithm, then p g converges to p r .
Revised: One important result of the original paper is that, if the discriminator is optimal at every step of the GAN algorithm, then p g converges to p r .

Original: In practice, density functions are not known and the optimal discriminator is approximated by optimizing the classiﬁcation performance of a parametric discriminator D ( · ; θ D ) on a discrete ﬁnite dataset D
Revised: In practice, density functions are not known and the optimal discriminator is approximated by optimizing the classiﬁcation performance of a parametric discriminator D ( · ; θ D ) on a discrete ﬁnite dataset D

Original: = D r ∪ D g .
Revised: = D r ∪ D g .

Original: We call a discriminator trained on a discrete ﬁnite dataset an empirical discriminator.
Revised: We call a discriminator trained on a discrete ﬁnite dataset an empirical discriminator.

Original: The empirically optimal discriminator is denoted by ˆ D ∗ .
Revised: The empirically optimal discriminator is denoted by ˆ D ∗ .

Original: Arora et al.
Revised: Arora et al.

Original: (2017) deﬁned generalization of a divergence d as follow:
Revised: (2017) deﬁned generalization of a divergence d as follow:

Original: A divergence d is said to have generalization error (cid:15) if
Revised: A divergence d is said to have generalization error (cid:15) if

Original: [Equation]
Revised: [Equation]

Original: A discriminator D deﬁnes a divergence between two distributions.
Revised: A discriminator D deﬁnes a divergence between two distributions.

Original: The performance of a discriminator with good generalization capability on the training dataset should be similar to that on the entire data space.
Revised: The performance of a discriminator with good generalization capability on the training dataset should be similar to that on the entire data space.

Original: In practice, generalization capability of D can be estimated by measuring the difference between its performance on the training dataset and a held-out dataset.
Revised: In practice, generalization capability of D can be estimated by measuring the difference between its performance on the training dataset and a held-out dataset.

Original:  G ENERALIZATION CAPABILITY OF DISCRIMINATORS 4. T HE EMPIRICALLY OPTIMAL DISCRIMINATOR DOES NOT APPROXIMATE THE THEORETICALLY OPTIMAL DISCRIMINATOR
Revised:  G ENERALIZATION CAPABILITY OF DISCRIMINATORS 4. T HE EMPIRICALLY OPTIMAL DISCRIMINATOR DOES NOT APPROXIMATE THE THEORETICALLY OPTIMAL DISCRIMINATOR

Original: It has been observed that if the discriminator is too good at discriminating real and fake samples, the generator cannot learn effectively (Goodfellow et al., 2014; Arjovsky & Bottou, 2017).
Revised: It has been observed that if the discriminator is too good at discriminating real and fake samples, the generator cannot learn effectively (Goodfellow et al., 2014; Arjovsky & Bottou, 2017).

Original: The phenomenon suggests that ˆ D ∗ does not well approximate D ∗ , and does not guarantee the convergence of p g to p r .
Revised: The phenomenon suggests that ˆ D ∗ does not well approximate D ∗ , and does not guarantee the convergence of p g to p r .

Original: In the following, we clarify the mismatch between ˆ D ∗ and D ∗ , and its implications.
Revised: In the following, we clarify the mismatch between ˆ D ∗ and D ∗ , and its implications.

Original: Proposition 1.

Original: The two datasets D r and D ( t ) g are disjoint with probability 1 regardless of how close the two distributions p r and p ( t ) g are.

Original: Powered by TCPDF (www.tcpdf.org)
Revised: Powered by TCPDF (www.tcpdf.org)

Original: Powered by TCPDF (www.tcpdf.org) (a) 2 1 0 1 2 3 41 02 3(b) 2 1 0 1 2 3 41 02 3(c) 2 1 0 1 2 3 41 02 3(d) 2 1 0 1 2 3 41 02 3(e) 2 1 0 1 2 3 41 02 30.0 0.
Revised: Powered by TCPDF (www.tcpdf.org) (a) 2 1 0 1 2 3 41 02 3(b) 2 1 0 1 2 3 41 02 3(c) 2 1 0 1 2 3 41 02 3(d) 2 1 0 1 2 3 41 02 3(e) 2 1 0 1 2 3 41 02 30.0 0.

Original: (f)
Revised: (f)

Revised: Proposition 1.

Revised: The two datasets D r and D ( t ) g are disjoint with probability 1 regardless of how close the two distributions p r and p ( t ) g are.

Original: Proof.
Revised: Proof.

Original: See appendix A.
Revised: See appendix A.

Original: D r and D ( t ) g are disjoint with probability 1 even when p g and p r are exactly the same.
Revised: D r and D ( t ) g are disjoint with probability 1 even when p g and p r are exactly the same.

Original: ˆ D ∗ perfectly classiﬁes the real and the fake datasets, and ˆ D ∗ ( x ) = 1 , ∀ x ∈ D r , ˆ D ∗ ( y )
Revised: ˆ D ∗ perfectly classiﬁes the real and the fake datasets, and ˆ D ∗ ( x ) = 1 , ∀ x ∈ D r , ˆ D ∗ ( y )

Original: = 0 , ∀ y ∈ D ( t ) g .
Revised: = 0 , ∀ y ∈ D ( t ) g .

Original: The value of ˆ D ∗ on D ( t ) does not depend on the distance between the two distributions and does not reﬂect the learning progress.
Revised: The value of ˆ D ∗ on D ( t ) does not depend on the distance between the two distributions and does not reﬂect the learning progress.

Original: The value of ˆ D ∗ on the training dataset approximates that of D ∗ in the beginning of the learning process but not when the two distributions are close.
Revised: The value of ˆ D ∗ on the training dataset approximates that of D ∗ in the beginning of the learning process but not when the two distributions are close.

Original: When trained using gradient descent on a discrete ﬁnite dataset with the loss in Eqn. 1, the discriminator D is pushed toward ˆ D ∗ , not D ∗ .
Revised: When trained using gradient descent on a discrete ﬁnite dataset with the loss in Eqn. 1, the discriminator D is pushed toward ˆ D ∗ , not D ∗ .

Original: This behavior does not depend on the size of training set (see Fig.
Revised: This behavior does not depend on the size of training set (see Fig.

Original: 1a, 1b), implying that the original GAN is not guaranteed to converge to the target distribution even when given enough data .
Revised: 1a, 1b), implying that the original GAN is not guaranteed to converge to the target distribution even when given enough data .

Original: 4. E MPIRICAL DISCRIMINATORS HAVE POOR GENERALIZATION CAPABILITY
Revised: 4. E MPIRICAL DISCRIMINATORS HAVE POOR GENERALIZATION CAPABILITY

Original: When the generator gets better, generated samples are more similar to samples from the target distribution.
Revised: When the generator gets better, generated samples are more similar to samples from the target distribution.

Original: However, regardless of their quality, generated samples are still labeled as fake in Eqn. 1.
Revised: However, regardless of their quality, generated samples are still labeled as fake in Eqn. 1.

Original: The training dataset D is a bad dataset as it contains many mislabeled examples.
Revised: The training dataset D is a bad dataset as it contains many mislabeled examples.

Original: A discriminator trained on such dataset will overﬁt to the mislabeled examples and has poor generalization capability.
Revised: A discriminator trained on such dataset will overﬁt to the mislabeled examples and has poor generalization capability.

Original: It will misclassify unseen samples and cannot teach the generator to generate these samples.
Revised: It will misclassify unseen samples and cannot teach the generator to generate these samples.

Original: Figure 1a and 1b demonstrate the problem on a synthetic dataset consisting of samples from two
Revised: Figure 1a and 1b demonstrate the problem on a synthetic dataset consisting of samples from two

Original: Gaussian distributions.
Revised: Gaussian distributions.

Original: The discriminator in Fig.
Revised: The discriminator in Fig.

Original: 1a overﬁts to the small dataset and does not generalize to new samples in Fig.
Revised: 1a overﬁts to the small dataset and does not generalize to new samples in Fig.

Original: 1b.
Revised: 1b.

Original: Although the discriminator in Fig.
Revised: Although the discriminator in Fig.

Original: 1b was trained on a larger dataset which is sufﬁcient to characterize the two distributions, it still overﬁts to the data and its value surface is very different from that of the theoretically optimal discriminator in Fig.
Revised: 1b was trained on a larger dataset which is sufﬁcient to characterize the two distributions, it still overﬁts to the data and its value surface is very different from that of the theoretically optimal discriminator in Fig.

Original: 1f.
Revised: 1f.

Original: An overﬁtted discriminator does not guide the model distribution toward target distribution but toward the real samples in the dataset.
Revised: An overﬁtted discriminator does not guide the model distribution toward target distribution but toward the real samples in the dataset.

Original: This explains why the original GAN usually exhibits mode collapse behavior.
Revised: This explains why the original GAN usually exhibits mode collapse behavior.

Original: Finding the empirically optimal discriminator using gradient descent usually requires many iterations.
Revised: Finding the empirically optimal discriminator using gradient descent usually requires many iterations.

Original: Heuristically, overﬁtting can be alleviated by limiting the number of discriminator updates per generator update.
Revised: Heuristically, overﬁtting can be alleviated by limiting the number of discriminator updates per generator update.

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2014) recommended to update the discriminator once every generator update.
Revised: (2014) recommended to update the discriminator once every generator update.

Original: In the next subsection, we show that limiting the number of discriminator updates per generator update prevents the discriminator from overﬁtting.
Revised: In the next subsection, we show that limiting the number of discriminator updates per generator update prevents the discriminator from overﬁtting.

Original: 4.2. (cid:15) - OPTIMAL DISCRIMINATORS
Revised: 4.2. (cid:15) - OPTIMAL DISCRIMINATORS

Original: ˆ D ∗ is costly to ﬁnd and maintain.
Revised: ˆ D ∗ is costly to ﬁnd and maintain.

Original: We consider here a weaker notion of optimality which can be achieved in practical settings.
Revised: We consider here a weaker notion of optimality which can be achieved in practical settings.

Original: Deﬁnition 1 ( (cid:15) -optimal discriminator) .
Revised: Deﬁnition 1 ( (cid:15) -optimal discriminator) .

Original: Given two disjoint datasets D r and D g , and a number (cid:15) > 0 , a discriminator D is (cid:15) -optimal if
Revised: Given two disjoint datasets D r and D g , and a number (cid:15) > 0 , a discriminator D is (cid:15) -optimal if

Original: [Equation]
Revised: [Equation]

Original: As observed in Goodfellow et al.
Revised: As observed in Goodfellow et al.

Original: (2014), ˆ D ∗ does not generate usable gradient for the generator.
Revised: (2014), ˆ D ∗ does not generate usable gradient for the generator.

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: proposed the non-saturating loss for the generator to circumvent this vanishing gradient problem.
Revised: proposed the non-saturating loss for the generator to circumvent this vanishing gradient problem.

Original: For an (cid:15) -optimal discriminator, if (cid:15) is relatively small, then the gradient of the discriminator w.r.t.
Revised: For an (cid:15) -optimal discriminator, if (cid:15) is relatively small, then the gradient of the discriminator w.r.t.

Original: fake datapoints might not vanish and can be used to guide the model distribution toward the target distribution.
Revised: fake datapoints might not vanish and can be used to guide the model distribution toward the target distribution.

Original: Proposition 2.
Revised: Proposition 2.

Original: Given two disjoint datasets D r and D g , and a number (cid:15) > 0 , an (cid:15) -optimal discriminator D (cid:15) exists and can be constructed as a one hidden layer MLP with O
Revised: Given two disjoint datasets D r and D g , and a number (cid:15) > 0 , an (cid:15) -optimal discriminator D (cid:15) exists and can be constructed as a one hidden layer MLP with O

Original: ( d x ( m + n )) parameters.
Revised: ( d x ( m + n )) parameters.

Original: Proof.
Revised: Proof.

Original: See appendix B.
Revised: See appendix B.

Original: Because deep networks are more powerful than shallow ones, the size of a deep (cid:15) -optimal discriminator can be much smaller than O ( d x ( m + n )) .
Revised: Because deep networks are more powerful than shallow ones, the size of a deep (cid:15) -optimal discriminator can be much smaller than O ( d x ( m + n )) .

Original: From the formula, the size of a shallow (cid:15) -optimal discriminator for real world datasets ranges from a few to hundreds of millions parameters.
Revised: From the formula, the size of a shallow (cid:15) -optimal discriminator for real world datasets ranges from a few to hundreds of millions parameters.

Original: That is comparable to the size of discriminators used in practice.
Revised: That is comparable to the size of discriminators used in practice.

Original: Arjovsky & Bottou (2017) showed that even when the generator can generate realistic samples, a discriminator that can perfectly classify real and fake samples can be found easily using gradient descent.
Revised: Arjovsky & Bottou (2017) showed that even when the generator can generate realistic samples, a discriminator that can perfectly classify real and fake samples can be found easily using gradient descent.

Original: The experiment veriﬁed that (cid:15) -optimal discriminator can be found using gradient descent in practical settings.
Revised: The experiment veriﬁed that (cid:15) -optimal discriminator can be found using gradient descent in practical settings.

Original: We observe that the norm of the gradient w.r.t.
Revised: We observe that the norm of the gradient w.r.t.

Original: the discriminator’s parameters decreases as fakes samples approach real samples.
Revised: the discriminator’s parameters decreases as fakes samples approach real samples.

Original: If the discriminator’s learning rate is ﬁxed, then the number of gradient descent steps that the discriminator has to take to reach (cid:15) -optimal state should increase.
Revised: If the discriminator’s learning rate is ﬁxed, then the number of gradient descent steps that the discriminator has to take to reach (cid:15) -optimal state should increase.

Original: Proposition 3.
Revised: Proposition 3.

Original: Alternating gradient descent with the same learning rate for discriminator and generator, and ﬁxed number of discriminator updates per generator update (Fixed-Alt-GD) cannot maintain the (empirical) optimality of the discriminator.
Revised: Alternating gradient descent with the same learning rate for discriminator and generator, and ﬁxed number of discriminator updates per generator update (Fixed-Alt-GD) cannot maintain the (empirical) optimality of the discriminator.

Original: Fixed-Alt-GD decreases the discriminative power of the discriminator to improve its generalization capability.
Revised: Fixed-Alt-GD decreases the discriminative power of the discriminator to improve its generalization capability.

Original: The proof for linear case is given in appendix C.
Revised: The proof for linear case is given in appendix C.

Original: In GANs trained with Two Timescale Update Rule (TTUR) (Heusel et al., 2017), the ratio between the learning rate of the discriminator and that of the generator goes to inﬁnity as the iteration number goes to inﬁnity.
Revised: In GANs trained with Two Timescale Update Rule (TTUR) (Heusel et al., 2017), the ratio between the learning rate of the discriminator and that of the generator goes to inﬁnity as the iteration number goes to inﬁnity.

Original: Therefore, the discriminator can learn much faster than the generator and might be able to maintain its optimality throughout the learning process.
Revised: Therefore, the discriminator can learn much faster than the generator and might be able to maintain its optimality throughout the learning process.

Original: 4.2. G RADIENT EXPLODING IN (cid:15) - OPTIMAL DISCRIMINATORS
Revised: 4.2. G RADIENT EXPLODING IN (cid:15) - OPTIMAL DISCRIMINATORS

Original: Let’s consider a simpliﬁed scenario where the real and the fake datasets each contains a single datapoint: D r = { x } , D ( t ) g
Revised: Let’s consider a simpliﬁed scenario where the real and the fake datasets each contains a single datapoint: D r = { x } , D ( t ) g

Original: = (cid:8) y
Revised: = (cid:8) y

Original: ( t ) (cid:9) .
Revised: ( t ) (cid:9) .

Original: Updating the generator according to the gradient from the discriminator will push y ( t ) toward x .
Revised: Updating the generator according to the gradient from the discriminator will push y ( t ) toward x .

Original: The absolute value of directional derivative of D in the direction u
Revised: The absolute value of directional derivative of D in the direction u

Original: = x − y ( t ) , at x is | ( ∇ [Equation] [Equation] | =
Revised: = x − y ( t ) , at x is | ( ∇ u D ) x | =

Original: lim y [Equation] t ) u −→ x (cid:12)(cid:12) D ( x ) − D
Revised: lim y ( t ) u −→ x (cid:12)(cid:12) D ( x ) − D

Original: ( y ( t ) )
Revised: y ( t ) )

Original: (cid:12)(cid:12) (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13)
Revised: (cid:12)(cid:12) (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13)

Original: If D is always (cid:15) -optimal, then (cid:12)(cid:12) D ( x ) − D
Revised: If D is always (cid:15) -optimal, then (cid:12)(cid:12) D ( x ) − D

Original: y ( t ) )
Revised: y ( t ) )

Original: (cid:12)(cid:12) ≥ (cid:15), ∀ t ∈ N , and | ( ∇ u D ) x | ≥ 
Revised: (cid:12)(cid:12) ≥ (cid:15), ∀ t ∈ N , and | ( ∇ u D ) x | ≥ lim

Original: [Equation]
Revised: [Equation]

Original: The directional derivate of the (cid:15) -optimal discriminator explodes as the fake datapoint approaches the real datapoint.
Revised: The directional derivate of the (cid:15) -optimal discriminator explodes as the fake datapoint approaches the real datapoint.

Original: Directional derivative exploding implies gradient exploding at datapoints on the line
Revised: Directional derivative exploding implies gradient exploding at datapoints on the line

Original: (a)    10 10.0 7. (b)    10 10.0 7. (c)    10 10.0 7. (d)    10 10.0 7.
Revised: [Figure]

Original: (e)    10 10.0 7. (f)    10 10.0 7. (g)    10 10.0 7. (h)    10 10.0 7.
Revised: [Figure]

Original: [Figure][Figure] [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: (i) segment connecting x and y ( t ) .
Revised: segment connecting x and y ( t ) .

Original: If in the next iteration, the generator produces a sample in a region where the gradient explodes, then the gradient w.r.t.
Revised: If in the next iteration, the generator produces a sample in a region where the gradient explodes, then the gradient w.r.t.

Original: the generator’s parameters explodes.
Revised: the generator’s parameters explodes.

Original: Let’s consider the following line integral
Revised: Let’s consider the following line integral

Original: [Equation]
Revised: [Equation]

Original: where C is the line segment from y ( t ) to x .
Revised: where C is the line segment from y ( t ) to x .

Original: As the model distribution gets closer to the target distribution, the length of C should be non increasing.
Revised: As the model distribution gets closer to the target distribution, the length of C should be non increasing.

Original: Therefore, maximizing D ( x ) − D ( y ( t ) ) , or the discriminative power of D , leads to the maximization of the directional derivative of D in the direction d s .
Revised: Therefore, maximizing D ( x ) − D ( y ( t ) ) , or the discriminative power of D , leads to the maximization of the directional derivative of D in the direction d s .

Original: The original GAN loss makes D to maximize its discriminative power, encouraging gradient exploding to occur.
Revised: The original GAN loss makes D to maximize its discriminative power, encouraging gradient exploding to occur.

Original: Gradient exploding happens in the discriminator trained with TTUR in Fig.
Revised: Gradient exploding happens in the discriminator trained with TTUR in Fig.

Original: 2c and 2d.
Revised: 2c and 2d.

Original: Because
Revised: Because

Original: TTUR can help the discriminator to maintain its optimality, gradient exploding happens and persists throughout the training process.
Revised: TTUR can help the discriminator to maintain its optimality, gradient exploding happens and persists throughout the training process.

Original: Without TTUR, the discriminator cannot maintain its optimality so gradient exploding can happen sometimes during the training but does not persist (Fig.
Revised: Without TTUR, the discriminator cannot maintain its optimality so gradient exploding can happen sometimes during the training but does not persist (Fig.

Original: 2a and 2b).
Revised: 2a and 2b).

Original: Because of the saturated regions in the sigmoid function used in neural network based discriminators, the gradient w.r.t.
Revised: Because of the saturated regions in the sigmoid function used in neural network based discriminators, the gradient w.r.t.

Original: datapoints in the training set could vanishes.
Revised: datapoints in the training set could vanishes.

Original: However, gradient exploding must happen at some datapoints on the path between a pair of samples, where the sigmoid function does not saturate.
Revised: However, gradient exploding must happen at some datapoints on the path between a pair of samples, where the sigmoid function does not saturate.

Original: In Fig.
Revised: In Fig.

Original: 1a, gradient exploding happens near the decision boundary.
Revised: 1a, gradient exploding happens near the decision boundary.

Original: In practice, D r and D g contain many datapoints and the generator is updated using the average of gradients of the discriminator w.r.t.
Revised: In practice, D r and D g contain many datapoints and the generator is updated using the average of gradients of the discriminator w.r.t.

Original: fake datapoints in the mini-batch.
Revised: fake datapoints in the mini-batch.

Original: If a fake datapoint y 0 is very close to a real datapoint x 0 , the gradient ( ∇ D ) y 0 might explode.
Revised: If a fake datapoint y 0 is very close to a real datapoint x 0 , the gradient ( ∇ D ) y 0 might explode.

Original: When the average gradient is computed over the mini-batch, ( ∇ D ) y 0 outweighs other gradients.
Revised: When the average gradient is computed over the mini-batch, ( ∇ D ) y 0 outweighs other gradients.

Original: The generator updated with this average gradient will move many fake datapoints in the direction of ( ∇ D ) y 0 , toward x 0 , making mode collapse visible.
Revised: The generator updated with this average gradient will move many fake datapoints in the direction of ( ∇ D ) y 0 , toward x 0 , making mode collapse visible.

Original:  I MPROVING GENERALIZATION CAPABILITY OF EMPIRICAL DISCRIMINATORS
Revised:  I MPROVING GENERALIZATION CAPABILITY OF EMPIRICAL DISCRIMINATORS

Original: Although the theoretically optimal discriminator D ∗ is generalizable, the original GAN loss does not push empirical discriminators toward D ∗ .
Revised: Although the theoretically optimal discriminator D ∗ is generalizable, the original GAN loss does not push empirical discriminators toward D ∗ .

Original: We aim to improve the generalization capability of empirical discriminators by pushing them toward D ∗ .
Revised: We aim to improve the generalization capability of empirical discriminators by pushing them toward D ∗ .

Original: 5. P USHING EMPIRICAL DISCRIMINATORS TOWARD D ∗
Revised: 5. P USHING EMPIRICAL DISCRIMINATORS TOWARD D ∗

Original: For any input v ∈ supp ( p r ) ∪ supp ( p g ) , the value of D ∗ ( v ) goes to 12 and the gradient ( ∇ D ) v goes to 0 as p g approaches p r .
Revised: For any input v ∈ supp ( p r ) ∪ supp ( p g ) , the value of D ∗ ( v ) goes to 12 and the gradient ( ∇ D ) v goes to 0 as p g approaches p r .

Original: Consider again the line integral in Eqn. 4.
Revised: Consider again the line integral in Eqn. 4.

Original: As D ∗ ( x ) and D ∗ ( y ) approach 12 for all x ∈ supp ( p r ) and y ∈ supp ( p g ) , we have
Revised: As D ∗ ( x ) and D ∗ ( y ) approach 12 for all x ∈ supp ( p r ) and y ∈ supp ( p g ) , we have

Original: [Equation]
Revised: [Equation]

Original: for all pairs of x and y and all paths C from y to x .
Revised: for all pairs of x and y and all paths C from y to x .

Original: That means, the discriminative power of D ∗ must decrease as the two distributions become more similar.
Revised: That means, the discriminative power of D ∗ must decrease as the two distributions become more similar.

Original: To push an empirical discriminator D toward D ∗ [Equation] we force D to satisfy two requirements:
Revised: To push an empirical discriminator D toward D ∗ , we force D to satisfy two requirements:

Original: [Equation]
Revised: [Equation]

Original: 5. Z ERO - CENTERED GRADIENT PENALTY
Revised: 5. Z ERO - CENTERED GRADIENT PENALTY

Original: The ﬁrst requirement can be implemented by sampling some datapoints v ∈ supp ( p r ) ∪ supp ( p g ) and force ( ∇ D ) v to be 0 .
Revised: The ﬁrst requirement can be implemented by sampling some datapoints v ∈ supp ( p r ) ∪ supp ( p g ) and force ( ∇ D ) v to be 0 .

Original: The second requirement can be implemented by sampling pairs of real and fake datapoints ( x , y ) and force D ( x ) − D ( y ) to be 0.
Revised: The second requirement can be implemented by sampling pairs of real and fake datapoints ( x , y ) and force D ( x ) − D ( y ) to be 0.

Original: The two requirements can be added to the discriminator’s objective as follows
Revised: The two requirements can be added to the discriminator’s objective as follows

Original: [Equation]
Revised: [Equation]

Original: D ( y )) 2 ] where L is the objective in Eqn. 1.
Revised: D ( y )) 2 ] where L is the objective in Eqn. 1.

Original: However, as discussed in section 4.2.2, an (cid:15) -optimal discriminator can have zero gradient on the training dataset and have gradient exploding outside of the training dataset.
Revised: However, as discussed in section 4.2.2, an (cid:15) -optimal discriminator can have zero gradient on the training dataset and have gradient exploding outside of the training dataset.

Original: The gradient norm could go to inﬁnity even when D ( x ) − D ( y ) is small.
Revised: The gradient norm could go to inﬁnity even when D ( x ) − D ( y ) is small.

Original: Regulating the difference between D ( x ) and D ( y ) is not an efﬁcient way to prevent gradient exploding.
Revised: Regulating the difference between D ( x ) and D ( y ) is not an efﬁcient way to prevent gradient exploding.

Original: We want to prevent gradient exploding on every path in supp ( p r ) ∪ supp ( p g ) .
Revised: We want to prevent gradient exploding on every path in supp ( p r ) ∪ supp ( p g ) .

Original: Because ( ∇ D ∗ ) v → 0 for all v ∈ supp ( p r ) ∪ supp ( p g ) as p g approach p r , we could push the gradient w.r.t.
Revised: Because ( ∇ D ∗ ) v → 0 for all v ∈ supp ( p r ) ∪ supp ( p g ) as p g approach p r , we could push the gradient w.r.t.

Original: every datapoint on every path C ∈ supp ( p r ) ∪ supp ( p g ) toward 0 .
Revised: every datapoint on every path C ∈ supp ( p r ) ∪ supp ( p g ) toward 0 .

Original: We note that, if ( ∇ D ) v → 0 , ∀ v ∈ C then (cid:82) C ( ∇ D ) v · d s → 0 .
Revised: We note that, if ( ∇ D ) v → 0 , ∀ v ∈ C then (cid:82) C ( ∇ D ) v · d s → 0 .

Original: Therefore, the two requirements can be enforced by a single zero-centered gradient penalty of the form
Revised: Therefore, the two requirements can be enforced by a single zero-centered gradient penalty of the form

Original: [Equation]
Revised: [Equation]

Original: The remaining problem is how to ﬁnd the path C from a fake to a real sample which lies inside supp ( p r ) ∪ supp ( p g ) .
Revised: The remaining problem is how to ﬁnd the path C from a fake to a real sample which lies inside supp ( p r ) ∪ supp ( p g ) .

Original: Because we do not have access to the full supports of p r and p g , and the supports of two distributions could be disjoint in the beginning of the training process, ﬁnding a path which lies completely inside the support is infeasible.
Revised: Because we do not have access to the full supports of p r and p g , and the supports of two distributions could be disjoint in the beginning of the training process, ﬁnding a path which lies completely inside the support is infeasible.

Original: In the current implementation, we approximate C with the straight line connecting a pair of samples, although there is no guarantee that all datapoints on that straight line are in supp ( p r ) ∪ supp ( p g ) .
Revised: In the current implementation, we approximate C with the straight line connecting a pair of samples, although there is no guarantee that all datapoints on that straight line are in supp ( p r ) ∪ supp ( p g ) .

Original: That results in the following objective
Revised: That results in the following objective

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: ˜ x
Revised: ˜ x

Original: = α x + (1 − α )
Revised: = α x + (1 − α )

Original: y , x ∼ p r , y ∼ p g , and α ∼ U (0 , 1) .
Revised: y , x ∼ p r , y ∼ p g , and α ∼ U (0 , 1) .

Original: We describe a more sophisticated way of ﬁnding a better path in appendix F.
Revised: We describe a more sophisticated way of ﬁnding a better path in appendix F.

Original: The larger λ is, the stronger ( ∇ D )
Revised: The larger λ is, the stronger ( ∇ D )

Original: ˜ x is pushed toward 0 .
Revised: ˜ x is pushed toward 0 .

Original: If λ is 0, then the discriminator will only focus on maximizing its discriminative power.
Revised: If λ is 0, then the discriminator will only focus on maximizing its discriminative power.

Original: If λ approaches inﬁnity, then the discriminator has maximum generalization capability and no discriminative power.
Revised: If λ approaches inﬁnity, then the discriminator has maximum generalization capability and no discriminative power.

Original: λ controls the tradeoff between discrimination and generalization in the discriminator.
Revised: λ controls the tradeoff between discrimination and generalization in the discriminator.

Original: 5. G ENERALIZATION CAPABILITY OF DIFFERENT GRADIENT PENALTIES
Revised: 5. G ENERALIZATION CAPABILITY OF DIFFERENT GRADIENT PENALTIES

Original: Mescheder et al.
Revised: Mescheder et al.

Original: (2018) proposed to force the gradient w.r.t.
Revised: (2018) proposed to force the gradient w.r.t.

Original: datapoints in the real and/or fake dataset(s) to be 0 to make the training of GANs convergent.
Revised: datapoints in the real and/or fake dataset(s) to be 0 to make the training of GANs convergent.

Original: In section 4, we showed that for discrete training dataset, an empirically optimal discriminator ˆ D ∗ always exists and could be found by gradient descent.
Revised: In section 4, we showed that for discrete training dataset, an empirically optimal discriminator ˆ D ∗ always exists and could be found by gradient descent.

Original: Although ( ∇ ˆ D ∗ ) v = 0 , ∀ v ∈ D , ˆ D ∗ does not satisfy the requirement in
Revised: Although ( ∇ ˆ D ∗ ) v = 0 , ∀ v ∈ D , ˆ D ∗ does not satisfy the requirement in

Original: Eqn. 5 and have gradient exploding when some fake datapoints approach a real datapoint.
Revised: Eqn. 5 and have gradient exploding when some fake datapoints approach a real datapoint.

Original: The discriminators in Fig.
Revised: The discriminators in Fig.

Original: 1a, 1b, 1d, 2c and 2d have vanishingly small gradients on datapoints in the training dataset and very large gradients outside.
Revised: 1a, 1b, 1d, 2c and 2d have vanishingly small gradients on datapoints in the training dataset and very large gradients outside.

Original: They have poor generalization capability and cannot teach the generator to generate unseen real datapoints.
Revised: They have poor generalization capability and cannot teach the generator to generate unseen real datapoints.

Original: Therefore, zero-centered gradient penalty on samples from p r and p g only cannot help improving the generalization of the discriminator .
Revised: Therefore, zero-centered gradient penalty on samples from p r and p g only cannot help improving the generalization of the discriminator .

Original: Non-zero centered GPs do not push an empirical discriminator toward D ∗ because the gradient does not converge to 0 .
Revised: Non-zero centered GPs do not push an empirical discriminator toward D ∗ because the gradient does not converge to 0 .

Original: A commonly used non-zero centered GP is the one-centered GP (1-GP) (Gulrajani et al., 2017) which has the following form
Revised: A commonly used non-zero centered GP is the one-centered GP (1-GP) (Gulrajani et al., 2017) which has the following form

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: ˜ x = α x +(1 − α )
Revised: ˜ x = α x +(1 − α )

Original: y , x ∼ p r , y ∼ p g , and α ∼ U (0 , 1) .
Revised: y , x ∼ p r , y ∼ p g , and α ∼ U (0 , 1) .

Original: Although the initial goal of 1-GP was to enforce Lipschitz constraint on the discriminator 1 , Fedus et al.
Revised: Although the initial goal of 1-GP was to enforce Lipschitz constraint on the discriminator 1 , Fedus et al.

Original: (2018) found that 1-GP prevents gradient exploding, making the original GAN more stable.
Revised: (2018) found that 1-GP prevents gradient exploding, making the original GAN more stable.

Original: 1-GP forces the norm of gradients w.r.t.
Revised: 1-GP forces the norm of gradients w.r.t.

Original: datapoints on the line segment connecting x and y to be 1.
Revised: datapoints on the line segment connecting x and y to be 1.

Original: If all gradients on the line segment have norm 1 , then the line integral in Eqn. 4 could be as large as (cid:107) x
Revised: If all gradients on the line segment have norm 1 , then the line integral in Eqn. 4 could be as large as (cid:107) x

Original: − y (cid:107) .
Revised: − y (cid:107) .

Original: Because the distance between random samples grows with the dimensionality, in high dimensional space (cid:107) x − y (cid:107) is greater than 1 with high probability.
Revised: Because the distance between random samples grows with the dimensionality, in high dimensional space (cid:107) x − y (cid:107) is greater than 1 with high probability.

Original: The discriminator could maximize the value of the line integral without violating the Lipschitz constraint.
Revised: The discriminator could maximize the value of the line integral without violating the Lipschitz constraint.

Original: The discriminator trained with 1-GP, therefore, can overﬁt to the training data and have poor generalization capability.
Revised: The discriminator trained with 1-GP, therefore, can overﬁt to the training data and have poor generalization capability.

Original: 5. C ONVERGENCE ANALYSIS FOR ZERO - CENTERED GRADIENT PENALTY
Revised: 5. C ONVERGENCE ANALYSIS FOR ZERO - CENTERED GRADIENT PENALTY

Original: Mescheder et al.
Revised: Mescheder et al.

Original: (2018) showed that zero-centered GP on real and/or fake samples (0-GP-sample) makes GANs convergent.
Revised: (2018) showed that zero-centered GP on real and/or fake samples (0-GP-sample) makes GANs convergent.

Original: The penalty is based on the convergence analysis for the Dirac GAN, an 1-dimensional linear GAN which learns the Dirac distribution.
Revised: The penalty is based on the convergence analysis for the Dirac GAN, an 1-dimensional linear GAN which learns the Dirac distribution.

Original: The intuition is that when p g is the same as p r , the gradient of the discriminator w.r.t.
Revised: The intuition is that when p g is the same as p r , the gradient of the discriminator w.r.t.

Original: the fake datapoints (which are also real datapoints) should be 

Revised: the fake datapoints (which are also real datapoints) should be

Original: so that generator will not move away when being updated using this gradient.
Revised: so that generator will not move away when being updated using this gradient.

Original: If the gradient from the discriminator is not 0 , then the generator will oscillate around the equilibrium.
Revised: If the gradient from the discriminator is not 0 , then the generator will oscillate around the equilibrium.

Original: Our GP forces the gradient w.r.t.
Revised: Our GP forces the gradient w.r.t.

Original: all datapoints on the line segment between a pair of samples (including the two endpoints) to be 0 .
Revised: all datapoints on the line segment between a pair of samples (including the two endpoints) to be 0 .

Original: As a result, our GP also prevents the generator from oscillating.
Revised: As a result, our GP also prevents the generator from oscillating.

Original: Therefore, our GP has the same convergence guarantee as the 0-GP-sample.
Revised: Therefore, our GP has the same convergence guarantee as the 0-GP-sample.

Original: 5. Z ERO - CENTERED GRADIENT PENALTY IMPROVES CAPACITY DISTRIBUTION
Revised: 5. Z ERO - CENTERED GRADIENT PENALTY IMPROVES CAPACITY DISTRIBUTION

Original: Discriminators trained with the original GAN loss tends to focus on the region of the where fake samples are close to real samples, ignoring other regions.
Revised: Discriminators trained with the original GAN loss tends to focus on the region of the where fake samples are close to real samples, ignoring other regions.

Original: The phenomenon can be seen in Fig.
Revised: The phenomenon can be seen in Fig.

Original: 2a, 2b, 2c, 2d, 2h and 2i.
Revised: 2a, 2b, 2c, 2d, 2h and 2i.

Original: Gradients in the region where fake samples are concentrated are large while gradients in other regions, including regions where real samples are located, are very small.
Revised: Gradients in the region where fake samples are concentrated are large while gradients in other regions, including regions where real samples are located, are very small.

Original: The generator cannot discover and generate real datapoints in regions where the gradient vanishes.
Revised: The generator cannot discover and generate real datapoints in regions where the gradient vanishes.

Original: When trained with the objective in Eqn.
Revised: When trained with the objective in Eqn.

Original: 6, the discriminator will have to balance between maximizing L and minimizing the GP.
Revised: 6, the discriminator will have to balance between maximizing L and minimizing the GP.

Original: For ﬁnite λ , the GP term will not be exactly 0.
Revised: For ﬁnite λ , the GP term will not be exactly 0.

Original: Let γ = λ E ˜ x
Revised: Let γ [Equation]

Original: [ (cid:107) ( ∇ D ) ˜ x (cid:107) 2 ] .

Original: Among discriminators with the same value of γ , gradient descent will ﬁnd the discriminator that maximizes L .
Revised: Among discriminators with the same value of γ , gradient descent will ﬁnd the discriminator that maximizes L .

Original: As discussed in section 4.2.2, maximizing L leads to the maximization of norms of gradients on the path from y to x .
Revised: As discussed in section 4.2.2, maximizing L leads to the maximization of norms of gradients on the path from y to x .

Original: The discriminator should maximize the value η = E ˜ x
Revised: The discriminator should maximize the value η = E ˜ x

Original: [ (cid:107) ( ∇ D )
Revised: [ (cid:107) ( ∇ D )

Original: ˜ x (cid:107) ] .
Revised: ˜ x (cid:107) ] .

Original: If γ is ﬁxed then η is maximized when (cid:107)∇ D ˜ x ( i ) (cid:107)
Revised: If γ is ﬁxed then η is maximized when (cid:107)∇ D ˜ x ( i ) (cid:107)

Original: =
Revised: =

Original: (cid:107)∇ D ˜ x ( j ) (cid:107) , ∀ i, j (Cauchy-Schwarz inequality).
Revised: (cid:107)∇ D ˜ x ( j ) (cid:107) , ∀ i, j (Cauchy-Schwarz inequality).

Original: Therefore, our zero-centered GP encourages the gradients at different regions of the real data space to have the same norm.
Revised: Therefore, our zero-centered GP encourages the gradients at different regions of the real data space to have the same norm.

Original: The capacity of D is distributed more equally between regions of the real data space, effectively reduce mode collapse.
Revised: The capacity of D is distributed more equally between regions of the real data space, effectively reduce mode collapse.

Original: The effect can be seen in
Revised: The effect can be seen in

Original: Fig.
Revised: Fig.

Original: 2e and 2f.
Revised: 2e and 2f.

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: That allows gradient norms to be smaller than 1 in some regions and larger than 1 in some other regions.
Revised: That allows gradient norms to be smaller than 1 in some regions and larger than 1 in some other regions.

Original: The problem can be seen in
Revised: The problem can be seen in

Original: Fig. 2h.
Revised: Fig. 2h.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The code for all experiments will be released after the review process.
Revised: The code for all experiments will be released after the review process.

Original: 6. Z ERO - CENTERED GRADIENT PENALTY PREVENTS OVERFITTING
Revised: 6. Z ERO - CENTERED GRADIENT PENALTY PREVENTS OVERFITTING

Original: To test the effectiveness of gradient penalties in preventing overﬁtting, we designed a dataset with real and fake samples coming from two Gaussian distributions and trained a MLP based discriminator on that dataset.
Revised: To test the effectiveness of gradient penalties in preventing overﬁtting, we designed a dataset with real and fake samples coming from two Gaussian distributions and trained a MLP based discriminator on that dataset.

Original: The result is shown in Fig.
Revised: The result is shown in Fig.

Original: As predicted in section 5.3, 0-GP-sample does not help to improve generalization.
Revised: As predicted in section 5.3, 0-GP-sample does not help to improve generalization.

Original: 1-GP helps to improve generalization.
Revised: 1-GP helps to improve generalization.

Original: The value surface in Fig.
Revised: The value surface in Fig.

Original: 1c is smoother than that in Fig.
Revised: 1c is smoother than that in Fig.

Original: 1a.
Revised: 1a.

Original: However, as discussed in section 5.3, 1-GP cannot help much in higher dimensional space where the pair-wise distances are large.
Revised: However, as discussed in section 5.3, 1-GP cannot help much in higher dimensional space where the pair-wise distances are large.

Original: The discriminator trained with our 0-GP has the best generalization capability, with a value surface which is the most similar to that of the theoretically optimal one.
Revised: The discriminator trained with our 0-GP has the best generalization capability, with a value surface which is the most similar to that of the theoretically optimal one.

Original: We increased the number of discriminator updates per generator update to 5 to see the effect of GPs in preventing overﬁtting.
Revised: We increased the number of discriminator updates per generator update to 5 to see the effect of GPs in preventing overﬁtting.

Original: On the MNIST dataset, GAN without GP and with other GPs cannot learn anything after 10,000 iterations.
Revised: On the MNIST dataset, GAN without GP and with other GPs cannot learn anything after 10,000 iterations.

Original: GAN with our 0-GP can still learn normally and start produce recognizable digits after only 1,000 iterations.
Revised: GAN with our 0-GP can still learn normally and start produce recognizable digits after only 1,000 iterations.

Original: The result conﬁrms that our GP is effective in preventing overﬁtting in the discriminator
Revised: The result conﬁrms that our GP is effective in preventing overﬁtting in the discriminator.

Original: . 6. Z ERO - CENTERED GRADIENT PENALTY IMPROVES GENERALIZATION AND ROBUSTNESS OF GAN S S YNTHETIC DATA
Revised: 6. Z ERO - CENTERED GRADIENT PENALTY IMPROVES GENERALIZATION AND ROBUSTNESS OF GAN S S YNTHETIC DATA

Original: We tested different gradient penalties on a number of synthetic datasets to compare their effectiveness.
Revised: We tested different gradient penalties on a number of synthetic datasets to compare their effectiveness.

Original: The ﬁrst dataset is a mixture of 8 Gaussians.
Revised: The ﬁrst dataset is a mixture of 8 Gaussians.

Original: The dataset is scaled up by a factor of 10 to simulate the situation in high dimensional space where random samples are far from each other.
Revised: The dataset is scaled up by a factor of 10 to simulate the situation in high dimensional space where random samples are far from each other.

Original: The result is shown in Fig.
Revised: The result is shown in Fig.

Original: GANs with other gradient penalties all fail to learn the distribution and exhibit mode collapse problem to different extents.
Revised: GANs with other gradient penalties all fail to learn the distribution and exhibit mode collapse problem to different extents.

Original: GAN with our 0-GP (GAN-0-GP) can successfully learn the distribution.
Revised: GAN with our 0-GP (GAN-0-GP) can successfully learn the distribution.

Original: Furthermore, GAN-0-GP can generate datapoints on the circle, demonstrating good generalization capability.
Revised: Furthermore, GAN-0-GP can generate datapoints on the circle, demonstrating good generalization capability.

Original: The original GAN collapses to some disconnected modes and cannot perform smooth interpolation between modes: small change in the input result in large, unpredictable change in the output.
Revised: The original GAN collapses to some disconnected modes and cannot perform smooth interpolation between modes: small change in the input result in large, unpredictable change in the output.

Original: GAN with zero-centered GP on real/fake samples only also exhibits the same ”mode jumping” behavior.
Revised: GAN with zero-centered GP on real/fake samples only also exhibits the same ”mode jumping” behavior.

Original: The behavior suggests that these GANs tend to remember the training dataset and have poor generalization capability.
Revised: The behavior suggests that these GANs tend to remember the training dataset and have poor generalization capability.

Original: Fig.
Revised: Fig.

Original: 9 in appendix D demonstrates the problem on MNIST dataset.
Revised: 9 in appendix D demonstrates the problem on MNIST dataset.

Original: We observe that GAN-0-GP behaves similar to Wasserstein GAN as it ﬁrst learns the overall structure of the distribution and then focuses on the modes.
Revised: We observe that GAN-0-GP behaves similar to Wasserstein GAN as it ﬁrst learns the overall structure of the distribution and then focuses on the modes.

Original: An evolution sequence of GAN-0-GP is shown in Fig.
Revised: An evolution sequence of GAN-0-GP is shown in Fig.

Original: 5 in appendix D. Results on other synthetic datasets are shown in appendix D.
Revised: 5 in appendix D. Results on other synthetic datasets are shown in appendix D.

Original: MNIST DATASET
Revised: MNIST DATASET

Original: The result on MNIST dataset is shown in Fig.
Revised: The result on MNIST dataset is shown in Fig.

Original: After 1,000 iterations, all other GANs exhibit mode collapse or cannot learn anything.
Revised: After 1,000 iterations, all other GANs exhibit mode collapse or cannot learn anything.

Original: GAN-0-GP is robust to changes in hyper parameters such as learning rate and optimizers.
Revised: GAN-0-GP is robust to changes in hyper parameters such as learning rate and optimizers.

Original: When Adam is initialized with large β 1 , e.g. 0 .
Revised: When Adam is initialized with large β 1 , e.g. 0 .

Original: 9 , GANs with other
Revised: 9 , GANs with other

Original: GPs cannot learn anything after many iterations.
Revised: GPs cannot learn anything after many iterations.

Original: More samples are given in appendix D.
Revised: More samples are given in appendix D.

Original: We observe that higher value of λ improves the diversity of generated samples.
Revised: We observe that higher value of λ improves the diversity of generated samples.

Original: For λ = 50 , we observe some similar looking samples in the generated data.
Revised: For λ = 50 , we observe some similar looking samples in the generated data.

Original: This is consistent with our conjecture that larger λ leads to better generalization.
Revised: This is consistent with our conjecture that larger λ leads to better generalization.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: I MAGE N ET
Revised: I MAGE N ET

Original: When trained on ImangeNet (Deng et al., 2009)
Revised: When trained on ImangeNet (Deng et al., 2009)

Original: GAN-0-GP can produce high quality samples from all 1,000 classes.
Revised: GAN-0-GP can produce high quality samples from all 1,000 classes.

Original: We compared our method with GAN with 0-GP-sample and WGAN-GP.
Revised: We compared our method with GAN with 0-GP-sample and WGAN-GP.

Original: GAN-0GP-sample is able to produce samples of state of the art quality without using progressive growing trick (Karras et al., 2018).
Revised: GAN-0GP-sample is able to produce samples of state of the art quality without using progressive growing trick (Karras et al., 2018).

Original: The result in Fig.
Revised: The result in Fig.

Original: 4 shows that our method consistently outperforms
Revised: 4 shows that our method consistently outperforms

Original: GAN-0-GP-sample.
Revised: GAN-0-GP-sample.

Original: GAN-0-GP and GAN-0-GP-sample outperform WGAN-GP by a large margin.
Revised: GAN-0-GP and GAN-0-GP-sample outperform WGAN-GP by a large margin.

Original: Image samples are given in appendix D.
Revised: Image samples are given in appendix D.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we clarify the reason behind the poor generalization capability of GAN.
Revised: In this paper, we clarify the reason behind the poor generalization capability of GAN.

Original: We show that the original GAN loss does not guide the discriminator and the generator toward a generalizable equilibrium.
Revised: We show that the original GAN loss does not guide the discriminator and the generator toward a generalizable equilibrium.

Original: We propose a zero-centered gradient penalty which pushes empirical discriminators toward the optimal discriminator with good generalization capability.
Revised: We propose a zero-centered gradient penalty which pushes empirical discriminators toward the optimal discriminator with good generalization capability.

Original: Our gradient penalty provides better generalization and convergence guarantee than other gradient penalties.
Revised: Our gradient penalty provides better generalization and convergence guarantee than other gradient penalties.

Original: Experiments on diverse datasets verify that our method signiﬁcantly improves the generalization and stability of
Revised: Experiments on diverse datasets verify that our method signiﬁcantly improves the generalization and stability of

Original: GANs.
Revised: GANs.

Original: A P ROOF FOR P ROPOSITION
Revised: A P ROOF FOR P ROPOSITION

Original: For continuous random variable V , P ( V = v ) = 0 for any v .
Revised: For continuous random variable V , P ( V = v ) = 0 for any v .

Original: The probability of ﬁnding a noise vector z such that G ( z ) is exactly equal to a real datapoint x ∈ D r via random sampling is 0.
Revised: The probability of ﬁnding a noise vector z such that G ( z ) is exactly equal to a real datapoint x ∈ D r via random sampling is 0.

Original: Therefore the probability of a real datapoint x i being in the fake dataset D g is 0.
Revised: Therefore the probability of a real datapoint x i being in the fake dataset D g is 0.

Original: Similarly, the probability of any fake datapoint being in the real dataset is 0.
Revised: Similarly, the probability of any fake datapoint being in the real dataset is 0.

Original: [Equation]
Revised: [Equation]

Original: [Equation] Furthermore, due to the curse of dimensionality the probability of sampling a datapoint which is close to another datapoint in high dimensional space also decrease exponentially.
Revised: Furthermore, due to the curse of dimensionality, the probability of sampling a datapoint which is close to another datapoint in high dimensional space also decrease exponentially.

Original: The distances between datapoints are larger in higher dimensional space.
Revised: The distances between datapoints are larger in higher dimensional space.

Original: That suggests that it is easier to separate
Revised: That suggests that it is easier to separate

Original: D r and D ( t ) g in higher dimensional space.
Revised: D r and D ( t ) g in higher dimensional space.

Original: B C ONSTRUCTING (cid:15) - OPTIMAL DISCRIMINATORS
Revised: B C ONSTRUCTING (cid:15) - OPTIMAL DISCRIMINATORS

Original: To make the construction process simpler, let’s assume that samples are normalized:
Revised: To make the construction process simpler, let’s assume that samples are normalized:

Original: [Equation]
Revised: [Equation]

Original: Let’s use the following new notations for real and fake samples:
Revised: Let’s use the following new notations for real and fake samples:

Original: [Equation]
Revised: [Equation]

Original: We construct the (cid:15) -optimal discriminator D as a MLP with 1 hidden layer.
Revised: We construct the (cid:15) -optimal discriminator D as a MLP with 1 hidden layer.

Original: Let W 1 ∈ R ( m + n ) × d x and W 2 ∈ R m + n be the weight matrices of D .
Revised: Let W 1 ∈ R ( m + n ) × d x and W 2 ∈ R m + n be the weight matrices of D .

Original: The total number of parameters in D is d x ( m + n ) + ( m + n ) =
Revised: The total number of parameters in D is d x ( m + n ) + ( m + n ) =

Original: O ( d x ( m + n )) .
Revised: O ( d x ( m + n )) .

Original: We set the value of W 1 as
Revised: We set the value of W 1 as

Original: [Equation]
Revised: [Equation]

Original:  and W 2 as
Revised: and W 2 as

Original: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: Given an input v ∈ D , the output is computed as:
Revised: Given an input v ∈ D , the output is computed as:

Original: [Equation]
Revised: [Equation]

Original: where σ is the softmax function.
Revised: where σ is the softmax function.

Original: Let a = W 1 v , we have
Revised: Let a = W 1 v , we have

Original: [Equation]
Revised: [Equation]

Original: As k → ∞ , σ ( W 1 v i ) becomes a one-hot vector with the i -th element being 1, all other elements being 0.
Revised: As k → ∞ , σ ( W 1 v i ) becomes a one-hot vector with the i -th element being 1, all other elements being 0.

Original: Thus, for large enough k , for any v j ∈ D , the output of the network is
Revised: Thus, for large enough k , for any v j ∈ D , the output of the network is

Original: D ( v j ) =
Revised: D ( v j ) =

Original: W (cid:62) 2 σ ( W 1 v j )
Revised: W (cid:62) 2 σ ( W 1 v j )

Original: ≈ W 2 ,j = (cid:26) >
Revised: ≈ W 2 ,j = (cid:26) >

Original: 12 + (cid:15) 2 , for j = 1 , ..., n < 12 − (cid:15) 2 , for j = n + 1 , ..., n + m
Revised: 12 + (cid:15) 2 , for j = 1 , ..., n < 12 − (cid:15) 2 , for j = n + 1 , ..., n + m

Original: D is a (cid:15) -optimal discriminator for dataset D .
Revised: D is a (cid:15) -optimal discriminator for dataset D .

Original: C F IXED -A LT -GD CANNOT MAINTAIN THE OPTIMALITY OF (cid:15) - DISCRIMINATORS
Revised: C F IXED -A LT -GD CANNOT MAINTAIN THE OPTIMALITY OF (cid:15) - DISCRIMINATORS

Original: Let’s consider the case where the real and the fake dataset each contain a single datapoint D r = { x } ,
Revised: Let’s consider the case where the real and the fake dataset each contain a single datapoint D r = { x } ,

Original: D ( t ) g
Revised: D ( t ) g

Original: = y ( t ) , and the discriminator and the generator are linear:
Revised: = y ( t ) , and the discriminator and the generator are linear:

Original: [Equation]
Revised: [Equation]

Original: and the objective is also linear (Wasserstein GAN’s objective):
Revised: and the objective is also linear (Wasserstein GAN’s objective):

Original: [Equation]
Revised: [Equation]

Original: The same learning rate α is used for D and G .
Revised: The same learning rate α is used for D and G .

Original: At step t , the discriminator is (cid:15) -optimal
Revised: At step t , the discriminator is (cid:15) -optimal

Original: [Equation]
Revised: [Equation]

Original: The gradients w.r.t.
Revised: The gradients w.r.t.

Original: θ D and θ G are
Revised: θ D and θ G are

Original: [Equation]
Revised: [Equation]

Original: If the learning rate α is small enough, (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13) should decrease as t increases.
Revised: If the learning rate α is small enough, (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13) should decrease as t increases.

Original: As the empirical fake distribution converges to the empirical real distribution, (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13) → 0 .
Revised: As the empirical fake distribution converges to the empirical real distribution, (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13) → 0 .

Original: The norm of gradient w.r.t.
Revised: The norm of gradient w.r.t.

Original: θ D , therefore, decreases as t increases and vanishes when the two empirical distributions are the same.
Revised: θ D , therefore, decreases as t increases and vanishes when the two empirical distributions are the same.

Original: From Eqn.
Revised: From Eqn.

Original: 10, we see that, in order to maintain D ’s (cid:15) -optimality when (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13) decreases, (cid:107) θ D (cid:107) has to increase.
Revised: 10, we see that, in order to maintain D ’s (cid:15) -optimality when (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13) decreases, (cid:107) θ D (cid:107) has to increase.

Original: From Eqn.
Revised: From Eqn.

Original: 10 and 12, we see that the gradient w.r.t.
Revised: 10 and 12, we see that the gradient w.r.t.

Original: θ G grows as the two empirical distributions are more similar.
Revised: θ G grows as the two empirical distributions are more similar.

Original: As (cid:13)(cid:13) x − y ( t ) [Equation] →
Revised: As (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13) →

Original: [Equation]
Revised: [Equation]

Original: Because the same learning rate α is used for both G and D , G will learn much faster than D .
Revised: Because the same learning rate α is used for both G and D , G will learn much faster than D .

Original: Furthermore, because (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13) decreases as t increases, the difference (cid:15)
Revised: Furthermore, because (cid:13)(cid:13) x − y ( t ) (cid:13)(cid:13) decreases as t increases, the difference (cid:15)

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: increases with t .
Revised: increases with t .

Original: The number of gradient steps that D has to take to reach the next (cid:15) -optimal state increases, and goes to inﬁnity as (cid:13)(cid:13)
Revised: The number of gradient steps that D has to take to reach the next (cid:15) -optimal state increases, and goes to inﬁnity as (cid:13)(cid:13)

Original: x − y ( t ) (cid:13)(cid:13)
Revised: x − y ( t ) (cid:13)(cid:13)

Original: → 0 .
Revised: → 0 .

Original: Therefore, gradient descent with ﬁxed number of updates to θ D cannot maintain the optimality of D .
Revised: Therefore, gradient descent with ﬁxed number of updates to θ D cannot maintain the optimality of D .

Original: The derivation for the objective in Eqn. 1 is similar.
Revised: The derivation for the objective in Eqn. 1 is similar.

Original: D R ESULTS ON DIFFERENT DATASETS
Revised: D R ESULTS ON DIFFERENT DATASETS

Original: [Figure]
Revised: [Figure]

Original: E I MPLEMENTATION DETAILS
Revised: E I MPLEMENTATION DETAILS

Original: We used Pytorch (Paszke et al., 2017) for development.
Revised: We used Pytorch (Paszke et al., 2017) for development.

Original: (a) GAN-0-GP    2 2.0 1.
Revised: (a) GAN-0-GP    2 2.0 1.

Original: (b) GAN-1-GP
Revised: (b) GAN-1-GP

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E. S YNTHETIC AND MNIST DATASETSGenerator architecture in synthetic and MNIST experiments
Revised: E. S YNTHETIC AND MNIST DATASETS

Revised: Generator architecture in synthetic and MNIST experiments

Original: Fully connected layer 2 × nhidden ↓
Revised: Fully connected layer 2 × nhidden ↓

Original: ReLU ↓
Revised: ReLU ↓

Original: Fully connected layer nhidden × nhidden ↓
Revised: Fully connected layer nhidden × nhidden ↓

Original: ReLU ↓
Revised: ReLU ↓

Original: Fully connected layer nhidden × nhidden ↓
Revised: Fully connected layer nhidden × nhidden ↓

Original: Fully connected layer nhidden ×
Revised: Fully connected layer nhidden ×

Original: Discriminator
Revised: Discriminator

Original: architecture in synthetic and
Revised: architecture in synthetic and

Original: MNIST experiments
Revised: MNIST experiments

Original: Fully connected layer 2 × nhidden ↓
Revised: Fully connected layer 2 × nhidden ↓

Original: ReLU ↓
Revised: ReLU ↓

Original: Fully connected layer nhidden × nhidden ↓
Revised: Fully connected layer nhidden × nhidden ↓

Original: ReLU ↓
Revised: ReLU ↓

Original: Fully connected layer nhidden × nhidden ↓
Revised: Fully connected layer nhidden × nhidden ↓

Original: Fully connected layer nhidden ×↓
Revised: Fully connected layer nhidden ×↓

Original: Sigmoid
Revised: Sigmoid

Original: [Table]
Revised: [Table]

Original: Learning rate 0.003 for both G and D
Revised: Learning rate 0.003 for both G and D

Original: Learning rate TTUR 0.003 for G , 0.009 for D
Revised: Learning rate TTUR 0.003 for G , 0.009 for D

Original: E. I MAGE N ET
Revised: E. I MAGE N ET

Original: The entire ImageNet dataset with all 1000 classes was used in the experiment.
Revised: The entire ImageNet dataset with all 1000 classes was used in the experiment.

Original: Because of our hardware limits, we used images of size 64 × 64 .
Revised: Because of our hardware limits, we used images of size 64 × 64 .

Original: We used the code from Mescheder et al.
Revised: We used the code from Mescheder et al.

Original: (2018), available at https://github . com/LMescheder/GAN stability , for our experiment.
Revised: (2018), available at https://github . com/LMescheder/GAN stability , for our experiment.

Original: Generator and Discriminator are ResNets, each contains 5 residual blocks.
Revised: Generator and Discriminator are ResNets, each contains 5 residual blocks.

Original: All GANs in our experiment have the same architectures and hyper parameters.
Revised: All GANs in our experiment have the same architectures and hyper parameters.

Original: The conﬁguration for WGAN-GP5 is as follows.
Revised: The conﬁguration for WGAN-GP5 is as follows.

Original: generator:
Revised: generator:

Original: name:resnet kwargs: nfilter: nfilter_max:embed_size: discriminator
Revised: name:resnet kwargs: nfilter: nfilter_max:embed_size:discriminator

Original: : name:resnet kwargs nfilter:nfilter_max:embed_size:
Revised: name:resnet kwargs: nfilter:nfilter_max:embed_size:z

Original: z_dist: type: gauss dim:training: out_dir: ../output/imagenet_wgangp5_TTUR gan_type: wgan reg_type: wgangp reg_param: 10.
Revised: dist type gauss dim:training: out_dir: ../output/imagenet_wgangp5_TTUR gan_type: wgan reg_type: wgangp reg_param: 10.

Original: batch_size:nworkers:take_model_average: true model_average_beta: 0.
Revised: batch_size:nworkers:take_model_average: true model_average_beta: 0.

Original: model_average_reinit: false monitoring: tensorboard sample_every:sample_nlabels:inception_every:
Revised: model_average_reinit: false monitoring: tensorboard sample_every:sample_nlabels:inception_every:

Original: save_every:backup_every:restart_every: - optimizer: adam lr_g: 0.
Revised: save_every:backup_every:restart_every: - optimizer: adam lr_g: 0.

Original: lr_d: 0.
Revised: lr_d: 0.

Original: lr_anneal: 1. lr_anneal_every:d_steps:equalize_lr: false
Revised: lr_anneal: 1. lr_anneal_every:d_steps:equalize_lr: false

Original: F F INDING A BETTER PATH BETWEEN A PAIR OF SAMPLES
Revised: F F INDING A BETTER PATH BETWEEN A PAIR OF SAMPLES

Original: Because the set of real data is unlikely to be convex, a linear interpolation between two datapoints is unlikely to be in the set.
Revised: Because the set of real data is unlikely to be convex, a linear interpolation between two datapoints is unlikely to be in the set.

Original: For example, the weighted average of two real images is often not a real image.
Revised: For example, the weighted average of two real images is often not a real image.

Original: If the generator is good enough then its output is likely to be on the real data manifold.
Revised: If the generator is good enough then its output is likely to be on the real data manifold.

Original: A path C from y to x can be found by transforming the straight line between the two corresponding latent codes, to the data space using the generator G .
Revised: A path C from y to x can be found by transforming the straight line between the two corresponding latent codes, to the data space using the generator G .

Original: To get the latent code of real datapoints, we use an encoder E which is trained to map real data to normally distributed latent codes.
Revised: To get the latent code of real datapoints, we use an encoder E which is trained to map real data to normally distributed latent codes.

Original: The pseudo code of the path ﬁnding algorithm is shown in Alg. 1.
Revised: The pseudo code of the path ﬁnding algorithm is shown in Alg. 1.

Original: Data: an encoder E ; a pair of samples x , y =
Revised: Data: an encoder E ; a pair of samples x , y =

Original: G ( z ) ;
Revised: G ( z ) ;

Original: Result: interpolated datapoint ˜ x
Revised: Result: interpolated datapoint ˜ x

Original: 1 Get the latent code of x : z x = E ( x ) 2 Calculate the interpolated latent code: ˜
Revised: 1 Get the latent code of x : z x = E ( x ) 2 Calculate the interpolated latent code: ˜

Original: z = α z x + (1 − α )
Revised: z = α z x + (1 − α )

Original: z 3 Generate the interpolated datapoint: ˜ x
Revised: z 3 Generate the interpolated datapoint: ˜ x

Original: = G (˜ z )
Revised: = G (˜ z )