Original: D ISTRIBUTIONAL C ONCAVITY R EGULARIZATION FOR GAN S
Revised: D ISTRIBUTIONAL C ONCAVITY R EGULARIZATION FOR GAN S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose Distributional Concavity (DC) regularization for GANs, a functional gradient-based method that promotes the entropy of the generator distribution and works against mode collapse.
Revised: We propose Distributional Concavity (DC) regularization for Generative Adversarial Networks (GANs), a functional gradient-based method that promotes the entropy of the generator distribution and works against mode collapse.

Original: Our DC regularization is an easy-to-implement method that can be used in combination with the current state of the art methods like Spectral Normalization and WGAN-GP to further improve the performance.
Revised: Our DC regularization is an easy-to-implement method that can be used in combination with the current state of the art methods like Spectral Normalization and Wasserstein GAN with gradient penalty to further improve the performance.

Original: We will not only show that our DC regularization can achieve highly competitive results on ImageNet and CIFAR datasets in terms of Inception score and FID score, but also provide a mathematical guarantee that our method can always increase the entropy of the generator distribution.
Revised: We will not only show that our DC regularization can achieve highly competitive results on ILSVRC2012 and CIFAR datasets in terms of Inception score and Fr ´ echet inception distance, but also provide a mathematical guarantee that our method can always increase the entropy of the generator distribution.

Original: We will also show an intimate theoretical connection between our method and the theory of optimal transport.
Revised: We will also show an intimate theoretical connection between our method and the theory of optimal transport.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: GANs (Goodfellow et al., 2014)  has been successful in numerous applications including image and movie generations (Isola et al., 2017; Zhu et al., 2017;
Revised: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) is a model consisting of two adversarial neural networks designed for the training of a generator distribution that mimics a target distribution deﬁned on a ( often ) high dimensional space, and it has been successful in numerous applications including image and movie generations (Isola et al., 2017; Zhu et al., 2017;

Original: Saito et al., 2017).
Revised: Saito et al., 2017).

Original: However, it has not yet completely established itself as a wieldly scientiﬁc tool because of the sheer computational difﬁculty of its training process.
Revised: However, it has not yet completely established itself as a wieldly scientiﬁc tool because of the sheer computational difﬁculty of its training process.

Original: As such, there has been numerous studies seeking for a way to stabilize the training process of GANs (Gulrajani et al., 2017;
Revised: As such, there has been numerous studies seeking for a way to stabilize the training process of GANs (Gulrajani et al., 2017;

Original: Miyato et al., 2018; Karras et al., 2018).
Revised: Miyato et al., 2018; Karras et al., 2018).

Original: Mode collapse is a particularly well known central problem for the training of  GANs. 
Revised: Mode collapse is a persistent central problem for the training of GANs, which collectively refers to the lack of diversity in generator distribution. For instance, without any countermeasure, GANs applied to multimodal mixture of Gaussians will often train a generator distribution with one mode (See Fig 22, Goodfellow (2016) for example). Mode collapse has been discussed in numerous literatures related GANs. To name a few, see Goodfellow (2016); Metz et al.

Original: Mode collapseis a problem that can be seen as a case of entropy degeneration.
Revised: ; Arjovsky & Bottou (2017); Arjovsky et al. ; Lin et al. (2017). In words of statistical machine learning, mode collapse can be described as a case of entropy degeneration.

Original: A naive countermeasure against mode collapse is therefore to augment the entropy of the generator distribution.
Revised: A naive countermeasure against mode collapse is therefore to augment the entropy of the generator distribution.

Original: Not many studies to date, however, tackled the problem of mode collapse using this direct approach.
Revised: Not many studies to date, however, tackled the problem of mode collapse using this direct approach.

Original: Dai et al.
Revised: Dai et al.

Original: (2017) partially realized this idea by actually evaluating the entropy of the generator distribution  and including it into the objective function.
Revised: (2017) partially realized this idea by actually evaluating the entropy of the generator distribution with variational inference and including it into the objective function.

Original: Precise estimation of  the entropy is, however, computationally difﬁcult, and not much further improvement can be expected from this strategy.
Revised: This type of strategy requires the user to continuously produce reliable estimates of the generators based on ﬁnite samples throughout the course of the training. As such, not much further improvement can be expected from this strategy, because precise estimation of the entropy tends to be computationally heavy, and empirical estimation is an excruciatingly difﬁcult problem on its own in high dimension, as is the case in image and movie generation. In fact, the performance of classical methods based on kernel density estimation, for example, does not scale well with respect to dimension, and the number of samples required to control the MSE can grow exponentially with the dimension (Cacoullos, 1966; Ozakin & Gray, 2009).

Original: In this study, we use the theory of functional gradient to develop a method that can promote the entropy of the generator distribution without directly estimating the entropy itself.
Revised: In this study, we use the theory of functional gradient to develop a method that can promote the entropy of the generator distribution without directly estimating the entropy itself.

Original: The functional gradient of an objective function with respect to a model function is an inﬁnite dimensional gradient computed over the inﬁnite dimensional space of all models.
Revised: The functional gradient of an objective function with respect to a model function is an inﬁnite dimensional gradient computed over the inﬁnite dimensional space of all models.

Original: As we will show, all variations of
Revised: As we will show, all variations of

Original: GANs to date are implicitly using the discriminator function to compute the functional derivative of the objective function with respect to the generator distribution function.
Revised: GANs to date are implicitly using the discriminator function to compute the functional derivative of the objective function with respect to the generator distribution function.

Original: In other words, the disrciminator determines the direction to which the algorithm should move the mass of the current generator distribution.
Revised: In other words, the disrciminator determines the direction to which the algorithm should move the mass of the current generator distribution.

Original: The work of Nitanda & Suzuki (2018) is a pioneer study that explicitly incorporated this idea into the training of neural generator models.
Revised: The work of Nitanda & Suzuki (2018) is a pioneer study that explicitly incorporated this idea into the training of neural generator models.

Original: Their study showed that one can carry out a faithful functional gradient-based update by inserting what they call gradient layer into the layers of neural generator function.
Revised: Their study showed that one can carry out a faithful functional gradient-based update by inserting what they call gradient layer into the layers of neural generator function.

Original: Johnson & Zhang (2018) further polished this strategy by periodically distilling the networks.
Revised: Johnson & Zhang (2018) further polished this strategy by periodically distilling the networks.

Original: One advantage of the functional gradient-based approach is that, at every stage in the training process of the generator distribution, the user can monitor what the next (target) distribution looks like in terms of the current distribution.
Revised: One advantage of the functional gradient-based approach is that, at every stage in the training process of the generator distribution, the user can monitor what the next (target) distribution looks like in terms of the current distribution.

Original: In contrast, the target distribution set forth by conventional parametric methods is usually expressed in a complicated parametric form (e.g. DNNs), and its behaviors are often difﬁcult to predict.
Revised: In contrast, the target distribution set forth by conventional parametric methods is usually expressed in a complicated parametric form (e.g. Deep Neural Nets (DNNs)), and its behaviors are often difﬁcult to predict.

Original: This advantage of the functional gradient based-update suggests the possibility that we can control the update rule in order to deliberately direct the generator distribution to a distribution with preferred properties–which, in our case, include high entropy .
Revised: This advantage of the functional gradient based-update suggests the possibility that we can control the update rule in order to deliberately direct the generator distribution to a distribution with preferred properties–which, in our case, include high entropy .

Original: We discovered that, by locally concavifying the discriminator function, we can manipulate the funcational gradient so that the next update for the generator will always target a distribution with higher entropy.
Revised: We discovered that, by locally concavifying the discriminator function, we can manipulate the funcational gradient so that the next update for the generator will always target a distribution with higher entropy.

Original: From now on, we should refer to our method by Distributional Concavity (DC) regularization .
Revised: From now on, we should refer to our method by Distributional Concavity (DC) regularization .

Original: Our method does not require the direct estimation of the entropy.
Revised: Our method does not require the direct estimation of the entropy.

Original: We will not only show that our DC regularization can help improve the performance of GANs in terms of Inception score  and FID score, but also give a mathematical guarantee that our regularization will always increase the entropy of the generator distribution. 
Revised: We will not only show that our DC regularization can help improve the performance of GANs in terms of Inception score (Salimans et al., 2016) and Fr´echet inception distance (FID) (Heusel et al., 2017), but also give a mathematical guarantee that our regularization will always increase the entropy of the generator distribution. We will also show that our method greatly outperforms the method of Dai et al.

Revised: (2017) that directly estimates the entropy based on classicaly techniques.

Original: The regularization strategy of monotnically increasing the entropy of the generator distribution over the course of its training is not without theoretical basis.
Revised: The regularization strategy of monotnically increasing the entropy of the generator distribution over the course of its training is not without theoretical basis.

Original: Our DC regularization has close relations to the theory of optimal transport.
Revised: Our DC regularization has close relations to the theory of optimal transport.

Original: We will show that, when the entropy of the true distribution is higher than the current generator distribution, the functional gradient that is properly derived from the optimal mass transport always increases the entropy of the distribution.
Revised: We will show that, when the entropy of the true distribution is higher than the current generator distribution, the functional gradient that is properly derived from the optimal mass transport always increases the entropy of the distribution.

Original: Moreover, the functional update used in our method is always a monotonic mapping, which also turns out to be one of the properties satisﬁed by the update with optimal transport.
Revised: Moreover, the functional update used in our method is always a monotonic mapping, which also turns out to be one of the properties satisﬁed by the update with optimal transport.

Original: This is a preferred property as well, because it tends to promote the smooth training of the generator.
Revised: This is a preferred property as well, because it tends to promote the smooth training of the generator.

Original: We summarize our contributions below:
Revised: We summarize our contributions below:

Original: • We propose an update for the generator of GANs that promotes the entropy of the generator without the need for an explicit estimation of the actual entropy.
Revised: • We propose an update for the generator of GANs that promotes the entropy of the generator without the need for an explicit estimation of the actual entropy.

Original: • We show that, when the entropy of the true distribution is larger than that of the current generator distribution, the functional gradient derived from the 2-Wasserstein ( W 2 ) optimal transport always increases the entropy of the distribution.
Revised: • We show that, when the entropy of the true distribution is larger than that of the current generator distribution, the functional gradient derived from the 2-Wasserstein ( W 2 ) optimal transport always increases the entropy of the distribution.

Original: • We provide a mathematical guarantee that our method increases the entropy of the generator distribution at every step.
Revised: • We provide a mathematical guarantee that our method increases the entropy of the generator distribution at every step.

Original: • We show that our method improves the results of GANs in terms of Inception score and FID score.
Revised: • We show that our method improves the results of GANs in terms of Inception score and FID score.

Original:  T HEORY 2. G ENERATIVE ADVERSARIAL NETWORKS
Revised:  T HEORY 2. G ENERATIVE ADVERSARIAL NETWORKS

Original: Let us ﬁrst review the formulation of the original GANs.
Revised: Let us ﬁrst review the formulation of the original GANs.

Revised: Unless otherwise noted, let us use boldface capital letters to refer to a random variable, and lower case letter for its realization.

Original: The training process of GANs (Goodfellow et al., 2014) is a two player min-max game in which the generator distribution µ θ is trained to minimize the divergence between the true distribution ν and the generator distribution µ θ measured by the current critic F , while the critic F is trained to most strongly discriminate µ θ from ν .
Revised: The training process of GANs (Goodfellow et al., 2014) is a two player min-max game in which the generator distribution µ θ is trained to minimize the divergence between the true distribution ν and the generator distribution µ θ measured by the current critic F , while the critic F is trained to most strongly discriminate µ θ from ν .

Original: Often, µ θ is produced by applying a generator function G to a random seed variable with some distribution µ z .
Revised: Often, µ θ is produced by applying a generator function G to a random seed variable with some distribution µ z .

Original: In a nutshell, GANs aims to ﬁnd optimal G ∗ that achieves the minimum value for
Revised: In a nutshell, GANs aims to ﬁnd optimal G ∗ that achieves the [Table]

Original: [Equation]
Revised: [Equation]

Original: where L r and L g are the functions of user’s choice.
Revised: where L r and L g are the functions of user’s choice.

Original: We can retrieve the formulation of the original GAN by letting L r ( F ( x ))
Revised: We can retrieve the formulation of the original GAN by letting L r ( F ( x ))

Original: =
Revised: =

Original: softplus ( − F ( x )) and L g ( F ( x ))
Revised: softplus ( − F ( x )) and L g ( F ( x ))

Original: =
Revised: =

Original: softplus ( F ( x )) .
Revised: softplus ( F ( x )) , where softplus ( · ) =

Original:  If we deﬁne the discriminator D ( x ) as sigmoid ( F ( x )) , the training of D with the equation (1) emerges as the training of a classiﬁer that distinguishes the samples of ν from the samples of µ θ .
Revised: log(1 + exp( · )) . If we deﬁne the discriminator D ( x ) as (1 + exp( − F ( x ))) − 1 , the training of D with the equation (1) emerges as the training of a classiﬁer that distinguishes the samples of ν from the samples of µ θ .

Original: It is known that the optimal D is given by D ∗ ( x ) = 
Revised: It is known that the optimal D is given by D ∗ ( x ) = [Equation] ( x [Equation]

Original: ν ( x ) ν ( x )+ µ θ ( x ) , and the problem of ﬁnding the optimal G ∗ in this case becomes the problem of ﬁnding the minimizer of Jensen Shannon divergence (Goodfellow et al., 2014).
Revised: x [Equation] + [Equation]

Revised: ( x

Revised: [Equation]

Revised: and the problem of ﬁnding the optimal G ∗ in this case becomes the problem of ﬁnding the minimizer of Jensen Shannon divergence (Goodfellow et al., 2014).

Original: There is a variety of choices for ( L r , L g )
Revised: There is a variety of choices for ( L r , L g )

Original: (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016).
Revised: (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016).

Original: 2. F UNCTIONAL G RADIENT INTERPRETATION OF THE GAN S UPDATE
Revised: 2. F UNCTIONAL G RADIENT INTERPRETATION OF THE GAN S UPDATE

Original: Next, we would provide an interpretation for the update rule of GANs from the perspective of functional gradient.
Revised: Next, we would provide an interpretation for the update rule of GANs from the perspective of functional gradient.

Revised: For ease of argument, we will assume the followings from here onward.

Original:  Let us  assume that G  is a parametric function of θ , and  consider the optimization problem of the equation (1) with gradient descent about θ .
Revised: First, as is the case in many application of GANs, we will assume that ν is a probability measure on the Euclidean space R d , and µ z is a probability measure on some Euclidean space U . Let us also assume that G : U → R d is a parametric function of θ that is almost everywhere differentiable with respect to both input and θ . We will ﬁrst consider the optimization problem of the equation (1) with gradient descent about θ .

Original: For ease of notation, let us write L =
Revised: For ease of notation, let us write L =

Original: L g ◦ F .
Revised: L g ◦ F .

Revised: We will assume that L is a continuously differrentiable and uniformly bounded function on R d , and that the Euclidean norms of its gradient and operator norm of Hessian are also uniformly bounded.

Revised: This type of regularity assumptions on the objective functions are used in other literatures as well (Nitanda & Suzuki (2018),

Revised: Johnson & Zhang (2018)), and it is not too all unrealistic because the support of both target distributions and generator distribution are often compact throughout the course of the training in the real applications.

Revised: We will also assume that the derivatives of all functions to appear are well deﬁned and bounded on R d .

Original: The update rule  is given by:
Revised: The update rule of θ is given by:

Original: [Equation]
Revised: [Equation]

Revised: where D θ represents the derivative operator with respect to θ .

Original: In general, almost all variations of GANs to date (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016) uses this type of update rule for  training of the generator.
Revised: In general, almost all variations of GANs to date (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016) uses this type of update rule for the training of the generator.

Original: Let us denote the law of G ( Z ; θ ) (or X ) by µ θ .
Revised: Let us denote the law of G ( Z ; θ ) (or X ) by µ θ .

Revised: Let L 2 ( µ θ ) denote the Hilbert space of L 2 integrable maps from R d to itself, granted with the inner product (cid:104) a, b (cid:105) µ θ = E µ θ [ a ( X ) T b ( X )] .

Original: We will show that we can re-derive the equation (3) using the theory of functional gradient.
Revised: We will show that we can re-derive the equation (3) using the theory of functional gradient.

Original: To begin with, instead of updating the parameter θ , we would like to consider directly updating the distribution µ θ by applying atransformation T  to a random variable generated from µ θ .
Revised: To begin with, instead of updating the parameter θ , we would like to consider directly updating the distribution µ θ by applying a T ∈ L 2 ( µ θ ) to a random variable generated from µ θ .

Original: This will result in a new distribution T # µ θ , which is a unique distribution that satisﬁes Eµ θ [ h ( T ( X )]
Revised: This will result in a new distribution T # µ θ , which is a unique distribution that satisﬁes E

Original: = E T # µ θ
Revised: [Equation]

Original: [ h ( X )] for all measurable h .
Revised: ] for all measurable h .

Original: Now, by using the functional gradient of E T # µ θ [ L ( X )] =
Revised: Now, in order to create an update rule for T , let us consider the Gˆateaux derivative of M ( T ) :=

Original: E  µ z
Revised: E T # µ θ

Original: [ L ( T ( G θ ( Z ))] with respect to T .
Revised: [ L ( X )] with respect to T into an arbitrary direction

Original:  we would like to search for a choice of  T that reduces E T # µ θ [ L ( X )] .
Revised: δ ∈ L 2 ( µ θ ) . In terse term, we would like to consider the effect of purturbing T into the direction of δ .

Original: If δ is a random functional perturbation,
Revised: By the compactnesss assumption on Ω together with the regularity assumptions for the functions deﬁned hereof, we can appeal to the dominated convergence theorem and see that

Original: [Equation]
Revised: [Equation]

Original:  This ’directional’ functional derivative  will always take a positive value when δ ∝ ∇ L ( T ( X )) .
Revised: is uniform for all δ ∈ L 2 ( µ ) . This way, the term ∇ L ( T ( · )) in the expression above is a Fr´echet derivative, or the type of derivative to which the usual set of derivative rules can be applied. We will refer to this derivative as functional derivative for short. Note that the directional functional derivative E µ θ [( ∇ L ( T ( X )) · δ )( · )] will always take a positive value when δ ∝ ∇ L ( T ( · )) .

Original: Thus, for α small enough, an update from the current choice of T to T − α ∇ L  will decrease the objective function.
Revised: Thus, for α small enough, an update from the current choice of T to T − α ∇ L ∈ L 2 ( µ θ ) will decrease the objective function.

Original: Here, we are interested in the derivative computed at T
Revised: Here, we are interested in the derivative computed at T

Original: = Id , so the transformation we would like to apply to µ θ is
Revised: = Id , so the transformation we would like to apply to µ θ is

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This is indeed a transportation of a mass into the direction of α ∇ L .
Revised: This is indeed a transportation of a mass into the direction of α ∇ L .

Original: Now, for the update of θ , we would like to design θ new such that µ θ new is closer to the target distribution T # µ θ old than µ θ old .
Revised: Now, for the update of θ , we would like to design θ new such that µ θ new is closer to the target distribution T # µ θ old than µ θ old .

Original: That is, we would like to choose θ that minimizes
Revised: That is, we would like to choose θ that minimizes

Original: [Equation]
Revised: [Equation]

Original: The gradient of this sub-objective function with respect to θ is given by
Revised: The gradient of this sub-objective function with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: Evaluating this at θ = θ old , we recover the gradient used in the equation (3).
Revised: Evaluating this at θ = θ old , we recover the gradient used in the equation (3).

Original: This formulation is practically equivalent to the one introduced in xICFG (Johnson & Zhang, 2018).
Revised: This formulation is practically equivalent to the one introduced in xICFG (Johnson & Zhang, 2018).

Original: 2. C HOICE OF THE TARGET DISTRIBUTION
Revised: 2. C HOICE OF THE TARGET DISTRIBUTION

Original: From the perspective of functional gradient, the min-max game of GANs can be decomposed into (i) the target construction step that looks for a next target distribution in the vicinity of the current generator distribution that is closer to the true distribution than the current distribution, and (ii) the distillation step that looks for a neural function that better approximates the target distribution.
Revised: From the perspective of functional gradient, the min-max game of GANs can be decomposed into (i) the target construction step that looks for a next target distribution in the vicinity of the current generator distribution that is closer to the true distribution than the current distribution, and (ii) the distillation step that looks for a neural function that better approximates the target distribution.

Original: Our proposal is to design a target distribution T # µ θ with favorable properties.
Revised: Our proposal is to design a target distribution T # µ θ with favorable properties.

Original: In this study, we would particularly like to design T that promotes higher entropy and satisﬁes monotonicity condition, which we will describe momentarily.
Revised: In this study, we would particularly like to design T that promotes higher entropy and satisﬁes monotonicity condition, which we will describe momentarily.

Original: Indeed, the promotion of entropy shall discourages mode collapse.
Revised: Indeed, the promotion of entropy shall discourages mode collapse.

Original: Meanwhile, the monotonicity is a property that assures that there is no crossings in the transport:
Revised: Meanwhile, the monotonicity is a property that assures that there is no crossings in the transport:

Original: [Equation]
Revised: [Equation]

Original: This is a preferred property because, as we will empirically show (see section 4.1) , the crossings in the transport tend to hinder the smooth training process.
Revised: This is a preferred property because, as we will empirically show (see section 4.1) , the crossings in the transport tend to hinder the smooth training process.

Original: This is in fact somewhat intuitive, because crossings lead to wasteful transportation of the mass.
Revised: This is in fact somewhat intuitive, because crossings lead to wasteful transportation of the mass.

Original: As it turns out, we can satisfy both of these conditions by simply requiring L to be concave.
Revised: As it turns out, we can satisfy both of these conditions by simply requiring L to be concave.

Original: Proposition 2.1.
Revised: Proposition 2.1.

Original: Let µ be a probability distribution.
Revised: Let µ be a probability distribution on R d .

Original: If L is concave on the support of µ , then H ( T # µ ) ≥ H ( µ ) .
Revised: If L is concave on the support of µ , then H ( T # µ ) ≥ H ( µ ) .

Original: Note that this statement is independent of the step size, because any positive scalar multiple of concave function is concave.
Revised: Note that this statement is independent of the step size, because any positive scalar multiple of concave function is concave.

Original: Also, by deﬁnition, the concavity of L implies the strong convexity of
Revised: Also, by deﬁnition, the concavity of L implies the strong convexity of

Original: T , and the strong convexity of T implies
Revised: T , and the strong convexity of T implies

Original: [Equation]
Revised: [Equation]

Original: (8) which is a stronger condition than the monotonicity.
Revised: which is a stronger condition than the monotonicity.

Original: Thus, by simply making L concave, we can not only construct a target distribution whose entropy is greater than that of the current generatator distribution, but also assure that the corresponding transport is monotonic.
Revised: Thus, by simply making L concave, we can not only construct a target distribution whose entropy is greater than that of the current generatator distribution, but also assure that the corresponding transport is monotonic.

Original: The proposal we would like to make in this study is simply to choose a distribution constructed from concave L in the target construction step.
Revised: The proposal we would like to make in this study is simply to choose a distribution constructed from concave L in the target construction step.

Original: 2. R ELATION TO OPTIMAL TRANSPORT
Revised: 2. R ELATION TO OPTIMAL TRANSPORT

Original: Our DC regularization has close relations to the theory of optimal transport.
Revised: Our DC regularization has close relations to the theory of optimal transport.

Original: In fact, an equation of form 5 is ubiquitous in the theory of optimal transport.
Revised: In fact, an equation of form 5 is ubiquitous in the theory of optimal transport.

Original: In order to further articulate this point, we would like to brieﬂy introduce the notion of optimal transport.
Revised: In order to further articulate this point, we would like to brieﬂy introduce the notion of optimal transport.

Original: For more rigorous introduction of the concept, please consult (Villani, 2008).
Revised: For more rigorous introduction of the concept, please consult (Villani, 2008).

Original: For p ≥ 1 , the p -Wasserstein distance W p ( µ, ν ) between two arbitrary distributions µ and ν with sufﬁcient regularity is given by
Revised: For p ≥ 1 , the p -Wasserstein distance W p ( µ, ν ) between two arbitrary distributions µ and ν with sufﬁcient regularity is given by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the inf above is taken over all  maps T satisfying T # µ = ν .
Revised: where the inf above is taken over all measurable maps T satisfying T # µ = ν .

Original: When p = 2 , it is known that the inﬁmum of this cost function is achieved by T ∗ that can be expressed as x − ∇ L ∗ for some L ∗ that renders (cid:107) x (cid:107) 2− L ∗ convex (Brenier, 1991).
Revised: When p = 2 , it is known that the inﬁmum of this cost function is achieved by T ∗ that can be expressed as x −∇ L ∗ for some L ∗ that renders (cid:107) x (cid:107) 2 / 2 − L ∗ convex (Brenier, 1991).

Original: The search for L ∗ and hence T ∗ therefore amounts to the search for the closest coupling between µ and ν in the L 2 sense.
Revised: The search for L ∗ and hence T ∗ therefore amounts to the search for the closest coupling between µ and ν in the L 2 sense.

Original: One surprising fact is that the movement of the particle along the direction of ∇ L ∗ monotonically decreases W2 (Villani, 2008).
Revised: One surprising fact is that the movement of the particle along the direction of ∇ L ∗ monotonically decreases W (Villani, 2008).

Original: That is, if T ∗ t ( x ) = x
Revised: That is, if T ∗ t ( x ) = x

Original: − t ∇ L ∗ ( x ) , then W 2 ( T ∗ t # µ, ν ) monotonically decreases with t (Villani, 2003).
Revised: − t ∇ L ∗ ( x ) , then W 2 ( T ∗ t # µ, ν ) monotonically decreases with t (Villani, 2003).

Original: Please compare this transport equation with the equation (5).
Revised: Please compare this transport equation with the equation (5).

Original: This T ∗ t is indeed the optimal transportation analogue of the functional gradient.
Revised: This T ∗ t is indeed the optimal transportation analogue of the functional gradient.

Original: The theorem in Brenier (1991) has a still surprising converse; for any convex function L , T = x − ∇ L turns out to be the unique optimal transport from µ to T # µ .
Revised: The theorem in Brenier (1991) has a still surprising converse; for any convex function L , T = x − ∇ L turns out to be the unique optimal transport from µ to T # µ .

Original: Using this fact, we can appeal to the following proposition 2.4 to claim that the functional update based on the optimal transport satisﬁes one important property that shall be intuitively fulﬁlled if we are in fact moving the distribution toward the true distribution—that is, if the current distribution has lower entropy than the true distribution, the sequence of the distributions produced by the updates should be monotonically increasing in the entropy:
Revised: Using this fact, we can appeal to the following proposition 2.4 to claim that the functional update based on the optimal transport satisﬁes one important property that shall be intuitively fulﬁlled if we are in fact moving the distribution toward the true distribution—that is, if the current distribution has lower entropy than the true distribution, the sequence of the distributions produced by the updates should be monotonically increasing in the entropy:

Original: Proposition 2.2.
Revised: Proposition 2.2.

Original: Suppose ν = T # µ for some T that can be written as a gradient of a strictly convex function.
Revised: Suppose ν = T # µ for some T that can be written as a gradient of a strictly convex function.

Original: If H ( ν ) ≥ H ( µ ) , then H ( T t # µ ) is monotonically increasing on t ∈ [0 , 1] .
Revised: If H ( ν ) ≥ H ( µ ) , then H ( T t # µ ) is monotonically increasing on t ∈ [0 , 1] .

Original: This proposition actually assures that the updates of DC regularization also satisfy the same property.
Revised: This proposition actually assures that the updates of DC regularization also satisfy the same property.

Original: Because we are designing the target distribution so that it will have higher entropy than the current distribution at every step, by letting ν in the proposition 2.4 to be the target distribution, we have a guarantee that the sequence of distributions constructed by our updates is also monotnically increasing in the entropy.
Revised: Because we are designing the target distribution so that it will have higher entropy than the current distribution at every step, by letting ν in the proposition 2.4 to be the target distribution, we have a guarantee that the sequence of distributions constructed by our updates is also monotnically increasing in the entropy.

Original: To see this, simply note that we can automatically make (cid:107) x (cid:107) 2 / 2 − L to be convex by making L concave,
Revised: To see this, simply note that we can automatically make (cid:107) x (cid:107) 2 / 2 − L to be convex by making L concave,

Original: The connection between our DC regularization and optimal transport is not limited to the property we introduced above.
Revised: The connection between our DC regularization and optimal transport is not limited to the property we introduced above.

Original: By the theory of Brenir, the choice T α
Revised: By the theory of Brenir, the choice T α

Original: = Id − α 1 / 2 ∇ L with concave and 1-Lipshitz L satisﬁesW 2 ( µ, T α # µ ) =
Revised: = Id − α 1 / 2 ∇ L with concave and 1-Lipshitz L satisﬁes

Original: (cid:112) E [ (cid:107) α 1 / 2 ∇ L ( X ) (cid:107) 2 ] = α.
Revised: [Equation] = α.

Original: Put in still other words, by constructing a target distribution with concave L , we can also assure that the target distribution is contained within the W 2 neighborhood of the current distribution function.
Revised: Put in still other words, by constructing a target distribution with concave L , we can also assure that the target distribution is contained within the W 2 neighborhood of the current distribution function.

Original: This is a property that cannot be guaranteed with the conventional parameter-based updates.
Revised: This is a property that cannot be guaranteed with the conventional parameter-based updates.

Original: Monotonicity condition is also a property that is satisﬁed by the optimal transport.
Revised: Monotonicity condition is also a property that is satisﬁed by the optimal transport.

Original: According the theory of Monge-Ampere (Villani (2008)), a transport map can be optimal only if it is monotonic.
Revised: According the theory of Monge-Ampere (Villani (2008)), a transport map can be optimal only if it is monotonic.

Original: Thus, while not being exactly based on the optimal transport from the generator distribution to the true distribution, our T shares many favorable properties in common with the optimal transport, and is very closely related to the theory of W 2 distance.
Revised: Thus, while not being exactly based on the optimal transport from the generator distribution to the true distribution, our T shares many favorable properties in common with the optimal transport, and is very closely related to the theory of W 2 distance.

Original: Reﬂecting on these facts, one might become tempted to say that we shall simply formulate GANs with the objective function that is solely based on W 2 distance between the generator distribution and the true distribution.
Revised: Reﬂecting on these facts, one might become tempted to say that we shall simply formulate GANs with the objective function that is solely based on W 2 distance between the generator distribution and the true distribution.

Original: However, as we will further articulate in the discussion section, the challenge remains to conduct faithful W 2 -based updates in the implementation of GANs.
Revised: However, as we will further articulate in the discussion section, the challenge remains to conduct faithful W 2 -based updates in the implementation of GANs.

Original:  M ETHOD
Revised:  M ETHOD

Original: If µ θ := G θ # µ z is our current parametric generator distribution and ν is the true distribution,
Revised: If µ θ := G θ # µ z is our current parametric generator distribution and ν is the true distribution,

Original: our
Revised: our

Original: DC regularization method ﬁrst (i) proposes a target distribution T # µ θ using a concave L , and then (ii) seek a measure µ θ new that well approximates the target distribution T # µ θ .
Revised: DC regularization method ﬁrst (i) proposes a target distribution T # µ θ using a concave L , and then (ii) seek a measure µ θ new that well approximates the target distribution T # µ θ .

Original: We use the following sampling-based penalty term in order to promote the concavity of L :
Revised: We use the following sampling-based penalty term in order to promote the concavity of L :

Original: [Equation]
Revised: [Equation]

Original: where x 1 , x 2 are samples from the support of µ θ , (cid:15) is a sample from the uniform distribution over [0 , 1] , and d is a positive scalar.
Revised: where x 1 , x 2 are samples from the support of µ θ , (cid:15) is a sample from the uniform distribution over [0 , 1] , and d is a positive scalar.

Original: Note that this term must be positive if L is concave over the support of µ θ .
Revised: Note that this term must be positive if L is concave over the support of µ θ .

Original: Our algorithm is summarized in Algorithm 1.
Revised: Our algorithm is summarized in Algorithm 1.

Original: The update rule in this algorithm uses the target distribution constructed by T ( x ) = x − ∇ L with concave L .
Revised: The update rule in this algorithm uses the target distribution constructed by T ( x ) = x − ∇ L with concave L .

Original: Intuitively speaking, the transport T has an effect of moving µ θ toward T # µ θ while dispersing the mass of µ θ .
Revised: Intuitively speaking, the transport T has an effect of moving µ θ toward T # µ θ while dispersing the mass of µ θ .

Original: See Fig 1 for a visual rendering of this interpretation.
Revised: See Fig 1 for a visual rendering of this interpretation.

Original: In most practical application, the training of GANs begins by dispersing the initial distribution with small entropy and gradually molds the mass into what resembles the true distribution.
Revised: In most practical application, the training of GANs begins by dispersing the initial distribution with small entropy and gradually molds the mass into what resembles the true distribution.

Original: Our transport ensures that this dispersion is consistently happening over the course of the training.
Revised: Our transport ensures that this dispersion is consistently happening over the course of the training.

Original: As we will show in the result section, this regularization works in favor of the inception score without any noticeable downfall from over-dispersion , suggesting that the generator distribution created with the current state of the art techniques are still not dispersed enough .
Revised: As we will show in the result section, this regularization works in favor of the inception score without any noticeable downfall from over-dispersion , suggesting that the generator distribution created with the current state of the art techniques are still not dispersed enough .

Original: for each iteration do  the target construction step : update critic by critic’s objective
Revised: for each iteration do θ old ← θ the target construction step : update critic by critic’s objective

Original: [Equation]
Revised: [Equation]

Original: (11) the distillation step : update generator by generator’s objective
Revised: (11) the distillation step : update generator by generator’s objective

Original: [Equation]
Revised: [Equation]

Original: (12) end for
Revised: (12) end for

Original: [Figure]
Revised: [Figure]

Original: transport function will make the points move away from each other.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We applied DC regularization to the training of GANs on CIFAR-10 (Torralba et al., 2008), CIFAR100 (Torralba et al., 2008) and ILSVRC2012 dataset (ImageNet) (Russakovsky et al., 2015) in various settings and evaluated its performance in terms of Inception score (Salimans et al., 2016) and
Revised: We applied DC regularization to the training of GANs on CIFAR-10 (Torralba et al., 2008), CIFAR100 (Torralba et al., 2008) and ILSVRC2012 dataset (ImageNet) (Russakovsky et al., 2015) in various settings and evaluated its performance in terms of Inception score (Salimans et al., 2016) and

Original: Fr´echet inception distance (FID) (Heusel et al., 2017).
Revised: Fr´echet inception distance (FID) (Heusel et al., 2017).

Original: Inception score and FID are performance measures that are commonly used to evaluate the severity of mode collapse.
Revised: Inception score and FID are performance measures that are commonly used to evaluate the severity of mode collapse.

Original: For the details of the evaluation, see Appendix B.1.
Revised: For the details of the evaluation, see Appendix B.1.

Revised: We also conducted an additional set of experiments with artiﬁcal dataset to investigate the properties of DC-regularization.

Original: We provide the results for additional experiments in the Appendix C as well.
Revised: We provide the results for additional experiments in the Appendix C as well.

Original: 4. I NTRINSIC PROPERTIES OF DC REGULARIZATION Effect of DC regularization on entropy
Revised: 4. I NTRINSIC PROPERTIES OF DC REGULARIZATION Effect of DC regularization on entropy

Original: Using a simple GMM as the true distribution, we evaluated the sheer ability of the DC regularization to promote the entropy of the generator distribution.
Revised: Using a simple Gaussian Mixture Model as the true distribution, we evaluated the sheer ability of the DC regularization to promote the entropy of the generator distribution.

Original: We used hinge loss for the objective function, and used MLPs to model both the discriminator and the generator.
Revised: We used hinge loss for the objective function, and used DNNs to model both the discriminator and the generator.

Original: We trained the model with and without the DC regularization and reported the entropy of the Generator at different stages of the training by explicitly computing the determinant of the Jacobian at each layer.
Revised: We trained the model with and without the DC regularization and reported the entropy of the Generator at different stages of the training by explicitly computing the determinant of the Jacobian at each layer.

Original: The result is illustrated in Fig 2 (a).
Revised: The result is illustrated in Fig 2 (a).

Original: We see that the regularization is positively affecting the entropy at all stages of the training.
Revised: We see that the regularization is positively affecting the entropy at all stages of the training.

Original: Indeed, the persistent pressure to increase the entropy can result in over-dispsersed ﬁnal product.
Revised: Indeed, the persistent pressure to increase the entropy can result in over-dispsersed ﬁnal product.

Original: However, this seems not to be a serious issue when it comes to the learning on big data like ImageNet.
Revised: However, this seems not to be a serious issue when it comes to the learning on big data like ImageNet.

Original: In terms of Inception score and FID score, all artiﬁcial generator distributions today are still far less diverse than the original dataset (Table 1), and there is still a large room left for the improvement of diversity.
Revised: In terms of Inception score and FID score, all artiﬁcial generator distributions today are still far less diverse than the original dataset (Table 1), and there is still a large room left for the improvement of diversity.

Original: Effect on monotonicity
Revised: Effect on monotonicity

Original: We also made an assessment on the effect of choosing a target distribution constructed with monotonic mapping.
Revised: We also made an assessment on the effect of choosing a target distribution constructed with monotonic mapping.

Original: Let K be a positive value.
Revised: Let K be a positive value.

Original: Starting from the initial distribution X =
Revised: Starting from the initial distribution X =

Original: G θ 

Revised: G θ

Original: ( Z ) , consider the following pair of maps to be applied to X :
Revised: ( Z ) , consider the following pair of maps to be applied to X :

Original: [Equation]
Revised: [Equation]

Original: It is evident that T (2) is a monotonic mapping and T (1) is not.
Revised: It is evident that T (2) is a monotonic mapping and T (1) is not.

Original: Denote the law of X by µ .
Revised: Denote the law of X by µ .

Original: We used
Revised: We used

Original: T (1) # µ and T (2) # µ as target distributions, and trained the θ using O m = E Z
Revised: T (1) # µ and T (2) # µ as target distributions, and trained the θ using O m = E Z

Original: [ (cid:107) T ( m ) ( G θ 

Revised: [ (cid:107) T ( m ) ( G θ

Original: ( Z ))
Revised: ( Z ))

Original: −
Revised: −

Original: G θ ( Z ) (cid:107) 2 ] as the objective function.
Revised: G θ ( Z ) (cid:107) 2 ] as the objective function.

Original: Note that, for both m = 1 , 2 , this objective function takes the same value, K 2 .
Revised: Note that, for both m = 1 , 2 , this objective function takes the same value, K 2 .

Original: However, as we can see in Fig 2 (b), the training with O 2 proceeds much faster than the training with O 1 .
Revised: However, as we can see in Fig 2 (b), the training with O 2 proceeds much faster than the training with O 1 .

Original: This outcome is against a naive intuition based on distributional distance because the W 2 ( T (1) # µ, µ ) <
Revised: This outcome is against a naive intuition based on distributional distance because the W 2 ( T (1) # µ, µ ) <

Original: W 2 ( T (2) # µ, µ ) .
Revised: W 2 ( T (2) # µ, µ ) .

Original: To see this, note that T 1 is a derivative of − K | X | and T (2) is a derivative of K | X | , which is a convex function.
Revised: To see this, note that T 1 is a derivative of − K | X | and T (2) is a derivative of K | X | , which is a convex function.

Original: This implies that T (2) is the optimal transport from µ to T (2) # µ .
Revised: This implies that T (2) is the optimal transport from µ to T (2) # µ .

Original: On the other hand, by the deﬁnition of the Wasserstein distance, K = W 2 ( T (2) # µ, µ ) while W 2 ( T [Equation]
Revised: On the other hand, by the deﬁnition of the Wasserstein distance, K = W 2 ( T (2) # µ, µ ) while W 2 ( T (1) # µ, µ ) 2 = inf T # µ = T 1 # µ E µ [ (cid:107) T ( X ) − X

Original: X ) − X (cid:107) 2 ] ≤
Revised: (cid:107) 2 ] ≤

Original: E µ [ (cid:107) T (1) ( X ) − X
Revised: E µ [ (cid:107) T (1) ( X ) − X

Original: (cid:107) 2 ]
Revised: (cid:107) 2 ]

Original: = K 2 .
Revised: = K 2 .

Original: This result (Fig 2 (b)) suggests that, when using the update rule similar to the equation 3, the training proceeds much faster when we choose a target distribution with monotonic mapping.
Revised: This result (Fig 2 (b)) suggests that, when using the update rule similar to the equation 3, the training proceeds much faster when we choose a target distribution with monotonic mapping.

Original: [Figure]
Revised: [Figure]

Original: 4. R ESULTS ON CIFAR-10 AND CIFARExperiments with different architectures and objective functions
Revised: 4. R ESULTS ON CIFAR-10 AND CIFARExperiments with different architectures and objective functions

Original: We tested our algorithm for the training of GANs with six types of objective functions and two network architectures, and reported the performance on all 12 combinations.
Revised: We tested our algorithm for the training of GANs with six types of objective functions and two network architectures, and reported the performance on all 12 combinations.

Original: For the details of the objective functions and the architectures, please see Appendix (B.2, B.3, B.4){+B.
+}
Revised: For the details of the objective functions and the architectures, please see Appendix B.2, B.3, {+B.
+}

Original: For the training of the network, we applied Spectral Normalization  (Miyato et al., 2018) to the full-connected layer and the convolution layer.
Revised: For the training of the network, we applied Spectral Normalization (SN) (Miyato et al., 2018) to the full-connected layer and the convolution layer.

Original: Fig 3 and Fig 8 (in Appendix) respectively summarize Inception scores and FID for all 12 settings.
Revised: Fig 3 and Fig 8 (in Appendix) respectively summarize Inception scores and FID for all 12 settings.

Original: We see that DC regularization is improving the performance irrespective of the choice of the architecture and the objective function.
Revised: We see that DC regularization is improving the performance irrespective of the choice of the architecture and the objective function.

Original: Experiments with different prior dimensions
Revised: Experiments with different prior dimensions

Original: In general, without careful selection of the dimension of the prior distribution, the training of GANs tends to suffer a serious case of mode collapse.
Revised: In general, without careful selection of the dimension of the prior distribution, the training of GANs tends to suffer a serious case of mode collapse.

Original: For image generation task, this will result in low inception score.
Revised: For image generation task, this will result in low inception score.

Original: We therefore applied DC regularization to the trainings of GANs on CIFAR-10 with very low prior dimensions as well as very large prior dimensions and evaluated the performance.
Revised: We therefore applied DC regularization to the trainings of GANs on CIFAR-10 with very low prior dimensions as well as very large prior dimensions and evaluated the performance.

Original: For this experiment, we used GAN-variant2 (Appendix B.2) for the objective function and used SNDCGAN for the architecture.
Revised: For this experiment, we used GAN-variant2 (Appendix B.2) for the objective function and used SNDCGAN for the architecture.

Original: As for the experimental details, please see Appendix B.4.
Revised: As for the experimental details, please see Appendix B.4.

Original: The results are summarized in Fig 4 and Fig 10 (in Appendix).
Revised: The results are summarized in Fig 4 and Fig 10 (in Appendix).

Original: When the prior dimension is below the inherent dimension of the dataset, the dimension of the generator distribution cannot match the dimension of the true distribution.
Revised: When the prior dimension is below the inherent dimension of the dataset, the dimension of the generator distribution cannot match the dimension of the true distribution.

Original: Thus, the inception score of the generator trained with low dimensional prior is bound to be low, irrespective of the application of DC regularization.
Revised: Thus, the inception score of the generator trained with low dimensional prior is bound to be low, irrespective of the application of DC regularization.

Original: However, for dim( z ) ≥ 5 , the inception score is as high as 7.5, and for all choices of dim( z ) , the generator trained with DC regularization consistently outperformed the generator trained without the DC regularization.
Revised: However, for dim( z ) ≥ 5 , the inception score is as high as 7.5, and for all choices of dim( z ) , the generator trained with DC regularization consistently outperformed the generator trained without the DC regularization.

Original: Similar argument applies to the performance of DC regularization for high dimensional prior.
Revised: Similar argument applies to the performance of DC regularization for high dimensional prior.

Original: Overall, the DC regularization provides some robustness against the choice of the prior dimension [Figure] [Figure] [Table]
Revised: Overall, the DC regularization provides some robustness against the choice of the prior dimension.

Original: Comparison with EGAN and other methods
Revised: Comparison with EGAN and other methods

Original: We compared the performance of our algorithm against EGAN (Dai et al., 2017), another framework that can be used to control the entropy of the generator.
Revised: We compared the performance of our algorithm against EGAN (Dai et al., 2017), another framework that can be used to control the entropy of the generator.

Original: In particular, we implemented EGAN-EntVI, which includes into its objective function a penalty against the negative entropy of the generator.
Revised: In particular, we implemented EGAN-EntVI, which includes into its objective function a penalty against the negative entropy of the generator.

Original: We conducted this comparative study using SNDCGAN (Table 2) and SNResNet (Table 3).
Revised: We conducted this comparative study using SNDCGAN (Table 2) and SNResNet (Table 3).

Original: For the experimental setting of this study, please see the Appendix B.4.
Revised: For the experimental setting of this study, please see the Appendix B.4.

Original: As we show in Table 
Revised: As we show in Table 1, DC

Revised: [Figure]

Original: [Table]
Revised: [Figure]

Original: DC regularization outperformed EGAN-Ent-VI on both models.
Revised: regularization outperformed EGAN-Ent-VI on both models.

Original: For the full version of the Table and the visuals of the generated samples, please see  8 and  11 in the Appendix)
Revised: For the full version of the Table and the visuals of the generated samples, please see Table 8 and Table 11 in the Appendix.

Original: We would like to emphasize here that EGAN-Ent-VI requires a separate decoder in addition to the generator and the discriminator, and that our algorithm is easier to implement.
Revised: We would like to emphasize here that EGAN-Ent-VI requires a separate decoder in addition to the generator and the discriminator [Figure] and that our algorithm is easier to implement.

Original: We compared our algorithm against other competitive methods as well.
Revised: We compared our algorithm against other competitive methods as well.

Original: The best performance of our method is almost on par with the state-of-the-art method (Karras et al., 2018).
Revised: The best performance of our method is almost on par with the state-of-the-art method (Karras et al., 2018).

Original: We are losing to progressive GAN (Karras et al., 2018) by a slight margin; we, however, would like to make a disclaimer that we are using much smaller architecture than Karras et al.
Revised: We are losing to progressive GAN (Karras et al., 2018) by a slight margin; we, however, would like to make a disclaimer that we are using much smaller architecture than Karras et al.

Original: (2018) for the performance evaluation of our method.
Revised: (2018) for the performance evaluation of our method.

Original: We also comﬁrmed that we can improve the result of Miyato et al.
Revised: We also comﬁrmed that we can improve the result of Miyato et al.

Original: by using DC regularization together with SN.
Revised: by using DC regularization together with SN.

Original: The results support that our method is helping the training process suppress mode collapse and is improving the overall performance.
Revised: The results support that our method is helping the training process suppress mode collapse and is improving the overall performance.

Original: For the results of the experiments conducted with WGAN-GP, please see the Fig 9 in Appendix. 
Revised: For the results of the experiments conducted with Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al., 2017), please see the Fig 9 in Appendix. [Table]

Original: 4. R ESULTS ON I MAGE N ET
Revised: 4. R ESULTS ON I MAGE N ET

Original: Additionaly, to evaluate our method’s effectiveness on a higher dimensional dataset, we applied our method on ImageNet with 1000 classes, with each class containing approximately 1300 images.
Revised: Additionaly, to evaluate our method’s effectiveness on a higher dimensional dataset, we applied our method on ImageNet with 1000 classes, with each class containing approximately 1300 images.

Original: We compressed the images to 64 × 64 pixels prior to the experiments.
Revised: We compressed the images to 64 × 64 pixels prior to the experiments.

Original: For the objective function, we chose GAN-hinge (Appendix B.2), which was being used in (Miyato et al., 2018).
Revised: For the objective function, we chose GAN-hinge (Appendix B.2), which was being used in Miyato et al.

Original: For the experimental settings, please see Appendix B.5 for the details.
Revised: For the experimental settings, please see Appendix B.5 for the details.

Original: We can conﬁrm on Fig 5 that our DC regularization is improving the Inception score throughout the course of the training.
Revised: We can conﬁrm on Fig 5 that our DC regularization is improving the Inception score throughout the course of the training.

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION Dilemma of GANs update
Revised:  D ISCUSSION Dilemma of GANs update

Original: As we have shown above, the experimental results support our claim that the DC regularization promotes the entropy of the generator distribution and stabilizes the training process of GANs.
Revised: As we have shown above, the experimental results support our claim that the DC regularization promotes the entropy of the generator distribution and stabilizes the training process of GANs.

Original: As mentioned brieﬂy in the theory section 2, however, we do not have the guarantee that our update rule consistently reduces the W 2 distance between the generator distribution and the true distribution.
Revised: As mentioned brieﬂy in the theory section 2, however, we do not have the guarantee that our update rule consistently reduces the W 2 distance between the generator distribution and the true distribution.

Original: This fault, however, is common to almost all GANs algorithms today.
Revised: This fault, however, is common to almost all GANs algorithms today.

Original: As we have shown, the conventional update rule is implicitly targetting a distribution of the form T # µ where T = Id − α ∇ L for some α and a discriminator L .
Revised: As we have shown, the conventional update rule is implicitly targetting a distribution of the form T # µ where T = Id − α ∇ L for some α and a discriminator L .

Original: As it turns out, optimal transport takes this form only when we are optimizing W 2 distance, while the common constraint of |∇ L | = 1 is a condition required for dual potential when p = 1 ( W 1 ) (Villani, 2008).
Revised: As it turns out, optimal transport takes this form only when we are optimizing W 2 distance, while the common constraint of |∇ L | = 1 is a condition required for dual potential when p = 1 ( W 1 ) (Villani, 2008).

Original: In other words, the conventional GANs are making the discriminator with W 1 criteria and updating the generator with W 2 criteria.
Revised: In other words, the conventional GANs are making the discriminator with W 1 criteria and updating the generator with W 2 criteria.

Original: If we are to faithfully create the discriminator with W 2 criteria, we must look for a legendre-pair of dual potential functions.
Revised: If we are to faithfully create the discriminator with W 2 criteria, we must look for a legendre-pair of dual potential functions.

Original: On the contrary, if we are to update the generator with W 1 criteria, one must look for a closed form solution for the W 1 transport.
Revised: On the contrary, if we are to update the generator with W 1 criteria, one must look for a closed form solution for the W 1 transport.

Original: This, however, is in general a highly complex mathematical problem for which there is a separate ﬁeld of study (Santambrogio, 2015).
Revised: This, however, is in general a highly complex mathematical problem for which there is a separate ﬁeld of study (Santambrogio, 2015).

Original: To the authors’ best knowledge, no studies have provided a solid solution to this dilemma.
Revised: To the authors’ best knowledge, no studies have provided a solid solution to this dilemma.

Original: Convexity vs Strong convexity
Revised: Convexity vs Strong convexity

Original: We would like to also mention that our regularization is asking for more than what is required by
Revised: We would like to also mention that our regularization is asking for more than what is required by

Original: W 2 theory.
Revised: W 2 theory.

Original: As mentioned above, in order for T to be the optimal transport from µ to T # µ , T only needs to be the gradient of a convex function.
Revised: As mentioned above, in order for T to be the optimal transport from µ to T # µ , T only needs to be the gradient of a convex function.

Original: On the other hand, by asking L to be concave, we are in fact asking for T to be the gradient of a strongly convex function.
Revised: On the other hand, by asking L to be concave, we are in fact asking for T to be the gradient of a strongly convex function.

Original: This ”overdo” is actually intentional.
Revised: This ”overdo” is actually intentional.

Original: Recall that the parameter α in the transport T α
Revised: Recall that the parameter α in the transport T α

Original: = Id − α ∇ L corresponds to a step size in the update of the generator.
Revised: = Id − α ∇ L corresponds to a step size in the update of the generator.

Original: If we train L that only guarantees the convexity of (cid:107) x (cid:107) 2 / 2 − L ( x ) , the functional update derived from such L can be non-monotonic when the step size is large, because such L only guarantees the convexity of (cid:107) x (cid:107) 2 / 2 − αL ( x ) when α ≤ 1 .
Revised: If we train L that only guarantees the convexity of (cid:107) x (cid:107) 2 / 2 − L ( x ) , the functional update derived from such L can be non-monotonic when the step size is large, because such L only guarantees the convexity of (cid:107) x (cid:107) 2 / 2 − αL ( x ) when α ≤ 1 .

Original: Should we require L to be concave, however, (cid:107) x (cid:107) 2 / 2 − αL ( x ) is concave for any positive α .
Revised: Should we require L to be concave, however, (cid:107) x (cid:107) 2 / 2 − αL ( x ) is concave for any positive α .

Original: In this context, we may therefore say that the DC regularization leaves much room to be playful about the learning schedule.
Revised: In this context, we may therefore say that the DC regularization leaves much room to be playful about the learning schedule.

Original: Finally, as can be inferred from our proofs for the proposition 2.3 and 2.4, the strong convexity of T has an effect of diffusing the mass of pdf in every direction.
Revised: Finally, as can be inferred from our proofs for the proposition 2.3 and 2.4, the strong convexity of T has an effect of diffusing the mass of pdf in every direction.

Original: We can therefore expect the DC regularization to disperse the collapsed masses in the case of mode collapse.
Revised: We can therefore expect the DC regularization to disperse the collapsed masses in the case of mode collapse.

Original: In the light of the fact that the training of GANs usually begins with dense distribution with small support, this dispersion effect should be helpful at the early stage of the training as well.
Revised: In the light of the fact that the training of GANs usually begins with dense distribution with small support, this dispersion effect should be helpful at the early stage of the training as well.

Original: Appendix (Supplemental Material for Distributional Concavity Regularization for GANs) A T HE PROOFS FOR THE PROPOSITIONS 2.3 AND 2. 
Revised: Appendix (Supplemental Material for Distributional Concavity Regularization for GANs) A T HE PROOFS FOR THE PROPOSITIONS 2.3 AND 2.

Original: We will ﬁrst prove the proposition 2.4.
Revised: We will ﬁrst prove the proposition 2.4.

Original: Let us begin with a useful lemma.
Revised: Let us begin with a useful lemma.

Revised: We will assume that all regularity assumptions made in Section 2 holds for all variables and functions that appear in this section.

Revised: Also, we will use the Df to denote the derivative of f .

Revised: Also assume that, unless otherwise noted, Df ( x ) indicates D x f ( x ) , the derivative of f with respect to x .

Original: Lemma A.1.
Revised: Lemma A.1.

Original: Let µ and ν be  distributions ona compact set in R n , and suppose that we can write ( Id − ∇ L )# µ
Revised: Let µ and ν be probability distributions on R n , and suppose that we can write ( Id − ∇ L )# µ

Original: = ν with one-to-one ∇ L .
Revised: = ν with one-to-one ∇ L .

Original: Let us also write T
Revised: Let us also write T

Original: = ( Id − ∇ L ) , and let T t
Revised: = ( Id − ∇ L ) , and let T t

Original: = Id − t ∇ L be its time-linear interpolation.
Revised: = Id − t ∇ L be its time-linear interpolation.

Original: Then, for all t ∈ [0 , 1] ,
Revised: Then, for all t ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let p µ be the pdf of µ .
Revised: Let p µ be the pdf of µ .

Original: If y = T ( x ) , by the one-to-one assumption det ( DT ( x )) >
Revised: If y = T ( x ) , by the one-to-one assumption det ( DT ( x )) >

Original: and we may let d y = det ( DT ( x ))
Revised: and we may let d y = det ( DT ( x ))

Original: d x .
Revised: d x .

Original: By appealing to the fact that E µ [ h ( T ( X ))]
Revised: By appealing to the fact that E µ [ h ( T ( X ))]

Original: =
Revised: =

Original: E ν [ h ( Y )] for arbitrary h ,
Revised: E ν [ h ( Y )] for arbitrary h ,

Original: [Equation]
Revised: [Equation]

Original: 15) and we can deduce the identity p T # µ ( T ( x )) det ( DT ( x ))
Revised: [Equation] and we can deduce the identity p T # µ ( T ( x )) det ( DT ( x ))

Original: = p µ ( x ) .
Revised: = p µ ( x ) .

Original: Building on this fact, with straightforward computation we can say
Revised: Building on this fact, with straightforward computation we can say

Original: [Equation]
Revised: [Equation]

Original: Assuming that the distributions are regular enough that we can swap the integral, we can appeal to
Revised: Assuming that the distributions are regular enough that we can swap the integral, we can appeal to

Original: Jacobi’s formula and deduce
Revised: Jacobi’s formula and deduce

Original: [Equation]
Revised: [Equation]

Original: Now, let us prove the proposition 2.4.
Revised: Now, let us prove the proposition 2.4.

Original: Without loss of generality, choose L so that DT = (cid:107) x (cid:107) 2 2 − L .
Revised: Without loss of generality, choose L so that DT = (cid:107) x (cid:107) 2 2 − L .

Original: Assuming that D 2 L is diagonalizable, let spec ( D 2 L ) =
Revised: Assuming that D 2 L is diagonalizable and that its spectrum is uniformly bounded away from 1 , let spec ( D 2 L ) =

Original: { λ k } and write diag { λ k } k = Λ .
Revised: { λ k } and write diag { λ k } k = Λ .

Original: Because (cid:107) x (cid:107) 2− L is assumed to be strictly convex, D 2 (cid:16) (cid:107) x (cid:107) 2− L (cid:17)
Revised: Because (cid:107) x (cid:107) 2− L is assumed to be strictly convex, D 2 (cid:16) (cid:107) [Equation] (cid:107) 2− L (cid:17)

Original: =
Revised: =

Original: I − D 2 L is positive deﬁnite, and 1 − λ k
Revised: I − D 2 L is positive deﬁnite, and 1 − λ k

Original: ( x ) > 0 for all k .
Revised: ( x ) > 0 for all k .

Original: Appealing to the line equation (19) we see that the assumption H ( ν ) > H ( µ ) is equivalent to
Revised: Appealing to the line equation (19) we see that the assumption H ( ν ) > H ( µ ) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Let us write max k { 1 − tλ k
Revised: Let us write max k { 1 − tλ k

Original: ( x ) } =
Revised: ( x ) } =

Original: C t ( x ) > 0 .
Revised: C t ( x ) > 0 .

Original: Using the assumption that the support of µ is compact, we can also say 0 < max supp ( µ ) { C t ( x ) } :=
Revised: Using the assumption that the support of µ is compact, we can also say 0 < max supp ( µ ) { C t ( x ) } :=

Original: c t < ∞ .
Revised: c t < ∞ .

Original: In general, if A is positive deﬁnite and diagonalizable, with straightforward argument we can say 
Revised: In general, if A is positive deﬁnite and diagonalizable, with straightforward argument we can say [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using this fact, we see that
Revised: Using this fact, we see that

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Suppose that D 2 L is diagonalizable with the change of basis matrix P .
Revised: Suppose that D 2 L is diagonalizable with the change of basis matrix P .

Original: Then 
Revised: Then −

Original: [Equation]
Revised: [Equation]

Original: This concludes the proof of the proposition 2.4.
Revised: This concludes the proof of the proposition 2.4.

Original: The proposition 2.3 is much simpler to prove.
Revised: The proposition 2.3 is much simpler to prove.

Original: In order to assure the H ( ν ) ≥ H ( µ ) , we only need to guarantee that E µ [log det ( DT ( X ))] ≥ 0 .
Revised: In order to assure the H ( ν ) ≥ H ( µ ) , we only need to guarantee that E µ [log det ( DT ( X ))] ≥ 0 .

Original: By requiring L to be concave, we would make (cid:107) x (cid:107) 2 / 2 − L ( x ) to be convex so that the eigenvalues of
Revised: By requiring L to be concave, we would make (cid:107) x (cid:107) 2 / 2 − L ( x ) to be convex so that the eigenvalues of

Original: DT will all become positive.
Revised: DT will all become positive.

Original: The result then follows trivially from the argument similar to the one that leads to equation (24).
Revised: The result then follows trivially from the argument similar to the one that leads to equation (24).

Original: B E XPERIMENTAL S ETTINGS B. P ERFORMANCE MEASURES
Revised: B E XPERIMENTAL S ETTINGS B. P ERFORMANCE MEASURES

Original: For the measure of GANs’ performance on the image dataset, we used Inception score(Salimans et al., 2016).
Revised: For the measure of GANs’ performance on the image dataset, we used Inception score (Salimans et al., 2016).

Original: Inception score was introduced originally as an exponentiated divergence measure based on the trained Inception convolutional neural network (Szegedy et al., 2015), which is often called Inception Model .
Revised: Inception score was introduced originally as an exponentiated divergence measure based on the trained Inception convolutional neural network (Szegedy et al., 2015), which is often called Inception Model .

Original: Using p ( y | x ) to denote the Inception model, the inception score is given by I ( { x n } Nn =1 ) := exp( E [ D KL [ p ( y | x ) || p ( y )]]) , which is often computed by approximating p ( y ) with
Revised: Using p ( y | x ) to denote the Inception model, the inception score is given by I ( { x n } N n =1 ) := exp(ˆ E [ D KL [ p ( y | x ) || p ( y )]]) , where ˆ E indicates the empirically approximated expectation.

Original:  N (cid:80)Nn
Revised: Often times, p ( y ) is approximated with 1 N (cid:80)

Original:  =1 p ( y | x n ) .
Revised: Nn =1 p ( y | x n ) .

Original: The dominating consensus among the machine learning community is that this score is strongly correlated with subjective human judgment of image quality.
Revised: The dominating consensus among the machine learning community is that this score is strongly correlated with subjective human judgment of image quality.

Original: Following the procedure in Salimans et al.
Revised: Following the procedure in Salimans et al.

Original: , we generated 5000 examples from each trained generator and calculated the Inception score on the samples.
Revised: , we generated 5000 examples from each trained generator and calculated the Inception score on the samples.

Original: We evaluated the score 10 times with different seeds for the generation of x n and reported the average and the standard deviation of the scores.
Revised: We evaluated the score 10 times with different seeds for the generation of x n and reported the average and the standard deviation of the scores.

Original: Fr´echet inception distance  (Heusel et al., 2017)(FID) is another measure for the quality of the generated examples that uses 2nd order information of the ﬁnal layer of the inception model.
Revised: Fr´echet inception distance (FID) (Heusel et al., 2017) is another measure for the quality of the generated examples that uses 2nd order information of the ﬁnal layer of the inception model.

Original: The FID is based on Fre ´chet distance (Dowson & Landau, 1982)
Revised: The FID is based on Fre ´ chet distance (Dowson & Landau, 1982)

Original: (Not to be confused with FID), which is the 2-Wasserstein distance between two distribution multivariate Gaussian distributions, p 1 and p 2 .
Revised: (Not to be confused with FID), which is the 2-Wasserstein distance between two distribution multivariate Gaussian distributions, p 1 and p 2 .

Original: The

Original:  Wasserstein distance for two Gaussian distributions have a closed form, and is given by
Revised: The Wasserstein distance for two Gaussian distributions have a closed form, and is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where { µ p 1 , C p 1 } , { µ p 2 , C p 2 } are the mean and covariance of samples from q and p , respectively.
Revised: where { µ p 1 , C p 1 } , { µ p 2 , C p 2 } are the mean and covariance of samples from q and p , respectively.

Original: Now, if f (cid:9) is the output of the ﬁnal layer of the inception model before the softmax, the Fr´echet inception distance (FID) between two distributions p 1 and p 2 on the images is the Fre´chet distance distance between f (cid:9) ◦ p 1 and f (cid:9) ◦ p 2 .
Revised: Now, if f (cid:9) is the output of the ﬁnal layer of the inception model before the softmax, the Fr´echet inception distance (FID) between two distributions p 1 and p 2 on the images is the Fre´chet distance distance between f (cid:9) ◦ p 1 and f (cid:9) ◦ p 2 .

Original: We empirically computed the Fr´echet inception distance between the true distribution and the generated distribution using 10000 samples from the true distribution and 5000 samples from the generator distribution.
Revised: We empirically computed the Fr´echet inception distance between the true distribution and the generated distribution using 10000 samples from the true distribution and 5000 samples from the generator distribution.

Original: B. GAN’ S OBJECTIVE FUNCTION
Revised: B. GAN’ S OBJECTIVE FUNCTION

Original: We used applied DC regularizaiton to the training with the following set of objective functions.{+functions (softplus ( x ) = log(
 + exp( x )) ). GAN-+}
Revised: We used applied DC regularizaiton to the training with the following set of objective {+functions (softplus ( x ) = log(
 + exp( x )) ). GAN-+}

Original: [Equation]
Revised: [Equation]

Revised: Goodfellow et al.

Revised: [Equation]

Revised: [Equation]

Original: B. N ETWORK A RCHITECTURES
Revised: [Equation] [Equation] [Equation] GAN

Original: Radford et al., 2016) for image Generation experiments on
Revised: [Equation] Ye [Equation] [Equation]

Original: CIFAR-10 and CIFAR-100.
Revised: [Equation]

Original: The slopes of all leaky-ReLU functions in the networks were set to 0 . z ∈ R 128 ∼ N (0 , I ) dense → M g × M g ×4 × 4, stride=2 deconv. BN 256 ReLU 4 × 4, stride=2 deconv. BN 128 ReLU 4 × 4, stride=2 deconv.
Revised: Feature Matching

Original: BN 64 ReLU 3 × 3, stride=1 conv.
Revised: [Equation] [Table] B. N ETWORK A RCHITECTURES

Original: 3 Tanh
Revised: [Table]

Original: RGB image x ∈ R M × M ×3 × 3, stride=1 conv 64 lReLU 4 × 4, stride=2 conv 64 lReLU 3 × 3, stride=1 conv 128 lReLU 4 × 4, stride=2 conv 128 lReLU 3 × 3, stride=1 conv 256 lReLU 4 × 4, stride=2 conv 256 lReLU 3 × 3, stride=1 conv.
Revised: [Figure] [Figure] [Figure]

Original: 512 lReLU dense →
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: Global sum pooling

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: dense →

Revised: 128 *(b) ResNet decoder.

Original: B. D ETAILS FOR THE EXPERIMENTS ON CIFAR-10 AND CIFAR- 
Revised: B. D ETAILS FOR THE EXPERIMENTS ON CIFAR-10 AND CIFAR-

Original: Experimental setting for the evaluation of the method’s robustness against the choice of objective
Revised: Experimental setting for the evaluation of the method’s robustness against the choice of objective

Original: functions
Revised: functions

Original: For the model, we chose the architecture of DCGAN(Radford et al., 2016)
Revised: For the model, we chose the architecture of DCGAN(Radford et al., 2016)

Original: (Table 2) and ResNet (Table 3) with spectral normalization applied to full connect layer and convolution layer(SNDCGAN,
Revised: (Table 2) and ResNet (Table 3) with spectral normalization applied to full connect layer and convolution layer(SNDCGAN,

Original: SNResNet).
Revised: SNResNet).

Original: For the optimization, we used Adam(Kingma & Ba, 2015) and chose ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ) for the hyperparameters.
Revised: For the optimization, we used Adam(Kingma & Ba, 2015) and chose ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ) for the hyperparameters.

Original: Also, we chose n dis = 1 , n gen = 1 for
Revised: Also, we chose n dis = 1 , n gen = 1 for

Original: SNDCGAN and ( n dis = 5 , n gen = 1) for SNResNet.
Revised: SNDCGAN and ( n dis = 5 , n gen = 1) for SNResNet.

Original: We updated the generator 100 k times and linearly decayed the learning rate over last 5 k iterations.
Revised: We updated the generator 100 k times and linearly decayed the learning rate over last 5 k iterations.

Original: Experimental setting for the evaluation of the method’s robustness against the choice of the prior dimension
Revised: Experimental setting for the evaluation of the method’s robustness against the choice of the prior dimension

Original: We chose SNDCGAN for the model, and optimized the network with Adam using the hyperparameter ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ).
Revised: We chose SNDCGAN for the model, and optimized the network with Adam using the hyperparameter ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ).

Original: For the update of the discriminator and the generator, we set n dis = 1 , n gen = 1 .
Revised: For the update of the discriminator and the generator, we set n dis = 1 , n gen = 1 .

Original: We trained the generator 50 k times and linearly decayed the learning rate over last 5 k iterations.
Revised: We trained the generator 50 k times and linearly decayed the learning rate over last 5 k iterations.

Original: For the objective function, we chose GAN-variant2 in Appendix B.2.
Revised: For the objective function, we chose GAN-variant2 in Appendix B.2.

Original: We also set λ = 3 . 0 , d = 0 .
Revised: We also set λ = 3 . 0 , d = 0 .

Original: 01 for the parameters in (10).
Revised: 01 for the parameters in (10).

Original: For this set of the experiments, we repeated the experiments three times with different seeds, and reported the maximum, mean and minimum.
Revised: For this set of the experiments, we repeated the experiments three times with different seeds, and reported the maximum, mean and minimum.

Original: We tested with prior dimensions of range dim( z ) = 3 , 4 , 5 , 7 , 10 , 500 , 1000 , 2000 , 4000 , 6000 , 8000 , 10000 .
Revised: We tested with prior dimensions of range dim( z ) = 3 , 4 , 5 , 7 , 10 , 500 , 1000 , 2000 , 4000 , 6000 , 8000 , 10000 .

Original: Comparison with EGAN and other methods
Revised: Comparison with EGAN and other methods

Original: For the model, we chose SNDCGAN , SNResNet and SNResNetLarge, and trained the networks using Adam with hyperparameters ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ).
Revised: For the model, we chose SNDCGAN , SNResNet and SNResNetLarge, and trained the networks using Adam with hyperparameters ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ).

Original: SNResNetLarge is a model that was used in (Miyato et al., 2018).
Revised: SNResNetLarge is a model that was used in (Miyato et al., 2018).

Original: It is a same model as SNResNet except that it is uses a larger generator (Table 4).
Revised: It is a same model as SNResNet except that it is uses a larger generator (Table 4).

Original: For both models, We trained the generator 100 k times and linearly decayed the learning rate over last 10 k iterations.
Revised: For both models, We trained the generator 100 k times and linearly decayed the learning rate over last 10 k iterations.

Original: For EGAN-Ent-VI (Dai et al., 2017), we used an additional decoder equipped with convolution layers.
Revised: For EGAN-Ent-VI (Dai et al., 2017), we used an additional decoder equipped with convolution layers.

Original: We used n dis = 1 , n gen = 1 , n dec = 5 for SNDCGAN, and n dis = 1 , n gen = 5 , n dec = 5 for SNResNet.
Revised: We used n dis = 1 , n gen = 1 , n dec = 5 for SNDCGAN, and n dis = 1 , n gen = 5 , n dec = 5 for SNResNet.

Original: The table below is the list of the choices of the hyperparameters ( λ, d ) in equation (10).
Revised: The table below is the list of the choices of the hyperparameters ( λ, d ) in equation (10).

Original: we used in our comparative study, sorted by models
Revised: we used in our comparative study, sorted by models

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B. I MAGE G ENERATION ON I MAGE N ET
Revised: B. I MAGE G ENERATION ON I MAGE N ET

Original: The images used in this set of experiments were resized to 64 × 64 pixels.
Revised: The images used in this set of experiments were resized to 64 × 64 pixels.

Original: The details of the architecture are given in Table 5.
Revised: The details of the architecture are given in Table 5.

Original: For the optimization, we used Adam with the same hyperparameters we used for ResNet on CIFAR-10 and CIFAR-100 dataset.
Revised: For the optimization, we used Adam with the same hyperparameters we used for ResNet on CIFAR-10 and CIFAR-100 dataset.

Original: We trained the networks with 250K generator updates, and applied linear decay for the learning rate after 200K iterations so that the rate would be 0 at the end.
Revised: We trained the networks with 250K generator updates, and applied linear decay for the learning rate after 200K iterations so that the rate would be 0 at the end.

Original: We set λ = 6 . 0 , d = 0 .
Revised: We set λ = 6 . 0 , d = 0 .

Original: 01 for the parameters in equation (10).
Revised: 01 for the parameters in equation (10).

Original: C
Revised: C

Original: A PPENDIX R ESULTS C. A PPENDIX R ESULT ON ARTIFICIAL DATA
Revised: A PPENDIX R ESULTS C. A PPENDIX R ESULT ON ARTIFICIAL DATA

Original: [Figure]
Revised: [Figure]

Original: (a) without DC regularization        8        data model
Revised: (a) without DC regularization        8        data model

Original: [Figure]
Revised: [Figure]

Original: C.
Revised: C.

Original: A PPENDIX R ESULT ON CIFAR-10 AND CIFARExperiment with different architectures
Revised: A PPENDIX R ESULT ON CIFAR-10 AND CIFARExperiment with different architectures

Original: [Figure]
Revised: [Figure]

Original: Experiment with different architectures on WGAN-GP
Revised: Experiment with different architectures on WGAN-GP

Original: [Figure]
Revised: [Figure]

Original: FID(lower is better)
Revised: FID(lower is better)

Original: [Figure]
Revised: [Figure]

Original: Comparison with other methods
Revised: Comparison with other methods

Original: [Table]
Revised: [Table]

Original: Image generation on CIFAR-10 and CIFAR- 
Revised: Image generation on CIFAR-10 and CIFAR-

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: C. A PPENDIXR ESULT ON I MAGE N ET
Revised: PPENDIX

Revised: ESULT ON MAGE N

Original: [Figure]
Revised: [Figure]