Original: D ISTRIBUTIONAL C ONCAVITY R EGULARIZATION FOR GAN S
Revised: D ISTRIBUTIONAL C ONCAVITY R EGULARIZATION FOR GAN S

Original: Preferred Networks { guguchi, masomatics } @preferred.jp
Revised: Preferred Networks { guguchi, masomatics } @preferred.jp

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose Distributional Concavity (DC) regularization for Generative Adversarial Networks (GANs), a functional gradient-based method that promotes the entropy of the generator distribution and works against mode collapse.
Revised: We propose Distributional Concavity (DC) regularization for Generative Adversarial Networks (GANs), a functional gradient-based method that promotes the entropy of the generator distribution and works against mode collapse.

Original: Our DC regularization is an easy-to-implement method that can be used in combination with the current state of the art methods like Spectral Normalization and Wasserstein GAN with gradient penalty to further improve the performance.
Revised: Our DC regularization is an easy-to-implement method that can be used in combination with the current state of the art methods like Spectral Normalization and Wasserstein GAN with gradient penalty to further improve the performance.

Original: We will not only show that our DC regularization can achieve highly competitive results on ILSVRC2012 and CIFAR datasets in terms of Inception score and Fr ´ echet inception distance, but also provide a mathematical guarantee that our method can always increase the entropy of the generator distribution.
Revised: We will not only show that our DC regularization can achieve highly competitive results on ILSVRC2012 and CIFAR datasets in terms of Inception score and Fr ´ echet inception distance, but also provide a mathematical guarantee that our method can always increase the entropy of the generator distribution.

Original: We will also show an intimate theoretical connection between our method and the theory of optimal transport
Revised: We will also show an intimate theoretical connection between our method and the theory of optimal transport

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) is a model consisting of two adversarial neural networks designed for the training of a generator distribution that mimics a target distribution deﬁned on a ( often ) high dimensional space, and it has been successful in numerous applications including image and movie generations (Isola et al., 2017; Zhu et al., 2017;
Revised: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) is a model consisting of two adversarial neural networks designed for the training of a generator distribution that mimics a target distribution deﬁned on a ( often ) high dimensional space, and it has been successful in numerous applications including image and movie generations (Isola et al., 2017; Zhu et al., 2017;

Original: Saito et al., 2017).
Revised: Saito et al., 2017).

Original: However, it has not yet completely established itself as a wildly scientiﬁc tool because of the sheer computational difﬁculty of its training process.
Revised: However, it has not yet completely established itself as a wildly scientiﬁc tool because of the sheer computational difﬁculty of its training process.

Original: As such, there has been numerous studies seeking for a way to stabilize the training process of GANs (Gulrajani et al., 2017;
Revised: As such, there has been numerous studies seeking for a way to stabilize the training process of GANs (Gulrajani et al., 2017;

Original: Miyato et al., 2018; Karras et al., 2018).
Revised: Miyato et al., 2018; Karras et al., 2018).

Original: Mode collapse is a persistent central problem for the training of GANs, which collectively refers to the lack of diversity in generator distribution.
Revised: Mode collapse is a persistent central problem for the training of GANs, which collectively refers to the lack of diversity in generator distribution.

Original: For instance, without any countermeasure, GANs applied to multimodal mixture of Gaussians will often train a generator distribution with one mode (See Fig 22, Goodfellow (2016) for example).
Revised: For instance, without any countermeasure, GANs applied to multimodal mixture of Gaussians will often train a generator distribution with one mode (See Fig 22, Goodfellow (2016) for example).

Original: Mode collapse has been discussed in numerous literatures related GANs.
Revised: Mode collapse has been discussed in numerous literatures related GANs.

Original: To name a few, see Goodfellow (2016); Metz et al.
Revised: To name a few, see Goodfellow (2016); Metz et al.

Original: ; Arjovsky &
Revised: ; Arjovsky &

Original: Bottou (2017); Arjovsky et al.
Revised: Bottou (2017); Arjovsky et al.

Original: ; Lin et al. (2017).
Revised: ; Lin et al. (2017).

Original: In words of statistical machine learning, mode collapse can be described as a case of entropy degeneration.
Revised: In words of statistical machine learning, mode collapse can be described as a case of entropy degeneration.

Original: A naive countermeasure against mode collapse is therefore to augment the entropy of the generator distribution.
Revised: A naive countermeasure against mode collapse is therefore to augment the entropy of the generator distribution.

Original: Not many studies to date, however, tackled the problem of mode collapse using this direct approach.
Revised: Not many studies to date, however, tackled the problem of mode collapse using this direct approach.

Original: Dai et al.
Revised: Dai et al.

Original: (2017) partially realized this idea by actually evaluating the entropy of the generator distribution with variational inference and including it into the objective function.
Revised: (2017) partially realized this idea by actually evaluating the entropy of the generator distribution with variational inference and including it into the objective function.

Original: This type of strategy requires the user to continuously produce reliable estimates of the generators based on ﬁnite samples throughout the course of the training.
Revised: This type of strategy requires the user to continuously produce reliable estimates of the generators based on ﬁnite samples throughout the course of the training.

Original: As such, not much further improvement can be expected from this strategy, because precise estimation of the entropy tends to be computationally heavy, and empirical estimation is an excruciatingly difﬁcult problem on its own in high dimension, as is the case in image and movie generation.
Revised: As such, not much further improvement can be expected from this strategy, because precise estimation of the entropy tends to be computationally heavy, and empirical estimation is an excruciatingly difﬁcult problem on its own in high dimension, as is the case in image and movie generation.

Original: In fact, the performance of classical methods based on kernel density estimation, for example, does not scale well with respect to dimension, and the number of samples required to control the MSE can grow exponentially with the dimension (Cacoullos, 1966; Ozakin & Gray, 2009).
Revised: In fact, the performance of classical methods based on kernel density estimation, for example, does not scale well with respect to dimension, and the number of samples required to control the MSE can grow exponentially with the dimension (Cacoullos, 1966; Ozakin & Gray, 2009).

Original: In this study, we use the theory of functional gradient to develop a method that can promote the entropy of the generator distribution without directly estimating the entropy itself.
Revised: In this study, we use the theory of functional gradient to develop a method that can promote the entropy of the generator distribution without directly estimating the entropy itself.

Original: All in all, the ob- jective of GANs is ﬁnd a good generator.
Revised: All in all, the ob- jective of GANs is ﬁnd a good generator.

Original: The motive of functional gradient method is pay attention to the update of the generator in the space of generator functions itself, instead of the updates in the parameter space.
Revised: The motive of functional gradient method is pay attention to the update of the generator in the space of generator functions itself, instead of the updates in the parameter space.

Original: In the function space, the functional properties of the generators are generally easier expressed than in the parameter space.
Revised: In the function space, the functional properties of the generators are generally easier expressed than in the parameter space.

Original: As such, the search for a generator with good functional properties may be much easier in the space of functions, as opposed to the space of parameters.
Revised: As such, the search for a generator with good functional properties may be much easier in the space of functions, as opposed to the space of parameters.

Original: Throughout, this philosophy will serve as the basis of our method.
Revised: Throughout, this philosophy will serve as the basis of our method.

Original: In more general technical term, the functional gradient of an objective function with respect to a model function is an inﬁnite dimensional gradient computed over the inﬁnite dimensional space of all models.
Revised: In more general technical term, the functional gradient of an objective function with respect to a model function is an inﬁnite dimensional gradient computed over the inﬁnite dimensional space of all models.

Original: In our case, our interest is the functional gradient of the GANs’ objective function with respect to the generator function.
Revised: In our case, our interest is the functional gradient of the GANs’ objective function with respect to the generator function.

Original: As we will show, all variations of GANs to date are implicitly using the discriminator function to compute the functional derivative of the objective function with respect to the generator distribution function.
Revised: As we will show, all variations of GANs to date are implicitly using the discriminator function to compute the functional derivative of the objective function with respect to the generator distribution function.

Original: In other words, the discriminator determines the direction to which the algorithm should move the mass of the current generator distribution; the discriminator determines what the next (target) distribution looks like.
Revised: In other words, the discriminator determines the direction to which the algorithm should move the mass of the current generator distribution; the discriminator determines what the next (target) distribution looks like.

Original: The work of Nitanda & Suzuki (2018) is a pioneer study that explicitly incorporated this idea into the training of neural generator models.
Revised: The work of Nitanda & Suzuki (2018) is a pioneer study that explicitly incorporated this idea into the training of neural generator models.

Original: Their study showed that one can carry out a faithful functional gradient-based update by inserting what they call gradient layer into the layers of neural generator function.
Revised: Their study showed that one can carry out a faithful functional gradient-based update by inserting what they call gradient layer into the layers of neural generator function.

Original: Johnson & Zhang (2018) further polished this strategy by periodically distilling the networks.
Revised: Johnson & Zhang (2018) further polished this strategy by periodically distilling the networks.

Original: A more precisely worded advantage of the functional gradient method is that, at every stage in the training process of the generator distribution, it allows the user to monitor what the next distribution (target distribution) looks like in terms of the current distribution.
Revised: A more precisely worded advantage of the functional gradient method is that, at every stage in the training process of the generator distribution, it allows the user to monitor what the next distribution (target distribution) looks like in terms of the current distribution.

Original: The ability of the functional gradient update to tell something about the next (target) distribution is a signiﬁcant advantage of the functional gradient method over the conventional parametric methods, because the target distribution set forth by conventional parametric methods is usually expressed in a complicated parametric form (e.g. Deep Neural Nets (DNNs)), and its behaviors are often difﬁcult to predict.
Revised: The ability of the functional gradient update to tell something about the next (target) distribution is a signiﬁcant advantage of the functional gradient method over the conventional parametric methods, because the target distribution set forth by conventional parametric methods is usually expressed in a complicated parametric form (e.g. Deep Neural Nets (DNNs)), and its behaviors are often difﬁcult to predict.

Original: This advantage of the functional gradient based-update suggests the possibility that we can control the update rule in order to deliberately direct the generator distribution to a distribution with preferred properties– which, in our case, include high entropy .
Revised: This advantage of the functional gradient based-update suggests the possibility that we can control the update rule in order to deliberately direct the generator distribution to a distribution with preferred properties– which, in our case, include high entropy .

Original: We discovered that, by locally concavifying the discriminator function, we can manipulate the functional gradient so that the next update for the generator will always target a distribution with higher entropy.
Revised: We discovered that, by locally concavifying the discriminator function, we can manipulate the functional gradient so that the next update for the generator will always target a distribution with higher entropy.

Original: From now on, we should refer to our method by Distributional Concavity (DC) regularization .
Revised: From now on, we should refer to our method by Distributional Concavity (DC) regularization .

Original: Our method does not require the direct estimation of the entropy.
Revised: Our method does not require the direct estimation of the entropy.

Original: We will not only show that our DC regularization can help improve the performance of GANs in terms of Inception score (Salimans et al., 2016) and Fr ´ echet inception distance (FID) (Heusel et al., 2017), but also give a mathematical guarantee that our regularization will always increase the entropy of the generator distribution.
Revised: We will not only show that our DC regularization can help improve the performance of GANs in terms of Inception score (Salimans et al., 2016) and Fr ´ echet inception distance (FID) (Heusel et al., 2017), but also give a mathematical guarantee that our regularization will always increase the entropy of the generator distribution.

Original: We will also show that our method greatly outperforms the method of Dai et al.
Revised: We will also show that our method greatly outperforms the method of Dai et al.

Original: (2017) that directly estimates the entropy based on classically techniques.
Revised: (2017) that directly estimates the entropy based on classically techniques.

Original: The regularization strategy of monotonically increasing the entropy of the generator distribution over the course of its training is not without theoretical basis.
Revised: The regularization strategy of monotonically increasing the entropy of the generator distribution over the course of its training is not without theoretical basis.

Original: Our DC regularization has close relations to the theory of optimal transport.
Revised: Our DC regularization has close relations to the theory of optimal transport.

Original: We will show that, when the entropy of the true distribution is higher than the current generator distribution, the functional gradient that is properly derived from the optimal mass transport always increases the entropy of the distribution.
Revised: We will show that, when the entropy of the true distribution is higher than the current generator distribution, the functional gradient that is properly derived from the optimal mass transport always increases the entropy of the distribution.

Original: Moreover, the functional update used in our method is always a monotonic mapping, which also turns out to be one of the properties satisﬁed by the update with optimal transport.
Revised: Moreover, the functional update used in our method is always a monotonic mapping, which also turns out to be one of the properties satisﬁed by the update with optimal transport.

Original: This is a preferred property as well, because it tends to promote the smooth training of the generator.
Revised: This is a preferred property as well, because it tends to promote the smooth training of the generator.

Original: We summarize our contributions below:
Revised: We summarize our contributions below:

Original: • We propose an update for the generator of GANs that promotes the entropy of the generator without the need for an explicit estimation of the actual entropy.
Revised: • We propose an update for the generator of GANs that promotes the entropy of the generator without the need for an explicit estimation of the actual entropy.

Original: • We show that, when the entropy of the true distribution is larger than that of the current generator distribution, the functional gradient derived from the 2-Wasserstein ( W 2 ) optimal transport always increases the entropy of the distribution.
Revised: • We show that, when the entropy of the true distribution is larger than that of the current generator distribution, the functional gradient derived from the 2-Wasserstein ( W 2 ) optimal transport always increases the entropy of the distribution.

Original: • We provide a mathematical guarantee that our method increases the entropy of the generator distribution at every step.
Revised: • We provide a mathematical guarantee that our method increases the entropy of the generator distribution at every step.

Original: • We show that our method improves the results of GANs in terms of Inception score
Revised: • We show that our method improves the results of GANs in terms of Inception score

Original: and
Revised: and

Original: FID score.
Revised: FID score.

Original:  T HEORY 2. G ENERATIVE ADVERSARIAL NETWORKS
Revised:  T HEORY 2. G ENERATIVE ADVERSARIAL NETWORKS

Original: Let us ﬁrst review the formulation of the original GANs.
Revised: Let us ﬁrst review the formulation of the original GANs.

Original: Unless otherwise noted, let us use boldface capital letters to refer to a random variable, and lower case letter for its realization.
Revised: Unless otherwise noted, let us use boldface capital letters to refer to a random variable, and lower case letter for its realization.

Original: The training process of GANs (Goodfellow et al., 2014) is a two player min-max game in which the generator distribution µ θ is trained to minimize the divergence between the true distribution ν and the generator distribution µ θ measured by the current critic F , while the critic F is trained to most strongly discriminate µ θ from ν .
Revised: The training process of GANs (Goodfellow et al., 2014) is a two player min-max game in which the generator distribution µ θ is trained to minimize the divergence between the true distribution ν and the generator distribution µ θ measured by the current critic F , while the critic F is trained to most strongly discriminate µ θ from ν .

Original: Often, µ θ is produced by applying a paramatric generator function G θ to a random seed variable with some distribution µ z , and it satisﬁes
Revised: Often, µ θ is produced by applying a paramatric generator function G θ to a random seed variable with some distribution µ z , and it satisﬁes

Original: [Equation]
Revised: [Equation]

Original: for all measurable statistics h .
Revised: for all measurable statistics h .

Original: In mathematical language, µ θ is a pushforward of µ z , and is often written as G θ # µ z .
Revised: In mathematical language, µ θ is a pushforward of µ z , and is often written as G θ # µ z .

Original: In a nutshell, GANs aim to ﬁnd an optimal parametric distribution µ θ that achieves the minimum value for
Revised: In a nutshell, GANs aim to ﬁnd an optimal parametric distribution µ θ that achieves the minimum value for

Original: [Equation]
Revised: [Equation]

Original: where L r and L g are the functions of user’s choice.
Revised: where L r and L g are the functions of user’s choice.

Original: We can retrieve the formulation of the original GANs (Goodfellow et al., 2014) by letting L r ( F ( x ))
Revised: We can retrieve the formulation of the original GANs (Goodfellow et al., 2014) by letting L r ( F ( x ))

Original: =
Revised: =

Original: softplus ( − F ( x )) and L g ( F ( x ))
Revised: softplus ( − F ( x )) and L g ( F ( x ))

Original: =
Revised: =

Original: softplus ( F ( x )) , where softplus ( · ) =
Revised: softplus ( F ( x )) , where softplus ( · ) =

Original: log(1 + exp( · ))
Revised: log(1 + exp( · ))

Original: There is a variety of choices for ( L r , L g )
Revised: There is a variety of choices for ( L r , L g )

Original: (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016).
Revised: (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016).

Original: Rewritten as the optimization problem about the function G , our objective function is given by
Revised: Rewritten as the optimization problem about the function G , our objective function is given by

Original: [Equation]
Revised: [Equation]

Original: 2. F UNCTIONAL G RADIENT INTERPRETATION OF THE GAN S UPDATE
Revised: 2. F UNCTIONAL G RADIENT INTERPRETATION OF THE GAN S UPDATE

Original: Let us elaborate further on the mechanism of the functional-gradient based update and its connection to the conventional update of GANs .
Revised: Let us elaborate further on the mechanism of the functional-gradient based update and its connection to the conventional update of GANs .

Original: For ease of notation, let us write L =
Revised: For ease of notation, let us write L =

Original: L g ◦ F in the equation (3), where the operator ◦ designates the function-composition.
Revised: L g ◦ F in the equation (3), where the operator ◦ designates the function-composition.

Original: Let us remind ourselves that the objective of the min part of the min-max game in each step is to ﬁnd a good update of µ θ that decreases E µ θ [ L ( X )] .
Revised: Let us remind ourselves that the objective of the min part of the min-max game in each step is to ﬁnd a good update of µ θ that decreases E µ θ [ L ( X )] .

Original: It is no exaggeration to say that the choice of the next target distribution of µ θ —or the distribution to which the µ θ will be updated—will completely determine the training efﬁciency of GANs.
Revised: It is no exaggeration to say that the choice of the next target distribution of µ θ —or the distribution to which the µ θ will be updated—will completely determine the training efﬁciency of GANs.

Original: The most canonical and obvious approach is to use the information of the discriminator L in making this decision.
Revised: The most canonical and obvious approach is to use the information of the discriminator L in making this decision.

Original: Note that, by appealing to a standard argument based on
Revised: Note that, by appealing to a standard argument based on

Original: Taylor expansion, L ( x ) ≥ L ( x − α ∇ L ( x )) for any value of x when α is sufﬁently small.
Revised: Taylor expansion, L ( x ) ≥ L ( x − α ∇ L ( x )) for any value of x when α is sufﬁently small.

Original: Therefore, leaving the mathematical technicalities aside, one may expect that a carefully chosen α can ensure
Revised: Therefore, leaving the mathematical technicalities aside, one may expect that a carefully chosen α can ensure

Original: [Equation]
Revised: [Equation]

Original: One naive suggestion based on this intuition is the update of µ θ to a distribution that can be constructed by taking a sample from µ θ (say, x ) and transporting it into the direction of − α ∇ L ( x ) .
Revised: One naive suggestion based on this intuition is the update of µ θ to a distribution that can be constructed by taking a sample from µ θ (say, x ) and transporting it into the direction of − α ∇ L ( x ) .

Original: Using the pushforward notation, this amounts to the update from µ θ to ( Id − α ∇ )# µ θ .
Revised: Using the pushforward notation, this amounts to the update from µ θ to ( Id − α ∇ )# µ θ .

Original: The presumed relation (4) is equivalent to
Revised: The presumed relation (4) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: X ))] , and the corresponding suggested update in terms of the function G θ is an update from G θ to ( Id − α ∇ L ) ◦
Revised: X ))] , and the corresponding suggested update in terms of the function G θ is an update from G θ to ( Id − α ∇ L ) ◦

Original: G θ .
Revised: G θ .

Original: That is, our suggestion amounts to the update from G θ # µ z to (( Id − α ∇ L ) ◦ G θ )#
Revised: That is, our suggestion amounts to the update from G θ # µ z to (( Id − α ∇ L ) ◦ G θ )#

Original: [Equation]
Revised: [Equation]

Original: z .
Revised: z .

Original: Denoting
Revised: Denoting

Original: [Equation]
Revised: [Equation]

Original: This is an update from G θ # µ z to ( T α ◦ G θ )
Revised: This is an update from G θ # µ z to ( T α ◦ G θ )

Original: # µ z .
Revised: # µ z .

Original: This update of G is called functional gradient update, and T α is called a transport function.
Revised: This update of G is called functional gradient update, and T α is called a transport function.

Original: With enough (reasonable) regularity assumption of the function space and probability space, one can justify the argument we have made above using a differential calculus in an inﬁnite dimensional space of functions(See appendix A).
Revised: With enough (reasonable) regularity assumption of the function space and probability space, one can justify the argument we have made above using a differential calculus in an inﬁnite dimensional space of functions(See appendix A).

Original: As a spoiler for what we will further elaborate later, our method regularizes L so that the target distribution ( T α ◦ G θ )# µ z has higher entropy than µ θ =
Revised: As a spoiler for what we will further elaborate later, our method regularizes L so that the target distribution ( T α ◦ G θ )# µ z has higher entropy than µ θ =

Original: G θ # µ z .
Revised: G θ # µ z .

Original: We are, however, still not done in our description of a functional gradient perspective of the GANs update.
Revised: We are, however, still not done in our description of a functional gradient perspective of the GANs update.

Original: In what we formulated above, there is no guarantee that the target distribution ( T α ◦ G θ )
Revised: In what we formulated above, there is no guarantee that the target distribution ( T α ◦ G θ )

Original: # µ z admits the same parametric representation as G θ ; if ( T α ◦ G θ )# µ z cannot be expressed by the DNN that we have prepared for the training, all suggestions we have made above is for naught.
Revised: # µ z admits the same parametric representation as G θ ; if ( T α ◦ G θ )# µ z cannot be expressed by the DNN that we have prepared for the training, all suggestions we have made above is for naught.

Original: The ﬁnal remaining task in this update procedure is therefore to ﬁnd the parameter θ ∗ such that G θ ∗ # µ z can best approximate the target distribution.
Revised: The ﬁnal remaining task in this update procedure is therefore to ﬁnd the parameter θ ∗ such that G θ ∗ # µ z can best approximate the target distribution.

Original: Letting θ old to denote the current choice of θ for the generator function, this can be done by solving the following optimization problem about θ :
Revised: Letting θ old to denote the current choice of θ for the generator function, this can be done by solving the following optimization problem about θ :

Original: [Equation]
Revised: [Equation]

Original: The gradient of this sub-objective function (6) with respect to θ evaluated at θ = θ old is given by
Revised: The gradient of this sub-objective function (6) with respect to θ evaluated at θ = θ old is given by

Original: [Equation]
Revised: [Equation]

Original: where ∂ θ designates the derivative operator with respect to θ .
Revised: where ∂ θ designates the derivative operator with respect to θ .

Original: This formulation is practically equivalent to the one introduced in xICFG (Johnson & Zhang, 2018).
Revised: This formulation is practically equivalent to the one introduced in xICFG (Johnson & Zhang, 2018).

Original: This turns out to be the familiar gradient update used in the usual GAN implementation.
Revised: This turns out to be the familiar gradient update used in the usual GAN implementation.

Original: Indeed, in the usual implementation, the parameter for the generator is updated with the rule:
Revised: Indeed, in the usual implementation, the parameter for the generator is updated with the rule:

Original: [Equation]
Revised: [Equation]

Original: In general, almost all variations of GANs to date (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016) uses this type of update rule for the training of the generator.
Revised: In general, almost all variations of GANs to date (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016) uses this type of update rule for the training of the generator.

Original: In other words, all methods to date has been implicitly doing the functional-gradient type update all along, and the choice of L has been determining the choice of the target distribution.
Revised: In other words, all methods to date has been implicitly doing the functional-gradient type update all along, and the choice of L has been determining the choice of the target distribution.

Original: As hinted a moment ago, this suggests that, by directly regularizing the L in the usual update scheme of GANs, one can realize a functional gradient update of the generator with a controlled target distribution.
Revised: As hinted a moment ago, this suggests that, by directly regularizing the L in the usual update scheme of GANs, one can realize a functional gradient update of the generator with a controlled target distribution.

Original: This is the very gist of our algorithm.
Revised: This is the very gist of our algorithm.

Original: 2. C HOICE OF THE TARGET DISTRIBUTION
Revised: 2. C HOICE OF THE TARGET DISTRIBUTION

Original: As inferred above, from the perspective of functional gradient, the min-max game of GANs can be decomposed into steps: (i) the target construction step that constructs the next target distribution using the functional gradient, and (ii) the distillation step that looks for a neural function that better approximates the target distribution.
Revised: As inferred above, from the perspective of functional gradient, the min-max game of GANs can be decomposed into steps: (i) the target construction step that constructs the next target distribution using the functional gradient, and (ii) the distillation step that looks for a neural function that better approximates the target distribution.

Original: Now, the next natural pressing question is: ”what type of L should we use to make sure that the next target distribution is nice?”
Revised: Now, the next natural pressing question is: ”what type of L should we use to make sure that the next target distribution is nice?”

Original: As inferred in the introduction, the goal of this particular study is to create a sequence of updates that is less likely to suffer from mode collapse.
Revised: As inferred in the introduction, the goal of this particular study is to create a sequence of updates that is less likely to suffer from mode collapse.

Original: Let us recall that, as long as we follow the standard update procedure of GANs, the choice of L will entirely determine the property of the next target distribution.
Revised: Let us recall that, as long as we follow the standard update procedure of GANs, the choice of L will entirely determine the property of the next target distribution.

Original: A proposal we would like to make in this study is to simply choose the discriminator
Revised: A proposal we would like to make in this study is to simply choose the discriminator

Original: L from the set functions that are concave on the support of the current distribution;
Revised: L from the set functions that are concave on the support of the current distribution;

Original: Proposition 2.1.
Revised: Proposition 2.1.

Original: Let µ be a probability distribution on R d .
Revised: Let µ be a probability distribution on R d .

Original: If L is concave on the support of µ , then H ( T α # µ ) ≥ H ( µ ) .
Revised: If L is concave on the support of µ , then H ( T α # µ ) ≥ H ( µ ) .

Original: Note that this statement is independent of the step size α , because any positive scalar multiple of concave function is concave.
Revised: Note that this statement is independent of the step size α , because any positive scalar multiple of concave function is concave.

Original: This result ensures that any update with a concave L will always increase the entropy; that is, the target distribution will be more dispersed than the current distribution, and the mode collapse is less likely to happen.
Revised: This result ensures that any update with a concave L will always increase the entropy; that is, the target distribution will be more dispersed than the current distribution, and the mode collapse is less likely to happen.

Original: Additionally, our choice of L guarantees that the transport function used to created the target distribution satisﬁes another preferable property called monotonicity.
Revised: Additionally, our choice of L guarantees that the transport function used to created the target distribution satisﬁes another preferable property called monotonicity.

Original: Monotonicity is a property that requires that there is no crossings in the transport:
Revised: Monotonicity is a property that requires that there is no crossings in the transport:

Original: [Equation]
Revised: [Equation]

Original: This is a preferred property because, as we will empirically show (see section 4.1) , the crossings in the transport tend to hinder the smooth training process.
Revised: This is a preferred property because, as we will empirically show (see section 4.1) , the crossings in the transport tend to hinder the smooth training process.

Original: This is in fact somewhat intuitive, because crossings lead to wasteful transportation of the mass.
Revised: This is in fact somewhat intuitive, because crossings lead to wasteful transportation of the mass.

Original: In fact, this is a property that is achieved by the distributional update based on Optimal transport as well (Villani, 2008).
Revised: In fact, this is a property that is achieved by the distributional update based on Optimal transport as well (Villani, 2008).

Original: Indeed, by the deﬁnition, the concavity of L implies the strong convexity of x 2 2 − αL ( x ) , which in turn implies
Revised: Indeed, by the deﬁnition, the concavity of L implies the strong convexity of x 2 2 − αL ( x ) , which in turn implies

Original: [Equation]
Revised: [Equation]

Original: which is a stronger condition than the monotonicity.
Revised: which is a stronger condition than the monotonicity.

Original: Thus, by simply making L concave, we can not only construct a target distribution whose entropy is greater than that of the current generator distribution, but also assure that the corresponding transport is monotonic.
Revised: Thus, by simply making L concave, we can not only construct a target distribution whose entropy is greater than that of the current generator distribution, but also assure that the corresponding transport is monotonic.

Original: In a way, this is a statement about how complicated the function is, because the presence of the crossings imply the existence of a discontinuity or a region of one to many mappings.
Revised: In a way, this is a statement about how complicated the function is, because the presence of the crossings imply the existence of a discontinuity or a region of one to many mappings.

Original: This is therefore a property that is likely to affect the distillation step.
Revised: This is therefore a property that is likely to affect the distillation step.

Original: In fact, we will empirically show that the property of monotonicity affects the distillation step in positive way.
Revised: In fact, we will empirically show that the property of monotonicity affects the distillation step in positive way.

Original: Indeed, we do not intend say that our suggestion for the property of L is optimal—the user has the freedom to choose the set of properties to be required for L so that the corresponding target distribution will have the desired properties that serves the purpose of the user.
Revised: Indeed, we do not intend say that our suggestion for the property of L is optimal—the user has the freedom to choose the set of properties to be required for L so that the corresponding target distribution will have the desired properties that serves the purpose of the user.

Original: 2. R ELATION TO OPTIMAL TRANSPORT
Revised: 2. R ELATION TO OPTIMAL TRANSPORT

Original: Our strategy for the distributional update that we have discussed so far has close relations to the optimal transport.
Revised: Our strategy for the distributional update that we have discussed so far has close relations to the optimal transport.

Original: In fact, an equation of the form (5) is ubiquitous in the theory of optimal transport.
Revised: In fact, an equation of the form (5) is ubiquitous in the theory of optimal transport.

Original: In this section, we will show the following:
Revised: In this section, we will show the following:

Original: By choosing L to be concave, the transport T α in the equation (5) becomes the optimal transport from the current distribution µ to the target distribution T α # µ . 2.
Revised: By choosing L to be concave, the transport T α in the equation (5) becomes the optimal transport from the current distribution µ to the target distribution T α # µ . 2.

Original: If the entropy of the true distribution ν is greater than that of the current distribution µ , any sequence of the updates of the distribution along the optimal transport from µ to ν increases the entropy monotonically.
Revised: If the entropy of the true distribution ν is greater than that of the current distribution µ , any sequence of the updates of the distribution along the optimal transport from µ to ν increases the entropy monotonically.

Original: We can also ensure the monotonic increase in the entropy by using T α with concave L . 3.
Revised: We can also ensure the monotonic increase in the entropy by using T α with concave L . 3.

Original: By choosing L to be concave, we can assure that the target distribution constructed from T α will always be within α distance of the current distribution in Wasserstein ( W 2 ) sense.
Revised: By choosing L to be concave, we can assure that the target distribution constructed from T α will always be within α distance of the current distribution in Wasserstein ( W 2 ) sense.

Original: In order to elaborate on these points, we would like to brieﬂy introduce the notion of optimal transport.
Revised: In order to elaborate on these points, we would like to brieﬂy introduce the notion of optimal transport.

Original: For a more rigorous introduction of the concept, please consult Villani (2008).
Revised: For a more rigorous introduction of the concept, please consult Villani (2008).

Original: For p ≥ 1 , the p -Wasserstein distance W p ( µ, ν ) between two arbitrary distributions µ and ν with sufﬁcient regularity is given by (cid:16)
Revised: For p ≥ 1 , the p -Wasserstein distance W p ( µ, ν ) between two arbitrary distributions µ and ν with sufﬁcient regularity is given by (cid:16)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the inf above is taken over all measurable maps T satisfying T # µ = ν .
Revised: where the inf above is taken over all measurable maps T satisfying T # µ = ν .

Original: When p = 2 , it is known that the inﬁmum of this cost function is achieved by T ∗ that can be expressed as x −∇ L ∗ for some L ∗ that renders (cid:107) x (cid:107) 2 / 2 − L ∗ convex (Brenier, 1991).
Revised: When p = 2 , it is known that the inﬁmum of this cost function is achieved by T ∗ that can be expressed as x −∇ L ∗ for some L ∗ that renders (cid:107) x (cid:107) 2 / 2 − L ∗ convex (Brenier, 1991).

Original: The search for L ∗ and hence T ∗ therefore amounts to the search for the closest coupling between µ and ν in the L 2 sense.
Revised: The search for L ∗ and hence T ∗ therefore amounts to the search for the closest coupling between µ and ν in the L 2 sense.

Original: Another surprising fact is that the movement of the particle along the direction of ∇ L ∗ monotonically decreases W 2 (Villani, 2008).
Revised: Another surprising fact is that the movement of the particle along the direction of ∇ L ∗ monotonically decreases W 2 (Villani, 2008).

Original: That is, if T ∗ t ( x ) = x
Revised: That is, if T ∗ t ( x ) = x

Original: − t ∇ L ∗ ( x ) , then W 2 ( T ∗ t # µ, ν ) monotonically decreases with t (Villani, 2003).
Revised: − t ∇ L ∗ ( x ) , then W 2 ( T ∗ t # µ, ν ) monotonically decreases with t (Villani, 2003).

Original: Please compare this transport equation with the equation (5).
Revised: Please compare this transport equation with the equation (5).

Original: This T ∗ t is indeed the optimal transportation analogue of the functional gradient.
Revised: This T ∗ t is indeed the optimal transportation analogue of the functional gradient.

Original: The theorem in Brenier (1991) has a still surprising converse; any function T that can be written as a gradient of a strictly convex function turns out to be the unique optimal transport from µ to T # µ .
Revised: The theorem in Brenier (1991) has a still surprising converse; any function T that can be written as a gradient of a strictly convex function turns out to be the unique optimal transport from µ to T # µ .

Original: Using this fact, we can appeal to the proposition 2.2 below to claim that the functional update based on the optimal transport satisﬁes the following important property that shall be intuitively fulﬁlled if we are in fact moving the distribution toward the true distribution: if the current distribution has lower entropy than the true distribution, the sequence of the distributions produced by the optimal transport based updates should be monotonically increasing in the entropy:
Revised: Using this fact, we can appeal to the proposition 2.2 below to claim that the functional update based on the optimal transport satisﬁes the following important property that shall be intuitively fulﬁlled if we are in fact moving the distribution toward the true distribution: if the current distribution has lower entropy than the true distribution, the sequence of the distributions produced by the optimal transport based updates should be monotonically increasing in the entropy:

Original: Proposition 2.2.
Revised: Proposition 2.2.

Original: Suppose ν = T # µ for some T that can be written as a gradient of a strictly convex function.
Revised: Suppose ν = T # µ for some T that can be written as a gradient of a strictly convex function.

Original: If H ( ν ) ≥ H ( µ ) , then H ( T t # µ ) is monotonically increasing on t ∈ [0 , 1] .
Revised: If H ( ν ) ≥ H ( µ ) , then H ( T t # µ ) is monotonically increasing on t ∈ [0 , 1] .

Original: This proposition actually assures that the updates of DC regularization also satisfy the same property.
Revised: This proposition actually assures that the updates of DC regularization also satisfy the same property.

Original: Because we are designing the target distribution so that it will have higher entropy than the current distribution at every step, by letting ν in the proposition 2.2 to be the target distribution, we have a guarantee that the sequence of distributions constructed by our updates is also monotonically increasing in the entropy.
Revised: Because we are designing the target distribution so that it will have higher entropy than the current distribution at every step, by letting ν in the proposition 2.2 to be the target distribution, we have a guarantee that the sequence of distributions constructed by our updates is also monotonically increasing in the entropy.

Original: To see this, simply note that we can automatically make (cid:107) x (cid:107) 2 / 2 − L to be convex by making L concave,
Revised: To see this, simply note that we can automatically make (cid:107) x (cid:107) 2 / 2 − L to be convex by making L concave,

Original: The connection between our DC regularization and optimal transport is not limited to the property we introduced above.
Revised: The connection between our DC regularization and optimal transport is not limited to the property we introduced above.

Original: By the theory of Brenir, the choice T α
Revised: By the theory of Brenir, the choice T α

Original: = Id − α ∇ L with concave and 1-Lipschitz L satisﬁes W 2 ( µ, T α # µ ) =
Revised: = Id − α ∇ L with concave and 1-Lipschitz L satisﬁes W 2 ( µ, T α # µ ) =

Original: (cid:112) E [ (cid:107) α ∇ L ( X ) (cid:107) 22 ] =
Revised: (cid:112) E [ (cid:107) α ∇ L ( X ) (cid:107) 22 ] =

Original: α.
Revised: α.

Original: Put in still other words, by constructing a target distribution with concave L , we can also assure that the target distribution is contained within the W 2 neighborhood of the current distribution function.
Revised: Put in still other words, by constructing a target distribution with concave L , we can also assure that the target distribution is contained within the W 2 neighborhood of the current distribution function.

Original: This is a property that cannot be guaranteed with the conventional parameter-based updates.
Revised: This is a property that cannot be guaranteed with the conventional parameter-based updates.

Original: Monotonicity condition is also a property that is satisﬁed by the optimal transport.
Revised: Monotonicity condition is also a property that is satisﬁed by the optimal transport.

Original: According the theory of Monge-Ampere (Villani (2008)), a transport map can be optimal only if it is monotonic.
Revised: According the theory of Monge-Ampere (Villani (2008)), a transport map can be optimal only if it is monotonic.

Original: Thus, while not being exactly based on the optimal transport from the generator distribution to the true distribution, our T shares many favorable properties in common with the optimal transport, and is very closely related to the theory of W 2 distance.
Revised: Thus, while not being exactly based on the optimal transport from the generator distribution to the true distribution, our T shares many favorable properties in common with the optimal transport, and is very closely related to the theory of W 2 distance.

Original: Reﬂecting on these facts, one might become tempted to say that we shall simply formulate GANs with the objective function that is solely based on W 2 distance between the generator distribution and the true distribution.
Revised: Reﬂecting on these facts, one might become tempted to say that we shall simply formulate GANs with the objective function that is solely based on W 2 distance between the generator distribution and the true distribution.

Original: However, as we will further articulate in the discussion section, the challenge remains to conduct faithful W 2 -based updates in the implementation of GANs.
Revised: However, as we will further articulate in the discussion section, the challenge remains to conduct faithful W 2 -based updates in the implementation of GANs.

Original:  M ETHOD
Revised:  M ETHOD

Original: If µ θ := G θ # µ z is our current parametric generator distribution and ν is the true distribution,
Revised: If µ θ := G θ # µ z is our current parametric generator distribution and ν is the true distribution,

Original: our
Revised: our

Original: DC regularization method ﬁrst (i) proposes a target distribution T # µ θ using a concave L , and then (ii) seek a measure µ θ new that well approximates the target distribution T # µ θ .
Revised: DC regularization method ﬁrst (i) proposes a target distribution T # µ θ using a concave L , and then (ii) seek a measure µ θ new that well approximates the target distribution T # µ θ .

Original: We use the following sampling-based penalty term in order to promote the concavity of L (= L g ◦ F ) :
Revised: We use the following sampling-based penalty term in order to promote the concavity of L (= L g ◦ F ) :

Original: [Equation]
Revised: [Equation]

Original: where x 1 , x 2 are samples from the support of µ θ , (cid:15) is a sample from the uniform distribution over [0 , 1] , and d is a positive scalar.
Revised: where x 1 , x 2 are samples from the support of µ θ , (cid:15) is a sample from the uniform distribution over [0 , 1] , and d is a positive scalar.

Original: Note that this term must be positive if L is concave over the support of µ θ .
Revised: Note that this term must be positive if L is concave over the support of µ θ .

Original: Our algorithm is summarized in Algorithm 1.
Revised: Our algorithm is summarized in Algorithm 1.

Original: The update rule in this algorithm uses the target distribution constructed by T ( x ) = x − ∇ L with concave L .
Revised: The update rule in this algorithm uses the target distribution constructed by T ( x ) = x − ∇ L with concave L .

Original: Intuitively speaking, the transport T has an effect of moving µ θ toward T # µ θ while dispersing the mass of µ θ .
Revised: Intuitively speaking, the transport T has an effect of moving µ θ toward T # µ θ while dispersing the mass of µ θ .

Original: See Fig 1 for a visual rendering of this interpretation.
Revised: See Fig 1 for a visual rendering of this interpretation.

Original: In most practical application, the training of GANs begins by dispersing the initial distribution with small entropy and gradually molds the mass into what resembles the true distribution.
Revised: In most practical application, the training of GANs begins by dispersing the initial distribution with small entropy and gradually molds the mass into what resembles the true distribution.

Original: Our transport ensures that this dispersion is consistently happening over the course of the training.
Revised: Our transport ensures that this dispersion is consistently happening over the course of the training.

Original: As we will show in the result section, this regularization works in favor of the inception score without any noticeable downfall from over-dispersion , suggesting that the generator distribution created with the current state of the art techniques are still not dispersed enough .
Revised: As we will show in the result section, this regularization works in favor of the inception score without any noticeable downfall from over-dispersion , suggesting that the generator distribution created with the current state of the art techniques are still not dispersed enough .

Original: Algorithm 1 GANs algorithm with DC regularization for each iteration do θ old ← θ the target construction step : ﬁnd F ∗ that optimizes
Revised: Algorithm 1 GANs algorithm with DC regularization for each iteration do θ old ← θ the target construction step : ﬁnd F ∗ that optimizes

Original: [Equation]
Revised: [Equation]

Original: (14) and construct T α from L :=
Revised: (14) and construct T α from L :=

Original: L g ◦ F as in the equation (5).
Revised: L g ◦ F as in the equation (5).

Original: the distillation step : update generator by generator’s objective
Revised: the distillation step : update generator by generator’s objective

Original: [Equation]
Revised: [Equation]

Original: (15) end for
Revised: (15) end for

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We applied DC regularization to the training of GANs on CIFAR-10 (Torralba et al., 2008), CIFAR100 (Torralba et al., 2008) and ILSVRC2012 dataset (ImageNet) (Russakovsky et al., 2015) in various settings and evaluated its performance in terms of Inception score (Salimans et al., 2016) and
Revised: We applied DC regularization to the training of GANs on CIFAR-10 (Torralba et al., 2008), CIFAR100 (Torralba et al., 2008) and ILSVRC2012 dataset (ImageNet) (Russakovsky et al., 2015) in various settings and evaluated its performance in terms of Inception score (Salimans et al., 2016) and

Original: Fr´echet inception distance (FID) (Heusel et al., 2017).
Revised: Fr´echet inception distance (FID) (Heusel et al., 2017).

Original: Inception score and FID are performance measures that are commonly used to evaluate the severity of mode collapse.
Revised: Inception score and FID are performance measures that are commonly used to evaluate the severity of mode collapse.

Original: For the details of the evaluation, see Appendix C.1.
Revised: For the details of the evaluation, see Appendix C.1.

Original: We also conducted an additional set of experiments with artiﬁcal
Revised: We also conducted an additional set of experiments with artiﬁcal

Original: [Figure]
Revised: [Figure]

Original: dataset to investigate the properties of DC-regularization.
Revised: dataset to investigate the properties of DC-regularization.

Original: We provide the results for additional experiments in the Appendix D as well.
Revised: We provide the results for additional experiments in the Appendix D as well.

Original: 4. I NTRINSIC PROPERTIES OF DC REGULARIZATION Effect of DC regularization on entropy
Revised: 4. I NTRINSIC PROPERTIES OF DC REGULARIZATION Effect of DC regularization on entropy

Original: Using a simple Gaussian Mixture Model with ﬁve modes as the true distribution, we evaluated the sheer ability of the DC regularization to promote the entropy of the generator distribution.
Revised: Using a simple Gaussian Mixture Model with ﬁve modes as the true distribution, we evaluated the sheer ability of the DC regularization to promote the entropy of the generator distribution.

Original: We used hinge loss for the objective function, and used DNNs to model both the discriminator and the generator.
Revised: We used hinge loss for the objective function, and used DNNs to model both the discriminator and the generator.

Original: We trained the model with and without the DC regularization and reported the entropy of the Generator at different stages of the training by explicitly computing the determinant of the
Revised: We trained the model with and without the DC regularization and reported the entropy of the Generator at different stages of the training by explicitly computing the determinant of the

Original: Jacobian at each layer.
Revised: Jacobian at each layer.

Original: As one of the baseline, we trained EGAN-Ent-VI (Dai et al., 2017), which includes into its objective function a penalty against the negative entropy of the generator.
Revised: As one of the baseline, we trained EGAN-Ent-VI (Dai et al., 2017), which includes into its objective function a penalty against the negative entropy of the generator.

Original: The result is illustrated in Fig 2 (a).
Revised: The result is illustrated in Fig 2 (a).

Original: We see that the regularization is positively affecting the entropy at all stages of the training.
Revised: We see that the regularization is positively affecting the entropy at all stages of the training.

Original: Although to a less extent, we can conﬁrm that our implementation of EGAN-Ent-VI is also preventing the degeneration of the entropy.
Revised: Although to a less extent, we can conﬁrm that our implementation of EGAN-Ent-VI is also preventing the degeneration of the entropy.

Original: Indeed, the persistent pressure to increase the entropy can result in over-dispsersed ﬁnal product.
Revised: Indeed, the persistent pressure to increase the entropy can result in over-dispsersed ﬁnal product.

Original: However, this seems not to be a serious issue when it comes to the learning on big data like ImageNet.
Revised: However, this seems not to be a serious issue when it comes to the learning on big data like ImageNet.

Original: In terms of Inception score and FID score, all artiﬁcial generator distributions today are still far less diverse than the original dataset (Table 1), and there is still a large room left for the improvement of diversity.
Revised: In terms of Inception score and FID score, all artiﬁcial generator distributions today are still far less diverse than the original dataset (Table 1), and there is still a large room left for the improvement of diversity.

Original: Effect of monotonicity in distillation step
Revised: Effect of monotonicity in distillation step

Original: Recall that each round of GANs update consists of the target construction step and the distillation step.
Revised: Recall that each round of GANs update consists of the target construction step and the distillation step.

Original: We conducted an experiment to verify the effect of the monotonicty on the distillation step.
Revised: We conducted an experiment to verify the effect of the monotonicty on the distillation step.

Original: We will show a case in which, even if the target distribution constructed from a non-monotonic map is further away from the current distribution than the target distribution constructed from a monotonic map, the projection of the latter distribution onto the parametric function-space is much easier.
Revised: We will show a case in which, even if the target distribution constructed from a non-monotonic map is further away from the current distribution than the target distribution constructed from a monotonic map, the projection of the latter distribution onto the parametric function-space is much easier.

Original: Let K be a positive value.
Revised: Let K be a positive value.

Original: Starting from an initial distribution X =
Revised: Starting from an initial distribution X =

Original: G θ 

Revised: G θ

Original: ( Z ) , consider the following pair of maps to be applied to X :
Revised: ( Z ) , consider the following pair of maps to be applied to X :

Original: [Equation]
Revised: [Equation]

Original: It is evident that T (2) is a monotonic mapping and T (1) is not.
Revised: It is evident that T (2) is a monotonic mapping and T (1) is not.

Original: Denote the law of X by µ .
Revised: Denote the law of X by µ .

Original: We used T (1) # µ and T (2) # µ as target distributions, and trained the parameter θ using O m =
Revised: We used T (1) # µ and T (2) # µ as target distributions, and trained the parameter θ using O m =

Original: E Z [ (cid:107) T ( m ) ( G θ 

Revised: E Z [ (cid:107) T ( m ) ( G θ

Original: ( Z ))
Revised: ( Z ))

Original: − G θ ( Z ) (cid:107) 2 ] as the objective function (Distillation).
Revised: − G θ ( Z ) (cid:107) 2 ] as the objective function (Distillation).

Original: Note that this objective function takes the same value of K 2 for both m = 1 , 2 .
Revised: Note that this objective function takes the same value of K 2 for both m = 1 , 2 .

Original: Also, by the deﬁnition of the Wasserstein distance, K = W 2 ( T (2) # µ, µ ) while W 2 ( T (1) # µ, µ ) 2 = inf T # µ = T 1 # µ E µ [ (cid:107) T ( X ) − X (cid:107) 2 ] ≤
Revised: Also, by the deﬁnition of the Wasserstein distance, K = W 2 ( T (2) # µ, µ ) while W 2 ( T (1) # µ, µ ) 2 = inf T # µ = T 1 # µ E µ [ (cid:107) T ( X ) − X (cid:107) 2 ] ≤

Original: E µ [ (cid:107) T (1) ( X ) − X
Revised: E µ [ (cid:107) T (1) ( X ) − X

Original: (cid:107) 2 ]
Revised: (cid:107) 2 ]

Original: = K 2 .
Revised: = K 2 .

Original: A naive intuition dictates that the distillation of T (1) # µ is easier, because T (1) # µ is distributionally closer to µ while the objective function evaluated at T (1) # µ is same as the evaluation at T (2) # µ .
Revised: A naive intuition dictates that the distillation of T (1) # µ is easier, because T (1) # µ is distributionally closer to µ while the objective function evaluated at T (1) # µ is same as the evaluation at T (2) # µ .

Original: However, as we can see in Fig 2 (b), the training about Oproceeds much faster than the training about O 1 .
Revised: However, as we can see in Fig 2 (b), the training about Oproceeds much faster than the training about O 1 .

Original: This seemingly unintuitive observation can be supported by the theory of optimal transport.
Revised: This seemingly unintuitive observation can be supported by the theory of optimal transport.

Original: Note that T 1 is a derivative of − K | X | and T (2) is a derivative of K | X | , and the latter is a convex function.
Revised: Note that T 1 is a derivative of − K | X | and T (2) is a derivative of K | X | , and the latter is a convex function.

Original: This implies that T (2) is the optimal transport from µ to T (2) # µ .
Revised: This implies that T (2) is the optimal transport from µ to T (2) # µ .

Original: This result (Fig 2 (b)) suggests that, when using the update rule similar to the equation 10, the training proceeds much faster when we choose a target distribution with monotonic mapping.
Revised: This result (Fig 2 (b)) suggests that, when using the update rule similar to the equation 10, the training proceeds much faster when we choose a target distribution with monotonic mapping.

Original: [Figure]
Revised: [Figure]

Original: 4. R ESULTS ON CIFAR-10 AND CIFARExperiments with different architectures and objective functions
Revised: 4. R ESULTS ON CIFAR-10 AND CIFARExperiments with different architectures and objective functions

Original: We tested our algorithm for the training of GANs with six types of objective functions and two network architectures, and reported the performance on all 12 combinations.
Revised: We tested our algorithm for the training of GANs with six types of objective functions and two network architectures, and reported the performance on all 12 combinations.

Original: For the details of the objective functions and the architectures, please see Appendix C.2, C.3, C.4 For the training of the network, we applied Spectral Normalization (SN) (Miyato et al., 2018) to the full-connected layer and the convolution layer.
Revised: For the details of the objective functions and the architectures, please see Appendix C.2, C.3, C.4 For the training of the network, we applied Spectral Normalization (SN) (Miyato et al., 2018) to the full-connected layer and the convolution layer.

Original: Fig 3 and Fig 8 (in Appendix) respectively summarize Inception scores and FID for all 12 settings.
Revised: Fig 3 and Fig 8 (in Appendix) respectively summarize Inception scores and FID for all 12 settings.

Original: We see that DC regularization is improving the performance irrespective of the choice of the architecture and the objective function.
Revised: We see that DC regularization is improving the performance irrespective of the choice of the architecture and the objective function.

Original: Experiments with different prior dimensions
Revised: Experiments with different prior dimensions

Original: In general, without careful selection of the dimension of the prior distribution, the training of
Revised: In general, without careful selection of the dimension of the prior distribution, the training of

Original: GANs tends to suffer a serious case of mode collapse.
Revised: GANs tends to suffer a serious case of mode collapse.

Original: For image generation task, this will result in low inception score.
Revised: For image generation task, this will result in low inception score.

Original: We therefore applied DC regularization to the trainings of GANs on
Revised: We therefore applied DC regularization to the trainings of GANs on

Original: CIFAR-10 with very low prior dimensions as well as very large prior dimensions and evaluated the performance.
Revised: CIFAR-10 with very low prior dimensions as well as very large prior dimensions and evaluated the performance.

Original: For this experiment, we used GAN-variant2 (Appendix C.2) for the objective function and used SNDCGAN for the architecture.
Revised: For this experiment, we used GAN-variant2 (Appendix C.2) for the objective function and used SNDCGAN for the architecture.

Original: As for the experimental details, please see Appendix
Revised: As for the experimental details, please see Appendix

Original: C.4.
Revised: C.4.

Original: The results are summarized in Fig 4 and Fig 10 (in Appendix).
Revised: The results are summarized in Fig 4 and Fig 10 (in Appendix).

Original: When the prior dimension is below the inherent dimension of the dataset, the dimension of the generator distribution cannot match the dimension of the true distribution.
Revised: When the prior dimension is below the inherent dimension of the dataset, the dimension of the generator distribution cannot match the dimension of the true distribution.

Original: Thus, the inception score of the generator trained with low dimensional prior is bound to be low, irrespective of the application of DC regularization.
Revised: Thus, the inception score of the generator trained with low dimensional prior is bound to be low, irrespective of the application of DC regularization.

Original: However, for dim( z ) ≥ 5 , the inception score is as high as 7.5, and for all choices of dim( z ) , the generator trained with DC regularization consistently outperformed the generator trained without the DC regularization.
Revised: However, for dim( z ) ≥ 5 , the inception score is as high as 7.5, and for all choices of dim( z ) , the generator trained with DC regularization consistently outperformed the generator trained without the DC regularization.

Original: Similar argument applies to the performance of DC regularization for high dimensional prior.
Revised: Similar argument applies to the performance of DC regularization for high dimensional prior.

Original: Overall, the DC regularization provides some robustness against the choice of the prior dimension.
Revised: Overall, the DC regularization provides some robustness against the choice of the prior dimension.

Original: [Figure]
Revised: [Figure]

Original: improves the inception score for all methods.
Revised: improves the inception score for all methods.

Original: [Figure]
Revised: [Figure]

Original: I
Revised: I

Original: [Figure]
Revised: [Figure]

Original: Comparison with EGAN and other methods
Revised: Comparison with EGAN and other methods

Original: We compared the performance of our algorithm against EGAN-Ent-VI (Dai et al., 2017), another framework that can be used to control the entropy of the generator.
Revised: We compared the performance of our algorithm against EGAN-Ent-VI (Dai et al., 2017), another framework that can be used to control the entropy of the generator.

Original: We conducted this comparative study using SNDCGAN (Table 2) and SNResNet (Table 3), which uses Spectral Normalization that is known to have an effect of preventing the degeneration of the feature space.
Revised: We conducted this comparative study using SNDCGAN (Table 2) and SNResNet (Table 3), which uses Spectral Normalization that is known to have an effect of preventing the degeneration of the feature space.

Original: SNDCGAN is a method that is known to perform well on Cifar10 on its own.
Revised: SNDCGAN is a method that is known to perform well on Cifar10 on its own.

Original: For the experimental setting of this study, please see the Appendix C.4.
Revised: For the experimental setting of this study, please see the Appendix C.4.

Original: As we show in Table 1, DC regularization outperformed
Revised: As we show in Table 1, DC regularization outperformed

Original: EGAN-Ent-VI on both models.
Revised: EGAN-Ent-VI on both models.

Original: We reported the result of EGAN-Ent-VI with spectral normalization, because it worked better than the version without SN.
Revised: We reported the result of EGAN-Ent-VI with spectral normalization, because it worked better than the version without SN.

Original: As we can see in the table 1, the EGAN with
Revised: As we can see in the table 1, the EGAN with

Original: SN performed worse than the vanilla SNDCGAN.
Revised: SN performed worse than the vanilla SNDCGAN.

Original: For high dimensional dataset like Cifar10, the variational inference for the negative entropy can be extremely difﬁcult.
Revised: For high dimensional dataset like Cifar10, the variational inference for the negative entropy can be extremely difﬁcult.

Original: It is possible that a poor variational inference in high dimensional space backﬁred for EGAN-VI’s performance on Cifar10.
Revised: It is possible that a poor variational inference in high dimensional space backﬁred for EGAN-VI’s performance on Cifar10.

Original: We would also like to emphasize here that EGAN-Ent-VI requires a separate decoder in addition to the generator and the discriminator, and that our algorithm is easier to implement.
Revised: We would also like to emphasize here that EGAN-Ent-VI requires a separate decoder in addition to the generator and the discriminator, and that our algorithm is easier to implement.

Original: For the full version of the Table and the visuals of the generated samples, please see Table 8 and Table 11 in the Appendix.
Revised: For the full version of the Table and the visuals of the generated samples, please see Table 8 and Table 11 in the Appendix.

Original: We compared our algorithm against other competitive methods as well.
Revised: We compared our algorithm against other competitive methods as well.

Original: The best performance of our method is almost on par with the state-of-the-art method (Karras et al., 2018).
Revised: The best performance of our method is almost on par with the state-of-the-art method (Karras et al., 2018).

Original: We are losing to progressive GAN (Karras et al., 2018) by a slight margin; we, however, would like to make a disclaimer that we are using much smaller architecture than Karras et al.
Revised: We are losing to progressive GAN (Karras et al., 2018) by a slight margin; we, however, would like to make a disclaimer that we are using much smaller architecture than Karras et al.

Original: (2018) for the performance evaluation of our method.
Revised: (2018) for the performance evaluation of our method.

Original: We also conﬁrmed that we can improve the result of Miyato et al.
Revised: We also conﬁrmed that we can improve the result of Miyato et al.

Original: by using DC regularization together with SN.
Revised: by using DC regularization together with SN.

Original: The results support that our method is helping the training process suppress mode collapse and is improving the overall performance.
Revised: The results support that our method is helping the training process suppress mode collapse and is improving the overall performance.

Original: For the results of the experiments conducted with Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al., 2017), please see the Fig 9 in Appendix.
Revised: For the results of the experiments conducted with Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al., 2017), please see the Fig 9 in Appendix.

Original: 4. R ESULTS ON I MAGE N ET
Revised: 4. R ESULTS ON I MAGE N ET

Original: Additionaly, to evaluate our method’s effectiveness on a higher dimensional dataset, we applied our method on ImageNet with 1000 classes, with each class containing approximately 1300 images.
Revised: Additionaly, to evaluate our method’s effectiveness on a higher dimensional dataset, we applied our method on ImageNet with 1000 classes, with each class containing approximately 1300 images.

Original: We compressed the images to 64 × 64 pixels prior to the experiments.
Revised: We compressed the images to 64 × 64 pixels prior to the experiments.

Original: [Figure]
Revised: [Figure]

Original: For the objective function, we chose GAN-hinge (Appendix C.2), which was being used in Miyato et al.
Revised: For the objective function, we chose GAN-hinge (Appendix C.2), which was being used in Miyato et al.

Original: For the experimental settings, please see Appendix C.5 for the details.
Revised: For the experimental settings, please see Appendix C.5 for the details.

Original: We can conﬁrm on Fig 5 that our DC regularization is improving the Inception score throughout the course of the training.
Revised: We can conﬁrm on Fig 5 that our DC regularization is improving the Inception score throughout the course of the training.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION Dilemma of GANs update
Revised:  D ISCUSSION Dilemma of GANs update

Original: As we have shown above, the experimental results support our claim that the DC regularization promotes the entropy of the generator distribution and stabilizes the training process of GANs.
Revised: As we have shown above, the experimental results support our claim that the DC regularization promotes the entropy of the generator distribution and stabilizes the training process of GANs.

Original: As mentioned brieﬂy in the theory section 2, however, we do not have the guarantee that our update rule consistently reduces the W 2 distance between the generator distribution and the true distribution.
Revised: As mentioned brieﬂy in the theory section 2, however, we do not have the guarantee that our update rule consistently reduces the W 2 distance between the generator distribution and the true distribution.

Original: This fault, however, is common to almost all GANs algorithms today.
Revised: This fault, however, is common to almost all GANs algorithms today.

Original: As we have shown, the conventional update rule is implicitly targetting a distribution of the form T # µ where T = Id − α ∇ L for some α and a discriminator L .
Revised: As we have shown, the conventional update rule is implicitly targetting a distribution of the form T # µ where T = Id − α ∇ L for some α and a discriminator L .

Original: As it turns out, optimal transport takes this form only when we are optimizing W 2 distance, while the common constraint of |∇ L | = 1 is a condition required for dual potential when p = 1 ( W 1 ) (Villani, 2008).
Revised: As it turns out, optimal transport takes this form only when we are optimizing W 2 distance, while the common constraint of |∇ L | = 1 is a condition required for dual potential when p = 1 ( W 1 ) (Villani, 2008).

Original: In other words, the conventional GANs are making the discriminator with W 1 criteria and updating the generator with W 2 criteria.
Revised: In other words, the conventional GANs are making the discriminator with W 1 criteria and updating the generator with W 2 criteria.

Original: If we are to faithfully create the discriminator with W 2 criteria, we must look for a Legendre-pair of dual potential functions.
Revised: If we are to faithfully create the discriminator with W 2 criteria, we must look for a Legendre-pair of dual potential functions.

Original: On the contrary, if we are to update the generator with W 1 criteria, one must look for a closed form solution for the W 1 transport.
Revised: On the contrary, if we are to update the generator with W 1 criteria, one must look for a closed form solution for the W 1 transport.

Original: This, however, is in general a highly complex mathematical problem for which there is a separate ﬁeld of study (Santambrogio, 2015).
Revised: This, however, is in general a highly complex mathematical problem for which there is a separate ﬁeld of study (Santambrogio, 2015).

Original: To the authors’ best knowledge, no studies have provided a solid solution to this dilemma.
Revised: To the authors’ best knowledge, no studies have provided a solid solution to this dilemma.

Original: Convexity vs Strong convexity
Revised: Convexity vs Strong convexity

Original: We would like to also mention that our regularization is asking for more than what is required by
Revised: We would like to also mention that our regularization is asking for more than what is required by

Original: W 2 theory.
Revised: W 2 theory.

Original: As mentioned above, in order for T to be the optimal transport from µ to T # µ , T only needs to be the gradient of a convex function.
Revised: As mentioned above, in order for T to be the optimal transport from µ to T # µ , T only needs to be the gradient of a convex function.

Original: On the other hand, by asking L to be concave, we are in fact asking for T to be the gradient of a strongly convex function.
Revised: On the other hand, by asking L to be concave, we are in fact asking for T to be the gradient of a strongly convex function.

Original: This ”overdo” is actually intentional.
Revised: This ”overdo” is actually intentional.

Original: Recall that the parameter α in the transport T α
Revised: Recall that the parameter α in the transport T α

Original: = Id − α ∇ L corresponds to a step size in the update of the generator.
Revised: = Id − α ∇ L corresponds to a step size in the update of the generator.

Original: If we train L that only guarantees the convexity of (cid:107) x (cid:107) 2 / 2 − L ( x ) , the functional update derived from such L can be non-monotonic when the step size is large, because such L only guarantees the convexity of (cid:107) x (cid:107) 2 / 2 − αL ( x ) when α ≤ 1 .
Revised: If we train L that only guarantees the convexity of (cid:107) x (cid:107) 2 / 2 − L ( x ) , the functional update derived from such L can be non-monotonic when the step size is large, because such L only guarantees the convexity of (cid:107) x (cid:107) 2 / 2 − αL ( x ) when α ≤ 1 .

Original: Should we require L to be concave, however, (cid:107) x (cid:107) 2 / 2 − αL ( x ) is concave for any positive α .
Revised: Should we require L to be concave, however, (cid:107) x (cid:107) 2 / 2 − αL ( x ) is concave for any positive α .

Original: In this context, we may therefore say that the DC regularization leaves much room to be playful about the learning schedule.
Revised: In this context, we may therefore say that the DC regularization leaves much room to be playful about the learning schedule.

Original: Finally, as can be inferred from our proofs for the proposition 2.1 and 2.2, the strong convexity of T has an effect of diffusing the mass of pdf in every direction.
Revised: Finally, as can be inferred from our proofs for the proposition 2.1 and 2.2, the strong convexity of T has an effect of diffusing the mass of pdf in every direction.

Original: We can therefore expect the DC regularization to disperse the collapsed masses in the case of mode collapse.
Revised: We can therefore expect the DC regularization to disperse the collapsed masses in the case of mode collapse.

Original: In the light of the fact that the training of GANs usually begins with dense distribution with small support, this dispersion effect should be helpful at the early stage of the training as well.
Revised: In the light of the fact that the training of GANs usually begins with dense distribution with small support, this dispersion effect should be helpful at the early stage of the training as well.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We would like to thank the members of PFN.Inc, particularly Daisuke Okanohara, Kouhei Hayashi,
Revised: We would like to thank the members of PFN.Inc, particularly Daisuke Okanohara, Kouhei Hayashi,

Original: Masaki Watabnabe, Shin-ichi Maeda, Sosuke Kobayashi, Takeru Miyato and Toshiki Kataoka for helpful comments and advices
Revised: Masaki Watabnabe, Shin-ichi Maeda, Sosuke Kobayashi, Takeru Miyato, Kenta Oono and Toshiki Kataoka for helpful comments and advices.

Revised: We would also like to thank Atsushi Nitanda for constructive advices

Original: Appendix (Supplemental Material for Distributional Concavity Regularization for GANs) A M ORE MATHEMATICAL RENDITION OF S ECTION
Revised: Appendix (Supplemental Material for Distributional Concavity Regularization for GANs) A M ORE MATHEMATICAL RENDITION OF S ECTION

Original: In this section, we re-explain the section 2 with more mathematical details.
Revised: In this section, we re-explain the section 2 with more mathematical details.

Original: For ease of argument, we will assume the followings from here onward.
Revised: For ease of argument, we will assume the followings from here onward.

Original: First, as is the case in many application of GANs, we will assume that ν is a probability measure on the Euclidean space R d , and µ z is a probability measure on some Euclidean space U .
Revised: First, as is the case in many application of GANs, we will assume that ν is a probability measure on the Euclidean space R d , and µ z is a probability measure on some Euclidean space U .

Original: Let us also assume that G θ : U → R d is a parametric function of θ that is almost everywhere differentiable with respect to both input and θ .
Revised: Let us also assume that G θ : U → R d is a parametric function of θ that is almost everywhere differentiable with respect to both input and θ .

Original: We will ﬁrst consider the optimization problem of the equation (3) with gradient descent about θ .
Revised: We will ﬁrst consider the optimization problem of the equation (3) with gradient descent about θ .

Original: For ease of notation, let us write L =
Revised: For ease of notation, let us write L =

Original: L g ◦ F , where the operator ◦ designates the function-composition.
Revised: L g ◦ F , where the operator ◦ designates the function-composition.

Original: We will assume that L is a continuously differrentiable and uniformly bounded function on R d , and that the Euclidean norms of its gradient and operator norm of Hessian are also uniformly bounded.
Revised: We will assume that L is a continuously differrentiable and uniformly bounded function on R d , and that the Euclidean norms of its gradient and operator norm of Hessian are also uniformly bounded.

Original: This type of regularity assumptions on the objective functions are used in other literatures as well (Nitanda & Suzuki (2018), Johnson & Zhang (2018)), and it is not too all unrealistic because the support of both target distributions and generator distribution are often compact throughout the course of the training in the real applications.
Revised: This type of regularity assumptions on the objective functions are used in other literatures as well (Nitanda & Suzuki (2018), Johnson & Zhang (2018)), and it is not too all unrealistic because the support of both target distributions and generator distribution are often compact throughout the course of the training in the real applications.

Original: We will also assume that the derivatives of all functions to appear are well deﬁned and bounded on R d .
Revised: We will also assume that the derivatives of all functions to appear are well deﬁned and bounded on R d .

Original: The update rule of θ is given by:
Revised: The update rule of θ is given by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (18) where ∂ θ designates the derivative operator with respect to θ .
Revised: (18) where ∂ θ designates the derivative operator with respect to θ .

Original: In general, almost all variations of
Revised: In general, almost all variations of

Original: GANs to date (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016) uses this type of update rule for the training of the generator.
Revised: GANs to date (Arjovsky et al., 2017; Lim & Ye, 2017; Mao et al., 2017; Nowozin et al., 2016) uses this type of update rule for the training of the generator.

Original: Let us denote the law of G θ ( Z ) (or X ) by µ θ .
Revised: Let us denote the law of G θ ( Z ) (or X ) by µ θ .

Original: Let L 2 ( µ θ ) denote the Hilbert space of L 2 integrable maps from R d to itself, granted with the inner product (cid:104) a, b (cid:105) µ θ = E µ θ [ a ( X ) T b ( X )] .
Revised: Let L 2 ( µ θ ) denote the Hilbert space of L 2 integrable maps from R d to itself, granted with the inner product (cid:104) a, b (cid:105) µ θ = E µ θ [ a ( X ) T b ( X )] .

Original: We will show that we can re-derive the equation (18) using the theory of functional gradient.
Revised: We will show that we can re-derive the equation (18) using the theory of functional gradient.

Original: To begin with, instead of updating the parameter θ , we would like to consider directly updating the distribution µ θ by applying a T ∈ L 2 ( µ θ ) to a random variable generated from µ θ .
Revised: To begin with, instead of updating the parameter θ , we would like to consider directly updating the distribution µ θ by applying a T ∈ L 2 ( µ θ ) to a random variable generated from µ θ .

Original: This will result in a new distribution T # µ θ , which is a unique distribution that satisﬁes E µ θ [ h ( T ( X )]
Revised: This will result in a new distribution T # µ θ , which is a unique distribution that satisﬁes E µ θ [ h ( T ( X )]

Original: =
Revised: =

Original: E T # µ θ
Revised: E T # µ θ

Original: [ h ( X )] for all measurable h .
Revised: [ h ( X )] for all measurable h .

Original: Now, in order to create an update rule for T , let us consider the G ˆ ateaux derivative of M ( T ) :=
Revised: Now, in order to create an update rule for T , let us consider the G ˆ ateaux derivative of M ( T ) :=

Original: E T # µ θ
Revised: E T # µ θ

Original: [ L ( X )] with respect to T into an arbitrary direction
Revised: [ L ( X )] with respect to T into an arbitrary direction

Original: δ ∈ L 2 ( µ θ ) .
Revised: δ ∈ L 2 ( µ θ ) .

Original: In terse term, we would like to consider the effect of purturbing T into the direction of δ .
Revised: In terse term, we would like to consider the effect of purturbing T into the direction of δ .

Original: By the compactnesss assumption on Ω together with the regularity assumptions for the functions deﬁned hereof, we can appeal to the dominated convergence theorem and see that
Revised: By the compactnesss assumption on Ω together with the regularity assumptions for the functions deﬁned hereof, we can appeal to the dominated convergence theorem and see that

Original: [Equation]
Revised: [Equation]

Original: is uniform for all δ ∈ L 2 ( µ ) .
Revised: is uniform for all δ ∈ L 2 ( µ ) .

Original: This way, the term ∇ L ( T ( · )) in the expression above is a Fr ´ echet derivative, or the type of derivative to which the usual set of derivative rules can be applied.
Revised: This way, the term ∇ L ( T ( · )) in the expression above is a Fr ´ echet derivative, or the type of derivative to which the usual set of derivative rules can be applied.

Original: We will refer to this derivative as functional derivative for short.
Revised: We will refer to this derivative as functional derivative for short.

Original: Note that the directional functional derivative E µ θ [( ∇ L ( T ( X )) · δ )( · )] will always take a positive value when δ ∝ ∇ L ( T ( · )) .
Revised: Note that the directional functional derivative E µ θ [( ∇ L ( T ( X )) · δ )( · )] will always take a positive value when δ ∝ ∇ L ( T ( · )) .

Original: Thus, for α small enough, an update from the current choice of T to T − α ∇ L ∈ L 2 ( µ θ ) will decrease the objective function.
Revised: Thus, for α small enough, an update from the current choice of T to T − α ∇ L ∈ L 2 ( µ θ ) will decrease the objective function.

Original: Here, we are interested in the derivative computed at T
Revised: Here, we are interested in the derivative computed at T

Original: = Id , so the transformation we would like to apply to µ θ is
Revised: = Id , so the transformation we would like to apply to µ θ is

Original: [Equation]
Revised: [Equation]

Original: This is indeed a transportation of a mass into the direction of α ∇ L .
Revised: This is indeed a transportation of a mass into the direction of α ∇ L .

Original: Now, for the update of θ , we would like to design θ new such that µ θ new is closer to the target distribution T # µ θ old than µ θ old .
Revised: Now, for the update of θ , we would like to design θ new such that µ θ new is closer to the target distribution T # µ θ old than µ θ old .

Original: That is, we would like to choose θ that minimizes
Revised: That is, we would like to choose θ that minimizes

Original: [Equation]
Revised: [Equation]

Original: The gradient of this sub-objective function with respect to θ is given by
Revised: The gradient of this sub-objective function with respect to θ is given by

Original: [Equation]
Revised: [Equation]

Original: Evaluating this at θ = θ old , we recover the gradient used in the equation (18).
Revised: Evaluating this at θ = θ old , we recover the gradient used in the equation (18).

Original: This formulation is practically equivalent to the one introduced in xICFG (Johnson & Zhang, 2018).
Revised: This formulation is practically equivalent to the one introduced in xICFG (Johnson & Zhang, 2018).

Original: B T HE PROOFS FOR THE PROPOSITIONS 2.1 AND 2. 
Revised: B T HE PROOFS FOR THE PROPOSITIONS 2.1 AND 2.

Original: We will ﬁrst prove the proposition 2.2.
Revised: We will ﬁrst prove the proposition 2.2.

Original: Let us begin with a useful lemma.
Revised: Let us begin with a useful lemma.

Original: We will assume that all regularity assumptions made in Section 2 holds for all variables and functions that appear in this section.
Revised: We will assume that all regularity assumptions made in Section 2 holds for all variables and functions that appear in this section.

Original: Also, we will use the Df to denote the derivative of f .
Revised: Also, we will use the Df to denote the derivative of f .

Original: Also assume that, unless otherwise noted, Df ( x ) indicates D x f ( x ) , the derivative of f with respect to x .
Revised: Also assume that, unless otherwise noted, Df ( x ) indicates D x f ( x ) , the derivative of f with respect to x .

Original: Lemma B.1.
Revised: Lemma B.1.

Original: Let µ and ν be probability distributions on R n , and suppose that we can write ( Id − ∇ L )# µ
Revised: Let µ and ν be probability distributions on R n , and suppose that we can write ( Id − ∇ L )# µ

Original: = ν with one-to-one ∇ L .
Revised: = ν with one-to-one ∇ L .

Original: Let us also write T
Revised: Let us also write T

Original: = ( Id − ∇ L ) , and let T t
Revised: = ( Id − ∇ L ) , and let T t

Original: = Id − t ∇ L be its time-linear interpolation.
Revised: = Id − t ∇ L be its time-linear interpolation.

Original: Then, for all t ∈ [0 , 1] ,
Revised: Then, for all t ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let p µ be the pdf of µ .
Revised: Let p µ be the pdf of µ .

Original: If y = T ( x ) , by the one-to-one assumption det ( DT ( x )) >
Revised: If y = T ( x ) , by the one-to-one assumption det ( DT ( x )) >

Original: and we may let d y = det ( DT ( x ))
Revised: and we may let d y = det ( DT ( x ))

Original: d x .
Revised: d x .

Original: By appealing to the fact that E µ [ h ( T ( X ))]
Revised: By appealing to the fact that E µ [ h ( T ( X ))]

Original: =
Revised: =

Original: E ν [ h ( Y )] for arbitrary h ,
Revised: E ν [ h ( Y )] for arbitrary h ,

Original: [Equation]
Revised: [Equation]

Original: and we can deduce the identity p T # µ ( T ( x )) det ( DT ( x ))
Revised: and we can deduce the identity p T # µ ( T ( x )) det ( DT ( x ))

Original: = p µ ( x ) .
Revised: = p µ ( x ) .

Original: Building on this fact, with straightforward computation we can say
Revised: Building on this fact, with straightforward computation we can say

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Assuming that the distributions are regular enough that we can swap the integral, we can appeal to
Revised: Assuming that the distributions are regular enough that we can swap the integral, we can appeal to

Original: Jacobi’s formula and deduce
Revised: Jacobi’s formula and deduce

Original: [Equation]
Revised: [Equation]

Original: Now, let us prove the proposition 2.2.
Revised: Now, let us prove the proposition 2.2.

Original: Without loss of generality, choose L so that DT = (cid:107) x (cid:107) 2 2 − L .
Revised: Without loss of generality, choose L so that DT = (cid:107) x (cid:107) 2 2 − L .

Original: Assuming that D 2 L is diagonalizable and that its spectrum is uniformly bounded away from 1 , let spec ( D 2 L ) =
Revised: Assuming that D 2 L is diagonalizable and that its spectrum is uniformly bounded away from 1 , let spec ( D 2 L ) =

Original: { λ k } and write diag { λ k } k = Λ .
Revised: { λ k } and write diag { λ k } k = Λ .

Original: Because (cid:107) x (cid:107) 2− L is assumed to be strictly convex,
Revised: Because (cid:107) x (cid:107) 2− L is assumed to be strictly convex,

Original: D 2 (cid:16)
Revised: D 2 (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: − L (cid:17) =
Revised: − L (cid:17) =

Original: I − D 2 L is positive deﬁnite, and 1 − λ k
Revised: I − D 2 L is positive deﬁnite, and 1 − λ k

Original: ( x ) > 0 for all k .
Revised: ( x ) > 0 for all k .

Original: Appealing to the line equation (29) we see that the assumption H ( ν ) > H ( µ ) is equivalent to
Revised: Appealing to the line equation (29) we see that the assumption H ( ν ) > H ( µ ) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Let us write max k { 1 − tλ k
Revised: Let us write max k { 1 − tλ k

Original: ( x ) } =
Revised: ( x ) } =

Original: C t ( x ) > 0 .
Revised: C t ( x ) > 0 .

Original: Using the assumption that the support of µ is compact, we can also say 0 < max supp ( µ ) { C t ( x ) } :=
Revised: Using the assumption that the support of µ is compact, we can also say 0 < max supp ( µ ) { C t ( x ) } :=

Original: c t < ∞ .
Revised: c t < ∞ .

Original: In general, if A is positive deﬁnite and diagonalizable, with straightforward argument we can say
Revised: In general, if A is positive deﬁnite and diagonalizable, with straightforward argument we can say

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using this fact, we see that
Revised: Using this fact, we see that

Original: [Equation]
Revised: [Equation]

Original: Suppose that D 2 L is diagonalizable with the change of basis matrix P .
Revised: Suppose that D 2 L is diagonalizable with the change of basis matrix P .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: This concludes the proof of the proposition 2.2.
Revised: This concludes the proof of the proposition 2.2.

Original: The proposition 2.1 is much simpler to prove.
Revised: The proposition 2.1 is much simpler to prove.

Original: In order to assure the H ( ν ) ≥ H ( µ ) , we only need to guarantee that E µ [log det ( DT ( X ))] ≥ 0 .
Revised: In order to assure the H ( ν ) ≥ H ( µ ) , we only need to guarantee that E µ [log det ( DT ( X ))] ≥ 0 .

Original: By requiring L to be concave, we would make (cid:107) x (cid:107) 2 / 2 − L ( x ) to be convex so that the eigenvalues of
Revised: By requiring L to be concave, we would make (cid:107) x (cid:107) 2 / 2 − L ( x ) to be convex so that the eigenvalues of

Original: DT will all become positive.
Revised: DT will all become positive.

Original: The result then follows trivially from the argument similar to the one that leads to equation (34).
Revised: The result then follows trivially from the argument similar to the one that leads to equation (34).

Original: C E XPERIMENTAL S ETTINGS C. P ERFORMANCE MEASURES
Revised: C E XPERIMENTAL S ETTINGS C. P ERFORMANCE MEASURES

Original: For the measure of GANs’ performance on the image dataset, we used Inception score (Salimans et al., 2016).
Revised: For the measure of GANs’ performance on the image dataset, we used Inception score (Salimans et al., 2016).

Original: Inception score was introduced originally as an exponentiated divergence measure based on the trained Inception convolutional neural network (Szegedy et al., 2015), which is often called Inception Model .
Revised: Inception score was introduced originally as an exponentiated divergence measure based on the trained Inception convolutional neural network (Szegedy et al., 2015), which is often called Inception Model .

Original: Using p ( y | x ) to denote the Inception model, the inception score is given by I ( { x n } Nn =1 ) :=
Revised: Using p ( y | x ) to denote the Inception model, the inception score is given by I ( { x n } Nn =1 ) :=

Original: exp(ˆ E [ D KL [ p ( y | x ) || p
Revised: exp(ˆ E [ D KL [ p ( y | x ) || p

Original: ( y )]]) , where ˆ E indicates the empirically approximated expectation.
Revised: ( y )]]) , where ˆ E indicates the empirically approximated expectation.

Original: Often times, p ( y ) is approximated with 1 N (cid:80)
Revised: Often times, p ( y ) is approximated with 1 N (cid:80)

Original: Nn =1 p ( y | x n ) .
Revised: Nn =1 p ( y | x n ) .

Original: The dominating consensus among the machine learning community is that this score is strongly correlated with subjective human judgment of image quality.
Revised: The dominating consensus among the machine learning community is that this score is strongly correlated with subjective human judgment of image quality.

Original: Following the procedure in Salimans et al.
Revised: Following the procedure in Salimans et al.

Original: , we generated 5000 examples from each trained generator and calculated the Inception score on the samples.
Revised: , we generated 5000 examples from each trained generator and calculated the Inception score on the samples.

Original: We evaluated the score 10 times with different seeds for the generation of x n and reported the average and the standard deviation of the scores.
Revised: We evaluated the score 10 times with different seeds for the generation of x n and reported the average and the standard deviation of the scores.

Original: Fr´echet inception distance (FID) (Heusel et al., 2017) is another measure for the quality of the generated examples that uses 2nd order information of the ﬁnal layer of the inception model.
Revised: Fr´echet inception distance (FID) (Heusel et al., 2017) is another measure for the quality of the generated examples that uses 2nd order information of the ﬁnal layer of the inception model.

Original: The
Revised: The

Original: FID is based on Fre´chet distance (Dowson & Landau, 1982)
Revised: FID is based on Fre´chet distance (Dowson & Landau, 1982)

Original: (Not to be confused with FID), which is the 2-Wasserstein distance between two distribution multivariate Gaussian distributions, p 1 and p 2 .
Revised: (Not to be confused with FID), which is the 2-Wasserstein distance between two distribution multivariate Gaussian distributions, p 1 and p 2 .

Original: The Wasserstein distance for two Gaussian distributions have a closed form, and is given by
Revised: The Wasserstein distance for two Gaussian distributions have a closed form, and is given by

Original: [Equation]
Revised: [Equation]

Original: (45) where { µ p 1 , C p 1 } , { µ p 2 , C p 2 } are the mean and covariance of samples from q and p , respectively.
Revised: (45) where { µ p 1 , C p 1 } , { µ p 2 , C p 2 } are the mean and covariance of samples from q and p , respectively.

Original: Now, if f (cid:9) is the output of the ﬁnal layer of the inception model before the softmax, the Fr´echet inception distance (FID) between two distributions p 1 and p 2 on the images is the Fre´chet distance distance between f (cid:9) ◦ p 1 and f (cid:9) ◦ p 2 .
Revised: Now, if f (cid:9) is the output of the ﬁnal layer of the inception model before the softmax, the Fr´echet inception distance (FID) between two distributions p 1 and p 2 on the images is the Fre´chet distance distance between f (cid:9) ◦ p 1 and f (cid:9) ◦ p 2 .

Original: We empirically computed the Fr´echet inception distance between the true distribution and the generated distribution using 10000 samples from the true distribution and 5000 samples from the generator distribution.
Revised: We empirically computed the Fr´echet inception distance between the true distribution and the generated distribution using 10000 samples from the true distribution and 5000 samples from the generator distribution.

Original: C. GAN’ S OBJECTIVE FUNCTION
Revised: C. GAN’ S OBJECTIVE FUNCTION

Original: We used applied DC regularizaiton to the training with the following set of objective functions (softplus ( x )
Revised: We used applied DC regularizaiton to the training with the following set of objective functions (softplus ( x )

Original: = log(

Revised: = log(

Original: + exp( x )) ).
Revised: + exp( x )) ).

Original: GAN-vanilla
Revised: GAN-vanilla

Original: [Equation]
Revised: [Equation]

Original: GAN-
Revised: GAN-

Original: [Equation]
Revised: [Equation]

Original: GAN
Revised: GAN

Original: [Equation]
Revised: [Equation]

Original: WGAN-GP
Revised: WGAN-GP

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: LSGAN (
Revised: LSGAN (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Feature Matching
Revised: Feature Matching

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: C. N ETWORK A RCHITECTURES
Revised: C. N ETWORK A RCHITECTURES

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 3 × 3, stride=1 conv 128 lReLU 4 × 4, stride=2 conv 128 lReLU 3 × 3, stride=1 conv 256 lReLU
Revised: 3 × 3, stride=1 conv 128 lReLU 4 × 4, stride=2 conv 128 lReLU 3 × 3, stride=1 conv 256 lReLU

Original: [Table]
Revised: [Table]

Original: 4, stride
Revised: 4, stride

Original: [Table]
Revised: [Table]

Original: 2 conv 256 lReLU
Revised: 2 conv 256 lReLU

Original: [Table]
Revised: [Table]

Original: 3, stride=1 conv.
Revised: 3, stride=1 conv.

Original: [Table]
Revised: [Table]

Original: lReLU
Revised: lReLU

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: C. D ETAILS FOR THE EXPERIMENTS ON CIFAR-10 AND CIFAR- 
Revised: C. D ETAILS FOR THE EXPERIMENTS ON CIFAR-10 AND CIFAR-

Original: Experimental setting for the evaluation of the method’s robustness against the choice of objective
Revised: Experimental setting for the evaluation of the method’s robustness against the choice of objective

Original: functions
Revised: functions

Original: For the model, we chose the architecture of DCGAN(Radford et al., 2016)
Revised: For the model, we chose the architecture of DCGAN(Radford et al., 2016)

Original: (Table 2) and ResNet (Table 3) with spectral normalization applied to full connect layer and convolution layer(SNDCGAN,
Revised: (Table 2) and ResNet (Table 3) with spectral normalization applied to full connect layer and convolution layer(SNDCGAN,

Original: SNResNet).
Revised: SNResNet).

Original: For the optimization, we used Adam(Kingma & Ba, 2015) and chose ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ) for the hyperparameters.
Revised: For the optimization, we used Adam(Kingma & Ba, 2015) and chose ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ) for the hyperparameters.

Original: Also, we chose n dis = 1 , n gen = 1 for
Revised: Also, we chose n dis = 1 , n gen = 1 for

Original: SNDCGAN and ( n dis = 5 , n gen = 1) for SNResNet.
Revised: SNDCGAN and ( n dis = 5 , n gen = 1) for SNResNet.

Original: We updated the generator 100 k times and linearly decayed the learning rate over last 5 k iterations.
Revised: We updated the generator 100 k times and linearly decayed the learning rate over last 5 k iterations.

Original: Experimental setting for the evaluation of the method’s robustness against the choice of the prior dimension
Revised: Experimental setting for the evaluation of the method’s robustness against the choice of the prior dimension

Original: We chose SNDCGAN for the model, and optimized the network with Adam using the hyperparameter ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ).
Revised: We chose SNDCGAN for the model, and optimized the network with Adam using the hyperparameter ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ).

Original: For the update of the discriminator and the generator, we set n dis = 1 , n gen = 1 .
Revised: For the update of the discriminator and the generator, we set n dis = 1 , n gen = 1 .

Original: We trained the generator 50 k times and linearly decayed the learning rate over last 5 k iterations.
Revised: We trained the generator 50 k times and linearly decayed the learning rate over last 5 k iterations.

Original: For the objective function, we chose GAN-variant2 in Appendix C.2.
Revised: For the objective function, we chose GAN-variant2 in Appendix C.2.

Original: We also set λ = 3 . 0 , d = 0 .
Revised: We also set λ = 3 . 0 , d = 0 .

Original: 01 for the parameters in (13).
Revised: 01 for the parameters in (13).

Original: For this set of the experiments, we repeated the experiments three times with different seeds, and reported the maximum, mean and minimum.
Revised: For this set of the experiments, we repeated the experiments three times with different seeds, and reported the maximum, mean and minimum.

Original: We tested with prior dimensions of range dim( z ) = 3 , 4 , 5 , 7 , 10 , 500 , 1000 , 2000 , 4000 , 6000 , 8000 , 10000 .
Revised: We tested with prior dimensions of range dim( z ) = 3 , 4 , 5 , 7 , 10 , 500 , 1000 , 2000 , 4000 , 6000 , 8000 , 10000 .

Original: Comparison with EGAN and other methods
Revised: Comparison with EGAN and other methods

Original: For the model, we chose SNDCGAN , SNResNet and SNResNetLarge, and trained the networks using Adam with hyperparameters ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ).
Revised: For the model, we chose SNDCGAN , SNResNet and SNResNetLarge, and trained the networks using Adam with hyperparameters ( α = 0 . 0002 , β 1 = 0 , β 2 = 0 . 9 ).

Original: SNResNetLarge is a model that was used in (Miyato et al., 2018).
Revised: SNResNetLarge is a model that was used in (Miyato et al., 2018).

Original: It is a same model as SNResNet except that it is uses a larger generator (Table 4).
Revised: It is a same model as SNResNet except that it is uses a larger generator (Table 4).

Original: For both models, We trained the generator 100 k times and linearly decayed the learning rate over last 10 k iterations.
Revised: For both models, We trained the generator 100 k times and linearly decayed the learning rate over last 10 k iterations.

Original: For EGAN-Ent-VI (Dai et al., 2017), we used an additional decoder equipped with convolution layers.
Revised: For EGAN-Ent-VI (Dai et al., 2017), we used an additional decoder equipped with convolution layers.

Original: We used n dis = 1 , n gen = 1 , n dec = 5 for SNDCGAN, and n dis = 1 , n gen = 5 , n dec = 5 for SNResNet.
Revised: We used n dis = 1 , n gen = 1 , n dec = 5 for SNDCGAN, and n dis = 1 , n gen = 5 , n dec = 5 for SNResNet.

Original: The table below is the list of the choices of the hyperparameters ( λ, d ) in equation (13).
Revised: The table below is the list of the choices of the hyperparameters ( λ, d ) in equation (13).

Original: we used in our comparative study, sorted by models
Revised: we used in our comparative study, sorted by models

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C. I MAGE G ENERATION ON I MAGE N ET
Revised: C. I MAGE G ENERATION ON I MAGE N ET

Original: The images used in this set of experiments were resized to 64 × 64 pixels.
Revised: The images used in this set of experiments were resized to 64 × 64 pixels.

Original: The details of the architecture are given in Table 5.
Revised: The details of the architecture are given in Table 5.

Original: For the optimization, we used Adam with the same hyperparameters we used for ResNet on CIFAR-10 and CIFAR-100 dataset.
Revised: For the optimization, we used Adam with the same hyperparameters we used for ResNet on CIFAR-10 and CIFAR-100 dataset.

Original: We trained the networks with 250K generator updates, and applied linear decay for the learning rate after 200K iterations so that the rate would be 0 at the end.
Revised: We trained the networks with 250K generator updates, and applied linear decay for the learning rate after 200K iterations so that the rate would be 0 at the end.

Original: We set λ = 6 . 0 , d = 0 .
Revised: We set λ = 6 . 0 , d = 0 .

Original: 01 for the parameters in equation (13).
Revised: 01 for the parameters in equation (13).

Original: D A PPENDIX R ESULTS D. A PPENDIX R ESULT ON ARTIFICIAL DATA
Revised: D A PPENDIX R ESULTS D. A PPENDIX R ESULT ON ARTIFICIAL DATA

Original: 8        data
Revised: 8        data

Original: [Figure]
Revised: [Figure]

Original: (a) without DC regularization        8        data model
Revised: (a) without DC regularization        8        data model

Original: [Figure]
Revised: [Figure]

Original: D.
Revised: D.

Original: A PPENDIX R ESULT ON CIFAR-10 AND CIFARExperiment with different architectures
Revised: A PPENDIX R ESULT ON CIFAR-10 AND CIFARExperiment with different architectures

Original: [Figure]
Revised: [Figure]

Original: Experiment with different architectures on WGAN-GP
Revised: Experiment with different architectures on WGAN-GP

Original: [Figure]
Revised: [Figure]

Original: (lower is better)
Revised: (lower is better)

Original: [Figure]
Revised: [Figure]

Original: Comparison with other methods
Revised: Comparison with other methods

Original: [Table]
Revised: [Table]

Original: Image generation on CIFAR-10 and CIFAR- 
Revised: Image generation on CIFAR-10 and CIFAR-

Original: [Figure]
Revised: [Figure]

Original: D. A PPENDIX R ESULT ON I MAGE N ET
Revised: D. A PPENDIX R ESULT ON I MAGE N ET

Original: [Figure]
Revised: [Figure]