Original: Q UANTITATIVELY E VALUATING GAN S W ITH D IVERGENCES P ROPOSED FOR T RAINING
Revised: Q UANTITATIVELY E VALUATING GAN S W ITH D IVERGENCES P ROPOSED FOR T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative adversarial networks (GANs) have made enormous progress in terms of theory and application in the machine learning and computer vision communities.
Revised: Generative adversarial networks (GANs) have made enormous progress in terms of theory and application in the machine learning and computer vision communities.

Original: Even though substantial progress has been made, a lack of quantitative model assessment is still a serious issue.
Revised: Even though substantial progress has been made, a lack of quantitative model assessment is still a serious issue.

Original: This has led to a huge number of GAN variants being proposed, with relatively little understanding of their relative abilities.
Revised: This has led to a huge number of GAN variants being proposed, with relatively little understanding of their relative abilities.

Original: In this paper, we evaluate the performance of various types of GANs using divergence and distance functions typically used only for training.
Revised: In this paper, we evaluate the performance of various types of GANs using divergence and distance functions typically used only for training.

Original: We observe consistency across the various proposed metrics, and interestingly, the test-time metrics do not favour networks that use the same training-time criterion.
Revised: We observe consistency across the various proposed metrics, and interestingly, the test-time metrics do not favour networks that use the same training-time criterion.

Original: Finally, we measure the statistically correlation between different metrics to human perceptual scores.
Revised: Finally, we measure the statistically correlation between different metrics to human perceptual scores.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative adversarial networks (GANs) aim to approximate a data distribution P , using a parameterized model distribution Q .
Revised: Generative adversarial networks (GANs) aim to approximate a data distribution P , using a parameterized model distribution Q .

Original: They achieve this by jointly optimizing generative and discriminative networks (Goodfellow et al., 2014).
Revised: They achieve this by jointly optimizing generative and discriminative networks (Goodfellow et al., 2014).

Original: GANs are end-to-end differentiable, and can produce exact samples from the distribution of the generative network Q .
Revised: GANs are end-to-end differentiable, and can produce exact samples from the distribution of the generative network Q .

Original: The samples of a generative network are propagated forward to a discriminative network, and the error signals are then propagated backwards to the generative network, passing through the discriminative network.
Revised: The samples of a generative network are propagated forward to a discriminative network, and the error signals are then propagated backwards to the generative network, passing through the discriminative network.

Original: This is often viewed as learning an adaptive loss function for a generative network by training a discriminative network.
Revised: This is often viewed as learning an adaptive loss function for a generative network by training a discriminative network.

Original: The success of learning a probability distribution depends heavily on the metric used to measure the distance between P and Q .
Revised: The success of learning a probability distribution depends heavily on the metric used to measure the distance between P and Q .

Original: Fortunately, GAN frameworks can be formulated as a divergence or distance function of various forms (involving supremum/inﬁmum over the function class),
Revised: Fortunately, GAN frameworks can be formulated as a divergence or distance function of various forms (involving supremum/inﬁmum over the function class),

Original: [Equation]
Revised: [Equation]

Original: (1) for some real-valued bounded class of functions F .
Revised: (1) for some real-valued bounded class of functions F .

Original: Several GAN variants, such as f-GAN (Nowozin et al., 2016), Wasserstein GAN (Arjovsky et al., 2017), and Least-Square GAN (Mao et al., 2017) have the above form with speciﬁc µ and v functions (see Table 1).
Revised: Several GAN variants, such as f-GAN (Nowozin et al., 2016), Wasserstein GAN (Arjovsky et al., 2017), and Least-Square GAN (Mao et al., 2017) have the above form with speciﬁc µ and v functions (see Table 1).

Original: The problem is relaxed by converting the supremum operator over the function class to have the maximum operator over a parametric model class, and minimizing the generative network.
Revised: The problem is relaxed by converting the supremum operator over the function class to have the maximum operator over a parametric model class, and minimizing the generative network.

Original: Finally, the GAN is formulated as a min-max problem:
Revised: Finally, the GAN is formulated as a min-max problem:

Original: [Equation]
Revised: [Equation]

Original: where θ and φ are the parameters of the generative and discriminative networks, D φ corresponds to the discriminative network, and Q θ ( x ) corresponds to the generative network’s distribution.
Revised: where θ and φ are the parameters of the generative and discriminative networks, D φ corresponds to the discriminative network, and Q θ ( x ) corresponds to the generative network’s distribution.

Original: Many practitioners who deploy GANs are primarily concerned with sample quality, as GANs are known to generate realistic-looking sharp samples compared to other popular generative models, such as variational autoencoders (Kingma & Welling, 2014).
Revised: Many practitioners who deploy GANs are primarily concerned with sample quality, as GANs are known to generate realistic-looking sharp samples compared to other popular generative models, such as variational autoencoders (Kingma & Welling, 2014).

Original: Nonetheless, there is no principled way of ﬁnding an architecture for high quality samples.
Revised: Nonetheless, there is no principled way of ﬁnding an architecture for high quality samples.

Original: Plenty of previous probabilistic generative models were evaluated based on the point-wise likelihood of the test data, and they were trained to maximize the point-wise likelihood.
Revised: Plenty of previous probabilistic generative models were evaluated based on the point-wise likelihood of the test data, and they were trained to maximize the point-wise likelihood.

Original: However, GANs do not use a point-wise criterion during training, so there is no easy way to evaluate the test likelihood without simplifying assumptions.
Revised: However, GANs do not use a point-wise criterion during training, so there is no easy way to evaluate the test likelihood without simplifying assumptions.

Original: But more importantly, it has been shown that the likelihood is generally uninformative about the quality of samples and vice versa (Theis et al., 2015), which is what GAN practitioners care about.
Revised: But more importantly, it has been shown that the likelihood is generally uninformative about the quality of samples and vice versa (Theis et al., 2015), which is what GAN practitioners care about.

Original: [Table]
Revised: [Table]

Original: Evaluating the quality of samples with human vision can be time-consuming and difﬁcult to reproduce.
Revised: Evaluating the quality of samples with human vision can be time-consuming and difﬁcult to reproduce.

Original: In order to quantitatively measure the sample quality, we propose to evaluate GANs by exploring the divergence and distance metrics that were used to train the GANs.
Revised: In order to quantitatively measure the sample quality, we propose to evaluate GANs by exploring the divergence and distance metrics that were used to train the GANs.

Original: We show that rankings produced using our proposed metrics are consistent and robust across metrics.
Revised: We show that rankings produced using our proposed metrics are consistent and robust across metrics.

Original: We use these metrics to quantitatively analyze three different families of GANs.
Revised: We use these metrics to quantitatively analyze three different families of GANs.

Original: This analysis includes measuring overall performance, as well as carrying out a sensitivity analysis with respect to various factors, such as the architecture size, noise dimension, update ratio between discriminator and generator, and number of data points.
Revised: This analysis includes measuring overall performance, as well as carrying out a sensitivity analysis with respect to various factors, such as the architecture size, noise dimension, update ratio between discriminator and generator, and number of data points.

Original: Our empirical results show that: i) the larger the GAN architecture the better the results, ii) having a generator network larger than the discriminator network does not yield good results, iii) the best ratio between discriminator and generator updates depended on the data set, and iv) the W-GAN and LS-GAN performance increases much faster than DCGAN as the number of training examples grows.
Revised: Our empirical results show that: i) the larger the GAN architecture the better the results, ii) having a generator network larger than the discriminator network does not yield good results, iii) the best ratio between discriminator and generator updates depended on the data set, and iv) the W-GAN and LS-GAN performance increases much faster than DCGAN as the number of training examples grows.

Original: Ultimately, we can tune the hyper-parameters and architectures of GANs based on our proposed method.
Revised: Ultimately, we can tune the hyper-parameters and architectures of GANs based on our proposed method.

Original:  E VALUATION M ETRICS
Revised:  E VALUATION M ETRICS

Original: Given a generator with ﬁxed parameters θ , we can simply compute the distance/divergence of P and
Revised: Given a generator with ﬁxed parameters θ , we can simply compute the distance/divergence of P and

Original: Q θ (See Algorithm 1).
Revised: Q θ (See Algorithm 1).

Original: We can consider metrics from two widely studied distance and divergence measures, f -divergence
Revised: We can consider metrics from two widely studied distance and divergence measures, f -divergence

Original: (Nguyen et al., 2008) and the Integral Probability Metric (IPM) (Muller, 1997).
Revised: (Nguyen et al., 2008) and the Integral Probability Metric (IPM) (Muller, 1997).

Original: Note that there is no overlap of the distance functions between the two classes, except for total variation distance (Sriperumbudur et al., 2009).
Revised: Note that there is no overlap of the distance functions between the two classes, except for total variation distance (Sriperumbudur et al., 2009).

Original: In our experiments, we consider the following four metrics that are commonly used to train GANs:
Revised: In our experiments, we consider the following four metrics that are commonly used to train GANs:

Original: Metric 1. Jensen-Shannon Divergence
Revised: Metric 1. Jensen-Shannon Divergence

Original: Training a standard GAN corresponds to minimizing the lower bound of Jensen-Shannon Divergence (Goodfellow et al., 2014):
Revised: Training a standard GAN corresponds to minimizing the lower bound of Jensen-Shannon Divergence (Goodfellow et al., 2014):

Original: [Equation]
Revised: [Equation]

Original: where p ( z ) is the prior distribution of the generative network, and G θ ( z ) is a differentiable function from z to the data space represented by a neural network with parameter θ .
Revised: where p ( z ) is the prior distribution of the generative network, and G θ ( z ) is a differentiable function from z to the data space represented by a neural network with parameter θ .

Original: Metric 2. Constrained Pearson X
Revised: Metric 2. Constrained Pearson X

Original: A Least-Squares GAN corresponds to training under Pearson X 2 divergence with constrained variables
Revised: A Least-Squares GAN corresponds to training under Pearson X 2 divergence with constrained variables

Original: b − a = 2 and b − c = 

Revised: b − a = 2 and b − c =

Original: (Mao et al., 2017):
Revised: (Mao et al., 2017):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where ϕ and θ are parameters of the discriminative and generative network, respectively.
Revised: ) where ϕ and θ are parameters of the discriminative and generative network, respectively.

Original: Typically a, b, c are set to 0 , 1 , 1 , respectively.
Revised: Typically a, b, c are set to 0 , 1 , 1 , respectively.

Original: Metric
Revised: Metric

Original: 3. Maximum Mean Discrepancy
Revised: 3. Maximum Mean Discrepancy

Original: [Equation]
Revised: [Equation]

Original: where H is a class of functions in a reproducing kernel Hilbert space (Gretton et al., 2012), k ( · , · ) is the kernel function, and m , n are the number of samples for P and Q .
Revised: where H is a class of functions in a reproducing kernel Hilbert space (Gretton et al., 2012), k ( · , · ) is the kernel function, and m , n are the number of samples for P and Q .

Original: Metric 4. Wasserstein Distance
Revised: Metric 4. Wasserstein Distance

Original: Arjovsky & Bottou (2017) propose the use of the dual representation of Wasserstein distance (Villani, 2009) for training GANs.
Revised: Arjovsky & Bottou (2017) propose the use of the dual representation of Wasserstein distance (Villani, 2009) for training GANs.

Original: The Wasserstein distance is an IPM which considers the 1-Lipschitz function class (cid:107) F (cid:107) L ≤ 1 .
Revised: The Wasserstein distance is an IPM which considers the 1-Lipschitz function class (cid:107) F (cid:107) L ≤ 1 .

Original: [Equation]
Revised: [Equation]

Original: Note that there were previous attempts on evaluating generative models via wasserstein distance and
Revised: Note that there were previous attempts on evaluating generative models via wasserstein distance and

Original: MMD (Danihelka et al., 2017; Sutherland et al., 2017) .
Revised: MMD (Danihelka et al., 2017; Sutherland et al., 2017) .

Original: Algorithm 

Revised: Algorithm

Original: Computing the divergence/distance 1: procedure D ISTANCE O PTIMIZATION P ROCEDURE
Revised: Computing the divergence/distance 1: procedure D ISTANCE O PTIMIZATION P ROCEDURE

Original: (Dataset { X tr , X vl , X te } , generative model
Revised: (Dataset { X tr , X vl , X te } , generative model

Original: G θ , learning rate η , metric D ) 2:
Revised: G θ , learning rate η , metric D ) 2:

Original: Initialize critic network parameter ϕ .
Revised: Initialize critic network parameter ϕ .

Original: 3: for i = 1 · · · N do 4:
Revised: 3: for i = 1 · · · N do 4:

Original: Sample data points from X, { x m } ∼ X tr .
Revised: Sample data points from X, { x m } ∼ X tr .

Original: Sample points from generative model, { s m } ∼ G θ .
Revised: Sample points from generative model, { s m } ∼ G θ .

Original: 6: ϕ ← ϕ + η ∇ ϕ D ( { x m } , { s m } ; ϕ ) .
Revised: 6: ϕ ← ϕ + η ∇ ϕ D ( { x m } , { s m } ; ϕ ) .

Original: Sample points from generative model, { s m } ∼ G θ .
Revised: Sample points from generative model, { s m } ∼ G θ .

Original: 8: return D ( X te , { s m } ) .
Revised: 8: return D ( X te , { s m } ) .

Original:  P REVIOUS E VALUATION M ETRICS
Revised:  P REVIOUS E VALUATION M ETRICS

Original: The Generative Adversarial Metric (Im et al., 2016a) measures the relative performance of two
Revised: The Generative Adversarial Metric (Im et al., 2016a) measures the relative performance of two

Original: GANs by measuring the likelihood ratio of the two models.
Revised: GANs by measuring the likelihood ratio of the two models.

Original: Consider two GANs with their respective trained partners, M 1 = ( D 1 , G 1 ) and M 2 = ( D 2 , G 2 ) , where G 1 and G 2 are the generators, and Dand D 2 are the discriminators.
Revised: Consider two GANs with their respective trained partners, M 1 = ( D 1 , G 1 ) and M 2 = ( D 2 , G 2 ) , where G 1 and G 2 are the generators, and Dand D 2 are the discriminators.

Original: The hypothesis H 1 is that M 1 is better than M 2 if G 1 fools D 2 more than G 2 fools D 1 , and vice versa for the hypothesis H 0 .
Revised: The hypothesis H 1 is that M 1 is better than M 2 if G 1 fools D 2 more than G 2 fools D 1 , and vice versa for the hypothesis H 0 .

Original: The likelihood-ratio is deﬁned as:
Revised: The likelihood-ratio is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where M (cid:48) 1 and M (cid:48) 2 are the swapped pairs ( D 1 , G 2 ) and ( D 2 , G 1 ) , and p ( x | y
Revised: where M (cid:48) 1 and M (cid:48) 2 are the swapped pairs ( D 1 , G 2 ) and ( D 2 , G 1 ) , and p ( x | y

Original: = 1 , M ) is the likelihood of x generated from the data distribution p ( x ) by model M .
Revised: = 1 , M ) is the likelihood of x generated from the data distribution p ( x ) by model M .

Original: Hence, the procedure continues by simply swapping the partners of the two GANs, where the discriminator and generator from the ﬁrst
Revised: Hence, the procedure continues by simply swapping the partners of the two GANs, where the discriminator and generator from the ﬁrst

Original: GAN is partnered with the discriminator and generator from the second GAN.
Revised: GAN is partnered with the discriminator and generator from the second GAN.

Original: Then, we can measure which generator fools the opponent’s discriminator more, D 1 ( S 2 ) D 2 ( S 1 ) where S 1 ∼ G 1 and S 2 ∼ G 2 .
Revised: Then, we can measure which generator fools the opponent’s discriminator more, D 1 ( S 2 ) D 2 ( S 1 ) where S 1 ∼ G 1 and S 2 ∼ G 2 .

Original: There are two main caveats to the Generative Adversarial Metric.
Revised: There are two main caveats to the Generative Adversarial Metric.

Original: First, the measurement is a relative comparison between the two models.
Revised: First, the measurement is a relative comparison between the two models.

Original: Second, the metric has a constraint where the two discriminators must have an approximately similar performance on a calibration dataset, which can be difﬁcult to satisfy in practice.
Revised: Second, the metric has a constraint where the two discriminators must have an approximately similar performance on a calibration dataset, which can be difﬁcult to satisfy in practice.

Original: The Inception Score (Salimans et al., 2016) measures the performance of a model using a welltrained third party neural network.
Revised: The Inception Score (Salimans et al., 2016) measures the performance of a model using a welltrained third party neural network.

Original: The Inception Score computes the expectation of divergence between the probability of class prediction, given samples and a prior distribution over the class,
Revised: The Inception Score computes the expectation of divergence between the probability of class prediction, given samples and a prior distribution over the class,

Original: [Equation]
Revised: [Equation]

Original: where the probability of class prediction given the samples is computed using a third party neural network that is trained by supervised learning.
Revised: where the probability of class prediction given the samples is computed using a third party neural network that is trained by supervised learning.

Original: In the original paper that proposed the Inception Score,
Revised: In the original paper that proposed the Inception Score,

Original: Google’s inception network (Szegedy et al., 2015) was used to compute p ( y | x ) .
Revised: Google’s inception network (Szegedy et al., 2015) was used to compute p ( y | x ) .

Original: Inception Score is the most widely used metric to measure GAN performance.
Revised: Inception Score is the most widely used metric to measure GAN performance.

Original: Nonetheless, it is an indirect way of measuring the generative model quality, as it considers the distance between the distribution over the class, rather than the distance between the data distribution P and the model distribution Q .
Revised: Nonetheless, it is an indirect way of measuring the generative model quality, as it considers the distance between the distribution over the class, rather than the distance between the data distribution P and the model distribution Q .

Original: As well, it requires another neural network that is trained separately via supervised learning.
Revised: As well, it requires another neural network that is trained separately via supervised learning.

Original: We demonstrate an example of a failure case of Inception Score in the experiments section.
Revised: We demonstrate an example of a failure case of Inception Score in the experiments section.

Original: Log-likelihood via AIS (Wu et al., 2016) measures the log-likelihood of decoder-based models using
Revised: Log-likelihood via AIS (Wu et al., 2016) measures the log-likelihood of decoder-based models using

Original: Annealed Importance Sampling (AIS).
Revised: Annealed Importance Sampling (AIS).

Original: AIS proceeds by considering many intermediate distributions that are deﬁned by taking a weighted geometric mean between the prior (initial) distribution, p ( z ) , and the joint (target) distribution, p σ ( x, z ) =
Revised: AIS proceeds by considering many intermediate distributions that are deﬁned by taking a weighted geometric mean between the prior (initial) distribution, p ( z ) , and the joint (target) distribution, p σ ( x, z ) =

Original: p ( x | z ) p ( z ) .
Revised: p ( x | z ) p ( z ) .

Original: The drawback of using the estimator as was done in (Wu et al., 2016) is that it makes a Gaussian observation model assumption with ﬁxed variance.
Revised: The drawback of using the estimator as was done in (Wu et al., 2016) is that it makes a Gaussian observation model assumption with ﬁxed variance.

Original: Unfortunately, neither a GANs’ objectives nor architectures have any particular observation model built in, whereas variational auto-encoder training objectives have a Gaussian observable model included.
Revised: Unfortunately, neither a GANs’ objectives nor architectures have any particular observation model built in, whereas variational auto-encoder training objectives have a Gaussian observable model included.

Original: In Section 4, we demonstrate that the estimate can vary considerably, sometimes depending on the size of the ﬁxed variance.
Revised: In Section 4, we demonstrate that the estimate can vary considerably, sometimes depending on the size of the ﬁxed variance.

Original: More importantly, recent work has shown that the log-likelihood does not reﬂect sample quality (Theis et al., 2015).
Revised: More importantly, recent work has shown that the log-likelihood does not reﬂect sample quality (Theis et al., 2015).

Original: Fréchet Inception Distance (Heusel et al., 2017)
Revised: Fréchet Inception Distance (Heusel et al., 2017)

Original: Fréchet Inception Distance (FID) computes the distance between the Gaussian in Wasserstein-2 distance,
Revised: Fréchet Inception Distance (FID) computes the distance between the Gaussian in Wasserstein-2 distance,

Original: [Equation]
Revised: [Equation]

Original: where m i , C i are the mean and covariance of the samples from distribution P i .
Revised: where m i , C i are the mean and covariance of the samples from distribution P i .

Original: In practice, the sufﬁcient statistics of Gaussian comes from the ﬁrst and second moment of the neural network features (convolutional neural network’s feature maps).
Revised: In practice, the sufﬁcient statistics of Gaussian comes from the ﬁrst and second moment of the neural network features (convolutional neural network’s feature maps).

Original: Compare to our proposed evaluation methods, FID has an advantage and disadvantage.
Revised: Compare to our proposed evaluation methods, FID has an advantage and disadvantage.

Original: The main advantage is the speed.
Revised: The main advantage is the speed.

Original: The calculating of FID distance is much faster than our evaluation methods.
Revised: The calculating of FID distance is much faster than our evaluation methods.

Original: The disadvantage of FID is that it only considers difference in ﬁrst two moments of the samples, which can be insufﬁcient unless the feature maps are Gaussian distributed.
Revised: The disadvantage of FID is that it only considers difference in ﬁrst two moments of the samples, which can be insufﬁcient unless the feature maps are Gaussian distributed.

Original: On the other hand, the four metrics that we consider does not make any assumptions about the distribution of the samples.
Revised: On the other hand, the four metrics that we consider does not make any assumptions about the distribution of the samples.

Original: Classiﬁer Two-sample Tests (Heusel et al., 2017) (C2ST) proposes to train a classiﬁer that can distinguish samples drawn from two distribution P and Q and accept/reject the null hypothesis.
Revised: Classiﬁer Two-sample Tests (Heusel et al., 2017) (C2ST) proposes to train a classiﬁer that can distinguish samples drawn from two distribution P and Q and accept/reject the null hypothesis.

Original: There is a relationship between the methods proposed and C2ST.
Revised: There is a relationship between the methods proposed and C2ST.

Original: One of the commonalities that shared between our proposed test methods and C2ST is that both require optimizing a function (training a neural network) at test time.
Revised: One of the commonalities that shared between our proposed test methods and C2ST is that both require optimizing a function (training a neural network) at test time.

Original: C2ST trains neural network to maximize the classiﬁcation between data and samples, whereas our method is training neural network to maximize the distance between the data and samples.
Revised: C2ST trains neural network to maximize the classiﬁcation between data and samples, whereas our method is training neural network to maximize the distance between the data and samples.

Original: Going further, every binary classiﬁer has a corresponding distance metric from IPM (Sriperumbudur et al., 2009) (see our detailed explanation in Appendix 5).
Revised: Going further, every binary classiﬁer has a corresponding distance metric from IPM (Sriperumbudur et al., 2009) (see our detailed explanation in Appendix 5).

Original: Thus, C2ST binary classiﬁer must have a distance function associated with it with speciﬁc set of functions F .
Revised: Thus, C2ST binary classiﬁer must have a distance function associated with it with speciﬁc set of functions F .

Original: For example, if the binary classiﬁer is KNN, then we are considering IPM metric with the topology induced by KNN, as well if the classiﬁer is neural networks then we are considering IPM metric with the topology induced by the neural network.
Revised: For example, if the binary classiﬁer is KNN, then we are considering IPM metric with the topology induced by KNN, as well if the classiﬁer is neural networks then we are considering IPM metric with the topology induced by the neural network.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conducted our experiments on three types of GANs: Deep Convolutional Generative Adversarial
Revised: We conducted our experiments on three types of GANs: Deep Convolutional Generative Adversarial

Original: Networks (DCGAN), Least-Squares GAN (LS-GAN), and Wasserstein GAN (W-GAN).
Revised: Networks (DCGAN), Least-Squares GAN (LS-GAN), and Wasserstein GAN (W-GAN).

Original: We evaluated these three families of GANs with six metrics.
Revised: We evaluated these three families of GANs with six metrics.

Original: Two existing metrics, the inception score (IS), MMD, and FID were chosen because IS and FID are the most commonly used metric for evaluating GANs, and MMD is well-known for two sample tests (Sutherland et al., 2017).
Revised: Two existing metrics, the inception score (IS), MMD, and FID were chosen because IS and FID are the most commonly used metric for evaluating GANs, and MMD is well-known for two sample tests (Sutherland et al., 2017).

Original: The other three metrics have the same criterion as the standard GAN, LS-GAN, and W-GAN, with a ﬁxed generator from section 2.
Revised: The other three metrics have the same criterion as the standard GAN, LS-GAN, and W-GAN, with a ﬁxed generator from section 2.

Original: In order to not confuse the test metric names with the actual trained GANs, we will denote the GAN,
Revised: In order to not confuse the test metric names with the actual trained GANs, we will denote the GAN,

Original: LS-GAN, and W-GAN metric at test time as JS, LS, and IW, respectively.
Revised: LS-GAN, and W-GAN metric at test time as JS, LS, and IW, respectively.

Original: Because the optimization of the discriminator is required both during training and test time, we will call the discriminator at test time the critic , in order to not confuse the two discriminators.
Revised: Because the optimization of the discriminator is required both during training and test time, we will call the discriminator at test time the critic , in order to not confuse the two discriminators.

Original: In our experiments, we considered the MNIST, CIFAR-10, and LSUN Bedroom datasets.
Revised: In our experiments, we considered the MNIST, CIFAR-10, and LSUN Bedroom datasets.

Original: MNIST consists of 60,000 training and 10,000 test images of size 28 × 28 pixels containing handwritten digits from the classes 0 to 9.
Revised: MNIST consists of 60,000 training and 10,000 test images of size 28 × 28 pixels containing handwritten digits from the classes 0 to 9.

Original: From the 60,000 training examples, we set aside 10,000 as validation examples to tune various hyper-parameters.
Revised: From the 60,000 training examples, we set aside 10,000 as validation examples to tune various hyper-parameters.

Original: The CIFAR-10 dataset 1 consists of images with a size of 32 × 32 × 3 pixels with ten different classes of objects.
Revised: The CIFAR-10 dataset 1 consists of images with a size of 32 × 32 × 3 pixels with ten different classes of objects.

Original: We used 45,000, 5,000 and 10,000 examples as training, validation, and test data, respectively.
Revised: We used 45,000, 5,000 and 10,000 examples as training, validation, and test data, respectively.

Original: The LSUN bedroom dataset consists of 64 ×images of bedrooms.
Revised: The LSUN bedroom dataset consists of 64 ×images of bedrooms.

Original: Among 3,033,342 images, we used 90,000 images as training data and another 90,000 images as validation data.
Revised: Among 3,033,342 images, we used 90,000 images as training data and another 90,000 images as validation data.

Original: The hyper-parameters, such as learning rate, number of epochs, and batch size, were selected from discrete ranges and chosen based on a held-out validation set.
Revised: The hyper-parameters, such as learning rate, number of epochs, and batch size, were selected from discrete ranges and chosen based on a held-out validation set.

Original: Table 12 in the supplementary materials (S.M.) section shows the learning rates and the convolutional kernel sizes that were used for each experiment.
Revised: Table 12 in the supplementary materials (S.M.) section shows the learning rates and the convolutional kernel sizes that were used for each experiment.

Original: The architecture of each network is presented in the S.M. in Figure 10.
Revised: The architecture of each network is presented in the S.M. in Figure 10.

Original: Additionally, we used exponential-mean-square kernels with several different sigma values for MMD.
Revised: Additionally, we used exponential-mean-square kernels with several different sigma values for MMD.

Original: A pre-trained logistic regression and pre-trained residual network are used for IS and FID on MNIST and CIFAR-10 datasets, respectively.
Revised: A pre-trained logistic regression and pre-trained residual network are used for IS and FID on MNIST and CIFAR-10 datasets, respectively.

Original: For every experiment, 10-fold evaluation was used 2 and the mean and standard deviation were displayed.
Revised: For every experiment, 10-fold evaluation was used 2 and the mean and standard deviation were displayed.

Original: We analyze the GANs in the following way: First, we present the result of the GANs under the six different metrics.
Revised: We analyze the GANs in the following way: First, we present the result of the GANs under the six different metrics.

Original: We then show the changes in performance with respect to the size of the architectures.
Revised: We then show the changes in performance with respect to the size of the architectures.

Original: Next, we investigate the performance with respect to a varying number of update ratios between discriminator and generator updates.
Revised: Next, we investigate the performance with respect to a varying number of update ratios between discriminator and generator updates.

Original: Lastly, we investigate how performance changes with respect to the amount of training data available.
Revised: Lastly, we investigate how performance changes with respect to the amount of training data available.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4. N OTES ON E XISTING M ETRICS
Revised: 4. N OTES ON E XISTING M ETRICS

Original: The log-likelihood measurement is the most commonly used metric for generative models.
Revised: The log-likelihood measurement is the most commonly used metric for generative models.

Original: However, our experiments show that the result of measuring the log-likelihood using AIS 3 on GANs is strange, as shown in Figure 1.
Revised: However, our experiments show that the result of measuring the log-likelihood using AIS 3 on GANs is strange, as shown in Figure 1.

Original: We measured the log-likelihood of DCGAN on MNIST with three different sizes of variance, 0 . 01 , 0 . 025 , and 0 . 05 .
Revised: We measured the log-likelihood of DCGAN on MNIST with three different sizes of variance, 0 . 01 , 0 . 025 , and 0 . 05 .

Original: The ﬁgure illustrates that the log-likelihood curve over the training epochs varies substantially depending on the variance, which indicates that the ﬁxed
Revised: The ﬁgure illustrates that the log-likelihood curve over the training epochs varies substantially depending on the variance, which indicates that the ﬁxed

Original: Gaussian observable model might not be the ideal assumption for GANs.
Revised: Gaussian observable model might not be the ideal assumption for GANs.

Original: Moreover, we observe a high log-likelihood at the beginning of training, followed by a drop in likelihood, which then returns to the high value, and we are unable to explain why this happens.
Revised: Moreover, we observe a high log-likelihood at the beginning of training, followed by a drop in likelihood, which then returns to the high value, and we are unable to explain why this happens.

Original: Among the metrics we considered, only IS and MMD were previously used to evaluate GANs.
Revised: Among the metrics we considered, only IS and MMD were previously used to evaluate GANs.

Original: It was difﬁcult to determine whether the IS and MMD matched with the proposed metrics.
Revised: It was difﬁcult to determine whether the IS and MMD matched with the proposed metrics.

Original: We observed that IS is consistent most of the time with the metrics that we proposed, and yet, we did ﬁnd a few outliers cases when IS and MMD failed.
Revised: We observed that IS is consistent most of the time with the metrics that we proposed, and yet, we did ﬁnd a few outliers cases when IS and MMD failed.

Original: For example, Figure 2 is an odd scenario, where the visual quality of samples did not match the score.
Revised: For example, Figure 2 is an odd scenario, where the visual quality of samples did not match the score.

Original: The left side of the ﬁgure shows the samples of DCGAN when it failed to train properly.
Revised: The left side of the ﬁgure shows the samples of DCGAN when it failed to train properly.

Original: Even though the failed DCGAN samples are much darker than the samples on the right, the IS score for the left samples is higher than the right samples (a higher score
Revised: Even though the failed DCGAN samples are much darker than the samples on the right, the IS score for the left samples is higher than the right samples (a higher score

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: is better for IS).
Revised: is better for IS).

Original: Another odd example for MMD is shown in Figure 5.
Revised: Another odd example for MMD is shown in Figure 5.

Original: The samples on the left are dark, like the previous examples, but still textually recognizable, whereas the samples on the right are totally meaningless.
Revised: The samples on the left are dark, like the previous examples, but still textually recognizable, whereas the samples on the right are totally meaningless.

Original: However, MMD gives lower scores to the right side samples (a lower score is better for MMD).
Revised: However, MMD gives lower scores to the right side samples (a lower score is better for MMD).

Original: This is because the average intensity of the pixels of the samples on the right.
Revised: This is because the average intensity of the pixels of the samples on the right.

Original: It seems that the common failure case of MMD is when the mean pixel intensities are a better match than texture matches (see Figure 5), and the common failure cases of IS happens to be when the samples are recognizable textures, but the intensity of the samples are either brighter or darker (see
Revised: It seems that the common failure case of MMD is when the mean pixel intensities are a better match than texture matches (see Figure 5), and the common failure cases of IS happens to be when the samples are recognizable textures, but the intensity of the samples are either brighter or darker (see

Original: Figure 2).
Revised: Figure 2).

Original: Other metrics probably have unusual failings as well, but they were not as obviously noticeable as those for IS and MMD.
Revised: Other metrics probably have unusual failings as well, but they were not as obviously noticeable as those for IS and MMD.

Original: In section 4.2.1, we conduct more systematic experiments by measuring the correlation between these metrics to human perceptual scores.
Revised: In section 4.2.1, we conduct more systematic experiments by measuring the correlation between these metrics to human perceptual scores.

Original: 4. T HE METRIC COMPARISON
Revised: 4. T HE METRIC COMPARISON

Original: DCGANs are known to be unstable for a few reasons: the support of the data and model distributions possibly being disjoint (Arjovsky & Bottou, 2017), and the Hessian of the GAN objective being non-Hermitian (Mescheder et al., 2017).
Revised: DCGANs are known to be unstable for a few reasons: the support of the data and model distributions possibly being disjoint (Arjovsky & Bottou, 2017), and the Hessian of the GAN objective being non-Hermitian (Mescheder et al., 2017).

Original: LS-DCGAN and W-DCGAN are proposed to address this by providing non-saturating gradients.
Revised: LS-DCGAN and W-DCGAN are proposed to address this by providing non-saturating gradients.

Original: Based on experience, it is found that LS-DCGAN is the simplest and most stable model to train.
Revised: Based on experience, it is found that LS-DCGAN is the simplest and most stable model to train.

Original: We visualize the 2D subspace of the loss surface of the GANs in
Revised: We visualize the 2D subspace of the loss surface of the GANs in

Original: S.M. Figure 24.
Revised: S.M. Figure 24.

Original: Here, we took the parameters of three trained models (corresponds to red vertices in the ﬁgure) and applied barycentric interpolation with respect to three parameters (see details from
Revised: Here, we took the parameters of three trained models (corresponds to red vertices in the ﬁgure) and applied barycentric interpolation with respect to three parameters (see details from

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: L S s c o r e
Revised: L S s c o r e

Original: Filter #: [3, 64, 128, 256] gan wgan lsgan
Revised: Filter #: [3, 64, 128, 256] gan wgan lsgan

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: L S s c o r e
Revised: L S s c o r e

Original: Filter #: [3, 128, 256, 512] gan wgan lsgan models 0.
Revised: Filter #: [3, 128, 256, 512] gan wgan lsgan models 0.

Original: l s d i s t a n c e
Revised: l s d i s t a n c e

Original: Filter #: [3, 256, 512, 1024] gan wgan lsgan models 0.
Revised: Filter #: [3, 256, 512, 1024] gan wgan lsgan models 0.

Original: L S s c o r e
Revised: L S s c o r e

Original: Filter #: [3, 320, 640, 1280]
Revised: Filter #: [3, 320, 640, 1280]

Original: (Im et al., 2016c)).
Revised: (Im et al., 2016c)).

Original: It appears that DCGAN surfaces have much sharper slopes when compared to
Revised: It appears that DCGAN surfaces have much sharper slopes when compared to

Original: LS-DCGAN and W-DCGAN 4 , and LS-DCGAN has the most gentle surfaces.
Revised: LS-DCGAN and W-DCGAN 4 , and LS-DCGAN has the most gentle surfaces.

Original: In what follows, we show that this geometric view is consistent with our ﬁnding that LS-DCGAN is the easiest and the most stable to train.
Revised: In what follows, we show that this geometric view is consistent with our ﬁnding that LS-DCGAN is the easiest and the most stable to train.

Original: Table 2 and Table 3 present the results on MNIST and CIFAR-10, respectively.
Revised: Table 2 and Table 3 present the results on MNIST and CIFAR-10, respectively.

Original: Considering the proposed method of evaluation, we need to ﬁrst ask whether the metric favours the samples from the family of model that was trained on the same metric; fortunately, we do not see this behaviour.
Revised: Considering the proposed method of evaluation, we need to ﬁrst ask whether the metric favours the samples from the family of model that was trained on the same metric; fortunately, we do not see this behaviour.

Original: Both for MNIST, CIFAR-10, and LSUN, all metrics (IW, JS, and LS) consistently agree with which samples are the closest to test data.
Revised: Both for MNIST, CIFAR-10, and LSUN, all metrics (IW, JS, and LS) consistently agree with which samples are the closest to test data.

Original: We speculate that this happens for a combination of two reasons, where we are training the critic at test time from scratch, and the dynamics of the optimization process for training and test time are different.
Revised: We speculate that this happens for a combination of two reasons, where we are training the critic at test time from scratch, and the dynamics of the optimization process for training and test time are different.

Original: Overall, every metric except for MMD showed that LS-DCGAN performed the best among the three
Revised: Overall, every metric except for MMD showed that LS-DCGAN performed the best among the three

Original: GANs under our chosen hyper-parameter range, and that MMD favoured W-DCGAN’s samples.
Revised: GANs under our chosen hyper-parameter range, and that MMD favoured W-DCGAN’s samples.

Original: The performance between DCGAN or W-DCGAN are quite similar, where the mean plus/minus standard deviation overlaps, especially for CIFAR-10.
Revised: The performance between DCGAN or W-DCGAN are quite similar, where the mean plus/minus standard deviation overlaps, especially for CIFAR-10.

Original: Moreover, we observed similar results for a range of different critic CNN architectures (number of feature maps in each convolutional layer): [3 , 64 , 128 , 256] , [3 , 128 , 256 , 512] , [3 , 256 , 512 , 1024] , and [3 , 320 , 640 , 1280] (see Figure 3 and
Revised: Moreover, we observed similar results for a range of different critic CNN architectures (number of feature maps in each convolutional layer): [3 , 64 , 128 , 256] , [3 , 128 , 256 , 512] , [3 , 256 , 512 , 1024] , and [3 , 320 , 640 , 1280] (see Figure 3 and

Original: S.M. Figure 12).
Revised: S.M. Figure 12).

Original: Additionally, we experimented consistency of the metric scores with respect to the size of validation set.
Revised: Additionally, we experimented consistency of the metric scores with respect to the size of validation set.

Original: We trained three GANs upto 100 epochs with 90,000 examples of LSUN bedroom images.
Revised: We trained three GANs upto 100 epochs with 90,000 examples of LSUN bedroom images.

Original: We measured both LS and IW scores with 300 and 90,000 validation examples respectively.
Revised: We measured both LS and IW scores with 300 and 90,000 validation examples respectively.

Original: Note that the scores on 300 validation data should be less reliable than 90K one.
Revised: Note that the scores on 300 validation data should be less reliable than 90K one.

Original: We found that the rank of the
Revised: We found that the rank of the

Original: GANs on 300 and 90K validation data agrees 88% of times for LS Score and agrees 55% of times for
Revised: GANs on 300 and 90K validation data agrees 88% of times for LS Score and agrees 55% of times for

Original: IW distance.
Revised: IW distance.

Original: This indicates that LS scores are more robust than IW distance.
Revised: This indicates that LS scores are more robust than IW distance.

Original: For LS-score, we can simply minimize its divergence, whereas IW involves minimizing the distance and regularizing the gradient term (the gradient penalty term in (Gulrajani et al., 2017)).
Revised: For LS-score, we can simply minimize its divergence, whereas IW involves minimizing the distance and regularizing the gradient term (the gradient penalty term in (Gulrajani et al., 2017)).

Original: Computing the gradient penalty term requires extra computational time and tuning its regularization coefﬁcient.
Revised: Computing the gradient penalty term requires extra computational time and tuning its regularization coefﬁcient.

Original: 4.2. M ETRIC COMPARISON TO H UMAN P ERCEPTUAL S CORE
Revised: 4.2. M ETRIC COMPARISON TO H UMAN P ERCEPTUAL S CORE

Original: More generally, because we see some overlap in the statistics of performance, the statistical significance becomes questionable.
Revised: More generally, because we see some overlap in the statistics of performance, the statistical significance becomes questionable.

Original: We computed the Wilcoxon rank sum test that the medians of the distributions are the same.
Revised: We computed the Wilcoxon rank sum test that the medians of the distributions are the same.

Original: The p-values in Table 7 indicate that the differences are signiﬁcant for LS.
Revised: The p-values in Table 7 indicate that the differences are signiﬁcant for LS.

Original: Above all, out of all metrics we consider, it is important to ﬁgure out which metric matches the human perceptual scores.
Revised: Above all, out of all metrics we consider, it is important to ﬁgure out which metric matches the human perceptual scores.

Original: Thus, we compared between the evaluation metrics to human perceptual scores.
Revised: Thus, we compared between the evaluation metrics to human perceptual scores.

Original: We gathered ﬁve volunteers and ask them to choose the better group of samples among two randomly select set of samples.
Revised: We gathered ﬁve volunteers and ask them to choose the better group of samples among two randomly select set of samples.

Original: Here, the better samples refers to the samples that looks like data distribution.
Revised: Here, the better samples refers to the samples that looks like data distribution.

Original: At the beginning of survey, the volunteers were trained by selecting between random samples generated by GANs versus samples from data distribution.
Revised: At the beginning of survey, the volunteers were trained by selecting between random samples generated by GANs versus samples from data distribution.

Original: They get a positive reward if they selected the data samples and a negaitve reward if they select the samples from the model.
Revised: They get a positive reward if they selected the data samples and a negaitve reward if they select the samples from the model.

Original: After
Revised: After

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: enough training, they begin the real test.
Revised: enough training, they begin the real test.

Original: S. M. Figure 13 displays the user interace for the participants.
Revised: S. M. Figure 13 displays the user interace for the participants.

Original: S. M. Figure 14 show the fraction of labels that ﬁve volunteers agree.
Revised: S. M. Figure 14 show the fraction of labels that ﬁve volunteers agree.

Original: (Table 6) presents the fraction of pairs on which each metric agrees with humans (higher the better) and the result of two-sided Fisher’s test (no multiple comparisons correction) that these fractions are these same or not in Table 8.
Revised: (Table 6) presents the fraction of pairs on which each metric agrees with humans (higher the better) and the result of two-sided Fisher’s test (no multiple comparisons correction) that these fractions are these same or not in Table 8.

Original: Table 6 shows that IW distance agreed the most with human perceptual scores, and then followed by
Revised: Table 6 shows that IW distance agreed the most with human perceptual scores, and then followed by

Original: LS, Inception score
Revised: LS, Inception score

Original: [Table]
Revised: [Table]

Original: and MMD.
Revised: and MMD.

Original: The results of Table 6 and Table 8 together demonstrate that IW, LS signiﬁcantly better than Inception Score and MMD, p < 0 .
Revised: The results of Table 6 and Table 8 together demonstrate that IW, LS signiﬁcantly better than Inception Score and MMD, p < 0 .

Original: Thus, this presents which metrics are more statistically correlated with human perceptual scores.
Revised: Thus, this presents which metrics are more statistically correlated with human perceptual scores.

Original: In terms of guidance on which metric to use, we recommend using the metric that is the closest to the human perceptual score
Revised: In terms of guidance on which metric to use, we recommend using the metric that is the closest to the human perceptual score

Original: . 4.2. E VALUATING ON ADDITIONAL GAN S
Revised: . 4.2. E VALUATING ON ADDITIONAL GAN S

Original: There has been several variants of GANs that were proposed recently other than DCGAN, LSDCGAN,
Revised: There has been several variants of GANs that were proposed recently other than DCGAN, LSDCGAN,

Original: WGAN.
Revised: WGAN.

Original: We additioally give the performance of other popular GANs including EBGAN(Junbo Zhao, 2016), BEGAN(Berthelot et al., 2017), WGAN GP(Gulrajani et al., 2017), and DRAGAN (Kodali et al., 2017).
Revised: We additioally give the performance of other popular GANs including EBGAN(Junbo Zhao, 2016), BEGAN(Berthelot et al., 2017), WGAN GP(Gulrajani et al., 2017), and DRAGAN (Kodali et al., 2017).

Original: We trained on MNIST and FashionMNIST using off-the-shelf pytorch package 5 .
Revised: We trained on MNIST and FashionMNIST using off-the-shelf pytorch package 5 .

Original: This means that we use the hyper-parameters that were already tuned by the package.
Revised: This means that we use the hyper-parameters that were already tuned by the package.

Original: S. M. Figuredemonstrates the samples from each trained models.
Revised: S. M. Figuredemonstrates the samples from each trained models.

Original: We use three evaluation metrics: LS, IW, and
Revised: We use three evaluation metrics: LS, IW, and

Original: FID.
Revised: FID.

Original: Table 5 presents the evaluation results.
Revised: Table 5 presents the evaluation results.

Original: MNIST table shows that the three metrics are quite consistent and they rank the DRAGAN the highest and followed by LSGAN and DCGAN.
Revised: MNIST table shows that the three metrics are quite consistent and they rank the DRAGAN the highest and followed by LSGAN and DCGAN.

Original: The top three models are scored very close to each other.
Revised: The top three models are scored very close to each other.

Original: On the other hand, the rest of other models, EBGAN, BEGAN, and WGAN GP, are quite far from top three models.
Revised: On the other hand, the rest of other models, EBGAN, BEGAN, and WGAN GP, are quite far from top three models.

Original: For Fashion MNIST, LS and IW agrees with each other and they show that the top three models are DRAGAN LSGAN, and DCGAN.
Revised: For Fashion MNIST, LS and IW agrees with each other and they show that the top three models are DRAGAN LSGAN, and DCGAN.

Original: Again, these three perform all about the same where the mean plus standard deviation overlaps with each other.
Revised: Again, these three perform all about the same where the mean plus standard deviation overlaps with each other.

Original: On the other hand, FID tells us that the top three models are LSGAN, DRAGAN and BEGAN.
Revised: On the other hand, FID tells us that the top three models are LSGAN, DRAGAN and BEGAN.

Original: However, only LSGAN and DRAGAN scores are close to each other and BEGAN score is somewhat distanced from LSGAN and DRAGAN.
Revised: However, only LSGAN and DRAGAN scores are close to each other and BEGAN score is somewhat distanced from LSGAN and DRAGAN.

Original: Overal, we consistently see that LSGAN and DRAGAN stay in the top rank.
Revised: Overal, we consistently see that LSGAN and DRAGAN stay in the top rank.

Original: 4. S ENTITIVITY A NALYSIS 4.3. P ERFORMANCE CHANGE WITH RESPECT TO THE SIZE OF THE NETWORK
Revised: 4. S ENTITIVITY A NALYSIS 4.3. P ERFORMANCE CHANGE WITH RESPECT TO THE SIZE OF THE NETWORK

Original: Several works have demonstrated an improvement in performance by enlarging deep network architectures (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;
Revised: Several works have demonstrated an improvement in performance by enlarging deep network architectures (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;

Original: He et al., 2015; Huang et al., 2017).
Revised: He et al., 2015; Huang et al., 2017).

Original: Here, we investigate performance changes with respect to the width and depth of the networks.
Revised: Here, we investigate performance changes with respect to the width and depth of the networks.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: First, we trained three GANs with different numbers of feature map sizes, as shown in Table 9 (a-d).
Revised: First, we trained three GANs with different numbers of feature map sizes, as shown in Table 9 (a-d).

Original: Note that we double the number of feature maps in Table 9 for both the discriminators and generators.
Revised: Note that we double the number of feature maps in Table 9 for both the discriminators and generators.

Original: In Figure 4, the performance of the LS-score increases logarithmically as the number of feature maps is doubled.
Revised: In Figure 4, the performance of the LS-score increases logarithmically as the number of feature maps is doubled.

Original: A similar behaviour is observed in other metrics as well (see S.M. Figure 15).
Revised: A similar behaviour is observed in other metrics as well (see S.M. Figure 15).

Original: [Table]
Revised: [Table]

Original: We then tested the importance of size in the discriminative versus generative network.
Revised: We then tested the importance of size in the discriminative versus generative network.

Original: We considered two extreme feature map sizes, where we choose a small and a large number of feature maps for the generator and discriminator, and vice versa (see Label (e) and (f) in Table 9), and results are shown in Table 10.
Revised: We considered two extreme feature map sizes, where we choose a small and a large number of feature maps for the generator and discriminator, and vice versa (see Label (e) and (f) in Table 9), and results are shown in Table 10.

Original: For LS-DCGAN, it can be seen that the large number of feature maps for the discriminator has a better score than the large number of feature maps for the generator.
Revised: For LS-DCGAN, it can be seen that the large number of feature maps for the discriminator has a better score than the large number of feature maps for the generator.

Original: This can also be qualitatively veriﬁed by looking at the samples from architectures (a), (e), (f), and (d) in Figure 6.
Revised: This can also be qualitatively veriﬁed by looking at the samples from architectures (a), (e), (f), and (d) in Figure 6.

Original: For W-DCGAN, we observe the agreement between the LS and IW metric, but conﬂicts with MMD and IS.
Revised: For W-DCGAN, we observe the agreement between the LS and IW metric, but conﬂicts with MMD and IS.

Original: But, when we look at the samples of W-DCGAN in Figure 5, it is clear that the model with the larger number of feature maps in the discriminator should get a better score.
Revised: But, when we look at the samples of W-DCGAN in Figure 5, it is clear that the model with the larger number of feature maps in the discriminator should get a better score.

Original: This is another example of false intuition propagated by MMD and IS.
Revised: This is another example of false intuition propagated by MMD and IS.

Original: One interesting observation is that when we compare the score and samples from architecture (a) and (e) from Table 9, architecture (a) is much better than (e) (see Figure 6).
Revised: One interesting observation is that when we compare the score and samples from architecture (a) and (e) from Table 9, architecture (a) is much better than (e) (see Figure 6).

Original: It demonstrates that having a large generator and small discriminator is worse than having small architectures for both networks.
Revised: It demonstrates that having a large generator and small discriminator is worse than having small architectures for both networks.

Original: Overall, we found that having a larger generator than a discriminator does not give good results, and that it is more desirable to have a larger discriminator than a generator.
Revised: Overall, we found that having a larger generator than a discriminator does not give good results, and that it is more desirable to have a larger discriminator than a generator.

Original: Similar results were also observed for
Revised: Similar results were also observed for

Original: MNIST, as shown in S.M. Figure 19.
Revised: MNIST, as shown in S.M. Figure 19.

Original: This result somewhat supports the theoretical result from Arora et al.
Revised: This result somewhat supports the theoretical result from Arora et al.

Original: (2017), where the generator capacity needs to be modulated in order for approximately pure equilibrium to exist for GANs.
Revised: (2017), where the generator capacity needs to be modulated in order for approximately pure equilibrium to exist for GANs.

Original: “Small” number of ﬁlters for both discriminator and generator (ref.
Revised: “Small” number of ﬁlters for both discriminator and generator (ref.

Original: (a) in Table 9).
Revised: (a) in Table 9).

Original: “Small” and “large” number of ﬁlters for discriminator and generator respectively (ref.
Revised: “Small” and “large” number of ﬁlters for discriminator and generator respectively (ref.

Original: (e) in
Revised: (e) in

Original: Table 9).
Revised: Table 9).

Original: “Large” and “small” number of ﬁlters for discriminator and generator respectively (ref.
Revised: “Large” and “small” number of ﬁlters for discriminator and generator respectively (ref.

Original: (f) in
Revised: (f) in

Original: Table 9).
Revised: Table 9).

Original: “Large” number of ﬁlters for both discriminator and generator (ref.
Revised: “Large” number of ﬁlters for both discriminator and generator (ref.

Original: (d) in Table 9).
Revised: (d) in Table 9).

Original: Lastly, we experimented with how performance changes with respect to the dimension of the noise vectors.
Revised: Lastly, we experimented with how performance changes with respect to the dimension of the noise vectors.

Original: The source of the sample starts by transforming a noise vector into a meaningful image.
Revised: The source of the sample starts by transforming a noise vector into a meaningful image.

Original: It is unclear how the size of noise affects the ability of the generator to generate a meaningful image.
Revised: It is unclear how the size of noise affects the ability of the generator to generate a meaningful image.

Original: Che et al.
Revised: Che et al.

Original: (2017) have observed that a 100-d noise vector preserves modes better than a 200-d noise vector for DCGAN.
Revised: (2017) have observed that a 100-d noise vector preserves modes better than a 200-d noise vector for DCGAN.

Original: Our experiments show that this depends on the model.
Revised: Our experiments show that this depends on the model.

Original: Given a ﬁxed size architecture (d) from Table 9, we observed the performance of LS-DCGAN and W-DCGAN by varying the size of noise vector z .
Revised: Given a ﬁxed size architecture (d) from Table 9, we observed the performance of LS-DCGAN and W-DCGAN by varying the size of noise vector z .

Original: Table 11 illustrates that LS-DCGAN gives the best score with a noise dimension of 50 and W-DCGAN gives best score with a noise dimension of 150 for both IW and LS.
Revised: Table 11 illustrates that LS-DCGAN gives the best score with a noise dimension of 50 and W-DCGAN gives best score with a noise dimension of 150 for both IW and LS.

Original: The outcome of LS-DCGAN is consistent with the result in (Che et al., 2017).
Revised: The outcome of LS-DCGAN is consistent with the result in (Che et al., 2017).

Original: It is possible that this is because both fall into the category of f -divergences, whereas the W-DCGAN behaves differently because its metric falls under a different category, the Integral Probability Metric.
Revised: It is possible that this is because both fall into the category of f -divergences, whereas the W-DCGAN behaves differently because its metric falls under a different category, the Integral Probability Metric.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4.3. P ERFORMANCE CHANGE WITH RESPECT TO THE RATIO OF NUMBER OF UPDATES BETWEEN THE GENERATOR AND DISCRIMINATOR
Revised: 4.3. P ERFORMANCE CHANGE WITH RESPECT TO THE RATIO OF NUMBER OF UPDATES BETWEEN THE GENERATOR AND DISCRIMINATOR

Original: In practice, we alternate the updates between the discriminator and generator
Revised: In practice, we alternate the updates between the discriminator and generator

Original: [Figure]
Revised: [Figure]

Original: and yet this is not guaranteed to give the same result as the solution to the min-max problem in Equation 2.
Revised: and yet this is not guaranteed to give the same result as the solution to the min-max problem in Equation 2.

Original: Hence, the update ratio can inﬂuence the performance of GANs.
Revised: Hence, the update ratio can inﬂuence the performance of GANs.

Original: We experimented with three different update ratios, 5 : 1 , 1 : 1 , and 1 : 5 , with respect to the discriminator and generator update.
Revised: We experimented with three different update ratios, 5 : 1 , 1 : 1 , and 1 : 5 , with respect to the discriminator and generator update.

Original: We applied these ratios to both the MNIST and CIFAR-10 datasets on all models.
Revised: We applied these ratios to both the MNIST and CIFAR-10 datasets on all models.

Original: Figure 7 presents the LS scores on both MNIST and CIFAR-10 and this result is consistent under the
Revised: Figure 7 presents the LS scores on both MNIST and CIFAR-10 and this result is consistent under the

Original: IW metric as well (see S.M. Figure 20).
Revised: IW metric as well (see S.M. Figure 20).

Original: However, we did not ﬁnd
Revised: However, we did not ﬁnd

Original: any pattern of one update ratio being superior over others among the two datasets.
Revised: any pattern of one update ratio being superior over others among the two datasets.

Original: For CIFAR-10, 1 : 1 update ratio worked the best for all models, and yet for MNIST, different ratios worked the best for different models.
Revised: For CIFAR-10, 1 : 1 update ratio worked the best for all models, and yet for MNIST, different ratios worked the best for different models.

Original: Hence, we conclude that number of update ratio needs to be dynamically tuned.
Revised: Hence, we conclude that number of update ratio needs to be dynamically tuned.

Original: The corresponding samples from the different models trained by different update ratios are shown in S.M. Figure 22.
Revised: The corresponding samples from the different models trained by different update ratios are shown in S.M. Figure 22.

Original: 4.3. P ERFORMANCE WITH RESPECT TO THE AMOUNT OF AVAILABLE TRAINING DATA
Revised: 4.3. P ERFORMANCE WITH RESPECT TO THE AMOUNT OF AVAILABLE TRAINING DATA

Original: [Figure]
Revised: [Figure]

Original: In practice, DCGANs are known to be unstable, and the generator suffers as the discriminator gets better, due to disjoint support between the data and generator distributions (Goodfellow et al., 2014; Arjovsky & Bottou, 2017).
Revised: In practice, DCGANs are known to be unstable, and the generator suffers as the discriminator gets better, due to disjoint support between the data and generator distributions (Goodfellow et al., 2014; Arjovsky & Bottou, 2017).

Original: W-DCGAN and LS-DCGAN offer alternative ways to solving this problem in GANs.
Revised: W-DCGAN and LS-DCGAN offer alternative ways to solving this problem in GANs.

Original: If the model is suffering from disjoint support, having more training examples will not help, and alternatively, if the model does not suffer from such a problem, having more training examples could potentially help.
Revised: If the model is suffering from disjoint support, having more training examples will not help, and alternatively, if the model does not suffer from such a problem, having more training examples could potentially help.

Original: Here, we explore how sensitive three different kinds of
Revised: Here, we explore how sensitive three different kinds of

Original: GANs are with respect to the number of training examples.
Revised: GANs are with respect to the number of training examples.

Original: We have trained GANs with 10,000, 20,000, 30,000, 40,000, and 45,000 examples on CIFAR-10.
Revised: We have trained GANs with 10,000, 20,000, 30,000, 40,000, and 45,000 examples on CIFAR-10.

Original: Figureshows that the LS score curve of DCGAN grows quite slowly when compared to W-DCGAN and LS-DCGAN.
Revised: Figureshows that the LS score curve of DCGAN grows quite slowly when compared to W-DCGAN and LS-DCGAN.

Original: The three GANs have relatively the same loss when they are trained with 10,000 training examples.
Revised: The three GANs have relatively the same loss when they are trained with 10,000 training examples.

Original: However, the DCGAN only gained 0 .
Revised: However, the DCGAN only gained 0 .

Original: 0124 ± 0 . 

Revised: 0124 ± 0 .

Original: by increasing from 10,000 to 40,000 training examples, whereas the performance of W-DCGAN and
Revised: by increasing from 10,000 to 40,000 training examples, whereas the performance of W-DCGAN and

Original: LS-DCGAN improved by 0 .
Revised: LS-DCGAN improved by 0 .

Original: 03016 ± 0 . 00469 and 0 . 0444 ± 0 . 0033 , respectively.
Revised: 03016 ± 0 . 00469 and 0 . 0444 ± 0 . 0033 , respectively.

Original: Thus, we empirically observe that W-DCGAN and LS-DCGAN increases their performance faster than DCGAN as the number of training examples grows.
Revised: Thus, we empirically observe that W-DCGAN and LS-DCGAN increases their performance faster than DCGAN as the number of training examples grows.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed four well-known distance functions as an evaluation metrics, and empirically investigated the DCGAN, W-DCGAN, and LS-DCGAN families under these metrics.
Revised: In this paper, we proposed four well-known distance functions as an evaluation metrics, and empirically investigated the DCGAN, W-DCGAN, and LS-DCGAN families under these metrics.

Original: Previously, these models were compared based visual assessment of sample quality and difﬁculty of training.
Revised: Previously, these models were compared based visual assessment of sample quality and difﬁculty of training.

Original: In our experiments, we showed that there are performance differences in terms of average experiments, but that some are not statistically signiﬁcant.
Revised: In our experiments, we showed that there are performance differences in terms of average experiments, but that some are not statistically signiﬁcant.

Original: Moreover, we thoroughly analyzed the performance of GANs under different hyper-parameter settings.
Revised: Moreover, we thoroughly analyzed the performance of GANs under different hyper-parameter settings.

Original: There still are many different type of GANs that need to be evaluated, such as GRAN (Im et al., 2016a), IWGAN (Gulrajani et al., 2017), BEGAN (Berthelot et al., 2017), MMDGAN (Li et al., 2017), and CramerGAN (Bellemare et al., 2017).
Revised: There still are many different type of GANs that need to be evaluated, such as GRAN (Im et al., 2016a), IWGAN (Gulrajani et al., 2017), BEGAN (Berthelot et al., 2017), MMDGAN (Li et al., 2017), and CramerGAN (Bellemare et al., 2017).

Original: We hope to evaluate all of these models under this framework and thoroughly analyze them in the future.
Revised: We hope to evaluate all of these models under this framework and thoroughly analyze them in the future.

Original: Moreover, there has been work on taking ensemble approaches to GANs, such as Generative Adversarial Parallelization Im et al.
Revised: Moreover, there has been work on taking ensemble approaches to GANs, such as Generative Adversarial Parallelization Im et al.

Original: (2016b).
Revised: (2016b).

Original: Ensemble approaches have been empirically shown to work well in many domains of research, so it would be interesting to ﬁnd out whether ensembles can also help in min-max problems.
Revised: Ensemble approaches have been empirically shown to work well in many domains of research, so it would be interesting to ﬁnd out whether ensembles can also help in min-max problems.

Original: Alternatively, we can also try to evaluate other log-likelihood-based
Revised: Alternatively, we can also try to evaluate other log-likelihood-based

Original: Model evaluation is an important and complex topic.
Revised: Model evaluation is an important and complex topic.

Original: Model selection, model design, and even research directions, can change depending on the evaluation metric.
Revised: Model selection, model design, and even research directions, can change depending on the evaluation metric.

Original: Thus, we need to continuously explore different metrics and rigorously evaluate new models.
Revised: Thus, we need to continuously explore different metrics and rigorously evaluate new models.

Original: A PPENDIX R ELATIONSHIP BETWEEN THE METRICS AND BINARY C LASSIFICATION
Revised: A PPENDIX R ELATIONSHIP BETWEEN THE METRICS AND BINARY C LASSIFICATION

Original: In this paper, we considered four distance metrics that belong to two class of metrics, φ -divergence and IPMs.
Revised: In this paper, we considered four distance metrics that belong to two class of metrics, φ -divergence and IPMs.

Original: Sriperumbudur et al.
Revised: Sriperumbudur et al.

Original: (2009) have shown that the optimal risk function is associated with a binary classiﬁer with P and Q distribution conditioned on class when the discriminant function is restricted to certain F (Theorem 17 from (Sriperumbudur et al., 2009)).
Revised: (2009) have shown that the optimal risk function is associated with a binary classiﬁer with P and Q distribution conditioned on class when the discriminant function is restricted to certain F (Theorem 17 from (Sriperumbudur et al., 2009)).

Original: Let the optimal risk function be
Revised: Let the optimal risk function be

Original: [Equation]
Revised: [Equation]

Original: where F is the set of discriminant functions (classiﬁer), y ∈ − 1 , 1 , and L is the loss function.
Revised: where F is the set of discriminant functions (classiﬁer), y ∈ − 1 , 1 , and L is the loss function.

Original: By following derivation, we can see that the optimal risk function becomes IPM:
Revised: By following derivation, we can see that the optimal risk function becomes IPM:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where L (1 , f ( x ))
Revised: where L (1 , f ( x ))

Original: = 1 /(cid:15) and L (0 , f ( x ))
Revised: = 1 /(cid:15) and L (0 , f ( x ))

Original: = − 1 / (1 − (cid:15) ) .
Revised: = − 1 / (1 − (cid:15) ) .

Original: The second equality is derived by separating the loss for class 1 and class 0.
Revised: The second equality is derived by separating the loss for class 1 and class 0.

Original: Third equality is from the way how we chose L(1,f(x)) and L(0,f(x)).
Revised: Third equality is from the way how we chose L(1,f(x)) and L(0,f(x)).

Original: The last equality follow by the fact that F is symmetric around zero ( f ∈ F = > − f ∈ F ) .
Revised: The last equality follow by the fact that F is symmetric around zero ( f ∈ F = > − f ∈ F ) .

Original: Hence, this shows that with appropriately choosing L, MMD,
Revised: Hence, this shows that with appropriately choosing L, MMD,

Original: Wasserstein distance can be understood as the optimal L-risk associated with binary classiﬁer with speciﬁc set of F functions.
Revised: Wasserstein distance can be understood as the optimal L-risk associated with binary classiﬁer with speciﬁc set of F functions.

Original: For example, Wasserstein distance and MMD distances are equivalent to optimal risk function with 1-Lipschitz classiﬁers and RKHS classiﬁer with an unit length.
Revised: For example, Wasserstein distance and MMD distances are equivalent to optimal risk function with 1-Lipschitz classiﬁers and RKHS classiﬁer with an unit length.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: M ORE EXPERIMENTS
Revised: M ORE EXPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Table] [Table]

Revised: We trained two critics on training data and validation data respectively and evaluated on test data from both critics.

Revised: We trained six GANs (GAN, LSGAN, WGAN GP, DRAGAN, BEGAN, EBGAN) on MNIST and Fashion MNIST.

Revised: We trained these GANs with 50K training examples.

Revised: At test time, we used 10k training and 10k validation examples for training the critics, and evaluated on 10k test examples.

Revised: Here, we present the test scores from the critics trained on training and validation data.

Original: [Figure]
Revised: The results shown in Table ?? Note that we also have the IW and FID evaluation on these models in the paper. For Fashion-MNIST, we ﬁnd that test scores with critic trained on training and validation data are very close. Hence, we don’t see any indication of overﬁtting.

Revised: On the other hand, there are gaps between the scores for the MNIST dataset and the test scores from critics trained on validation set gives better performance than the ones that are trained on the training set.

Original: [Figure]
Revised: [Figure]

Original: I n c e p t
Revised: I n c e p t

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: c o
Revised: c o

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 1e
Revised: 1e

Original: [Figure]
Revised: [Figure]

Original: 5e
Revised: 5e

Original: [Figure]
Revised: [Figure]

Original: 6e
Revised: 6e

Original: [Figure]
Revised: [Figure]

Original: J S d i v e r g e n c e
Revised: J S d i v e r g e n c e

Original: DCGAN
Revised: DCGAN

Original: LS
Revised: LS

Original: [Figure]
Revised: [Figure]

Original: DCGAN
Revised: DCGAN

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Ratio
Revised: Ratio

Original: DCGAN Samples 1: 1:
Revised: DCGAN Samples 1: 1:

Original: Ratio
Revised: Ratio

Original: W-DCGAN Samples 5: 1: 1:
Revised: W-DCGAN Samples 5: 1: 1:

Original: Ratio
Revised: Ratio

Original: LS-DCGAN Samples 5: 1: 1:
Revised: LS-DCGAN Samples 5: 1: 1:

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]