Paper:
Published as a conference paper at ICLR 2018 QUANTITATIVELY EVALUATING GANS WITH DIVERGENCES PROPOSED FOR TRAINING
Generative adversarial networks (GANs) have been extremely effective in approximating complex distributions of high-dimensional, input data samples, and substantial progress has been made in understanding and improving GAN performance in terms of both theory and application. However, we currently lack quantitative methods for model assessment. Because ot this, while many GAN variants being proposed, we have relatively little understanding of their relative abilities. In this paper, we evaluate the performance of various types of GANs using divergence and distance functions typically used only for training. We observe consistency across the various proposed metrics and, interestingly, the test-time metrics do not favour networks that use the same training-time criterion. We also compare the proposed metrics to human perceptual scores.

Review:
This paper proposes using divergence and distance functions typically used for generative model training to evaluate the performance of various types of GANs. Through numerical evaluation, the authors observed that the behavior is consistent across various proposed metrics and the test-time metrics do not favor networks that use the same training-time criterion. 
More specifically, the evaluation metric used in the paper are: 1) Jensen-Shannon divergence, 2) Constrained Pearson chi-squared, 3) Maximum Mean Discrepancy, 4) Wasserstein Distance, and 5) Inception Score. They applied those metrics to compare three different GANs: the standard DCGAN, Wasserstein DCGAN, and LS-DCGAN on MNIST and CIFAR-10 datasets. 
Summary:
——
In summary, it is an interesting topic, but I think that the paper does not have sufficient novelty. Some empirical results are still preliminary. It is hard to judge the effectiveness of the proposed metrics for model selection and is not clear that those metrics are better qualitative descriptors to replace visual assessment. In addition, the writing should be improved. See comments below for details and other points.
Comments:
——
1.	In Section 3, the evaluation metrics are existing metrics and some of them have already been used in comparing GAN models.  Maximum mean discrepancy has been used before in work by Yujia Li et al. (2016, 2017)
2.	In the experiments, the proposed metrics were only tested on small scale datasets; the authors should evaluate on larger datasets such as CIFAR-100, Toronto Faces, LSUN bedrooms or CelebA.
3.	In the experiments, the authors noted that “Gaussian observable model might not be the ideal assumption for GANs. Moreover, we observe a high log-likelihood at the beginning of training, followed by a drop in likelihood, which then returns to the high value, and we are unable to explain why this happens.” Could the authors give explanation to this phenomenon? The authors should look into this more carefully.
4.	In algorithm 1, it seems that the distance is computed via gradient decent. Is it possible to show that the optimization always converges? Is it meaningful to compare the metrics if some of them cannot be properly computed?
5.     With many different metrics for assessing GANs, how should people choose? How do we trust the scores? Recently, Fréchet Inception Distance (FID) was proposed to evaluate the samples generated from GANs (Heusel et al. 2017), how are the above scores compared with FID?
Minor Comments:
——
1.	Writing should be fixed: “It seems that the common failure case of MMD is when the mean pixel intensities are a better match than texture matches (see Figure 5), and the common failure cases of IS happens to be when the samples are recognizable textures, but the intensity of the samples are either brighter or darker (see Figure 2).”