Paper:
Despite the widespread interest in generative adversarial networks (GANs), few works have studied the metrics that quantitatively evaluate GANs' performance. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the important problem of how to evaluate the evaluation metrics. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. Then with a series of carefully designed experiments, we are able to comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbour (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far these state-of-the-art GANs are from perfect.

Review:
This paper introduces a comparison between several approaches for evaluating GANs. The authors consider the setting of a pre-trained image models as generic representations of generated and real images to be compared. They compare the evaluation methods based on five criteria termed disciminability, mode collapsing and mode dropping, sample efficiency,computation efficiency, and robustness to transformation. This paper has some interesting insights and a few ideas of how to validate an evaluation method. The topic is an important one and a very difficult one. However, the work has some problems in rigor and justification and the conclusions are overstated in my view.
Pros
-Several interesting ideas for evaluating evaluation metrics are proposed
-The authors tackle a very challenging subject
Cons
-It is not clear why GANs are the only generative model considered
-Unprecedented visual quality as compared to other generative models has brought the GAN to prominence and yet this is not really a big factor in this paper.
-The evaluations rely on using a pre-trained imagenet model as a representation. The authors point out that different architectures yield similar results for their analysis, however it is not clear how the biases of the learned representations affect the results. The use of learned representations needs more rigorous justification
-The evaluation for discriminative metric, increased score when mix of real and unreal increases, is interesting but it is not convincing as the sole evaluation for “discriminativeness” and seems like something that can be gamed. 
- The authors implicitly contradict the argument of Theis et al against monolithic evaluation metrics for generative models, but this is not strongly supported.
Several references I suggest:
https://arxiv.org/abs/1706.08500 (FID score)
https://arxiv.org/abs/1511.04581 (MMD as evaluation)