Paper:
Generative networks are known to be difficult to assess. Recent works on generative models, especially on generative adversarial networks, produce nice samples of varied categories of images. But the validation of their quality is highly dependent on the method used. A good generator should generate data which contain meaningful and varied information and that fit the distribution of a dataset. This paper presents a new method to assess a generator. Our approach is based on training a classifier with a mixture of real and generated samples. We train a generative model over a labeled training set, then we use this generative model to sample new data points that we mix with the original training data. This mixture of real and generated data is thus used to train a classifier which is afterwards tested on a given labeled test dataset. We compare this result with the score of the same classifier trained on the real training data mixed with noise. By computing the classifier's accuracy with different ratios of samples from both distributions (real and generated) we are able to estimate if the generator successfully fits and is able to generalize the distribution of the dataset. Our experiments compare different generators from the VAE and GAN framework on MNIST and fashion MNIST dataset.

Review:
The authors propose to evaluate how well generative models fit the training set by analysing their data augmentation capacity, namely the benefit brought by training classifiers on mixtures of real/generated data, compared to training on real data only. Despite the the idea of exploiting generative models to perform data augmentation is interesting, using it as an evaluation metric does not constitute an innovative enough contribution. 
In addition, there is a fundamental matter which the paper does not address: when evaluating a generative model, one should always ask himself what purpose the data is generated for. If the aim is to have realistic samples, a visual turing test is probably the best metric. If instead the purpose is to exploit the generated data for classification, well, in this case an evaluation of the impact of artificial data over training is a good option.
PROS:
The idea is interesting. 
CONS:
1. The authors did not relate the proposed evaluation metric to other metrics cited (e.g., the inception score, or a visual turing test, as discussed in the introduction). It would be interesting to understand how the different metrics relate. Moreover, the new metric is introduced with the following motivation “[visual Turing test and Inception Score] do not indicate if the generator collapses to a particular mode of the data distribution”. The mode collapse issue is never discussed elsewhere in the paper. 
2. Only two datasets were considered, both extremely simple: generating MNIST digits is nearly a toy task nowadays. Different works on GANs make use of CIFAR-10 and SVHN, since they entail more variability: those two could be a good start. 
3. The authors should clarify if the method is specifically designed for GANs and VAEs. If not, section 2.1 should contain several other works (as in Table 1). 
4. One of the main statements of the paper “Our approach imposes a high entropy on P(Y) and gives unbiased indicator about entropy of both P(Y|X) and P(X|Y)” is never proved, nor discussed.
5. Equation 2 (the proposed metric) is not convincing: taking the maximum over tau implies training many models with different fractions of generated data, which is expensive. Further, how many tau’s one should evaluate? In order to evaluate a generative model one should test on the generated data only (tau=1) I believe. In the worst case, the generator experiences mode collapse and performs badly. Differently, it can memorize the training data and performs as good as the baseline model. If it does actual data augmentation, it should perform better.
6. The protocol of section 3 looks inconsistent with the aim of the work, which is to evaluate data augmentation capability of generative models. In fact, the limit of training with a fixed dataset is that the model ‘sees’ the data multiple times across epochs with the risk of memorizing. In the proposed protocol, the model ‘sees’ the generated data D_gen (which is fixed before training) multiple time across epochs. This clearly does not allow to fully evaluate the capability of the generative model to generate newer and newer samples with significant variability.
Minor: 
Section 2.2 might be more readable it divided in two (exploitation and evaluation).