Paper:
Deep generative models can learn to generate realistic-looking images, but many of the most effective methods are adversarial and involve a saddlepoint optimization, which require careful balancing of training between a generator network and a critic network. Maximum mean discrepancy networks (MMD-nets) avoid this issue by using kernel as a fixed adversary, but unfortunately they have not on their own been able to match the generative quality of adversarial training. In this work, we take their insight of using kernels as fixed adversaries further and present a novel method for training deep generative models that does not involve saddlepoint optimization. We call our method generative ratio matching or GRAM for short. In GRAM, the generator and the critic networks do not play a zero-sum game against each other, instead they do so against a fixed kernel. Thus GRAM networks are not only stable to train like MMD-nets but they also match and beat the generative quality of adversarially trained generative networks.

Review:
The proposed approach amounts to a non-adversarial way to update the parameters theta of a kernel K_theta(x, y) = k(f_theta(x), f_theta(y)) in an MMD-net, rather than adversarially as in MMD-GANs. Avoiding the saddle point problem clearly gives a more stable optimization problem, as shown in Section 3.4, which is very nice.
Theory/motivation:
The motivation based on density ratio estimation, however, very much assumes that these densities exist. As convincingly argued by Arjovsky and Bottou (ICLR 17, "Towards principled methods for training generative adversarial networks"), the densities p_x, q_x of (4) typically do not exist (w.r.t. Lebesgue measure), and moreover the support of the distributions are typically almost disjoint. Algorithmically, this doesn't seem to be a major issue for your approach: the estimator (8) corresponds to some kind of smoothing by the kernel. But it would be nice from a motivation point of view to have a better understanding of:
- What assumptions exactly does (8) make about the existence of densities? For example, in the case of a Gaussian kernel, does it correspond to the distance between distributions convolved with Gaussian noises with a smaller Gaussian kernel? Does the algorithm "make sense" when distributions don't exist?
- Can we say something about when a density exists for the dimensionality-reduced data f_theta(x) even when it doesn't exist for the data x? 
- Relatedly, the dimension-reduced Pearson divergence is clearly somehow different from the plain Pearson divergence. Can we understand a little bit more how it's different? How does the choice of architecture of f_theta affect the distance metric?
- Is it the case, as in Arjovsky et al. (2017) or in Arbel et al. (NeurIPS 2018, "On gradient regularizers for MMD GANs"), that the overall loss function with an optimal f_theta is continuous with respect to the GAN distribution?
Certainly not all of these theoretical questions require answers for an initial ICLR paper, but they would be illuminating for understanding the method.
Motivating experiments:
- In Figure 2, it seems that except in the first image (h=2) the mapping f_theta becomes essentially the identity. Is this expected to be generally true in your 2d cases, because the density ratio can already be well-represented there? A more interesting example for this aspect of the model might involve higher-dimensional distributions with some low-dimensional manifold structure; would f_theta pick up some reasonable representation of the manifold in those cases?
- Can you plot the density ratio estimate in a simple 2d example like this? How does it look visually? Is a good estimate necessary for good performance of your generative model?
Experiments:
- As shown by Binkowski et al., it's extremely important to specify the sample size for FID scores, which you don't do.
- That said, the FID scores of the MMD GAN models of Binkowski et al using DCGAN discriminators (which replace the FSR and AE penalties of Li et al. with a single gradient penalty) found substantially smaller penalties for performance of MMD-GANs with using a smaller discriminator (1/4 as many hidden units rather than 1/2 as you use) than you did. They also reported substantially better numbers on CelebA than you did. If not too difficult code-wise, it might be worth trying either their code or that of the followup Arbel et al. (citation above) to compare to your approach.
- The results of, for example, Mescheder et al. (ICML-18, "Which Training Methods for GANs do actually Converge?") and Arbel et al. (NeurIPS-18, "On gradient regularizers for MMD GANs") seem to be much better visually with comparably-sized models than the results of Figure 8 (and of course StyleGAN/etc are *far* better but with drastically larger models). It would be good to compare empirically to some more recent models than original MMD GANs (whose empirical results were substantially improved with only minor tweaks by Binkowski et al. mere months later) or DCGAN, which are fairly old approaches in this space.
- Two more extremely relevant methods it would be worth comparing to or at least mentioning:
   - Ravuri et al. (ICML 2018, "Learning Implicit Generative Models with the Method of Learned Moments") can be interpreted as learning f_theta for an MMD-net as you do by taking the gradients of an auxiliary classification network. Although it is still somewhat "adversarial," they need to update the kernel function extremely rarely (every 2,000 generator update steps) and the optimization seems overall much more stable.
   - dos Santos et al. (https://arxiv.org/abs/1904.02762 "Learning Implicit Generative Models by Matching Perceptual Features") effectively use an MMD-net (with some slight tweaks) with a *fixed* kernel. Though results aren't quite as good, it's worth at least mentioning.
Minor typos/etc:
- Top of page 3: it is sufficient to choose F a unit ball in an RKHS with a _characteristic_ kernel.
- Page 4: "this issue in next."
- Figure 1 caption: (both) should probably be (bottom).
Overall initial thoughts:
This seems like a nice alternative to adversarial methods, but it does not compare to more recent (last 2 years) models in this space or very thoroughly establish the applicability of its motivation. I think it's worthy of a weak accept as is, but could be much more convinced with some additional work.