Paper:
Conditional generation refers to the process of sampling from an unknown distribution conditioned on semantics of the data. This can be achieved by augmenting the generative model with the desired semantic labels, albeit it is not straightforward in an unsupervised setting where the semantic label of every data sample is unknown. In this paper, we address this issue by proposing a method that can generate samples conditioned on the properties of a latent distribution engineered in accordance with a certain data prior. In particular, a latent space inversion network is trained in tandem with a generative adversarial network such that the modal properties of the latent space distribution are induced in the data generating distribution. We demonstrate that our model, despite being fully unsupervised, is effective in learning meaningful representations through its mode matching property. We validate our method on multiple unsupervised tasks such as conditional generation, dataset attribute discovery and inference using three real world image datasets namely MNIST, CIFAR-10 and CelebA and show that the results are comparable to the state-of-the-art methods.

Review:
This paper is concerned with the so-called conditional generation, which was descried as the task of sampling from an unknown distribution conditioned on semantics of the data. This proposed method aims at achieving this by using a latent distribution engineered according to a certain data prior. Experimental results showed that the proposed method seems to produce good results.
I have several questions about the motivation and the method in the paper. First, it is not clear to me how the "semantics" of the data was defined. Is it given by visual inspection? Is it possible to find it with some automated method? Second, the authors seem to advocate the idea that data of a k-mode distribution should be generated from a k-mode latent distribution. It might be useful in certain scenarios; however, it is not clear why the transformation from the latent to the observed does not change the number of modes or why keeping the same number of modes would endow the latent distribution a "semantics" meaning. We know that a k-mode distribution can be obtained by applying a smooth nonlinear transformation to a Gaussian or uniform distribution and, similarly, a k-mode distribution can be transformed to a single-mode distribution with a smooth mapping. So I am not sure why engineering the latent distribution this way can give it a "semantics" meaning. Should we try to enforce a kind of smoothness of the transformation, by, say, penalizing high derivative values? Third, the experimental results seem nice, but the lack of comparisons blurs the advantage of the proposed method. How is the result produced by GAN compared to the reported one? How did the original GAN with the engineering latent distribution work? It would be appreciated if the authors could address these issues more clearly.