Paper:
Published as a conference paper at ICLR 2018 LATENT CONSTRAINTS: LEARNING TO GENERATE CONDITIONALLY FROM UNCONDITIONAL GENERATIVE MODELS
Deep generative neural networks have proven effective at both conditional and unconditional modeling of complex data distributions. Conditional generation enables interactive control, but creating new controls often requires expensive retraining. In this paper, we develop a method to condition generation without retraining the model. By post-hoc learning latent constraints, value functions that identify regions in latent space that generate outputs with desired attributes, we can conditionally sample from these regions with gradient-based optimization or amortized actor functions. Combining attribute constraints with a universal "realism" constraint, which enforces similarity to the data distribution, we generate realistic conditional images from an unconditional variational autoencoder. Further, using gradient-based optimization, we demonstrate identity-preserving transformations that make the minimal adjustment in latent space to modify the attributes of an image. Finally, with discrete sequences of musical notes, we demonstrate zero-shot conditional generation, learning latent constraints in the absence of labeled data or a differentiable reward function.

Review:
UPDATE: I think the authors' rebuttal and updated draft address my points sufficiently well for me to update my score and align myself with the other reviewers.
-----
ORIGINAL REVIEW: The paper proposes a method for learning post-hoc to condition a decoder-based generative model which was trained unconditionally. Starting from a VAE trained with an emphasis on good reconstructions (and at the expense of sample quality, via a small hard-coded standard deviation on the conditional p(x | z)), the authors propose to train two "critic" networks on the latent representation:
1. The "realism" critic receives either a sample z ~ q(z) (which is implicitly defined as the marginal of q(z | x) over all empirical samples) or a sample z ~ p(z) and must tell them apart.
2. The "attribute" critic receives either a (latent code, attribute) pair from the dataset or a synthetic (latent code, attribute) pair (obtained by passing both the attribute and a prior sample z ~ p(z) through a generator) and must tell them apart.
The goal is to find a latent code which satisfies both the realism and the attribute-exhibiting criteria, subject to a regularization penalty that encourages it to stay close to its starting point.
It seems to me that the proposed realism constraint hinges exclusively on the ability to implictly capture the marginal distribution q(z) via a trained discriminator. Because of that, any autoencoder could be used in conjunction with the realism constraint to obtain good-looking samples, including the identity encoder-decoder pair (in which case the problem reduces to generative adversarial training). I fail to see why this observation is VAE-specific. The authors do mention that the VAE semantics allow to provide some weak form of regularization on q(z) during training, but the way in which the choice of decoder standard deviation alters the shape of q(z) is not explained, and there is no justification for choosing one standard deviation value in particular.
With that in mind, the fact that the generator mapping prior samples to "realistic" latent codes works is expected: if the VAE is trained in a way that encourages it to focus almost exclusively on reconstruction, then its prior p(z) and its marginal q(z) have almost nothing to do with each other, and it is more convenient to view the proposed method as a two-step procedure in which an autoencoder is first trained, and an appropriate prior on latent codes is then learned. In other words, the generator represents the true prior by definition.
The paper is also rather sparse in terms of comparison with existing work. Table 1 does compare with Perarnau et al., but as the caption mentions, the two methods are not directly comparable due to differences in attribute labels.
Some additional comments:
- BiGAN [1] should be cited as concurrent work when citing (Dumoulin et al., 2016).
- [2] and [3] should be cited as concurrent work when citing (Ulyanov et al., 2016).
Overall, the relative lack of novelty and comparison with previous work make me hesitant to recommend the acceptance of this paper.
References:
[1] Donahue, J., Krähenbühl, P., and Darrell, T. (2017). Adversarial feature learning. In Proceedings of the International Conference on Learning Representations.
[2] Li, C., and Wand, M. (2016). Precomputed real-time texture synthesis with markovian generative adversarial networks. In European Conference on Computer Vision.
[3] Johnson, J., Alahi, A., and Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision.