Paper:
Variational Autoencoders (VAEs) are a popular generative model, but one in which conditional inference can be challenging. If the decomposition into query and evidence variables is fixed, conditional VAEs provide an attractive solution. To support arbitrary queries, one is generally reduced to Markov Chain Monte Carlo sampling methods that can suffer from long mixing times. In this paper, we propose an idea we term cross-coding to approximate the distribution over the latent variables after conditioning on an evidence assignment to some subset of the variables. This allows generating query samples without retraining the full VAE. We experimentally evaluate three variations of cross-coding showing that (i) they can be quickly optimized for different decompositions of evidence and query and (ii) they quantitatively and qualitatively outperform Hamiltonian Monte Carlo.  (Girolami & Calderhead, 2011;Daniel Levy, 2018) are difficult to adapt to these problems, and can suffer from long mixing times as we show empirically.
To remedy the limitations of existing methods for conditional inference in VAEs, we aim to approximate the distribution over the latent variables after conditioning on an evidence assignment through a variational Bayesian methodology. In doing this, we reuse the decoder of the VAE and show that the error of the distribution over query variables is controlled by that over latent variables via a fortuitous cancellation in the KL-divergence. This avoids the computational expense of re-training the decoder as done by the CVAE approach. We term the network that generates the conditional latent distribution the cross-coder.
We experiment with two cross-coding alternatives: Gaussian variational inference via a linear transform (GVI) and Normalizing Flows (NF). We also provide some comparison to a fully connected network (FCN), which suffers from some technical and computational issues but provides a useful point of reference for experimental comparison purposes. Overall, our results show that the GVI and NF variants of cross-coding can be optimized quickly for arbitrary decompositions of query and evidence and compare favorably against a ground truth provided by rejection sampling for low latent dimensionality. For high dimensionality, we observe that HMC often fails to mix despite our systematic efforts to tune its parameters and hence demonstrates poor performance compared to cross-coding in both quantitative and qualitative evaluation.
1 Under review as a conference paper at ICLR 2019 Algorithm 1 Conditional Inference via Cross-coding. Input (a) Pre-trained VAE p(z)p θ (t|z) with p θ (t|z) based on Decoder θ (z). (Encoder ignored.) (b) Single query x (any subset of t) for which to predict y. (Rest of t.) Optimize Define q( )q ψ (z| ) with q ψ (z| ) based on XCoder ψ ( ). Find ψ to maximize C-ELBO[q ψ (Z) p θ (Z, x)] (Defined in Theorem 2). Estimate stochastic gradients by drawing random ∼ q( ) and using the reparameterization trick. Predict Draw a sample {z m } M m=1 ∼ q ψ (z) by setting z m = XCoder ψ ( m ) for m ∼ q( ). Predict p θ (y|x) ≈

Review:
(apologies for this belated review)
Summary
The authors consider the task of imputing missing data using variational auto-encoders. To do so, they assume a fixed pre-trained generative model, perform variational inference to infer a posterior on latent variables given a partial image, and then use this approximate posterior to predict missing pixels. They compare a variety of parameterizations of the variational distribution to HMC inference, and evaluate on MNIST, Celeb-A and the Anime data. 
Comments
There are many things about this paper that I don’t understand. My main concern is that I fail to follow why the authors are interested in this task. In what settings would we be interested in performing non-autoencoding variational inference in order to impute missing data? Moreover, in cases where are interested in performing such imputations, what would we like to use the results for? This paper seems like a nice demo, but I’m not entirely convinced I see a compelling application. 
My second concern is about the baselines that are considered. If I were interested in carrying out this inference task, my inclination would not be to run an HMC chain to convergence, but instead to do something like annealed importance sampling (AIS), where at each step I run an iteration of HMC on a large batch of samples on a sequence of target densities that interpolate between the prior and full joint p(x, Z). If computational cost is a concern, I imagine this would not be more expensive than training a density estimator. Moreover, whereas HMC is generally not known to be a good method for estimating marginal likelihoods, AIS methods generally perform much better.
Finally I find the language used in this paper confusing. Cross-coding seems a misnomer for the technique that the authors propose. Isn’t this simply a form of variational inference in which qψ(Z) approximates pθ(Ζ | x)? The term “-coding” suggests that we somehow define an encoder that accepts the query as input. Moreover, isn’t the XCoder network just a neural density estimator? 
Finally, Lemma 1 seems like a really roundabout way of deriving a lower bound. The authors could instead just write:
	log p(x)
	>=
	E_q(Z,Y)[log p(x, Y, Z) - log q(Z, Υ)]
	=
	E_q(Z,Y)[log p(x | Ζ) + log p(Y | Z) + log p(Z) - log q(Z) - log p(Υ | Z)]
	=
	E_q(Z,Y)[log p(x | Ζ) + log p(Z) - log q(Z)]
	=
	E_q(Z)[log p(x, Ζ) - log q(Z)]
This avoids confusing terminology such as cross-coding, and shows that what the authors are doing is in fact just variational inference. Am I missing something here?
I am also confused about how the comparison to HMC is set up. If you’re training qψ(Z), then you presumably need generate a certain number samples at training time. Shouldn’t you add this number of samples number of samples you generate in HMC, in order to get a more apples to apples comparison in terms of the amount of computation performed? As it stands, it is hard to evaluate whether these methods are given a similar number of samples. 
Finally, I am not quite sure what to make of the experimental evaluation. We see some scatter plots on MNIST with a 2D latent space, and some faces of celebrities in which there is arguably some sample diversity, although most of this diversity arises in blurry looking hairstyles. However, since the authors condition on the eyes, rather than, say, the nose or mouth, it is hard to know how good a job the network is doing at generalizing to multiple plausible faces. 
Overall, I find it difficult to judge the merit of this paper. Is this task in fact hard? Is it useful? Are the results good? Maybe the authors can give us some additional guidance on these questions.
Questions
- I’m a bit worried that not all the samples that we see in Figure 6 may have equally high probability under the posterior. Could the authors compute and report importance weights?
	W = p(x, Z) / q(Z)
	
- Could the authors say something about the effective sample size that we obtain when using the learned distribution q(Z) as a proposal? 
	
	ESS = (Σ_k w^k)^2 / (Σ_k (w^k)^2)
- Should it be the case that the ESS is low, and the weights are high variance, could the authors generate a sufficient number of samples to ensure the the ESS = 25 (i.e. the number of images in the figure) and then show the 25 highest-weight samples (or resample 25 images with probability proportional to their weight)?
	
Minor 
- Equation (3): There’s an extra p_θ in the first integral
- In the proof in Appendix 6.1 
	KL[ qψ(Z) ‖ pθ(Z | x) ] + KL[ qψ(Y | Z) ‖ pθ(Y | Z, x)]
it would be clearer to explicitly denote the expectation over qψ(Z)
	KL[ qψ(Z) ‖ pθ(Z | x) ] + E_qψ(Z)[ KL[ qψ(Y | Z) ‖ pθ(Y | Z, x)] ]
	
(I had to google lecture notes to find out that this expectation is sometimes implicit, which 
as far as I know is not very standard).