Paper:
Gaussian Process Conditional Density Estimation
Conditional Density Estimation (CDE) models deal with estimating conditional distributions. The conditions imposed on the distribution are the inputs of the model. CDE is a challenging task as there is a fundamental trade-off between model complexity, representational capacity and overfitting. In this work, we propose to extend the model's input with latent variables and use Gaussian processes (GPs) to map this augmented input onto samples from the conditional distribution. Our Bayesian approach allows for the modeling of small datasets, but we also provide the machinery for it to be applied to big data using stochastic variational inference. Our approach can be used to model densities even in sparse data regions, and allows for sharing learned structure between conditions. We illustrate the effectiveness and wide-reaching applicability of our model on a variety of real-world problems, such as spatio-temporal density estimation of taxi drop-offs, non-Gaussian noise modeling, and few-shot learning on omniglot images.

Review:
Edit after the rebuttal period: I have read the other reviews and the author feedback. My evaluation and rating about this submission will remain unchanged.  Summary: The paper considers approximate inference and learning for the Gaussian process latent variables model in which some latent variables are observed. An alternative view of the model considered here is Gaussian process regression in which some dimensions of the inputs are unobserved. Variational inference based on pseudo-points is used. The posterior over the pseudo-points are approximately integrated out by quadrature or approximated by a Gaussian parameterised by a recognition model. Natural gradients and linear input and output warpings are also considered. Extensive experiments on density estimation and regression were provided to support the proposed inference scheme.  Comments:  The contribution of this work is, in my opinion, a well-executed framework for inference and learning in Gaussian process latent variable models with some latent variables being known beforehand. I think the contributions listed at the end of section 1 are incremental, as i. linear input and output warpings have been considered before in multiple contexts, for example: multi-task learning, ii. natural gradients for the variational approximation over the non-linear function have been considered before, e.g. Hensman et al's GP for big data paper. In addition, recognition models have been used before to parameterise the variational distribution in the GP latent variable model, and similar models have been considered before -- see the list at the end of the review. Having said this, this submission is well-structured and is an interesting and, perhaps, non-trivial combination of all these things that seems to work well.  re analytically optimal q(w): I agree this approach could be useful for low-dimensional w and there will be less variational parameters to deal with [and this could perform well as in the regression experiment considered], but I'm pedantically unclear about the bound being tighter here. The gaps between the bound and the true marginal likelihood are: in the free-form q(w) case: log \int dw p(w) \exp [ KL(q(f) || p(f|w, x, y)) ] in the fixed-form case: KL( q(f)q(w) || p(f, w|x, y) ) Is this true that the gap in the first case above is always smaller than that in the second case?  Some questions about the experiments: + linear model baselines should be included in the density modelling tasks + taxi trip density estimation: the test set has 4 folds with 50 taxi trips each -- this seems rather small compared to the whole dataset of 1.5 mil trips. + few shot learning on omniglot: the generated images are not very crisp. Is there some quantitative or qualitative comparison to alternative models here, e.g. VAE? + regression task: this seems like the only task the proposed model + inference technique have the competitive edge on average. Is this because the number of latent/input dimensions is small and no input + output warpings are used? The additional variational approximation for the input warping could make things worse in other tasks. + MNIST density estimation: why not use the full binarised MNIST as often considered in generative modelling literature? It is hard to judge the number provided table 2 as they are very different to what often reported. The Gaussian likelihood could be straightforwardly replaced by another likelihood for binary outputs, right? I'm also slightly puzzled by table 2 and the full table in the appendix: why is the first row for CVAE: sigma^2 optimised made bold? For fixed variances, GP-CDE is like 100-500 nats better than CVAE but for optimised variance, GP-CDE is, disappointedly, way worse compared to CVAE? Could you please enlighten me on this: perhaps the capacity of the GP-CDE model is not great, or is that limited by the inference scheme? Would CVAE + early stopping work much better than GP-CDE overall then?  Clarity: The paper is very well-written and the experiments seem to be well thought out.  Perhaps the following works should be mentioned and discussed in details, as they are closely related to this submission:  Lawrence, Neil D., and Joaquin Quiñonero-Candela. "Local distance preservation in the GP-LVM through back constraints", ICML 2006 -- this was briefly mentioned, but the idea of using recognition models for GP latent variables was first introduced in this paper, albeit only for MAP.  Thang Bui and Richard Turner, "Stochastic variational inference for Gaussian process latent variable models using back constraints", NIPS workshop on Black Box Learning and Inference, 2015 --- this paper uses recognition models for GP latent variable models  Zhenwen Dai, Andreas Damianou, Javier González, Neil Lawrence, "Variational Auto-encoded Deep Gaussian Processes", ICLR 2016 -- using structured recognition models for deep GPs.  A. Damianou and N. Lawrence, "Semi-described and semi-supervised learning with Gaussian processes", UAI 2015 --- This paper is very related to the submission here, though the set up is slightly different: some inputs are observed and some inputs are unobserved/missing and treated as latent variables.  Depeweg S., Hernández-Lobato J. M., Doshi-Velez F. and Udluft S., "Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks", ICLR 2017 --- the set up in this paper is very similar to what proposed here, except that Bayesian neural networks were considered instead of GPs