Paper:
MCMC for Variationally Sparse Gaussian Processes
Gaussian process (GP) models form a core part of probabilistic machine learning. Considerable research effort has been made into attacking three issues with GP models: how to compute efficiently when the number of data is large; how to approximate the posterior when the likelihood is not Gaussian and how to estimate covariance function parameter posteriors. This paper simultaneously addresses these, using a variational approximation to the posterior which is sparse in support of the function but otherwise free-form. The result is a Hybrid Monte-Carlo sampling scheme which allows for a non-Gaussian approximation over the function values and covariance parameters simultaneously, with efficient computations based on inducing-point sparse GPs. Code to replicate each experiment in this paper is available at github.com/sparseMCMC.

Review:
0. SummaryThis paper presents a method for sampling from the (approximate) posterior in variational sparse Gaussian process models with factorizing likelihoods. The main claimed contribution is that allowing a "free-form"for the approximate posterior, instead of constraining it to belong to a pre-specifiedfamily of distributions. 1. Clarity This is paper is well written with the contributions clearly stated, describing also how the the paper differentiates from previous work. 2. Quality, originality and significance The paper addresses a significant problem in probabilistic inference, that of approximating posterior distributions in models with Gaussian process priors and factorizing likelihoods, allowing efficient sampling of inducing variables and covariance hyper-parameters. The paper is technically sound, buildingupon previous work on variational approaches to sparse GP models (reference 5 in the paper) with the main contribution appearing in Equations (5) and (6),rewriting the variational objective so that one can use MCMC (HMC) machinery to to draw samples from the joint approximate posterior over inducing variables and hyper-parameters. I note in passing that Equation (5) seems to have typo, as the expectation of the conditional likelihood over the conditional prior should be exponentiated (?).The very same idea of sampling from the approximate posterior within the inducing-point framework has been proposed in the past (see Equation 1.27 and corresponding text in [1] below) but I am unaware of its implementation. I am surprised the authors do not reference this work. Nevertheless, the contribution of the paper is still significant as it carries out the development of a practical framework that does not involve the commonly used assumption of optimizing over the parameters of a fixed distribution, while still maintaining the computational efficiency of sparse GP models. I do not see major concerns about this paper, although it somehow gives the readers the impression that the proposed method is sampling from the true posterior of a sparse GP model, which I do not think is the case giving the assumed form for the posterior approximation in the denominator of Equation (3). It would be useful if the authors discuss this.The paper presents results on a wide variety of experiments on small datasets and on MNIST, comparing to the full Gaussian approximation and to 'gold standard' MCMC. It is interesting to see that there is not a large practical difference between a free-form approximation and a Gaussian approximation but I think the proposed approach it is still useful on its own right. On the MNIST data, it would be useful to know how long it took to run the whole experiment.  3. Minor comments - Figures 1 and 4 show some weird numbers '000, 001, ...' - line 370: "As Girolami and S' -> 'As Girolami and Rogers'- line 401: "approaches to to classify" -> "approaches to classify"4. References[1] . M. K. Titsias, M. Rattray and N.D. Lawrence. Markov chain Monte Carlo algorithms for Gaussian processes. Chapter to appear in the book "Inference and Learning in Dynamic Models"