Paper:
Scalable Inference for Gaussian Process Models with Black-Box Likelihoods
We propose a sparse method for scalable automated variational inference (AVI) in a large class of models with Gaussian process (GP) priors, multiple latent functions, multiple outputs and non-linear likelihoods. Our approach maintains the statistical efficiency property of the original AVI method, requiring only expectations over univariate Gaussian distributions to approximate the posterior with a mixture of Gaussians. Experiments on small datasets for various problems including regression, classification, Log Gaussian Cox processes, and warped GPs show that our method can perform as well as the full method under high sparsity levels. On larger experiments using the MNIST and the SARCOS datasets we show that our method can provide superior performance to previously published scalable approaches that have been handcrafted to specific likelihood models.

Review:
This paper takes the pre-existing Gaussian Process inference method of "Automatic Variational Inference" and adapts it for scaling to larger datasets. Specifically, it draws on the method of inducing points to develop a new variational bound that factorises across the data, enabling the use of stochastic optimisation. It uses a mixture of Gaussians to generate approximate posteriors, without being tailored to a specific likelihood model.I think this article represents a very good contribution to the field: previous work on non-Gaussian likelihoods is generally either an expensive Monte Carlo method, likelihood-specific, or stuck with O(n^3) computational bound. This work represents a method to perform scalable and accurate inference for generic likelihood models: a previously unachieved target that this work makes good progress towards.The only strong objections I have to the presentation of the work is the representation of experimental results. There is no comparison to competing methods: it would be greatly of interest to compare against "Scalable variational Gaussian process classification." for binary classification, although for other likelihoods there are no clearly competing methods at this scale, so their absence is fair. There is also no timing data: without indication of how quickly the method takes to work, readers will not be able to make a judgement of the accuracy/time tradeoff involved. Ideally, there would be a figure plotting average test error against training time, so the model's learning behaviour can be elucidated. This should be achievable, however, by slight modification to existing code.Some comments working through the paper:The introduction is good: the preceding work on Automatic Variational Inference is presented well, as is the method of inducing variables.Introduction of uncoupled latent Gaussian processes is presented sensibly, along with iid likelihood properties.The scalability modifications to AVI, drawing on inducing point framework is presented well. Calculation of the of each term is presented well and very thoroughly, to a degree might make as much sense in a supplement.The introduction of stochastic optimisation is interesting, and could possibly be made more of: unbiased subsampling of data is a hugely useful property for large datasets, and competitors to existing SVI methods would be a good contribution to the literature.The experimental section is impressive, containing some results that are unprecedented in the literature. As mentioned above, it would greatly benefit from the presentation of timing results, and with comparison against existing methods for scalable Gaussian Process Binary Classification.A separate conclusion would be valuable to reiterate the main contributions of the work, and the related work section could possibly be moved to earlier in the article.The language is consistently clear, and the references are well-presented. The supplementary material is accurate and complete.----- after author rebuttal -----After the authors' rebuttal, I would be happy to see this paper accepted, if the points raised are fulfilled, i.e. comparison to SVIGPC, timing data and a full conclusion.