Paper:
Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
Deep Gaussian Processes (DGPs) are hierarchical generalizations of Gaussian Processes that combine well calibrated uncertainty estimates with the high flexibility of multilayer models. One of the biggest challenges with these models is that exact inference is intractable. The current state-of-the-art inference method, Variational Inference (VI), employs a Gaussian approximation to the posterior distribution. This can be a potentially poor unimodal approximation of the generally multimodal posterior. In this work, we provide evidence for the non-Gaussian nature of the posterior and we apply the Stochastic Gradient Hamiltonian Monte Carlo method to generate samples. To efficiently optimize the hyperparameters, we introduce the Moving Window MCEM algorithm. This results in significantly better predictions at a lower computational cost than its VI counterpart. Thus our method establishes a new state-of-the-art for inference in DGPs.

Review:
The paper addresses the problem of doing inference in deep Gaussian processes (DGPs). The authors propose to use Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) as a way to relax the assumption of Gaussianity for the inducing point posterior as assumed in the standard variational inference setting. To motivate this, the authors argue based on empirical evidence that the posterior distributions of interest are not always Gaussian nor unimodal.  In order optimize the hyperparameters during training, they propose a new variant of the Markov Chain Expectation Maximization (MCEM) algorithm called Moving Window MCEM, which is claimed to perform better results than the original MCEM algorithm.  Finally, the authors demonstrate the performance ofthe proposed method on a set of benchmark datasets (UCI datasets, MNIST classification, and Harvard Clean Energy). The experiments show that the proposed method outperforms the benchmark methods.   The technical details of the submission appear sound and correct. While none of the claims in the paper is supported by any theoretical analysis, all claims are to some degree supported by empirical evidence. However, the claim that Moving Window MCEM performs better than MCEM is only supported by a plot of the training log likelihood vs run time for a single data set. Similarly, the claim that SGHMC converges faster than the reference method (DSVI) is also only supported by a plot of the test log likelihood vs run time for a single data set. However, the experiments based on the UCI datasets do show that the proposed method achieved better predictive performance than the benchmark methods, especially for the larger datasets.   The paper is in general clear, well-organized, and well-written. However, the proposed method has many moving parts (SGHMC, the autotuning method for the SGHMC parameters and the Moving Window MCEM) and it’s not entirely clear how each of these is executed relative to each other. Therefore, it would be an improvement to include pseudo-code for the entire method in the supplementary material and not just for the Moving Window EM algorithm.  Furthermore, there are a few statements that could be made more precise, see comments below.   The combination of SGHMC and deep Gaussian processes is novel. Furthermore, the proposed variant of the MCEM is also original.   Constructing deep models that are capable of quantifying uncertainty in a reasonable way is an important topic. Therefore, it is most likely that other researchers will use the proposed ideas and improve them. Additionally, the proposed method achieves state of the art performance in terms of predictive accuracy for a range of benchmark datasets and therefore practitioners will most likely also be interested in the method.   Further comments: Line 61: “One might expect a sampling method to be more computationally intensive than an approximate method such as DSVI.” Sampling-based methods are also approximate  Line 62: “However, in DGPs, sampling from the posterior is inexpensive, since it does not require the recomputation of the inverse covariance matrix, which only depends on the Hyperparameters.” The authors should clarify if the is always true or if it’s only true for the “sampling phase”, where the hyperparameters are kept fixed  Line 82: “The probability of y...” This is not a meaningful quantity - the authors probably mean the conditional distribution of y   Line 88: “The GP distribution f given the inducing” typo  Line 126: “First, we illustrate with a toy problem that the posterior distribution in DGPs can be multimodal.” The authors state that there are two modes for the toy problem, but they should clarify of these modes are related to each other in any way. Furthermore, the plots in figure 1 only show f1 vs x and y vs f1, but it would be more interesting to see what y vs x looks like for the two modes.   Line 140: "We examine the posterior samples generated by SGHMC for each  inducing output" What measures have you taken to make sure that SGHMC have  actually converged to the target distributions? If you cannot guarantee that the samples from SGHMC are representative of the true target distributions, then these tests lose their value.  Line 194: Missing a log in the M-step?   Line 212: “We plotted the predictive log-likelihood on the training set against the runtime of the algorithm to demonstrate the superior  performance of Moving Window MCEM over MCEM (the test log-likelihood shows the same trend)” In that case, it would be more convincing to show the test log likelihood  After the rebuttal ------------------------------ I have read the rebuttal, but it does not change my opinion of the paper as the authors did properly address all of my concerns. For example, it would have been nice if the authors would have commented on the nature of the multimodality in terms of symmetries and what measures they have taken to guarantee that SGHMC produces samples from the true posterior distribution for their statistical significance tests.