Paper:
Orthogonally Decoupled Variational Gaussian Processes
Gaussian processes (GPs) provide a powerful non-parametric framework for reasoning over functions. Despite appealing theory, its superlinear computational and memory complexities have presented a long-standing challenge. State-of-the-art sparse variational inference methods trade modeling accuracy against complexity. However, the complexities of these methods still scale superlinearly in the number of basis functions, implying that that sparse GP methods are able to learn from large datasets only when a small model is used. Recently, a decoupled approach was proposed that removes the unnecessary coupling between the complexities of modeling the mean and the covariance functions of a GP. It achieves a linear complexity in the number of mean parameters, so an expressive posterior mean function can be modeled. While promising, this approach suffers from optimization difficulties due to ill-conditioning and non-convexity. In this work, we propose an alternative decoupled parametrization. It adopts an orthogonal basis in the mean function to model the residues that cannot be learned by the standard coupled approach. Therefore, our method extends, rather than replaces, the coupled approach to achieve strictly better performance. This construction admits a straightforward natural gradient update rule, so the structure of the information manifold that is lost during decoupling can be leveraged to speed up learning. Empirically, our algorithm demonstrates significantly faster convergence in multiple experiments.

Review:
Edit 2: While reviewing the paper and discussing with other reviewers, I reframed the approximation of Cheng and Boots as a structured variational approximation in the Titsias's framework like this: https://imgur.com/a/E6uUKXh, and similarly for the method proposed in this paper: https://imgur.com/a/tbx1fMs. While these might seem trivial in hindsight and that they didn't bring much intuition, I think these could be useful in making connections between methods and previously published results. I hope the authors find these useful.  Edit after the rebuttal period 1: The author response has sufficiently addressed my concerns, though I would still want to understand how the approximation was discovered/derived. I also think that it could be good, as a future research direction, to relate the approximation of Cheng and Boots and the approximation that this paper is using to the Titsias' framework without the need of RKHS, and the natural gradient version of Hensman et al.   Summary: This paper introduces an orthogonally decoupled sparse variational approximation for Gaussian process regression and classification models, extending the decoupled framework of Cheng and Boots (2017). The idea is that a combination of a particular partitioning of the inducing points and a specific parameterisation of the mean and covariance functions lead to an efficient sparse posterior such that natural gradients can be obtained cheaply, enabling natural gradient methods to be used to optimise the variational lower bound. Experiments on multiple regression and classification datasets demonstrate the performance of the new decoupled approach and its version using natural gradients.  Comments:  1. the parameterisation of the mean and covariance in equations 9 is particularly interesting, as it allows useful cancellations to happen when moving to equations 10. 2. I'm not sure how this particular parameterisation pops up in the first place, could the thought process be elaborated -- I think this could be useful for deriving further approximations based on this decoupled framework? for example, the natural gradient version of Cheng and Boots as in appendix C2 was considered first, and then somehow you realised that inserting (I - \phi_\beta K_\beta^{-1} \phi_\beta) in front of \phi_gamma gives decoupled natural gradients. This does not look trivial. 3. just an observation: the hybrid approach seems to perform poorly compared to the natural gradient version of Orth and Coupled, but seems to be better than Orth. 4. I am particularly intrigued that the natural version of the coupled approach -- which I think is the natural gradient approach of Hensman et al -- performs *much better* than the decoupled approach of Cheng and Boots (2017). However, Cheng and Boots compared their decoupled approach to SVI [I thought this is the approach of Hensman et al], and the decoupled approach outperforms SVI in all experiments they considered. Have you looked at this contradictory results in detail? Or I'm missing something here? 5. While I think the experimental results, especially on the regression dataset, are pretty solid, I'm wondering what is the fairest comparison protocol here? Previous sparse GP works basically assumed the same number of pseudo points for all methods and looked at the predictive performance after a number of iterations [provided that all methods have the same computational complexity]. The decoupled approaches have two set of pseudo points, and the complexity is a little bit different compared to previous approaches. The deltas between methods in the experimental tables are also very small to concretely say one method is better than the other, given they all have slightly different complexities and hence running times.  6. Making some of the ranking results in table 2 bold is, in my opinion, a bit misleading, as the methods perform very similarly and the std error for the mean rank seems to be very large here. 7. The paper is generally well-written, but I found it's hard to understand what the approximation is intuitively doing. Perhaps some visualisation of the methods [coupled vs decoupled vs orthogonally decoupled] on some 1D, 2D tasks could be useful, i.e. what the pseudo-points are doing, what happens when you optimise them, will the methods get to exact GP when the number of pseudo points = number of training points, does the FITC pathology described in Bauer et al (2016) exist here, ...