Paper:
GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
Despite advances in scalable models, the inference tools used for Gaussian processes (GPs) have yet to fully capitalize on developments in computing hardware. We present an efficient and general approach to GP inference based on Blackbox Matrix-Matrix multiplication (BBMM). BBMM inference uses a modified batched version of the conjugate gradients algorithm to derive all terms for training and inference in a single call. BBMM reduces the asymptotic complexity of exact GP inference from O(n 3 ) to O(n 2 ). Adapting this algorithm to scalable approximations and complex GP models simply requires a routine for efficient matrix-matrix multiplication with the kernel and its derivative. In addition, BBMM uses a specialized preconditioner to substantially speed up convergence. In experiments we show that BBMM effectively uses GPU hardware to dramatically accelerate both exact GP inference and scalable approximations. Additionally, we provide GPyTorch, a software platform for scalable GP inference via BBMM, built on PyTorch.

Review:
Summary ========================  The authors propose a generic inference algorithm for GP regression that attempts to exploit the advantages of modern GPU based architectures to perform fast matrix multiplications. They proposed a modification of PCG (preconditioned conjugate gradient descent) to solve “Ab = c” that computes simultaneously the solution for multiple r.h.s vectors it computes, at the same time, the tri-diagonalization of the matrix w.r.t. the r.h.s vectors. Based on these outputs, the author propose efficient estimates of the three main quantities needed for computing the marginal log-likelihood: the projection of the output using the inverse of the kernel matrix , the log-determinant of the kernel matrix and the trace of the kernel matrix times its derivative. In the experimental section they show a significant speed-up with respect to state-of-the-art inference methods for scalable GP regression and improvements in terms of accuracy due to the better stability of the proposal.   Details =========================  The paper is focused on scalable inference method for GP regression (although the authors claim the method can be easily extended to other scenarios, e.g. classification). The main idea is to take advantage of modern GPUs and its high performance to perform fast matrix multiplications. To do so, they cleverly modify PCG to solve several r.h.s. vectors simultaneously where the matrix is the kernel matrix of the training set. More specifically, they solve for the output variable "y" and a set of random i.i.d vectors sampled from a standard multivariate gaussian. The random vectors are used to approximate the log determinant term of the kernel matrix in the log-marginal likelihood. In addition, the method outputs the tri-diagonalizations of the kernel matrix w.r.t. the random vectors, that are used to compute the trace term in the log-marginal likelihood. In addition, they propose preconditioning to reduce the number of sequential steps.  One of the selling points of the algorithm is that it is black-box: the user only have to define a matrix multiplication operator for the kernel and its derivative. However, the paper focused on regression. It is true that it can be easily extended to other problems like classification, but in that case the algorithm needs specific inference algorithms for the problem at hand and it could be argued that it is not that black-box anymore.   In the experimental section the authors perform a wide set of experiments where they show how their method outperforms the baselines in the majority of the scenarios in terms of speed, and in some cases, in terms of accuracy. For the improvement in terms of accuracy the authors hypothesis is that it is related to the better stability of their method, although further research is encouraged. Overall, it is a strong well-written paper with nice theoretical and practical contributions.  Minor: Should not be K instead of T at the beginning of eq. (6)?