Structured Kernel Interpolation with Sparse Grids
Abstract
Structured kernel interpolation (SKI) accelerates Gaussian process (GP) inference by interpolating the kernel covariance function using a dense grid of inducing points, whose corresponding kernel matrix is highly structured and thus amenable to fast linear algebra.
Unfortunately, SKI scales poorly in the dimension of the input points, since the dense grid size grows exponentially with the dimension.
To mitigate this issue, we propose the use of sparse grids within the SKI framework.
These grids enable accurate interpolation, but with a number of points growing more slowly with dimension.
We contribute a novel nearly linear time matrixvector multiplication algorithm for the sparse grid kernel matrix.
We also describe how sparse grids can be combined with an efficient interpolation scheme based on simplices.
With these modifications, we demonstrate that SKI can be scaled to higher dimensions while maintaining accuracy.
Introduction
Gaussian processes (GPs) are popular prior distributions over continuous functions for use in Bayesian inference [14].
Due to their simple mathematical structure, closed form expressions can be given for posterior inference [20].
Unfortunately, a well-established limitation of GPs is that they are difficult to scale to large datasets.
In particular, for both exact posterior inference and the exact log-likelihood computation for hyperparameter learning, one must invert a dense kernel covariance matrix K ∈ R n × n where n is the number of training points.
Naively, this operation requires O ( n 3 ) time and O ( n 2 ) memory.
Structured Kernel Interpolation.
Many techniques have been proposed to mitigate this scalability issue [23, 19,
Recently, structured kernel interpolation (SKI) has emerged as a promising approach [9].
In SKI, the kernel matrix is approximated via interpolation onto a dense rectilinear grid of m inducing points.
In particular, K is approximated as WK G W T , where K G ∈ R m × m is the kernel matrix on the inducing points and W ∈ R n × m is an interpolation weight matrix mapping training points to nearby grid points.
Typically, W is sparse, and K G is highly structured — e.g., for shift invariant kernels, K G is multi-level Toeplitz.
Thus, W , K G , and in turn the approximate kernel matrix WK G W T admit fast matrix-vector multiplication.
This allows fast approximate inference and log-likelihood computation via the use of iterative methods, e.g., the conjugate gradient algorithm.
SKI’s Curse of Dimensionality.
Unfortunately, SKI does not scale well to high-dimensional input data: the number of points in the dense grid, and hence the size of K G , grows exponentially in the dimension d .
Moreover, SKI typically employs local cubic interpolation, which leads to an interpolation weight matrix W with row sparsity that also scales exponentially in d .
This curse of 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
dimensionality is a well-known issue with the use of dense grid interpolation.
It has been studied extensively, e.g., in the context of high-dimensional interpolation and numerical integration [17, 5].
In the computational mathematics community, an important technique for interpolating functions in high dimensions is sparse grids [2].
Roughly a sparse grid is a union of rectilinear grids with different resolutions in each dimension.
In particular, it is a union of all 2 ℓ 1 × 2 ℓ 2 × . . .
× 2 ℓ d sized grids, where (cid:80) di =1 ℓ i ≤ ℓ , for some maximum total resolution ℓ .
This upper bound on the total resolution limits the number of points in each grid — while a grid can be dense in a few dimensions, no grid can be dense in all dimensions.
See Figure 1 for an illustration.
Sparse grids have interpolation accuracy comparable to dense grids under certain smoothness assumptions on the interpolated function [22], while using significantly fewer points.
Concretely, for any function f : R d → R with bounded mixed partial derivatives, a sparse grid containing O (2 ℓ ℓ d − 1 ) points can interpolate as accurately as a dense grid with O (2 ℓd ) points, where ℓ is the maximum grid resolution [21].
Combining Sparse Grids with SKI.
Our main contribution is to demonstrate that sparse grids can be used within the SKI framework to significantly improve scaling with dimension.
Doing so requires several algorithmic developments.
When the inducing point grid is sparse, the kernel matrix on the grid, K G , no longer has simple structure.
E.g., it is not Toeplitz when the kernel is shift invariant.
Thus, naive matrix-vector multiplications with K G require time that scales quadratically, rather than near-linearly in the grid size.
This would significantly limit the scope of performance improvement from using a sparse grid.
To handle this issue, we develop a near-linear 1 time matrixvector multiplication (MVM) algorithm for any sparse grid kernel matrix.
Our algorithm is recursive, and critically leverages the fact that sparse grids can be constructed from smaller dense grids and that they have tensor product structure across dimensions.
For an illustration of our algorithm’s complexity versus that of naive quadratic time MVMs, see Figure 1 (c).
l 2 =   l 2 =  1 l 1 =  l 2 =  1 l 1 = 1 l 1 =(a)
[Figure]
A s y m p t
o t
i c
o r d e r o f g r o w t h
[Figure]
A second key challenge is that, while sparse grids allow for a grid size that grows as a much more mild exponential function of the dimension d , the bottleneck for applying SKI on large datasets can come in computing MVMs with the interpolation weight matrix W ∈ R n × m .
For classic high-dimensional interpolation schemes, like cubic interpolation, each row of W has O (2 d ) non-zero entries, i.e., the kernel covariance for each training point is approximated by a weighted sum of the covariance at O (2 d ) grid points.
When the number of training points n is large, storing W in memory, and multiplying by it, can become prohibitively expensive.
To handle this issue, we take an approach similar to that of Kapoor et al.
[13] and employ simplicial basis functions for interpolation, whose support grows linearly with d .
Combined with our fast MVM algorithm for sparse grid kernel matrices, this interpolation scheme lets us scale SKI to higher dimensions.
In summary, we propose the use of sparse grids to improve the scalability of kernel interpolation for GP inference relative to the number of dimensions.
To this end, we develop an efficient nearly linear time matrix-vector multiplication algorithm for the sparse grid kernel matrix.
Furthermore, we also propose the use of simplicial interpolation to improve scalability of SKI for both dense and sparse grids.
We show empirically that these ideas allow SKI to scale to at least 10 dimensions and perform competitively with state-of-the art GP regression methods.
We provide an efficient GPU implementation of the proposed algorithm compatible with GPyTorch [9], which is available at https://github.com/ymohit/skisg and licensed under the MIT license.
Background
Notation.
We let N and N 0 denote the natural numbers and N ∪ { 0 } respectively.
Matrices are represented by capital letters, and vectors by bold letters.
I denotes the identity matrix, with dimensions apparent from context.
For a matrix M , mvm( M ) denotes the number of operations required to multiply M by any admissible vector.
In GP regression, the training data are modeled as noisy measurements of a random function f drawn from a GP prior, denoted f ∼ N (0 , k ( · , · )) , where k : R d × R d → R is a covariance kernel.
For an input x i , the observed value is modeled as y i =
f
( x i )
+ ϵ i , with ϵ i ∼ N (0 , σ 2 ) .
Observed training pairs ( x i , y i ) are collected as X =
[ x 1 , . . .
, x n ] ∈ R n × d and y =
[ y 1 , . . .
, y n ] ∈ R n .
The kernel matrix (on training data) is K X
=
[ k ( x i , x j )] ni,j =1 ∈ R n × n .
The GP inference tasks are to compute the posterior distribution of f given ( X, y ) , which itself is a Gaussian process, and to compute the marginal log-likelihood log p ( y ) .
Naive approaches rely on the Cholesky decomposition of the matrix
K X =
K X
+ σ 2 I , which takes Θ( n 3 ) time; see Rasmussen [20] for more details.
To avoid the Θ( n 3 ) running time of naive inference, many modern methods use iterative algorithms such as the conjugate gradient (CG) algorithm to perform GP inference in a way that accesses the kernel matrix only through matrix vector multiplication (MVM), i.e., the mapping v (cid:55)→ K X v [9].
These methods support highly accurate approximate solutions to GP posterior inference task as well as hyper-parameter optimization.
The complexity of posterior inference and one step of hyper-parameter optimization is Θ( pn 2 ) for p CG iterations, as mvm( K X ) = n 2 .
In practice, p ≪ n suffices [27, 9].
2.1 SKI: Structured Kernel Interpolation
SKI further accelerates iterative GP inference by approximating the kernel matrix in a way that makes matrix-vector multiplications faster [27].
Given a set of inducing points U ⊂ R d , SKI approximates the kernel function as ˜ k
( x , x ′ ) ≜
w T x
K
U w x ′ , where K U ∈ R | U |×| U | is the kernel matrix for the set of inducing points U , and the vector w x ∈ R | U | contains interpolation weights to interpolate from U to any x .
The SKI approximate kernel matrix is ˜ K X
=
WK U W T , where W ∈ R n ×| U | is the matrix with i th row equal to w x i .
To accelerate matrix-vector multiplications with the approximate kernel matrix, SKI places inducing points on a regular grid and uses grid-based interpolation.
This leads to a sparse interpolation weight matrix W —for example, with cubic interpolation there are O (4 d ) entries per row, so that mvm( W )
=
O ( n 4 d ) —and to a kernel matrix K U that is multi-level Toeplitz (if k is stationary) [27], so that mvm( K U ) =
O ( | U | log | U | ) .
Overall, mvm( ˜ K X ) =
O ( n 4 d + | U | log | U | ) , which is much faster than n 2 for small d .
However, SKI quickly becomes infeasible in higher dimensions due to the 4 d entries per row of W and curse of dimensionality for the number of points in a high-dimensional grid: specifically, | U | =
m d for a grid with m points in each dimension.
2.2 Sparse Grids
Rectilinear grids.
We first give a formal construction of rectilinear grids, which will later be the foundation for sparse grids [7].
For a resolution index l ∈ N 0 , define the 1 -d grid Ω l as the centers of 2 l equal partitions of the interval [0 , 1] , which gives Ω l := { i/ 2 l +1 | 1 ≤ i ≤ 2 l [-+ 
-]{++
+}
and i is odd } .
The fact that the position index i must be odd implies that grids for any two different resolutions are disjoint .
Moreover, resolution-position index pairs ( l, i ) uniquely specify grid points in the union (cid:83) l ∈ N 0 Ω l of 1 -d rectilinear grids.
To extend rectilinear grids to d dimensions, let l ∈ N d 0 denote a resolution vector.
The corresponding rectilinear grid is given by Ω l := ⊗ dj
=1 Ω l j , where ⊗ denotes the Cartesian product.
A grid point in Ω l is indexed by the pair ( l , i ) of a resolution vector l and position vector i , where ( l j , i j ) gives the position in the 1 -d grid Ω j for dimension j .
This construction of rectilinear grids yields three essential properties that will facilitate formalizing sparse grids: (1) a grid Ω l is uniquely determined by its resolution vector l , (2) grids Ω l and Ω l ′ with different resolution vectors are disjoint, (3) the size | Ω l | = 2 ∥ l ∥ 1 of a grid is determined by the L 1 norm of its resolution vector.
Construction of Sparse Grids.
Sparse grids use rectilinear grids as their fundamental building block [22] and exploit the fact that resolution vectors uniquely identify different rectilinear grids.
Larger grids are formed as the union of rectilinear grids with different resolution vectors.
Sparse grids use all rectilinear grids with resolution vector having L 1 norm below a specified threshold.
Formally, for a resolution index ℓ ∈ N 0 , the sparse grid G ℓ,d in d dimensions is G ℓ,d :
= (cid:83) l :
∥ l ∥ 1 ≤ ℓ Ω l .
Figureillustrates the construction of the sparse grid G 2 , 2 from smaller 2-d rectilinear grids with maximum resolution 2 , i.e., { Ω l | ∥ l ∥ 1 ≤ 2 } .
The figure also illustrates another important fact: the sparse grid G ℓ,d has a total of Θ(2 ℓ ) distinct and equally spaced coordinates in each dimension, but many fewer total points than a dense d -fold Cartesian product of such 1-d grids, which would have Θ(2 ℓd ) points.
Sparse grids have a number of formal properties that are useful in algorithms and applications [22, 7].
Proposition 1 below summarizes the most relevant ones for our work.
For completeness, a proof appears in appendix A. For more details, see Valentin [25].
Proposition 1 ( Properties of Sparse Grid ) .
Let G ℓ,d ⊂ [0 , 1] d be a sparse grid with any resolution ℓ ∈ N 0 and dimension d ∈ N .
Then the following properties hold:
[Equation]
Property P1 shows that the size of sparse grid with Θ(2 ℓ ) points in each dimension grows more slowly than a dense grid with the same number of points in each dimension, since O (2 ℓ ℓ d − 1 ) ≪
O (2 ℓd ) .
Properties P2 and P3 are consequences of the structure of the set {
∥ l ∥ 1 ≤ ℓ } and the sparse grid construction.
Property P2 says that a sparse grid with smaller resolution is contained in one with higher resolution.
Property P3 is a crucial property, and says that a d -dimensional sparse grid can be constructed via Cartesian products of 1-dimensional dense grids with sparse grids in d − 1 dimensions.
Structured Kernel Interpolation on Sparse Grids
To scale kernel interpolation to higher dimensions, we propose to select inducing points U =
G ℓ,d on a sparse grid and approximate the kernel matrix as WK G ℓ,d W T for a suitable interpolation matrix W adapted to sparse grids.
This will require fast matrix-vector multiplications with the sparse grid kernel matrix K G ℓ,d and the interpolation matrix W .
We show how to accomplish these two tasks in Sections 3.1 and 3.2 for the important case of stationary product kernels [10].
3.1 Fast Multiplication with the Sparse Grid Kernel Matrix
Algorithm 1 is an algorithm to compute K G ℓ,d v for any vector v .
The algorithm uses the following definitions.
For any finite set U , let K U = (cid:2) k
( x , x ′ )
(cid:3) x , x ′ ∈ U .
The rows and columns of K U are “ U -indexed”, meaning the entries correspond to elements of U under some arbitrary fixed ordering.
For U ⊆ V , we introduce a selection matrix S U,V to map between U -indexed and V -indexed vectors.
It has entries ( S U,V ) ij equal to one if the i th element of U is equal to the j th element of V , and zero otherwise.
Also, S V,U := S TU,V .
For a V -indexed vector z V , the multiplication S U,V z V produces a U -indexed vector by selecting entries corresponding to elements in U , and for a U -indexed vector
z U , the multiplication S V,
U z U produces a V -indexed vector by inserting zeros for elements not in U .
Input: v ∈ R [Equation] and
[Equation]
d | Output: u = mvm (cid:0) K G ℓ,d , v (cid:1) , where, mvm ( K, v ) denotes K v obtained using this algorithm. 1: Let V i be the result of reshaping S Ω i ⊗G ℓ − i,d − 1 , G ℓ,d v into a | Ω i | × |G ℓ − i,d − 1 | matrix; this contains entries of v corresponding to the i th grid in the decomposition of P3.
2: if d = 1 then 3: return u = K G ℓ, 1 v ▷ Base case 4: end if 5: ▷ Pre-computation 6: for i = 0 to ℓ do 7: A i =
K G i, 1 S G i, 1 , Ω
i V i 8: B Ti = mvm (cid:0) K G ℓ − i,d − 1 , V Ti (cid:1) ▷ Recursively multiply K G ℓ − i,d − 1 with columns of V Ti . 9: end for 10: ▷ Main loop 11: for i = 0 to ℓ do 12:
A
[Equation]
(cid:32)(cid:80)
[Equation]
(cid:33) T  
▷ Recurse like line 8. B i = S Ω i , G i, 1 K G i,(cid:32)(cid:80)
j ≤ i S
G i, 1 , Ω j B j S G ℓ − j,d − 1 , G ℓ − i,d −(cid:33) 14: u i = vec
[ A i ] + vec [ B i ] 15: end for
Theorem 1.
Let K G
ℓ,d
be the kernel matrix for a d -dimensional sparse grid with resolution ℓ for a stationary product kernel.
For any v ∈ R |G ℓ,d | , Algorithm 1 computes K G ℓ,d v in O ( ℓ d 2 ℓ ) time.
The formal analysis and proof of Theorem 1 appears in Appendix B .
The running time of O ( ℓ d 2 ℓ ) is nearly linear in |G ℓ,d | and much faster asymptotically than the naive MVM algorithm that materializes the full matrix and has running time quadratic in [Equation]
Algorithm 1 is built on two high-level observations.
First, the decomposition of Property P3 from Proposition 1 and the fact that the kernel follows product structure across dimensions are used to decompose the MVM into blocks, each of which is between sub-grids which are the product of a 1-dimensional rectilinear grid and a sparse grid in d − 1 dimensions.
Therefore, the overall MVM computation can be recursively decomposed into MVMs with sparse grid kernel matrices in d −dimensions.
This observation is in part inspired by Zeiser [28], which also decomposes computation with the matrix on sparse grids by the resolution of first dimension.
The base case occurs when d = 1 .
We assume that Toeplitz structure, which arises due to the kernel being stationary, is leveraged to perform this base case MVM in O ( ℓ 2 ℓ ) time.
Algorithm 1 can also be extended to non-stationary product kernels by using a standard MVM routine for the base case, which changes the overall running-time analysis but is still more efficient than the naive algorithm.
Secondly, by Property P2, the kernel matrix multiplication for any grid of resolution ℓ also includes the result of the multiplication for grids of lower resolution and the same number of dimensions.
Thus, the results of the multiplications for many individual blocks can be obtained by using the appropriate selection operators with the result of the multiplication with the kernel matrix K G ℓ − i,d − 1 in Line 12 of the algorithm.
Further intuition and explanation are provided in the Appendix B.1.
Improving batching efficiency.
The recursions in Lines 8 and 12 can be batched for efficiency, since both are multiplications with the same symmetric kernel matrix K G ℓ,d .
Similarly, the recursion spawns many recursive multiplications with kernel matrices of the form G ℓ ′ ,d ′ for 0 ≤ ℓ ′ < ℓ and≤ d ′ < d , and the calculation can be reorganized to batch all multiplications with each K G ℓ ′ ,d ′ .
This is a significant savings, because there are only d ( ℓ +1) distinct kernel matrices, but the recursion has a branching factor of ( ℓ + 1) , so spawns many recursive calls with the same kernel matrices.
3.2 Sparse Interpolation For Sparse Grids
We now seek to construct the matrix W , which interpolates function values from the sparse grid G ℓ,d to training points x i ∈ R d , while ensuring that each row of W is sparse enough to preserve efficiency of matrix-vector multiplications with W .
This requires a sparse interpolation rule for sparse grids.
To set up the problem, we consider interpolating a function f observed at points in a generic set U .
Let f U =
( f ( x )) u ∈ U .
A linear interpolation rule for U is a mapping x (cid:55)→ w x ∈ R | U | used to approximate f ( x ) at an arbitrary point as f ( x ) ≈ w T x f U .
The density of an interpolation rule is the maximum number of non-zeros in w x for any x .
The combination technique for sparse grids constructs an interpolation rule by combining interpolation rules for the constituent rectilinear grids.
Proposition 2.
For each l , let w lx be an interpolation rule for the rectilinear grid Ω l with maximum density c .
The combination technique gives an interpolation rule w x for the sparse grid G ℓ,d with density at most c × (cid:0) ℓ + d − 1 d − 1 (cid:1) .
The factor (cid:0) ℓ + d − 1 d − 1 (cid:1) is the number of rectilinear grids included in G ℓ,d .
We use the combination technique to construct the sparse-grid interpolation coefficients w x i for each training point x i and stack them in the rows of W , which can be done in time proportional to the number of nonzeros.
Details are given in Appendix A.
The combination technique can use any base interpolation rule for the rectilinear grids, such as multilinear, cubic, or simplicial interpolation.
3.3 Simplicial Interpolation
The density of the interpolation rule is a critical consideration for kernel interpolation techniques— with or without sparse grids.
Linear and cubic interpolation in d dimensions have density Θ(2 d ) and Θ(4 d ) , respectively.
This “second curse of dimensionality” makes computations with W intractable in higher dimensions independently of operations with the grid kernel matrix.
For sparse grids, the density of W increases by an additional factor of (cid:0) ℓ + d − 1 d − 1 (cid:1) .
We propose to use simplicial interpolation [11] for the underlying interpolation rule to avoid exponential growth of the density.
Simplicial interpolation refers to a scheme where R d is partitioned into simplices and a point x is interpolated using only the d +1 extreme points of the enclosing simplex, so the density of the interpolation rule is exactly d + 1 .
Simplicial interpolation was previously proposed for sparse grid classifiers in [8].
In work closely related to ours, Kapoor et al.
[13] used simplicial interpolation for GP kernel interpolation, with the key difference that they use the permutohedral lattice as the underlying grid, which has a number of nice properties but does not come equipped with fast specialized routines for kernel matrix multiplication.
In contrast to Kapoor et al.
[13], we maintain rectilinear and/or sparse underlying grids, which preserve structure that enables fast kernel matrix multiplication.
For rectilinear grids, this requires partitioning each hyper-rectangle into simplices, so the entire space is partitioned by simplices whose extreme points belong to the rectilinear grid.
Then, within each simplex, the values at the extreme points are interpolated linearly.
In general, there are different ways to partition hyper-rectangles into simplices—we use the specific scheme detailed in [11].
For sparse grids, we then use the combination technique, leading to overall density of ( d + 1) (cid:0) ℓ + d − 1 d − 1 (cid:1) .
Table 1 provides the MVM complexities of
W and kernel matrices for different interpolation schemes and both grids.
More details on how to perform simplicial interpolation with rectilinear and sparse grids are given in the Appendix B.2. [Table]
Experiments
In this section
[Table] we empirically evaluate the time and memory taken by Algorithm 1 for matrix-vector multiplication with the sparse grid kernel matrix, the accuracy of sparse grid interpolation and GP regression as the data dimension d increases [Table] and the accuracy of sparse grid kernel interpolation for GP regression on real higher-dimensional datasets from UCI.
Hyper-parameters, data processing steps, and optimization details are given in Appendix C.1.
[Figure]
M V
M t i m
[Figure]
i n
[Figure]
Sparse grid kernel MVM complexity.
First, we evaluate the efficiency of MVM algorithms.
We compare the basic and efficient implementations of Algorithm 1 to the naive algorithm, which constructs the full kernel matrix and scales quadratically with the sparse grid size.
Algorithmhas a significant theoretical advantage in terms of both time and memory requirements as the grid size grows.
Figure 2 illustrates this for d = 6 by comparing MVM time and memory requirements for sparse grids with resolutions ℓ ∈ { 2 , . .
, 9 } (roughly 100 to 1M grid points).
The MVM time, preprocessing time, and memory consumption of Algorithm 1 all grow more slowly than the naive algorithm, and the efficient implementation of Algorithm 1 is faster for |G| larger than about 10 4 , after which the naive algorithm also exceeds the 10 GB memory limit.
For comparison, at ℓ = 6 (about 40K grid points), Algorithm 1 uses only 0 . 05 GB of memory.
These results indicate that the proposed algorithm is crucial for enabling sparse grid kernel interpolation in higher dimensions.
Figure 2 also depicts the typical time to run GP inference (i.e., preprocessing time plus [-
-]50 MVM operations).
Sparse grid interpolation and GP inference accuracy on synthetic data.
A significant advantage of sparse grids over dense rectilinear grids is their ability to perform accurate interpolation in higher dimensions.
We demonstrate this for d = 6 by interpolating the function f ( x ) = cos( ∥ x ∥ 1 ) from observation locations on sparse grids of increasing resolution onto 200 random points sampled uniformly from [0 , 1] d .
Figure 3, left, shows the interpolation error for dense and sparse grids with both cubic and simplicial interpolation.
Sparse (cubic) is significantly more accurate than Dense (cubic) , and Sparse (simplicial) is more accurate than Dense (simplicial) .
We next evaluate the accuracy of GP inference in increasing dimensions.
We keep the same function f ( x ) and generate observations as y = f ( x ) + N (0 , 0 . 05) .
For all d , we use ℓ = 4 for the sparse grid and compare to the dense grid with the closest possible total number of grid points (i.e., ⌈G 1 /d 4 ,d ⌉ points in each dimension) 3 .
For d ≥ 8 , performance is better with sparse grids than dense grids for both interpolation schemes.
Remarkably, our proposal to use simplicial interpolation with dense grids allows SKI to scale to d = 10 , which is a significant improvement over prior work, in which SKI is typically infeasible for d ≥ 4 . [Figure] I n t e r p o l a t i o n E rr o r
[Figure]
GP regression performance on UCI datasets.
To evaluate the effectiveness of our proposed methods for scaling GP kernel interpolation to higher dimensions, we consider all UCI [4] data sets with dimension 8 ≤ d ≤ 10 .
We compare our proposed methods, Dense-grid (dense SKI with simplicial interpolation) and Sparse-grid (sparse-grid SKI with simplicial interpolation) to SGPR [24], SKIP [10], and Simplex-GP [13].
For SGPR, we report the best results using 256 or 512 inducing points.
For SKIP, 100 points per dimension are used.
For Simplex-GP [13], the blur stencil order is set to 1 .
Table 3 shows the root mean squared error (RMSE) for all methods.
Our methods have performance comparable to and often better than SGPR and prior methods for scaling kernel interpolation to higher dimensions.
This shows that sparse grids and simplicial interpolation can effectively scale SKI to higher dimensions to give a GP regression framework that is competitive with state-of-the-art approaches.
We report additional results and analysis in Appendix C. [Table] [Table]
Related Works
Beyond SKI and its variants [Table] a number scalable GP approximations have been investigated.
Most notable are the different variants of sparse GP approximations [26, 23, 18].
For m inducing points, these methods require either Ω( nm 2 ) time for direct solves or Ω( nm ) time for approximate kernel MVMs in iterative solvers.
While these methods generally do not leverage structured matrix algebra like the SKI framework, and thus have worse scaling in terms of the number of inducing points m , they may achieve comparable accuracy with smaller m , especially in higher dimensions.
Some work utilizes the SKI framework to further boost the performance of sparse GP approximations [12].
A closely related work to ours is on improving the dimension scaling of SKI through the use of low-rank approximation and product structure [10].
In another very closely related work, Kapoor et al.
[13] recently proposed to scale SKI to higher dimensions via interpolation on a permutohedral lattice.
Like our work, they use simplicial interpolation.
Unlike our work, the kernel matrix on the permutohedral lattice does not have special structure that admits fast exact multiplications; they instead use a locality-based approximation that takes into account the length scale of the kernel function by only considering pairs of grid points within a certain distance.
Another related direction of research is the adaptation of sparse grid techniques for machine learning problems, e.g., classification and regression [16] and data mining [1].
These methods construct feature representations using sparse grid points [3], often by selecting a subset of grid points relevant for the dataset [6, 3].
Discussion
This work demonstrates that two classic numerical techniques, namely, sparse grids and simplicial interpolation, can be used to scale GP kernel interpolation to higher dimensions.
SKI with sparse grids and simplicial interpolation has better or competitive regression accuracy compared to state-of-the-art GP regression approaches on several UCI benchmarking datasets with 8 to 10 dimensions.
Limitations and future work.
Sparse grids and simplicial interpolation address two important bottlenecks when scaling kernel interpolation to higher dimensions.
Sparse grids allow scalable matrix-vector multiplications with the grid kernel matrix, and simplicial interpolation allows scalable multiplications with the interpolation matrix W .
The relatively large number of rectilinear grids used to form a sparse grid—i.e., the factor of (cid:0) ℓ + d − 1 d − 1 (cid:1) in Proposition 2—is one limiting factor that makes multiplication by W more costly.
Future research could investigate methods to mitigate this extra cost, and explore the limits of scaling to even higher dimensions with sparse grids.
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant Nos.
1661259, 1749854, 2046235 and 1763618, along with Adobe and Google Research Grants.
We thank
Javier Burroni, Abhinav Agarwal, Shiv Shankar, Shreyas Chaudhari for their valuable feedback.