Structured Kernel Interpolation with Sparse Grids
Abstract
Structured kernel interpolation (SKI) accelerates Gaussian process (GP)inference by interpolating the kernel covariance function using a densegrid of inducing points, whose corresponding kernel matrix is highlystructured and thus amenable to fast linear algebra.
Unfortunately, SKIscales poorly in the dimension of the input points, since the dense gridsize grows exponentially with the dimension.
To mitigate this issue, wepropose the use of sparse grids within the SKI framework.
These gridsenable accurate interpolation, but with a number of points growing moreslowly with dimension.
We contribute a novel nearly linear time matrixvector multiplication algorithm for the sparse grid kernel matrix.
We alsodescribe how sparse grids can be combined with an efficient interpolationscheme based on simplices.
With these modifications, we demonstrate that
SKI can be scaled to higher dimensions while maintaining accuracy.
Introduction
Gaussian processes (GPs) are popular prior distributions over continuous functions foruse in Bayesian inference [ 14 ].
Due to their simple mathematical structure, closed formexpressions can be given for posterior inference [ 20 ].
Unfortunately, a well-establishedlimitation of GPs is that they are difficult to scale to large datasets.
Exact posterior inferenceand log-likelihood computation for hyperparameter learning require the inversion of atypically dense kernel covariance matrix K ∈ R n × n , where n is the number of trainingpoints.
Naively, this operation requires O ( n 3 ) time and O ( n 2 ) memory.
Structured Kernel Interpolation.
Many techniques have been proposed to mitigate thisscalability issue [ 24 , 19 , 25 , 9 ].
Recently, structured kernel interpolation (SKI) has emergedas a promising approach [ 9 ].
In SKI, the kernel matrix is approximated via interpolationonto a dense rectilinear grid of m inducing points.
In particular, K is approximated as
WK G W T , where K G ∈ R m × m is the kernel matrix on the inducing points and W ∈ R n × m isan interpolation weight matrix mapping training points to nearby grid points.
Typically, Wis sparse, and K G is highly structured — e.g., for shift invariant kernels, K G is multi-level
Toeplitz.
Thus, W , K G , and in turn the approximate kernel matrix WK G W T admit fastmatrix-vector multiplication.
This allows fast approximate inference and log-likelihoodcomputation via the use of iterative methods, like the conjugate gradient and Lanczosalgorithms.
SKI’s Curse of Dimensionality.
Unfortunately, SKI does not scale well to high-dimensionalinput data: the number of points in the dense grid, and hence the size of K G , grows exponentially in the dimension d .
Moreover, SKI typically employs local cubic interpolation,which leads to an interpolation weight matrix W with row sparsity that also scales ex- ponentially in d .
This curse of dimensionality is a well-known issue with the use of densegrid interpolation.
It has been studied extensively, e.g., in the context of high-dimensionalpolynomial interpolation and numerical integration [ 17 , 5 ].
In the computational mathematics community, an important technique for interpolatingfunctions in high dimensions is sparse grids [ 2 ].
Roughly a sparse grid is a union ofrectilinear grids with different resolutions in each dimension.
In particular, it is a union ofall 2 ℓ 1 × 2 ℓ 2 × . . .
× 2 ℓ d sized grids, where ∑ di = 1 ℓ i ≤ ℓ , for some maximum total resolutionℓ .
This upper bound on the total resolution limits the number of points in each grid —while a grid can be dense in a few dimensions, no grid can be dense in all dimensions.
See
Figure 1 for an illustration.
It is well-known that sparse grids have interpolation accuracy comparable to dense gridsunder certain smoothness assumptions on the interpolated function [ 22 ], while usingsignificantly fewer points.
Concretely, for any function f : R d → R with bounded mixedpartial derivatives, a sparse grid containing O ( 2 ℓ ℓ d − 1 ) points can interpolate as accuratelyas a dense grid with O ( 2 ℓ d ) points, where ℓ is the maximum grid resolution [ 21 ].
Combining Sparse Grids with SKI.
Our main contribution is to demonstrate that sparsegrids can be used within the SKI framework to significantly improve scaling with dimension.
Doing so requires several algorithmic developments.
When the inducing point grid issparse, the kernel matrix on the grid, K G , no longer has simple structure.
E.g., it is not
Toeplitz when the kernel is shift invariant.
Thus, naive matrix-vector multiplications with
K G require time that scales quadratically, rather than near-linearly in the grid size.
Thiswould significantly limit the scope of performance improvement from using a sparse grid.
To handle this issue, we develop a near-linear time matrix-vector multiplication (MVM)algorithm for any sparse grid kernel matrix.
Our algorithm is recursive, and criticallyleverages the fact that sparse grids can be constructed from smaller dense grids and thatthey have tensor product structure across dimensions.
For an illustration of our algorithm’scomplexity versus that of naive quadratic time MVMs, see Figure 1 (c).
l 2 =   l 2 =  1 l 1 =  l 2 =  1 l 1 = 1 l 1 =(a)
[Figure] [Figure]
x i
[Figure]
For
) improves significantly
A second key challenge is that, while sparse grids allow for a grid size that grows as amuch more mild exponential function of the dimension d , the bottleneck for applying
SKI on large datasets can come in computing MVMs with the interpolation weight matrix
W ∈ R n × m .
For classic high-dimensional interpolation schemes, like cubic interpolation,each row of W has O ( 2 d ) non-zero entries, i.e., the kernel covariance for each trainingpoint is approximated by a weighted sum of the covariance at O ( 2 d ) grid points.
Whenthe number of training points n is large, storing W in memory, and multiplying by it, canbecome prohibitively expensive.
To handle this issue, we take an approach similar to thatof Kapoor et al.
[ 13 ] and employ simplicial basis functions for interpolation, whose supportgrows linearly with d .
Combined with our fast MVM algorithm for sparse grid kernelmatrices, this interpolation scheme lets us scale SKI to higher dimensions.
In summary, we propose the use of sparse grids to improve the scalibility of kernel interpolation for GP inference relative to the number of dimensions.
To this end, we developan efficient nearly linear time matrix-vector multiplication algorithm for the sparse gridkernel matrix.
Furthermore, we also propose the use of simplicial interpolation to improvescalability of SKI for both dense and sparse grids.
We show empirically that these ideasallow SKI to scale to at least 10 dimensions and perform competitively with state-of-theart GP regression methods.
We provide an efficient GPU implementation of the proposedalgorithm compatible with GPyTorch [ 9 ], which is licensed under the MIT license.
Background
Notation.
We let N and N 0 denote the natural numbers and N ∪ { 0 } respectively.
Matricesare represented by capital letters, and vectors by bold letters.
I denotes the identity matrix,with dimensions apparent from context.
For a matrix M , mvm ( M ) denotes the number ofoperations required to multiply M by any admissible vector.
In Gaussian process (GP) regression, the training data are modeled as noisy measurementsof a random function f drawn from a GP prior, denoted f ∼ N ( 0, k ( · , · )) , where k : 89 R d × R d → R is a covariance kernel.
For an input x i , the observed value is modeledas y i
=
f
( x i )
+ ϵ i , with ϵ i ∼ N ( 0, σ 2 ) .
Observed training pairs ( x i , y i ) are collected as
X =
[ x 1 , . . .
, x n ] ∈ R n × d and y =
[ y 1 , . . .
, y n ] ∈ R n .
The kernel matrix (on training data)is K X
=
[ k ( x i , x j )] ni , j = 1 ∈ R n × n .
The GP inference tasks are to compute the posteriordistribution of f given ( X , y ) , which is itself a Gaussian process, and to compute themarginal log-likelihood log p ( y ) .
Naive approaches rely on the Cholesky decomposition ofthe matrix K X
= K X
+ σ 2 I , which takes Θ ( n 3 ) time; see Rasmussen [ 20 ] for more details.
To avoid the Θ ( n 3 ) running time of naive inference, many modern methods use iterativealgorithms such as the conjugate gradient (CG) algorithm to perform GP inference in away that accesses the kernel matrix only through matrix vector multiplication (MVM),
i.e.,the mapping v (cid:55)→ K X v [ 9 ].
These methods support highly accurate approximate solutionsto GP posterior inference task as well as hyper-parameter optimization.
The complexityof posterior inference and one step of hyper-parameter optimization is Θ ( pn 2 ) for p CGiterations, as mvm ( K X ) = n 2 .
Typically, p ≪ n suffices in practice [ 28 , 9 ].
2 . 1 SKI: Structured Kernel Interpolation
SKI further accelerates iterative GP inference by approximating the kernel matrix in a waythat makes matrix-vector multiplications faster [ 28 ].
Given a set of inducing points U ⊂ R d ,
SKI approximates the kernel function as ˜ k
( x , x ′ ) ≜
w T x
K
U w x ′ , where K U ∈ R | U |×| U | isthe kernel matrix for the set of inducing points U , and the vector w x ∈ R | U | containsinterpolation weights to interpolate from U to any x .
The SKI approximate kernel matrixis ˜ K X
=
WK U W T , where W ∈ R n ×| U | is the matrix with i th row equal to w x i .
To accelerate matrix-vector multiplications with the approximate kernel matrix, SKI placesinducing points on a regular grid and uses grid-based interpolation.
This leads to a sparse interpolation weight matrix W —for example, with cubic interpolation there are
O ( 4 d ) entries per row, so that mvm ( W ) =
O ( n 4 d ) —and to a kernel matrix K U that ismulti-level Toeplitz (if k is stationary) [ 28 ], so that mvm ( K U ) =
O ( | U | log | U | ) .
Overall,mvm ( ˜ K X ) =
O ( n 4 d + | U | log | U | ) , which is much faster than n 2 for small d .
However, SKIquickly becomes infeasible in higher dimensions due to the 4 d entries per row of W andcurse of dimensionality for the number of points in a high-dimensional grid: specifically,| U |
= m d for a grid with m points in each dimension.
2 . 2 Sparse Grids
Rectilinear grids.
We first give a formal construction of rectilinear grids, which willlater be the foundation for sparse grids [ 7 ].
For a resolution index l ∈ N 0 , definethe 1-d grid Ω l as the centers of 2 l equal partitions of the interval [ 0, 1 ] , which givesΩ l : =
{ i /2 l + 1 | 1 ≤ i ≤ 2 l +
and i is odd } .
The fact that the position index i must be oddimplies that grids for any two different resolutions are disjoint .
1 Moreover, resolutionposition index pairs ( l , i ) uniquely specify grid points in the union (cid:83) l ∈ N 0 Ω l of 1-d grids.
To extend rectilinear grids to d dimensions, let l ∈ N d 0 denote a resolution vector.
Thecorresponding rectilinear grid is given by Ω l : = ⊗ dj
= 1 Ω l j , where ⊗ denotes the Cartesianproduct.
A grid point in Ω l is indexed by the pair ( l , i ) of a resolution vector l and positionvector i , where ( l j , i j ) gives the position in the 1-d grid Ω j for dimension j .
This constructionof rectilinear grids yields three essential properties that will facilitate formalizing sparsegrids: ( 1 ) a grid Ω l is uniquely determined by its resolution vector l , ( 2 ) grids Ω l and Ω l ′with different resolution vectors are disjoint, ( 3 ) the size | Ω l | = 2 ∥ l ∥ 1 of a grid is determinedby the L 1 norm of its resolution vector.
Construction of Sparse Grids.
Sparse grids use rectilinear grids as their fundamentalbuilding block [ 22 ] and exploit the fact that resolution vectors uniquely identify differentrectilinear grids.
Larger grids are formed as the union of rectilinear grids with differentresolution vectors.
Sparse grids use all rectilinear grids with resolution vector having L 1norm below a specified threshold.
Formally, for a resolution index ℓ ∈ N 0 , the sparse grid
G ℓ , d in d dimensions is G ℓ , d : =
(cid:83) l :
∥ l ∥ 1 ≤ ℓ Ω l .
Figure 1 illustrates the construction of the sparse grid G 2,2 from smaller 2 -d rectilineargrids with maximum resolution 2, i.e., { Ω l | ∥ l ∥ 1 ≤ 2 } .
The figure also illustrates anotherimportant fact: the sparse grid G ℓ , d has a total of Θ ( 2 ℓ ) distinct and equally spacedcoordinates in each dimension, but many fewer total points than a dense d -fold Cartesianproduct of such 1 -d grids, which would have Θ ( 2 ℓ d ) points.
Sparse grids have a number of formal properties that are useful in algorithms and applications [ 22 , 7 ].
Proposition 1 below summarizes the most relevant ones for our work.
Forcompleteness, a proof appears in appendix A. For more details, see Valentin [ 26 ].
Proposition 1 ( Properties of Sparse Grid ) .
Let G ℓ , d ⊂ [ 0, 1 ] d be a sparse grid with any resolution ℓ ∈ N 0 and dimension d ∈ N .
Then the following properties hold:
[Equation]
Property P 1 shows that the size of sparse grid with Θ ( 2 ℓ ) points in each dimension growsmore slowly than a dense grid with the same number of points in each dimension, since
O ( 2 ℓ ℓ d − 1 ) ≪
O ( 2 ℓ d ) .
Properties P 2 and P 3 are consequences of the structure of the set
{∥ l ∥ 1 ≤ ℓ } and the sparse grid construction.
Property P 2 says that a sparse grid with 152 smaller resolution is contained in one with higher resolution.
Property P 3 is a crucialproperty, and says that a d -dimensional sparse grid can be constructed recursively via
Cartesian products of 1 -dimensional dense grids with sparse grids in d − 1 dimensions.
Structured Kernel Interpolation on Sparse Grids
To scale kernel interpolation to higher dimensions, we propose to select inducing points
U = G ℓ , d on a sparse grid and approximate the kernel matrix as WK G ℓ , d W T for a suitableinterpolation matrix W adapted to sparse grids.
This will require fast matrix-vectormultiplications with the sparse grid kernel matrix K G ℓ , d and the interpolation matrix W .
We show how to accomplish these two tasks in Sections 3 . 1 and 3 . 2 .
3 . 1 Fast Multiplication with the Sparse Grid Kernel Matrix
Algorithm 1 is an algorithm to compute K G ℓ , d v for any vector v .
The algorithm uses thefollowing definitions.
For any finite set U , let K U = (cid:2) k
( x , x ′ )
(cid:3) x , x ′ ∈ U .
The rows and columnsof K U are “ U -indexed”, meaning the entries correspond to elements of U under somearbitrary fixed ordering.
For U ⊆ V , we introduce a selection matrix S U , V to map between 166 U -indexed
and V -indexed vectors.
It has entries (
S U , V ) ij equal to one if the i th element of
U is equal to the j th element of V , and zero otherwise.
We also define S V , U = S TU , V .
For a
V -indexed vector z V , the multiplication S U , V z V produces a U -indexed vector by selectingentries corresponding to elements in U , and for a U -indexed vector
z U , the multiplication
S V ,
U z U produces a V -indexed vector by inserting zeros for elements not in U .
Input: v ∈ R |G ℓ , d | and K G ℓ , d ∈ R |G ℓ , d |×|G ℓ , d |
Output: u = K G ℓ , d v ∈ R |G ℓ ,
[Equation]
| 1 : Let V i be the result of reshaping S Ω i ⊗G ℓ − i , d − 1 , G ℓ , d v into a | Ω i | × |G ℓ − i , d − 1 | matrix; this contains entries of v corresponding to the i th grid in the decomposition of P 3 .
2 : if d = 1 then: return u = K G ℓ ,1 v ▷
[Table]
4 : end if 5 : ▷ Pre-computation: for i = 0 to ℓ do: a i =
K G i ,
[Equation]
G i ,1 , Ω
[Equation]
V i: b i =
V
i K G ℓ
[Equation]
i ,
[Equation]
▷ Recurse with d − 1 dimensions: end for 10 : ▷ Main loop: for i = 0 to ℓ do 12 :
[Equation]
▷ Recurse with d − 1 dimensions 13 :
[Equation]
14 : u i = vec [ A i ] + vec [ B i ]: end for
Theorem 1 .
For any v ∈ R |G ℓ , d | , Algorithm 1 computes K G ℓ , d v in O ( ℓ d + 1 2 ℓ ) time.
The formal analysis and proof of Theorem 1 appears in Appendix B .
The running time of
O ( ℓ d + 1 2 ℓ ) is nearly linear in |G ℓ , d | and much faster asymptotically than the naive MVMalgorithm that materializes the full matrix and has running time quadratic in |G ℓ , d | .
Algorithm 1 is built on two high-level observations.
First, the decomposition of Property P 3from Proposition 1 is used to decompose the MVM into blocks, each of which is betweensubgrids which are the product of a 1 -dimensional rectilinear grid and a sparse grid ind − 1 dimensions.
Therefore, the overall MVM computation can be recursively decomposedinto MVMs with sparse grid kernel matrices in d − 1 dimensions.
This observation isin part inspired by Zeiser [ 29 ], which also decomposes computation with the matrix onsparse grids by the resolution of first dimension.
Second, by Property P 2 , the kernel matrixmultiplication for any grid of resolution ℓ also includes the result of the multiplicationfor grids of lower resolution and the same number of dimensions.
Thus, the results ofthe multiplications for many individual blocks can be obtained by using the appropriateselection operators with the result of the multiplication with the kernel matrix K G ℓ − i , d − 1 in
Line 12 of the algorithm.
Further explanation is provided in Appendix B. 1 .
Improving batching efficiency.
The recursions in Lines 8 and 12 can be batched forefficiency, since both are multiplications with the same symmetric kernel matrix K G ℓ , d .
Similarly, the recursion spawns many recursive multiplications with kernel matrices of theform G ℓ ′ , d ′ for 0 ≤ ℓ ′ < ℓ and 1 ≤ d ′ < d , and the calculation can be reorganized to batch 190 all multiplications with each K
G ℓ ′ , d ′ .
This is a significant savings, because there are onlyd ( ℓ + 1 ) distinct kernel matrices, but the recursion has a branching factor of ( ℓ + 1 ) , sospawns many recursive calls with the same kernel matrices.
We provide the pseudocode ofthis efficient implementation of Algorithm 1 in the Appendix B. 1 .
3 . 2 Sparse Interpolation For Sparse Grids
We now seek to construct the matrix W , which interpolates function values from the sparsegrid G ℓ , d to training points x i ∈ R d , while ensuring that each row of W is sparse enoughto preserve efficiency of matrix-vector multiplications with W .
This requires a sparseinterpolation rule for sparse grids.
To set up the problem, we consider interpolating a function f observed at points ina generic set U .
Let f U =
( f ( x )) u ∈ U .
A linear interpolation rule for U is a mappingx (cid:55)→ w x ∈ R | U | used to approximate f ( x ) at an arbitrary point as f ( x ) ≈ w T x f U .
Thedensity of an interpolation rule is the maximum number of nonzeros in w x for any x .
The combination technique for sparse grids constructs an interpolation rule by combininginterpolation rules for the constituent rectilinear grids.
Proposition 2 .
For each l , let w lx be an interpolation rule for the rectilinear grid Ω l with maximumdensity c.
The combination technique gives an interpolation rule w x for the sparse grid G ℓ , d withdensity at most c × ( ℓ + d − 1 d − 1 ) .
The factor ( ℓ + d − 1 d − 1 ) is the number of rectilinear grids included in G ℓ , d .
We use the combination technique to construct the sparse-grid interpolation coefficientsw x i for each training point x i and stack them in the rows of W , which can be done in timeproportional to the number of nonzeros.
Details are given in Appendix A.
The combinationtechnique can use any base interpolation rule for the rectilinear grids, such as multilinear,cubic, or simplicial interpolation.
3 . 3 Simplicial Interpolation
The density of the base interpolation rule is a critical consideration for kernel interpolationtechniques—with or without sparse grids.
Linear and cubic interpolation in d dimensionshave density Θ ( 2 d ) and Θ ( 4 d ) , respectively.
This “second curse of dimensionality” makescomputations with W intractable in higher dimensions independently of operations withthe grid kernel matrix.
For sparse grids, the density of W increases by an additional factorof ( ℓ + d − 1 d − 1 ) .
We propose to use simplicial interpolation [ 11 ] for the underlying interpolation rule to avoidexponential growth of the density.
Simplicial interpolation refers to a scheme where R d ispartitioned into simplices and a point x is interpolated using only the d + 1 extreme pointsof the enclosing simplex, so the density of the interpolation rule is exactly d + 1.
Simplicialinterpolation was previously proposed for sparse grid classifiers in [ 8 ].
In work closelyrelated to ours, Kapoor et al.
[ 13 ] used simplicial interpolation for GP kernel interpolation,with the key difference that they use the permutohedral lattice as the underlying grid,which has a number of nice properties but does not come equipped with fast specializedroutines for kernel matrix multiplication.
In contrast, we maintain rectilinear and/or sparseunderlying grids, which preserve structure that enables fast kernel matrix multiplication.
For rectilinear grids, this requires partitioning each hyperrectangle into simplices, so theentire space is partitioned by simplices whose extreme points belong to the rectilinear grid.
Then, within each simplex, the values at the extreme points are interpolated linearly.
Ingeneral, there are different ways to partition hypperrectangles into simplices—we use thespecific scheme detailed in [ 11 ].
For sparse grids, we then use the combination technique,leading to overall density of ( d + 1 )( ℓ + d − 1 d − 1 ) .
More details on simplicial interpolation aregiven in the Appendix B. 2 .
Experiments
In this section, we empirically evaluate the time and memory taken by Algorithm 1 formatrix-vector multiplication with the sparse grid kernel matrix, the accuracy of sparsegrid interpolation and GP regression as the data dimension d increases, and the accuracyof sparse grid kernel interpolation for GP regression on real higher-dimensional datasetsfrom UCI.
Hyper-parameters, data processing steps, and optimization details are given in
Appendix C. 1 .
[Figure]
|G|
[Figure]
M
[Figure]
M t i m
[Figure]
n
[Figure]
M
[Figure]
m o r y
( i n
G B )
[Figure]
Sparse grid kernel MVM complexity.
First, we evaluate the efficiency of MVM algorithms.
We compare the basic and efficient implementations of Algorithm 1 to the naivealgorithm, which constructs the full kernel matrix and scales quadratically with the sparsegrid size.
Algorithm 1 has a significant theoretical advantage in terms of both time andmemory requirements as the grid size grows.
Figure 2 illustrates this for d = 6 by comparing MVM time and memory requirements for sparse grids with resolutions ℓ ∈ { 2, . .
, 9 }(roughly 100 to 1 M grid points).
The MVM time, preprocessing time, and memory consumption of Algorithm 1 all grow more slowly than the naive algorithm, and the efficientimplementation of Algorithm 1 is faster for |G| larger than about 10 4 , after which the naivealgorithm also exceeds the 10 GB memory limit.
For comparison, at ℓ = 6 (about 40 K gridpoints), Algorithm 1 uses only 0.05 GB of memory.
These results indicate that the proposedalgorithm is crucial for enabling sparse grid kernel interpolation in higher dimensions.
Figure 2 also depicts the typical time to run GP inference (i.e., preprocessing time plus
MVM operations).
Sparse grid interpolation and GP inference accuracy on synthetic data.
A significantadvantage of sparse grids over dense rectilinear grids is their ability to perform accurateinterpolation in higher dimensions.
We demonstrate this for d = 6 by interpolatingthe function f ( x ) = cos ( ∥ x ∥ 1 ) from observation locations on sparse grids of increasingresolution onto 200 random points sampled uniformly from [ 0, 1 ] d .
Figure 3 , left, shows theinterpolation error for dense and sparse grids with both cubic and simplicial interpolation.
Sparse (cubic) is significantly more accurate than Dense (cubic) , and Sparse (simplicial) is moreaccurate than Dense (simplicial) .
We next evaluate the accuracy of GP inference in increasing dimensions.
We keep the samefunction f ( x ) and generate observations as y = f ( x ) + N ( 0, 0.05 ) .
For all d , we use ℓ = 4for the sparse grid and compare to the dense grid with the closest possible total number ofgrid points (i.e., ⌈G 1/ d 4, d ⌉ points in each dimension).
For d ≥ 8, performance is better withsparse grids than dense grids for both interpolation schemes.
Remarkably, our proposalto use simplicial interpolation with dense grids allows SKI to scale to d = 10, which is asignificant improvement over prior work, in which SKI is typically infeasible for d ≥ 4.
[Figure]
GP regression performance on UCI datasets.
To evaluate the effectiveness of our proposed methods for scaling GP kernel interpolation to higher dimensions, we considerall UCI [ 4 ] data sets with dimension 8 ≤ d ≤ 10.
We compare our proposed methods,
Dense-grid (dense SKI with simplicial interpolation) and Sparse-grid (sparse-grid SKI withsimplicial interpolation) to SGPR [ 25 ], SKIP [ 10 ], and Simplex-GP [ 13 ].
For SGPR, wereport the best results using 256 or 512 inducing points.
For SKIP, 100 points per dimensionare used.
For Simplex-GP [ 13 ], the blur stencil order is set to 1.
Table 2 shows the rootmean squared error (RMSE) for all methods.
Our methods have performance comparableto and often better than SGPR and prior methods for scaling kernel interpolation to higher
[Table] [Table]
dimensions.
This shows that sparse grids and simplicial interpolation can effectively scale
SKI to higher dimensions to give a GP regression framework that is competitive withstate-of-the-art approaches.
We report additional results and analysis in Appendix C.
Related Works
Beyond SKI and its variants, a number scalable GP approximations have been investigated.
Most notable are the different variants of sparse GP approximations [ 27 , 23 , 18 ].
For minducing points, these methods require either Ω ( nm 2 ) time for direct solves or Ω ( nm ) timefor approximate kernel MVMs in iterative solvers.
While these methods generally do notleverage structured matrix algebra like the SKI framework, and thus have worse scaling interms of the number of inducing points m , they may achieve comparable accuracy withsmaller m , especially in higher dimensions.
Some work utilizes the SKI framework tofurther boost the performance of sparse GP approximations [ 12 ].
A closely related work to ours is on improving the dimension scaling of SKI through the useof low-rank approximation and product structure [ 10 ].
In another very closely related work,
Kapoor et al.
[ 13 ] recently proposed to scale SKI to higher dimensions via interpolationon a permutohedral lattice.
Like our work, they use simplicial interpolation.
Unlike ourwork, the kernel matrix on the permutohedral lattice does not have special structure thatadmits fast exact multiplications; they instead use a locality-based approximation thattakes into account the length scale of the kernel function by only considering pairs of gridpoints within a certain distance.
Another related direction of research is the adaptation ofsparse grid techniques for machine learning problems, e.g., classification and regression[ 16 ] and data mining [ 1 ].
These methods construct feature representations using sparsegrid points [ 3 ], often by selecting a subset of grid points relevant for the dataset [ 6 , 3 ].
Discussion
This work demonstrates that sparse grids and simplicial interpolation on a rectilinear grid,which are both old numerical techniques, can be used to scale GP kernel interpolation tohigher dimensions.
On several UCI benchmarking datasets with 8 to 10 dimensions, SKIwith sparse grids and simplicial interpolation achieves better or competitive regressionaccuracy compared to state-of-the-art GP regression approaches.
Limitations and future work.
Sparse grids and simplicial interpolation address twoimportant bottlenecks when scaling kernel interpolation to higher dimensions.
Sparse gridsallow scalable matrix-vector multiplications with the grid kernel matrix, and simplicialinterpolation allows scalable multiplications with the interpolation matrix W .
The relativelylarge number of rectilinear grids used to form a sparse grid—i.e., the factor of ( ℓ + d − 1 d − 1 ) in
Proposition 2 —is one limiting factor that makes multiplication by W more costly for sparsegrids.
Future research could investigate methods to mitigate this extra cost, and explorethe limits of scaling to even higher dimensions with sparse grids. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6 .
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
As our method is an improvement on existing methods therefore it doesn’t
have any direct negative consquences.
(d) Have you read the ethics review guidelines and ensured that your paperconforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the
main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section C.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, howthey were chosen)?
[Yes] See Section C. 1 .(c)
Did you report error bars (e
g., with respect to the random seed after runningexperiments multiple times)?
[Yes] See Figures 2 and 3 .(d)
Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See C.
If you are using existing assets (e
g., code, data, models) or curating/releasing newassets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a
URL?
[Yes] See Section C. 2 .(d)
Did you discuss whether
and how consent was obtained from people whose
[Table]
data you’re using/curating?
[N/A] We use only publicly available datasets.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally
[Table]
identifiable information or offensive content?
[N/A] We use only publiclyavailable datasets.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional
[Table]
Review Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total
[Table]
amount spent on participant compensation?
[N/A]