Original: Log-Linear-Time Gaussian Processes Using Binary Tree Kernels
Revised: Linear-Time Gaussian Processes Using Binary Tree Kernels

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: Afﬁliation email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Gaussian processes (GPs) produce good probabilistic models of functions, butmost GP kernels require O (( n + m ) n 2 ) time, where n is the number of data pointsand m the number of predictive locations.
Revised: Gaussian processes (GPs) produce good probabilistic models of functions, butmost GP kernels require O (( n + m ) n 2 ) time, where n is the number of data pointsand m the number of predictive locations.

Original: We present a new kernel that allows for
Revised: We present a new kernel that allows for

Original: Gaussian process regression in O (( n + m )log( n + m )) time.
Revised: Gaussian process regression in O ( n + m ) time.

Original: Our “binary tree”kernel places all data points on the leaves of a binary tree, with the kernel dependingonly on the depth of the deepest common ancestor.
Revised: Our “binary tree” kernel places alldata points on the leaves of a binary tree, with the kernel depending only on thedepth of the deepest common ancestor.

Original: We can store the resulting kernelmatrix in O ( n ) space in O ( nlog n ) time, as a sum of sparse rank-one matrices,and approximately invert the kernel matrix in O ( n ) time.
Revised: We can store the resulting kernel matrix in O ( n ) space, as a sum of sparse rank-one matrices, and approximately invert thekernel matrix in O ( n ) time.

Original: Sparse GP methods alsooffer linear run time, but they predict less well than higher dimensional kernels.
Revised: Sparse GP methods also offer O ( n ) run time, but theypredict less well than higher dimensional kernels.

Original: Ona classic suite of regression tasks, we compare our kernel against Matérn, sparse,and sparse variational kernels.
Revised: On a classic suite of regressiontasks, we compare our kernel against Matérn, sparse, and sparse variational kernels.

Original: The binary tree GP assigns the highest likelihood tothe test data on a plurality of datasets, usually achieves lower mean squared errorthan the sparse methods, and often ties or beats the Matérn GP.
Revised: The binary tree GP assigns the highest likelihood to the test data on a plurality ofdatasets, usually achieves lower mean squared error than the sparse methods, andoften ties or beats the Matérn GP.

Original: On large datasets,the binary tree GP is fastest, and much faster than a Matérn GP.
Revised: On large datasets, the binary tree GP is fastest,and much faster than a Matérn GP.

Original:  Introduction
Revised:  Introduction

Original: Gaussian processes (GPs) can be used to perform regression with high-quality uncertainty estimates,but they are slow.
Revised: Gaussian processes (GPs) can be used to perform regression with high-quality uncertainty estimates,but they are slow.

Original: Naïvely, GP regression requires O ( n 3 + n 2 m ) computation time and O ( n 2 )computation space when predicting at m locations given n data points [25].
Revised: Naïvely, GP regression requires O ( n 3 + n 2 m ) computation time and O ( n 2 )computation space when predicting at m locations given n data points [24].

Original: A kernel matrix of sizen ⇥ n must be inverted (or Cholesky decomposed) and then m matrix-vector multiplications mustbe done with that inverse matrix (or m linear solves with the Cholesky factors).
Revised: A kernel matrix of sizen ⇥ n must be inverted (or Cholesky decomposed) and then m matrix-vector multiplications mustbe done with that inverse matrix (or m linear solves with the Cholesky factors).

Original: A few methods thatwe will discuss later achieve O ( n 2 m ) time complexity [22, 27]
Revised: A few methods thatwe will discuss later achieve O ( n 2 m ) time complexity [21, 26].

Original: With special kernels, GP regression can be faster and use less space.
Revised: With special kernels GP regression can be faster and use less space.

Original: Inducing point methods, usingz inducing points, allow regression to be done in O ( z 2 ( n + m )) time and in O ( z 2 + zn ) space[18, 19, 20, 10].
Revised: Inducing point methods, usingz inducing points, allow regression to be done in O ( z 2 ( n + m )) time and in O ( z 2 + zn ) space[16, 17, 18, 9].

Original: We will discuss the details of these inducing point kernels later, but they are kernelsin their own right, not just approximations to other kernels.
Revised: We will discuss the details of these inducing point kernels later, but they are kernelsin their own right, not just approximations to other kernels.

Original: Unfortunately, if we have observed thefunction value at a location x , and have to predict the function value at that location x , these inducingpoint kernels do not automatically predict the observed value at x with very little uncertainty (unlessx is one of our inducing points).
Revised: Unfortunately, if we have observed thefunction value at a location x , and have to predict the function value at that location x , these inducingpoint kernels do not automatically predict the observed value at x with very little uncertainty (unlessx is one of our inducing points).

Original: This is the easiest possible prediction problem—simple recall.
Revised: This is the easiest possible prediction problem—simple recall.

Original: Inducing point methods fail it.
Revised: Inducing point methods fail it.

Original: We present a new kernel, the binary tree kernel , that also allows for GP regression in O ( n + m)space and O (( n + m ) log( n + m )) time (both model ﬁtting and prediction).
Revised: We present a new kernel, the binary tree kernel , that also allows for GP regression in O ( n + m ) timeand space (both model ﬁtting and prediction).

Original: The time and spacecomplexity of our method is also linear in the depth of the binary tree, which is naïvely linear in thedimension of the data, although in practice we can increase the depth sublinearly.

Original: Training somekernel parameters takes time quadratic in the depth of the tree.

Original: Unlike inducing point kernels, withthe binary tree kernel, if x is both a training data point and a predictive location, it will produce anaccurate and appropriately conﬁdent prediction.
Revised: Unlike inducing point kernels, with the binary treekernel, if x is both a training data point and a predictive location, it will produce an accurate andappropriately conﬁdent prediction.

Original: The fact that inducing point kernels fail this sanitycheck should make us suspicious of their performance in general
Revised: The fact that inducing point kernels fail this sanity check shouldmake us suspicious of their performance in general.

Original: In this example, k ( x 1 , x 1 ) = 1 , k ( x 1 , x 2 ) = 0 , k ( x 1 , x 3 )

Original: = 0 . 8 , and k ( x 1 , x 4 )

Original: = 0 .

Original: A simple depiction of our kernel is shown in Figure 1, which we will deﬁne precisely in Section 3.
Revised: A simple depiction of our kernel is shown in Figure 1, which we will deﬁne precisely in Section3.

Original: First, we create a procedure for placingall data points on the leaves of a binary tree.
Revised: First, we create a procedure for placing all data points on the leaves of a binary tree.

Original: Given the binary tree, the kernel between twopoints depends only on the depth of the deepestcommon ancestor.
Revised: Given the binary tree, the kernel between two points depends only on the depth of the deepest common ancestor.

Original: Because very different treestructures are possible for the data, we can easilyform an ensemble of diverse GP regression models.
Revised: Because very different tree structures are possible for the data, we can easily form an ensembleof diverse GP regression models.

Original: Figure 2 depicts a schematic sample from abinary tree kernel.
Revised: Figure 2 depicts a schematic sample from a binary tree kernel.

Original: Note how the posterior meanis piecewise ﬂat, but the pieces can be small.
Revised: Tree-based models are a well-known tool in machine learning [20]. In this example, k ( x 1 , x 1 ) = 1 , k ( x 1 , x 2 ) = 0 , k ( x 1 , x 3 ) = 0 . 8 , and k ( x 1 , x 4 ) = 0 .

Original: On a standard suite of benchmark regression tasks [22], we show that our kernel usually achievesbetter negative log likelihood (NLL) than state-of-the-art sparse methods and conjugate-gradientbased “exact” methods, at lower computational cost in the big-data regime.
Revised: On a standard suite of benchmark regressiontasks [21], we show that, on average, ourkernel achieves better negative log likelihood(NLL) than state-of-the-art sparse methods andconjugate-gradient-based “exact” methods, atlower computational cost in the big-data regime.

Original: There are not many limitations to using our kernel.
Revised: There are not many limitations to using our kernel.

Original: The main limitation is that other kernels sometimescapture the relationships in the data better.
Revised: The main limitation is that other kernelssometimes capture the relationships in the databetter.

Original: We do not have a good procedure for understandingwhen data has more Matérn character or more binary tree character (except through running bothand comparing training NLL).
Revised: We do not have a good procedure for understanding when data has more Matérn character or more binary tree character (except throughrunning both and comparing training NLL).

Original: But given that the binary tree kernel usually outperforms the Matérn,we’ll tentatively say the best ﬁrst guess is that a new dataset has more binary tree character.
Revised: Butgiven that the binary tree kernel usually outperforms the Matérn, we’ll tentatively say the best ﬁrst guess is that a new dataset has more binary treecharacter.

Original: Oneconcrete limitation for some applications, like Bayesian Optimization, is that the posterior mean ispiecewise-ﬂat, so gradient-based heuristics for ﬁnding extrema would not work.
Revised: One concrete limitation for some applications, like Bayesian Optimization, is that theposterior mean is piecewise-ﬂat, so gradient-based heuristics for ﬁnding extrema would not work.

Original: In contexts where a piecewise-ﬂat posterior mean is suitable, we struggle to see when one wouldprefer a sparse or sparse variational GP to a binary tree kernel.
Revised: In contexts where a piecewise-ﬂat posterior mean is suitable, we struggle to see when one wouldprefer a sparse or sparse variational GP to a binary tree kernel.

Original: The most thorough approach wouldbe to run both and see which has a better training NLL, but if you had to pick one, the binary tree
Revised: The most thorough approach wouldbe to run both and see which has a better training NLL, but if you had to pick one, the binary tree

Original: GP seems to be better performing and comparably fast.
Revised: GP seems to be better performing and comparably fast.

Original: If minimizing mean-squared error is theobjective, the Matérn kernel seems to do slightly better than the binary tree.
Revised: If minimizing mean-squared error is theobjective, the Matérn kernel seems to do slightly better than the binary tree.

Original: If the dataset is small,and one needs a very fast prediction, a Matérn kernel may be the best option.
Revised: If the dataset is small,and one needs a very fast prediction, a Matérn kernel may be the best option.

Original: But otherwise, if onecares about well-calibrated predictions, these initial results we present tentatively suggest using abinary tree kernel over the widely-used Matérn kernel.
Revised: But otherwise, if onecares about well-calibrated predictions, these initial results we present tentatively suggest using abinary tree kernel over the widely-used Matérn kernel.

Original: The log-linear time andlinear space complexity of the binary tree GP, with performance exceeding a“normal” kernel, could profoundly expand the viability of GP regression to larger datasets.
Revised: The linear time and space complexity of the binary tree GP, with performance exceeding a “normal”kernel, could profoundly expand the viability of Gaussian process regression to larger datasets.

Original:  Preliminaries
Revised:  Preliminaries

Original: Our problem setting is regression.
Revised: Our problem setting is regression.

Original: Given a function f : X !
Revised: Given a function f : X !

Original: R , for some arbitrary set X , we would 71 like to predict f ( x ) for various x 2 X .
Revised: R , for some arbitrary set X , we would 69 like to predict f ( x ) for various x 2 X .

Original: What we have are observations of f ( x ) for various (other) 72 x 2 X .
Revised: What we have are observations of f ( x ) for various (other) 70 x 2 X .

Original: Let X 2 X n be an n -tuple of elements of X , and let y 2 R n be an n -tuple of real numbers,such that y i ⇠ f
Revised: Let X 2 X n be an n -tuple of elements of X , and let y 2 R n be an n -tuple of real numbers,such that y i ⇠ f

Original: ( X i )
Revised: ( X i )

Original: + N (0 ,  ) , for  2 R  0 .
Revised: + N (0 ,  ) , for  2 R  0 .

Original: X and y comprise our training data.
Revised: X and y comprise our training data.

Original: With an m -tuple of test locations X 0 2 X m , let y 0 2 R m , with y 

Revised: With an m -tuple of test locations X 0 2 X m , let y 0 2 R m , with y

Original: i = f
Revised: i = f

Original: ( X 0 i ) .
Revised: ( X 0 i ) .

Original: y 0 is the ground truthfor the target locations.
Revised: y 0 is the ground truthfor the target locations.

Original: Given training data, we would like to produce a distribution over R for eachtarget location X 0 i , such that it assigns high marginal probability to the unknown y 0 i .
Revised: Given training data, we would like to produce a distribution over R for eachtarget location X 0 i , such that it assigns high marginal probability to the unknown y 0 i .

Original: Alternatively, wesometimes would like to produce point estimates ˆ y 0 i in order to minimize the squared error (ˆ y 

Revised: Alternatively, wesometimes would like to produce point estimates ˆ y 0 i in order to minimize the squared error (ˆ y

Original: i  y 0 i ) 2 .
Revised: i  y 0 i ) 2 .

Original: A GP prior over functions is deﬁned by a mean function m : X !
Revised: A GP prior over functions is deﬁned by a mean function m : X !

Original: R , and a kernel k : X ⇥ X !
Revised: R , and a kernel k : X ⇥ X !

Original: R .
Revised: R .

Original: The expected function value at a point x is deﬁned to be m ( x ) , and the covariance of the function values at two points x 1 and x 2 is deﬁned to be k ( x 1 , x 2 ) .
Revised: The expected function value at a point x is deﬁned to be m ( x ) , and the covariance of the functionvalues at two points x 1 and x 2 is deﬁned to be k ( x 1 , x 2 ) .

Original: Let K XX 2 R n ⇥ n be the matrix of kernelvalues ( K XX ) ij =k
Revised: Let K XX 2 R n ⇥ n be the matrix of kernel values ( K XX ) ij =

Original:  ( X i , X j ) , and let m X 2 R n be the vector of mean values ( m X ) i
Revised: k ( X i , X j ) , and let m X 2 R nbe the vector of mean values ( m X ) i

Original: = m ( X i ) .
Revised: = m ( X i ) .

Original: For a GP to be well-deﬁned, the kernel must be such that K XX is positive semideﬁnite for any X 2 X n .
Revised: For a GP to be well-deﬁned, the kernelmust be such that K XX is positive semideﬁnite for any X 2 X n .

Original: For a point x 2 X ,  Let K Xx 2 R n be the vector of kernel values: ( K Xx )
Revised: For a point x 2 X , 82 Let K Xx 2 R n be the vector of kernel values: ( K Xx )

Original: i =
Revised: i =

Original: k
Revised: k

Original: ( X i , x ) ,and let K xX =
Revised: ( X i , x ) , and let K xX =

Original: K > Xx .
Revised: K > Xx .

Original: Let   0 be the variance of observation noise.
Revised: Let   0 be the variance of observation noise.

Original: Let µ x and  2 x be the meanand variance of our posterior predictive distribution at x .
Revised: Let µ x and  2 x be the mean and variance of our posterior predictive distribution at x .

Original: Then, with K  inv XX =
Revised: Then, with K  inv XX =

Original: ( K XX +  I )  1 ,
Revised: ( K XX +  I )  1 ,

Original: [Equation]
Revised: [Equation]

Original: See Williams and Rasmussen [25] for a derivation.
Revised: See Williams and Rasmussen [24] for a derivation.

Original: We compute Equations 1 and 2 for all x 2 X 0 .
Revised: We compute Equations 1 and 2 for all x 2 X 0 .

Original:  Binary tree kernel
Revised:  Binary tree kernel

Original: We now introduce the binary tree kernel.
Revised: We now introduce the binary tree kernel.

Original: First, we encode our data points as binary strings.
Revised: First, we encode our data points as binary strings.

Original: So wehave X = B q , where B = { 0 , 1 } , and q 2 N .
Revised: So wehave X = B q , where B = { 0 , 1 } , and q 2 N .

Original: If X = R d , we must map R d 7!
Revised: If X = R d , we must map R d 7!

Original: B q .
Revised: B q .

Original: First, we rescale all points (training points and test points) to liewithin the box [0 , 1] d .
Revised: First, we rescale all points (training points and test points) to liewithin the box [0 , 1] d .

Original: (If we have a stream of test points, and one lands outside of the box [0 , 1] d , wecan either set K xX to 0 for that point or we rescale and retrain in O ( nlog n ) time.)
Revised: (If we have a stream of test points, and one lands outside of the box [0 , 1] d ,we can either set K xX to 0 for that point or we rescale and retrain in O ( n ) time.)

Original: Then, for eachx 2 [0 , 1] d , for each dimension, we take the binary expansion up to some precision p , and for thosed ⇥ p bits, we permute them using some ﬁxed permutation.
Revised: Then, for eachx 2 [0 , 1] d , for each dimension, we take the binary expansion up to some precision p , and for thosed ⇥ p bits, we permute them using some ﬁxed permutation.

Original: We call this permutation the bit order, andit is the same for all x 2 [0 , 1] d .
Revised: We call this permutation the bit order, andit is the same for all x 2 [0 , 1] d .

Original: Note that now q = dp .
Revised: Note that now q = dp .

Original: See Figure 3 for an example.
Revised: See Figure 3 for an example.

Original: We optimizethe bit order during training, and we can also form an ensemble of GPs using different bit orders.
Revised: We optimizethe bit order during training, and we can also form an ensemble of GPs using different bit orders.

Original: For x 2 B q , let x  i be the ﬁrst i bits of x .
Revised: For x 2 B q , let x  i be the ﬁrst i bits of x .

Original: [[ expression ]] evaluates to1, if expression is true, otherwise 0.
Revised: [[ expression ]] evaluates to1, if expression is true, otherwise 0.

Original: We now deﬁne the kernel:
Revised: We now deﬁne the kernel:

Original: Deﬁnition 1 (Binary Tree Kernel) .
Revised: Deﬁnition 1 (Binary Tree Kernel) .

Original: Given a weight vector w 2 R q ,with w ⌫ 0 and || w || 1 = 1 ,
Revised: Given a weight vector w 2 R q ,with w ⌫ 0 and || w || 1 = 1 ,

Original: [Equation]
Revised: [Equation]

Original: ii
Revised: ii

Original: So the more leading bits shared by x 1 and x 2 , the larger the covariance between the function values.
Revised: So the more leading bits shared by x 1 and x 2 , the larger the covariance between the function values.

Original: Consider, for example, points x 1and x 4 from Figure 1, where x 1 is ( left, left, right ) , and x 4 is ( left, right, right ) ; they share only theﬁrst leading “bit”.
Revised: Consider, for example, points x 1and x 4 from Figure 1, where x 1 is ( left, left, right ) , and x 4 is ( left, right, right ) ; they share only theﬁrst leading “bit”.

Original: We train the weight vector w to maximize the likelihood of the training data.
Revised: We train the weight vector w to maximize the likelihood of the training data.

Original: Proposition 1 (Positive Semideﬁniteness) .
Revised: Proposition 1 (Positive Semideﬁniteness) .

Original: For X 2 X n , for k = k w , K XX ⌫ 0 .
Revised: For X 2 X n , for k = k w , K XX ⌫ 0 .

Original: Proof.
Revised: Proof.

Original: Let s
Revised: Let s

Original: S qi = 1 B i be a binary string, and let | s | be the length of s .
Revised: S qr =1 B r be a binary string, and let | s | be the length of s .

Original: Let X [ s ] 2 R n with( X [ s ] ) 
Revised: Let X [ s ] 2 R n with( X [ s ] ) i

Original: j = hh
Revised: = hh

Original: X  | s | j = s ii .
Revised: X  | s | i = s ii .

Original: X [ s ] X > [ s ] is clearly positive semideﬁnite.
Revised: X [ s ] X > [ s ] is clearly positive semideﬁnite.

Original: Finally,
Revised: Finally,

Original: [Equation]
Revised: [Equation]

Original:  (3) and recall w i  0 , so K XX ⌫ 0 .
Revised: ] (3) and recall w i  0 , so K XX ⌫ 0 .

Original:  Sparse rank one sum representation
Revised:  Sparse rank one sum representation

Original: In order to do GP regression in O ( n ) space andO ( n log n ) time, we develop a “Sparse Rank One Sum” representation of linear operators (SROS).
Revised: In order to do GP regression in O ( n ) space and time, we develop a “Sparse Rank One Sum”representation of linear operators (SROS).

Original: In SROS form, linear transformation of a vector canbe done in O ( n ) time instead of O ( n 2 ) .
Revised: In SROS form, linear transformation of a vector can bedone in O ( n ) time instead of O ( n 2 ) .

Original: We will store our kernel matrix and inverse kernel matrix in
Revised: We will store our kernel matrix and inverse kernel matrix in

Original: SROS form.
Revised: SROS form.

Original: The proof of Proposition 1 exempliﬁes representing a matrix as the sum of sparse rankone matrices.
Revised: The proof of Proposition 1 exempliﬁes representing a matrix as the sum of sparse rankone matrices.

Original: Note that each X [ s ] is sparse—if q is large, most X [ s ] ’s are the zero vector.
Revised: Note that each X [ s ] is sparse—if q is large, most X [ s ] ’s are the zero vector.

Original: We now show how to interpret an SROS representation of an n ⇥ n matrix.
Revised: We now show how to interpret an SROS representation of an n ⇥ n matrix.

Original: Let [ n ] = { 1 , 2 , ..., n } .
Revised: Let [ n ] = { 1 , 2 , ..., n } .

Original: For r 2 N , let L : [ r ] n ⇥ [ r ] n ⇥ R n ⇥ R n !
Revised: For m 2 N , let L : [ m ] n ⇥ [ m ] n ⇥ R n ⇥ R n !

Original: R n ⇥ n construct a linear operator from four vectors.
Revised: R n ⇥ n construct a linear operator from four vectors.

Original: Deﬁnition 2 (Linear Operator from Simple SROS Representation) .
Revised: Deﬁnition 2 (Linear Operator from Simple SROS Representation) .

Original: Let p, p 0 2 [ r ] n , and let u, u 0 [-
-] R n .
Revised: Let p, p 0 2 [ m ] n , and letu, u 0 [-
-]2 R n .

Original: For l 2 [ r ] , let u p =
Revised: Let u p =

Original: l 2 R n be the vector where u p = l j = u j [[ p j = l ]] ,  likewise for u 0 and p 0 .
Revised: i 2 R n be the vector where u p = i j = u j [[ p j = i ]] , and likewise for u 0 and p 0 .

Original: Then:
Revised: Then:

Original: L ( p, p 0 , u, u 0 ) 7! r
Revised: L [Equation] p, [Equation]

Original: X l = ( u p = l )(( u 0 ) p 0 = l ) > .
Revised: [Equation]

Original: We depict Deﬁnition 2 in Figure 4.
Revised: We depict Deﬁnition 2 in Figure 4.

Original: p and p 0 represent partitions over n elements: all elements with the same integervalue in the vector p belong to the same partition.
Revised: p and p 0 represent partitions over n elements: all elements with the same integervalue in the vector p belong to the same partition.

Original: Note thatr , the number of parts in the partition, need not exceed n , thenumber of elements being partitioned.

Original: If p = p 0 (which isalmost always the case for us) and the elements of p , u , and u 0were shufﬂed so that all elements in the same partition werenext to each other, then L ( p, p 0 , u, u 0 ) would be block diagonal.
Revised: If p = p 0(which is almost always the case for us) and the elements ofp , u , and u 0 were shufﬂed so that all elements in the samepartition were next to each other, then L ( p, p 0 , u, u 0 ) wouldbe block diagonal.

Original: Note that L ( p, p 0 , u, u 0 ) isnot necessarily low rank. 
Revised: Note that L ( p, p 0 , u, u 0 ) is low rank. L ( p p 0 , u, u

Original: Ifp is the ﬁnest possible partition, and p = p 0 , L ( p, p 0 , u, u 0 ) isdiagonal.
Revised: ) is diagonal.

Original: SROS matrices can be thought of as a generalizationof two types of matrix that are famously amenable to fast computation: rank one matrices (all points in the same partition)and diagonal matrices(each point in its own partition).
Revised: SROS matrices can be thought ofas a generalization of two types of matrix that are famouslyamenable to fast computation one matrices ( diagonal matrices

Original: We now extend the deﬁnition of L to allow for multiple p , p 0 ,u , and u 0 vectors.
Revised: We now extend the deﬁnition of L to allow for multiple p , p 0 ,u , and u 0 vectors.

Original: Deﬁnition 3 (Linear Operator from SROS Representation) .
Revised: Deﬁnition 3 (Linear Operator from SROS Representation) .

Original: Let L : [ r ] n ⇥ q ⇥ [ r ] n ⇥ q ⇥ R n ⇥ q ⇥ R n ⇥ q ! R n ⇥ n .
Revised: Let L : [ m ] n ⇥ q ⇥ [ m ] n ⇥ q ⇥ R n ⇥ q ⇥ R n ⇥ q ! R n ⇥ n .

Original: Let P, P 0 2[ r ] n ⇥ q , and let U, U 0 2 R n ⇥ q .
Revised: Let P, P 0 2 [ m ] n ⇥ q , and let U, U 0 2 R n ⇥ q .

Original: Let P : ,i , U : ,i , etc.
Revised: Let P : ,i , U : ,i , etc.

Original: be the i th columns of the respective arrays.
Revised: bethe i th columns of the respective arrays.

Original: Then: L ( P, P 0 , U, U 0 ) 7!
Revised: Then: L ( P, P 0 , U, U 0 ) 7!

Original: P qi =
Revised: P qi =

Original: 1 L ( P : ,i , P 0 : ,i , U : ,i , U 0 : ,i ) .
Revised: 1 L ( P : ,i , P 0 : ,i , U : ,i , U 0 : ,i ) .

Original: Algorithm1 performs linear transformation of a vector using SROS representation in O ( nq ) time. Algorithm 1 Linear Transformation with SROS Linear Operator. This can bevectorized on a Graphical Processing Unit (GPU), using e.g. torch.
Revised: Algorithm be

Original: Tensor.index_add_ for Line 5 and non-slice indexing for Line 6 [16].
Revised: Processing Unit e. non-slice indexing for Line 6 [14].

Original: Slight restructuring allows vectorization over [ q ] as well.
Revised: Slight restructuring allows vectorization over [ q ] as well.

Original: Require: P, P 0 2 [ r ] n ⇥ q , U, U 0 2 R n ⇥ q , x 2 R n
Revised: Require: P, P 0 2 [ m ] n ⇥ q , U, U 0 2 R n ⇥ q , x 2 R n

Original: Ensure:
Revised: Ensure:

Original: y =
Revised: y =

Original: L ( P, P 0 , U, U 0 )
Revised: L ( P, P 0 , U, U 0 )

Original: x 1: y 0 2 R n 2: for i 2 [ q ] do .
Revised: x 1: y 0 2 R n 2: for i 2 [ q ] do .

Original: O ( nq ) time 3: p, p 0 , u, u
Revised: O ( nq ) time 3: p, p 0 , u, u

Original: P : ,i , P 0 : ,i , U : ,i , U 0 : ,i 4: z 0 2 R r .
Revised: P : ,i , P 0 : ,i , U : ,i , U 0 : ,i 4: z 0 2 R m .

Original: z l will store the dot product (( u 0 ) p 0{+
+} = l ) > x p 

Revised: z i will store the dot product (( u 0 ) p {+
+} = i ) > x p

Original: = l 5: for j 2 [ n ] do z p 

Revised: = i 5: for j 2 [ n ] do z p

Original: j z p 

Revised: j z p

Original: j + u 0 j x j .
Revised: j + u 0 j x j .

Original: O ( n ) time 6: for j 2 [ n ] do y j y j
Revised: O ( n ) time 6: for j 2 [ n ] do y j y j

Original: + z p j u j .
Revised: + z p j u j .

Original: O ( n ) time return y
Revised: O ( n ) time return y

Original: We now discuss how to approximately invert a certain kind of symmetric SROS matrix, but ourmethods could be extended to asymmetric matrices.
Revised: We now discuss how to approximately invert a certain kind of symmetric SROS matrix, but ourmethods could be extended to asymmetric matrices.

Original: First, we deﬁne a partial ordering over partitions.
Revised: First, we deﬁne a partial ordering over partitions.

Original: For two partitions p, p 0 , we say p 0  p if p 0 is ﬁner than or equal to p ; that is, p 

Revised: For two partitions p, p 0 , we say p 0  p if p 0 is ﬁner than or equal to p ; that is, p

Original: j = p 

Revised: i = p

Original: j0 = ) p j = p j0 .
Revised: j = ) p i = p j .

Original: Using that partial ordering, a symmetric SROS matrix can be approximately inverted efﬁciently if forall 1  i, i 0  q , P : ,i  P : ,i 0 or P : ,i 0  P : ,i .
Revised: Using that partial ordering, a symmetric SROS matrix can be approximately inverted efﬁciently if forall 1  i, j  q , P : ,i  P : ,j or P : ,j  P : ,i .

Original: As the reader may have recognized, our kernel matrix 146 K
Revised: As the reader may have recognized, our kernel matrix 145 K

Original: XX can be written as an SROS matrix with this property.
Revised: XX can be written as an SROS matrix with this property.

Original: We will write symmetric SROS matrices in a slightly more convenient form.
Revised: We will write symmetric SROS matrices in a slightly more convenient form.

Original: All ( u 0 ) p
Revised: All ( u 0 ) p

Original: = l must be aconstant times u p = l .
Revised: = i must be aconstant times u p = i .

Original: We will store these constants in an array C .
Revised: We will store these constants in an array C .

Original: Let L ( P, C, U ) be shorthand for
Revised: Let L ( P, C, U ) be shorthand for

Original: L ( P, P, U, C  U ) , where  denotes element-wise multiplication.
Revised: L ( P, P, U, C  U ) , where  denotes element-wise multiplication.

Original: For L ( P, C, U ) to be symmetric,it must be the case that P ji = P j 0 i = )
Revised: For L ( P, C, U ) to be symmetric,it must be the case that P kj = P lj = )

Original: C ji = C j 0 i .
Revised: C kj = C lj .

Original: Then, all elements of U corresponding toa given u p =l
Revised: Then, all elements of U corresponding toa given u p =

Original:  are multiplied by the same constant.
Revised: i are multiplied by the same constant.

Original: We now present an algorithm for calculating( L ( P, C, U ) +  I )  1 , for  6 = 0, which is an approximate inversion of L ( P, C, U ) .
Revised: We now present an algorithm for calculating( L ( P, C, U ) +  I )  1 , for  6 = 0 .

Original: We have notyet analyzed numerical sensitivity for  ! 0 , but we conjecture that all ﬂoating point numbers involved need to be stored to at least log 2 (1 /  ) bits.
Revised: We have not yet analyzed numerical sensitivity for  ! 0 , butwe conjecture that all ﬂoating point numbers involved need to be stored to at least log 2 (1 /  ) bits.

Original: Without loss of generality, let  = 1 , and note( L ( P, C, U ) +  I )  

Revised: Without loss of generality, let  = 1 , and note ( L ( P, C, U ) +  I )

Original: =   1 ( L ( P,   1 C, U ) + I )  1 .
Revised: =   1 ( L ( P,   1 C, U ) + I )  1 .

Original: By assumption, all columns of P are comparable with respect to the partial ordering above, so wecan reorder the columns of P such that P : ,i  P : ,j for i < j .
Revised: By assumption, all columns of P are comparable with respect to the partial ordering above, so wecan reorder the columns of P such that P : ,i  P : ,j for i < j .

Original: The key identity that we use to developour fast inversion algorithm is the Sherman–Morrison Formula:
Revised: The key identity that we use to developour fast inversion algorithm is the Sherman–Morrison Formula:

Original: [Equation]
Revised: [Equation]

Original: Starting with A
Revised: Starting with A

Original: =
Revised: =

Original: I , we add the sparse rank one matrices iteratively, from the ﬁnest partition to thecoarsest one, updating A  1 as we go.
Revised: I , we add the sparse rank one matrices iteratively, from the ﬁnest partition to thecoarsest one, updating A  1 as we go.

Original: We represent ( L ( P, C, U )+ I )  1 in the form I + L ( P, C 0 , U 0 ) ,so we write an algorithm that returns C 0 and U 0 .
Revised: We represent ( L ( P, C, U )+ I )  1 in the form I + L ( P, C 0 , U 0 ) ,so we write an algorithm that returns C 0 and U 0 .

Original: We can also quickly calculate log | L ( P, C, U )
Revised: We can also quickly calculate log | L ( P, C, U )

Original: + I |at the same time, using the matrix determinant lemma: | A + cuu > | =
Revised: + I |at the same time, using the matrix determinant lemma: | A + cuu > | =

Original: (1 + cu > A  1 u ) | A | .
Revised: (1 + cu > A  1 u ) | A | .

Original: Theorem 1 (Fast Inversion) .
Revised: Theorem 1 (Fast Inversion) .

Original: For P 2 [ r ] n ⇥ q and C, U 2 R n ⇥ q , if P : ,i (is coarser than) 
Revised: For P 2 [ m ] n ⇥ q and C, U 2 R n ⇥ q , if P : ,i (is coarser than)

Original: P : ,j for i <
Revised: P : ,j for i <

Original: j ,then there exists C 0 , U 0 2 R n ⇥ q , such that ( L ( P, C, U ) + I )  

Revised: j ,then there exists C 0 , U 0 2 R n ⇥ q , such that ( L ( P, C, U ) + I )

Original: = I + L ( P, C 0 , U 0 ) .
Revised: = I + L ( P, C 0 , U 0 ) .

Original: There exists analgorithm for computing C 

Revised: There exists analgorithm for computing C

Original: and U 0 that takes O ( nq 2 ) time.
Revised: and U 0 that takes O ( nq 2 ) time.

Original: Proof.
Revised: Proof.

Original: For X 2 R n ⇥ q , let X : ,i +1: q 2 R n ⇥ ( q  i ) be columns i + 1 through q of matrix X (inclusive).
Revised: For X 2 R n ⇥ m , let X : ,i +1: m 2 R n ⇥ ( m  i ) be columns i + 1 through m of matrix X(inclusive).

Original: Let A i
Revised: Let A i

Original: = I + L ( P : ,i +1: q , C : ,i +1: q , U : ,i +1: q ) , and A q = I .
Revised: = I + L ( P : ,i +1: q , C : ,i +1: q , U : ,i +1: q ) , and A q = I .

Original: Now suppose A  1 i can be written as 
Revised: Now suppose A  1 i can bewritten as I + L ( P : ,i +1:

Original: I + L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) for some C 0 and U 0 .
Revised: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) for some C 0 and U 0 .

Original: For the base case of i = q , this holds trivially.
Revised: For the base case of i = q , thisholds trivially.

Original: We show it also holds for i  1 , and we can compute C 0 : ,i : q , U 0 : ,i : q in O  n ( q  i )  time.
Revised: We show it also holds for i  1 , and we can compute C 0 : ,i : q , U 0 : ,i : q in O  n ( q  i )  time.

Original: Let p
Revised: Let p

Original: =
Revised: =

Original: P : ,i ,u
Revised: P : ,i , u

Original: = U : ,i , and c = C : ,i .
Revised: = U : ,i , and c = C : ,i .

Original: Consider u p = l , where each element is zero unless the corresponding elementof p equals l .
Revised: Consider u p = j , where each element is zero unless thecorresponding element of p equals j .

Original: What do we know about the product
Revised: What do we know about the product

Original: A  1 i u p = l (as seen in Equation 4)?
Revised: A  1 i u p = j (as seen in Equation 4)?

Original: Because  the columns of Pgo from coarser partitions to ﬁner ones, all of the vectors generatingthe sparse rank one components of L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) are from partitions that are equalto or ﬁner than p .
Revised: Because we have reordered the columns of P from coarse partitions to ﬁne ones, all of the vectorsgenerating the sparse rank one components of L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) are from partitionsthat are equal to or ﬁner than p .

Original: Thus, they are either zero everywhere u p =
Revised: Thus, they are either zero everywhere u p =

Original: l is zero, or zero everywhere u p = l isnonzero.
Revised: j is zero, or zeroeverywhere u p = j is nonzero.

Original: Vectors v of the second kind can be ignored, as cvv > u p = l = 0 .
Revised: Vectors v of the second kind can be ignored, as cvv > u p = j = 0 .

Original: Thus, when multiplying L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) by u p = l , the only relevant vectors are ﬁlled  with zeros except wherethe corresponding element of p equals l .
Revised: Thus, when multiplying L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) by u p = j , the only relevant vectors are ﬁlled 176 with zeros except where the corresponding element of p equals j .

Original: So we can get rid of those rows of P : ,i +1: q , C0 : ,i +1: q , and U0 : ,i +1: q .
Revised: So we can get rid of thoserows of P : ,i +1: q , C : ,i +1: q , and U : ,i +1: q .

Original: Suppose there are n l elements of p that equal l .
Revised: Suppose there are n j elements of p that equal j .

Original: Then L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) u 
Revised: Then L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) u p =

Original: p = linvolves n l rows, and can be computed in O  n [Equation] i 
Revised: j involves n j rows, and can be computed in O  n j ( q  i )  time.

Original: [Equation]  time. Moreover, this product, which we’llcall ( u 0 ) p = l , is only nonzero when the corresponding element of p equals l , so it has the same sparsitypattern as u p = l .
Revised: Moreover, this product, which we will call ( u 0 ) p = j , is only nonzero when the corresponding elementof p equals j , so it has the same sparsity pattern as u p = j .

Original: The other component of A  1 i is the identity matrix, and Iu p
Revised: The other component of A  1 i is the identitymatrix, and Iu p

Original: =
Revised: =

Original: l clearly has the samesparsity as u p = l .
Revised: j clearly has the same sparsity as u p = j .

Original: Thus, returning to Equation 4, when we add u p = l ( u p = l ) > to A i , we update A  1 iwith an outer product of vectors whose sparsity pattern is the same as that of u p = l .
Revised: Thus, returning to Equation 4, when we addu p = j ( u p = j ) > to A i , we update A  1 i with an outer product of vectors whose sparsity pattern is thesame as that of u p = j .

Original: For each l , A  1 i need not be updated with each u p =
Revised: For each j , A  1 i need not be updated with each u p =

Original: l one at a time.
Revised: j one at a time.

Original: For l 6 = l 0 , u p = l and u p = l 0are nonzero at separate indices, so u p = l and ( u 0 ) p=
Revised: For j 6 = j 0 , u p = j and u p = j 0are nonzero at separate indices, so u p = j and ( u 0 ) p

Original: l 0 are nonzero at separate indices, so the extracomponent of A  1 i that appears after  the u p = l 0 update is irrelevant to the u p = l update, because( u p = l ) >
Revised: = j 0 are nonzero at separate indices, so the extracomponent of A  1 i that appears after after the u p = j 0 update is irrelevant to the u p = j update.

Original: ( u 0 ) p=
Revised: Since theu p

Original: l 0 =0 .
Revised: =

Original: Since the u p = l update takes O ( n l ( q  i )) time, all of them together take O (
Revised: j update takes O ( n j ( q  i )) time, all of them together take O (

Original: P l n l ( q  i )) time, which equals O ( n ( q  i )) time.
Revised: P j n j ( q  i )) time, which equals O ( n ( q  i )) time.

Original: Calculatingan element of c 0{+
 i+} only involvescomputing the denominator in Equation 4, using a matrix-vector product already computed.
Revised: Calculating c {+
 i+} only involves computing the denominator in Equation 4, using amatrix-vector product already computed.

Original: So wecan write C 0 : ,i : q and U 0 : ,i : q by adding a preceding column to C 0 : ,i +1:
Revised: So we can write C 0 : ,i : q and U 0 : ,i : q by adding a precedingcolumn to C 0 : ,i +1:

Original: q and U 0 : ,i +1: q , using the samepartition p , and  it takes O ( n ( q  i )) time.
Revised: q and U 0 : ,i +1: q , using the same partition p , and we can do it in O ( n ( q  i )) time.

Original: Following the induction down to i = 0 , we have ( L ( P, C, U ) + I )  

Revised: Following the induction down to i = 0 , we have ( L ( P, C, U ) + I )

Original: =
Revised: =

Original: I + L ( P, C 0 , U 0 ) , and a totaltime of O ( nq 2 ) .
Revised: I + L ( P, C 0 , U 0 ) , and a totaltime of O ( nq 2 ) .

Original: Algorithm 2 also performs approximate inversion, which we prove in Appendix A. It differs slightlyfrom the algorithm in the proof, but can take full advantage of a GPU speedup.
Revised: Algorithm 2 also performs approximate inversion, which we prove in Appendix A, but differs slightlyfrom the algorithm in the proof above. Algorithm 2 can take full advantage of a GPU speedup.

Original: In the setting whereall columns of U are identical, observe that in Lines 10 and 11, the same computation is  repeated forall k 

Revised: Inthe special setting where all the columns of U are identical, observe that in Lines 10 and 11, thesame computation is being repeated for all k

Original: [ i ] .
Revised: [ i ] .

Original: Indeed, in this setting, this block of code can be modiﬁed to run in O ( n ) time  rather than O ( ni ) , making the whole algorithm runin O ( nq ) time, as shown in Proposition4 in Appendix A.
Revised: Indeed, in this setting, this block of code can bemodiﬁed to run in O ( n ) time and space rather than O ( ni ) , making the whole algorithm run O ( nq )time Proposition

Revised: Algorithm

Original: Require:
Revised: Require:

Original: P 2 [ r ] n ⇥ q , C, U 2 R n ⇥ q
Revised: P 2 [ m ] n ⇥ q , C, U 2 R n ⇥ q

Original: Ensure: I + L ( P, C 0 , U 0 )
Revised: Ensure: I + L ( P, C 0 , U 0 )

Original: =
Revised: =

Original: ( I + L ( P, C, U ))  1 ; x = log | I + L
Revised: ( I + L ( P, C, U ))  1 ; x = log | I + L

Original: ( P, C, U )
Revised: ( P, C, U )

Original: | 1: x, C 0 , U0 , 0 2 R n ⇥ q , U 2: for i 2 ( q, q  1 , ..., 1) do .
Revised: | 1: x, C 0 , U0 , 0 2 R n ⇥ q , U 2: for i 2 ( q, q  1 , ..., 1) do .

Original: O ( nq 2 ) time 3: p, c, u, u
Revised: O ( nq 2 ) time 3: p, c, u, u

Original: P : ,i , C : ,i , U : ,i , U 0 : ,i 4: z 0 2 R r .
Revised: P : ,i , C : ,i , U : ,i , U 0 : ,i 4: z 0 2 R m .

Original: z l will store c ( l ) (( u 0 ) p = l ) > u p = l , where c ( l ) =
Revised: z i will store c ( i ) (( u 0 ) p = i ) > u p = i , where c ( i ) =

Original: c k if p k = l 5: for j 2 [ n ] do z p j z p j
Revised: c k if p k = i 5: for j 2 [ n ] do z p j z p j

Original: + c j u 0 j u j .
Revised: + c j u 0 j u j .

Original: O ( n ) time 6: for j 2 [ n ] do C 0 ji  c j / (1 + z
Revised: O ( n ) time 6: for j 2 [ n ] do C 0 ji  c j / (1 + z

Original: p j ) .
Revised: p j ) .

Original: O ( n ) time 7: for l 2 [ r ] do x x + log(

Revised: O ( n ) time 7: for i 2 [ m ] do x x + log(

Original: + z l ) .
Revised: + z i ) .

Original: O ( n ) timebecause r  n 8: if i > 0 then 9: y 0 2 R n ⇥ i .
Revised: O ( n ) time 8: if i > 0 then 9: y 0 2 R n ⇥ i .

Original: O ( ni ) time 10: for j, k 2 [ n ] ⇥
Revised: O ( ni ) time 10: for j, k 2 [ n ] ⇥

Original: [ i  1] do y p
Revised: [ i  1] do y p

Original: j k y p
Revised: j k y p

Original: j k + u 

Revised: j k + u

Original: j U jk .
Revised: j U jk .

Original: O ( ni ) time 11: for j, k 2 [ n ] ⇥
Revised: O ( ni ) time 11: for j, k 2 [ n ] ⇥

Original: [ i  1] do U 0 jk
Revised: [ i  1] do U 0 jk

Original: U 0 jk + C 0 ji u 0 j y p
Revised: U 0 jk + C 0 ji u 0 j y p

Original: j k .
Revised: j k .

Original: O ( ni ) time return C 0 , U 0 , x
Revised: O ( ni ) time return C 0 , U 0 , x

Original:  Binary tree Gaussian process
Revised:  Binary tree Gaussian process

Original: We now show that our kernel matrix K XX can be written in SROS form, with P containing successively ﬁner partitions.
Revised: We now show that our kernel matrix K XX can be written in SROS form, with P containing successively ﬁner partitions.

Original: Thus, K XX can be approximately inverted quickly, for use in Equations 1 and2.
Revised: Thus, K XX can be approximately inverted quickly, for use in Equations 1 and2.

Original: Next, we’ll show that we can efﬁciently optimize the log likelihood of the training data by tuningthe weight vector w along with the bit order.
Revised: Next, we’ll show that we can efﬁciently optimize the log likelihood of the training data by tuningthe weight vector w along with the bit order.

Original: The log likelihood can be calculated in O ( nq log n )time and then the gradient w.r.t.
Revised: The log likelihood can be calculated in O ( nq ) time andthe gradient w.r.t.

Original: w in O ( nq 2 ) time.
Revised: w in O ( nq 2 ) time.

Original: Recall Equation 3 in the proof of Proposition 1 : K XX =
Revised: Recall Equation 3 in the proof of Proposition 1 : K XX =

Original: P qi =
Revised: P qi =

Original: P s 2 B i w i X
Revised: P s 2 B i w i X

Original: [ s ] X > [ s ] , where X [ s ] 2 R nwith ( X [ s ] ) j = hh
Revised: [ s ] X > [ s ] , where X [ s ] 2 R nwith ( X [ s ] ) i = hh

Original: X  | s | j = s ii .
Revised: X  | s | i = s ii .

Original: So we will set P : ,i , C : ,i , and U : ,i , so that L ( P : ,i , C : ,i , U : ,i ) =
Revised: So we will set P : ,i , C : ,i , and U : ,i , so that L ( P : ,i , C : ,i , U : ,i ) =

Original: P s 2 B i w i X
Revised: P s 2 B i w i X

Original: [ s ] X > [ s ] .
Revised: [ s ] X > [ s ] .

Original: Let P : ,i partition the set of points X
Revised: Let P : ,i partition the set of points X

Original: so that points are in the same partition if theﬁrst i bits match.
Revised: so that points are in the same partition if theﬁrst i bits match.

Original: Now, requiring the ﬁrst i + 1 bits to match is a stricter criterion than requiring theﬁrst i bits to match, so the P : ,i grow successively ﬁner.
Revised: Now, requiring the ﬁrst i + 1 bits to match is a stricter criterion than requiring theﬁrst i bits to match, so the P : ,i grow successively ﬁner.

Original: For any piece of the partition where the ﬁrst ibits of the constituent points equals the bitstring s , the corresponding sparse rank one component of
Revised: For any piece of the partition where the ﬁrst ibits of the constituent points equals the bitstring s , the corresponding sparse rank one component of

Original: K XX is w i X
Revised: K XX is w i X

Original: [ s ] X > [ s ] .
Revised: [ s ] X > [ s ] .

Original: So let U : ,i = 1 n , and let C : ,i =
Revised: So let U : ,i = 1 n , and let C : ,i =

Original: w i 1 n .
Revised: w i 1 n .

Original: Proposition 2 (SROS Form Kernel) .
Revised: Proposition 2 (SROS Form Kernel) .

Original: K XX =
Revised: K XX =

Original: L ( P, C, U ) , as deﬁned above.
Revised: L ( P, C, U ) , as deﬁned above.

Original: This follows immediately from the deﬁnitions.
Revised: This follows immediately from the deﬁnitions.

Original: To compute these partitions P : ,i , we sort X , which isa set of bit strings.
Revised: To compute these partitions P : ,i , we sort X , which isa set of bit strings.

Original: And then we can easily compute which points have the same ﬁrst i bits.
Revised: And then we can easily compute which points have the same ﬁrst i bits.

Original: This alltakes O ( nq log n ) time.
Revised: This alltakes O ( nq log n ) time.

Original: Now note that U : ,i = U : ,i 0 for all i, i 0 , so
Revised: Now note that U : ,i = U : ,j for all i, j , so

Original: ( K XX +  I )  1 and | K XX +  I |can be computed in O ( nq ) time, rather than O ( nq 2 ) .
Revised: ( K XX +  I )  1 and | K XX +  I |can be computed in O ( nq ) time, rather than O ( nq 2 ) .

Original: The training negative log likelihood of a GP is that of the corresponding multivariate Gaussian onthe training data.
Revised: The training negative log likelihood of a GP is that of the corresponding multivariate Gaussian onthe training data.

Original: So: NLL ( w ) =
Revised: So: NLL ( w [Equation] =

Original: [Equation]
Revised: [Equation]

Original: This can be computed in O ( nq ) time, since matrix-vector multiplication takes O ( nq ) time for amatrix in SROS form.
Revised: This can be computed in O ( nq ) time, since matrix-vector multiplication takes O ( nq ) time for amatrix in SROS form.

Original: So if the bit order is unchanged, an optimization step can be done in O ( nq )time, and if the data needs to be resorted, then in O ( nq log n ) time.
Revised: So if the bit order is unchanged, an optimization step can be done in O ( nq )time, and if the data needs to be resorted, then in O ( nq log n ) time.

Original: On the largest dataset we tested(House Electric), with n ⇡ 1.3 million and q = 88 , sorting the data and computing P takes about0.96 seconds on a GPU, and then calculating the negative log likelihood takes about another 1.08seconds.
Revised: On the largest dataset we tested(House Electric), with n ⇡ 1.3 million and q = 88 , sorting the data and computing P takes about0.96 seconds on a GPU, and then calculating the negative log likelihood takes about another 1.08seconds.

Original: We show in Appendix B how to compute r w NLL in O ( nq 2 ) time.
Revised: We show in Appendix B how to compute r w NLL in O ( nq 2 ) time.

Original: To optimize the bit order and weight vector at the same time, we represent both with a single parametervector ✓ 2 R q + , with || ✓ || 1 = 1 .
Revised: To optimize the bit order and weight vector at the same time, we represent both with a single parametervector ✓ 2 R q + , with || ✓ || 1 = 1 .

Original: To get the bit order from ✓ , we start with a default bit order and 227 permute the bit order according to a permutation that would sort ✓ in descending order.
Revised: To get the bit order from ✓ , we start with a default bit order and 227 permute the bit order according to a permutation that would sort ✓ in descending order.

Original: To get theweight vector, we sort ✓ in descending order, add a 0 at the end, and compute the differences betweenadjacent elements.
Revised: To get theweight vector, we sort ✓ in descending order, add a 0 at the end, and compute the differences betweenadjacent elements.

Original: When there are ties in the elements of ✓ , the choice of bit order does not affect thenegative log likelihood (or the kernel at all) because the relevant associated weight is 0 .
Revised: When there are ties in the elements of ✓ , the choice of bit order does not affect thenegative log likelihood (or the kernel at all) because the relevant associated weight is 0 .

Original: The negative log likelihood is continuous with respect to ✓ , and when all values of ✓ are unique, it is differentiablewith respect to ✓ .
Revised: The negative log likelihood is continuous with respect to ✓ , and when all values of ✓ are unique, it is differentiablewith respect to ✓ .

Original: Letting ✓ = e  / || e  || 1 , we minimize loss w.r.t.
Revised: Letting ✓ = e  / || e  || 1 , we minimize loss w.r.t.

Original:  using BFGS [7].
Revised:  using BFGS [6].

Original: To calculate the predictive mean at a list of predictive locations X 0 , we ﬁrst multiply y by ( K XX
Revised: To calculate the predictive mean at a list of predictive locations X 0 , we ﬁrst multiply y by ( K XX

Original: + I )  1 , and then we multiply that vector by K XX 0 .
Revised: + I )  1 , and then we multiply that vector by K XX 0 .

Original: We obtain both K XX and K XX 0 in SROS formas follows.
Revised: We obtain both K XX and K XX 0 in SROSform using the following procedure.

Original: Let ˜ X
Revised: Let ˜ X

Original: = X  X 0 be the concatenation of the two tuples, now an ( n + m ) -tuple.
Revised: = X  X 0 be the concatenation of the two tuples,now an ( n + m ) -tuple.

Original: Writing K ˜ X ˜ X
Revised: Writing K ˜ X ˜ X

Original: = L ( ˜ P, ˜ C, ˜ U ) , the arrays on the r.h.s.
Revised: = L ( ˜ P, ˜ C, ˜ U ) , the arrays on the r.h.s.

Original: can be computed in O (( n + m ) q log( n + m )) time.
Revised: can be computed in O (( n + m ) q log( n + m )) time.

Original: Then, with P , C , and U being the ﬁrst n rows of ˜ P , ˜ C , ˜ U , K XX =
Revised: Then, with P , C , and U being the ﬁrst n rows of ˜ P , ˜ C , ˜ U , K XX =

Original: L ( P, C, U ) .
Revised: L ( P, C, U ) .

Original: And letting P 00 and U 00 be the last m rows, K XX 

Revised: And letting P 00 and U 00 be the last m rows, K XX

Original: = L ( P, P 00 , C  U, U 00 ) .
Revised: = L ( P, P 00 , C  U, U 00 ) .

Original: Thus, the predictive mean µ x from Equation1 can be computed at m locations in O (( n + m ) q log( n + m )) time.
Revised: Thus, thepredictive mean µ x from Equation 1 can be computed at m locations in O (( n + m ) q log( n + m ))time.

Original: The predictive covariance matrix, which extends the predictive variance from Equation 2, is calculated⌃ X 

Revised: The predictive covariance matrix, which extends the predictive variance from Equation 2, is calculated⌃ X

Original: = K X 0 X 

Revised: = K X 0 X

Original: + 
Revised: +

Original: I m  K X
Revised: I m  K X

Original: 0 X
Revised: 0 X

Original: ( K XX +  I n )  1 K XX 0 =
Revised: ( K XX +  I n )  1 K XX 0 =

Original: ( K ˜ X ˜ X
Revised: ( K ˜ X ˜ X

Original: + 
Revised: +

Original: I m + n ) /K
Revised: I m + n ) /K

Original: XX , where / denotes 242 the Schur complement.
Revised: XX , where / denotes 243 the Schur complement.

Original: From a property of block matrix inversion, the last m columns of the lastm rows of ( K ˜ X ˜ X
Revised: From a property of block matrix inversion, the last m columns of the lastm rows of ( K ˜ X ˜ X

Original: +  I )  1 equals ((
Revised: +  I )  1 equals ((

Original: K ˜ X ˜ X
Revised: K ˜ X ˜ X

Original: + 
Revised: +

Original: I m + n ) /K
Revised: I m + n ) /K

Original: XX )  1 .
Revised: XX )  1 .

Original: So we get the predictive precisionmatrix in O (( n + m ) q log( n + m )) time by inverting K ˜ X
Revised: So we get the predictive precisionmatrix in O (( n + m ) q log( n + m )) time by inverting K ˜ X

Original: ˜ X
Revised: ˜ X

Original: +  I and taking the bottom right m ⇥ mblock.
Revised: +  I and taking the bottom right m ⇥ mblock.

Original: Then, we get the predictive covariance matrix by inverting that.
Revised: Then, we get the predictive covariance matrix by inverting that.

Original: This takes O ( mq 2 ) time,since it does not have the property of all the columns of U being equal.
Revised: This takes O ( mq 2 ) time,since it does not have the property of all the columns of U being equal.

Original: If we only want the diagonalelements of an SROS matrix (the independent predictive variances in this case), we can simply sumthe rows of C  U  U in O ( mq ) time.
Revised: If we only want the diagonalelements of an SROS matrix (the independent predictive variances in this case), we can simply sumthe rows of C  U  U in O ( mq ) time.

Original: Thus, in total, computing the independent predictive variancesrequires O (( n + m ) q log( n + m ) + mq 2 ) time.
Revised: Thus, in total, computing the independent predictive variancesrequires O (( n + m ) q log( n + m ) + mq 2 ) time.

Original: See Algorithm 3.
Revised: See Algorithm 3.

Original: Algorithm

Original: Require: X 2 B n ⇥ q , y 2 R n , X

Original: [Equation]
Revised: [Equation]

Original:  2 B m ⇥ q , w
Revised: Ensure: µ X 0 and  2 X 0 are the predictive means and variances at X 0 , and nll the training negative log likelihood. 1: ˜ X

Original: [Equation]
Revised: [Equation]

Original: R q

Original: [Equation]
Revised: [Equation]

Original:  2 R + Ensure: µ X 0 and  2 X 0 are the predictive means and variances at X 0 , and nll the training negative log likelihood. 1: ˜ X X  X 0 2: ˜ X " , perm 
Revised: , perm Sort (

Original: Sort ( ˜ X ).
Revised: X )

Original: Therows of X are sorted lexically from leading bit to trailing bit.
Revised: The of X are sorted lexically from leading bit to trailing bit.

Original: O (( n + m ) q log( n + m )) time.
Revised: ( n + m ) q log( n + m ) time.

Original: 3: for j, i 2 [ n + m ] 
Revised: : for i, j n + m ] ⇥

Original: ⇥
Revised: q

Original: [ q ] do˜ P " ji # of unique rows in X " 1: j, 1: i 4: . M 1: j, 1: i is the ﬁrst j rows and i columns of M . O (( n + m ) q ) time.
Revised: do

Original: 5: ˜ P perm( P
Revised: ﬁrst P perm

Original: This “unsorts” the input.
Revised: This “unsorts” the input.

Original: O (( n + m ) q ) time.
Revised: O (( n + m ) q ) time.

Original: 6: P , P˜ P 1: n , ˜ P n + n + m
Revised: U, U

Original: U, U 0 , ˜ U 1 n ⇥ q , 1 m ⇥ q , 1 ( n + m ) ⇥ q  C, ˜ C 1 n w T , 1 n + m w T
Revised: U 1 n ⇥ q , 1 m ⇥ q , 1 ( n + m ) ⇥ q 8: C, ˜ C 1 n w T , 1 n + m w T

Original:  C  1  , U  1 , logdet 
Revised: : C  1  , U  1 , logdet

Original: Invert ( P,   1 C, U ) .
Revised: Invert ( P,   1 C, U ) .

Original: Uses Algorithm 2.
Revised: Uses Algorithm 2.

Original: Speedup to O ( nq ) time because columns of   1 U are identical.
Revised: Speedup to O ( nq ) time because columns of   1 U are identical.

Original:  C  1 , logdet   1 C  1  , logdet  + n log(  ) 11: z
Revised: 10: C  1 , logdet   1 C  1  , logdet  + n log(  ) 11: z

Original: LinTransform ( P, P, U  1 , C  1  U  1 , y )
Revised: LinTransform ( P, P, U  1 , C  1  U  1 , y )

Original: +   1 y .
Revised: +   1 y .

Original: Uses Algorithm 1 to compute the
Revised: Uses Algorithm 1 to compute the

Original: Woodbury vector.
Revised: Woodbury vector.

Original: O ( nq ) time.
Revised: O ( nq ) time.

Original: 12: µ X
Revised: 12: µ X

Original: LinTransform ( P 0 , P, U 0 , C  U, z ) .
Revised: LinTransform ( P 0 , P, U 0 , C  U, z ) .

Original: O (( n + m ) q ) time.
Revised: O (( n + m ) q ) time.

Original: 13: nll ( y > z + logdet + n log(2 ⇡ )) /14:
Revised: 13: nll ( y > z + logdet + n log(2 ⇡ )) /14:

Original: ˜ C prec , ˜
Revised: ˜ C prec , ˜

Original: U prec
Revised: U prec

Original: Invert ( ˜ P,   1 ˜ C,   1 ˜ U ) .
Revised: Invert ( ˜ P,   1 ˜ C,   1 ˜ U ) .

Original: O(( n + m ) q ) time.
Revised: O

Original: 15: C prec , U prec˜ C prec n +1: n + m , ˜ U prec n +1: n + m 16: Ccov , U cov Invert ( P 0 , C prec , U prec ) .
Revised: 15: C prec , U prec 16: C

Original: O ( mq 2 ) time; extra factor of q because columns of U prec are not identical.
Revised: , U cov Invert extra factor of q because columns of U prec are not identical.

Original: 17:  2 X ( 1 m + SumEachRow ( C cov  U cov  U cov )) .
Revised: 17:  2 X

Original: O ( mq ) time.
Revised: O ( mq ) time.

Original: 18: return µ X 0 ,  2 X 0 , nll
Revised: 18: return µ X 0 ,  2 X 0 , nll

Original:  Related Work
Revised:  Related Work

Original: All existing kernels of which we are aware for linear time GP regression on unstructured data involveinducing points (related to the Nyström approximation [24]) or inducing frequencies.
Revised: All existing kernels of which we are aware for O ( n ) time GP regression on unstructured data involveinducing points (related to the Nyström approximation [23]) or inducing frequencies.

Original: For a given setof inducing points Z , for some base kernel k , the inducing point kernel (in its most basic form) is thefollowing, although subtle variants exist: k Z ( x, x 0 )
Revised: For a given setof inducing points Z , for some base kernel k , the inducing point kernel (in its most basic form) is thefollowing, although subtle variants exist: k Z ( x, x 0 )

Original: =
Revised: =

Original: K xZ K  1 ZZ K Zx 0 [18].{+
+}
Revised: K xZ K  1 ZZ K Zx {+
+}

Original: Sparse Gaussian Process Regression (SGPR) involves selecting Z , and then using k Z (or a variant).
Revised: Sparse Gaussian Process Regression (SGPR) involves selecting Z , and then using k Z (or a variant).

Original: Notably, K ZXX =
Revised: Notably, K ZXX =

Original: K XZ K  1 ZZ K ZX is low rank, providing computational efﬁciency.
Revised: K XZ K  1 ZZ K ZX is low rank, providing computational efﬁciency.

Original: The predictivemean and covariance have compact form, with observational noise  : µ x ( Z ) =
Revised: The predictivemean and covariance have compact form, with observational noise  : µ x ( Z ) =

Original: K xZ (  K ZZ +k ZX k XZ )  1 K ZX y and  2 xx 

Revised: K xZ (  K ZZ +k ZX k XZ )  1 K ZX y and  2 xx

Original: ( Z ) =
Revised: ( Z ) =

Original: K xZ ( K ZZ +   1 k ZX k XZ )  1 K Zx 0 .
Revised: K xZ ( K ZZ +   1 k ZX k XZ )  1 K Zx 0 .

Original: Titsias’s [21] sparse variational kernel is also low rank and uses inducing points.
Revised: Titsias’s [19] sparse variational kernel is also low rank and uses inducing points.

Original: The sparse variational
Revised: The sparse variational

Original: GP (SVGP) is constructed as the solution to a variational inference problem.
Revised: GP (SVGP) is constructed as the solution to a variational inference problem.

Original: It depends on inducingpoints Z , data points X , and observed function values y .
Revised: It depends on inducingpoints Z , data points X , and observed function values y .

Original: We have focused on Gaussian processes with0 mean, but the SVGP method uses a nonzero prior mean along with a kernel: m SVGP ( x )
Revised: We have focused on Gaussian processes with0 mean, but the SVGP method uses a nonzero prior mean along with a kernel: m SVGP ( x )

Original: = µ x ( Z )and k SVGP ( x, x 0 )
Revised: = µ x ( Z )and k SVGP ( x, x 0 )

Original: =
Revised: =

Original: k ( x, x 0 )  K xZ K  1 ZZ K Zx 

Revised: k ( x, x 0 )  K xZ K  1 ZZ K Zx

Original: +  2 xx 0 ( Z ) .
Revised: +  2 xx 0 ( Z ) .

Original: Given the dependence on X and y , thisis not a true probability distribution over function space.
Revised: Given the dependence on X and y , thisis not a true probability distribution over function space.

Original: The variational problem underlying thiskernel also provides guidance in how to select the inducing points Z .
Revised: The variational problem underlying thiskernel also provides guidance in how to select the inducing points Z .

Original: For further discussion of thekernel underlying the SVGP method, see Wild et al.
Revised: For further discussion of thekernel underlying the SVGP method, see Wild et al.

Original: An inducing point kernel with z inducing points produces a z -dimensional reproducing kernel Hilbertspace (RKHS).
Revised: An inducing point kernel with z inducing points produces a z -dimensional reproducing kernel Hilbertspace (RKHS).

Original: The dimensionality of the RKHS relates to the expressivity of the kernel.
Revised: The dimensionality of the RKHS relates to the expressivity of the kernel.

Original: Whereas aninducing point method buys a z -dimensional RKHS for the price of O ( z 2 n ) time and O ( zn ) space,the binary tree kernel produces a 2 q -dimensional RKHS in O ( qn ) time and space—an exponentialimprovement.
Revised: Whereas aninducing point method buys a z -dimensional RKHS for the price of O ( z 2 n ) time and O ( zn ) space,the binary tree kernel produces a 2 q -dimensional RKHS in O ( qn ) time and space—an exponentialimprovement.

Original: (Observe that we can ﬁnd 2 q linearly independent functions of the form k ( · , x ) —onefor each of the 2 q leaves x might belong to.)
Revised: (Observe that we can ﬁnd 2 q linearly independent functions of the form k ( · , x ) —onefor each of the 2 q leaves x might belong to.)

Original: Wilson and Nickisch [26] develop a method for speedingup inducing point methods signiﬁcantly, especially in low-dimensional settings.
Revised: Wilson and Nickisch [25] develop a method for speedingup inducing point methods signiﬁcantly, especially in low-dimensional settings.

Original: Lázaro-Gredilla et al.
Revised: Lázaro-Gredilla et al.

Original: [13] propose an inducing frequencies kernel: given a set of m inducing vectorss i , k ( x, x 0 )
Revised: [11] propose an inducing frequencies kernel: given a set of m inducing vectorss i , k ( x, x 0 )

Original: = 1 /m
Revised: = 1 /m

Original: P mi =1 cos(2 ⇡ s > i ( x  x 0 )) .
Revised: P mi =1 cos(2 ⇡ s > i ( x  x 0 )) .

Original: Dutordoir et al.
Revised: Dutordoir et al.

Original: [5] propose an inducing frequencieskernel for low dimensional data, in which k ( x, x 0 ) is a special function of x > x 0 .
Revised: [5] propose an inducing frequencieskernel for low dimensional data, in which k ( x, x 0 ) is a special function of x > x 0 .

Original: On one-dimensional data, ﬁltering/smoothing methods perform Bayesian inference over functionsin O ( n ) time [11, 9].
Revised: On one-dimensional data, ﬁltering/smoothing methods perform Bayesian inference over functionsin O ( n ) time [10, 8].

Original: A few non- O ( n ) methods bear mentioning.
Revised: A few non- O ( n ) methods bear mentioning.

Original: We are not the ﬁrst to considera kernel over points on the leaves of a tree [15, 14] or on the leaves of multiple trees [6], but theirmethods take O ( n 3 ) time.
Revised: We are not the ﬁrst to consider akernel over points on the leaves of a tree [13, 12], but their methods take O ( n 3 ) time.

Original: On certain kinds of structured data, Toeplitz solvers achieve O ( n 2 ) timecomplexity [27].
Revised: On certain kindsof structured data, Toeplitz solvers achieve O ( n 2 ) time complexity [26].

Original: Cutajar et al.’s
Revised: Cutajar et al.’s

Original: [2], Wang et al.’s
Revised: [2], Wanget al.’s

Original: [22], and others’ use of a conjugate gradients solverto replace inversion/factorization has unclear time complexity between O ( n 2 ) and O ( n 3 ) , dependingon the kernel matrix spectrum.
Revised: [21], and others’ use of a conjugate gradients solver to replace inversion/factorization hasunclear time complexity between O ( n 2 ) and O ( n 3 ) , depending on the kernel matrix spectrum.

Original:  Experiments
Revised:  Experiments

Original: In Table 5, we compare our binary tree kernel and a binary tree ensemble (see Appendix C for detailson the ensemble) against three baseline methods: exact GP regression using a Matérn kernel, sparse
Revised: In Table 1, we compare our binary tree kernel and a binary tree ensemble (see Appendix C for detailson the ensemble) against three baseline methods: exact GP regression using a Matérn kernel, sparse

Original: Gaussian process regression (SGPR)
Revised: Gaussian process regression (SGPR)

Original: [20], and a stochastic variational Gaussian process (SVGP)[10].
Revised: [18], and a stochastic variational Gaussian process (SVGP)[9].

Original: We evaluate our method on the same open-access UCI datasets [3] as Wang et al.
Revised: We evaluate our method on the same open-access UCI datasets [3] as Wang et al.

Original: [22], usingtheir same training, validation, and test partitions, and we compare against the baseline results theyreport.
Revised: [21], usingtheir same training, validation, and test partitions, and we compare against the baseline results theyreport.

Original: For the binary tree (BT) kernels, we use p = min(8 , b 150 /d c + 1) , and recall q = pd .
Revised: For the binary tree (BT) kernels, we use p = min(8 , b 150 /d c + 1) , and recall q = pd .

Original: We set = 1 /n .
Revised: We set = 1 /n .

Original: We train the bit order and weights to minimize training NLL.
Revised: We train the bit order and weights to minimize training NLL.

Original: For the binary tree ensemble(BTE), we use 20 kernels.
Revised: For the binary tree ensemble(BTE), we use 20 kernels.

Original: For the Matérn kernel, we use Blackbox Matrix-Matrix multiplication(BBMM)
Revised: For the Matérn kernel, we use Blackbox Matrix-Matrix multiplication(BBMM)

Original: [8], which uses the conjugate gradients method to calculate ( K XX +  I )  1 .
Revised: [7], which uses the conjugate gradients method to calculate ( K XX +  I )  1 .

Original: SGPR uses512 data points and SVGP uses 1,024 inducing points.
Revised: SGPR uses512 data points and SVGP uses 1,024 inducing points.

Original: We report the mean and two standard errorsacross 3 replications with different dataset splits.
Revised: We report the mean and two standard errorsacross 3 replications with different dataset splits.

Original: For further experimental details, see Appendix C.
Revised: For further experimental details, see Appendix C.

Original: BTE achieves the best NLL on 6/12 datasets, and best RMSE on 5/12 datasets (including some ties).
Revised: BTE achieves the best NLL on 6/12 datasets, and best RMSE on 5/12 datasets (including some ties).

Original: Out of the 4 largest datasets, BT/BTE is fastest on 3.
Revised: Out of the 4 largest datasets, BT/BTE is fastest on 3.

Original: The run times are plotted in Figure 5.
Revised: The run times are plotted in Figure 5.

Original: The codeis included in the supplementary material, and we will open source our code upon publication.
Revised: The codeis included in the supplementary material, and we will open source our code upon publication.

Original: BT performs noticeably worse than BTE for test NLL on CTSlice due to over-ﬁtting.
Revised: BT performs noticeably worse than BTE for test NLL on CTSlice due to over-ﬁtting.

Original: There areenough degrees of freedom when optimizing the bit order ( d = 385 ) that BT kernel can over-ﬁt tothe training data.
Revised: There areenough degrees of freedom when optimizing the bit order ( d = 385 ) that BT kernel can over-ﬁt tothe training data.

Original: The ensemble over multiple bit orders is much more robust.
Revised: The ensemble over multiple bit orders is much more robust.

Original: [Table]
Revised: [Table]

Original:  Discussion
Revised:  Discussion

Original: 5: Run times given dataset size.

Original: For BT, the trendline is calculated controlling for log( q ) with afﬁne regression, and then setting q = 150 .
Revised: controlling for log( q ) with afﬁne regression, and then setting q = 150 .

Original: The slope w.r.t.
Revised: The slope w.r.t.

Original: log( q ) is 2 .
Revised: log( q ) is 2 .

Original: 82 ± 1 . 06 .
Revised: 82 ± 1 . 06 .

Original: Theoretically, all slopes are too low except for that of SVGP, presumably because of overhead in the small-data regime.
Revised: Theoretically, all slopes are too low except for that of SVGP, presumably because of overhead in the small-data regime.

Original: We have proven that the binary tree kernel GP is scalable.
Revised: We have proven that the binary treekernel GP is scalable.

Original: Our empirical resultssuggest that it often outpredicts not justother scalable methods, but even the popular Matérn GP.
Revised: Our empiricalresults suggest that it often outpredictsnot just other scalable methods, buteven the popular Matérn GP.

Original: If the results in this paperreplicate in other domains, it could obviatewide usage of classic GP kernels like the Matérn kernel, as well as inducing pointkernels.
Revised: If the results in this paper replicate in other domains, it could obviate wide usage ofclassic GP kernels like the Matérn kernel, as well as inducing point kernels.

Original: Sometimes, our kernel fails to capture patterns in the data; some functions’values simply do not covary this way.
Revised: Sometimes, our kernel fails to capturepatterns in the data; some functions’values simply do not covary this way.

Original: Butother kernels we tested seemed to fail likethat even more.
Revised: But other kernels we tested seemed tofail like that even more.

Original: The SROS representation of linear operators, which we developed for this kernel, isconceptually general, and could potentiallybe useful in other domains where linear algebra is applied.
Revised: The SROS representation of linear operators, which we developed for thiskernel, is conceptually general, andcould potentially be useful in otherdomains where linear algebra is applied.

Original: Low-rank approximationsare widely used to speed up linear transformations, and SROS representations mightbe an alternative in many settings.
Revised: Low-rank approximations arewidely used to speed up linear transformations, and SROS representations might be an alternative in many settings.

Original: Our contributions to linear algebra and kernel design may signiﬁcantly increase the size of data setson which GPs can do state-of-the-art modelling. 
Revised: Our contributions to linear algebra and kernel design may signiﬁcantly increase the size of data setson which GPs can do state-of-the-art modelling. Checklist

Revised: The checklist follows the references.

Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Revised: For example:

Revised: • Did you include the license to the code and datasets?

Revised: [Yes] See Section ?? .•

Revised: Did you include the license to the code and datasets?

Revised: [No] The code and the data areproprietary.•

Revised: Did you include the license to the code and datasets?

Revised: [N/A]

Revised: Please do not modify the questions and only use the provided macros for your answers.

Revised: Note that the

Revised: Checklist section does not count towards the page limit.

Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Revised: For all authors...

Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Revised: [Yes](b) Did you describe the limitations of your work?

Revised: [Yes] In the introduction.(c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] Yes inSection D.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Revised: [Yes]2.

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] In Section 3(b) Did you include complete proofs of all theoretical results?

Revised: [Yes] In Sections 3 and 4and Appendix A3.

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] In thesupplemental material.(b) Did you specify all the training details (e.g., data splits, hyperparameters,

Revised: how they

Revised: were chosen)?

Revised: [Yes]

Revised: In Section 7 and Appendix C.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Revised: ments multiple times)?

Revised: [Yes] See Table 1.

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See Table 1.

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] The UCI datasets in

Revised: Section 7.

Revised: (b) Did you mention the license of the assets?

Revised: [Yes]

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [N/A]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Revised: information or offensive content?

Revised: [N/A]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: applicable?

Revised: [N/A]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: spent on participant compensation?

Revised: [N/A]