Original: Log-Linear-Time Gaussian Processes Using Binary Tree Kernels
Revised: Log-Linear-Time Gaussian Processes Using Binary Tree Kernels

Original: Anonymous Author(s)

Original: Afﬁliation email

Original: Abstract
Revised: Abstract

Original: Gaussian processes (GPs) produce good probabilistic models of functions, butmost GP kernels require O (( n + m ) n 2 ) time, where n is the number of data pointsand m the number of predictive locations.
Revised: Gaussian processes (GPs) produce good probabilistic models of functions, but most GP kernels require O (( n + m ) n 2 ) time, where n is the number of data points and m the number of predictive locations.

Original: We present a new kernel that allows for
Revised: We present a new kernel that allows for

Original: Gaussian process regression in O (( n + m ) log( n + m )) time.
Revised: Gaussian process regression in O (( n + m ) log( n + m )) time.

Original: Our “binary tree”kernel places all data points on the leaves of a binary tree, with the kernel dependingonly on the depth of the deepest common ancestor.
Revised: Our “binary tree” kernel places all data points on the leaves of a binary tree, with the kernel depending only on the depth of the deepest common ancestor.

Original: We can store the resulting kernelmatrix in O ( n ) space in O ( n log n ) time, as a sum of sparse rank-one matrices,and approximately invert the kernel matrix in O ( n ) time.
Revised: We can store the resulting kernel matrix in O ( n ) space in O ( n log n ) time, as a sum of sparse rank-one matrices, and approximately invert the kernel matrix in O ( n ) time.

Original: Sparse GP methods alsooffer linear run time, but they predict less well than higher dimensional kernels.
Revised: Sparse GP methods also offer linear run time, but they predict less well than higher dimensional kernels.

Original: Ona classic suite of regression tasks, we compare our kernel against Matérn, sparse,and sparse variational kernels.
Revised: On a classic suite of regression tasks, we compare our kernel against Matérn, sparse, and sparse variational kernels.

Original: The binary tree GP assigns the highest likelihood tothe test data on a plurality of datasets, usually achieves lower mean squared errorthan the sparse methods, and often ties or beats the Matérn GP.
Revised: The binary tree GP assigns the highest likelihood to the test data on a plurality of datasets, usually achieves lower mean squared error than the sparse methods, and often ties or beats the Matérn GP.

Original: On large datasets,the binary tree GP is fastest, and much faster than a Matérn GP.
Revised: On large datasets, the binary tree GP is fastest, and much faster than a Matérn GP.

Original:  Introduction
Revised:  Introduction

Original: Gaussian processes (GPs) can be used to perform regression with high-quality uncertainty estimates,but they are slow.
Revised: Gaussian processes (GPs) can be used to perform regression with high-quality uncertainty estimates, but they are slow.

Original: Naïvely, GP regression requires O ( n 3 + n 2 m ) computation time and O ( n 2 )computation space when predicting at m locations given n data points [25].
Revised: Naïvely, GP regression requires O ( n 3 + n 2 m ) computation time and O ( n 2 ) computation space when predicting at m locations given n data points [28].

Original: A kernel matrix of sizen ⇥ n must be inverted (or Cholesky decomposed) and then m matrix-vector multiplications mustbe done with that inverse matrix (or m linear solves with the Cholesky factors).
Revised: A kernel matrix of size n × n must be inverted (or Cholesky decomposed), and then m matrix-vector multiplications must be done with that inverse matrix (or m linear solves with the Cholesky factors).

Original: A few methods thatwe will discuss later achieve O ( n 2 m ) time complexity [22, 27]
Revised: A few methods that we will discuss later achieve O ( n 2 m ) time complexity [25, 30].

Original: With special kernels, GP regression can be faster and use less space.
Revised: With special kernels, GP regression can be faster and use less space.

Original: Inducing point methods, usingz inducing points, allow regression to be done in O ( z 2 ( n + m )) time and in O ( z 2 + zn ) space[18, 19, 20, 10].
Revised: Inducing point methods, using z inducing points, allow regression to be done in O ( z 2 ( n + m )) time and in O ( z 2 + zn ) space [21, 22, 23, 13].

Original: We will discuss the details of these inducing point kernels later, but they are kernelsin their own right, not just approximations to other kernels.
Revised: We will discuss the details of these inducing point kernels later, but they are kernels in their own right, not just approximations to other kernels.

Original: Unfortunately,if we have observed thefunction value at a location x , and have to predict the function value at that location x , theseinducingpoint kernels do not automatically predict the observed value at x with very little uncertainty (unlessx is one ofour inducing points). This is the easiest possible prediction problem—simple recall. Inducing point methods fail it.
Revised: Unfortunately, these kernels are low dimensional (having a z -dimensional Hilbert space), which limits the expressivity of the GP model.

Original: We present a new kernel, the binary tree kernel , that also allows for GP regression in O ( n + m )space and O (( n + m ) log( n + m )) time (both model ﬁtting and prediction).
Revised: We present a new kernel, the binary tree kernel , that also allows for GP regression in O ( n + m ) space and O (( n + m ) log( n + m )) time (both model fitting and prediction).

Original: The time and spacecomplexity of our method is also linear in the depth of the binary tree, which is naïvely linear in thedimension of the data, although in practice we can increase the depth sublinearly.
Revised: The time and space complexity of our method is also linear in the depth of the binary tree, which is naïvely linear in the dimension of the data, although in practice we can increase the depth sublinearly.

Original: Training somekernel parameters takes time quadratic in the depth of the tree.
Revised: Training some kernel parameters takes time quadratic in the depth of the tree.

Original: Unlike inducing point kernels, withthe binary tree kernel, if x is both a training data point and a predictive location, it will produce anaccurate and appropriately conﬁdent prediction. The fact that inducing point kernels fail this sanitycheck should make us suspicious of their performance in general
Revised: The dimensionality of the binary tree kernel is exponential in the depth of the tree, making it much more expressive than an inducing points kernel. Whereas for an inducing points kernel, the runtime is quadratic in the dimension of the

Original: In this example, k ( x 1 , x 1 ) = 1 , k ( x 1 , x 2 ) = 0 , k ( x 1 , x 3 ) = 0 . 8 , and k ( x 1 , x 4 ) = 0 .
Revised: Hilbert space, for the binary tree kernel, it is only logarithmic—an exponential speedup.

Original: A simple depiction of our kernel is shown in Figure 1, which we will deﬁne precisely in Section 3.
Revised: A simple depiction of our kernel is shown in Figure 1, which we will define precisely in Section 3.

Original: First, we create a procedure for placingall data points on the leaves of a binary tree.
Revised: First, we create a procedure for placing all data points on the leaves of a binary tree.

Original: Given the binary tree, the kernel between twopoints depends only on the depth of the deepestcommon ancestor.
Revised: Given the binary tree, the kernel between two points depends only on the depth of the deepest common ancestor.

Original: Because very different treestructures are possible for the data, we can easilyform an ensemble of diverse GP regression models.
Revised: Because very different tree structures are possible for the data, we can easily form an ensemble of diverse GP regression models.

Original: Figure 2 depicts a schematic sample from abinary tree kernel.
Revised: Figure 2 depicts a schematic sample from a binary tree kernel.

Original: Note how the posterior meanis piecewise ﬂat, but the pieces can be small.
Revised: Note how the posterior mean is piecewise flat, but the pieces can be small.

Original:  On a standard suite of benchmark regression tasks [22], we show that our kernel usually achievesbetter negative log likelihood (NLL) than state-of-the-art sparse methods and conjugate-gradientbased “exact” methods, at lower computational cost in the big-data regime.
Revised: In this example, k ( x 1 , x 1 ) = 1 , k ( x 1 , x 2 ) = 0 , k ( x 1 , x 3 ) = 0 . 8 , and k ( x 1 , x 4 ) = 0 . On a standard suite of benchmark regression tasks [25], we show that our kernel usually achieves better negative log likelihood (NLL) than state-of-the-art sparse methods and conjugate-gradient-based “exact” methods, at lower computational cost in the big-data regime.

Original: There are not many limitations to using our kernel.
Revised: There are not many limitations to using our kernel.

Original: The main limitation is that other kernels sometimescapture the relationships in the data better.
Revised: The main limitation is that other kernels sometimes capture the relationships in the data better.

Original: We do not have a good procedure for understandingwhen data has more Matérn character or more binary tree character (except through running bothand comparing training NLL).
Revised: We do not have a good procedure for understanding when data has more Matérn character or more binary tree character (except through running both and comparing training NLL).

Original: But given that the binary tree kernel usually outperforms the Matérn,we’ll tentatively say the best ﬁrst guess is that a new dataset has more binary tree character.
Revised: But given that the binary tree kernel usually outperforms the Matérn, we’ll tentatively say the best first guess is that a new dataset has more binary tree character.

Original: Oneconcrete limitation for some applications, like Bayesian Optimization, is that the posterior mean ispiecewise-ﬂat, so gradient-based heuristics for ﬁnding extrema would not work.
Revised: One concrete limitation for some applications, like Bayesian Optimization, is that the posterior mean is piecewise-flat, so gradient-based heuristics for finding extrema would not work.

Original: In contexts where a piecewise-ﬂat posterior mean is suitable, we struggle to see when one wouldprefer a sparse or sparse variational GP to a binary tree kernel.
Revised: In contexts where a piecewise-flat posterior mean is suitable, we struggle to see when one would prefer a sparse or sparse variational GP to a binary tree kernel.

Original: The most thorough approach wouldbe to run both and see which has a better training NLL, but if you had to pick one, the binary tree
Revised: The most thorough approach would be to run both and see which has a better training NLL, but if you had to pick one, the binary tree

Original: GP seems to be better performing and comparably fast.
Revised: GP seems to be better performing and comparably fast.

Original: If minimizing mean-squared error is theobjective, the Matérn kernel seems to do slightly better than the binary tree.
Revised: If minimizing mean-squared error is the objective, the Matérn kernel seems to do slightly better than the binary tree.

Original: If the dataset is small,and one needs a very fast prediction, a Matérn kernel may be the best option.
Revised: If the dataset is small, and one needs a very fast prediction, a Matérn kernel may be the best option.

Original: But otherwise, if onecares about well-calibrated predictions, these initial results we present tentatively suggest using abinary tree kernel over the widely-used Matérn kernel.
Revised: But otherwise, if one cares about well-calibrated predictions, these initial results we present tentatively suggest using a binary tree kernel over the widely-used Matérn kernel.

Original: The log-linear time and linear space complexity of the binary tree GP, with performance exceeding a“normal” kernel, could profoundly expand the viability of GP regression to larger datasets.
Revised: The log-linear time and linear space complexity of the binary tree GP, with performance exceeding a “normal” kernel, could profoundly expand the viability of GP regression to larger datasets.

Original:  Preliminaries
Revised:  Preliminaries

Original: Our problem setting is regression.
Revised: Our problem setting is regression.

Original: Given a function f : X ! R , for some arbitrary set X , we would71 like to predict f ( x ) for various x 2 X .
Revised: Given a function f : X → R , for some arbitrary set X , we would like to predict f ( x ) for various x ∈ X .

Original: What we have are observations of f ( x ) for various (other)72 x 2 X .
Revised: What we have are observations of f ( x ) for various (other) x ∈ X .

Original: Let X 2 X n be an n -tuple of elements of X , and let y 2 R n be an n -tuple of real numbers,such that y i ⇠ f
Revised: Let X ∈ X n be an n -tuple of elements of X , and let y ∈ R n be an n -tuple of real numbers, such that y i ∼ f

Original: ( X i )
Revised: ( X i )

Original: + N (0 ,  ) , for  2 R  0 .
Revised: + N (0 , λ ) , for λ ∈ R ≥ 0 .

Original: X and y comprise our training data.
Revised: X and y comprise our training data.

Original: With an m -tuple of test locations X 0 2 X m , let y [-0 2 R m , with y 
-]
Revised: With an m -tuple of test locations X ′ ∈ X m , let y [-0 2 R m , with y 
-]′ ∈

Original:  i =f
Revised: R m , with y ′ i =

Original:  ( X 0 i ) .
Revised: f ( X ′ i ) .

Original: y 0 is the ground truthfor the target locations.
Revised: y ′ is the ground truth for the target locations.

Original: Given training data, we would like to produce a distribution over R for eachtarget location X 0 i , such that it assigns high marginal probability to the unknown y 0 i .
Revised: Given training data, we would like to produce a distribution over R for each target location X ′ i , such that it assigns high marginal probability to the unknown y ′ i .

Original: Alternatively, wesometimes would like to produce point estimates ˆ y 0 i in order to minimize the squared error (ˆ y [-
-]
Revised: Alternatively, we sometimes would like to produce point estimates ˆ y ′ i in order to minimize the squared error (ˆ y [-
-]′

Original: i  y 0 i ) 2 .
Revised: i − y ′ i ) 2 .

Original: A GP prior over functions is deﬁned by a mean function m : X ! R , and a kernel k : X ⇥ X!
Revised: A GP prior over functions is defined by a mean function m : X → R , and a kernel k : X × X

Original:  R .
Revised: → R .

Original: The expected function value at a point x is deﬁned to be m ( x ) , and the covariance of the function values at two points x 1 and x 2 is deﬁned to be k ( x 1 , x 2 ) .
Revised: The expected function value at a point x is defined to be m ( x ) , and the covariance of the function values at two points x 1 and x 2 is defined to be k ( x 1 , x 2 ) .

Original: Let K XX 2 R n ⇥ n be the matrix of kernelvalues ( K XX ) ij =
Revised: Let K XX ∈ R n × n be the matrix of kernel values ( K XX ) ij =

Original: k
Revised: k

Original: ( X i , X j ) , and let m X 2 R n be the vector of mean values ( m X ) i
Revised: ( X i , X j ) , and let m X ∈ R n be the vector of mean values ( m X ) i

Original: = m ( X i ) .
Revised: = m ( X i ) .

Original: For a GP to be well-deﬁned, the kernel must be such that K XX is positive semideﬁnite for any
Revised: For a GP to be well-defined, the kernel must be such that K XX is positive semidefinite for any

Original: X 2 X n .
Revised: X ∈ X n .

Original: For a point x 2 X , Let K Xx 2 R n be the vector of kernel values: ( K Xx )
Revised: For a point x ∈ X , Let K Xx ∈ R n be the vector of kernel values: ( K Xx )

Original: i =
Revised: i =

Original: k ( X i , x ) ,and let K xX =
Revised: k ( X i , x ) , and let K xX =

Original: K > Xx .
Revised: K ⊤ Xx .

Original: Let   0 be the variance of observation noise.
Revised: Let λ ≥ 0 be the variance of observation noise.

Original: Let µ x and  2 x be the meanand variance of our posterior predictive distribution at x .
Revised: Let µ x and σ 2 x be the mean and variance of our posterior predictive distribution at x .

Original: Then, with K  inv XX =
Revised: Then, with K λ inv XX =

Original: ( K XX +  I )  1 , 
Revised: ( K XX + λI ) − 1 , µ x := ( y − m X ) ⊤ K λ inv XX K Xx + m

Original: [Equation]
Revised: [Equation]

Original: See Williams and Rasmussen [25] for a derivation.
Revised: See Williams and Rasmussen [28] for a derivation.

Original: We compute Equations 1 and 2 for all x 2 X 0 .
Revised: We compute Equations 1 and 2 for all x ∈ X ′ .

Original:  Binary tree kernel
Revised:  Binary tree kernel

Original: We now introduce the binary tree kernel.
Revised: We now introduce the binary tree kernel.

Original: First, we encode our data points as binary strings.
Revised: First, we encode our data points as binary strings.

Original: So wehave X = B q , where B = { 0 , 1 } , and q 2 N .
Revised: So we have X = B q , where B = { 0 , 1 } , and q ∈ N .

Original: If X = R d , we must map R d 7! B q .
Revised: If X = R d , we must map R d (cid:55)→ B q .

Original: First, we rescale all points (training points and test points) to liewithin the box [0 , 1] d .
Revised: First, we rescale all points (training points and test points) to lie within the box [0 , 1] d .

Original: (If we have a stream of test points, and one lands outside of the box [0 , 1] d , wecan either set K xX to 0 for that point or we rescale and retrain in O ( n log n ) time.)
Revised: (If we have a stream of test points, and one lands outside of the box [0 , 1] d , we can either set K xX to 0 for that point or we rescale and retrain in O ( n log n ) time.)

Original: Then, for eachx 2 [0 , 1] d , for each dimension, we take the binary expansion up to some precision p , and for thosed ⇥ p bits, we permute them using some ﬁxed permutation.
Revised: Then, for each x ∈ [0 , 1] d , for each dimension, we take the binary expansion up to some precision p , and for those d × p bits, we permute them using some fixed permutation.

Original: We call this permutation the bit order, andit is the same for all x 2 [0 , 1] d .
Revised: We call this permutation the bit order, and it is the same for all x ∈ [0 , 1] d .

Original: Note that now q = dp .
Revised: Note that now q = dp .

Original: See Figure 3 for an example.
Revised: See Figure 3 for an example.

Original: We optimizethe bit order during training, and we can also form an ensemble of GPs using different bit orders.
Revised: We optimize the bit order during training, and we can also form an ensemble of GPs using different bit orders.

Original: For x 2 B q , let x  i be the ﬁrst i bits of x .
Revised: For x ∈ B q , let x ≤ i be the first i bits of x .

Original: [[ expression ]] evaluates to1, if expression is true, otherwise 0.
Revised: [[ expression ]] evaluates to 1, if expression is true, otherwise 0.

Original: We now deﬁne the kernel:
Revised: We now define the kernel:

Original: Deﬁnition 1 (Binary Tree Kernel) .
Revised: Definition 1 (Binary Tree Kernel) .

Original: Given a weight vector w 2 R q ,with w ⌫ 0 and || w || 1 = 1 ,
Revised: Given a weight vector w ∈ R q , with w ⪰ 0 and || w || 1 = 1 ,

Original: [Equation]
Revised: [Equation]

Original: ii So the more leading bits shared by x 1 and x 2 , the larger the covariance between the function values.
Revised: So the more leading bits shared by x 1 and x 2 , the larger the covariance between the function values.

Original: Consider, for example, points x 1and x 4 from Figure 1, where x 1 is ( left, left, right ) , and x 4 is ( left, right, right ) ; they share only theﬁrst leading “bit”.
Revised: Consider, for example, points xand x 4 from Figure 1, where x 1 is ( left, left, right ) , and x 4 is ( left, right, right ) ; they share only the first leading “bit”.

Original: We train the weight vector w to maximize the likelihood of the training data.
Revised: We train the weight vector w to maximize the likelihood of the training data.

Original: Proposition 1 (Positive Semideﬁniteness) .
Revised: Proposition 1 (Positive Semidefiniteness) .

Original: For X 2 X n , for k = k w , K XX ⌫ 0 .
Revised: For X ∈ X n , for k = k w , K XX ⪰ 0 .

Original: Proof.
Revised: Proof.

Original: Let s S qi=
Revised: Let s ∈ (cid:83) qi

Original: 1 B i be a binary string, and let | s | be the length of s .
Revised: =1 B i be a binary string, and let | s | be the length of s .

Original: Let X [ s ] 2 R n with( X [ s ] ) 
Revised: Let X [ s ] ∈ R n with ( X [ s ] ) j = (cid:104)(cid:104)

Original: j = hh X  | s | j = s ii .
Revised: X ≤| s | j = s (cid:105)(cid:105) .

Original: X [ s ] X > [ s ] is clearly positive semideﬁnite.
Revised: X [ s ] X ⊤ [ s ] is clearly positive semidefinite.

Original: Finally, 
Revised: Finally, K XX = (cid:80) qi

Original: [Equation]
Revised: =1 (cid:80) s ∈ B i w i X

Original: (3) and recall w i  0 , so K XX ⌫ 0 .
Revised: [ s ] X ⊤ [ s ] , and recall w i ≥ 0 , so K XX ⪰ 0 .

Original:  Sparse rank one sum representation
Revised:  Sparse rank one sum representation

Original: In order to do GP regression in O ( n ) space and O ( n log n ) time, we develop a “Sparse Rank One
Revised: In order to do GP regression in O ( n ) space and O ( n log n ) time, we develop a “Sparse Rank One

Original: Sum” representation of linear operators (SROS).
Revised: Sum” representation of linear operators (SROS).

Revised: This was developed separately from the very similar

Original:  In SROS form, linear transformation of a vector canbe done in O ( n ) time instead of O ( n 2 ) .
Revised: Hierarchical matrices [1], which we discuss below. In SROS form, linear transformation of a vector can be done in O ( n ) time instead of O ( n 2 ) .

Original: We will store our kernel matrix and inverse kernel matrix in SROS form.
Revised: We will store our kernel matrix and inverse kernel matrix in SROS form.

Original: The proof of Proposition 1 exempliﬁes representing a matrix as the sum of sparse rankone matrices.
Revised: The proof of Proposition 1 exemplifies representing a matrix as the sum of sparse rank one matrices.

Original: Note that each X [ s ] is sparse—if q is large, most X [ s ] ’s are the zero vector.
Revised: Note that each X [ s ] is sparse—if q is large, most X [ s ] ’s are the zero vector.

Original: We now show how to interpret an SROS representation of an n ⇥ n matrix.
Revised: We now show how to interpret an SROS representation of an n × n matrix.

Original: Let [ n ] = { 1 , 2 , ..., n } .
Revised: Let [ n ] = { 1 , 2 , ..., n } .

Original: For r 2 N , let L : [ r ] n ⇥ [ r ] n ⇥ R n ⇥ R n ! R n ⇥ n construct a linear operator from four vectors.
Revised: For r ∈ N , let L : [ r ] n × [ r ] n × R n × R n → R n × n construct a linear operator from four vectors.

Original: Deﬁnition 2 (Linear Operator from Simple SROS Representation) .
Revised: Definition 2 (Linear Operator from Simple SROS Representation) .

Original: Let p, p 0 2 [ r ] n , and let u, u [-0 
-]
Revised: Let p, p ′ ∈ [ r ] n , and let u, u [-0 
-]′ ∈

Original: R n .
Revised: R n .

Original: For l 2 [ r ] , let u p = l 2 R n be the vector where u p = l j = u j [[ p j = l ]] , likewise for u 0 and p 0 .
Revised: For l ∈ [ r ] , let u p = l ∈ R n be the vector where u p = l j = u j [[ p j = l ]] , likewise for u ′ and p ′ .

Original: Then:
Revised: Then:

Original: L ( p, p 0 , u, u 0 )7! r
Revised: L ( p, p ′ , u, u ′ )

Original: X l = ( u p = l )(( u 0 ) p 0 = l ) > .
Revised: (cid:55)→ (cid:80) mi =1 u p = i ( u ′ ) p ′ = i ⊤ .

Original: We depict Deﬁnition 2 in Figure 4.
Revised: We depict Definition 2 in Figure 4.

Original: p and p 0 represent partitions over n elements: all elements with the same integervalue in the vector p belong to the same partition.
Revised: p and p ′ represent partitions over n elements: all elements with the same integer value in the vector p belong to the same partition.

Original: Note thatr , the number of parts in the partition, need not exceed n , thenumber of elements being partitioned.
Revised: Note that r , the number of parts in the partition, need not exceed n , the number of elements being partitioned.

Original: If p = p 0 (which isalmost always the case for us) and the elements of p , u , and u 0were shufﬂed so that all elements in the same partition werenext to each other, then L ( p, p 0 , u, u 0 ) would be block diagonal.
Revised: If p = p ′ (which is almost always the case for us) and the elements of p , u , and u ′ were shuffled so that all elements in the same partition were next to each other, then L ( p, p ′ , u, u ′ ) would be block diagonal.

Original: Note that L ( p, p 0 , u, u 0 ) is not necessarily low rank.
Revised: Note that L ( p, p ′ , u, u ′ ) is not necessarily low rank.

Original: Ifp is the ﬁnest possible partition, and p = p 0 , L ( p, p 0 , u, u 0 ) isdiagonal.
Revised: If p is the finest possible partition, and p = p ′ , L ( p, p ′ , u, u ′ ) is diagonal.

Original: SROS matrices can be thought of as a generalizationof two types of matrix that are famously amenable to fast computation: rank one matrices (all points in the same partition)and diagonal matrices (each point in its own partition).
Revised: SROS matrices can be thought of as a generalization of two types of matrix that are famously amenable to fast computation: rank one matrices (all points in the same partition) and diagonal matrices (each point in its own partition).

Original: We now extend the deﬁnition of L to allow for multiple p , p 0 ,u , and u 0 vectors.
Revised: We now extend the definition of L to allow for multiple p , p ′ , u , and u ′ vectors.

Original: Deﬁnition 3 (Linear Operator from SROS Representation) .
Revised: Definition 3 (Linear Operator from SROS Representation) .

Original: Let L : [ r ] n ⇥ q ⇥
Revised: Let L : [ r ] n × q ×

Original: [ r ] n ⇥ q ⇥ R n ⇥ q ⇥ R n ⇥ q ! R n ⇥ n .
Revised: [ r ] n × q × R n × q × R n × q → R n × n .

Original: Let P, P 0 2[ r ] n ⇥ q , and let U, U 0 2 R n ⇥ q .
Revised: Let P, P ′ ∈ [ r ] n × q , and let U, U ′ ∈ R n × q .

Original: Let P : ,i , U : ,i , etc.
Revised: Let P : ,i , U : ,i , etc.

Original: be the i th columns of the respective arrays.
Revised: be the i th columns of the respective arrays.

Original: Then:
Revised: Then:

Original: L ( P, P 0 , U, U 0 )7!
Revised: L ( P, P ′ , U, U ′ )

Original: P qi=
Revised: (cid:55)→ (cid:80) qi

Original: 1 L ( P : ,i , P 0 : ,i , U : ,i , U 0 : ,i ) .
Revised: =1 L ( P : ,i , P ′ : ,i , U : ,i , U ′ : ,i ) .

Original: Algorithm 1 performs linear transformation of a vector using SROS representation in O ( nq ) time.
Revised: Algorithm 1 performs linear transformation of a vector using SROS representation in O ( nq ) time.

Original: Algorithm 1 Linear Transformation with SROS Linear Operator.
Revised: Algorithm 1 Linear Transformation with SROS Linear Operator.

Original: This can be vectorized on a
Revised: This can be vectorized on a

Original: Graphical Processing Unit(GPU), using e.g. torch. Tensor.index_add_ for Line 5 and non-slice indexing for Line 6 [16].
Revised: Unit for Line 6 [19].

Original: Slight restructuring allows vectorization over [ q ] as well.
Revised: Slight restructuring allows vectorization over [ q ] as well.

Original: Require: P, P 0 2 [ r ] n ⇥ q , U, U 0 2 R n ⇥ q , x 2 R n
Revised: Require: P, P ′ ∈ [ r ] n × q , U, U ′ ∈ R n × q , x ∈ R n

Original: Ensure:
Revised: Ensure:

Original: y =
Revised: y =

Original: L ( P, P 0 , U, U 0 )
Revised: L ( P, P ′ , U, U ′ )

Original: x 1: y  0 2 R n 2: for i 2 [ q ] do . O ( nq ) time 3: p, p 0 , u, u  P : ,i , P 0 : ,i , U : ,i , U 0 : ,i 4:z 0 2 R r .
Revised: x 1: y ← 0 ∈ R n 2: for i ∈ [ q ] do ▷ O ( nq ) time 3: p, p ′ , u, u ′ ← P : ,i , P ′ : ,i , U : ,i , U ′ : ,i 4:

Original: z  l will store the dot product (( u 0 ) p 0 = l ) > x p [-
-]
Revised: z ← 0 ∈ R r ▷ z l will store the dot product (( u ′ ) p ′ = l ) ⊤ x p [-
-]′

Original: = l 5: for j 2 [ n ] do z p [-
 j z p 
-]
Revised: = l 5: for j ∈ [ n ] do z p [-
 j z p 
-]′

Original: j  + u 0 j x j . O ( n ) time 6: for j 2 [ n ] do y j  y j
Revised: j ← z p ′ j + u ′ j x j ▷ O ( n ) time 6: for j ∈ [ n ] do y j ← y j

Original: + z p j u j . O ( n ) time return y
Revised: + z p j u j ▷ O ( n ) time return y

Original: We now discuss how to approximately invert a certain kind of symmetric SROS matrix, but ourmethods could be extended to asymmetric matrices.
Revised: We now discuss how to approximately invert a certain kind of symmetric SROS matrix, but our methods could be extended to asymmetric matrices.

Original: First, we deﬁne a partial ordering over partitions.
Revised: First, we define a partial ordering over partitions.

Original: For two partitions p, p 0 , we say p 0  p if p 0 is ﬁner than or equal to p ; that is, p [-
-]
Revised: For two partitions p, p ′ , we say p ′ ≤ p if p ′ is finer than or equal to p ; that is, p [-
-]′

Original: j = p [-
-]
Revised: j = p [-
-]′

Original: j 0 = ) p
Revised: j ′ = ⇒ p

Original: j = p j 0 .
Revised: j = p j ′ .

Original: Using that partial ordering, a symmetric SROS matrix can be approximately inverted efﬁciently if forall 1  i, i 0  q , P : ,i  P : ,i 0 or P : ,i 0  P : ,i .
Revised: Using that partial ordering, a symmetric SROS matrix can be approximately inverted efficiently if for all 1 ≤ i, i ′ ≤ q , P : ,i ≤ P : ,i ′ or P : ,i ′ ≤ P : ,i .

Original: As the reader may have recognized, our kernel matrix146 K
Revised: As the reader may have recognized, our kernel matrix

Original:  XX can be written as an SROS matrix with this property.
Revised: K XX can be written as an SROS matrix with this property.

Original: We will write symmetric SROS matrices in a slightly more convenient form.
Revised: We will write symmetric SROS matrices in a slightly more convenient form.

Original: All ( u 0 ) p
Revised: All ( u ′ ) p

Original: = l must be aconstant times u p = l .
Revised: = l must be a constant times u p = l .

Original: We will store these constants in an array C .
Revised: We will store these constants in an array C .

Original: Let L ( P, C, U ) be shorthand for
Revised: Let L ( P, C, U ) be shorthand for

Original: L ( P, P, U, C  U ) , where  denotes element-wise multiplication.
Revised: L ( P, P, U, C ⊙ U ) , where ⊙ denotes element-wise multiplication.

Original: For L ( P, C, U ) to be symmetric,it must be the case that P ji = P j 0 i =)
Revised: For L ( P, C, U ) to be symmetric, it must be the case that P ji = P j ′ i =

Original:  C ji = C
Revised: ⇒ C ji = C

Original: j 0 i .
Revised: j ′ i .

Original: Then, all elements of U corresponding toa given u p =
Revised: Then, all elements of U corresponding to a given u p =

Original: l are multiplied by the same constant.
Revised: l are multiplied by the same constant.

Original: We now present an algorithm for calculating( L ( P, C, U ) +  I )  1 , for  6 = 0 , which is an approximate inversion of L ( P, C, U ) .
Revised: We now present an algorithm for calculating ( L ( P, C, U ) + λI ) − 1 , for λ ̸ = 0 , which is an approximate inversion of L ( P, C, U ) .

Original: We have notyet analyzed numerical sensitivity for  ! 0 , but we conjecture that all ﬂoating point numbers involved need to be stored to at least log 2 (1 /  ) bits.
Revised: We have not yet analyzed numerical sensitivity for λ → 0 , but we conjecture that all floating point numbers involved need to be stored to at least log 2 (1 /λ ) bits.

Original: Without loss of generality, let  = 1 , and note( L ( P, C, U ) +  I ) [- 
-] =   1 ( L ( P,   1 C, U ) + I )  1 .
Revised: Without loss of generality, let λ = 1 , and note ( L ( P, C, U ) + λI ) [- 
-]− 1 = λ − 1 ( L ( P, λ − 1 C, U ) + I ) − 1 .

Original: By assumption, all columns of P are comparable with respect to the partial ordering above, so wecan reorder the columns of P such that P : ,i  P : ,j for i < j .
Revised: By assumption, all columns of P are comparable with respect to the partial ordering above, so we can reorder the columns of P such that P : ,i ≥ P : ,j for i < j .

Original: The key identity that we use to developour fast inversion algorithm is the Sherman–Morrison Formula:
Revised: The key identity that we use to develop our fast inversion algorithm is the Sherman–Morrison Formula:

Original: [Equation]
Revised: [Equation]

Original: Starting with A
Revised: Starting with A

Original: =
Revised: =

Original: I , we add the sparse rank one matrices iteratively, from the ﬁnest partition to thecoarsest one, updating A  1 as we go.
Revised: I , we add the sparse rank one matrices iteratively, from the finest partition to the coarsest one, updating A − 1 as we go.

Original: We represent ( L ( P, C, U )+ I )  1 in the form I + L ( P, C 0 , U 0 ) ,so we write an algorithm that returns C 0 and U 0 .
Revised: We represent ( L ( P, C, U )+ I ) − 1 in the form I + L ( P, C ′ , U ′ ) , so we write an algorithm that returns C ′ and U ′ .

Original: We can also quickly calculate log | L ( P, C, U )
Revised: We can also quickly calculate log | L ( P, C, U )

Original: + I |at the same time, using the matrix determinant lemma: | A + cuu > | = (1 + cu > A  1 u ) | A | .
Revised: + I | at the same time, using the matrix determinant lemma: | A + cuu ⊤ | = (1 + cu ⊤ A − 1 u ) | A | .

Original: Theorem 1 (Fast Inversion) .
Revised: Theorem 1 (Fast Inversion) .

Original: For P 2 [ r ] n ⇥ q and C, U 2 R n ⇥ q , if P : ,i (is coarser than) 
Revised: For P ∈ [ r ] n × q and C, U ∈ R n × q , if P : ,i (is coarser than) ≥

Original: P : ,j for i <
Revised: P : ,j for i <

Original: j ,then there exists C 0 , U 0 2 R n ⇥ q , such that ( L ( P, C, U ) + I )  

Revised: j , then there exists C ′ , U ′ ∈ R n × q , such that ( L ( P, C, U ) + I ) −

Original: = I + L ( P, C 0 , U 0 ) .
Revised: = I + L ( P, C ′ , U ′ ) .

Original: There exists analgorithm for computing C [-
-] and U 0 that takes O ( nq 2 ) time.
Revised: There exists an algorithm for computing C [-
-]′ and U ′ that takes O ( nq 2 ) time.

Original: Proof.
Revised: Proof.

Original: For X 2 R n ⇥ q , let X : ,i +1: q 2 R n ⇥ ( q  i ) be columns i + 1 through q of matrix X (inclusive).
Revised: For X ∈ R n × q , let X : ,i +1: q ∈ R n × ( q − i ) be columns i + 1 through q of matrix X (inclusive).

Original: Let A i
Revised: Let A i

Original: = I + L ( P : ,i +1: q , C : ,i +1: q , U : ,i +1: q ) , and A q = I .
Revised: = I + L ( P : ,i +1: q , C : ,i +1: q , U : ,i +1: q ) , and A q = I .

Original: Now suppose A  1 i can be written as
Revised: Now suppose A − 1 i can be written as

Original: I + L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) for some C 0 and U 0 .
Revised: I + L ( P : ,i +1: q , C ′ : ,i +1: q , U ′ : ,i +1: q ) for some C ′ and U ′ .

Original: For the base case of i = q , this holds trivially.
Revised: For the base case of i = q , this holds trivially.

Original: We show it also holds for i  1 , and we can compute C 0 : ,i : q , U 0 : ,i : q in O  n ( q  i )  time.
Revised: We show it also holds for i − 1 , and we can compute C ′ : ,i : q , U ′ : ,i : q in O (cid:0) n ( q − i ) (cid:1) time.

Original: Let p
Revised: Let p

Original: =
Revised: =

Original: P : ,i ,u
Revised: P : ,i , u

Original: = U : ,i , and c = C : ,i .
Revised: = U : ,i , and c = C : ,i .

Original: Consider u p = l , where each element is zero unless the corresponding elementof p equals l .
Revised: Consider u p = l , where each element is zero unless the corresponding element of p equals l .

Original: What do we know about the product 
Revised: What do we know about the product A − 1 i u p =

Original: A  1 i u p = l (as seen in Equation 4)?
Revised: l (as seen in Equation 3)?

Original: Because the columns of P go from coarser partitions to ﬁner ones, all of the vectors generatingthe sparse rank one components of L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) are from partitions that are equalto or ﬁner than p .
Revised: Because the columns of P go from coarser partitions to finer ones, all of the vectors generating the sparse rank one components of L ( P : ,i +1: q , C ′ : ,i +1: q , U ′ : ,i +1: q ) are from partitions that are equal to or finer than p .

Original: Thus, they are either zero everywhere u p =
Revised: Thus, they are either zero everywhere u p =

Original: l is zero, or zero everywhere u p = l isnonzero.
Revised: l is zero, or zero everywhere u p = l is nonzero.

Original: Vectors v of the second kind can be ignored, as cvv > u p = l = 0 .
Revised: Vectors v of the second kind can be ignored, as cvv ⊤ u p = l = 0 .

Original: Thus, when multiplying
Revised: Thus, when multiplying

Original: L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) by u p = l , the only relevant vectors are ﬁlled with zeros except wherethe corresponding element of p equals l .
Revised: L ( P : ,i +1: q , C ′ : ,i +1: q , U ′ : ,i +1: q ) by u p = l , the only relevant vectors are filled with zeros except where the corresponding element of p equals l .

Original: So we can get rid of those rows of P : ,i +1: q , C 0 : ,i +1: q , and
Revised: So we can get rid of those rows of P : ,i +1: q , C ′ : ,i +1: q , and

Original: U 0 : ,i +1: q .
Revised: U ′ : ,i +1: q .

Original: Suppose there are n l elements of p that equal l .
Revised: Suppose there are n l elements of p that equal l .

Original: Then L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) u p = linvolves n l rows, and can be computed in O  n [Equation]
Revised: Then L ( P : ,i +1: q , C ′ : ,i +1: q , U ′ : ,i +1: q ) u p = l involves n l rows, and can be computed in O (cid:0) n l ( q − i ) (cid:1) time. Moreover, this product, which we’ll call ( u ′ )

Original: i
Revised: p =

Original: [Equation]  time. Moreover, this product, which we’llcall ( u 0 ) p = l , is only nonzero when the corresponding element of p equals l , so it has the same sparsitypattern as u p = l .
Revised: l , is only nonzero when the corresponding element of p equals l , so it has the same sparsity pattern as u p = l .

Original: The other component of A  1 i is the identity matrix, and Iu p
Revised: The other component of A − 1 i is the identity matrix, and Iu p

Original: =
Revised: =

Original: l clearly has the samesparsity as u p = l .
Revised: l clearly has the same sparsity as u p = l .

Original: Thus, returning to Equation 4, when we add u p = l ( u p = l ) > to A i , we update A  1 iwith an outer product of vectors whose sparsity pattern is the same as that of u p = l .
Revised: Thus, returning to Equation 3, when we add u p = l ( u p = l ) ⊤ to A i , we update A − 1 i with an outer product of vectors whose sparsity pattern is the same as that of u p = l .

Original: For each l , A  1 i need not be updated with each u p =
Revised: For each l , A − 1 i need not be updated with each u p =

Original: l one at a time.
Revised: l one at a time.

Original: For l 6 = l 0 , u p = l and u p = l 0are nonzero at separate indices, so u p = l and ( u 0 ) p=
Revised: For l ̸ = l ′ , u p = l and u p = l ′ are nonzero at separate indices, so u p = l and ( u ′ ) p

Original:  l 0 are nonzero at separate indices, so the extracomponent of A  1 i that appears after the u p = l 0 update is irrelevant to the u p =
Revised: = l ′ are nonzero at separate indices, so the extra component of A − 1 i that appears after the u p = l ′ update is irrelevant to the u p =

Original: l update, because( u p = l ) > ( u 0 )p =
Revised: l update, because ( u p = l ) ⊤ ( u ′ )

Original:  l 0 = 0 .
Revised: p = l ′ = 0 .

Original: Since the u p = l update takes O ( n l ( q  i )) time, all of them together take
Revised: Since the u p = l update takes O ( n l ( q − i )) time, all of them together take

Original: O ( P l n l ( q  i )) time, which equals O ( n ( q  i )) time.
Revised: O ( (cid:80) l n l ( q − i )) time, which equals O ( n ( q − i )) time.

Original: Calculating an element of c 0 only involvescomputing the denominator in Equation 4, using a matrix-vector product already computed.
Revised: Calculating an element of c ′ only involves computing the denominator in Equation 3, using a matrix-vector product already computed.

Original: So wecan write C 0 : ,i : q and U 0 : ,i : q by adding a preceding column to C 0 : ,i +1:
Revised: So we can write C ′ : ,i : q and U ′ : ,i : q by adding a preceding column to C ′ : ,i +1:

Original: q and U 0 : ,i +1: q , using the samepartition p , and it takes O ( n ( q  i )) time.
Revised: q and U ′ : ,i +1: q , using the same partition p , and it takes O ( n ( q − i )) time.

Original: Following the induction down to i = 0 , we have ( L ( P, C, U ) + I ) [- 
-] =
Revised: Following the induction down to i = 0 , we have ( L ( P, C, U ) + I ) [- 
-]− 1 =

Original: I + L ( P, C 0 , U 0 ) , and a totaltime of O ( nq 2 ) .
Revised: I + L ( P, C ′ , U ′ ) , and a total time of O ( nq 2 ) .

Original: Algorithm 2 also performs approximate inversion, which we prove in Appendix A.
Revised: Algorithm 2 also performs approximate inversion, which we prove in Appendix A.

Original: It differs slightlyfrom the algorithm in the proof, but can take full advantage of a GPU speedup.
Revised: It differs slightly from the algorithm in the proof, but can take full advantage of a GPU speedup.

Original: In the setting whereall columns of U are identical, observe that in Lines 10 and 11, the same computation is repeated forall k [-
-]
Revised: In the setting where all columns of U are identical, observe that in Lines 10 and 11, the same computation is repeated for all k [-
-]∈

Original: [ i ] .
Revised: [ i ] .

Original: Indeed, in this setting, this block of code can be modiﬁed to run in O ( n ) time rather than
Revised: Indeed, in this setting, this block of code can be modified to run in O ( n ) time rather than

Original: O ( ni ) , making the whole algorithm run in O ( nq ) time, as shown in Proposition 4 in Appendix A.
Revised: O ( ni ) , making the whole algorithm run in O ( nq ) time, as shown in Proposition 4 in Appendix A.

Revised: A Hierarchical matrix is a matrix which is either represented as a low-rank matrix or as a 2 ×block matrix of Hierarchical matrices [1].

Revised: In our SROS format, many of the sparse rank one matrices overlap, whereas in a Hierarchical matrix, the low-rank matrices do not overlap, and converting an SROS matrix into a Hierarchical matrix would typically be inefficient.

Revised: Hierarchical matrices admit approximate inversion in O ( na 2 log 2 n ) time, where a is the maximum rank of the component submatrices [11].

Revised: However, this is not an approximation in a technical sense, as there is no error bound.

Revised: At many successive steps in the algorithm, a rank 2 a matrix is approximated by a rank a

Original:  Require:
Revised: Algorithm Require:

Original: P 2 [ r ] n ⇥ q , C, U 2 R n ⇥ q
Revised: ∈ × q , C, U ∈ R n × q

Original: Ensure: I + L ( P, C 0 , U 0 )
Revised: Ensure: I + L ( P, C ′ , U ′ )

Original: =
Revised: =

Original: ( I + L ( P, C, U ))  1 ; x = log | I + L ( P, C, U ) | 1: x, C 0 , U0 , 0 2 R n ⇥ q , U 2: for i 2 ( q, q  1 , ..., 1) do . O ( nq 2 ) time 3: p, c, u, u 
Revised: ( I + L ( P, C, U )) − 1 ; x = log | I + L ( P, C, U ) | 1: x, C ′ , U ′ ← 0 , 0 ∈ R n × q , U 2: for i ∈ ( q, q − 1 , ..., 1) do ▷ O ( nq 2 ) time 3: p, c, u, u ′ ← P : ,i , C : ,i , U : ,i , U ′ : ,i 4:

Original: P : ,i , C : ,i , U : ,i , U 0 : ,i 4: z 0 2 R r .

Original: z  l will store c ( l ) (( u 0 ) p = l ) > u p = l , where c ( l ) =
Revised: z ← 0 ∈ R r ▷ z l will store c ( l ) (( u ′ ) p = l ) ⊤ u p = l , where c ( l ) =

Original: c k if p k = l 5: for j 2 [ n ] do z p j  z p j 
Revised: c k if p k = l 5: for j ∈ [ n ] do z p j ← z p j + c j u ′ j u j ▷ O ( n ) time 6: for j ∈ [ n ] do C ′

Original: + c j u 0 j u j .
Revised: ji ← − c j / (1 + z p j )

Original:  O ( n ) time 6: for j 2 [ n ] do C 0 ji  c j / (1 + z p j ) . O ( n ) time 7: for l 2 [ r [-] do x x + log(
-]
Revised: ▷ O ( n ) time 7: for l ∈ [ r ] do x ← x + log(1 + z l ) ▷ O ( n ) time because r [-] do x x + log(
-]≤ n 8: if i > 0 then 9: y ← 0 ∈ R n × i

Original: + z l ) . O ( n ) time because r  n 8: if i > 0 then 9: y 0 2 R n ⇥ i .
Revised: ▷

Original: O ( ni ) time 10: for j, k 2 [ n ] ⇥
Revised: O ( ni ) time 10: for j, k ∈ [ n ] ×

Original: [ i  1] do y p
Revised: [ i − 1] do y p

Original: j k  y p
Revised: j k ← y p

Original: j k + u [-
-] j U jk.
Revised: j k + u [-
-]′ j U jk

Original:  O ( ni ) time 11: for j, k 2 [ n ] ⇥
Revised: ▷ O ( ni ) time 11: for j, k ∈ [ n ] ×

Original: [ i  1] do U 0 jk  U 0 jk + C 0 ji u 0 j y p
Revised: [ i − 1] do U ′ jk ← U ′ jk + C ′ ji u ′ j y p

Original: j k . O ( ni ) time return C 0 , U 0 , x 
Revised: j k ▷ O ( ni ) time return C ′ , U ′ , x matrix [10]; to our knowledge there is no analysis of how resulting errors might cascade. After converting an SROS matrix to hierarchical form, this rough inversion would take O ( nq 2 log 2 n ) time.

Original:  Binary tree Gaussian process
Revised:  Binary tree Gaussian process

Original: We now show that our kernel matrix K XX can be written in SROS form, with P containing successively ﬁner partitions.
Revised: We now show that our kernel matrix K XX can be written in SROS form, with P containing successively finer partitions.

Original: Thus, K XX can be approximately inverted quickly, for use in Equations 1 and2.
Revised: Thus, K XX can be approximately inverted quickly, for use in Equations 1 and 2.

Original: Next, we’ll show that we can efﬁciently optimize the log likelihood of the training data by tuningthe weight vector w along with the bit order.
Revised: Next, we’ll show that we can efficiently optimize the log likelihood of the training data by tuning the weight vector w along with the bit order.

Original: The log likelihood can be calculated in O ( nq log n )time and then the gradient w.r.t.
Revised: The log likelihood can be calculated in O ( nq log n ) time and then the gradient w.r.t.

Original: w in O ( nq 2 ) time.
Revised: w in O ( nq 2 ) time.

Original: Recall Equation 3 in the proof of Proposition 1 : K XX = 
Revised: Recall from the proof of Proposition 1 : K XX = (cid:80) qi

Original: P qi = P s 2 B i w i X
Revised: =1 (cid:80) s ∈ B i w i X

Original: [ s ] X > [ s ] , where X [ s ] 2 R nwith ( X [ s ] )
Revised: [ s ] X ⊤ [ s ] , where X [ s ] ∈ R n with ( X [ s ] )

Original: j = hh
Revised: j = (cid:104)(cid:104)

Original: X  | s | j = s ii .
Revised: X ≤| s | j = s (cid:105)(cid:105) .

Original: So we will set P : ,i , C : ,i , and U : ,i , so that L ( P : ,i , C : ,i , U : ,i ) = 
Revised: So we will set P : ,i , C : ,i , and U : ,i , so that L ( P : ,i , C : ,i , U : ,i ) = (cid:80) s

Original: P s 2 B i w i X
Revised: ∈ B i w i X

Original: [ s ] X > [ s ] .
Revised: [ s ] X ⊤ [ s ] .

Original: Let P : ,i partition the set of points X
Revised: Let P : ,i partition the set of points X

Original: so that points are in the same partition if theﬁrst i bits match.
Revised: so that points are in the same partition if the first i bits match.

Original: Now, requiring the ﬁrst i + 1 bits to match is a stricter criterion than requiring theﬁrst i bits to match, so the P : ,i grow successively ﬁner.
Revised: Now, requiring the first i + 1 bits to match is a stricter criterion than requiring the first i bits to match, so the P : ,i grow successively finer.

Original: For any piece of the partition where the ﬁrst ibits of the constituent points equals the bitstring s , the corresponding sparse rank one component of
Revised: For any piece of the partition where the first i bits of the constituent points equals the bitstring s , the corresponding sparse rank one component of

Original: K XX is w i X
Revised: K XX is w i X

Original: [ s ] X > [ s ] .
Revised: [ s ] X ⊤ [ s ] .

Original: So let U : ,i = 1 n , and let C : ,i =
Revised: So let U : ,i = 1 n , and let C : ,i =

Original: w i 1 n .
Revised: w i 1 n .

Original: Proposition 2 (SROS Form Kernel) .
Revised: Proposition 2 (SROS Form Kernel) .

Original: K XX =
Revised: K XX =

Original: L ( P, C, U ) , as deﬁned above.
Revised: L ( P, C, U ) , as defined above.

Original: This follows immediately from the deﬁnitions.
Revised: This follows immediately from the definitions.

Original: To compute these partitions P : ,i , we sort X , which isa set of bit strings.
Revised: To compute these partitions P : ,i , we sort X , which is a set of bit strings.

Original: And then we can easily compute which points have the same ﬁrst i bits.
Revised: And then we can easily compute which points have the same first i bits.

Original: This alltakes O ( nq log n ) time.
Revised: This all takes O ( nq log n ) time.

Original: Now note that U : ,i =
Revised: Now note that U : ,i =

Original: U : ,i 0 for all i, i 0 , so ( K XX +  I )  1 and | K XX +  I |can be computed in O ( nq ) time, rather than O ( nq 2 ) .
Revised: U : ,i ′ for all i, i ′ , so ( K XX + λI ) − 1 and | K XX + λI | can be computed in O ( nq ) time, rather than O ( nq 2 ) .

Original: The training negative log likelihood of a GP is that of the corresponding multivariate Gaussian onthe training data.
Revised: The training negative log likelihood of a GP is that of the corresponding multivariate Gaussian on the training data.

Original: So: NLL( w ) =
Revised: So: NLL

Original: [Equation]
Revised: [Equation]

Original: This can be computed in O ( nq ) time, since matrix-vector multiplication takes O ( nq ) time for amatrix in SROS form.
Revised: This can be computed in O ( nq ) time, since matrix-vector multiplication takes O ( nq ) time for a matrix in SROS form.

Original: So if the bit order is unchanged, an optimization step can be done in O ( nq )time, and if the data needs to be resorted, then in O ( nq log n ) time.
Revised: So if the bit order is unchanged, an optimization step can be done in O ( nq ) time, and if the data needs to be resorted, then in O ( nq log n ) time.

Original: On the largest dataset we tested(House Electric), with n ⇡ 1.3 million and q = 88 , sorting the data and computing P takes about0.96 seconds on a GPU, and then calculating the negative log likelihood takes about another 1.08seconds.
Revised: On the largest dataset we tested (House Electric), with n ≈ 1.3 million and q = 88 , sorting the data and computing P takes about 0.96 seconds on a GPU, and then calculating the negative log likelihood takes about another 1.

Original:  We show in Appendix B how to compute r w NLL in O ( nq 2 ) time.
Revised: seconds. We show in Appendix B how to compute ∇ w NLL in O ( nq 2 ) time.

Original: To optimize the bit order and weight vector at the same time, we represent both with a single parametervector ✓ 2 R q + , with || ✓ || 1 = 1 .
Revised: To optimize the bit order and weight vector at the same time, we represent both with a single parameter vector θ ∈ R q + , with || θ || ∞ = 1 .

Original: To get the bit order from ✓ , we start with a default bit order and227 permute the bit order according to a permutation that would sort ✓ in descending order. 
Revised: To get the bit order from θ , we start with a default bit order and permute the bit order according to a permutation that would sort θ in descending order. To get the Require: X ∈ B n × q , y ∈ R n , X ′ ∈ B m × q , w ∈ R q , λ ∈ R +

Original: To get theweight vector, we sort ✓ in descending order, add a 0 at the end, and  compute thedifferences betweenadjacent elements.
Revised: Ensure: µ X ′ and σ are the predictive means and variances at X ′ , and nll the training negative log likelihood. Algorithm 2. Speedup to time of are identical Uses Algorithm 1 to compute the

Revised: Woodbury vector.

Revised: O nq ) time.

Revised: ▷ O ( n m q time.

Revised: [Table]

Revised: O (

Revised: [Table]

Revised: n + m

Revised: [Table]

Revised: q

Revised: [Table]

Revised: ▷ O ( mq

Revised: [Table]

Revised: time

Revised: [Table]

Revised: extra factor of q because columns of U prec are not identical

Revised: [Table]

Revised: ▷ O ( mq ) time

Revised: [Table]

Original:  When there are ties in the elements of ✓ , the choice of bit order does not affect thenegative log likelihood (or the kernel at all) because the relevant associated weight is 0 .
Revised: 18: return [Table] weight vector, we sort θ in descending order, add a 0 at the end, and compute the differences between adjacent elements. When there are ties in the elements of θ , the choice of bit order does not affect the negative log likelihood (or the kernel at all) because the relevant associated weight is 0 .

Original: The negative log likelihood is continuous with respect to ✓ , and when all values of ✓ are unique, it is differentiablewith respect to ✓ .
Revised: The negative log likelihood is continuous with respect to θ , and when all values of θ are unique, it is differentiable with respect to θ .

Original: Letting ✓ = e  / || e  || 1 , we minimize loss w.r.t.
Revised: Letting θ = e ϕ / || e ϕ || ∞ , we minimize loss w.r.t.

Original:  using BFGS [7].
Revised: ϕ using BFGS [8].

Original: To calculate the predictive mean at a list of predictive locations X 0 , we ﬁrst multiply y by ( K XX + I )  1 , and then we multiply that vector by K XX 0 .
Revised: To calculate the predictive mean at a list of predictive locations X ′ , we first multiply y by ( K XX + λI ) − 1 , and then we multiply that vector by K XX ′ .

Original: We obtain both K XX and K XX 0 in SROS formas follows.
Revised: We obtain both K XX and K XX ′ in SROS form as follows.

Original: Let ˜ X
Revised: Let ˜ X

Original: = X  X 0 be the concatenation of the two tuples, now an ( n + m ) -tuple.
Revised: = X ◦ X ′ be the concatenation of the two tuples, now an ( n + m ) -tuple.

Original: Writing
Revised: Writing

Original: K ˜ X ˜ X
Revised: K ˜ X ˜ X

Original: = L ( ˜ P, ˜ C, ˜ U ) , the arrays on the r.h.s.
Revised: = L ( ˜ P, ˜ C, ˜ U ) , the arrays on the r.h.s.

Original: can be computed in O (( n + m ) q log( n + m )) time.
Revised: can be computed in O (( n + m ) q log( n + m )) time.

Original: Then, with P , C , and U being the ﬁrst n rows of ˜ P , ˜ C , ˜ U , K XX =
Revised: Then, with P , C , and U being the first n rows of ˜ P , ˜ C , ˜ U , K XX =

Original: L ( P, C, U ) .
Revised: L ( P, C, U ) .

Original: And letting P 00 and
Revised: And letting P ′′ and

Original: U 00 be the last m rows, K XX [-
-]
Revised: U ′′ be the last m rows, K XX [-
-]′

Original: = L ( P, P 00 , C  U, U 00 ) .
Revised: = L ( P, P ′′ , C ⊙ U, U ′′ ) .

Original: Thus, the predictive mean µ x from Equation1 can be computed at m locations in O (( n + m ) q log( n + m )) time.
Revised: Thus, the predictive mean µ x from Equation 1 can be computed at m locations in O (( n + m ) q log( n + m )) time.

Original: The predictive covariance matrix, which extends the predictive variance from Equation 2, is calculated⌃ X [-
-]
Revised: The predictive covariance matrix, which extends the predictive variance from Equation 2, is calculated Σ X [-
-]′ =

Original: = K X 0 X[-
-]
Revised: K X ′ X[-
-]

Original: +  I m  K X
Revised: + λI m − K X

Original: 0 X
Revised: ′ X

Original: ( K XX +  I n )  1 K XX 0 =
Revised: ( K XX + λI n ) − 1 K XX ′ =

Original: ( K ˜ X ˜ X +  I m + n ) /K
Revised: ( K ˜ X ˜ X + λI m + n ) /K

Original: XX , where / denotes242 the Schur complement.
Revised: XX , where / denotes the Schur complement.

Original: From a property of block matrix inversion, the last m columns of the lastm rows of ( K ˜ X ˜ X +  I )  1 equals (( K ˜ X ˜ X
Revised: From a property of block matrix inversion, the last m columns of the last m rows of ( K ˜ X ˜ X + λI ) − 1 equals (( K ˜ X ˜ X

Original: +  I m + n ) /K
Revised: + λI m + n ) /K

Original: XX )  1 .
Revised: XX ) − 1 .

Original: So we get the predictive precisionmatrix in O (( n + m ) q log( n + m )) time by inverting K ˜ X
Revised: So we get the predictive precision matrix in O (( n + m ) q log( n + m )) time by inverting K ˜ X

Original: ˜ X +  I and taking the bottom right m ⇥ mblock.
Revised: ˜ X + λI and taking the bottom right m × m block.

Original: Then, we get the predictive covariance matrix by inverting that.
Revised: Then, we get the predictive covariance matrix by inverting that.

Original: This takes O ( mq 2 ) time,since it does not have the property of all the columns of U being equal.
Revised: This takes O ( mq 2 ) time, since it does not have the property of all the columns of U being equal.

Original: If we only want the diagonalelements of an SROS matrix (the independent predictive variances in this case), we can simply sumthe rows of C  U  U in O ( mq ) time.Thus, in total, computing the independent predictive variancesrequires O (( n + m ) q log( n + m ) + mq 2 ) time. See Algorithm 3. Algorithm
Revised: If we only want the diagonal elements of an SROS matrix (the independent predictive variances in this case), we can simply sum the rows of C ⊙ U ⊙ U in O ( mq ) time.

Original: Require: X 2 B n ⇥ q , y 2 R n , X

Original: [Equation]

Original: 2 B m ⇥ q , w

Original: [Equation]

Original: R q

Original: [Equation]

Original:  2 R +

Original: Ensure: µ X 0 and  2 X 0 are the predictive means and variances at X 0 , and nll the training negative log likelihood.

Original: 1: ˜ X X  X 0 2: ˜ X " , perm

Original: Sort ( ˜ X ) .

Original: The rows of X are sorted lexically from leading bit to trailing bit.

Original: O (( n + m ) q log( n + m )) time.

Original: 3: for j, i 2 [ n + m ]

Original: ⇥

Original: [ q ] do ˜ P " ji # of unique rows in X " 1: j, 1: i 4: .

Original: M 1: j, 1: i is the ﬁrst j rows and i columns of M .

Original: O (( n + m ) q ) time.

Original: 5: ˜ P perm

Original: ( P This “unsorts” the input. O (( n + m ) q  ) time.6: P
Revised: Thus, in total, computing the independent predictive variances requires O (( n + m ) q log( n + m ) + mq 2 ) time.

Original: , P˜ P 1: n , ˜ P n +

Original: n + m

Original: U, U 0 , ˜ U 1 n ⇥ q , 1 m ⇥ q , 1 ( n + m ) ⇥

Original: q

Original: C, ˜ C 1 n w T , 1 n + m w T

Original: C  1  , U  1 , logdet

Original: Invert ( P,   1 C, U ) .

Original: Uses Algorithm 2.
Revised: See Algorithm 3.

Original: Speedup to O ( nq ) time because columns of   1 U are identical.

Original: C  1 , logdet

Original:   1 C  1  , logdet  + n log(  ) 11:

Original: z

Original: LinTransform ( P, P, U  1 , C  1  U  1 , y )

Original: +   1 y .

Original: Uses Algorithm 1 to compute the

Original: Woodbury vector.

Original: O ( nq ) time.

Original: 12: µ X

Original: LinTransform ( P 0 , P, U 0 , C  U, z ) .

Original: O (( n + m ) q ) time.

Original: 13: nll ( y > z + logdet + n log(2 ⇡ )) /14:

Original: ˜ C prec , ˜

Original: U prec

Original: Invert ( ˜ P,   1 ˜ C,   1 ˜ U ) .

Original: O (( n + m ) q ) time.

Original: 15: C prec , U prec ˜ C prec n +1: n + m , ˜ U prec n +1: n + m 16: C cov , U cov

Original: Invert ( P 0 , C prec , U prec ) .

Original: O ( mq 2 ) time; extra factor of q because columns of U prec are not identical.

Original: 17:  2 X  ( 1 m + SumEachRow ( C cov  U cov  U cov )) .

Original: O ( mq ) time.

Original: 18: return µ X 0 ,  2 X 0 , nll

Original:  Related Work
Revised:  Related Work

Original: All existing kernels of which we are aware for linear time GP regression on unstructured data involveinducing points (related to the Nyström approximation [24]) or inducing frequencies.
Revised: All existing kernels of which we are aware for linear time GP regression on unstructured data involve inducing points (related to the Nyström approximation [27]) or inducing frequencies.

Original: For a given setof inducing points Z , for some base kernel k , the inducing point kernel (in its most basic form) is thefollowing, although subtle variants exist: k Z ( x, x 0 )
Revised: For a given set of inducing points Z , for some base kernel k , the inducing point kernel (in its most basic form) is the following, although subtle variants exist: k Z ( x, x ′ )

Original: =
Revised: =

Original: K xZ K  1 ZZ K Zx 0 [18].
Revised: K xZ K − 1 ZZ K Zx ′ [21].

Original: Sparse Gaussian Process Regression (SGPR) involves selecting Z , and then using k Z (or a variant).
Revised: Sparse Gaussian Process Regression (SGPR) involves selecting Z , and then using k Z (or a variant).

Original: Notably, K ZXX =
Revised: Notably, K ZXX =

Original: K XZ K  1 ZZ K ZX is low rank, providing computational efﬁciency.
Revised: K XZ K − 1 ZZ K ZX is low rank, providing computational efficiency.

Original: The predictivemean and covariance have compact form, with observational noise  : µ x ( Z )=
Revised: The predictive mean and covariance have compact form, with observational noise λ : µ x ( Z )

Original:  K xZ (  K ZZ +k ZX k XZ )  1 K ZX y and  2 xx [-
-]
Revised: = K xZ ( λK ZZ + k ZX k XZ ) − 1 K ZX y and σ 2 xx [-
-]′

Original: ( Z ) =
Revised: ( Z ) =

Original: K xZ ( K ZZ +   1 k ZX k XZ )  1 K Zx 0 .
Revised: K xZ ( K ZZ + λ − 1 k ZX k XZ ) − 1 K Zx ′ .

Original: Titsias’s [21] sparse variational kernel is also low rank and uses inducing points.
Revised: Titsias’s [24] sparse variational kernel is also low rank and uses inducing points.

Original: The sparse variational
Revised: The sparse variational

Original: GP (SVGP) is constructed as the solution to a variational inference problem.
Revised: GP (SVGP) is constructed as the solution to a variational inference problem.

Original: It depends on inducingpoints Z , data points X , and observed function values y .
Revised: It depends on inducing points Z , data points X , and observed function values y .

Original: We have focused on Gaussian processes with0 mean, but the SVGP method uses a nonzero prior mean along with a kernel: m SVGP ( x )
Revised: We have focused on Gaussian processes with 0 mean, but the SVGP method uses a nonzero prior mean along with a kernel: m SVGP ( x )

Original: = µ x ( Z )and k SVGP ( x, x 0 )
Revised: = µ x ( Z ) and k SVGP ( x, x ′ )

Original: =
Revised: =

Original: k ( x, x 0 )  K xZ K  1 ZZ K Zx[-
-]
Revised: k ( x, x ′ ) − K xZ K − 1 ZZ K Zx[-
-]

Original:  +  2 xx 0 ( Z ) .
Revised: ′ + σ 2 xx ′ ( Z ) .

Original: Given the dependence on X and y , thisis not a true probability distribution over function space.
Revised: Given the dependence on X and y , this is not a true probability distribution over function space.

Original: The variational problem underlying thiskernel also provides guidance in how to select the inducing points Z .
Revised: The variational problem underlying this kernel also provides guidance in how to select the inducing points Z .

Original: For further discussion of thekernel underlying the SVGP method, see Wild et al.
Revised: For further discussion of the kernel underlying the SVGP method, see Wild et al.

Original: An inducing point kernel with z inducing points produces a z -dimensional reproducing kernel Hilbertspace (RKHS).
Revised: An inducing point kernel with z inducing points produces a z -dimensional reproducing kernel Hilbert space (RKHS).

Original: The dimensionality of the RKHS relates to the expressivity of the kernel.
Revised: The dimensionality of the RKHS relates to the expressivity of the kernel.

Original: Whereas aninducing point method buys a z -dimensional RKHS for the price of O ( z 2 n ) time and O ( zn ) space,the binary tree kernel produces a 2 q -dimensional RKHS in O ( qn ) time and space—an exponentialimprovement.
Revised: Whereas an inducing point method buys a z -dimensional RKHS for the price of O ( z 2 n ) time and O ( zn ) space, the binary tree kernel produces a 2 q -dimensional RKHS in O ( qn ) time and space—an exponential improvement.

Original: (Observe that we can ﬁnd 2 q linearly independent functions of the form k ( · , x ) —onefor each of the 2 q leaves x might belong to.)
Revised: (Observe that we can find 2 q linearly independent functions of the form k ( · , x ) —one for each of the 2 q leaves x might belong to.)

Original: Wilson and Nickisch [26] develop a method for speedingup inducing point methods signiﬁcantly, especially in low-dimensional settings.
Revised: Wilson and Nickisch [29] develop a method for speeding up inducing point methods significantly, especially in low-dimensional settings.

Original: Lázaro-Gredilla et al.
Revised: Lázaro-Gredilla et al.

Original: [13] propose an inducing frequencies kernel: given a set of m inducing vectorss i , k ( x, x 0 ) 
Revised: [16] propose an inducing frequencies kernel: given a set of m inducing vectors s i , k ( x, x ′ ) = 1 /m (cid:80) mi =1 cos(2 πs ⊤ i ( x − x ′ )) .

Revised: Dutordoir et al.

Revised: [6] propose an inducing frequencies kernel for low dimensional data, in which k ( x, x ′ ) is a special function of x ⊤ x ′ .

Original: = 1 /m
Revised: On one-dimensional data, filtering/smoothing methods perform Bayesian inference over functions in O ( n ) time [14, 12]. A few non- O ( n ) methods bear mentioning. We are not the first to consider a kernel over points on the leaves of a tree [18, 17] or on the leaves of multiple trees [7], but their methods take O ( n 3 ) time. On certain kinds of structured data, Toeplitz solvers achieve O ( n 2 ) time complexity [30].

Revised: Cutajar et al.

Revised: ’s

Revised: [3], Wang et al.’s

Revised: [25], and others’ use of a conjugate gradients solver to replace inversion/factorization has unclear time complexity between O ( n 2 ) and O ( n 3 ) , depending on the kernel matrix spectrum.

Original: P mi =1 cos(2 ⇡ s > i ( x  x 0 )) .
Revised: In Table 1, we compare our binary tree kernel and a binary tree ensemble (see Appendix C for details on the ensemble) against three baseline methods: exact GP regression using a Matérn kernel, sparse Gaussian process regression (SGPR) [23], and a stochastic variational Gaussian process (SVGP) [13].

Original: Dutordoir et al. 
Revised: We evaluate our method on the same open-access UCI datasets [4] as Wang et al. [25], using their same training, validation, and test partitions, and we compare against the baseline results they report. For the binary tree (BT) kernels, we use p = min(8 , ⌊ 150 /d ⌋ + 1) , and recall q = pd . We set λ = 1 /n .

Revised: We train the bit order and weights to minimize training NLL.

Original: [5] propose an inducing frequencieskernel for low dimensional data, in which k ( x, x 0 ) is a special function of x > x 0 . 
Revised: For the binary tree ensemble (BTE), we use 20 kernels. For the Matérn kernel, we use Blackbox Matrix-Matrix multiplication (BBMM) [9], which uses the conjugate gradients method to calculate ( K XX + λI ) − 1 . SGPR uses 512 data points and SVGP uses 1,024 inducing points. We report the mean and two standard errors across 3 replications with different dataset splits.

Original: On one-dimensional data, ﬁltering/smoothing methods perform Bayesian inference over functionsin O ( n ) time [11, 9].
Revised: For further experimental details, see Appendix C. BTE achieves the best NLL on 6/12 datasets, and best RMSE on 5/12 datasets (including some ties). Out of the 4 largest datasets, BT/BTE is fastest on 3. The run times are plotted in Figure 5.

Revised: The code is available at https://github.com/mkc1000/btgp and https://tinyurl.com/btgp-colab .

Revised: [Table]

Original: A few non- O ( n ) methods bear mentioning.
Revised: BT performs noticeably worse than BTE for test NLL on CTSlice due to over-fitting. There are enough degrees of freedom when optimizing the bit order ( d = 385 ) that BT kernel can over-fit to the training data. The ensemble over multiple bit orders is much more robust.  Discussion

Revised: We have proven that the binary tree kernel GP is scalable.

Revised: Our empirical results suggest that it often outpredicts not just other scalable methods but even the popular Matérn GP.

Revised: If the results in this paper replicate in other domains, it could obviate wide usage of classic GP kernels like the Matérn kernel, as well as inducing point kernels.

Original: We are not the ﬁrst to considera kernel over points on the leaves of a tree [15, 14] or on the leaves of multiple trees [6], but theirmethods take O ( n 3 ) time.
Revised: Sometimes, our kernel fails to capture patterns in the data; some functions’ values simply do not covary this way. But other kernels we tested seemed to fail like that even more. Our contributions to linear algebra and kernel design may significantly increase the size of data sets on which GPs can do state-of-the-art modelling.

Revised: Checklist

Revised: The checklist follows the references.

Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: On certain kinds of structured data, Toeplitz solvers achieve O ( n 2 ) timecomplexity [27].
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] . You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?

Original: Cutajar et al.’s
Revised: [Yes] See Section ?? . • Did you include the license to the code and datasets? [No] The code and the data are proprietary. • Did you include the license to the code and datasets?

Original: [2], Wang et al.’s
Revised: [N/A] Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit.

Original: [22], and others’ use of a conjugate gradients solverto replace inversion/factorization has unclear time complexity between O ( n 2 ) and O ( n 3 ) , dependingon the kernel matrix spectrum.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below. For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes]

Revised: In the introduction.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] Yes in Section E. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] In Section(b) Did you include complete proofs of all theoretical results?

Revised: [Yes]

Revised: In Sections 3 andand Appendix A 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] As a URL.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] In Section 7 and Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] See Table 1.

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See Table 1. 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] The UCI datasets in Section 7.

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [N/A] 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]

Revised: A Correctness of Algorithm

Revised: In this section, we show that Algorithm 2 performs approximate inversion.

Revised: And then we show how to modify the algorithm in the setting were all columns of U are identical, for a factor of q speedup.

Revised: We begin by writing the exact form of ( u ′ )

Revised: p =

Revised: l from the proof of Theorem 1, and the corresponding c ′ ( l ) .

Revised: Recall the Sherman–Morrison Formula:

Revised: [Equation]

Revised: At the time that we update A with the rank one matrix c ( l ) u p = l u p = l ⊤ , what does A equal?

Revised: Let c and u originate from the i th column of C and U .

Revised: So our update is C ( l ) : ,i U P : ,i

Original: Experiments
Revised: = l : ,i [Equation] ,i

Revised: [Equation]

Original: {+Then A ← A i +
+}
Revised: {+Then A ← A i +
+}

Revised: = I + (cid:80)

Revised: qk = i +1 (cid:80) ℓ ∈

Revised: [ r ] C ( ℓ ) : ,k U P : ,k = ℓ : ,k

Revised: U P : ,k = ℓ : ,k ⊤ , recalling r is the largest integer in

Revised: P .

Revised: And likewise, all the relevant entries of C ′ and U ′ have been calculated for A − 1 .

Revised: So we have

Original: {+A − 1 i +
+}
Revised: {+A − 1 i +
+}

Revised: = I + (cid:80)

Revised: qk = i +1 (cid:80) ℓ ∈

Revised: [ r ] C ′ ( ℓ ) : ,k U ′ P : ,k = ℓ : ,k

Revised: [Equation]

Revised: The main computation we need to do is A − 1 u .

Revised: We’ll say that k ℓ ⊑ i l if set l from partition P : ,i is a superset of set ℓ from partition P : ,k .

Revised: That is, the rows where P : ,k takes the value ℓ are a subset of the rows where P : ,i takes the value l .

Revised: The key simplification we use is that if k ℓ ̸⊑ i l , then

Revised: U P : ,i =

Revised: l : ,i ⊤ U ′ P : ,k = ℓ : ,k = 0 .

Revised: The rows at which those two vectors have nonzero elements are disjoint.

Revised: This logic is explained in the proof of Theorem 1 without all the notation.

Revised: So we let

Revised: [Equation]

Revised: Equation 6 follows because the terms in the dot product U ′ P : ,k = ℓ : ,k ⊤ U P : ,i

Revised: = l : ,i are only nonzero when

Revised: P : ,k = ℓ and P : ,i = l , but the latter is implied by k ℓ ⊑ i l , so this is equivalent to the elements where

Revised: P : ,k = ℓ .

Revised: So U ′ P : ,k = ℓ : ,k ⊤

Revised: [Equation]

Revised: P : ,i =

Revised: l : ,i

Revised: [Equation]

Revised: = ℓ :

Revised: [Equation]

Revised: , which gives us Equation 6.

Revised: Then, we let

Revised: [Equation]

Revised: [Equation]

Revised: Proposition 3 (Correctness of Algorithm 2) .

Revised: In Algorithm 2, U ′ and C ′ take the values defined in

Revised: Equations 6 and 7.

Revised: Proof.

Revised: We’ll assume that for k > i , C ′ : ,k and U ′ : ,k have the right values, and we’ll show that C ′ : ,i and U ′ : ,i get the right values.

Revised: Starting with U ′ : ,q , the (cid:80) qk = q +1 in Equation 6 is empty, so U ′ : ,q = U : ,q .

Revised: In Algorithm 2, U ′ is initialized to U , and the q th column is never updated.

Revised: Now we see that C ′ : ,i is correct assuming U ′ : ,i is.

Revised: In Line 5, we ensure z l = C ( l ) : ,i U ′ P : ,i =

Revised: l : ,i ⊤ U P : ,i

Revised: = l : ,i .

Revised: A dot product is the sum of elementwise multiplications, and one can inspect that each such multiplication gets added to the right slot in z .

Revised: Then, Line 6 implements Equation 7, with numerator and denominator multiplied by C ( l ) : ,i .

Revised: (It stores the same value in multiple locations).

Revised: Now we turn to U ′ : ,i .

Revised: U ′ : ,i is updated in every preceding loop.

Revised: It starts out initialized to U : ,i , which accounts for the first term in Equation 6.

Revised: In the sum from k = i + 1 to q , each term is accounted for in a separate loop of the algorithm.

Revised: We check that each term gets added at some point.

Revised: So consider the term (cid:80) ℓ : k ℓ ⊑ i l C ′ ( ℓ ) : ,k U ′ P : ,k = ℓ : ,k

Revised: U ′ P : ,k = ℓ : ,k ⊤ U P : ,i = l : ,i .

Revised: This term gets added to U : ,i when i from Algorithm 2 equals k from Equation 6, and when k from

Revised: Algorithm 2 equals i from Equation 6.

Revised: We are very sorry about this correspondence, but it would have to happen either here or above in the discussion of C ′ .

Revised: Observe that in Line 10, y p

Revised: j k takes the value ( U ′ )

Revised: P : ,i = p

Revised: j : ,i ⊤ U P : ,i = p

Revised: j : ,k .

Revised: Then, in Line 11, y p

Revised: j k gets multiplied by ( U ′ )

Revised: P : ,i = p

Revised: j : ,i and C ′ ( p j ) : ,i , and added to U ′ : ,k .

Revised: Swapping the i ’s and k ’s, and letting p j from Algorithm 2 equal ℓ from Equation 6,

Revised: Lines 10 and 11 add to U ′ : ,k the terms in Equation 6.

Revised: Thus, doing induction from i = q down to 1, Algorithm 2 assigns C ′ : ,i and U ′ : ,i the correct values.

Revised: Now, we modify Algorithm 2 for the setting where all the columns U : ,i are the same, allowing a speedup of O ( q ) .

Revised: Require: P ∈

Revised: [ r ] n × q , C ∈ R n × q , u ∈ R n

Revised: Ensure: I + L ( P, C ′ , U ′ )

Revised: =

Revised: ( I + L ( P, C, u ( 1 q ) ⊤ )) − 1 ; x = log | I + L ( P, C, u ( 1 q ) ⊤ ) | 1: x, C ′ , U ′ ← 0 , 0 ∈ R n × q , u ( 1 q ) ⊤ 2: for i ∈ ( q, q − 1 , ..., 1) do ▷ O ( nq ) time 3: p, c, u ′ ← P : ,i , C : ,i , U ′ : ,i 4:

Revised: z ← 0 ∈ R r ▷ z i will store c ( l ) (( u ′ )

Revised: p = l ) ⊤ u p = l , where c k =

Revised: c

Revised: ( l ) if p k = l 5: for j ∈ [ n ] do z p j ← z p j + c j u ′ j u j ▷ O ( n ) time 6: for j ∈ [ n ] do C ′

Revised: ji ← − c j / (1 + z p j )

Revised: ▷ O ( n ) time 7: for i ∈ [ r ] do x ← x + log(1 + z i )

Revised: ▷ O ( n ) time 8:

Revised: if i > 0 then 9: if False then ▷ This block is the slow version.

Revised: What follows below is equivalent.

Revised: 10: y ← 0 ∈ R n × i 11: for j, k ∈ [ n ] × [ i − 1] do y p

Revised: j k ← y p

Revised: j k + u ′ j u j 12: for j, k ∈ [ n ] ×

Revised: [ i − 1] do U ′ jk ← U ′ jk + C ′ ji u ′ j y p

Revised: j k 13: y ← 0 ∈ R n ▷ O ( n ) time 14:

Revised: U ′ : , ( i − 1) ← U ′ : ,i ▷ O ( n ) time 15: for j ∈ [ n ] do y p

Revised: j ← y p j + u ′ j u j ▷ O ( n ) time 16: for j ∈ [ n ] do U ′

Revised: j ( i − 1) ← U ′ j ( i − 1)

Revised: + C ′ ji u ′ j y p

Revised: j ▷ O ( n ) time return C ′ , U ′ , x

Revised: Proposition 4 ( O ( nq ) inversion) .

Revised: Algorithm 4 performs approximate inversion in O ( nq ) time.

Revised: Proof.

Revised: The fact that Algorithm 4 runs in O ( nq ) time is easily verified.

Revised: Up to Line 12, Algorithm 4 is the same as Algorithm 2, except U has been replaced with u ( 1 q ) ⊤ .

Revised: So all we have to show is that

Revised: Lines 13-16 produce the same result that Lines 10-12 would have.

Revised: Observe that at the start of the algorithm, U ′ : ,k and U ′ : ,k +1 are initialized to the same value.

Revised: Observe that Lines 11 and 12 repeat the same computation i − 1 times.

Revised: So in Lines 11 and 12, U ′ : ,k and U ′ : ,k + are updated by the same amount if i > k + 1 , and they aren’t updated at all if i ≤ k .

Revised: So the difference between U ′ : ,k and U ′ : ,k +1 comes from only the former being updated when i = k

Revised: + 1 .

Revised: Therefore, Line 14 initializes U ′ : ,k to U ′ : ,k +1 .

Revised: Then Lines 15 and 16 update U ′ : ,k with the appropriate difference; they copy Lines 11 and 12, setting k to i − 1 .

Revised: B Gradient of loss with respect to weights

Revised: In this section, we show how to calculate ∇ w NLL in O ( nq 2 ) time.

Revised: Recall:

Revised: [Equation]

Revised: Differentiating gives:

Revised: [Equation]

Revised: We begin by evaluuating ∂K XX /∂w i .

Revised: Recall from Proposition 2 that K XX

Revised: =

Revised: L ( P, C, U ) , where

Revised: C = 1 n w ⊤ , and U = 1 n × q .

Revised: It follows easily from the definition of L that the elements of L ( P, C, U ) are linear in the elements of C .

Revised: So,

Revised: [Equation]

Revised: Algorithm 3 shows how to calculate ( K XX + λ ) − 1 in O ( nq ) time, and represent it as L ( P, C − 1 , U − 1 ) .

Revised: (Recall C − 1 and U − 1 are not true inverses; we just the notation to denote their purpose.)

Revised: Now we turn to the question of how to calculate Tr[ L ( P : ,i , 1 n , 1 n ) L ( P, C − 1 , U − 1 )] .

Revised: We are considering symmetric matrices, so the trace of the product is the sum of the elements of the elementwise product.

Revised: We expand and simplify:

Revised: [Equation]

Revised: where ( a ) follows from the fact that the ( i, j ) th element of L ( p, c, u ) is zero unless p i =

Revised: p j , in which case, it is c

Revised: i u i u j ; when multiplying elementwise by L ( p ′ , c ′ , u ′ ) , where p ′ is a finer partition,

Revised: L ( p, c, u ) ⊙ L ( p ′ , c ′ , u ′ ) =

Revised: L ( p ′ , c, u ) ⊙ L ( p ′ , c ′ , u ′ ) , because L ( p, c, u ) and L ( p ′ , c, u ) only differ on elements where L ( p ′ , c ′ , u ′ ) is 0 anyway.

Revised: In the context of ( a ) , P : , max( i,j ) is a finer partition than P : , min( i,j ) .

Revised: ( b ) follows because we are doing elementwise multiplication between the two matrices; anywhere L ( P : , max( i,j ) , 1 n , 1 n ) is 0 , L ( P : , max( i,j ) , C − 1 : ,j , U − 1 : ,j ) is already 0 , and elsewhere, multiplying elements by 1 does not effect the matrix.

Revised: ( c ) follows from the construction of L .

Revised: It is straightforward to compute this in O ( nq ) time.

Revised: See Algorithm 5, which runs in O ( n ) time and can be iterated over the q terms in Equation 16.

Revised: Now we can see that Equation 9 can be computed for all w i in O ( nq 2 ) time.

Revised: First, ( K XX + λ ) −can be computed in O ( nq ) time, in the form L ( P, C − 1 , U − 1 ) , as shown in Algorithm 3.

Revised: Then, z =

Revised: ( K XX + λ ) − 1 y can be computed in O ( nq ) time, also as shown in Algorithm 3.

Revised: Then, for each i ∈ [ q ] , we can calculate − z ⊤ ∂K XX ∂w

Revised: i z = − z ⊤ L ( P : ,i , 1 n , 1 n ) z in O ( n ) time.

Revised: Thus, handling the first term in Equation 9 for all i takes a total of O ( nq ) time.

Revised: As just shown, the second term can be computed in O ( nq ) time for each i , giving a total run time of O ( nq 2 ) .

Revised: Algorithm

Revised: (cid:80) c ( ℓ ) ||

Revised: = ℓ ||

Revised: Require: p ∈

Original: In Table 5, we compare our binary tree kernel and a binary tree ensemble (see Appendix C for detailson the ensemble) against three baseline methods: exact GP regression using a Matérn kernel, sparse
Revised: [ r ] n , c, u ∈ R n Ensure: x = (cid:80) mℓ =1 c ( ℓ ) || u p = ℓ ||1: y ← 0 m 2: for j ∈ [ n ] do y p j ← y p j + √ c j u j ▷ √ c

Original: Gaussian process regression (SGPR)
Revised: j may be imaginary, but it will later be squared. With modifications, we could avoid complex types. 3: x ←4: for j ∈ [ r ] do x ← x + y 2 j return x C Experimental Details

Original: [20], and a stochastic variational Gaussian process (SVGP)[10].
Revised: We initialize 160 random bit orders. For each one, we initialize three weight vectors w : uniform, uniform except the last bit is 0.5, and uniform except the last bit is 0.9. Out of these 480 initializations, we draw 20 samples via Boltzman sampling [5] on the log likelihood of the training data (after standardizing the values to have zero mean and unit variance). Then, we optimize the weights and bit order with BFGS as described in Section 5, using line search with Wofle conditions, with no extra gradient computations during line search.

Revised: This allows fewer calculations of the gradient relative to the cheaper calculation of the loss.

Original: We evaluate our method on the same open-access UCI datasets [3] as Wang et al.
Revised: The BT column in Table 1 refers to the performance of the binary tree kernel, using the weights and bit order that gave the lowest training NLL out of these 20 trained models. BTE produces a Gaussian mixture model at each predictive location, mixing over the predictive Gaussians produced by each of these 20 trained models. The relative weights of each Gaussian in the mixture depends on the training NLL of the model that produced it.

Original: [22], usingtheir same training, validation, and test partitions, and we compare against  the baseline results theyreport.
Revised: We weight the models according to the softmax of the per-data-point NLL with a temperature of 0.01. We follow the same train/test/validation splits as Wang et al. [25], but we never use the validation set, which the methods we compare against need. Thus, we could add the validation data to the training data for the binary tree kernel and call it a fair comparison, but we didn’t do this, so as not to confuse the origin of the binary tree kernel’s success.

Original: For the binary tree (BT) kernels, we use p = min(8 , b 150 /d c + 1) , and recall q = pd .
Revised: In Table 2, we evaluate the performance of the BT and BTE kernels on the precision p for p ∈ (2 , 4 , 8) .

Original: We set = 1 /n .
Revised: In all problems, we find that RMSE and NLL decrease monotonically as p increases and wall time increases monotonically. For best predictive performance, p should be set as large as possible subject resource constraints. This validates that our heuristic rule for setting p is a reasonable choice in a variety of settings. D.2 A Simple Performance Improvement

Original: We train the  bit order and weights to minimize training NLL.
Revised: As shown in Table 3, the heuristic rule for setting p results in bit strings that preserve the uniqueness of the raw training set for most datasets. However, for the 3dRoad, Song, and Buzz datasets, the percentage of unique bit strings is very low relative to the percentage of unique training rows. We note that this observation is from exploratory data analysis (EDA) and can be made before model fitting. There are two key ways that mapping training rows to bit strings can lower the percent of unique examples.

Original: For the binary tree ensemble(BTE), we use 20 kernels.
Revised: First, a low percentage of unique bit strings can arise if a given input feature has a nonuniform input distribution which can lead to multiple different inputs mapping to the same discrete bucket. To alleviate, this problem EDA can be used to determine a suitable feature transformation. For example, we apply a strictly increasing, piecewise linear transformation to the data, mapping the k th percentile of each dimension to k/ 100 , for k ∈ { 0 , 10 , 20 , ..., 100 } . This resembles an empirical cumulative density function (ECDF).

Original: For the Matérn kernel, we use Blackbox Matrix-Matrix multiplication(BBMM)
Revised: Second, if the precision p is set too low, then multiple different inputs can map to the same bit string. A simple solution is to iteratively increase the precision p [Table] based on the difference between the percentage of unique rows in the raw training set and unique bit strings under precision p .

Original: [8], which uses the conjugate gradients method to calculate ( K XX +  I )  1 .
Revised: The last column of Table 3 reports the percentage of unique bit strings on the 3dRoad, Song [Table]

Original: SGPR uses512 data points and SVGP uses 1,024 inducing points.
Revised: and Buzz datasets after applying transforming each feature through its ECDF and increasing the precision if need be. These changes (all made through EDA) lead to significantly more unique bit strings. Table 4 shows the performance on these datasets under the proposed ECDF transformations and settings of p .

Original: We report the mean and two standard errorsacross 3 replications with different dataset splits.
Revised: We find that the BTE outperforms all methods on 3dRoad and Buzz with respect to RSME and NLL under these proposed changes.

Original: For further experimental details, see Appendix C.
Revised: The performance on Song also improves. [Table] [Table]

Original:  BTE achieves the best NLL on 6/12 datasets, and best RMSE on 5/12 datasets (including some ties).
Revised: a single GPU (Tesla V100-SXM2-16GB for BT and BTE and Tesla V100-SXM2-32GB for the other methods) after transforming each feature through its ECDF and using precision p new .

Original: Out of the 4 largest datasets, BT/BTE is fastest on 3. The run times are plotted in Figure 5.
Revised: The asterisk indicates an estimate of the time from the reported training time on 8 GPUS, assuming linear speedup in number of GPUs and independent noise in training times per GPU.

Original: The codeis included in the supplementary material, and we will open source our code upon publication.
Revised: Here, we compare against Simplex-GP [15] using the 4 datasets that are common to both papers.

Original: BT performs noticeably worse than BTE for test NLL  on CTSlice due to over-ﬁtting.
Revised: We find that BTE outperforms Simplex GP on all datasets with respect to NLL and on 3 out of 4 datasets with respect to RMSE.

Original: There areenough degrees of freedom when optimizing the bit order ( d = 385 ) that BT kernel can over-ﬁt tothe training data. The ensemble over multiple bit orders is much more robust.
Revised: Furthermore, Simplex-GP is not the top performing method on any dataset.

Original: [Table]
Revised: [Table]

Original: Discussion
Revised: E Societal Impacts

Original: 5: Run times given dataset size.
Revised: More accurate machine learning models allow for better decision making.

Original: For BT, the trendline is calculated controlling for log( q ) with afﬁne regression, and then setting q = 150 . The slope w.r.t. log( q ) is 2 .
Revised: Improvements in decision making can lead to improved societal outcomes in many applications such as early disease detection.

Original: 82 ± 1 . 06 . Theoretically, all slopes are too low except for that of SVGP, presumably because of overhead in the small-data regime.
Revised: However when predictions of a machine learning algorithm are confident, we may be more compelled to act on them, which could lead to increased risk in delicate settings. Therefore, confident and wrong predictions, resulting from poor and confident generalization due to overfitting, can lead to worse outcomes.

Original: We have proven that the binary tree kernel GP is scalable.
Revised: When using binary tree kernels for Gaussian processes, overfitting is possible.

Original: Our empirical resultssuggest that it often outpredicts not justother scalable methods, but even the popular Matérn GP. If the results in this paperreplicate in other domains, it could obviatewide usage of classic GP kernels like the Matérn kernel, as well as inducing pointkernels. Sometimes, our kernel fails to capture patterns in the data; some functions’values simply do not covary this way.
Revised: That said, Gaussian processes do quantify their uncertainty very naturally, which can be helpful for knowing when to take their predictions with a grain, or a heap, of salt.

Original: Butother kernels we tested seemed to fail likethat even more. The SROS representation of linear operators, which we developed for this kernel, isconceptually general, and could potentiallybe useful in other domains where linear algebra is applied. Low-rank approximationsare widely used  to speed up linear transformations, and SROS representations mightbe an alternative in many settings. Our contributions to linear algebra and kernel design may signiﬁcantly increase the size of data setson which GPs can do state-of-the-art modelling.
Revised: As with all machine learning models, what outcomes are predicted and how those predictions are used are ultimately decided by to the practitioner, and those decisions contribute to whether the model impacts society in positive or negative ways.