Original: Log-Linear-Time Gaussian Processes Using Binary Tree Kernels
Revised: Log-Linear-Time Gaussian Processes Using Binary Tree Kernels

Original: Abstract
Revised: Abstract

Original: Gaussian processes (GPs) produce good probabilistic models of functions, but most GP kernels require O (( n + m ) n 2 ) time, where n is the number of data points and m the number of predictive locations.
Revised: Gaussian processes (GPs) produce good probabilistic models of functions, but most GP kernels require O (( n + m ) n 2 ) time, where n is the number of data points and m the number of predictive locations.

Original: We present a new kernel that allows for
Revised: We present a new kernel that allows for

Original: Gaussian process regression in O (( n + m ) log( n + m )) time.
Revised: Gaussian process regression in O (( n + m ) log( n + m )) time.

Original: Our “binary tree” kernel places all data points on the leaves of a binary tree, with the kernel depending only on the depth of the deepest common ancestor.
Revised: Our “binary tree” kernel places all data points on the leaves of a binary tree, with the kernel depending only on the depth of the deepest common ancestor.

Original: We can store the resulting kernel matrix in O ( n ) space in O ( n log n ) time, as a sum of sparse rank-one matrices, and approximately invert the kernel matrix in O ( n ) time.
Revised: We can store the resulting kernel matrix in O ( n ) space in O ( n log n ) time, as a sum of sparse rank-one matrices, and approximately invert the kernel matrix in O ( n ) time.

Original: Sparse GP methods also offer linear run time, but they predict less well than higher dimensional kernels.
Revised: Sparse GP methods also offer linear run time, but they predict less well than higher dimensional kernels.

Original: On a classic suite of regression tasks, we compare our kernel against Matérn, sparse, and sparse variational kernels.
Revised: On a classic suite of regression tasks, we compare our kernel against Matérn, sparse, and sparse variational kernels.

Original: The binary tree GP assigns the highest likelihood to the test data on a plurality of datasets, usually achieves lower mean squared error than the sparse methods, and often ties or beats the Matérn GP.
Revised: The binary tree GP assigns the highest likelihood to the test data on a plurality of datasets, usually achieves lower mean squared error than the sparse methods, and often ties or beats the Matérn GP.

Original: On large datasets, the binary tree GP is fastest, and much faster than a Matérn GP.
Revised: On large datasets, the binary tree GP is fastest, and much faster than a Matérn GP.

Original:  Introduction
Revised:  Introduction

Original: Gaussian processes (GPs) can be used to perform regression with high-quality uncertainty estimates, but they are slow.
Revised: Gaussian processes (GPs) can be used to perform regression with high-quality uncertainty estimates, but they are slow.

Original: Naïvely, GP regression requires O ( n 3 + n 2 m ) computation time and O ( n 2 ) computation space when predicting at m locations given n data points [28].
Revised: Naïvely, GP regression requires O ( n 3 + n 2 m ) computation time and O ( n 2 ) computation space when predicting at m locations given n data points [28].

Original: A kernel matrix of size n × n must be inverted (or Cholesky decomposed), and then m matrix-vector multiplications must be done with that inverse matrix (or m linear solves with the Cholesky factors).
Revised: A kernel matrix of size n ⇥ n must be inverted (or Cholesky decomposed), and then m matrix-vector multiplications must be done with that inverse matrix (or m linear solves with the Cholesky factors).

Original: A few methods that we will discuss later achieve O ( n 2 m ) time complexity [25, 30].
Revised: A few methods that we will discuss later achieve O ( n 2 m ) time complexity [25, 30].

Original: With special kernels, GP regression can be faster and use less space.
Revised: With special kernels, GP regression can be faster and use less space.

Original: Inducing point methods, using z inducing points, allow regression to be done in O ( z 2 ( n + m )) time and in O ( z 2 + zn ) space [21, 22, 23, 13].
Revised: Inducing point methods, using z inducing points, allow regression to be done in O ( z 2 ( n + m )) time and in O ( z 2 + zn ) space [21, 22, 23, 13].

Original: We will discuss the details of these inducing point kernels later, but they are kernels in their own right, not just approximations to other kernels.
Revised: We will discuss the details of these inducing point kernels later, but they are kernels in their own right, not just approximations to other kernels.

Original: Unfortunately, these kernels are low dimensional (having a z -dimensional Hilbert space), which limits the expressivity of the GP model.
Revised: Unfortunately, these kernels are low dimensional (having a z -dimensional Hilbert space), which limits the expressivity of the GP model.

Original: We present a new kernel, the binary tree kernel , that also allows for GP regression in O ( n + m ) space and O (( n + m ) log( n + m )) time (both model fitting and prediction).
Revised: We present a new kernel, the binary tree kernel , that also allows for GP regression in O ( n + m ) space and O (( n + m ) log( n + m )) time (both model ﬁtting and prediction).

Original: The time and space complexity of our method is also linear in the depth of the binary tree, which is naïvely linear in the dimension of the data, although in practice we can increase the depth sublinearly.
Revised: The time and space complexity of our method is also linear in the depth of the binary tree, which is naïvely linear in the dimension of the data, although in practice we can increase the depth sublinearly.

Original: Training some kernel parameters takes time quadratic in the depth of the tree.
Revised: Training some kernel parameters takes time quadratic in the depth of the tree.

Original: The dimensionality of the binary tree kernel is exponential in the depth of the tree, making it much more expressive than an inducing points kernel.
Revised: The dimensionality of the binary tree kernel is exponential in the depth of the tree, making it much more expressive than an inducing points kernel.

Original: Whereas for an inducing points kernel, the runtime is quadratic in the dimension of the
Revised: Whereas for an inducing points kernel, the runtime is quadratic in the dimension of the

Original: Hilbert space, for the binary tree kernel, it is only logarithmic—an exponential speedup.
Revised: Hilbert space, for the binary tree kernel, it is only logarithmic—an exponential speedup.

Original: A simple depiction of our kernel is shown in Figure 1, which we will define precisely in Section 3.
Revised: A simple depiction of our kernel is shown in Figure 1, which we will deﬁne precisely in Section 3.

Original: First, we create a procedure for placing all data points on the leaves of a binary tree.
Revised: First, we create a procedure for placing all data points on the leaves of a binary tree.

Original: Given the binary tree, the kernel between two points depends only on the depth of the deepest common ancestor.
Revised: Given the binary tree, the kernel between two points depends only on the depth of the deepest common ancestor.

Original: Because very different tree structures are possible for the data, we can easily form an ensemble of diverse GP regression models.
Revised: Because very different tree structures are possible for the data, we can easily form an ensemble of diverse GP regression models.

Original: Figure 2 depicts a schematic sample from a binary tree kernel.
Revised: Figure 2 depicts a schematic sample from a binary tree kernel.

Original: Note how the posterior mean is piecewise flat, but the pieces can be small.
Revised: Note how the posterior mean is piecewise ﬂat, but the pieces can be small.

Original: In this example, k ( x 1 , x 1 ) = 1 , k ( x 1 , x 2 ) = 0 , k ( x 1 , x 3 )
Revised: In this example, k ( x 1 , x 1 ) = 1 , k ( x 1 , x 2 ) = 0 , k ( x 1 , x 3 )

Original: = 0 . 8 , and k ( x 1 , x 4 )
Revised: = 0 . 8 , and k ( x 1 , x 4 )

Original: = 0 .
Revised: = 0 .

Original: On a standard suite of benchmark regression tasks [25], we show that our kernel usually achieves better negative log likelihood (NLL) than state-of-the-art sparse methods and conjugate-gradient-based “exact” methods, at lower computational cost in the big-data regime.
Revised: On a standard suite of benchmark regression tasks [25], we show that our kernel usually achieves better negative log likelihood (NLL) than state-of-the-art sparse methods and conjugate-gradient-based “exact” methods, at lower computational cost in the big-data regime.

Original: There are not many limitations to using our kernel.
Revised: There are not many limitations to using our kernel.

Original: The main limitation is that other kernels sometimes capture the relationships in the data better.
Revised: The main limitation is that other kernels sometimes capture the relationships in the data better.

Original: We do not have a good procedure for understanding when data has more Matérn character or more binary tree character (except through running both and comparing training NLL).
Revised: We do not have a good procedure for understanding when data has more Matérn character or more binary tree character (except through running both and comparing training NLL).

Original: But given that the binary tree kernel usually outperforms the Matérn, we’ll tentatively say the best first guess is that a new dataset has more binary tree character.
Revised: But given that the binary tree kernel usually outperforms the Matérn, we’ll tentatively say the best ﬁrst guess is that a new dataset has more binary tree character.

Original: One concrete limitation for some applications, like Bayesian Optimization, is that the posterior mean is piecewise-flat, so gradient-based heuristics for finding extrema would not work.
Revised: One concrete limitation for some applications, like Bayesian Optimization, is that the posterior mean is piecewise-ﬂat, so gradient-based heuristics for ﬁnding extrema would not work.

Original: In contexts where a piecewise-flat posterior mean is suitable, we struggle to see when one would prefer a sparse or sparse variational GP to a binary tree kernel.
Revised: In contexts where a piecewise-ﬂat posterior mean is suitable, we struggle to see when one would prefer a sparse or sparse variational GP to a binary tree kernel.

Original: The most thorough approach would be to run both and see which has a better training NLL, but if you had to pick one, the binary tree
Revised: The most thorough approach would be to run both and see which has a better training NLL, but if you had to pick one, the binary tree

Original: GP seems to be better performing and comparably fast.
Revised: GP seems to be better performing and comparably fast.

Original: If minimizing mean-squared error is the objective, the Matérn kernel seems to do slightly better than the binary tree.
Revised: If minimizing mean-squared error is the objective, the Matérn kernel seems to do slightly better than the binary tree.

Original: If the dataset is small, and one needs a very fast prediction, a Matérn kernel may be the best option.
Revised: If the dataset is small, and one needs a very fast prediction, a Matérn kernel may be the best option.

Original: But otherwise, if one cares about well-calibrated predictions, these initial results we present tentatively suggest using a binary tree kernel over the widely-used Matérn kernel.
Revised: But otherwise, if one cares about well-calibrated predictions, these initial results we present tentatively suggest using a binary tree kernel over the widely-used Matérn kernel.

Original: The log-linear time and linear space complexity of the binary tree GP, with performance exceeding a “normal” kernel, could profoundly expand the viability of GP regression to larger datasets.
Revised: The log-linear time and linear space complexity of the binary tree GP, with performance exceeding a “normal” kernel, could profoundly expand the viability of GP regression to larger datasets.

Original:  Preliminaries
Revised:  Preliminaries

Original: Our problem setting is regression.
Revised: Our problem setting is regression.

Original: Given a function f : X → R , for some arbitrary set X , we would like to predict f ( x ) for various x ∈ X .
Revised: Given a function f : X ! R , for some arbitrary set X , we would like to predict f ( x ) for various x 2 X .

Original: What we have are observations of f ( x ) for various (other) x ∈ X .
Revised: What we have are observations of f ( x ) for various (other) x 2 X .

Original: Let X ∈ X n be an n -tuple of elements of X , and let y ∈ R n be an n -tuple of real numbers, such that y i ∼ f
Revised: Let X 2 X n be an n -tuple of elements of X , and let y 2 R n be an n -tuple of real numbers, such that y i ⇠ f

Original: ( X i )
Revised: ( X i )

Original: + N (0 , λ ) , for λ ∈ R ≥ 0 .
Revised: + N (0 ,  ) , for  2 R  0 .

Original: X and y comprise our training data.
Revised: X and y comprise our training data.

Original: With an m -tuple of test locations X ′ ∈ X m , let y ′ ∈{+0 2 R m , with y 
+}
Revised: With an m -tuple of test locations X 0 2 X m , let y {+0 2 R m , with y 
+}

Original: R m , with y ′ i = 
Revised: i = f

Original: f ( X ′ i ) .
Revised: ( X 0 i ) .

Original: y ′ is the ground truth for the target locations.
Revised: y 0 is the ground truth for the target locations.

Original: Given training data, we would like to produce a distribution over R for each target location X ′ i , such that it assigns high marginal probability to the unknown y ′ i .
Revised: Given training data, we would like to produce a distribution over R for each target location X 0 i , such that it assigns high marginal probability to the unknown y 0 i .

Original: Alternatively, we sometimes would like to produce point estimates ˆ y ′ i in order to minimize the squared error (ˆ y ′{+
+}
Revised: Alternatively, we sometimes would like to produce point estimates ˆ y 0 i in order to minimize the squared error (ˆ y {+
+}

Original: i − y ′ i ) 2 .
Revised: i  y 0 i ) 2 .

Original: A GP prior over functions is defined by a mean function m : X → R , and a kernel k : X × X 
Revised: A GP prior over functions is deﬁned by a mean function m : X ! R , and a kernel k : X ⇥ X !

Original: → R .
Revised: R .

Original: The expected function value at a point x is defined to be m ( x ) , and the covariance of the function values at two points x 1 and x 2 is defined to be k ( x 1 , x 2 ) .
Revised: The expected function value at a point x is deﬁned to be m ( x ) , and the covariance of the function values at two points x 1 and x 2 is deﬁned to be k ( x 1 , x 2 ) .

Original: Let K XX ∈ R n × n be the matrix of kernel values ( K XX ) ij =
Revised: Let K XX 2 R n ⇥ n be the matrix of kernel values ( K XX ) ij =

Original: k
Revised: k

Original: ( X i , X j ) , and let m X ∈ R n be the vector of mean values ( m X ) i
Revised: ( X i , X j ) , and let m X 2 R n be the vector of mean values ( m X ) i

Original: = m ( X i ) .
Revised: = m ( X i ) .

Original: For a GP to be well-defined, the kernel must be such that K XX is positive semidefinite for any
Revised: For a GP to be well-deﬁned, the kernel must be such that K XX is positive semideﬁnite for any

Original: X ∈ X n .
Revised: X 2 X n .

Original: For a point x ∈ X , Let K Xx ∈ R n be the vector of kernel values: ( K Xx )
Revised: For a point x 2 X , Let K Xx 2 R n be the vector of kernel values: ( K Xx )

Original: i =
Revised: i =

Original: k ( X i , x ) , and let K xX =
Revised: k ( X i , x ) , and let K xX =

Original: K ⊤ Xx .
Revised: K > Xx .

Original: Let λ ≥ 0 be the variance of observation noise.
Revised: Let   0 be the variance of observation noise.

Original: Let µ x and σ 2 x be the mean and variance of our posterior predictive distribution at x .
Revised: Let µ x and  2 x be the mean and variance of our posterior predictive distribution at x .

Original: Then, with K λ inv XX =
Revised: Then, with K  inv XX =

Original: ( K XX + λI ) − 1 , µ x :=
Revised: ( K XX +  I )  1 , µ x :=

Original: ( y − m X ) 
Revised: ( y  m X ) >

Original: ⊤ K λ inv XX K Xx+ m
Revised: K  inv [Equation] Xx

Original: [Equation]
Revised: [Equation]

Original:  See Williams and Rasmussen [28] for a derivation.
Revised: m [Equation] See Williams and Rasmussen [28] for a derivation.

Original: We compute Equations 1 and 2 for all x ∈ X ′ .
Revised: We compute Equations 1 and 2 for all x 2 X 0 .

Original:  Binary tree kernel
Revised:  Binary tree kernel

Original: We now introduce the binary tree kernel.
Revised: We now introduce the binary tree kernel.

Original: First, we encode our data points as binary strings.
Revised: First, we encode our data points as binary strings.

Original: So we have X = B q , where B = { 0 , 1 } , and q ∈ N .
Revised: So we have X = B q , where B = { 0 , 1 } , and q 2 N .

Original: If X = R d , we must map R d (cid:55)→ B q .
Revised: If X = R d , we must map R d 7! B q .

Original: First, we rescale all points (training points and test points) to lie within the box [0 , 1] d .
Revised: First, we rescale all points (training points and test points) to lie within the box [0 , 1] d .

Original: (If we have a stream of test points, and one lands outside of the box [0 , 1] d , we can either set K xX to 0 for that point or we rescale and retrain in O ( n log n ) time.)
Revised: (If we have a stream of test points, and one lands outside of the box [0 , 1] d , we can either set K xX to 0 for that point or we rescale and retrain in O ( n log n ) time.)

Original: Then, for each x ∈ [0 , 1] d , for each dimension, we take the binary expansion up to some precision p , and for those d × p bits, we permute them using some fixed permutation.
Revised: Then, for each x 2 [0 , 1] d , for each dimension, we take the binary expansion up to some precision p , and for those d ⇥ p bits, we permute them using some ﬁxed permutation.

Original: We call this permutation the bit order, and it is the same for all x ∈ [0 , 1] d .
Revised: We call this permutation the bit order, and it is the same for all x 2 [0 , 1] d .

Original: Note that now q = dp .
Revised: Note that now q = dp .

Original: See Figure 3 for an example.
Revised: See Figure 3 for an example.

Original: We optimize the bit order during training, and we can also form an ensemble of GPs using different bit orders. 
Revised: We optimize the bit order during training, and we can also form an ensemble of GPs using different bit orders. [Equation]

Original: For x ∈ B q , let x ≤ i be the first i bits of x .
Revised: For x 2 B q , let x  i be the ﬁrst i bits of x .

Original: [[ expression ]] evaluates to 1, if expression is true, otherwise 0.
Revised: [[ expression ]] evaluates to 1, if expression is true, otherwise 0.

Original: We now define the kernel:
Revised: We now deﬁne the kernel:

Original: Definition 1 (Binary Tree Kernel) .
Revised: Deﬁnition 1 (Binary Tree Kernel) .

Original: Given a weight vector w ∈ R q , with w ⪰ 0 and || w || 1 = 1 ,
Revised: Given a weight vector w 2 R q , with w ⌫ 0 and || w || 1 = 1 ,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: So the more leading bits shared by x 1 and x 2 , the larger the covariance between the function values.
Revised: So the more leading bits shared by x 1 and x 2 , the larger the covariance between the function values.

Original: Consider, for example, points xand x 4 from Figure 1, where x 1 is ( left, left, right ) , and x 4 is ( left, right, right ) ; they share only the first leading “bit”.
Revised: Consider, for example, points xand x 4 from Figure 1, where x 1 is ( left, left, right ) , and x 4 is ( left, right, right ) ; they share only the ﬁrst leading “bit”.

Original: We train the weight vector w to maximize the likelihood of the training data.
Revised: We train the weight vector w to maximize the likelihood of the training data.

Original: Proposition 1 (Positive Semidefiniteness) .
Revised: Proposition 1 (Positive Semideﬁniteness) .

Original: For X ∈ X n , for k = k w , K XX ⪰ 0 .
Revised: For X 2 X n , for k = k w , K XX ⌫ 0 .

Original: Proof.
Revised: Proof.

Original: Let s ∈ (cid:83) qi 
Revised: Let s S qi =

Original: =1 B i be a binary string, and let | s | be the length of s .
Revised: 1 B i be a binary string, and let | s | be the length of s .

Original: Let X [ s ] ∈ R n with ( X [ s ] )j = (cid:104)(cid:104)
Revised: Let X [ s ] 2 R n with ( X [ s ] )

Original:  X ≤| s | j = s (cid:105)(cid:105) .
Revised: j = hh X  | s | j = s ii .

Original: X [ s ] X ⊤ [ s ] is clearly positive semidefinite.
Revised: X [ s ] X > [ s ] is clearly positive semideﬁnite.

Original: Finally, K XX =(cid:80) qi
Revised: Finally, K XX =

Original: =1 (cid:80) s ∈ B i w i X
Revised: P qi = P s 2 B i w i X

Original: [ s ] X ⊤ [ s ] , and recall w i ≥ 0 , so K XX ⪰ 0 .
Revised: [ s ] X > [ s ] , and recall w i  0 , so K XX ⌫ 0 .

Original:  Sparse rank one sum representation
Revised:  Sparse rank one sum representation

Original: In order to do GP regression in O ( n ) space and O ( n log n ) time, we develop a “Sparse Rank One
Revised: In order to do GP regression in O ( n ) space and O ( n log n ) time, we develop a “Sparse Rank One

Original: Sum” representation of linear operators (SROS).
Revised: Sum” representation of linear operators (SROS).

Original: This was developed separately from the very similar
Revised: This was developed separately from the very similar

Original: Hierarchical matrices [1], which we discuss below.
Revised: Hierarchical matrices [1], which we discuss below.

Original: In SROS form, linear transformation of a vector can be done in O ( n ) time instead of O ( n 2 ) .
Revised: In SROS form, linear transformation of a vector can be done in O ( n ) time instead of O ( n 2 ) .

Original: We will store our kernel matrix and inverse kernel matrix in SROS form.
Revised: We will store our kernel matrix and inverse kernel matrix in SROS form.

Original: The proof of Proposition 1 exemplifies representing a matrix as the sum of sparse rank one matrices.
Revised: The proof of Proposition 1 exempliﬁes representing a matrix as the sum of sparse rank one matrices.

Original: Note that each X [ s ] is sparse—if q is large, most X [ s ] ’s are the zero vector.
Revised: Note that each X [ s ] is sparse—if q is large, most X [ s ] ’s are the zero vector.

Original: We now show how to interpret an SROS representation of an n × n matrix.
Revised: We now show how to interpret an SROS representation of an n ⇥ n matrix.

Original: Let [ n ] = { 1 , 2 , ..., n } .
Revised: Let [ n ] = { 1 , 2 , ..., n } .

Original: For r ∈ N , let L : [ r ] n × [ r ] n × R n × R n → R n × n construct a linear operator from four vectors.
Revised: For r 2 N , let L : [ r ] n ⇥ [ r ] n ⇥ R n ⇥ R n ! R n ⇥ n construct a linear operator from four vectors.

Original: Definition 2 (Linear Operator from Simple SROS Representation) .
Revised: Deﬁnition 2 (Linear Operator from Simple SROS Representation) .

Original: Let p, p ′ ∈ [ r ] n , and let u, u ′ ∈{+
+}
Revised: Let p, p 0 2 [ r ] n , and let u, u {+
+}

Original: R n .
Revised: R n .

Original: For l ∈ [ r ] , let u p = l ∈ R n be the vector where u p = l j = u j [[ p j = l ]] , likewise for u ′ and p ′ .
Revised: For l 2 [ r ] , let u p = l 2 R n be the vector where u p = l j = u j [[ p j = l ]] , likewise for u 0 and p 0 .

Original: Then:
Revised: Then:

Original: L ( p, p ′ , u, u ′ ) 
Revised: L ( p, p 0 , u, u 0 ) 7!

Original: (cid:55)→ (cid:80) mi =1 u p = i ( u ′ ) p ′ =
Revised: [Equation]

Original: i ⊤ . We depict Definition 2 in Figure 4.
Revised: We depict Deﬁnition 2 in Figure 4.

Original: p and p ′ represent partitions over n elements: all elements with the same integer value in the vector p belong to the same partition.
Revised: p and p 0 represent partitions over n elements: all elements with the same integer value in the vector p belong to the same partition.

Original: Note that r , the number of parts in the partition, need not exceed n , the number of elements being partitioned.
Revised: Note that r , the number of parts in the partition, need not exceed n , the number of elements being partitioned.

Original: If p = p ′ (which is almost always the case for us) and the elements of p , u , and u ′ were shuffled so that all elements in the same partition were next to each other, then L ( p, p ′ , u, u ′ ) would be block diagonal.
Revised: If p = p 0 (which is almost always the case for us) and the elements of p , u , and uwere shufﬂed so that all elements in the same partition were next to each other, then L ( p, p 0 , u, u 0 ) would be block diagonal.

Original: Note that L ( p, p ′ , u, u ′ ) is not necessarily low rank.
Revised: Note that L ( p, p 0 , u, u 0 ) is not necessarily low rank.

Original: If p is the finest possible partition, and p = p ′ , L ( p, p ′ , u, u ′ ) is diagonal.
Revised: If p is the ﬁnest possible partition, and p = p 0 , L ( p, p 0 , u, u 0 ) is diagonal.

Original: SROS matrices can be thought of as a generalization of two types of matrix that are famously amenable to fast computation: rankone matrices (all points in the same partition) and diagonal matrices(each point in itsown partition).
Revised: SROS matrices can be thought of as a generalization of two types of matrix that are famously amenable to fast computation: rank matrices its

Original: We now extend the definition of L to allow for multiple p , p ′ , u , and u ′ vectors.
Revised: We now extend the deﬁnition of L to allow for multiple p , p 0 , u , and u 0 vectors.

Original: Definition 3 (Linear Operator from SROS Representation) .
Revised: Deﬁnition 3 (Linear Operator from SROS Representation) .

Original: Let L : [ r ] n × q ×
Revised: Let L : [ r ] n ⇥ q ⇥

Original: [ r ] n × q × R n × q × R n × q → R n × n .
Revised: [ r ] n ⇥ q ⇥ R n ⇥ q ⇥ R n ⇥ q ! R n ⇥ n .

Original: Let P, P ′ ∈{+
+}
Revised: Let P, P {+
+}

Original: [ r ] n × q , and let U, U ′ ∈ R n × q .
Revised: [ r ] n ⇥ q , and let U, U 0 2 R n ⇥ q .

Original: Let P : ,i , U : ,i , etc.
Revised: Let P : ,i , U : ,i , etc.

Original: be the i th columns of the respective arrays.
Revised: be the i th columns of the respective arrays.

Original: Then:
Revised: Then:

Original: L ( P, P ′ , U, U ′ ) 
Revised: L ( P, P 0 , U, U 0 ) 7! P

Original: (cid:55)→ (cid:80) qi
Revised: [Equation]

Original: =1 L ( P : ,i , P ′ : ,i , U : ,i , U ′ : ,i ) . Algorithm 1 performs linear transformation of a vector using SROS representation in O ( nq ) time.
Revised: Algorithm 1 performs linear transformation of a vector using SROS representation in O ( nq ) time.

Original: Algorithm 1 Linear Transformation with SROS Linear Operator.
Revised: with SROS Linear Operator.

Original: This can be vectorized on a
Revised: This can be vectorized on a

Original:  Unit  for Line 6 [19].
Revised: Graphical Processing Unit (GPU), using e.g. torch. Tensor.index_add_ for Line 5 and non-slice indexing for Line 6 [19].

Original: Slight restructuring allows vectorization over [ q ] as well.
Revised: Slight restructuring allows vectorization over [ q ] as well.

Original: Require: P, P ′ ∈ [ r ] n × q , U, U ′ ∈ R n × q , x ∈ R n
Revised: Require: P, P 0 2 [ r ] n ⇥ q , U, U 0 2 R n ⇥ q , x 2 R n

Original: Ensure:
Revised: Ensure:

Original: y =
Revised: y =

Original: L ( P, P ′ , U, U ′ )
Revised: L ( P, P 0 , U, U 0 )

Original: x 1: y← 0 ∈ R n 2: for i ∈ [ q ] do ▷ O ( nq ) time 3: p, p ′ , u, u′ ← P : ,i , P ′ : ,i , U : ,i , U ′ : ,i 4: 
Revised: x 1: y 0 2 R n 2: for i 2 [ q ] do . O ( nq ) time 3: p, p 0 , u, u P : ,i , P 0 : ,i , U : ,i , U 0 : ,i 4: z 0 2 R r .

Original: z← 0 ∈ R r ▷ z l will store the dot product (( u ′ ) p ′ = l ) ⊤ x p ′{+
+}
Revised: z l will store the dot product (( u 0 ) p 0 = l ) > x p {+
+}

Original: = l 5: for j ∈ [ n ] do z p ′{+
 j z p 
+}
Revised: = l 5: for j 2 [ n ] do z p {+
 j z p 
+}

Original: j← z p ′ j + u ′ j x j ▷ O ( n ) time 6: for j ∈ [ n ] do y j← y j
Revised: j + u 0 j x j . O ( n ) time 6: for j 2 [ n ] do y j y j

Original: + z p j u j ▷ O ( n ) time return y
Revised: + z p j u j . O ( n ) time return y

Original: We now discuss how to approximately invert a certain kind of symmetric SROS matrix, but our methods could be extended to asymmetric matrices.
Revised: We now discuss how to approximately invert a certain kind of symmetric SROS matrix, but our methods could be extended to asymmetric matrices.

Original: First, we define a partial ordering over partitions.
Revised: First, we deﬁne a partial ordering over partitions.

Original: For two partitions p, p ′ , we say p ′ ≤ p if p ′ is finer than or equal to p ; that is, p ′{+
+}
Revised: For two partitions p, p 0 , we say p 0  p if p 0 is ﬁner than or equal to p ; that is, p {+
+}

Original: j = p ′{+
+}
Revised: j = p {+
+}

Original: j ′ = ⇒ p
Revised: j 0 = ) p

Original: j = p j ′ .
Revised: j = p j 0 .

Original: Using that partial ordering, a symmetric SROS matrix can be approximately inverted efficiently if for all 1 ≤ i, i ′ ≤ q , P : ,i ≤ P : ,i ′ or P : ,i ′ ≤ P : ,i .
Revised: Using that partial ordering, a symmetric SROS matrix can be approximately inverted efﬁciently if for all 1  i, i 0  q , P : ,i  P : ,i 0 or P : ,i 0  P : ,i .

Original: As the reader may have recognized, our kernel matrix 
Revised: As the reader may have recognized, our kernel matrix K

Original: K XX can be written as an SROS matrix with this property.
Revised: XX can be written as an SROS matrix with this property.

Original: We will write symmetric SROS matrices in a slightly more convenient form.
Revised: We will write symmetric SROS matrices in a slightly more convenient form.

Original: All ( u ′ ) p
Revised: All ( u 0 ) p

Original: = l must be a constant times u p = l .
Revised: = l must be a constant times u p = l .

Original: We will store these constants in an array C .
Revised: We will store these constants in an array C .

Original: Let L ( P, C, U ) be shorthand for
Revised: Let L ( P, C, U ) be shorthand for

Original: L ( P, P, U, C ⊙ U ) , where ⊙ denotes element-wise multiplication.
Revised: L ( P, P, U, C  U ) , where  denotes element-wise multiplication.

Original: For L ( P, C, U ) to be symmetric, it must be the case that P ji = P j ′ i = 
Revised: For L ( P, C, U ) to be symmetric, it must be the case that P ji = P j 0 i = )

Original: ⇒ C ji = C
Revised: C ji = C

Original: j ′ i .
Revised: j 0 i .

Original: Then, all elements of U corresponding to a given u p =
Revised: Then, all elements of U corresponding to a given u p =

Original: l are multiplied by the same constant.
Revised: l are multiplied by the same constant.

Original: We now present an algorithm for calculating ( L ( P, C, U ) + λI ) − 1 , for λ ̸ = 0 , which is an approximate inversion of L ( P, C, U ) .
Revised: We now present an algorithm for calculating ( L ( P, C, U ) +  I )  1 , for  6 = 0 , which is an approximate inversion of L ( P, C, U ) .

Original: We have not yet analyzed numerical sensitivity for λ → 0 , but we conjecture that all floating point numbers involved need to be stored to at least log 2 (1 /λ ) bits.
Revised: We have not yet analyzed numerical sensitivity for  ! 0 , but we conjecture that all ﬂoating point numbers involved need to be stored to at least log 2 (1 /  ) bits.

Original: Without loss of generality, let λ = 1 , and note ( L ( P, C, U ) + λI ) − 1{+ 
+} = λ − 1 ( L ( P, λ − 1 C, U ) + I ) − 1 .
Revised: Without loss of generality, let  = 1 , and note ( L ( P, C, U ) +  I ) {+ 
+} =   1 ( L ( P,   1 C, U ) + I )  1 .

Original: By assumption, all columns of P are comparable with respect to the partial ordering above, so we can reorder the columns of P such that P : ,i ≥ P : ,j for i < j .
Revised: By assumption, all columns of P are comparable with respect to the partial ordering above, so we can reorder the columns of P such that P : ,i  P : ,j for i < j .

Original: The key identity that we use to develop our fast inversion algorithm is the Sherman–Morrison Formula:
Revised: The key identity that we use to develop our fast inversion algorithm is the Sherman–Morrison Formula:

Original: [Equation]
Revised: [Equation]

Original: Starting with A
Revised: Starting with A

Original: =
Revised: =

Original: I , we add the sparse rank one matrices iteratively, from the finest partition to the coarsest one, updating A − 1 as we go.
Revised: I , we add the sparse rank one matrices iteratively, from the ﬁnest partition to the coarsest one, updating A  1 as we go.

Original: We represent ( L ( P, C, U )+ I ) − 1 in the form I + L ( P, C ′ , U ′ ) , so we write an algorithm that returns C ′ and U ′ .
Revised: We represent ( L ( P, C, U )+ I )  1 in the form I + L ( P, C 0 , U 0 ) , so we write an algorithm that returns C 0 and U 0 .

Original: We can also quickly calculate log | L ( P, C, U )
Revised: We can also quickly calculate log | L ( P, C, U )

Original: + I | at the same time, using the matrix determinant lemma: | A + cuu ⊤ | = (1 + cu ⊤ A − 1 u ) | A | .
Revised: + I | at the same time, using the matrix determinant lemma: | A + cuu > | = (1 + cu > A  1 u ) | A | .

Original: Theorem 1 (Fast Inversion) .
Revised: Theorem 1 (Fast Inversion) .

Original: For P ∈ [ r ] n × q and C, U ∈ R n × q , if P : ,i (is coarser than) ≥
Revised: For P 2 [ r ] n ⇥ q and C, U 2 R n ⇥ q , if P : ,i (is coarser than)

Original: P : ,j for i <
Revised: P : ,j for i <

Original: j , then there exists C ′ , U ′ ∈ R n × q , such that ( L ( P, C, U ) + I ) − 

Revised: j , then there exists C 0 , U 0 2 R n ⇥ q , such that ( L ( P, C, U ) + I )

Original: = I + L ( P, C ′ , U ′ ) .
Revised: = I + L ( P, C 0 , U 0 ) .

Original: There exists an algorithm for computing C ′{+
+} and U ′ that takes O ( nq 2 ) time.
Revised: There exists an algorithm for computing C {+
+} and U 0 that takes O ( nq 2 ) time.

Original: Proof.
Revised: Proof.

Original: For X ∈ R n × q , let X : ,i +1: q ∈ R n × ( q − i ) be columns i + 1 through q of matrix X (inclusive).
Revised: For X 2 R n ⇥ q , let X : ,i +1: q 2 R n ⇥ ( q  i ) be columns i + 1 through q of matrix X (inclusive).

Original: Let A i
Revised: Let A i

Original: = I + L ( P : ,i +1: q , C : ,i +1: q , U : ,i +1: q ) , and A q = I .
Revised: = I + L ( P : ,i +1: q , C : ,i +1: q , U : ,i +1: q ) , and A q = I .

Original: Now suppose A − 1 i can be written as
Revised: Now suppose A  1 i can be written as

Original: I + L ( P : ,i +1: q , C ′ : ,i +1: q , U ′ : ,i +1: q ) for some C ′ and U ′ .
Revised: I + L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) for some C 0 and U 0 .

Original: For the base case of i = q , this holds trivially.
Revised: For the base case of i = q , this holds trivially.

Original: We show it also holds for i − 1 , and we can compute C ′ : ,i : q , U ′ : ,i : q in O (cid:0) n ( q − i ) (cid:1) time.
Revised: We show it also holds for i  1 , and we can compute C 0 : ,i : q , U 0 : ,i : q in O  n ( q  i )  time.

Original: Let p
Revised: Let p

Original: =
Revised: =

Original: P : ,i , u
Revised: P : ,i , u

Original: = U : ,i , and c = C : ,i .
Revised: = U : ,i , and c = C : ,i .

Original: Consider u p = l , where each element is zero unless the corresponding element of p equals l .
Revised: Consider u p = l , where each element is zero unless the corresponding element of p equals l .

Original: What do we know about the productA − 1 i u p =
Revised: What do we know about the product

Original:  l (as seen in Equation 3)?
Revised: A  1 i u p = l (as seen in Equation 3)?

Original: Because the columns of P go from coarser partitions to finer ones, all of the vectors generating the sparse rank one components of L ( P : ,i +1: q , C ′ : ,i +1: q , U ′ : ,i +1: q ) are from partitions that are equal to or finer than p .
Revised: Because the columns of P go from coarser partitions to ﬁner ones, all of the vectors generating the sparse rank one components of L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) are from partitions that are equal to or ﬁner than p .

Original: Thus, they are either zero everywhere u p =
Revised: Thus, they are either zero everywhere u p =

Original: l is zero, or zero everywhere
Revised: l is zero, or zero everywhere

Original: u p =
Revised: u p =

Original: l is nonzero.
Revised: l is nonzero.

Original: Vectors v of the second kind can be ignored, as cvv ⊤ u p = l = 0 .
Revised: Vectors v of the second kind can be ignored, as cvv > u p = l = 0 .

Original: Thus, when multiplying
Revised: Thus, when multiplying

Original: L ( P : ,i +1: q , C ′ : ,i +1: q , U ′ : ,i +1: q ) by u p = l , the only relevant vectors are filled with zeros except where the corresponding element of p equals l .
Revised: L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) by u p = l , the only relevant vectors are ﬁlled with zeros except where the corresponding element of p equals l .

Original: So we can get rid of those rows of P : ,i +1: q , C ′ : ,i +1: q , and
Revised: So we can get rid of those rows of P : ,i +1: q , C 0 : ,i +1: q , and

Original: U ′ : ,i +1: q .
Revised: U 0 : ,i +1: q .

Original: Suppose there are n l elements of p that equal l .
Revised: Suppose there are n l elements of p that equal l .

Original: Then L ( P : ,i +1: q , C ′ : ,i +1: q , U ′ : ,i +1: q ) u p = l involves n l rows, and can be computed in O (cid:0) n l ( q − i ) (cid:1) time.
Revised: Then L ( P : ,i +1: q , C 0 : ,i +1: q , U 0 : ,i +1: q ) u p = l involves n l rows, and can be computed in O  n l ( q  i )  time.

Original: Moreover, this product, which we’ll call ( u ′ ) p = l , is only nonzero when the corresponding element of p equals l , so it has the same sparsity pattern as u p = l .
Revised: Moreover, this product, which we’ll call ( u 0 ) p = l , is only nonzero when the corresponding element of p equals l , so it has the same sparsity pattern as u p = l .

Original: The other component of A − 1 i is the identity matrix, and Iu p
Revised: The other component of A  1 i is the identity matrix, and Iu p

Original: =
Revised: =

Original: l clearly has the same sparsity as u p = l .
Revised: l clearly has the same sparsity as u p = l .

Original: Thus, returning to Equation 3, when we add u p = l ( u p = l ) ⊤ to A i , we update A − 1 i with an outer product of vectors whose sparsity pattern is the same as that of u p = l .
Revised: Thus, returning to Equation 3, when we add u p = l ( u p = l ) > to A i , we update A  1 i with an outer product of vectors whose sparsity pattern is the same as that of u p = l .

Original: For each l , A − 1 i need not be updated with each u p =
Revised: For each l , A  1 i need not be updated with each u p =

Original: l one at a time.
Revised: l one at a time.

Original: For l ̸ = l ′ , u p = l and u p = l ′ are nonzero at separate indices, so u p = l and ( u ′ ) p
Revised: For l 6 = l 0 , u p = l and u p = lare nonzero at separate indices, so u p = l and ( u 0 ) p

Original: = l ′ are nonzero at separate indices, so the extra component of A − 1 i that appears after the u p = l ′ update is irrelevant to the u p =
Revised: = l 0 are nonzero at separate indices, so the extra component of A  1 i that appears after the u p = l 0 update is irrelevant to the u p =

Original: l update, because ( u p = l ) ⊤ ( u ′ ) 
Revised: l update, because ( u p = l ) > ( u 0 ) p =

Original: p = l ′ = 0 .
Revised: l 0 = 0 .

Original: Since the u p = l update takes O ( n l ( q − i )) time, all of them together take
Revised: Since the u p = l update takes O ( n l ( q  i )) time, all of them together take

Original: O ( (cid:80) l n l ( q − i )) time, which equals O ( n ( q − i )) time.
Revised: O ( P l n l ( q  i )) time, which equals O ( n ( q  i )) time.

Original: Calculating an element of c ′ only involves computing the denominator in Equation 3, using a matrix-vector product already computed.
Revised: Calculating an element of c 0 only involves computing the denominator in Equation 3, using a matrix-vector product already computed.

Original: So we can write C ′ : ,i : q and U ′ : ,i : q by adding a preceding column to C ′ : ,i +1:
Revised: So we can write C 0 : ,i : q and U 0 : ,i : q by adding a preceding column to C 0 : ,i +1:

Original: q and U ′ : ,i +1: q , using the same partition p , and it takes O ( n ( q − i )) time.
Revised: q and U 0 : ,i +1: q , using the same partition p , and it takes O ( n ( q  i )) time.

Original: Following the induction down to i = 0 , we have ( L ( P, C, U ) + I ) − 1{+ 
+} =
Revised: Following the induction down to i = 0 , we have ( L ( P, C, U ) + I ) {+ 
+} =

Original: I + L ( P, C ′ , U ′ ) , and a total time of O ( nq 2 ) .
Revised: I + L ( P, C 0 , U 0 ) , and a total time of O ( nq 2 ) .

Original: Algorithm 2 also performs approximate inversion, which we prove in Appendix A.
Revised: Algorithm 2 also performs approximate inversion, which we prove in Appendix A.

Original: It differs slightly from the algorithm in the proof, but can take full advantage of a GPU speedup.
Revised: It differs slightly from the algorithm in the proof, but can take full advantage of a GPU speedup.

Original: In the setting where all columns of U are identical, observe that in Lines 10 and 11, the same computation is repeated for all k ∈{+
+}
Revised: In the setting where all columns of U are identical, observe that in Lines 10 and 11, the same computation is repeated for all k {+
+}

Original: [ i ] .
Revised: [ i ] .

Original: Indeed, in this setting, this block of code can be modified to run in O ( n ) time rather than
Revised: Indeed, in this setting, this block of code can be modiﬁed to run in O ( n ) time rather than

Original: O ( ni ) , making the whole algorithm run in O ( nq ) time, as shown in Proposition 4 in Appendix A.
Revised: O ( ni ) , making the whole algorithm run in O ( nq ) time, as shown in Proposition 4 in Appendix A.

Original: A Hierarchical matrix is a matrix which is either represented as a low-rank matrix or as a 2 ×block matrix of Hierarchical matrices [1].
Revised: A Hierarchical matrix is a matrix which is either represented as a low-rank matrix or as a 2 ⇥block matrix of Hierarchical matrices [1].

Original: In our SROS format, many of the sparse rank one matrices overlap, whereas in a Hierarchical matrix, the low-rank matrices do not overlap, and converting an SROS matrix into a Hierarchical matrix would typically be inefficient.
Revised: In our SROS format, many of the sparse rank one matrices overlap, whereas in a Hierarchical matrix, the low-rank matrices do not overlap, and converting an SROS matrix into a Hierarchical matrix would typically be inefﬁcient.

Original: Hierarchical matrices admit approximate inversion in O ( na 2 log 2 n ) time, where a is the maximum rank of the component submatrices [11].
Revised: Hierarchical matrices admit approximate inversion in O ( na 2 log 2 n ) time, where a is the maximum rank of the component submatrices [11].

Original: However, this is not an approximation in a technical sense, as there is no error bound.
Revised: However, this is not an approximation in a technical sense, as there is no error bound.

Original: At many successive steps in the algorithm, a rank 2 a matrix is approximated by a rank aAlgorithm
Revised: At many successive steps in the algorithm, a rank 2 a matrix is approximated by a rank a

Original: Require:
Revised: Require:

Original: ∈ × q , C, U ∈ R n × q
Revised: P 2 [ r ] n ⇥ q , C, U 2 R n ⇥ q

Original: Ensure: I + L ( P, C ′ , U ′ )
Revised: Ensure: I + L ( P, C 0 , U 0 )

Original: =
Revised: =

Original: ( I + L ( P, C, U )) − 1 ; x = log | I + L ( P, C, U ) | 1: x, C′ , U ′ ← 0 ,  0 ∈ R n × q , U 2: for i ∈ ( q, q − 1 , ..., 1) do ▷ O ( nq 2 ) time 3: p, c, u, u′ ← P : ,i , C : ,i , U : ,i , U ′ : ,i 4:
Revised: ( I + L ( P, C, U ))  1 ; x = log | I + L ( P, C, U ) | 1: x, C 0 , U0 , 0 2 R n ⇥ q , U 2: for i 2 ( q, q  1 , ..., 1) do . O ( nq 2 ) time 3: p, c, u, u

Revised: P : ,i , C : ,i , U : ,i , U 0 : ,i 4: z 0 2 R r .

Original: z← 0 ∈ R r ▷ z l will store c ( l ) (( u ′ ) p = l ) ⊤ u p = l , where c ( l ) =
Revised: z l will store c ( l ) (( u 0 ) p = l ) > u p = l , where c ( l ) =

Original: c k if p k = l 5: for j ∈ [ n ] do z p j← z p j+ c j u ′ j u j ▷ O ( n ) time 6: for j ∈ [ n ] do C ′
Revised: c k if p k = l 5: for j 2 [ n ] do z p j z p j

Original: ji ← − c j / (1 + z p j )
Revised: + c j u 0 j u j .

Original: ▷ O ( n ) time 7: for l ∈ [ r ] do x ← x + log(1 + z l ) ▷ O ( n ) time because r ≤ n 8: if i > 0 then 9: y ← 0 ∈ R n × i{+] do x x + log(
+}
Revised: O ( n ) time 6: for j 2 [ n ] do C 0 ji  c j / (1 + z p j ) . O ( n ) time 7: for l 2 [ r {+] do x x + log(
+}

Original: ▷
Revised: + z l ) . O ( n ) time because r  n 8: if i > 0 then 9: y 0 2 R n ⇥ i .

Original: O ( ni ) time 10: for j, k ∈ [ n ] ×
Revised: O ( ni ) time 10: for j, k 2 [ n ] ⇥

Original: [ i − 1] do y p
Revised: [ i  1] do y p

Original: j k← y p
Revised: j k y p

Original: j k + u ′{+
+} j U jk 
Revised: j k + u {+
+} j U jk .

Original: ▷ O ( ni ) time 11: for j, k ∈ [ n ] ×
Revised: O ( ni ) time 11: for j, k 2 [ n ] ⇥

Original: [ i − 1] do U ′ jk← U ′ jk + C ′ ji u ′ j y p
Revised: [ i  1] do U 0 jk U 0 jk + C 0 ji u 0 j y p

Original: j k ▷ O ( ni ) time return C ′ , U ′ , x matrix [10]; to our knowledge there is no analysis of how resulting errors might cascade.
Revised: j k . O ( ni ) time return C 0 , U 0 , x matrix [10]; to our knowledge there is no analysis of how resulting errors might cascade.

Original: After converting an SROS matrix to hierarchical form, this rough inversion would take O ( nq 2 log 2 n ) time.
Revised: After converting an SROS matrix to hierarchical form, this rough inversion would take O ( nq 2 log 2 n ) time.

Original:  Binary tree Gaussian process
Revised:  Binary tree Gaussian process

Original: We now show that our kernel matrix K XX can be written in SROS form, with P containing successively finer partitions.
Revised: We now show that our kernel matrix K XX can be written in SROS form, with P containing successively ﬁner partitions.

Original: Thus, K XX can be approximately inverted quickly, for use in Equations 1 and 2.
Revised: Thus, K XX can be approximately inverted quickly, for use in Equations 1 and 2.

Original: Next, we’ll show that we can efficiently optimize the log likelihood of the training data by tuning the weight vector w along with the bit order.
Revised: Next, we’ll show that we can efﬁciently optimize the log likelihood of the training data by tuning the weight vector w along with the bit order.

Original: The log likelihood can be calculated in O ( nq log n ) time and then the gradient w.r.t.
Revised: The log likelihood can be calculated in O ( nq log n ) time and then the gradient w.r.t.

Original: w in O ( nq 2 ) time.
Revised: w in O ( nq 2 ) time.

Original: Recall from the proof of Proposition 1 : K XX = (cid:80) qi
Revised: Recall from the proof of Proposition 1 : K XX = P qi

Original: =1 (cid:80) s ∈ B i w i X
Revised: [Equation]

Original:  [ s ] X ⊤ [ s ] , where X [ s ] ∈ R n with ( X [ s ] )
Revised: X [ [Equation] ] X > [ s ] , where X [ s ] 2 R n with ( X [ s ] )

Original: j = (cid:104)(cid:104)
Revised: j = hh

Original: X ≤| s | j = s (cid:105)(cid:105) .
Revised: X  | s | j = s ii .

Original: So we will set P : ,i , C : ,i , and U : ,i , so that L ( P : ,i , C : ,i , U : ,i ) =(cid:80) s
Revised: So we will set P : ,i , C : ,i , and U : ,i , so that L ( P : ,i , C : ,i , U : ,i ) =

Original: ∈ B i w i X
Revised: P s 2 B i w i X

Original: [ s ] X ⊤ [ s ] .
Revised: [ s ] X > [ s ] .

Original: Let P : ,i partition the set of points X
Revised: Let P : ,i partition the set of points X

Original: so that points are in the same partition if the first i bits match.
Revised: so that points are in the same partition if the ﬁrst i bits match.

Original: Now, requiring the first i + 1 bits to match is a stricter criterion than requiring the first i bits to match, so the P : ,i grow successively finer.
Revised: Now, requiring the ﬁrst i + 1 bits to match is a stricter criterion than requiring the ﬁrst i bits to match, so the P : ,i grow successively ﬁner.

Original: For any piece of the partition where the first i bits of the constituent points equals the bitstring s , the corresponding sparse rank one component of
Revised: For any piece of the partition where the ﬁrst i bits of the constituent points equals the bitstring s , the corresponding sparse rank one component of

Original: K XX is w i X
Revised: K XX is w i X

Original: [ s ] X ⊤ [ s ] .
Revised: [ s ] X > [ s ] .

Original: So let U : ,i = 1 n , and let C : ,i =
Revised: So let U : ,i = 1 n , and let C : ,i =

Original: w i 1 n .
Revised: w i 1 n .

Original: Proposition 2 (SROS Form Kernel) .
Revised: Proposition 2 (SROS Form Kernel) .

Original: K XX =
Revised: K XX =

Original: L ( P, C, U ) , as defined above.
Revised: L ( P, C, U ) , as deﬁned above.

Original: This follows immediately from the definitions.
Revised: This follows immediately from the deﬁnitions.

Original: To compute these partitions P : ,i , we sort X , which is a set of bit strings.
Revised: To compute these partitions P : ,i , we sort X , which is a set of bit strings.

Original: And then we can easily compute which points have the same first i bits.
Revised: And then we can easily compute which points have the same ﬁrst i bits.

Original: This all takes O ( nq log n ) time.
Revised: This all takes O ( nq log n ) time.

Original: Now note that U : ,i =
Revised: Now note that U : ,i =

Original: U : ,i ′ for all i, i ′ , so ( K XX + λI ) − 1 and | K XX + λI | can be computed in O ( nq ) time, rather than O ( nq 2 ) .
Revised: U : ,i 0 for all i, i 0 , so ( K XX +  I )  1 and | K XX +  I | can be computed in O ( nq ) time, rather than O ( nq 2 ) .

Original: The training negative log likelihood of a GP is that of the corresponding multivariate Gaussian on the training data.
Revised: The training negative log likelihood of a GP is that of the corresponding multivariate Gaussian on the training data.

Original: So: NLL
Revised: So: NLL

Original: [Equation]
Revised: [Equation]

Original: This can be computed in O ( nq ) time, since matrix-vector multiplication takes O ( nq ) time for a matrix in SROS form.
Revised: This can be computed in O ( nq ) time, since matrix-vector multiplication takes O ( nq ) time for a matrix in SROS form.

Original: So if the bit order is unchanged, an optimization step can be done in O ( nq ) time, and if the data needs to be resorted, then in O ( nq log n ) time.
Revised: So if the bit order is unchanged, an optimization step can be done in O ( nq ) time, and if the data needs to be resorted, then in O ( nq log n ) time.

Original: On the largest dataset we tested (House Electric), with n ≈ 1.3 million and q = 88 , sorting the data and computing P takes about 0.96 seconds on a GPU, and then calculating the negative log likelihood takes about another 1.
Revised: On the largest dataset we tested (House Electric), with n ⇡ 1.3 million and q = 88 , sorting the data and computing P takes about 0.96 seconds on a GPU, and then calculating the negative log likelihood takes about another 1.

Original: seconds.
Revised: seconds.

Original: We show in Appendix B how to compute ∇ w NLL in O ( nq 2 ) time.
Revised: We show in Appendix B how to compute r w NLL in O ( nq 2 ) time.

Original: To optimize the bit order and weight vector at the same time, we represent both with a single parameter vector θ ∈ R q + , with || θ || ∞ = 1 .
Revised: To optimize the bit order and weight vector at the same time, we represent both with a single parameter vector ✓ 2 R q + , with || ✓ || 1 = 1 .

Original: To get the bit order from θ , we start with a default bit order and permute the bit order according to a permutation that would sort θ in descending order.
Revised: To get the bit order from ✓ , we start with a default bit order and permute the bit order according to a permutation that would sort ✓ in descending order.

Original: To get the
Revised: To get the

Revised: Algorithm

Original: Require: X ∈ B n × q , y∈ R n , X ′ ∈
Revised: Require: X 2 B [Equation] , y

Revised: [Equation]

Revised: , X

Revised: [Equation]

Original:  B m × q , w ∈ R q , λ ∈ R+
Revised: 2 B [Equation] , w 2 R

Revised: [Equation]

Revised: ,  2 R +

Original: Ensure: µ X ′ and σ are the predictive means and variances at X ′ , and nll the training negative log likelihood. 
Revised: Ensure: µ X 0 and  2 X 0 are the predictive means and variances at X 0 , and nll the training negative log likelihood. 1: ˜ X X  X 0 2: ˜ X " , perm Sort ( ˜ X ) .

Revised: The rows of X are sorted lexically from leading bit to trailing bit.

Original: Algorithm 2.
Revised: O (( n + m ) q log( n + m )) time. 3: for j, i 2 [ n + m ] ⇥ [ q ] do ˜ P " ji # of unique rows in X " 1: j, 1: i 4: . M 1: j, 1: i is the ﬁrst j rows and i columns of M .

Revised: O (( n + m ) q ) time.

Revised: 5: ˜ P perm  1 ( P " ) .

Revised: This “unsorts” the input.

Revised: O (( n + m ) q ) time.

Revised: 6: P, P˜ P 1: n , ˜ P n +1: n + m 7: U, U 0 , ˜ U 1 n ⇥ q , 1 m ⇥ q , 1 ( n + m ) ⇥ q 8: C, ˜ C 1 n w T , 1 n + m w T 9: C  1  , U  1 , logdet

Revised: Invert ( P,   1 C, U ) .

Revised: Uses Algorithm 2.

Original: Speedup to  time  of  are identical
Revised: Speedup to O ( nq ) time because columns of U are identical. 10: C  1 , logdet   1 C  1  , logdet  + n log(  ) 11: z LinTransform ( P, P, U  1 , C  1  U  1 , y ) +   1 y .

Original: Uses Algorithm 1 to compute the
Revised: Uses Algorithm 1 to compute the

Original: Woodbury vector.
Revised: Woodbury vector.

Original: O  nq ) time.▷ O ( n m q time.
Revised: O ( nq ) time.

Original: [Table]
Revised: 12: µ X LinTransform ( P 0 , P, U 0 , C  U, z ) .

Original: O ( [Table] n + m [Table]
Revised: O (( n + m ) q ) time.

Original: q
Revised: 13: nll ( y > z + logdet + n log(2 ⇡ )) /14: ˜ C prec , ˜ U prec Invert ( ˜ P,   1 ˜ C,   1 ˜ U ) .

Original: [Table] ▷ O ( mq [Table] time
Revised: O (( n + m ) q ) time.

Original: [Table]
Revised: 15: C prec , U prec ˜ C prec n +1: n + m , ˜ U prec n +1: n + m 16: C cov , U cov Invert ( P 0 , C prec , U prec ) .

Original:  extra factor of q because columns of U prec are not identical
Revised: O ( mq 2 ) time; extra factor of q because columns of U prec are not identical.

Original: [Table]
Revised: 17:  2 X  ( 1 m + SumEachRow ( C cov  U cov  U cov )) .

Original: ▷ O ( mq ) time
Revised: O ( mq ) time.

Original: [Table] 18: return [Table] weight vector, we sort θ in descending order, add a 0 at the end, and compute the differences between adjacent elements.
Revised: 18: return µ X 0 ,  2 X 0 , nll weight vector, we sort ✓ in descending order, add a 0 at the end, and compute the differences between adjacent elements.

Original: When there are ties in the elements of θ , the choice of bit order does not affect the negative log likelihood (or the kernel at all) because the relevant associated weight is 0 .
Revised: When there are ties in the elements of ✓ , the choice of bit order does not affect the negative log likelihood (or the kernel at all) because the relevant associated weight is 0 .

Original: The negative log likelihood is continuous with respect to θ , and when all values of θ are unique, it is differentiable with respect to θ .
Revised: The negative log likelihood is continuous with respect to ✓ , and when all values of ✓ are unique, it is differentiable with respect to ✓ .

Original: Letting θ = e ϕ / || e ϕ || ∞ , we minimize loss w.r.t.
Revised: Letting ✓ = e  / || e  || 1 , we minimize loss w.r.t.

Original: ϕ using BFGS [8].
Revised:  using BFGS [8].

Original: To calculate the predictive mean at a list of predictive locations X ′ , we first multiply y by ( K XX + λI ) − 1 , and then we multiply that vector by K XX ′ .
Revised: To calculate the predictive mean at a list of predictive locations X 0 , we ﬁrst multiply y by ( K XX +  I )  1 , and then we multiply that vector by K XX 0 .

Original: We obtain both K XX and K XX ′ in SROS form as follows.
Revised: We obtain both K XX and K XX 0 in SROS form as follows.

Original: Let ˜ X
Revised: Let ˜ X

Original: = X ◦ X ′ be the concatenation of the two tuples, now an ( n + m ) -tuple.
Revised: = X  X 0 be the concatenation of the two tuples, now an ( n + m ) -tuple.

Original: Writing
Revised: Writing

Original: K ˜ X ˜ X
Revised: K ˜ X ˜ X

Original: = L ( ˜ P, ˜ C, ˜ U ) , the arrays on the r.h.s.
Revised: = L ( ˜ P, ˜ C, ˜ U ) , the arrays on the r.h.s.

Original: can be computed in O (( n + m ) q log( n + m )) time.
Revised: can be computed in O (( n + m ) q log( n + m )) time.

Original: Then, with P , C , and U being the first n rows of ˜ P , ˜ C , ˜ U , K XX =
Revised: Then, with P , C , and U being the ﬁrst n rows of ˜ P , ˜ C , ˜ U , K XX =

Original: L ( P, C, U ) .
Revised: L ( P, C, U ) .

Original: And letting P ′′ and
Revised: And letting P 00 and

Original: U ′′ be the last m rows, K XX ′{+
+}
Revised: U 00 be the last m rows, K XX {+
+}

Original: = L ( P, P ′′ , C ⊙ U, U ′′ ) .
Revised: = L ( P, P 00 , C  U, U 00 ) .

Original: Thus, the predictive mean µ x from Equation 1 can be computed at m locations in O (( n + m ) q log( n + m )) time.
Revised: Thus, the predictive mean µ x from Equation 1 can be computed at m locations in O (( n + m ) q log( n + m )) time.

Original: The predictive covariance matrix, which extends the predictive variance from Equation 2, is calculated Σ X ′ ={+
+}
Revised: The predictive covariance matrix, which extends the predictive variance from Equation 2, is calculated ⌃ X {+
+}

Original:  K X ′ X {+
+}
Revised: = K X 0 X {+
+}

Original: + λI m − K X
Revised: +  I m  K X

Original: ′ X
Revised: 0 X

Original: ( K XX + λI n ) − 1 K XX ′ =
Revised: ( K XX +  I n )  1 K XX 0 =

Original: ( K ˜ X ˜ X + λI m + n ) /K
Revised: ( K ˜ X ˜ X +  I m + n ) /K

Original: XX , where / denotes the Schur complement.
Revised: XX , where / denotes the Schur complement.

Original: From a property of block matrix inversion, the last m columns of the last m rows of ( K ˜ X ˜ X + λI ) − 1 equals (( K ˜ X ˜ X
Revised: From a property of block matrix inversion, the last m columns of the last m rows of ( K ˜ X ˜ X +  I )  1 equals (( K ˜ X ˜ X

Original: + λI m + n ) /K
Revised: +  I m + n ) /K

Original: XX ) − 1 .
Revised: XX )  1 .

Original: So we get the predictive precision matrix in O (( n + m ) q log( n + m )) time by inverting K ˜ X
Revised: So we get the predictive precision matrix in O (( n + m ) q log( n + m )) time by inverting K ˜ X

Original: ˜ X + λI and taking the bottom right m × m block.
Revised: ˜ X +  I and taking the bottom right m ⇥ m block.

Original: Then, we get the predictive covariance matrix by inverting that.
Revised: Then, we get the predictive covariance matrix by inverting that.

Original: This takes O ( mq 2 ) time, since it does not have the property of all the columns of U being equal.
Revised: This takes O ( mq 2 ) time, since it does not have the property of all the columns of U being equal.

Original: If we only want the diagonal elements of an SROS matrix (the independent predictive variances in this case), we can simply sum the rows of C ⊙ U ⊙ U in O ( mq ) time.
Revised: If we only want the diagonal elements of an SROS matrix (the independent predictive variances in this case), we can simply sum the rows of C  U  U in O ( mq ) time.

Original: Thus, in total, computing the independent predictive variances requires O (( n + m ) q log( n + m ) + mq 2 ) time.
Revised: Thus, in total, computing the independent predictive variances requires O (( n + m ) q log( n + m ) + mq 2 ) time.

Original: See Algorithm 3.
Revised: See Algorithm 3.

Original:  Related Work
Revised:  Related Work

Original: All existing kernels of which we are aware for linear time GP regression on unstructured data involve inducing points (related to the Nyström approximation [27]) or inducing frequencies.
Revised: All existing kernels of which we are aware for linear time GP regression on unstructured data involve inducing points (related to the Nyström approximation [27]) or inducing frequencies.

Original: For a given set of inducing points Z , for some base kernel k , the inducing point kernel (in its most basic form) is the following, although subtle variants exist: k Z ( x, x ′ )
Revised: For a given set of inducing points Z , for some base kernel k , the inducing point kernel (in its most basic form) is the following, although subtle variants exist: k Z ( x, x 0 )

Original: =
Revised: =

Original: K xZ K − 1 ZZ K Zx ′ [21].{+
+}
Revised: K xZ K  1 ZZ K Zx {+
+}

Original: Sparse Gaussian Process Regression (SGPR) involves selecting Z , and then using k Z (or a variant).
Revised: Sparse Gaussian Process Regression (SGPR) involves selecting Z , and then using k Z (or a variant).

Original: Notably, K ZXX =
Revised: Notably, K ZXX =

Original: K XZ K − 1 ZZ K ZX is low rank, providing computational efficiency.
Revised: K XZ K  1 ZZ K ZX is low rank, providing computational efﬁciency.

Original: The predictive mean and covariance have compact form, with observational noise λ : µ x ( Z )
Revised: The predictive mean and covariance have compact form, with observational noise  : µ x ( Z )

Original: = K xZ ( λK ZZ + k ZX k XZ ) − 1 K ZX y and σ 2 xx ′{+
+}
Revised: = K xZ (  K ZZ + k ZX k XZ )  1 K ZX y and  2 xx {+
+}

Original: ( Z ) =
Revised: ( Z ) =

Original: K xZ ( K ZZ + λ − 1 k ZX k XZ ) − 1 K Zx ′ .
Revised: K xZ ( K ZZ +   1 k ZX k XZ )  1 K Zx 0 .

Original: Titsias’s [24] sparse variational kernel is also low rank and uses inducing points.
Revised: Titsias’s [24] sparse variational kernel is also low rank and uses inducing points.

Original: The sparse variational
Revised: The sparse variational

Original: GP (SVGP) is constructed as the solution to a variational inference problem.
Revised: GP (SVGP) is constructed as the solution to a variational inference problem.

Original: It depends on inducing points Z , data points X , and observed function values y .
Revised: It depends on inducing points Z , data points X , and observed function values y .

Original: We have focused on Gaussian processes with 0 mean, but the SVGP method uses a nonzero prior mean along with a kernel: m SVGP ( x )
Revised: We have focused on Gaussian processes with 0 mean, but the SVGP method uses a nonzero prior mean along with a kernel: m SVGP ( x )

Original: = µ x ( Z ) and k SVGP ( x, x ′ )
Revised: = µ x ( Z ) and k SVGP ( x, x 0 )

Original: =
Revised: =

Original: k ( x, x ′ ) − K xZ K − 1 ZZ K Zx {+
+}
Revised: k ( x, x 0 )  K xZ K  1 ZZ K Zx {+
+}

Original: ′ + σ 2 xx ′ ( Z ) .
Revised: +  2 xx 0 ( Z ) .

Original: Given the dependence on X and y , this is not a true probability distribution over function space.
Revised: Given the dependence on X and y , this is not a true probability distribution over function space.

Original: The variational problem underlying this kernel also provides guidance in how to select the inducing points Z .
Revised: The variational problem underlying this kernel also provides guidance in how to select the inducing points Z .

Original: For further discussion of the kernel underlying the SVGP method, see Wild et al.
Revised: For further discussion of the kernel underlying the SVGP method, see Wild et al.

Original: An inducing point kernel with z inducing points produces a z -dimensional reproducing kernel Hilbert space (RKHS).
Revised: An inducing point kernel with z inducing points produces a z -dimensional reproducing kernel Hilbert space (RKHS).

Original: The dimensionality of the RKHS relates to the expressivity of the kernel.
Revised: The dimensionality of the RKHS relates to the expressivity of the kernel.

Original: Whereas an inducing point method buys a z -dimensional RKHS for the price of O ( z 2 n ) time and O ( zn ) space, the binary tree kernel produces a 2 q -dimensional RKHS in O ( qn ) time and space—an exponential improvement.
Revised: Whereas an inducing point method buys a z -dimensional RKHS for the price of O ( z 2 n ) time and O ( zn ) space, the binary tree kernel produces a 2 q -dimensional RKHS in O ( qn ) time and space—an exponential improvement.

Original: (Observe that we can find 2 q linearly independent functions of the form k ( · , x ) —one for each of the 2 q leaves x might belong to.)
Revised: (Observe that we can ﬁnd 2 q linearly independent functions of the form k ( · , x ) —one for each of the 2 q leaves x might belong to.)

Original: Wilson and Nickisch [29] develop a method for speeding up inducing point methods significantly, especially in low-dimensional settings.
Revised: Wilson and Nickisch [29] develop a method for speeding up inducing point methods signiﬁcantly, especially in low-dimensional settings.

Original: Lázaro-Gredilla et al.
Revised: Lázaro-Gredilla et al.

Original: [16] propose an inducing frequencies kernel: given a set of m inducing vectors s i , k ( x, x ′ )
Revised: [16] propose an inducing frequencies kernel: given a set of m inducing vectors s i , k ( x, x 0 )

Original: = 1 /m (cid:80) mi =1 cos(2 πs ⊤ i ( x − x ′ )) .
Revised: = 1 /m P mi =1 cos(2 ⇡ s > i ( x  x 0 )) .

Original: Dutordoir et al.
Revised: Dutordoir et al.

Original: [6] propose an inducing frequencies kernel for low dimensional data, in which k ( x, x ′ ) is a special function of x ⊤ x ′ .
Revised: [6] propose an inducing frequencies kernel for low dimensional data, in which k ( x, x 0 ) is a special function of x > x 0 .

Original: On one-dimensional data, filtering/smoothing methods perform Bayesian inference over functions in O ( n ) time [14, 12].
Revised: On one-dimensional data, ﬁltering/smoothing methods perform Bayesian inference over functions in O ( n ) time [14, 12].

Original: A few non- O ( n ) methods bear mentioning.
Revised: A few non- O ( n ) methods bear mentioning.

Original: We are not the first to consider a kernel over points on the leaves of a tree [18, 17] or on the leaves of multiple trees [7], but their methods take O ( n 3 ) time.
Revised: We are not the ﬁrst to consider a kernel over points on the leaves of a tree [18, 17] or on the leaves of multiple trees [7], but their methods take O ( n 3 ) time.

Original: On certain kinds of structured data, Toeplitz solvers achieve O ( n 2 ) time complexity [30].
Revised: On certain kinds of structured data, Toeplitz solvers achieve O ( n 2 ) time complexity [30].

Original: Cutajar et al.
Revised: Cutajar et al.

Original: ’s
Revised: ’s

Original: [3], Wang et al.’s
Revised: [3], Wang et al.’s

Original: [25], and others’ use of a conjugate gradients solver to replace inversion/factorization has unclear time complexity between O ( n 2 ) and O ( n 3 ) , depending on the kernel matrix spectrum.
Revised: [25], and others’ use of a conjugate gradients solver to replace inversion/factorization has unclear time complexity between O ( n 2 ) and O ( n 3 ) , depending on the kernel matrix spectrum.

Original:  Experiments
Revised:  Experiments

Original: In Table 1, we compare our binary tree kernel and a binary tree ensemble (see Appendix C for details on the ensemble) against three baseline methods: exact GP regression using a Matérn kernel, sparse
Revised: In Table 1, we compare our binary tree kernel and a binary tree ensemble (see Appendix C for details on the ensemble) against three baseline methods: exact GP regression using a Matérn kernel, sparse

Original: Gaussian process regression (SGPR)
Revised: Gaussian process regression (SGPR)

Original: [23], and a stochastic variational Gaussian process (SVGP) [13].
Revised: [23], and a stochastic variational Gaussian process (SVGP) [13].

Original: We evaluate our method on the same open-access UCI datasets [4] as Wang et al.
Revised: We evaluate our method on the same open-access UCI datasets [4] as Wang et al.

Original: [25], using their same training, validation, and test partitions, and we compare against the baseline results they report.
Revised: [25], using their same training, validation, and test partitions, and we compare against the baseline results they report.

Original: For the binary tree (BT) kernels, we use p = min(8 , ⌊ 150 /d ⌋ + 1) , and recall q = pd .
Revised: For the binary tree (BT) kernels, we use p = min(8 , b 150 /d c + 1) , and recall q = pd .

Original: We set λ = 1 /n .
Revised: We set  = 1 /n .

Original: We train the bit order and weights to minimize training NLL.
Revised: We train the bit order and weights to minimize training NLL.

Original: For the binary tree ensemble (BTE), we use 20 kernels.
Revised: For the binary tree ensemble (BTE), we use 20 kernels.

Original: For the Matérn kernel, we use Blackbox Matrix-Matrix multiplication (BBMM) [9], which uses the conjugate gradients method to calculate  ( K XX + λI ) − 1 .
Revised: For the Matérn kernel, we use Blackbox Matrix-Matrix multiplication (BBMM) [9], which uses the conjugate gradients method to calculate matrix-vector products with ( K XX +  I )  1 .

Original: SGPR uses 512 data points and SVGP uses 1,024 inducing points.
Revised: SGPR uses 512 data points and SVGP uses 1,024 inducing points.

Original: We report the mean and two standard errors across 3 replications with different dataset splits.
Revised: We report the mean and two standard errors across 3 replications with different dataset splits.

Original: For further experimental details, see Appendix C. BTE achieves the best NLL on 6/12 datasets, and best RMSE on 5/12 datasets (including some ties).
Revised: For further experimental details, see Appendix C. BTE achieves the best NLL on 6/12 datasets, and best RMSE on 5/12 datasets (including some ties).

Original: Out of the 4 largest datasets, BT/BTE is fastest on 3.The run times are plotted in Figure 5.
Revised: Out of the 4 largest datasets, BT/BTE is fastest on 3.

Original: The code is available at https://github.com/mkc1000/btgp and https://tinyurl.com/btgp-colab . [Table]
Revised: The run

Original: BT performs noticeably worse than BTE for test NLL on CTSlice due to over-fitting.
Revised: BT performs noticeably worse than BTE for test NLL on CTSlice due to over-ﬁtting.

Original: There are enough degrees of freedom when optimizing the bit order ( d = 385 ) that BT kernel can over-fit to the training data.
Revised: There are enough degrees of freedom when optimizing the bit order ( d = 385 ) that BT kernel can over-ﬁt to the training data.

Original: The ensemble over multiple bit orders is much more robust.
Revised: The ensemble over multiple bit orders is much more robust

Revised: [Figure]

Revised: [Table]

Original:  Discussion
Revised:  Discussion

Original: We have proven that the binary tree kernel GP is scalable.
Revised: We have proven that the binary tree kernel GP is scalable.

Original: Our empirical results suggest that it often outpredicts not just other scalable methods but even the popular Matérn GP.
Revised: Our empirical results suggest that it often outpredicts not just other scalable methods, but even the popular Matérn GP.

Original: If the results in this paper replicate in other domains, it could obviate wide usage of classic GP kernels like the Matérn kernel, as well as inducing point kernels.
Revised: If the results in this paper replicate in other domains, it could obviate wide usage of classic GP kernels like the Matérn kernel, as well as inducing point kernels.

Original: Sometimes, our kernel fails to capture patterns in the data; some functions’ values simply do not covary this way.
Revised: Sometimes, our kernel fails to capture patterns in the data; some functions’ values simply do not covary this way.

Original: But other kernels we tested seemed to fail like that even more.
Revised: But other kernels we tested seemed to fail like that even more.

Original: Our contributions to linear algebra and kernel design may significantly increase the size of data sets on which GPs can do state-of-the-art modelling.Checklist The checklist follows the references.
Revised: Our contributions to linear algebra and kernel design may signiﬁcantly increase the size of data sets on which GPs can do state-of-the-art modelling.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes]

Original: In the introduction.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Yes in Section E. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] In Section(b) Did you include complete proofs of all theoretical results?

Original: [Yes]

Original: In Sections 3 andand Appendix A 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] As a URL.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] In Section 7 and Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Table 1.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Table 1. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] The UCI datasets in Section 7.

Original: (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: A Correctness of Algorithm

Original: In this section, we show that Algorithm 2 performs approximate inversion.

Original: And then we show how to modify the algorithm in the setting were all columns of U are identical, for a factor of q speedup.

Original: We begin by writing the exact form of ( u ′ )

Original: p =

Original: l from the proof of Theorem 1, and the corresponding c ′ ( l ) .

Original: Recall the Sherman–Morrison Formula:

Original: [Equation]

Original: At the time that we update A with the rank one matrix c ( l ) u p = l u p = l ⊤ , what does A equal?

Original: Let c and u originate from the i th column of C and U .

Original: So our update is C ( l ) : ,i U P : ,i

Original: =

Original: l : ,i

Original: [Equation]

Original: ,i

Original: [Equation]

Original: [-Then A ← A i +
-]
Revised: [-Then A ← A i +
-]

Original: = I + (cid:80)

Original: qk = i +1 (cid:80) ℓ ∈

Original: [ r ] C ( ℓ ) : ,k U P : ,k = ℓ : ,k

Original: U P : ,k = ℓ : ,k ⊤ , recalling r is the largest integer in

Original: P .

Original: And likewise, all the relevant entries of C ′ and U ′ have been calculated for A − 1 .

Original: So we have

Original: [-A − 1 i +
-]
Revised: [-A − 1 i +
-]

Original: = I + (cid:80)

Original: qk = i +1 (cid:80) ℓ ∈

Original: [ r ] C ′ ( ℓ ) : ,k U ′ P : ,k = ℓ : ,k

Original: [Equation]

Original: The main computation we need to do is A − 1 u .

Original: We’ll say that k ℓ ⊑ i l if set l from partition P : ,i is a superset of set ℓ from partition P : ,k .

Original: That is, the rows where P : ,k takes the value ℓ are a subset of the rows where P : ,i takes the value l .

Original: The key simplification we use is that if k ℓ ̸⊑ i l , then

Original: U P : ,i =

Original: l : ,i ⊤ U ′ P : ,k = ℓ : ,k = 0 .

Original: The rows at which those two vectors have nonzero elements are disjoint.

Original: This logic is explained in the proof of Theorem 1 without all the notation.

Original: So we let

Original: [Equation]

Original: Equation 6 follows because the terms in the dot product U ′ P : ,k = ℓ : ,k ⊤ U P : ,i

Original: = l : ,i are only nonzero when

Original: P : ,k = ℓ and P : ,i = l , but the latter is implied by k ℓ ⊑ i l , so this is equivalent to the elements where

Original: P : ,k = ℓ .

Original: So U ′ P : ,k = ℓ : ,k ⊤

Original: [Equation]

Original: P : ,i =

Original: l : ,i

Original: [Equation]

Original: = ℓ :

Original: [Equation]

Original: , which gives us Equation 6.

Original: Then, we let

Original: [Equation]

Original: [Equation]

Original: Proposition 3 (Correctness of Algorithm 2) .

Original: In Algorithm 2, U ′ and C ′ take the values defined in

Original: Equations 6 and 7.

Original: Proof.

Original: We’ll assume that for k > i , C ′ : ,k and U ′ : ,k have the right values, and we’ll show that C ′ : ,i and U ′ : ,i get the right values.

Original: Starting with U ′ : ,q , the (cid:80) qk = q +1 in Equation 6 is empty, so U ′ : ,q = U : ,q .

Original: In Algorithm 2, U ′ is initialized to U , and the q th column is never updated.

Original: Now we see that C ′ : ,i is correct assuming U ′ : ,i is.

Original: In Line 5, we ensure z l = C ( l ) : ,i U ′ P : ,i =

Original: l : ,i ⊤ U P : ,i

Original: = l : ,i .

Original: A dot product is the sum of elementwise multiplications, and one can inspect that each such multiplication gets added to the right slot in z .

Original: Then, Line 6 implements Equation 7, with numerator and denominator multiplied by C ( l ) : ,i .

Original: (It stores the same value in multiple locations).

Original: Now we turn to U ′ : ,i .

Original: U ′ : ,i is updated in every preceding loop.

Original: It starts out initialized to U : ,i , which accounts for the first term in Equation 6.

Original: In the sum from k = i + 1 to q , each term is accounted for in a separate loop of the algorithm.

Original: We check that each term gets added at some point.

Original: So consider the term (cid:80) ℓ : k ℓ ⊑ i l C ′ ( ℓ ) : ,k U ′ P : ,k = ℓ : ,k

Original: U ′ P : ,k = ℓ : ,k ⊤ U P : ,i = l : ,i .

Original: This term gets added to U : ,i when i from Algorithm 2 equals k from Equation 6, and when k from

Original: Algorithm 2 equals i from Equation 6.

Original: We are very sorry about this correspondence, but it would have to happen either here or above in the discussion of C ′ .

Original: Observe that in Line 10, y p

Original: j k takes the value ( U ′ )

Original: P : ,i = p

Original: j : ,i ⊤ U P : ,i = p

Original: j : ,k .

Original: Then, in Line 11, y p

Original: j k gets multiplied by ( U ′ )

Original: P : ,i = p

Original: j : ,i and C ′ ( p j ) : ,i , and added to U ′ : ,k .

Original: Swapping the i ’s and k ’s, and letting p j from Algorithm 2 equal ℓ from Equation 6,

Original: Lines 10 and 11 add to U ′ : ,k the terms in Equation 6.

Original: Thus, doing induction from i = q down to 1, Algorithm 2 assigns C ′ : ,i and U ′ : ,i the correct values.

Original: Now, we modify Algorithm 2 for the setting where all the columns U : ,i are the same, allowing a speedup of O ( q ) .

Original: Require: P ∈

Original: [ r ] n × q , C ∈ R n × q , u ∈ R n

Original: Ensure: I + L ( P, C ′ , U ′ )

Original: =

Original: ( I + L ( P, C, u ( 1 q ) ⊤ )) − 1 ; x = log | I + L ( P, C, u ( 1 q ) ⊤ ) | 1: x, C ′ , U ′ ← 0 , 0 ∈ R n × q , u ( 1 q ) ⊤ 2: for i ∈ ( q, q − 1 , ..., 1) do ▷ O ( nq ) time 3: p, c, u ′ ← P : ,i , C : ,i , U ′ : ,i 4:

Original: z ← 0 ∈ R r ▷ z i will store c ( l ) (( u ′ )

Original: p = l ) ⊤ u p = l , where c k =

Original: c

Original: ( l ) if p k = l 5: for j ∈ [ n ] do z p j ← z p j + c j u ′ j u j ▷ O ( n ) time 6: for j ∈ [ n ] do C ′

Original: ji ← − c j / (1 + z p j )

Original: ▷ O ( n ) time 7: for i ∈ [ r ] do x ← x + log(1 + z i )

Original: ▷ O ( n ) time 8:

Original: if i > 0 then 9: if False then ▷ This block is the slow version.

Original: What follows below is equivalent.

Original: 10: y ← 0 ∈ R n × i 11: for j, k ∈ [ n ] × [ i − 1] do y p

Original: j k ← y p

Original: j k + u ′ j u j 12: for j, k ∈ [ n ] ×

Original: [ i − 1] do U ′ jk ← U ′ jk + C ′ ji u ′ j y p

Original: j k 13: y ← 0 ∈ R n ▷ O ( n ) time 14:

Original: U ′ : , ( i − 1) ← U ′ : ,i ▷ O ( n ) time 15: for j ∈ [ n ] do y p

Original: j ← y p j + u ′ j u j ▷ O ( n ) time 16: for j ∈ [ n ] do U ′

Original: j ( i − 1) ← U ′ j ( i − 1)

Original: + C ′ ji u ′ j y p

Original: j ▷ O ( n ) time return C ′ , U ′ , x

Original: Proposition 4 ( O ( nq ) inversion) .

Original: Algorithm 4 performs approximate inversion in O ( nq ) time.

Original: Proof.

Original: The fact that Algorithm 4 runs in O ( nq ) time is easily verified.

Original: Up to Line 12, Algorithm 4 is the same as Algorithm 2, except U has been replaced with u ( 1 q ) ⊤ .

Original: So all we have to show is that

Original: Lines 13-16 produce the same result that Lines 10-12 would have.

Original: Observe that at the start of the algorithm, U ′ : ,k and U ′ : ,k +1 are initialized to the same value.

Original: Observe that Lines 11 and 12 repeat the same computation i − 1 times.

Original: So in Lines 11 and 12, U ′ : ,k and U ′ : ,k + are updated by the same amount if i > k + 1 , and they aren’t updated at all if i ≤ k .

Original: So the difference between U ′ : ,k and U ′ : ,k +1 comes from only the former being updated when i = k

Original: + 1 .

Original: Therefore, Line 14 initializes U ′ : ,k to U ′ : ,k +1 .

Original: Then Lines 15 and 16 update U ′ : ,k with the appropriate difference; they copy Lines 11 and 12, setting k to i − 1 .

Original: B Gradient of loss with respect to weights

Original: In this section, we show how to calculate ∇ w NLL in O ( nq 2 ) time.

Original: Recall:

Original: [Equation]

Original: Differentiating gives:

Original: [Equation]

Original: We begin by evaluuating ∂K XX /∂w i .

Original: Recall from Proposition 2 that K XX

Original: =

Original: L ( P, C, U ) , where

Original: C = 1 n w ⊤ , and U = 1 n × q .

Original: It follows easily from the definition of L that the elements of L ( P, C, U ) are linear in the elements of C .

Original: So,

Original: [Equation]

Original: Algorithm 3 shows how to calculate ( K XX + λ ) − 1 in O ( nq ) time, and represent it as L ( P, C − 1 , U − 1 ) .

Original: (Recall C − 1 and U − 1 are not true inverses; we just the notation to denote their purpose.)

Original: Now we turn to the question of how to calculate Tr[ L ( P : ,i , 1 n , 1 n ) L ( P, C − 1 , U − 1 )] .

Original: We are considering symmetric matrices, so the trace of the product is the sum of the elements of the elementwise product.

Original: We expand and simplify:

Original: [Equation]

Original: where ( a ) follows from the fact that the ( i, j ) th element of L ( p, c, u ) is zero unless p i =

Original: p j , in which case, it is c

Original: i u i u j ; when multiplying elementwise by L ( p ′ , c ′ , u ′ ) , where p ′ is a finer partition,

Original: L ( p, c, u ) ⊙ L ( p ′ , c ′ , u ′ ) =

Original: L ( p ′ , c, u ) ⊙ L ( p ′ , c ′ , u ′ ) , because L ( p, c, u ) and L ( p ′ , c, u ) only differ on elements where L ( p ′ , c ′ , u ′ ) is 0 anyway.

Original: In the context of ( a ) , P : , max( i,j ) is a finer partition than P : , min( i,j ) .

Original: ( b ) follows because we are doing elementwise multiplication between the two matrices; anywhere L ( P : , max( i,j ) , 1 n , 1 n ) is 0 , L ( P : , max( i,j ) , C − 1 : ,j , U − 1 : ,j ) is already 0 , and elsewhere, multiplying elements by 1 does not effect the matrix.

Original: ( c ) follows from the construction of L .

Original: It is straightforward to compute this in O ( nq ) time.

Original: See Algorithm 5, which runs in O ( n ) time and can be iterated over the q terms in Equation 16.

Original: Now we can see that Equation 9 can be computed for all w i in O ( nq 2 ) time.

Original: First, ( K XX + λ ) −can be computed in O ( nq ) time, in the form L ( P, C − 1 , U − 1 ) , as shown in Algorithm 3.

Original: Then, z =

Original: ( K XX + λ ) − 1 y can be computed in O ( nq ) time, also as shown in Algorithm 3.

Original: Then, for each i ∈ [ q ] , we can calculate − z ⊤ ∂K XX ∂w

Original: i z = − z ⊤ L ( P : ,i , 1 n , 1 n ) z in O ( n ) time.

Original: Thus, handling the first term in Equation 9 for all i takes a total of O ( nq ) time.

Original: As just shown, the second term can be computed in O ( nq ) time for each i , giving a total run time of O ( nq 2 ) .

Original: Algorithm

Original: (cid:80) c ( ℓ ) ||

Original: = ℓ ||

Original: Require: p ∈

Original: [ r ] n , c, u ∈ R n

Original: Ensure: x = (cid:80) mℓ =1 c ( ℓ ) ||

Original: u p = ℓ ||1: y ← 0 m 2: for j ∈ [ n ] do y p

Original: j ← y p j + √ c j u j ▷ √ c

Original: j may be imaginary, but it will later be squared.

Original: With modifications, we could avoid complex types.

Original: 3: x ←4: for j ∈ [ r ] do x ← x + y 2 j return x

Original: C Experimental Details

Original: We initialize 160 random bit orders.

Original: For each one, we initialize three weight vectors w : uniform, uniform except the last bit is 0.5, and uniform except the last bit is 0.9.

Original: Out of these 480 initializations, we draw 20 samples via Boltzman sampling [5] on the log likelihood of the training data (after standardizing the values to have zero mean and unit variance).

Original: Then, we optimize the weights and bit order with BFGS as described in Section 5, using line search with Wofle conditions, with no extra gradient computations during line search.

Original: This allows fewer calculations of the gradient relative to the cheaper calculation of the loss.

Original: The BT column in Table 1 refers to the performance of the binary tree kernel, using the weights and bit order that gave the lowest training NLL out of these 20 trained models.

Original: BTE produces a Gaussian mixture model at each predictive location, mixing over the predictive

Original: Gaussians produced by each of these 20 trained models.

Original: The relative weights of each Gaussian in the mixture depends on the training NLL of the model that produced it.

Original: We weight the models according to the softmax of the per-data-point NLL with a temperature of 0.01.

Original: We follow the same train/test/validation splits as Wang et al.

Original: [25], but we never use the validation set, which the methods we compare against need.

Original: Thus, we could add the validation data to the training data for the binary tree kernel and call it a fair comparison, but we didn’t do this, so as not to confuse the origin of the binary tree kernel’s success.

Original: D Additional Empirical Evaluation D.1 Sensitivity Analysis on Precision p

Original: In Table 2, we evaluate the performance of the BT and BTE kernels on the precision p for p ∈ (2 , 4 , 8) .

Original: In all problems, we find that RMSE and NLL decrease monotonically as p increases and wall time increases monotonically.

Original: For best predictive performance, p should be set as large as possible subject resource constraints.

Original: This validates that our heuristic rule for setting p is a reasonable choice in a variety of settings.

Original: D.2 A Simple Performance Improvement

Original: As shown in Table 3, the heuristic rule for setting p results in bit strings that preserve the uniqueness of the raw training set for most datasets.

Original: However, for the 3dRoad, Song, and Buzz datasets, the percentage of unique bit strings is very low relative to the percentage of unique training rows.

Original: We note that this observation is from exploratory data analysis (EDA) and can be made before model fitting.

Original: There are two key ways that mapping training rows to bit strings can lower the percent of unique examples.

Original: First, a low percentage of unique bit strings can arise if a given input feature has a nonuniform input distribution which can lead to multiple different inputs mapping to the same discrete bucket.

Original: To alleviate, this problem EDA can be used to determine a suitable feature transformation.

Original: For example, we apply a strictly increasing, piecewise linear transformation to the data, mapping the k th percentile of each dimension to k/ 100 , for k ∈ { 0 , 10 , 20 , ..., 100 } .

Original: This resembles an empirical cumulative density function (ECDF).

Original: Second, if the precision p is set too low, then multiple different inputs can map to the same bit string.

Original: A simple solution is to iteratively increase the precision p

Original: [Table]

Original: based on the difference between the percentage of unique rows in the raw training set and unique bit strings under precision p .

Original: The last column of Table 3 reports the percentage of unique bit strings on the 3dRoad, Song

Original: [Table]

Original: and Buzz datasets after applying transforming each feature through its ECDF and increasing the precision if need be.

Original: These changes (all made through EDA) lead to significantly more unique bit strings.

Original: Table 4 shows the performance on these datasets under the proposed ECDF transformations and settings of p .

Original: We find that the BTE outperforms all methods on 3dRoad and

Original: Buzz with respect to RSME and NLL under these proposed changes.

Original: The performance on Song also improves.

Original: [Table]

Original: [Table]

Original: a single GPU (Tesla V100-SXM2-16GB for BT and BTE and Tesla V100-SXM2-32GB for the other methods) after transforming each feature through its ECDF and using precision p new .

Original: The asterisk indicates an estimate of the time from the reported training time on 8 GPUS, assuming linear speedup in number of GPUs and independent noise in training times per GPU.

Original: D.3 Comparison with Simplex-GP

Original: Here, we compare against Simplex-GP [15] using the 4 datasets that are common to both papers.

Original: We find that BTE outperforms Simplex GP on all datasets with respect to NLL and on 3 out of 4 datasets with respect to RMSE.

Original: Furthermore, Simplex-GP is not the top performing method on any dataset.

Original: [Table]

Original: E Societal Impacts

Original: More accurate machine learning models allow for better decision making.

Original: Improvements in decision making can lead to improved societal outcomes in many applications such as early disease detection.

Original: However when predictions of a machine learning algorithm are confident, we may be more compelled to act on them, which could lead to increased risk in delicate settings.

Original: Therefore, confident and wrong predictions, resulting from poor and confident generalization due to overfitting, can lead to worse outcomes.

Original: When using binary tree kernels for Gaussian processes, overfitting is possible.

Original: That said, Gaussian processes do quantify their uncertainty very naturally, which can be helpful for knowing when to take their predictions with a grain, or a heap, of salt.

Original: As with all machine learning models, what outcomes are predicted and how those predictions are used are ultimately decided by to the practitioner, and those decisions contribute to whether the model impacts society in positive or negative ways.