Original: Adaptive Cholesky Gaussian Processes
Revised: Adaptive Cholesky Gaussian Processes

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We present a method to ﬁt exact Gaussian process models to large datasets byconsidering only a subset of the data.
Revised: We present a method to fit exact Gaussian process models to large datasets byconsidering only a subset of the data.

Original: Our approach is novel in that the size ofthe subset is selected on the ﬂy during exact inference with little computationaloverhead.
Revised: Our approach is novel in that the size ofthe subset is selected on the fly during exact inference with little computationaloverhead.

Original: From an empirical observation that the log-marginal likelihood oftenexhibits a linear trend once a sufﬁcient subset of a dataset has been observed,we conclude that many large datasets contain redundant information that onlyslightly affects the posterior.
Revised: From an empirical observation that the log-marginal likelihood oftenexhibits a linear trend once a sufficient subset of a dataset has been observed,we conclude that many large datasets contain redundant information that onlyslightly affects the posterior.

Original: Based on this, we provide probabilistic bounds onthe full model evidence that can identify such subsets.
Revised: Based on this, we provide probabilistic bounds onthe full model evidence that can identify such subsets.

Original: Remarkably, these boundsare largely composed of terms that appear in intermediate steps of the standard
Revised: Remarkably, these boundsare largely composed of terms that appear in intermediate steps of the standard

Original: Cholesky decomposition, allowing us to modify the algorithm to adaptively stopthe decomposition once enough data have been observed.
Revised: Cholesky decomposition, allowing us to modify the algorithm to adaptively stopthe decomposition once enough data have been observed.

Original: Empirically, we showthat our method can be directly plugged into well-known inference schemes to ﬁtexact Gaussian process models to large datasets.
Revised: Empirically, we showthat our method can be directly plugged into well-known inference schemes to fitexact Gaussian process models to large datasets

Original:  Introduction
Revised:  Introduction

Original: It has been observed (Chalupka et al., 2013) that the random-subset-of-data approximation canbe a hard-to-beat baseline for approximate Gaussian process inference.
Revised: It has been observed (Chalupka et al., 2013) that the random-subset-of-data approximation canbe a hard-to-beat baseline for approximate Gaussian process inference.

Original: However, the question ofhow to choose the size of the subset is non-trivial to answer.
Revised: However, the question ofhow to choose the size of the subset is non-trivial to answer.

Original: Here we make an attempt.
Revised: Here we make an attempt.

Original: The keycomputational challenge in Gaussian process regression is to evaluate the log-marginal likelihood ofthe N observed data points, which is known to have cubic complexity (Rasmussen & Williams, 2006).
Revised: The keycomputational challenge in Gaussian process regression is to evaluate the log-marginal likelihood ofthe N observed data points, which is known to have cubic complexity (Rasmussen & Williams, 2006).

Original: In order to arrive at a computationally less expensive approximation of this log-marginal likelihood,we ﬁrst empirically study its behavior as we increase the number of observations.
Revised: In order to arrive at a computationally less expensive approximation of this log-marginal likelihood,we first empirically study its behavior as we increase the number of observations.

Original: Figure 1 show thisprogression for a variety of models.
Revised: Figure 1 show thisprogression for a variety of models.

Original: We elaborate on this ﬁgure in Section 3.1, but for now note thatafter a certain number of observations, determined by the model and the dataset, the log-marginallikelihood starts to progress with a linear trend.
Revised: We elaborate on this figure in Section 3.1, but for now note thatafter a certain number of observations, determined by the model and the dataset, the log-marginallikelihood starts to progress with a linear trend.

Original: This suggest that we may leverage this near-linearityto estimate the log-marginal likelihood of the full dataset after having seen only a subset of the data.
Revised: This suggest that we may leverage this near-linearityto estimate the log-marginal likelihood of the full dataset after having seen only a subset of the data.

Original: However, as the point-of-linearity differs between models, this must be estimated on-the-ﬂy to keepcomputations tractable.
Revised: However, as the point-of-linearity differs between models, this must be estimated on-the-fly to keepcomputations tractable.

Original: In this paper, we approach the problem from a  probabilistic numerics perspective (Hennig et al.,2015).
Revised: In this paper, we approach the problem from a (frequentist) probabilistic numerics perspective(Hennig et al., 2015).

Original: By treating the dataset as a collection of independent and identically distributed randomvariables, we provide expected upper and lower bounds on the log-marginal likelihood, which becometight when the above-mentioned linear trend arises.
Revised: By treating the dataset as a collection of independent and identically distributedrandom variables, a common assumption in the frequentist literature, we provide expected upper andlower bounds on the log-marginal likelihood, which become tight when the above-mentioned lineartrend arises.

Original: We provide a particularly efﬁcient algorithmfor computing the bounds that leverage the intermediate computations performed by the Choleskydecomposition that is commonly used for evaluating the log-marginal likelihood.
Revised: We provide a particularly efficient algorithm for computing the bounds that leveragethe intermediate computations performed by the Cholesky decomposition that is commonly used forevaluating the log-marginal likelihood.

Original: The bounds aretherefore practically free to evaluate.
Revised: The bounds are therefore practically free to evaluate.

Original: We further show that these bounds allow us to predict whenthe linear trend determines the full-data log-marginal likelihood, such that we can phrase an optimalstopping problem to determine a suitable subset of the data for a particular model.
Revised: Wefurther show that these bounds allow us to predict when the linear trend determines the full-datalog-marginal likelihood, such that we can phrase an optimal stopping problem to determine a suitablesubset of the data for a particular model.

Original: We refer to ourmethod as Adaptive Cholesky Gaussian Process (ACGP).
Revised: We refer to our method as Adaptive Cholesky Gaussian Process (ACGP).

Original: Our approach has a complexity of O ( M 3 ) , where M is the processed subset-size, inducing an overhead of O ( M ) to the Cholesky.
Revised: Our approach has a complexity of O ( M 3 ) , where M is the processed subset-size, inducing an overhead of O ( M ) to the Cholesky.

Original: The maindifference to previous work is that our algorithm does not necessarily look at the whole dataset.
Revised: The main difference to previous work is that ouralgorithm does not necessarily look at the whole dataset, which makes it particularly useful in settingswhere even O ( N ) operations are intractable.

Revised: When a dataset contains a large amount of redundantdata, ACGP allows the inference procedure to stop early, saving precious compute—especially whenthe kernel function is expensive to evaluate.

Original:  Background
Revised:  Background

Original: We use a PYTHON -inspired index notation, abbreviating for example [ y 1 , . . .
Revised: We use a PYTHON -inspired index notation, abbreviating for example [ y 1 , . . .

Original: , y n − 1 ] (cid:62) as y : n ; observethat the indexing starts at 1.
Revised: , y n − 1 ] ⊤ as y : n ; observethat the indexing starts at 1.

Original: With Diag we deﬁne the operator that sets all off-diagonal entries of amatrix to 0 .
Revised: With Diag we define the operator that sets all off-diagonal entries of amatrix to 0 .

Original: 2.1 Gaussian Process Regression
Revised: 2.1 Gaussian Process Regression

Original: We start by brieﬂy reviewing Gaussian process (GP) models and how they are trained (see Rasmussen& Williams (2006, Chapter 2)).
Revised: We start by briefly reviewing Gaussian process (GP) models and how they are trained (see Rasmussen& Williams (2006, Chapter 2)).

Original: We consider the training dataset D = { x n , y n } Nn =1 with inputsx n ∈ R p and outputs y n ∈ R .
Revised: We consider the training dataset D = { x n , y n } Nn =1 with inputsx n ∈ R p and outputs y n ∈ R .

Original: The inputs are collected in the matrix X =
Revised: The inputs are collected in the matrix X =

Original: [ x 1 , x 2 , . . .
Revised: [ x 1 , x 2 , . . .

Original: , x N ] (cid:62) ∈ 47 R N × p .
Revised: , x N ] ⊤ ∈ 51 R N × p .

Original: A GP f ∼ GP ( m ( x ) , k
Revised: A GP f ∼ GP ( m ( x ) , k

Original: ( x , x (cid:48) )) is a collection of random variables deﬁned in terms ofa mean function, m ( x ) , and a covariance function or kernel , k
Revised: ( x , x ′ )) is a collection of random variables defined in terms ofa mean function, m ( x ) , and a covariance function or kernel , k

Original: ( x , x (cid:48) ) =
Revised: ( x , x ′ ) =

Original: cov( f ( x ) , f ( x (cid:48) )) , suchthat any ﬁnite amount of random variables has a Gaussian distribution.
Revised: cov( f ( x ) , f ( x ′ )) , suchthat any finite amount of random variables has a Gaussian distribution.

Original: Hence, the prior overf := f ( X ) is N ( f ; m ( X ) , K ff ) , where we have used the shorthand notation K ff = k
Revised: Hence, the prior overf := f ( X ) is N ( f ; m ( X ) , K ff ) , where we have used the shorthand notation K ff = k

Original: ( X , X ) .
Revised: ( X , X ) .

Original: Wewill consider the observations y as being noise-corrupted versions of the function values f , and weshall parameterize this corruption through the likelihood function p ( y | f ) , which for regression tasksis typically assumed to be Gaussian, p ( y | f ) =
Revised: Wewill consider the observations y as being noise-corrupted versions of the function values f , and weshall parameterize this corruption through the likelihood function p ( y | f ) , which for regression tasksis typically assumed to be Gaussian, p ( y | f ) =

Original: N ( f , σ 2 I ) .
Revised: N ( f , σ 2 I ) .

Original: For such a model, the posterior over testinputs X ∗ can be computed in closed-form: p ( f ∗ | y )
Revised: For such a model, the posterior over testinputs X ∗ can be computed in closed-form: p ( f ∗ | y )

Original: =
Revised: =

Original: N ( m ∗ , S ∗ ) , where
Revised: N ( m ∗ , S ∗ ) , where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: with K :
Revised: with K :

Original: = K ff + σ 2 I .
Revised: = K ff + σ 2 I .

Original: By marginalizing over the function values of the likelihood distribution,we obtain the marginal likelihood, p ( y ) =
Revised: By marginalizing over the function values of the likelihood distribution,we obtain the marginal likelihood, p ( y ) =

Original: (cid:82) p ( y | f ) p ( f ) d f , the de facto metric for comparingthe performance of models in the Bayesian framework.
Revised: (cid:82) p ( y | f ) p ( f ) d f , the de facto metric for comparingthe performance of models in the Bayesian framework.

Original: While this integral is not tractable ingeneral, it does have a closed-form solution for Gaussian process regression.
Revised: While this integral is not tractable ingeneral, it does have a closed-form solution for Gaussian process regression.

Original: Given the GP prior,p ( f ) = N ( 0 , K ff ) , and the Gaussian likelihood, the log-marginal likelihood distribution can befound to be
Revised: Given the GP prior,p ( f ) = N ( 0 , K ff ) , and the Gaussian likelihood, the log-marginal likelihood distribution can befound to be

Original: [Equation]
Revised: [Equation]

Revised: Background

Original: 2.2 Background onthe Cholesky decomposition
Revised: on

Revised: the Cholesky decomposition

Original: Inverting covariance matrices such as K is a slow and numerically unstable procedure.
Revised: Inverting covariance matrices such as K is a slow and numerically unstable procedure.

Original: Therefore, inpractice, one typically leverages the Cholesky decomposition of the covariance matrices to compute the inverses.
Revised: Therefore, inpractice, one typically leverages the Cholesky decomposition of the covariance matrices to computethe inverses.

Original: The Cholesky decomposition of a symmetric and positive deﬁnite matrix K is theunique, lower 1 triangular matrix L such that K = LL (cid:62) (Golub & Van Loan, 2013, Theorem 4.2.7).
Revised: The Cholesky decomposition of a symmetric and positive definite matrix K is theunique, lower 1 triangular matrix L such that K = LL ⊤ (Golub & Van Loan, 2013, Theorem 4.2.7).

Original: The advantage of having such a decomposition is that inversion with triangular matrices amounts to
Revised: The advantage of having such a decomposition is that inversion with triangular matrices amounts to

Original: Gaussian elimination.
Revised: Gaussian elimination.

Original: There are different options to compute L .
Revised: There are different options to compute L .

Original: The Cholesky of a 1 × 1 matrix isthe square root of the scalar.
Revised: The Cholesky of a 1 × 1 matrix isthe square root of the scalar.

Original: For larger matrices,
Revised: For larger matrices,

Original: [Equation]
Revised: [Equation]

Original: where T :=
Revised: where T :=

Original: K : s,s : chol
Revised: K : s,s : chol

Original: [ K : s, : s ] −(cid:62) and s is any integer between 1 and the size of K .
Revised: [ K : s, : s ] −⊤ and s is any integer between 1 and the size of K .

Original: Hence, extendinga given Cholesky to a larger matrix requires three steps:
Revised: Hence, extendinga given Cholesky to a larger matrix requires three steps:

Original: solve the linear equation system T ,2.
Revised: solve the linear equation system T ,2.

Original: apply the downdate K s : ,s : − T T (cid:62) and3.
Revised: apply the downdate K s : ,s : − T T ⊤ and3.

Original: compute the Cholesky of the down-dated matrix.
Revised: compute the Cholesky of the down-dated matrix.

Original: An important observation is that K s : ,s : − T T (cid:62) is the posterior covariance matrix S ∗ + σ 

Revised: An important observation is that K s : ,s : − T T ⊤ is the posterior covariance matrix S ∗ + σ

Original: I whenconsidering X s : as test points.
Revised: I whenconsidering X s : as test points.

Original: We will make use of this observation in Section 3.5.
Revised: We will make use of this observation in Section 3.5.

Original: The logdeterminant of K can be obtained from the Cholesky using log det [ K ] = 2 (cid:80) Nn =1 log L nn .
Revised: The logdeterminant of K can be obtained from the Cholesky using log det [ K ] = 2 (cid:80) Nn =1 log L nn .

Original: Asimilar recursive relationship exists between the quadratic form y (cid:62) K − 1 y and L − 1 y (see appendix,
Revised: Asimilar recursive relationship exists between the quadratic form y ⊤ K − 1 y and L − 1 y (see appendix,

Original: Equation (22)).
Revised: Equation (22)).

Original: 2.3 Related work
Revised: 2.3 Related work

Original: Much work has gone into tractable approximations to the log-marginal likelihood.
Revised: Much work has gone into tractable approximations to the log-marginal likelihood.

Original: Arguably, themost popular approximation methods for GPs are inducing point methods (Quiñonero-Candela &
Revised: Arguably, themost popular approximation methods for GPs are inducing point methods (Quiñonero-Candela &

Original: Rasmussen, 2005; Snelson & Ghahramani, 2006; Titsias, 2009; Hensman et al., 2013, 2017; Shi et al.
Revised: Rasmussen, 2005; Snelson & Ghahramani, 2006; Titsias, 2009; Hensman et al., 2013, 2017; Shi et al.

Original: ,2020; Artemev et al., 2021), where the dataset is approximated through a set of pseudo-data points(inducing points), summarizing information from nearby data.
Revised: ,2020; Artemev et al., 2021), where the dataset is approximated through a set of pseudo-data points(inducing points), summarizing information from nearby data.

Original: Other approaches involve buildingapproximations to K (Fine & Scheinberg, 2001; Harbrecht et al., 2012; Wilson & Nickisch, 2015;
Revised: Other approaches involve buildingapproximations to K (Fine & Scheinberg, 2001; Harbrecht et al., 2012; Wilson & Nickisch, 2015;

Original: Rudi et al., 2017; Wang et al., 2019) or aggregating of distributed local approximations (Gal et al.
Revised: Rudi et al., 2017; Wang et al., 2019) or aggregating of distributed local approximations (Gal et al.

Original: ,2014; Deisenroth & Ng, 2015).
Revised: ,2014; Deisenroth & Ng, 2015).

Original: One may also consider separately the approximation of the quadraticform via linear solvers such as conjugate gradients (Hestenes & Stiefel, 1952; Cutajar et al., 2016)and the approximation of the log-determinant (Fitzsimons et al., 2017a,b; Dong et al., 2017).
Revised: One may also consider separately the approximation of the quadraticform via linear solvers such as conjugate gradients (Hestenes & Stiefel, 1952; Cutajar et al., 2016)and the approximation of the log-determinant (Fitzsimons et al., 2017a,b; Dong et al., 2017).

Original: Anotherline of research is scaling the hardware (Nguyen et al., 2019).
Revised: Another line of research is scaling the hardware (Nguyen et al., 2019).

Original: All above referenced approaches have computational complexity at least O ( N ) (with the exceptionof Hensman et al.
Revised: All above referenced approaches have computational complexity at least O ( N ) (with the exceptionof Hensman et al.

Original: (2013) since it uses mini-batching).
Revised: (2013) since it uses mini-batching).

Original: However, the size of a dataset is seldom aparticularly chosen value but rather the ad-hoc end of the sampling procedure.
Revised: However, the size of a dataset is seldom aparticularly chosen value but rather the ad-hoc end of the sampling procedure.

Original: The dependence onthe dataset size implies that more data requires more computational budget even though more datamight not be helpful.
Revised: The dependence onthe dataset size implies that more data requires more computational budget even though more datamight not be helpful.

Original: This is the main motivation for our work: to derive an approximation algorithmwhere computational complexity does not depend on redundant data.
Revised: This is the main motivation for our work: to derive an approximation algorithm where computational complexity does not depend on redundant data.

Original: The work closest in spirit to the present paper is by Artemev et al.
Revised: The work closest in spirit to the present paper is by Artemev et al.

Original: (2021), who also propose lower andupper bounds on quadratic form and log-determinant.
Revised: (2021), who also propose lower andupper bounds on quadratic form and log-determinant.

Original: There are a number of differences, however.
Revised: There are a number of differences, however.

Original: Their bound relies on the method of conjugate gradients where we work directly with the Choleskydecomposition.
Revised: Their bound relies on the method of conjugate gradients where we work directly with the Cholesky decomposition.

Original: Furthermore, while their bounds are deterministic, ours are probabilistic, which canmake them tighter in certain cases, as they do not need to hold for all worst-case scenarios.
Revised: Furthermore, while their bounds are deterministic, ours are probabilistic, which can make them tighter in certain cases, as they do not need to hold for all worst-case scenarios.

Original: This isalso the main difference to the work of Hensman et al.
Revised: This is also the main difference to the work of Hensman et al.

Original: Their bounds allow for mini-batching,but these are inherently deterministic when applied with full batch size.
Revised: Their bounds allow for mini-batching,but these are inherently deterministic when applied with full batch size.

Original:  Methodology
Revised:  Methodology

Original: In the following, we will sketch our method.
Revised: In the following, we will sketch our method.

Original: Our main goal is to convey the idea and intuition.
Revised: Our main goal is to convey the idea and intuition.

Original: To
Revised: Tothis end, we use suggestive notation. We refer the reader to the appendix for a more thorough andformal treatment.

Original: 3.1 Intuition on the linear extrapolation
Revised: 3.1 Intuition on the linear extrapolation

Original: The marginal likelihood is typically presented as a joint distribution, but, one can also view it from acumulative perspective as the sum of log-conditionals:
Revised: The marginal likelihood is typically presented as a joint distribution, but, one can also view it from acumulative perspective as the sum of log-conditionals:

Original: [Equation]
Revised: [Equation]

Original: With this equation in hand, the phenomena in Figure 1 becomes much clearer.
Revised: With this equation in hand, the phenomena in Figure 1 becomes much clearer.

Original: The ﬁgure showsthe value of Equation (3) for an increasing number of observations n .
Revised: The figure showsthe value of Equation (3) for an increasing number of observations n .

Original: When the plot exhibits alinear trend, it is because the summands log p ( y n | y : n ) become approximately constant, implyingthat the model is not gaining additional knowledge.
Revised: When the plot exhibits alinear trend, it is because the summands log p ( y n | y : n ) become approximately constant, implyingthat the model is not gaining additional knowledge.

Original: In other words, new outputs are conditionallyindependent given the output observations seen so far.
Revised: In other words, new outputs are conditionallyindependent given the output observations seen so far.

Original: The key problem addressed in this paper is how to estimate the full marginal likelihood, p ( y ) , fromonly a subset of M observations.
Revised: The key problem addressed in this paper is how to estimate the full marginal likelihood, p ( y ) , fromonly a subset of M observations.

Original: The cumulative view of the log-marginal likelihood in Equation (3)is our starting point.
Revised: The cumulative view of the log-marginal likelihood in Equation (3)is our starting point.

Original: In particular, we will provide bounds, which are functions of seen observations,on the estimate of the full marginal likelihood.
Revised: In particular, we will provide bounds, which are functions of seen observations,on the estimate of the full marginal likelihood.

Original: These bounds will allow us to decide, on the ﬂy, whenwe have seen enough observations to accurately estimate the full marginal likelihood.
Revised: These bounds will allow us to decide, on the fly, whenwe have seen enough observations to accurately estimate the full marginal likelihood.

Original: 3.2 Stopping strategy
Revised: 3.2 Stopping strategy

Original: Suppose that we have processed M data points with N − M data points yet to be seen.
Revised: Suppose that we have processed M data points with N − M data points yet to be seen.

Original: We can thendecompose Equation (3) into a sum of terms which have already been computed and a remaining sum 
Revised: We can thendecompose Equation (3) into a sum of terms which have already been computed and a remaining sum [Equation]

Original: [Equation]
Revised: [Equation]

Revised: B : remaining .

Original: Recall that we consider the x i , y i as independent and identically distributed random variables.
Revised: Recall that we consider the x i , y i as independent and identically distributed random variables.

Original: Hence, we could estimate B as ( N − M ) A /M .
Revised: Hence, we could estimate B as ( N − M ) A /M .

Original: Yet this is estimator is biased, since( x M +1 , y M +1 ) , . . .
Revised: Yet this is estimator is biased, since( x M +1 , y M +1 ) , . . .

Original: , ( x N , y N ) interact non-linearly through the kernel function.
Revised: , ( x N , y N ) interact non-linearly through the kernel function.

Original: Instead, we willderive unbiased lower and upper bounds, L and U .
Revised: Instead, we willderive unbiased lower and upper bounds, L and U .

Original: To obtain unbiased estimates, we use the last- mprocessed points, such that conditioned on the points up to s :
Revised: To obtain unbiased estimates, we use the last- mprocessed points, such that conditioned on the points up to s :

Original: =
Revised: =

Original: M − m , log p ( y ) can be boundedfrom above and below:
Revised: M − m , log p ( y ) can be boundedfrom above and below:

Original: [Equation]
Revised: [Equation]

Original: and the observations from s to M can be used to estimate L and U .
Revised: and the observations from s to M can be used to estimate L and U .

Original: We can then detect when theupper and lower bounds are sufﬁciently near each other, and stop computations early when theapproximation is sufﬁciently good.
Revised: We can then detect when theupper and lower bounds are sufficiently near each other, and stop computations early when theapproximation is sufficiently good.

Original: More precisely, given a desired relative error r , we stop when
Revised: More precisely, given a desired relative error r , we stop when

Original: [Equation]
Revised: [Equation]

Original: If the bounds hold, then the estimator ( L + U ) / 2 achieves the desired relative error (Lemma 16 inappendix).
Revised: If the bounds hold, then the estimator ( L + U ) / 2 achieves the desired relative error (Lemma 16 inappendix).

Original: This is in contrast to other approximations, where one speciﬁes a computational budget,rather than a desired accuracy.
Revised: This is in contrast to other approximations, where one specifies a computational budget,rather than a desired accuracy.

Original: 3.3 Bounds
Revised: 3.3 Bounds

Original: From Equation (1), we see that the log-marginal likelihood decomposes into the log-determinant ofthe kernel matrix, a quadratic term, and a constant term.
Revised: From Equation (1), we see that the log-marginal likelihood decomposes into the log-determinant ofthe kernel matrix, a quadratic term, and a constant term.

Original: In the following we present upper and lowerbounds for both the log-determinant ( U D and L D , respectively) and the quadratic term ( U Q and L Q ).
Revised: In the following we present upper and lowerbounds for both the log-determinant ( U D and L D , respectively) and the quadratic term ( U Q and L Q ).

Original: 140 We will need the posterior equations for the observations, that is p ( y n | y : n ) , and we will need them 141 as functions of test inputs x ∗ and x (cid:48)∗ .
Revised: We will need the posterior equations for the observations, that is p ( y n | y : n ) , and we will need themas functions of test inputs x ∗ and x ′∗ .

Original: To this end, deﬁne
Revised: To this end, define

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: such that p ( y n | y : n ) = N ( y n ; m ( n ) ∗ ( x n ) , Σ ( n ) ∗ ( x n , x n )) , which allows us to rewrite Equation (3)as
Revised: such that p ( y n | y : n ) = N ( y n ; m ( n ) ∗ ( x n ) , Σ ( n ) ∗ ( x n , x n )) , which allows us to rewrite Equation (3)as

Original: [Equation]
Revised: [Equation]

Original: This reveals that the log-determinant can be written as a sum of posterior variances and the quadraticform has an expression as normalized square errors.
Revised: This reveals that the log-determinant can be written as a sum of posterior variances and the quadraticform has an expression as normalized square errors.

Original: Other key ingredients for our bounds areestimates for average posterior variance and average covariance.
Revised: Other key ingredients for our bounds areestimates for average posterior variance and average covariance.

Original: Therefore deﬁne the shorthands
Revised: Therefore define the shorthands

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where e j ∈ R m is the j -th standard basis vector.
Revised: where e j ∈ R m is the j -th standard basis vector.

Original: The matrix V is simply the diagonal of the posteriorcovariance matrix Σ ∗ .
Revised: The matrix V is simply the diagonal of the posteriorcovariance matrix Σ ∗ .

Original: The matrix C consists of every second entry of the ﬁrst off-diagonal of Σ ∗ .
Revised: The matrix C consists of every second entry of the first off-diagonal of Σ ∗ .

Original: These elements are placed on the diagonal with every second element being 0 .
Revised: These elements are placed on the diagonal with every second element being 0 .

Original: The reason for takingevery second element is of theoretical nature, see Remark 2 in the appendix.
Revised: The reason for takingevery second element is of theoretical nature, see Remark 2 in the appendix.

Original: In practice we use thefull off-diagonal.
Revised: In practice we use thefull off-diagonal.

Original: 3.3.1 Bounds on the log-determinant
Revised: 3.3.1 Bounds on the log-determinant

Original: Both bounds, lower and upper, use that log det [ K ] = log det 
Revised: Both bounds, lower and upper, use that log det [ K ] = log det [ K : s, : s ] + log det (cid:104) Σ ( s ) ∗ ( X s : ,

Original: [Equation]
Revised: X s : )

Original: which follows from the matrix-determinant lemma.
Revised: (cid:105)which follows from the matrix-determinant lemma.

Original: The ﬁrst term is available from the alreadyprocessed datapoints.
Revised: The first term is available from the alreadyprocessed datapoints.

Original: It is the second addend that needs to be estimated, which we approach from theperspective of Equation (5).
Revised: It is the second addend that needs to be estimated, which we approach from theperspective of Equation (5).

Original: It is well-established that, for a ﬁxed input, more observations decreasethe posterior variance, and this decrease cannot cross the threshold σ 2 (Rasmussen & Williams,2006, Question 2.9.4).
Revised: It is well-established that, for a fixed input, more observations decreasethe posterior variance, and this decrease cannot cross the threshold σ 2 (Rasmussen & Williams,2006, Question 2.9.4).

Original: This remains true when taking the expectation over the input.
Revised: This remains true when taking the expectation over the input.

Original: Hence, theaverage of the posterior variances for inputs X s : M is with high probability an overestimate of theaverage posterior variance for inputs with higher index.
Revised: Hence, theaverage of the posterior variances for inputs X s : M is with high probability an overestimate of theaverage posterior variance for inputs with higher index.

Original: This motivates our upper bound on thelog-determinant:
Revised: This motivates our upper bound on thelog-determinant:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: To arrive at the lower bound on the log-determinant, we need an argument about how fast theaverage posterior variance could decrease which is governed by the covariance between inputs.
Revised: To arrive at the lower bound on the log-determinant, we need an argument about how fast theaverage posterior variance could decrease which is governed by the covariance between inputs.

Original: Thevariable ρ D measures the average covariance, and we show in Theorem 6 in the appendix that thisoverestimates the decrease per step with high probability.
Revised: Thevariable ρ D measures the average covariance, and we show in Theorem 6 in the appendix that thisoverestimates the decrease per step with high probability.

Original: Since the decrease cannot exceed σ 2 , weintroduce ψ D to denote the step which would cross this threshold.
Revised: Since the decrease cannot exceed σ 2 , weintroduce ψ D to denote the step which would cross this threshold.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Both bounds collapse to the exact solution when s = N .
Revised: Both bounds collapse to the exact solution when s = N .

Original: The bounds are close when the averagecovariance between inputs, ρ D , is small.
Revised: The bounds are close when the averagecovariance between inputs, ρ D , is small.

Original: This occurs for example when the average variance is closeto σ 2 since the variance is an upper bound to the covariance.
Revised: This occurs for example when the average variance is closeto σ 2 since the variance is an upper bound to the covariance.

Original: Another case where ρ D is small is whenpoints are not correlated to begin with.
Revised: Another case where ρ D is small is whenpoints are not correlated to begin with.

Original: 3.3.2 Bounds on the quadratic term
Revised: Bounds on the quadratic term

Original: Denote with r ∗ := y s : − m ( s ) ∗ ( X s : ) the prediction errors (the residuals), when considering theﬁrst s points as training set and the remaining inputs as test set.
Revised: Denote with r ∗ := y s : − m ( s ) ∗ ( X s : ) the prediction errors (the residuals), when considering thefirst s points as training set and the remaining inputs as test set.

Original: Analogous to the bounds on thelog-determinant, one can show with the matrix inversion lemma that y (cid:62) K − 1 y =
Revised: Analogous to the bounds on thelog-determinant, one can show with the matrix inversion lemma that y ⊤ K − 1 y =

Original: y (cid:62) : s K − 1 : s, : s y : s + r (cid:62)∗ ( Σ ( s ) ∗ ( X s : )) − 1 r ∗ .
Revised: y ⊤ : s K − 1 : s, : s y : s +r ⊤∗ ( Σ ( s ) ∗ ( X s : )) − 1 r ∗ .

Original: Again, the ﬁrst term will turn out to be already computed.
Revised: Again, the first term will turn out to be already computed.

Original: With a slight abuseof notation let r ∗ :=
Revised: With a slight abuseof notation let r ∗ :=

Original: y s : M − m ( s ) ∗ ( X s : M ) , that is, we consider only the ﬁrst m entries.
Revised: y s : M − m ( s ) ∗ ( X s : M ) , that is, we consider only the first m entries.

Original: Our lowerbound arises from another well-known lower bound: a (cid:62) A − 1 a ≥ 2 a (cid:62) b − b (cid:62) Ab for all b (seefor example Kim & Teh (2018);
Revised: Our lowerbound arises from another well-known lower bound: a ⊤ A − 1 a ≥ 2 a ⊤ b − b ⊤ Ab for all b (seefor example Kim & Teh (2018);

Original: Artemev et al.
Revised: Artemev et al.

Original: We choose b := α 1 where α is chosento maximize the bound.
Revised: We choose b := α 1 where α is chosento maximize the bound.

Original: The result, after some cancellations, is the following lower bound on thequadratic term:
Revised: The result, after some cancellations, is the following lower bound on thequadratic term:

Original: [Equation]
Revised: [Equation]

Original: The α maximizing above bound is µ Q / ρ 2 Q , which is the value we chose in our implementation.
Revised: The α maximizing above bound is µ Q / ρ 2 Q , which is the value we chose in our implementation.

Original: However, note that L Q is an expected lower bound only if α depends on variables with index smallerthan s .
Revised: However, note that L Q is an expected lower bound only if α depends on variables with index smallerthan s .

Original: Our upper bound arises from the element-wise perspective of Equation (5).
Revised: Our upper bound arises from the element-wise perspective of Equation (5).

Original: We assume that theexpected mean square error ( y n − m ( n − 1) ∗ ( x n )) 2 decreases with more observations.
Revised: We assume that theexpected mean square error ( y n − m ( n − 1) ∗ ( x n )) 2 decreases with more observations.

Original: However, thoughmean square error and variance decrease, their expected ratio may increase or decrease dependingon the choice of kernel, dataset and number of processed points.
Revised: However, thoughmean square error and variance decrease, their expected ratio may increase or decrease dependingon the choice of kernel, dataset and number of processed points.

Original: Using the average error calibrationwith a correction for the decreasing variance, we arrive at our upper bound on the quadratic term:
Revised: Using the average error calibrationwith a correction for the decreasing variance, we arrive at our upper bound on the quadratic term:

Original: [Equation]
Revised: [Equation]

Original: In our implementation we use a slightly different upper bound.
Revised: In our implementation we use a slightly different upper bound.

Original: The estimate of the possible decreaseof the variance uses the same technique as the lower bound for the log-determinant.
Revised: The estimate of the possible decreaseof the variance uses the same technique as the lower bound for the log-determinant.

Original: Therefore we candeﬁne an analogue to Equation (8) determining the step when the variance estimate falls below σ 2 .
Revised: Therefore we candefine an analogue to Equation (8) determining the step when the variance estimate falls below σ 2 .

Original: Inour implementation, addends of the quadratic after this step are estimated by the more conservativeσ − 2 µ Q .
Revised: Inour implementation, addends of the quadratic after this step are estimated by the more conservativeσ − 2 µ Q .

Original: Again, the bounds collapse to the true quantity when s = N .
Revised: Again, the bounds collapse to the true quantity when s = N .

Original: The bounds will give goodestimates when the average covariance between inputs, represented by the matrix C , is low or whenthe model can predict new data well, that is, when r ∗ is close to 0 .
Revised: The bounds will give goodestimates when the average covariance between inputs, represented by the matrix C , is low or whenthe model can predict new data well, that is, when r ∗ is close to 0 .

Original: 3.4 Validity of bounds and stopping condition
Revised: 3.4 Validity of bounds and stopping condition

Original: For the upper bound on the quadratic form, we need to make a (technical) assumption.
Revised: For the upper bound on the quadratic form, we need to make a (technical) assumption.

Original: It expressesthe intuition that the  mean square error should decrease with more data, which, empirically, appearsto be true.
Revised: It expressesthe intuition that the (expected) mean square error should not increase with more data—a modelshould not become worse as its training set increases.

Original: Assumption 1.
Revised: Assumption 1.

Original: Assume that
Revised: Assume that

Original: [Equation]
Revised: [Equation]

Original: for all s ∈ { 1 , . . .
Revised: for all s ∈ { 1 , . . .

Original: , N } and for all s < j ≤ N , where f ( x , x (cid:48) ) is either
Revised: , N } and for all s < j ≤ N , where f ( x , x ′ ) is either

Original: [Equation]
Revised: [Equation]

Original: x ) .

Original: Theorem 2.
Revised: Theorem 2.

Original: Assume that ( x 1 , y 1 ) , . . .
Revised: Assume that ( x 1 , y 1 ) , . . .

Original: , ( x N , y N ) are independent and identically distributed, assumethat Assumption 1 holds, and assume that α in the deﬁnition of L Q depends only on x 1 , y 1 , . . .
Revised: , ( x N , y N ) are independent and identically distributed, assumethat Assumption 1 holds, and assume that α in the definition of L Q depends only on x 1 , y 1 , . . .

Original: x s , y s .
Revised: x s , y s .

Original: For any s ∈ { 1 , . . . , N } , the bounds deﬁned in Equations (6) , (7) , (9) and (10) hold in expectation:
Revised: For any s ∈ { 1 , . . . , N } , the bounds defined in Equations (6) , (7) , (9) and (10) hold in expectation:

Original: [Equation]
Revised: [Equation]

Original: Proof and a proof sketch can be found in the appendix.
Revised: Proof and a proof sketch can be found in the appendix.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let r > 0 be a desired relative error and set U :=
Revised: Let r > 0 be a desired relative error and set U :=

Original: U D + U Q and L :=
Revised: U D + U Q and L :=

Original: L D + L Q .
Revised: L D + L Q .

Original: If 207 the stopping conditions hold, that is, sign(
Revised: If 211 the stopping conditions hold, that is, sign(

Original: U )
Revised: U )

Original: =
Revised: =

Original: sign( L ) and Equation (4) is true, then log p ( y ) canbe estimated from ( U + L ) / 2 such that, under the condition L D ≤ log(det [ K ]) ≤ U D and L Q ≤y (cid:62) K − 1 y ≤ U Q , the relative error is smaller than r , formally:
Revised: sign( L ) and Equation (4) is true, then log p ( y ) canbe estimated from ( U + L ) / 2 such that, under the condition L D ≤ log(det [ K ]) ≤ U D and L Q ≤y ⊤ K − 1 y ≤ U Q , the relative error is smaller than r , formally:

Original: [Equation]
Revised: [Equation]

Original: The proof follows from Lemma 16 in the appendix.
Revised: The proof follows from Lemma 16 in the appendix.

Original: Theorem 2 is a ﬁrst step to obtain a probabilistic statement for Equation (11), that is, a statement ofthe form P (cid:16)(cid:12)(cid:12)(cid:12) log p ( y ) − 12 ( U + (cid:15) U ,δ + L− (cid:15) L ,δ ) log p ( y ) (cid:12)(cid:12)(cid:12) >
Revised: Theorem 2 is a first step to obtain a probabilistic statement for Equation (11), that is, a statement ofthe form P (cid:16)(cid:12)(cid:12)(cid:12) [Equation] U + [Equation]

Original:  r (cid:17) ≤ δ .
Revised: + [Equation] (cid:12)(cid:12)(cid:12) > r (cid:17) ≤ δ .

Original: Theoretically, we can obtain such a statementusing standard concentration inequalities and a union bound over s .
Revised: Theoretically, we can obtain such a statementusing standard concentration inequalities and a union bound over s .

Original: In practice, the error guardingconstants (cid:15) would render the result trivial.
Revised: In practice, the error guardingconstants ϵ would render the result trivial.

Original: A union bound can be avoided using Hoeffding’s inequalityfor martingales (Fan et al., 2012).
Revised: A union bound can be avoided using Hoeffding’s inequalityfor martingales (Fan et al., 2012).

Original: However, this requires to replace s :=
Revised: However, this requires to replace s :=

Original: M − m by a stopping timeindependent of M , which we regard as future work.
Revised: M − m by a stopping timeindependent of M , which we regard as future work.

Original: 3.5 Practical implementation
Revised: 3.5 Practical implementation

Original: The proposed bounds turn out to be surprisingly cheap to compute.
Revised: The proposed bounds turn out to be surprisingly cheap to compute.

Original: If we set the block-size of the
Revised: If we set the block-size of the

Original: Cholesky decomposition to be m , the matrix Σ ( s ) ∗ is exactly the downdated matrix in step 2 of the Cholesky decomposition algorithm outlined in Section 2.2.
Revised: Cholesky decomposition to be m , the matrix Σ ( s ) ∗ is exactly the downdated matrix in step 2 ofthe algorithm outlined in Section 2.2.

Original: Similarly, the expressions for the boundson the quadratic form appear while solving the linear equation system L − 1 y .
Revised: Similarly, the expressions for the bounds on the quadraticform appear while solving the linear equation system L − 1 y .

Original: A slight modiﬁcationto the Cholesky algorithm is enough to compute these bounds on the ﬂy during the decompositionwith little overhead.
Revised: A slight modification to the Choleskyalgorithm is enough to compute these bounds on the fly during the decomposition with little overhead.

Original: The stopping conditions can be checked before or after Step 3 of the Cholesky decomposition(Section 2.2).
Revised: The stopping conditions can be checked before or after Step 3 of the Cholesky decomposition(Section 2.2).

Original: Here, we explore the former option since Step 3 is the bottleneck due to being lessparallelizable than the other steps.
Revised: Here, we explore the former option since Step 3 is the bottleneck due to being lessparallelizable than the other steps.

Original: Note that the deﬁnition of the bounds does not involve variables x , y which have not been processed.
Revised: Note that the definition of the bounds does not involve variables x , y which have not been processed.

Original: This allows an on-the-ﬂy construction of the kernel matrix, avoiding potentially expensive kernelfunction evaluations.
Revised: This allows an on-the-fly construction of the kernel matrix, avoiding potentially expensive kernelfunction evaluations.

Original: Furthermore, it is not necessary to allocate O ( N 2 ) memory in advance; a usercan specify a maximal amount of processed datapoints, hoping that stopping occurs before hittingthat limit.
Revised: Furthermore, it is not necessary to allocate O ( N 2 ) memory in advance; a usercan specify a maximal amount of processed datapoints, hoping that stopping occurs before hittingthat limit.

Original: We provide the pseudo-code for this modiﬁed algorithm, our key algorithmic contribution,in the appendix.
Revised: We provide the pseudo-code for this modified algorithm, our key algorithmic contribution,in the appendix.

Original: Additionally, we provide a P YTHON implementation of our modiﬁed Choleskydecomposition and scripts to replicate the experiments of this paper.
Revised: Additionally, we provide a P YTHON implementation of our modified Choleskydecomposition and scripts to replicate the experiments of this paper.

Original:  Experiments
Revised:  Experiments

Original: In this section we examine  bounds and stopping strategy for ACGP.
Revised: We now examine the bounds and stopping strategy for ACGP.

Original: When running experimentswithout GPU support, all linear algebra operations are substituted for direct calls to the O PEN BLASlibrary (Wang et al., 2013), for efﬁcient realization of in-place operations.
Revised: When running experiments without GPU support, all linear algebra operations are substituted for direct calls to the O PEN BLAS library(Wang et al., 2013), for efficient realization of in-place operations.

Original: To still beneﬁt fromautomatic differentiation, we used P Y T ORCH (Paszke et al., 2019) with a custom backward functionfor log p ( y ) which wraps O PEN BLAS.
Revised: To still benefit from automaticdifferentiation, we used P Y T ORCH (Paszke et al., 2019) with a custom backward function for log p ( y )which wraps O PEN BLAS.

Original: The details of our experimental setup and the datasets usedcan be found in Appendix A.
Revised: The details of our experimental setup can be found in Appendix A.

Original: 4.1 Bound quality
Revised: 4.1 Bound quality

Original: In this section we examine the bounds presented in Section 3 and compare them to those proposed by
Revised: In this section we examine the bounds presented in Section 3 and compare them to those proposed by

Original: Artemev et al.
Revised: Artemev et al.

Original: (2021, Lemma 2 and Lemma 3) (CGLB).
Revised: (2021, Lemma 2 and Lemma 3) (CGLB).

Original: Speciﬁcally, for the determinant we compareto their O ( N ) upper bound (Artemev et al., 2021, Eq. 11) and their log(det [ Q ]) as lower bound.
Revised: Specifically, for the determinant we compareto their O ( N ) upper bound (Artemev et al., 2021, Eq. 11) and their log(det [ Q ]) as lower bound.

Original: We set the number of inducing inputs M for CGLB to 512, 1024 and 2048.
Revised: We set the number of inducing inputs M for CGLB to 512, 1024, 2048, and 4096.

Original: For ACGP, we deﬁnem := 40 × 256 = 10 240 which is the number of cores times the default O PEN BLAS blocksize for our machines.
Revised: For ACGP, wedefine m := 40 · 256 = 10 240 which is the number of cores times the default O PEN BLAS block [Figure] size for our machines.

Original: We compare both methods using squared exponential kernel (SE) and the
Revised: We compare both methods using squared exponential kernel (SE) and the

Original: Ornstein-Uhlenbeck kernel (OU),
Revised: Ornstein-Uhlenbeck kernel (OU),

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure] where we ﬁx σ 2 := 10 − 3 and θ := 1 , and we vary (cid:96) as log (cid:96) ∈ {− 1 , 0 , 1 , 2, 3 } .
Revised: where we fix σ 2 := 10 − 3 and θ := 1 , and we vary ℓ as log ℓ ∈ {− 1 , 0 , 1 , 2 } .

Original: As benchmarkingdatasets we use the two datasets consisting of more than 20 000 instances used by Artemev et al.(2021): kin40k and protein .
Revised: As benchmarkingdatasets we use the two datasets consisting of more than 20 000 instances used by Artemev et al.(2021): kin40k and protein .

Original: We further consider two additional datasets from the UCI repository(Dua & Graff, 2019): metro and pm25 (Liang et al., 2015).
Revised: We further consider two additional datasets from the UCI repository(Dua & Graff, 2019): metro and pm25 (Liang et al., 2015).

Original: We chose these datasets in addition asthey are of similar size, they are marked as regression tasks and no data points are missing.
Revised: We chose these datasets in addition asthey are of similar size, they are marked as regression tasks and no data points are missing.

Original: The typical picture we observe is that CGLB provides better estimates for the quadratic term, whereas ACGP is faster to identify the log-determinant.
Revised: Empirically, CGLB seems to better estimate the quadratic term, whereas ACGP is faster to identifythe log-determinant.

Original: Figures 2a and 2b show typical examples.
Revised: Figure 2 shows a typical example.

Original: Note that Figure 2a is on a log-scale: for the quadratic form, the upper bounds tend to be less tight than thelower lower bounds.
Revised: Note that, for the quadratic form, the upperbounds tend to be less tight than the lower bounds.

Original: Generally, there is no clear winner; sometimes ACGP estimates both quantitiesfaster and sometimes CGLB.
Revised: Generally, there is no clear winner; sometimes ACGP estimates both quantities faster and sometimes CGLB.

Original: For other results, see the appendix.
Revised: For other results, see the appendix.

Original: The reason why CGLB has more difﬁculties to approximate the log-determinant is that the boundinvolves trace[ K − Q ] where Q is a low rank approximation to K .
Revised: The reason why CGLB has more difficulties to approximate the log-determinant is that the boundinvolves trace[ K − Q ] where Q is a low rank approximation to K .

Original: If K ff is of high rank, the gapin the trace can be large.
Revised: If K ff is of high rank, the gapin the trace can be large.

Original: For CGLB the time to compute the bounds is dominated by the pivoted
Revised: For CGLB the time to compute the bounds is dominated by the pivoted

Original: Cholesky decomposition to select the inducing inputs.
Revised: Cholesky decomposition to select the inducing inputs.

Original: This overhead becomes irrelevant for thefollowing hyper-parameter tuning experiments, since the selection is computed only once in thebeginning.
Revised: This overhead becomes irrelevant for thefollowing hyper-parameter tuning experiments, since the selection is computed only once in thebeginning.

Original: One conclusion from these experiments is to keep in mind that when high precision isrequired, simply computing the exact solution can be a hard-to-beat baseline.
Revised: One conclusion from these experiments is to keep in mind that when high precision isrequired, simply computing the exact solution can be a hard-to-beat baseline.

Original: 4.2 Application in hyper-parameter tuning
Revised: 4.2 Application in hyper-parameter tuning

Original: We repeat the hyper-parameter tuning experiments performed by Artemev et al.
Revised: We repeat the hyper-parameter tuning experiments performed by Artemev et al.

Original: (2021) using thesame set-up, see Appendix A for details.
Revised: (2021) using thesame set-up, see Appendix A for details.

Original: We use the same kernel function, a Matérn 32 , and the sameoptimizer:
Revised: We use the same kernel function, a Matérn 32 , and the sameoptimizer:

Original: L-BFGS-B (Liu & Nocedal, 1989) with S CI P Y (Virtanen et al., 2020) default parameters.
Revised: L-BFGS-B (Liu & Nocedal, 1989) with S CI P Y (Virtanen et al., 2020) default parameters.

Original: Artemev et al.
Revised: Artemev et al.

Original: (2021) report their best results using M = 2048 inducing inputs.
Revised: (2021) report their best results using M = 2048 inducing inputs.

Original: For reference,we also compare against Sparse Variational Gaussian process regression (SGPR) by Titsias (2009)initialized with the same 512, 1024 and 2048 inducing inputs as CGLB.
Revised: For reference,we also compare against Sparse Variational Gaussian process regression (SGPR) by Titsias (2009)initialized with the same 512, 1024 and 2048 inducing inputs as CGLB.

Original: We use root mean squareerror (RMSE), negative log predictive density (NLPD) and exact, marginal log-likelihood on thetraining set, log p ( y ) , as performance metrics.
Revised: We use root mean squareerror (RMSE), negative log predictive density (NLPD) and exact, marginal log-likelihood on thetraining set, log p ( y ) , as performance metrics.

Original: The results for all experiments discussed in this sectioncan be found in Appendix B.1.
Revised: The results for all experiments discussed in this sectioncan be found in Appendix B.1.

Original: Here, we will focus on the behavior of each method during training.
Revised: Here, we will focus on the behavior of each method during training.

Original: A possible application of ACGP is that an optimizer can decide how precise function evaluationsneed to be.
Revised: A possible application of ACGP is that an optimizer can decide how precise function evaluationsneed to be.

Original: To explore this possibility, we successively decrease the “relative change in functionvalue” ( ftol ) convergence criterion of L-BFGS-B as (2 / 3) restart +1 and set this as value for r . Withthis choice, ACGP does not have any more free parameters than a standard optimizer. The block sizeis a problem independent parameter and it is set to the same value as in Section 4.1.
Revised: To explore this possibility, we successively decrease the “relative change in function

Original: [Figure]
Revised: [Figure]

Original: (a) protein dataset.
Revised: (a) protein dataset.

Original: The iteratively increasing preci- sion may allow ACGP to provide better solutions faster than exact optimization at the price of later conver- gence.
Revised: The iteratively increasing preci- sion may allow ACGP to reach better solutions faster than exact inference at the price of later convergence

Original: [Figure]
Revised: [Figure]

Revised: value” ( ftol ) convergence criterion of L-BFGS-B as (2 / 3) restart +1 and set this as value for r .

Revised: Withthis choice, ACGP does not have any more free parameters than a standard optimizer.

Revised: The block sizeis a problem independent parameter

Revised: and it is set to the same value as in Section 4.1.

Original: We explore two different computing environments.
Revised: We explore two different computing environments.

Original: For datasets smaller than 20 000 data points, weran our experiments on a single GPU.
Revised: For datasets smaller than 20 000 data points, weran our experiments on a single GPU.

Original: The results can be summarized in one paragraph: all methodsconverge the latest after two minutes.
Revised: The results can be summarized in one paragraph: all methodsconverge the latest after two minutes.

Original: The time difference between methods is less than twentyseconds.
Revised: The time difference between methods is less than twentyseconds.

Original: Exact Gaussian process regression is fastest, more often than not.
Revised: Exact Gaussian process regression is fastest, more often than not.

Original: The results can be foundin Appendix B.1.
Revised: The results can be foundin Appendix B.1.

Original: We conclude that in an environment with signiﬁcantly more processing resourcesthan memory, approximation may just cause overhead.
Revised: We conclude that in an environment with significantly more processing resourcesthan memory, approximation may just cause overhead.

Original: For datasets larger than 20 000 datapoints, our setup differs from Artemev et al.
Revised: For datasets larger than 20 000 datapoints, our setup differs from Artemev et al.

Original: (2021) in that we useonly CPUs on machines where the kernel matrix still ﬁts fully into memory.
Revised: (2021) in that we useonly CPUs on machines where the kernel matrix still fits fully into memory.

Original: On all datasets, ACGPis essentially exhibiting the same optimization behavior as the exact Gaussian process regressor,just stretched out.
Revised: On all datasets, ACGPis essentially exhibiting the same optimization behavior as the exact Gaussian process regressor,just stretched out.

Original: ACGP can provide results faster than exact optimization but may be slower inconvergence as Figure 3a shows for the protein dataset.
Revised: ACGP can provide results faster than exact optimization but may be slower inconvergence as Figure 3a shows for the protein dataset.

Original: This observation is as expected.
Revised: This observation is as expected.

Original: However,approximation can also hinder fast convergence as Figure 3b reveals on for the metro dataset.
Revised: However,approximation can also hinder fast convergence as Figure 3b reveals on for the metro dataset.

Original: CGLBbeneﬁts from caching the chosen inducing inputs and reusing the solution from the last solved linearequation system.
Revised: CGLBbenefits from caching the chosen inducing inputs and reusing the solution from the last solved linearequation system.

Original: The algorithm is faster, though it often plateaus at higher objective function values.
Revised: The algorithm is faster, though it often plateaus at higher objective function values.

Original: The results for kin40k are similar to protein and the results for pm25 are similar to metro .
Revised: The results for kin40k are similar to protein and the results for pm25 are similar to metro .

Original: Theseresults and the evolution of the root mean square error over time can be found in the appendix.
Revised: Theseresults and the evolution of the root mean square error over time can be found in the appendix.

Original: Again,when the available memory permits, the exact computation is a hard-to-beat baseline.
Revised: Again,when the available memory permits, the exact computation is a hard-to-beat baseline.

Original: However,the Cholesky as a standard numerical routine has been engineered over decades, whereas for theimplementations of CGLB and ACGP there is opportunity for improvement.
Revised: However,the Cholesky as a standard numerical routine has been engineered over decades, whereas for theimplementations of CGLB and ACGP there is opportunity for improvement.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper we have revisited the use of Cholesky decompositions in Gaussian process regression.
Revised: In this paper we have revisited the use of Cholesky decompositions in Gaussian process regression.

Original: We have shown that the Cholesky decomposition almost computes expected lower and upper boundson the marginal log-likelihood associated with GP regression.
Revised: We have shown that the Cholesky decomposition almost computes expected lower and upper boundson the marginal log-likelihood associated with GP regression.

Original: With only small modiﬁcations tothis classic matrix decomposition, we can use these bounds to stop the decomposition before allobservations have been processed.
Revised: With only small modifications tothis classic matrix decomposition, we can use these bounds to stop the decomposition before allobservations have been processed.

Original: This has the practical beneﬁt that the kernel matrix K does nothave to computed prior to performing the decomposition, but can rather be computed on-the-ﬂy.
Revised: This has the practical benefit that the kernel matrix K does nothave to computed prior to performing the decomposition, but can rather be computed on-the-fly.

Original: Empirical results indicate that the approach carries signiﬁcant promise, but no clear winner canbe crowned from our experiments.
Revised: Empirical results indicate that the approach carries significant promise, but no clear winner canbe crowned from our experiments.

Original: In general, we ﬁnd that exact GP inference leads to betterbehaved optimization than approximations such as CGLB and inducing point methods, and that awell-optimized Cholesky implementation is surprisingly competitive in terms of performance.
Revised: In general, we find that exact GP inference leads to betterbehaved optimization than approximations such as CGLB and inducing point methods, and that awell-optimized Cholesky implementation is surprisingly competitive in terms of performance.

Original: Anadvantage of our approach is that it is essentially parameter-free.
Revised: Anadvantage of our approach is that it is essentially parameter-free.

Original: The user has to specify a requestednumerical accuracy and the computational demands will be scaled accordingly.
Revised: The user has to specify a requestednumerical accuracy and the computational demands will be scaled accordingly.

Original: Finally, we note that
Revised: Finally, we note that

Original: ACGP is complementary to much existing work, and should be seen as an addition to the GP toolbox,rather than a substitute for all existing tools.
Revised: ACGP is complementary to much existing work, and should be seen as an addition to the GP toolbox,rather than a substitute for all existing tools.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The 50MBﬁle size limit of OpenReview, may prevent us from providing everything during thereviewing phase.
Revised: [Yes] The 50MBfile size limit of OpenReview, may prevent us from providing everything during thereviewing phase.

Original: In any case, all data, code and results will be available from a github repository after acceptance.
Revised: In any case, all data, code and results will be available from a githubrepository after acceptance.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e
Revised: [Yes](d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a)
Revised: (a)

Original: If your work uses existing assets, did you cite the creators?
Revised: If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes](b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: The URL to our github repository iscurrently
Revised: The URL to our github repository is

Revised: currently

Original: anonymized but will be made available
Revised: anonymized but will be made available

Original: [Table]

Original: after acceptance.
Revised: after acceptance.

Original: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?
Revised: using/curating?

Original: [No] This questions seems not applicable.
Revised: [No] This questions seems not applicable.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [No] We do not process sensitive information.[Table]
Revised: [No] We do not process sensitive information.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if[Table]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]