Original: G RADIENT B OOSTING P ERFORMS G FAUSSIAN P ROCESS I NFERENCE
Revised: G RADIENT B OOSTING P ERFORMS G AUSSIAN P ROCESS I NFERENCE

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper shows that gradient boosting based on symmetric decision trees can be equivalently reformulated as a kernel method that converges to the solution of a certain Kernel Ridge Regression problem.
Revised: This paper shows that gradient boosting based on symmetric decision trees can be equivalently reformulated as a kernel method that converges to the solution of a certain Kernel Ridge Regression problem.

Original: Thus, we obtain the convergence to a
Revised: Thus, we obtain the convergence to a

Original: Gaussian Process’ posterior mean, which, in turn, allows us to easily transform gradient boosting into a sampler from the posterior to provide better knowledge uncertainty estimates through Monte-Carlo estimation of the posterior variance.
Revised: Gaussian Process’ posterior mean, which, in turn, allows us to easily transform gradient boosting into a sampler from the posterior to provide better knowledge uncertainty estimates through Monte-Carlo estimation of the posterior variance.

Original: We show that the proposed sampler allows for better knowledge uncertainty estimates leading to improved out-of-domain detection
Revised: We show that the proposed sampler allows for better knowledge uncertainty estimates leading to improved out-of-domain detection.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Gradient boosting (Friedman, 2001) is a classic machine learning algorithm successfully used for web search, recommendation systems, weather forecasting, and other problems (Roe et al., 2005;
Revised: Gradient boosting (Friedman, 2001) is a classic machine learning algorithm successfully used for web search, recommendation systems, weather forecasting, and other problems (Roe et al., 2005;

Original: Caruana & Niculescu-Mizil, 2006; Richardson et al., 2007; Wu et al., 2010; Burges, 2010; Zhang &
Revised: Caruana & Niculescu-Mizil, 2006; Richardson et al., 2007; Wu et al., 2010; Burges, 2010; Zhang &

Original: Haghani, 2015).
Revised: Haghani, 2015).

Original: In a nutshell, gradient boosting methods iteratively combine simple models (usually decision trees), minimizing a given loss function.
Revised: In a nutshell, gradient boosting methods iteratively combine simple models (usually decision trees), minimizing a given loss function.

Original: Despite the recent success of neural approaches in various areas, gradient-boosted decision trees (GBDT) are still state-of-the-art algorithms for tabular datasets containing heterogeneous features (Gorishniy et al., 2021; Katzir et al., 2021).
Revised: Despite the recent success of neural approaches in various areas, gradient-boosted decision trees (GBDT) are still state-of-the-art algorithms for tabular datasets containing heterogeneous features (Gorishniy et al., 2021; Katzir et al., 2021).

Original: This paper aims at a better theoretical understanding of GBDT methods for regression problems assuming the widely used RMSE loss function.
Revised: This paper aims at a better theoretical understanding of GBDT methods for regression problems assuming the widely used RMSE loss function.

Original: First, we show that the gradient boosting with regularization can be reformulated as an optimization problem in some Reproducing Kernel Hilbert
Revised: First, we show that the gradient boosting with regularization can be reformulated as an optimization problem in some Reproducing Kernel Hilbert

Original: Space (RKHS) with implicitly deﬁned kernel structure.
Revised: Space (RKHS) with implicitly deﬁned kernel structure.

Original: After obtaining that connection between
Revised: After obtaining that connection between

Original: GBDT and kernel methods, we introduce a technique for sampling from prior Gaussian process distribution with the same kernel that deﬁnes RKHS so that the ﬁnal output would converge to a sample from the Gaussian process posterior.
Revised: GBDT and kernel methods, we introduce a technique for sampling from prior Gaussian process distribution with the same kernel that deﬁnes RKHS so that the ﬁnal output would converge to a sample from the Gaussian process posterior.

Original: Without this technique, we can view the output of
Revised: Without this technique, we can view the output of

Original: GBDT as the mean function of the Gaussian process.
Revised: GBDT as the mean function of the Gaussian process.

Original: Importantly, our theoretical analysis assumes the regularized gradient boosting procedure (Algorithm 2) without any simpliﬁcations — we only need decision trees to be symmetric (oblivious) and properly randomized (Algorithm 1).
Revised: Importantly, our theoretical analysis assumes the regularized gradient boosting procedure (Algorithm 2) without any simpliﬁcations — we only need decision trees to be symmetric (oblivious) and properly randomized (Algorithm 1).

Original: These assumptions are non-restrictive and are satisﬁed in some popular gradient boosting implementations, e.g., CatBoost (Prokhorenkova et al., 2018).
Revised: These assumptions are non-restrictive and are satisﬁed in some popular gradient boosting implementations, e.g., CatBoost (Prokhorenkova et al., 2018).

Original: Our experiments conﬁrm that the proposed sampler from the Gaussian process posterior outperforms the previous approaches (Malinin et al., 2021) and gives better knowledge uncertainty estimates and improved out-of-domain detection.
Revised: Our experiments conﬁrm that the proposed sampler from the Gaussian process posterior outperforms the previous approaches (Malinin et al., 2021) and gives better knowledge uncertainty estimates and improved out-of-domain detection.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Assume that we are given a distribution (cid:37) over X × Y , where X ⊂ R d is called a feature space and
Revised: Assume that we are given a distribution (cid:37) over X × Y , where X ⊂ R d is called a feature space and

Original: Y ⊂ R — a target space.
Revised: Y ⊂ R — a target space.

Original: Further assume that we are given a dataset z = { ( x i , y i ) } N i =1 ⊂ X × Y of size N ≥ 1 sampled i.i.d.
Revised: Further assume that we are given a dataset z = { ( x i , y i ) } Ni =1 ⊂ X × Y of size N ≥ 1 sampled i.i.d.

Original: from (cid:37) .
Revised: from (cid:37) .

Original: Let us denote by ρ (d x )
Revised: Let us denote by ρ (d x )

Original: = (cid:82) Y d (cid:37) (d x, d y ) .
Revised: = (cid:82) Y d (cid:37) (d x, d y ) .

Original: W.l.o.g.
Revised: W.l.o.g.

Original: , we also assume that X = supp ρ = (cid:8) x ∈ R d : ∀ (cid:15) > 0 , ρ ( { x (cid:48)
Revised: , we also assume that X = supp ρ = (cid:8) x ∈ R d : ∀ (cid:15) > 0 , ρ ( { x (cid:48)

Original: ∈ R
Revised: ∈ R

Original: d : (cid:107) x − x (cid:48) (cid:107) R d
Revised: d : (cid:107) x − x (cid:48) (cid:107) R d

Original: <
Revised: <

Original: (cid:15) } ) > 0 (cid:9) which is a closed subset of R d .
Revised: (cid:15) } ) > 0 (cid:9) which is a closed subset of R d .

Original: Moreover, for technical reasons, we assume that 12 N (cid:80) Ni =

Revised: Moreover, for technical reasons, we assume that 12 N (cid:80) Ni =

Original: y 2 i ≤ R 2 for some constant R > 0 almost surely, which can always be enforced by clipping.
Revised: y 2 i ≤ R 2 for some constant R > 0 almost surely, which can always be enforced by clipping.

Original: Throughout the paper, we also denote by x N and y N the matrix of all feature vectors and the vector of targets.
Revised: Throughout the paper, we also denote by x N and y N the matrix of all feature vectors and the vector of targets.

Original: 2. G RADIENT BOOSTED DECISION TREES
Revised: 2. G RADIENT BOOSTED DECISION TREES

Original: Given a loss function L : R 2 → R , a classic gradient boosting algorithm (Friedman, 2001) iteratively combines weak learners (usually decision trees) to reduce the average loss over the train set z :
Revised: Given a loss function L : R 2 → R , a classic gradient boosting algorithm (Friedman, 2001) iteratively combines weak learners (usually decision trees) to reduce the average loss over the train set z :

Original: L ( f ) =
Revised: L ( f ) =

Original: E z
Revised: E z

Original: [ L ( f ( x ) , y )] .
Revised: [ L ( f ( x ) , y )] .

Original: At each iteration τ , the model is updated as: f τ ( x ) =
Revised: At each iteration τ , the model is updated as: f τ ( x ) =

Original: f
Revised: f

Original: τ − 1 ( x ) + (cid:15)w τ ( x ) , where w τ ( · ) ∈ W is a weak learner chosen from some family of functions W , and (cid:15) is a learning rate.
Revised: τ − 1 ( x ) + (cid:15)w τ ( x ) , where w τ ( · ) ∈ W is a weak learner chosen from some family of functions W , and (cid:15) is a learning rate.

Original: The weak learner w τ is usually chosen to approximate the negative gradient of the loss function 
Revised: The weak learner w τ is usually chosen to approximate the negative gradient of the loss function − g τ ( x, y ) : = − ∂L ( s,y ) ∂s (cid:12) (cid:12) s = f τ − 1 ( x ) : w τ

Original: [Equation]
Revised: [Equation]

Original: The family W usually consists of decision trees.
Revised: The family W usually consists of decision trees.

Original: In this case, the algorithm is called GBDT (Gradient
Revised: In this case, the algorithm is called GBDT (Gradient

Original: Boosted Decision Trees).
Revised: Boosted Decision Trees).

Original: A decision tree is a model that recursively partitions the feature space into disjoint regions called leaves.
Revised: A decision tree is a model that recursively partitions the feature space into disjoint regions called leaves.

Original: Each leaf R j of the tree is assigned to a value, which is the estimated response y in the corresponding region.
Revised: Each leaf R j of the tree is assigned to a value, which is the estimated response y in the corresponding region.

Original: Thus, we can write w ( x ) =
Revised: Thus, we can write w ( x ) =

Original: (cid:80) dj
Revised: (cid:80) dj

Original: =
Revised: =

Original: θ j 1 { x ∈ R j } , so the decision tree is a linear function of the leaf values θ j .
Revised: θ j 1 { x ∈ R j } , so the decision tree is a linear function of the leaf values θ j .

Original: A recent paper Ustimenko & Prokhorenkova (2021) proposes a modiﬁcation of classic stochastic gradient boosting (SGB) called
Revised: A recent paper Ustimenko & Prokhorenkova (2021) proposes a modiﬁcation of classic stochastic gradient boosting (SGB) called

Original: Stochastic Gradient Langevin Boosting (SGLB).
Revised: Stochastic Gradient Langevin Boosting (SGLB).

Original: SGLB combines gradient boosting with stochastic gradient Langevin dynamics to achieve global convergence even for non-convex loss functions.
Revised: SGLB combines gradient boosting with stochastic gradient Langevin dynamics to achieve global convergence even for non-convex loss functions.

Original: The algorithm has two modiﬁcations compared to the classic boosting.

Original: First, Gaussian noise (parametrized by a temperature parameter) is injected into the gradients, and the temperature regulates the level of exploration.

Original: Second, shrinkage is added to the boosting process: at each iteration, the currently built model is multiplied by a constant smaller than one.

Original: As a result, the obtained algorithm provably converges to some stationary distribution (invariant measure) concentrated near the global optimum of the loss function.
Revised: As a result, the obtained algorithm provably converges to some stationary distribution (invariant measure) concentrated near the global optimum of the loss function.

Original: We mention this method because it samples from similar distribution as our method but with a different kernel.
Revised: We mention this method because it samples from similar distribution as our method but with a different kernel.

Original: 2. E STIMATING UNCERTAINTY
Revised: 2. E STIMATING UNCERTAINTY

Original: In addition to the predictive quality, it is often important to detect when the system is uncertain and can be mistaken.
Revised: In addition to the predictive quality, it is often important to detect when the system is uncertain and can be mistaken.

Original: For this, different measures of uncertainty can be used.
Revised: For this, different measures of uncertainty can be used.

Original: There are two main sources of uncertainty: data uncertainty (a.k.a.
Revised: There are two main sources of uncertainty: data uncertainty (a.k.a.

Original: aleatoric uncertainty) and knowledge uncertainty (a.k.a.
Revised: aleatoric uncertainty) and knowledge uncertainty (a.k.a.

Original: epistemic uncertainty).
Revised: epistemic uncertainty).

Original: Data uncertainty arises due to the inherent complexity of the data, such as additive noise or overlapping classes.
Revised: Data uncertainty arises due to the inherent complexity of the data, such as additive noise or overlapping classes.

Original: For instance, if the target is distributed as y | x ∼
Revised: For instance, if the target is distributed as y | x ∼

Original: N ( f ( x ) , σ 2 ( x )) , then σ ( x ) reﬂects the level of data uncertainty.
Revised: N ( f ( x ) , σ 2 ( x )) , then σ ( x ) reﬂects the level of data uncertainty.

Original: This uncertainty can be assessed if the model is probabilistic.
Revised: This uncertainty can be assessed if the model is probabilistic.

Original: Knowledge uncertainty arises when the model gets input from a region either sparsely covered by or far from the training data.
Revised: Knowledge uncertainty arises when the model gets input from a region either sparsely covered by or far from the training data.

Original: Since the model does not have enough data in this region, it will likely make a mistake.
Revised: Since the model does not have enough data in this region, it will likely make a mistake.

Original: A standard approach to estimating knowledge uncertainty is based on ensembles (Gal, 2016; Malinin, 2019).
Revised: A standard approach to estimating knowledge uncertainty is based on ensembles (Gal, 2016; Malinin, 2019).

Original: Assume that we have trained an ensemble of several independent models.
Revised: Assume that we have trained an ensemble of several independent models.

Original: If all the models understand an input (low knowledge uncertainty), they will give similar predictions.
Revised: If all the models understand an input (low knowledge uncertainty), they will give similar predictions.

Original: However, for out-of-domain examples (high knowledge uncertainty), the models are likely to provide diverse predictions.
Revised: However, for out-of-domain examples (high knowledge uncertainty), the models are likely to provide diverse predictions.

Original: For regression tasks, one can obtain knowledge uncertainty by measuring the variance of the predictions provided by multiple models (Malinin, 2019).
Revised: For regression tasks, one can obtain knowledge uncertainty by measuring the variance of the predictions provided by multiple models (Malinin, 2019).

Original: Such ensemble-based approaches are standard for neural networks (Lakshminarayanan et al., 2017).
Revised: Such ensemble-based approaches are standard for neural networks (Lakshminarayanan et al., 2017).

Original: Recently, ensembles were also tested for GBDT models (Malinin et al., 2021).
Revised: Recently, ensembles were also tested for GBDT models (Malinin et al., 2021).

Original: The authors consider two ways of generating ensembles: ensembles of independent SGB models and ensembles of independent SGLB models.
Revised: The authors consider two ways of generating ensembles: ensembles of independent SGB models and ensembles of independent SGLB models.

Original: While empirically methods are very similar, SGLB has better theoretical properties: the convergence of parameters to the stationary distribution allows one to sample models from a particular posterior distribution.
Revised: While empirically methods are very similar, SGLB has better theoretical properties: the convergence of parameters to the stationary distribution allows one to sample models from a particular posterior distribution.

Original: One drawback of SGLB is that its convergence rate is unknown, as the proof is asymptotic.
Revised: One drawback of SGLB is that its convergence rate is unknown, as the proof is asymptotic.

Original: However, the convergence rate can be upper bounded by that of
Revised: However, the convergence rate can be upper bounded by that of

Original: Stochastic Gradient Langevin Dynamics for log-concave functions, e.g., (Zou et al., 2020), which is not dimension-free.
Revised: Stochastic Gradient Langevin Dynamics for log-concave functions, e.g., Zou et al. (2021), which is not dimension-free.

Original: In contrast, our rate is dimension-free and scales linearly with inverse precision.
Revised: In contrast, our rate is dimension-free and scales linearly with inverse precision.

Original: 2. G AUSSIAN PROCESS INFERENCE
Revised: 2. G AUSSIAN PROCESS INFERENCE

Original: In this section, we brieﬂy describe the basics of Bayesian inference with Gaussian processes that is closely related to our analysis.
Revised: In this section, we brieﬂy describe the basics of Bayesian inference with Gaussian processes that is closely related to our analysis.

Original: A random variable f with values in L 2 ( ρ ) is said to be a Gaussian pro- cess f ∼ GP (cid:0) f 0 , σ 2 K + δ 2 Id L 2 (cid:1) with covariance deﬁned via a kernel K ( x, x (cid:48) )
Revised: A random variable f with values in L 2 ( ρ ) is said to be a Gaussian process f ∼ GP (cid:0) f 0 , σ 2 K + δ 2 Id L 2 (cid:1) with covariance deﬁned via a kernel K ( x, x (cid:48) )

Original: =
Revised: =

Original: cov (cid:0) f
Revised: cov (cid:0) f

Original: ( x ) , f ( x (cid:48) ) (cid:1) and mean value f 0 ∈
Revised: ( x ) , f ( x (cid:48) ) (cid:1) and mean value f 0 ∈

Original: L 2 ( ρ ) iff ∀ g ∈
Revised: L 2 ( ρ ) iff ∀ g ∈

Original: L 2 ( ρ ) we have (cid:90)
Revised: L 2 ( ρ ) we have (cid:90)

Revised: X f

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: N (cid:16)

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Table]

Revised: K

Revised: [Table]

Revised: [Table]

Original:  A typical Gaussian Process setup is to assume that the target y | x is distributed as GP (cid:0) 

Revised: [Table] + [Table] A typical Gaussian Process setup is to assume that the target y | x is distributed as GP (cid:0)

Original: L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 (cid:1) for some kernel function 1 K ( x, x (cid:48) ) with scales σ > 0 and δ > 0 : [Equation]
Revised: L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 (cid:1) for some kernel function 1 K ( x, x (cid:48) ) with scales σ > 0 and δ > 0 : y | x = f ( x ) , f ∼ GP (

Original: [Equation]
Revised: 0 L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 ) .

Original: The posterior distribution f ( x ) | x, x N , y N is again a Gaussian Process GP ( f ∗ , σ 2 (cid:101) K
Revised: The posterior distribution f ( x ) | x, x N , y N is again a Gaussian Process GP ( f ∗ , σ 2 (cid:101) K

Original: + δ 2 Id L 2 ) with mean and covariance given by (see Rasmussen & Williams (2006)):
Revised: + δ 2 Id L 2 ) with mean and covariance given by (see Rasmussen & Williams (2006)):

Original: [Equation]
Revised: [Equation]

Original: with λ = δ 2 σ 2 .
Revised: with λ = δ 2 σ 2 .

Original: To estimate the posterior mean f λ ∗ ( x ) = (cid:82) R fp ( f | x, x N , y N ) d f , we can use the maximum a posteriori probability estimate (MAP) — the only solution for the following Kernel
Revised: To estimate the posterior mean f λ ∗ ( x ) = (cid:82) R fp ( f | x, x N , y N ) d f , we can use the maximum a posteriori probability estimate (MAP) — the only solution for the following Kernel

Original: Ridge Regression (KRR) problem:
Revised: Ridge Regression (KRR) problem:

Original: [Equation]
Revised: [Equation]

Original: where H = span (cid:8) K ( · , x ) (cid:12)(cid:12) x ∈ X (cid:9) ⊂
Revised: where H = span (cid:8) K ( · , x ) (cid:12)(cid:12) x ∈ X (cid:9) ⊂

Original: L 2 ( ρ ) is the reproducing kernel Hilbert space (RKHS) for the kernel K ( · , · ) and L ( f ) → min f ∈H means that we are seeking minimizers of this function L ( f ) .
Revised: L 2 ( ρ ) is the reproducing kernel Hilbert space (RKHS) for the kernel K ( · , · ) and L ( f ) → min f ∈H means that we are seeking minimizers of this function L ( f ) .

Original: We refer to Appendix D for the details on KRR and RKHS.
Revised: We refer to Appendix D for the details on KRR and RKHS.

Original: To solve the KRR problem, one can apply the gradient descent (GD) method in the functional space:
Revised: To solve the KRR problem, one can apply the gradient descent (GD) method in the functional space:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) > 0 is a learning rate.
Revised: where (cid:15) > 0 is a learning rate.

Original: Since this objective is strongly convex due to the regularization, the gradient descent rapidly converges to the unique optimum:
Revised: Since this objective is strongly convex due to the regularization, the gradient descent rapidly converges to the unique optimum:

Original: [Equation]
Revised: [Equation]

Original: see Appendices C and E for the details.
Revised: see Appendices C and E for the details.

Original: Gradient descent guides f τ to the posterior mean f λ ∗ of the Gaussian Process with kernel σ 2 K + δ 2 Id L 2 .
Revised: Gradient descent guides f τ to the posterior mean f λ ∗ of the Gaussian Process with kernel σ 2 K + δ 2 Id L 2 .

Original: To obtain the posterior variance estimate (cid:101) K ( x, x ) for any x , one can use sampling and introduce a source of randomness in the above iterative scheme as follows:
Revised: To obtain the posterior variance estimate (cid:101) K ( x, x ) for any x , one can use sampling and introduce a source of randomness in the above iterative scheme as follows:

Original: sample f init ∼ GP ( 0 L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 ); 2.
Revised: sample f init ∼ GP ( 0 L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 ); 2.

Original: set new labels y
Revised: set new labels y

Original: newN =
Revised: newN =

Original: y N − f init ( x N ); 3.
Revised: y N − f init ( x N ); 3.

Original: ﬁt GD f τ ( · ) on y
Revised: ﬁt GD f τ ( · ) on y

Original: newN assuming F 0 ( · ) =
Revised: newN assuming F 0 ( · ) =

Original: L 2 ( ρ ) ; 4.
Revised: L 2 ( ρ ) ; 4.

Original: output f init ( · ) + f τ ( · ) as ﬁnal model .
Revised: output f init ( · ) + f τ ( · ) as ﬁnal model .

Original: This method is known as Sample-then-Optimize (Matthews et al., 2017) and is widely adopted for
Revised: This method is known as Sample-then-Optimize (Matthews et al., 2017) and is widely adopted for

Original: Bayesian inference.
Revised: Bayesian inference.

Original: As τ → ∞ , we get f init + f ∞ distributed as a Gaussian Process posterior with the desired mean and variance.
Revised: As τ → ∞ , we get f init + f ∞ distributed as a Gaussian Process posterior with the desired mean and variance.

Original: Formally, the following result holds:
Revised: Formally, the following result holds:

Original: Lemma 2.1.
Revised: Lemma 2.1.

Original: f init + f ∞ follows the Gaussian Process posterior GP ( f λ ∗ , σ 2 (cid:101) K
Revised: f init + f ∞ follows the Gaussian Process posterior GP ( f λ ∗ , σ 2 (cid:101) K

Original: + δ 2 Id L 2 ) with:
Revised: + δ 2 Id L 2 ) with:

Original: [Equation]
Revised: [Equation]

Original:  E VOLUTION OF GBDT IN RKHS 3. P RELIMINARIES
Revised:  E VOLUTION OF GBDT IN RKHS 3. P RELIMINARIES

Original: In our analysis, we assume that we are given a ﬁnite set V of weak learners used for the gradient boosting.
Revised: In our analysis, we assume that we are given a ﬁnite set V of weak learners used for the gradient boosting.

Original: Each ν corresponds to a decision tree that deﬁnes a partition of the feature space into disjoint regions (leaves).
Revised: Each ν corresponds to a decision tree that deﬁnes a partition of the feature space into disjoint regions (leaves).

Original: For each ν ∈ V , we denote the number of leaves in the tree by L ν ≥ 1 .
Revised: For each ν ∈ V , we denote the number of leaves in the tree by L ν ≥ 1 .

Original: Also, let φ ν : X → { 0 , 1 } L ν be a mapping that maps x to the vector indicating its leaf index in the tree ν .
Revised: Also, let φ ν : X → { 0 , 1 } L ν be a mapping that maps x to the vector indicating its leaf index in the tree ν .

Original: This mapping deﬁnes a decomposition of X into the disjoint union: X = ∪ L ν j =1 (cid:8) x ∈
Revised: This mapping deﬁnes a decomposition of X into the disjoint union: X = ∪ L ν j =1 (cid:8) x ∈

Original: X (cid:12)(cid:12) φ ( j ) ν
Revised: X (cid:12)(cid:12) φ ( j ) ν

Original: ( x ) = 1 (cid:9) .
Revised: ( x ) = 1 (cid:9) .

Original: Having φ ν , we deﬁne a weak learner associated with it as x (cid:55)→ (cid:104) θ, φ ν ( x ) (cid:105) R Lν for any choice of θ ∈ R L ν which we refer to as ‘leaf values’.
Revised: Having φ ν , we deﬁne a weak learner associated with it as x (cid:55)→ (cid:104) θ, φ ν ( x ) (cid:105) R Lν for any choice of θ ∈ R L ν which we refer to as ‘leaf values’.

Original: In other words, θ corresponds to predictions assigned to each region of the space deﬁned by ν .
Revised: In other words, θ corresponds to predictions assigned to each region of the space deﬁned by ν .

Original: Let us deﬁne a linear space F ⊂ L 2 ( ρ ) of all possible ensembles of trees from V :
Revised: Let us deﬁne a linear space F ⊂ L 2 ( ρ ) of all possible ensembles of trees from V :

Original: [Equation]
Revised: [Equation]

Original: We note that the space F can be data-dependent since V may depend on z , but we omit this dependence in the notation for simplicity.
Revised: We note that the space F can be data-dependent since V may depend on z , but we omit this dependence in the notation for simplicity.

Original: Note that we do not take the closure w.r.t.
Revised: Note that we do not take the closure w.r.t.

Original: the topology of L 2 ( ρ ) since we assume that V is ﬁnite and therefore F is ﬁnite-dimensional and thus topologically closed.
Revised: the topology of L 2 ( ρ ) since we assume that V is ﬁnite and therefore F is ﬁnite-dimensional and thus topologically closed.

Original: 3. GBDT ALGORITHM UNDER CONSIDERATION
Revised: 3. GBDT ALGORITHM UNDER CONSIDERATION

Original: Our theoretical analysis holds for classic GBDT algorithms discussed in Section 2.1 equipped with regularization from Ustimenko & Prokhorenkova (2021).
Revised: Our theoretical analysis holds for classic GBDT algorithms discussed in Section 2.1 equipped with regularization from Ustimenko & Prokhorenkova (2021).

Original: The only requirement we need is that the procedure of choosing each new tree has to be properly randomized.
Revised: The only requirement we need is that the procedure of choosing each new tree has to be properly randomized.

Original: Let us discuss a tree selection algorithm that we assume in our analysis.
Revised: Let us discuss a tree selection algorithm that we assume in our analysis.

Original: Each new tree approximates the gradients of the loss function with respect to the current predictions of the model.
Revised: Each new tree approximates the gradients of the loss function with respect to the current predictions of the model.

Original: Since we consider the RMSE loss function, the gradients are proportional to the residuals r j =
Revised: Since we consider the RMSE loss function, the gradients are proportional to the residuals r j =

Original: y
Revised: y

Original: j − f ( x j ) , where f is the currently built model.
Revised: j − f ( x j ) , where f is the currently built model.

Original: The tree structure is deﬁned by the features and the corresponding thresholds used to split the space.
Revised: The tree structure is deﬁned by the features and the corresponding thresholds used to split the space.

Original: The analysis in this paper assumes the SampleTree procedure (see Algorithm 1), which is a classic approach equipped with proper randomization.
Revised: The analysis in this paper assumes the SampleTree procedure (see Algorithm 1), which is a classic approach equipped with proper randomization.

Original: SampleTree builds an oblivious decision tree (Prokhorenkova et al., 2018), i.e., all nodes at a given level share the same splitting criterion (feature and threshold).
Revised: SampleTree builds an oblivious decision tree (Prokhorenkova et al., 2018), i.e., all nodes at a given level share the same splitting criterion (feature and threshold).

Original: To limit the number of candidate splits, each feature is quantized into n + 1 bins.
Revised: To limit the number of candidate splits, each feature is quantized into n + 1 bins.

Original: In other words, for each feature, we have n thresholds that can be chosen arbitrarily.
Revised: In other words, for each feature, we have n thresholds that can be chosen arbitrarily.

Original: The maximum tree depth is limited by m .
Revised: The maximum tree depth is limited by m .

Original: Recall that we denote the set of all possible tree structures by V .
Revised: Recall that we denote the set of all possible tree structures by V .

Original: We build the tree in a top-down greedy manner.
Revised: We build the tree in a top-down greedy manner.

Original: At each step, we choose one split among all the remaining candidates based on the following score deﬁned for ν ∈ V and residuals r :
Revised: At each step, we choose one split among all the remaining candidates based on the following score deﬁned for ν ∈ V and residuals r :

Original: [Equation]
Revised: [Equation]

Original: In classic gradient boosting, one builds a tree recursively by choosing such split s that maximizes the score D (( ν i , s ) , r ) (Ibragimov & Gusev, 2019).
Revised: In classic gradient boosting, one builds a tree recursively by choosing such split s that maximizes the score D (( ν i , s ) , r ) (Ibragimov & Gusev, 2019).

Original: 5 Random noise is often added to the scores to improve generalization.
Revised: 5 Random noise is often added to the scores to improve generalization.

Original: In SampleTree , we choose a split that maximizes
Revised: In SampleTree , we choose a split that maximizes

Original: D (cid:0) ( ν i , s ) , r (cid:1) + ε, where ε ∼ Gumbel(0 , β ) .
Revised: D (cid:0) ( ν i , s ) , r (cid:1) + ε, where ε ∼ Gumbel(0 , β ) .

Original: [Equation]

Original: Algorithm input: residuals r =
Revised: Algorithm input: residuals r =

Original: ( r i ) Ni = output: oblivious tree structure ν ∈ V hyper-parameters: number of feature splits n , max.
Revised: ( r i ) Ni = output: oblivious tree structure ν ∈ V hyper-parameters: number of feature splits n , max.

Original: tree depth m , random strength β ∈ [0 , ∞ ) deﬁnitions:
Revised: tree depth m , random strength β ∈ [0 , ∞ ) deﬁnitions:

Original: S = (cid:8) ( j, k ) (cid:12)(cid:12) j ∈ { 1 , . . .
Revised: S = (cid:8) ( j, k ) (cid:12)(cid:12) j ∈ { 1 , . . .

Original: , d } , k ∈ { 1 , . .
Revised: , d } , k ∈ { 1 , . .

Original: . , n } (cid:9) —
Revised: . , n } (cid:9) —

Original: indices of all possible splits instructions: initialize i = 0 , ν 0 = ∅ , S (0) =
Revised: indices of all possible splits instructions: initialize i = 0 , ν 0 = ∅ , S (0) =

Original: S repeat sample ( u i ( s )) s ∈S ( i ) ∼ U (cid:0) [0 , 1] nd − i (cid:1) choose next split as { s i +1 }
Revised: S repeat sample ( u i ( s )) s ∈S ( i ) ∼ U (cid:0) [0 , 1] nd − i (cid:1) choose next split as { s i +1 }

Original: = arg max s ∈S ( i ) (cid:16) D (cid:0) ( ν i , s ) , r (cid:1) − β log( − log u
Revised: = arg max s ∈S ( i ) (cid:16) D (cid:0) ( ν i , s ) , r (cid:1) − β log( − log u

Original: i ( s )) (cid:17) update tree: ν i +1 =
Revised: i ( s )) (cid:17) update tree: ν i +1 =

Original: ( ν i , s i +1 ) update candidate splits: S ( i +1)
Revised: ( ν i , s i +1 ) update candidate splits: S ( i +1)

Original: =
Revised: =

Original: S ( i ) \{ s i +1 } i
Revised: S ( i ) \{ s i +1 } i

Original: =
Revised: =

Original: i +until i ≥ m or S ( i ) =
Revised: i +until i ≥ m or S ( i ) =

Original: ∅ return:
Revised: ∅ return:

Original: ν i
Revised: ν i

Original: Algorithm; (cid:15), input: dataset z = ( x N , y N ) hyper-parameters: learning rate (cid:15) > 0 , regularization λ > 0 , iterations of boosting T , parameters of SampleTree m, n, β instructions: initialize τ = 0 , f 0 ( · ) = 

Revised: Algorithm input: dataset z = ( x N , y N ) hyper-parameters: learning rate (cid:15) > 0 , regularization λ > 0 , iterations of boosting T , parameters of SampleTree m, n, β instructions: initialize τ = 0 , f 0 ( · ) =

Original: L 2 ( ρ ) repeat r τ = y N − f τ ( x N ) — compute residuals ν τ = SampleTree ( r τ ; m, n, β ) — construct a tree θ τ = (cid:16) (cid:80) Ni =1 φ ( j ) ντ ( x i )
Revised: L 2 ( ρ ) repeat r τ = y N − f τ ( x N ) — compute residuals ν τ = SampleTree ( r τ ; m, n, β ) — construct a tree θ [Equation] = (cid:16) (cid:80) [Equation]

Original: r
Revised: =

Revised: [Equation]

Revised: ( j ) ντ ( x i ) r

Original: ( i ) τ (cid:80)
Revised: ( i ) τ (cid:80)

Original: Ni =1 φ ( j ) ντ ( x i ) (cid:17)
Revised: Ni =1 φ ( j ) ντ ( x i ) (cid:17)

Original: L ντ j =1 — set values in leaves f τ +1 ( · ) =
Revised: L ντ j =1 — set values in leaves f [Equation] [Equation]

Original: (1 − λ(cid:15) N ) f τ ( · ) + (cid:15) (cid:68) φ ν τ ( · ) , θ τ (cid:69) R Lντ — update model τ = τ +until τ ≥ T return: f T ( · )
Revised: (cid:15) [Equation] [Equation] θ τ (cid:69) R Lντ — update model τ = τ +until τ ≥ T return: f T ( · )

Original: Here β is random strength: β = 0 gives the standard greedy approach, while β → ∞ gives the uniform distribution among all possible split candidates.
Revised: Here β is random strength: β = 0 gives the standard greedy approach, while β → ∞ gives the uniform distribution among all possible split candidates.

Original: To sum up, SampleTree is a classic oblivious tree construction but with added random noise.
Revised: To sum up, SampleTree is a classic oblivious tree construction but with added random noise.

Original: We do this to make the distribution of trees regular in a certain sense: roughly speaking, the distributions should stabilize with iterations by converging to some ﬁxed distribution.
Revised: We do this to make the distribution of trees regular in a certain sense: roughly speaking, the distributions should stabilize with iterations by converging to some ﬁxed distribution.

Original: Given the algorithm SampleTree , we describe the gradient boosting procedure assumed in our analysis in Algorithm 2.
Revised: Given the algorithm SampleTree , we describe the gradient boosting procedure assumed in our analysis in Algorithm 2.

Original: It is a classic GBDT algorithm
Revised: It is a classic GBDT algorithm

Original: but with the update rule f τ +1 ( · ) = (1 − λ(cid:15)/N ) f τ ( · ) + (cid:15)w τ ( x ) .
Revised: but with the update rule f τ +1 ( · ) = (1 − λ(cid:15)/N ) f τ ( · ) + (cid:15)w τ ( x ) .

Original: In other words, we shrink the model at each iteration, which serves as regularization (Ustimenko & Prokhorenkova, 2021).
Revised: In other words, we shrink the model at each iteration, which serves as regularization (Ustimenko & Prokhorenkova, 2021).

Original: Such shrinkage is available, e.g., in the
Revised: Such shrinkage is available, e.g., in the

Original: CatBoost library.
Revised: CatBoost library.

Original: 3. D ISTRIBUTION OF TREES
Revised: 3. D ISTRIBUTION OF TREES

Original: The SampleTree algorithm induces a local family of distributions p ( ·| f, β ) for each f ∈ F :p (d ν | f, β )
Revised: The SampleTree algorithm induces a local family of distributions p ( ·| f, β ) for each f ∈ F :

Original: = P (cid:16)(cid:110) SampleTree (cid:0) y N − f ( x N ); m, n, β (cid:1) ∈
Revised: [Equation]

Original: d ν (cid:111)(cid:17) .
Revised: (cid:111)(cid:17) .

Original: Remark 3.1 .
Revised: Remark 3.1 .

Original: Lemma 3.7 ensure that such distribution coincides with the one where we use f ∗ ( x N ) instead of y N .
Revised: Lemma D.5 ensure that such distribution coincides with the one where we use f ∗ ( x N ) instead of y N .

Original: This is due to the fact that D (cid:0) ν, y N − f ( x N ) (cid:1) =
Revised: This is due to the fact that D (cid:0) ν, y N − f ( x N ) (cid:1) =

Original: D (cid:0) ν, f ∗ − f ( x N ) (cid:1) ∀ ν ∈ V , f ∈ F .
Revised: D (cid:0) ν, f ∗ − f ( x N ) (cid:1) ∀ ν ∈ V , f ∈ F .

Original: The following lemma describes the distribution p (d ν | f, β ) , see Appendix F for the proof.
Revised: The following lemma describes the distribution p (d ν | f, β ) , see Appendix F for the proof.

Original: Note that for oblivious decision trees, changing the order of splits does not affect the obtained partition.

Original: Hence, we assume that each tree is deﬁned by an unordered set of splits.

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: (Probability of a tree)
Revised: (Probability of a tree 6 )

Original: [Equation]
Revised: [Equation]

Original: where the sum is over all permutations
Revised: where the sum is over all permutations

Original: ς ∈ P m , ν ς,i =
Revised: ς ∈ P m , ν ς,i =

Original: ( s ς (1) , . . .
Revised: ( s ς (1) , . . .

Original: , s ς ( i ) ) , and ν = ( s 1 , . . .
Revised: , s ς ( i ) ) , and ν = ( s 1 , . . .

Original: , s m ) .
Revised: , s m ) .

Original: Let us deﬁne the stationary distribution of trees as π ( · ) =
Revised: Let us deﬁne the stationary distribution of trees as π ( · ) =

Original: lim β →∞ p ( ·| f, β ) .
Revised: lim β →∞ p ( ·| f, β ) .

Original: It follows from Remark 3.1 that we also have π ( · )
Revised: It follows from Remark 3.1 that we also have π ( · )

Original: =
Revised: =

Original: p ( ·| f ∗ , β ) .
Revised: p ( ·| f ∗ , β ) .

Original: Corollary 3.3.
Revised: Corollary 3.3.

Original: (Stationary distribution is the uniform distribution over tree structures)
Revised: (Stationary distribution is the uniform distribution over tree structures)

Original: We have π (d ν ) =
Revised: We have π (d ν ) =

Original: (cid:12)(cid:12)
Revised: (cid:12)(cid:12)

Original: d ν (cid:12)(cid:12)(cid:14)(cid:0) nd m (cid:1) , where (cid:0) nd m (cid:1) = ( nd )!
Revised: d ν (cid:12)(cid:12)(cid:14)(cid:0) ndm (cid:1) , where (cid:0) ndm (cid:1) = ( nd )!

Original: ( nd − m )!
Revised: ( nd − m )!

Original: m ! .
Revised: m ! .

Original: 3. RKHS S TRUCTURE
Revised: 3. RKHS STRUCTURE

Original: In this section, we describe the evolution of GBDT in a certain Reproducing Kernel Hilbert Space (RKHS).
Revised: In this section, we describe the evolution of GBDT in a certain Reproducing Kernel Hilbert Space (RKHS).

Original: Even though the problem is ﬁnite dimensional, treating it as functional regression is more beneﬁcial as dimension of the ensembles space grows rapidly and therefore we want to obtain dimension-free constants which is impossible if we treat it as ﬁnite dimensional optimization problem.
Revised: Even though the problem is ﬁnite dimensional, treating it as functional regression is more beneﬁcial as dimension of the ensembles space grows rapidly and therefore we want to obtain dimension-free constants which is impossible if we treat it as ﬁnite dimensional optimization problem.

Original: Let us start with deﬁning necessary kernels.
Revised: Let us start with deﬁning necessary kernels.

Original: For convenience, we also provide a diagram illustrating the introduced kernels and relations between them in Appendix A.
Revised: For convenience, we also provide a diagram illustrating the introduced kernels and relations between them in Appendix A.

Original: Deﬁnition 3.4.
Revised: Deﬁnition 3.4.

Original: A weak learner’s kernel k ν ( · , · ) is a kernel function associated with a tree structure ν ∈ V which can be deﬁned as:
Revised: A weak learner’s kernel k ν ( · , · ) is a kernel function associated with a tree structure ν ∈ V which can be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: This weak learner’s kernel is a building block for any other possible kernel in boosting and is used to deﬁne the iterations of the boosting algorithm analytically.
Revised: This weak learner’s kernel is a building block for any other possible kernel in boosting and is used to deﬁne the iterations of the boosting algorithm analytically.

Original: Deﬁnition 3.5.
Revised: Deﬁnition 3.5.

Original: We also deﬁne a greedy kernel of the gradient boosting algorithm as follows:
Revised: We also deﬁne a greedy kernel of the gradient boosting algorithm as follows:

Original: [Equation]
Revised: [Equation]

Original: This greedy kernel is a kernel that guides the GBDT iterations, i.e., we can think of each iteration as SGD with a kernel from 3.5, and 3.4 is used as a stochastic gradient estimator of the Fr´echet derivative in RKHS deﬁned by the kernel from 3.5.
Revised: This greedy kernel is a kernel that guides the GBDT iterations, i.e., we can think of each iteration as SGD with a kernel from 3.5, and 3.4 is used as a stochastic gradient estimator of the Fr ´ echet derivative in RKHS deﬁned by the kernel from 3.5.

Original: Deﬁnition 3.6.
Revised: Deﬁnition 3.6.

Original: Finally, there is a stationary kernel K ( x, x (cid:48) )
Revised: Finally, there is a stationary kernel K ( x, x (cid:48) )

Original: that is independent from f :
Revised: that is independent from f :

Original: [Equation]
Revised: [Equation]

Original: which we call a prior kernel of the gradient boosting.
Revised: which we call a prior kernel of the gradient boosting.

Original: This kernel deﬁnes the limiting solution since the gradient projection on RKHS converges to zero, and thus 3.5 converges to 3.6.
Revised: This kernel deﬁnes the limiting solution since the gradient projection on RKHS converges to zero, and thus 3.5 converges to 3.6.

Original: Notice that F = span (cid:8) K ( · , x ) | x ∈ X (cid:9) .
Revised: Notice that F = span (cid:8) K ( · , x ) | x ∈ X (cid:9) .

Original: Having the space of functions F , we deﬁne a reproducing kernel Hilbert space (RKHS) structure H = (cid:0) F , (cid:104)· , ·(cid:105) H (cid:1) on it using a scalar product deﬁned as
Revised: Having the space of functions F , we deﬁne a reproducing kernel Hilbert space (RKHS) structure H = (cid:0) F , (cid:104)· , ·(cid:105) H (cid:1) on it using a scalar product deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Now, let us deﬁne the empirical error of a model f :
Revised: Now, let us deﬁne the empirical error of a model f :

Original: [Equation]
Revised: [Equation]

Revised: Using it we deﬁne V ( f, λ ) =

Revised: L ( f, λ ) − inf f (cid:48)

Revised: ∈F L ( f (cid:48) , λ ) .

Original: Let us also deﬁne the following unique functions:
Revised: Let us also deﬁne the following unique functions:

Original: [Equation]
Revised: [Equation]

Original: And f ∗ =
Revised: And f ∗ =

Original: lim λ → 0 f λ ∗ ∈ arg min f ∈F V ( f ) where V ( f )
Revised: lim λ → 0 f λ ∗ ∈ arg min f ∈F V ( f ) where V ( f )

Original: =
Revised: =

Original: V ( f, 0) .
Revised: V ( f, 0) .

Original: It is known that such f ∗ exists and is unique since the set of all solutions is convex, and therefore there is a unique minimizer of the norm (cid:107) · (cid:107) H .
Revised: It is known that such f ∗ exists and is unique since the set of all solutions is convex, and therefore there is a unique minimizer of the norm (cid:107) · (cid:107) H .

Original: Finally, the following lemmas will be helpful for our analysis. In particular, we use them to analyze the distribution of trees in Section 3.3 and to derive the formula of the GBDT iteration in terms of kernels in Lemma 3.9.
Revised: Finally, the following lemma gives the formula of the GBDT iteration in terms of kernels in Lemma 3.7 which will be useful in proving our results.

Original: See Appendix D for the proofs.
Revised: See Appendix D for the proofs.

Original: Lemma 3.7.
Revised: Lemma 3.7.

Original: (cid:104)

Original: y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N = 0 for any f ∈ F .

Original: Lemma 3.8.

Original: V ( f, λ ) = 12 N (cid:107) f ∗ ( x N ) − f ( x N ) (cid:107) 2 R N + λ 2 N (cid:107) f

Original: (cid:107) 2 H + const .

Original: Lemma 3.9.

Original: Iterations f
Revised: Iterations f

Original: τ of Gradient Boosting (Algorithm 2) can be written in the form:
Revised: τ of Gradient Boosting (Algorithm 2) can be written in the form:

Original: [Equation]
Revised: [Equation]

Original: 3. K ERNEL G RADIENT B OOSTING C ONVERGENCE TO KRR
Revised: 3. K ERNEL G RADIENT B OOSTING CONVERGENCE TO KRR

Original: Consider the sequence { f τ } τ ∈ N generated by the gradient boosting algorithm.
Revised: Consider the sequence { f τ } τ ∈ N generated by the gradient boosting algorithm.

Original: Its evolution is described by Lemma 3.9.
Revised: Its evolution is described by Lemma 3.7.

Original: The following theorem estimates the expected (w.r.t.
Revised: The following theorem estimates the expected (w.r.t.

Original: the randomness of tree selection) empirical error of f T relative to the best possible ensemble.
Revised: the randomness of tree selection) empirical error of f T relative to the best possible ensemble.

Original: The full statement of the theorem and its proof can be found in Appendix G.
Revised: The full statement of the theorem and its proof can be found in Appendix G.

Original: Theorem 3.10.
Revised: Theorem 3.8.

Original: Assume that β, T 1 are sufﬁciently large and (cid:15) is sufﬁciently small (see Appendix G).
Revised: Assume that β, T 1 are sufﬁciently large and (cid:15) is sufﬁciently small (see Appendix G).

Original: Then, ∀ T ≥ T 1 ,
Revised: Then, ∀ T ≥ T 1 ,

Original: [Equation]
Revised: [Equation]

Original: Corollary 3.11.
Revised: Corollary 3.9.

Original: (Convergence to the solution of the KRR problem)
Revised: (Convergence to the solution of the KRR problem)

Original: Under the assumptions of the previous theorem, we have the following dimension-free bound:
Revised: Under the assumptions of the previous theorem, we have the following dimension-free bound:

Original: [Equation]
Revised: [Equation]

Original: This bound is dimension-free thanks to functional treatment and exponentially decaying to small value with iterations and therefore justiﬁes the observed rapid convergence of gradient boosting algorithms in practice even though dimension of space H is enormous.
Revised: This bound is dimension-free thanks to functional treatment and exponentially decaying to small value with iterations and therefore justiﬁes the observed rapid convergence of gradient boosting algorithms in practice even though dimension of space H is enormous.

Original:  G AUSSIAN INFERENCE
Revised:  G AUSSIAN INFERENCE

Original: So far, the main result of the paper proved in Section 3.5 shows that Algorithm 2 solves the Kernel
Revised: So far, the main result of the paper proved in Section 3.5 shows that Algorithm 2 solves the Kernel

Original: Ridge Regression problem which can be interpreted as learning Gaussian Process posterior mean f λ ∗ under the assumption
Revised: Ridge Regression problem which can be interpreted as learning Gaussian Process posterior mean f λ ∗ under the assumption

Original: that f ∼ GP (0 , σ 2 K + δ 2 Id L 2 ) where λ = σ 2 δ 2 .
Revised: that f ∼ GP (0 , σ 2 K + δ 2 Id L 2 ) where λ = σ 2 δ 2 .

Original: I.e. Algorithm 2 do not give us posterior variance, only mean but as we mentioned earlier we can estimate posterior variance through Monte-Carlo sampling in sample-then-optimize way.
Revised: I.e. Algorithm 2 do not give us posterior variance, only mean but as we mentioned earlier we can estimate posterior variance through Monte-Carlo sampling in sample-then-optimize way.

Original: For that we need to somehow sample from prior distribution GP (0 , σ 2 K + δ 2 Id L 2 ) itself.
Revised: For that we need to somehow sample from prior distribution GP (0 , σ 2 K + δ 2 Id L 2 ) itself.

Original: 4. P RIOR S AMPLING
Revised: 4. P RIOR SAMPLING

Original: We introduce Algorithm 3 for sampling from the prior distribution.
Revised: We introduce Algorithm 3 for sampling from the prior distribution.

Original: SamplePrior generates an ensemble of random trees (with random splits and random values in leaves).
Revised: SamplePrior generates an ensemble of random trees (with random splits and random values in leaves).

Original: Note that while being random, the tree structure depends on the dataset features x N since candidate splits are based on x N .
Revised: Note that while being random, the tree structure depends on the dataset features x N since candidate splits are based on x N .

Original: We ﬁrst note that the process h T ( · ) is centered with covariance operator K :
Revised: We ﬁrst note that the process h T ( · ) is centered with covariance operator K :

Original: [Equation]
Revised: [Equation]

Original: Then, we show that h T ( · ) converges to the
Revised: Then, we show that h T ( · ) converges to the

Original: Gaussian Process in the limit.
Revised: Gaussian Process in the limit.

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: The following convergence holds almost surely in x ∈ X :
Revised: The following convergence holds almost surely in x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: Algorithm hyper number of iterations T , parameters of SampleTree m, n instructions: initialize τ = 0 , h 0 ( x ) =repeat ν τ = SampleTree ( 0 R N ; m, n, 1) — sample random tree θ τ ∼ N (cid:16) 0 R Lντ , diag (cid:16)
Revised: Algorithm hyper-parameters: number of iterations T , parameters of SampleTree m, n instructions: initialize τ = 0 , h 0 ( x ) =repeat ν τ = SampleTree ( 0 R N ; m, n, 1) — sample random tree θ τ ∼ N (cid:16) 0 R Lντ , diag (cid:16)

Original: N max { N ( j ) ντ , 1 } : j ∈ { 1 , . . .
Revised: N max { N ( j ) ντ , 1 } : j ∈ { 1 , . . .

Original: , L ν τ } (cid:17)(cid:17) — generate random values in leavesh τ
Revised: , L ν τ } (cid:17)(cid:17) — generate random values in leaves

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: 1 √ T (cid:10)

Original: [Equation]
Revised: [Equation]

Original: 1 √ T (cid:10) φ ν τ ( · ), θ τ (cid:11) R Lντ — update model
Revised: ν τ ( · )

Original: [Equation]
Revised: [Equation]

Original: =
Revised: θ τ (cid:11) R Lντ — update model

Original: [Equation]
Revised: [Equation]

Original: +until τ ≥ T return: h T ( · )
Revised: until τ ≥ T return: h T ( · )

Original: 4. P OSTERIOR S AMPLING
Revised: 4. P OSTERIOR SAMPLING

Original: Now we are ready to introduce Algorithm 4 for sampling from the posterior.
Revised: Now we are ready to introduce Algorithm 4 for sampling from the posterior.

Original: The procedure is simple: we ﬁrst perform T 0 iterations of SamplePrior to obtain a function h T 0 ( · ) and then we train a standard GBDT model f T 1 ( · ) approximating y N − σh T 0 ( x N ) + N ( 0 N , δ 2 I N ) .
Revised: The procedure is simple: we ﬁrst perform T 0 iterations of SamplePrior to obtain a function h T 0 ( · ) and then we train a standard GBDT model f T 1 ( · ) approximating y N − σh T 0 ( x N ) + N ( 0 N , δ 2 I N ) .

Original: Our ﬁnal model is σh T 0 ( · ) + f T 1 ( · ) .
Revised: Our ﬁnal model is σh T 0 ( · ) + f T 1 ( · ) .

Original: [Figure]
Revised: [Figure]

Original: We further refer to this procedure as
Revised: We further refer to this procedure as

Original: SamplePosterior or KGB (Kernel Gradient Boosting) for brevity.
Revised: SamplePosterior or KGB (Kernel Gradient Boosting) for brevity.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst limit is with respect to the point-wise convergence of stochastic processes and the second one with respect to L 2 ( ρ ) convergence.
Revised: where the ﬁrst limit is with respect to the point-wise convergence of stochastic processes and the second one with respect to L 2 ( ρ ) convergence.

Original: The following theorem shows that KGB indeed samples from the desired posterior.
Revised: The following theorem shows that KGB indeed samples from the desired posterior.

Original: The proof directly follows from
Revised: The proof directly follows from

Original: Lemmas 4.1 and 2.1.
Revised: Lemmas 4.1 and 2.1.

Original: input: dataset z = ( x N , y N ) hyper-parameters: learning rate (cid:15) > 0 , boosting iteration T 1 , SamplePrior iterations T 0 , parameters of SampleTree m, n, β , kernel scale σ > 0 (default σ = 1 ), noise scale δ > 0 (default: δ = 0 .
Revised: input: dataset z = ( x N , y N ) hyper-parameters: learning rate (cid:15) > 0 , boosting iteration T 1 , SamplePrior iterations T 0 , parameters of SampleTree m, n, β , kernel scale σ > 0 (default σ = 1 ), noise scale δ > 0 (default: δ = 0 .

Original: 01 ) instructions:
Revised: 01 ) instructions:

Original: [Equation]
Revised: [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Original: , m, n, β,
Revised: TrainGBDT

Original: [Equation]
Revised: [Equation]

Original: 2 σ

Original: [Equation]
Revised: [Equation]

Original:  (cid:1) return: σh T 0 ( · ) + f T 1 ( · )
Revised: N , y newN ); (cid:15), T 1 , m, n, β, δ 2 σ 2 (cid:1) return: σh T 0 ( · ) + f T 1 ( · )

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: In the limit, the output of Algorithm 4 follows the Gaussian process posterior:
Revised: In the limit, the output of Algorithm 4 follows the Gaussian process posterior:

Original: [Equation]
Revised: [Equation]

Original: with mean (cid:101) f
Revised: with mean (cid:101) f

Original: ( x ) =
Revised: ( x ) =

Original: K ( x, x N ) (cid:0) K ( x N , x N ) + λI N (cid:1) − 1 y N and covariance (cid:101) K ( x, x )
Revised: K ( x, x N ) (cid:0) K ( x N , x N ) + λI N (cid:1) − 1 y N and covariance (cid:101) K ( x, x )

Original: = δ 2 + σ 2 (cid:0) K
Revised: = δ 2 + σ 2 (cid:0) K

Original: ( x, x ) − K ( x, x N ) (cid:0) K ( x N , x N ) + λI N (cid:1) − 1 K ( x N , x ) (cid:1) .
Revised: ( x, x ) − K ( x, x N ) (cid:0) K ( x N , x N ) + λI N (cid:1) − 1 K ( x N , x ) (cid:1) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section empirically evaluates the proposed KGB algorithm and shows that it indeed allows for better knowledge uncertainty estimates.
Revised: This section empirically evaluates the proposed KGB algorithm and shows that it indeed allows for better knowledge uncertainty estimates.

Original: Synthetic experiment To illustrate the KGB algorithm in a controllable setting, we ﬁrst conduct a synthetic experiment.
Revised: Synthetic experiment To illustrate the KGB algorithm in a controllable setting, we ﬁrst conduct a synthetic experiment.

Original: For this, we deﬁned the feature distribution as uniform over D =
Revised: For this, we deﬁned the feature distribution as uniform over D =

Original: { ( x, y ) ∈ [0 , 1] 2 : 110 ≤ ( x − 12 ) 2 − ( y − 12 ) 2 ≤ 14 ∧ ( x ≤ 25 ∨ x ≥ 35 ) ∧ ( y ≤ 25 ∨ y ≥ 35 ) } .
Revised: { ( x, y ) ∈ [0 , 1] 2 : 110 ≤ ( x − 12 ) 2 − ( y − 12 ) 2 ≤ 14 ∧ ( x ≤ 25 ∨ x ≥ 35 ) ∧ ( y ≤ 25 ∨ y ≥ 35 ) } .

Original: We sample 10K points from U ([0 , 1] 2 ) and take into the train set only those that fall into D .
Revised: We sample 10K points from U ([0 , 1] 2 ) and take into the train set only those that fall into D .

Original: The target is deﬁned as f ( x, y ) =
Revised: The target is deﬁned as f ( x, y ) =

Original: x + y .
Revised: x + y .

Original: Figure 1(a) illustrates the training dataset colored with the target values.
Revised: Figure 1(a) illustrates the training dataset colored with the target values.

Original: For evaluation, we take the same 10K points without restricting them to D .
Revised: For evaluation, we take the same 10K points without restricting them to D .

Original: For KGB, we ﬁx (cid:15) = 0 .
Revised: For KGB, we ﬁx (cid:15) = 0 .

Original: 3 , T 0 = 100 , T 1 = 900 , σ = 10 − 2 , δ = 10 − 4 β = 0 .
Revised: 3 , T 0 = 100 , T 1 = 900 , σ = 10 − 2 , δ = 10 − 4 β = 0 .

Original: 1 , m = 4 , n = 64 , and sampled 100 KGB models.
Revised: 1 , m = 4 , n = 64 , and sampled 100 KGB models.

Original: Figure 1(b) shows the estimated by Monte-Carlo posterior mean (cid:101) µ .
Revised: Figure 1(b) shows the estimated by Monte-Carlo posterior mean (cid:101) µ .

Original: On
Revised: On

Original: Figure 1(c), we show log (cid:101) σ , where (cid:101) σ 2 is the posterior variance estimated by Monte-Carlo.
Revised: Figure 1(c), we show log (cid:101) σ , where (cid:101) σ 2 is the posterior variance estimated by Monte-Carlo.

Original: One can see that the posterior variance is small in-domain and grows when we move outside the dataset D , as desired.
Revised: One can see that the posterior variance is small in-domain and grows when we move outside the dataset D , as desired.

Original: Experiment on real datasets Uncertainty estimates for GBDTs have been previously analyzed by Malinin et al.
Revised: Experiment on real datasets Uncertainty estimates for GBDTs have been previously analyzed by Malinin et al.

Original: Our experiments on real datasets closely follow their setup, and we compare the proposed KGB with SGB, SGLB, and their ensembles.
Revised: Our experiments on real datasets closely follow their setup, and we compare the proposed KGB with SGB, SGLB, and their ensembles.

Original: For the experiments, we use several standard regression datasets (Gal & Ghahramani, 2016).
Revised: For the experiments, we use several standard regression datasets (Gal & Ghahramani, 2016).

Original: The implementation details can be found in Appendix H.
Revised: The implementation details can be found in Appendix H.

Original: The code of our experiments can be found on GitHub.
Revised: The code of our experiments can be found on GitHub.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We note that in our setup, we cannot compute likelihoods as kernel K is deﬁned implicitly [Table] and its evaluation requires summing up among all possible trees structures number of which grows as ( nd ) m which is unfeasible, not to mention the requirement to inverse the kernel which requires
Revised: We note that in our setup, we cannot compute likelihoods as kernel K is deﬁned implicitly, and its evaluation requires summing up among all possible trees structures number of which grows as ( nd ) m which is unfeasible, not to mention the requirement to inverse the kernel which requires

Original: O ( N 2+ ω ) operations which additionally rules out the applicability of classical Gaussian Processes methods with our kernel.
Revised: O ( N 2+ ω ) operations which additionally rules out the applicability of classical Gaussian Processes methods with our kernel.

Original: Therefore, a typical Bayesian setup is not applicable, and we resort to the uncertainty estimation setup described in Malinin et al.
Revised: Therefore, a typical Bayesian setup is not applicable, and we resort to the uncertainty estimation setup described in Malinin et al.

Original: Also, the intractability of the kernel does not allow us to treat σ, δ in a fully Bayesian way, as it will require estimating the likelihood.
Revised: Also, the intractability of the kernel does not allow us to treat σ, δ in a fully Bayesian way, as it will require estimating the likelihood.

Original: Therefore, we ﬁx them as constants, but we note that this will not affect the evaluation metrics for our setup as they are scale and translation invariant.
Revised: Therefore, we ﬁx them as constants, but we note that this will not affect the evaluation metrics for our setup as they are scale and translation invariant.

Original: First, we compare KGB with SGLB since they both sample from similar posterior distributions.
Revised: First, we compare KGB with SGLB since they both sample from similar posterior distributions.

Original: Thus, this comparison allows us to ﬁnd out which of the algorithms does a better sampling from the posterior and thus provides us with more reliable estimates of knowledge uncertainty.
Revised: Thus, this comparison allows us to ﬁnd out which of the algorithms does a better sampling from the posterior and thus provides us with more reliable estimates of knowledge uncertainty.

Original: Moreover, we consider the SGB approach as the most “straightforward” way to generate an ensemble of models.
Revised: Moreover, we consider the SGB approach as the most “straightforward” way to generate an ensemble of models.

Original: In Table 1, we compare the predictive performance of the methods.
Revised: In Table 1, we compare the predictive performance of the methods.

Original: Interestingly, we obtain improvements on almost all the datasets.
Revised: Interestingly, we obtain improvements on almost all the datasets.

Original: Here we perform cross-validation to estimate statistical signiﬁcance with paired t -test and highlight the approaches that are insigniﬁcantly different from the best one (p-value > 0 . 05 ).
Revised: Here we perform cross-validation to estimate statistical signiﬁcance with paired t -test and highlight the approaches that are insigniﬁcantly different from the best one (p-value > 0 . 05 ).

Original: Then, we check whether uncertainty measured as the variance of the model’s predictions can be used to detect errors and out-of-domain inputs.
Revised: Then, we check whether uncertainty measured as the variance of the model’s predictions can be used to detect errors and out-of-domain inputs.

Original: Detecting errors can be evaluated via the Prediction-Rejection Ratio (PRR) (Malinin, 2019; Malinin et al., 2020).
Revised: Detecting errors can be evaluated via the Prediction-Rejection Ratio (PRR) (Malinin, 2019; Malinin et al., 2020).

Original: PRR measures how well uncertainty estimates correlate with errors and rank-order them.
Revised: PRR measures how well uncertainty estimates correlate with errors and rank-order them.

Original: Out-of-domain (OOD) detection is usually assessed via the area under the ROC curve (AUC-ROC) for the binary task of detecting whether a sample is OOD (Hendrycks & Gimpel, 2016).
Revised: Out-of-domain (OOD) detection is usually assessed via the area under the ROC curve (AUC-ROC) for the binary task of detecting whether a sample is OOD (Hendrycks & Gimpel, 2017).

Original: For this, one needs an OOD test set.
Revised: For this, one needs an OOD test set.

Original: We use the same OOD test sets (sampled from other datasets) as Malinin et al.
Revised: We use the same OOD test sets (sampled from other datasets) as Malinin et al.

Original: The results of this experiment are given in Table 2.
Revised: The results of this experiment are given in Table 2.

Original: We can see that the proposed method signiﬁcantly outperforms the baselines for out-of-domain detection.
Revised: We can see that the proposed method signiﬁcantly outperforms the baselines for out-of-domain detection.

Original: These improvements can be explained by the theoretical soundness of KGB: convergence properties are theoretically grounded and non-asymptotic.
Revised: These improvements can be explained by the theoretical soundness of KGB: convergence properties are theoretically grounded and non-asymptotic.

Original: In contrast, for SGB, there are no general results applicable in our setting, while for SGLB the guarantees are asymptotic.
Revised: In contrast, for SGB, there are no general results applicable in our setting, while for SGLB the guarantees are asymptotic.

Original: In summary, these results show that our approach is superior to SGB and SGLB, achieving smaller values of RMSE and having better knowledge uncertainty estimates.
Revised: In summary, these results show that our approach is superior to SGB and SGLB, achieving smaller values of RMSE and having better knowledge uncertainty estimates.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper theoretically analyses the classic gradient boosting algorithm.
Revised: This paper theoretically analyses the classic gradient boosting algorithm.

Original: In particular, we show that
Revised: In particular, we show that

Original: GBDT converges to the solution of a certain Kernel Ridge Regression problem.
Revised: GBDT converges to the solution of a certain Kernel Ridge Regression problem.

Original: We also introduce a simple modiﬁcation of the classic algorithm allowing one to sample from the Gaussian posterior.
Revised: We also introduce a simple modiﬁcation of the classic algorithm allowing one to sample from the Gaussian posterior.

Original: The proposed method gives much better knowledge uncertainty estimates than the existing approaches.
Revised: The proposed method gives much better knowledge uncertainty estimates than the existing approaches.

Original: We highlight the following important directions for future research.
Revised: We highlight the following important directions for future research.

Original: First, to explore how one can control the kernel and use it for better knowledge uncertainty estimates.
Revised: First, to explore how one can control the kernel and use it for better knowledge uncertainty estimates.

Original: Also, we do not analyze generalization in the current work, which is another important research topic.
Revised: Also, we do not analyze generalization in the current work, which is another important research topic.

Original: Finally, we need to establish universal approximation property which further justiﬁes need for functional formalism.
Revised: Finally, we need to establish universal approximation property which further justiﬁes need for functional formalism.

Original: A N OTATION USED IN THE PAPER
Revised: A N OTATION USED IN THE PAPER

Original: For convenience, let us list some frequently used notation:
Revised: For convenience, let us list some frequently used notation:

Original: • X ⊂ R d — feature space; • d — dimension of feature vectors; • Y ⊂ R — target space; • ρ — distribution of features; • N — number of samples; • z = ( x N , y N ) — dataset; • V — set of all possible tree structures; • L ν : V → N — number of leaves for ν ∈ V ; • D ( ν, r ) — score used to choose a split (3); • S — indices of all possible splits; • n — number of borders in our implementation of SampleTree; • m — depth of the tree in our implementation of SampleTree; • β — random strength; • (cid:15) — learning rate; • λ — regularization; • F — space of all possible ensembles of trees from V ; • φ ν : X → { 0 , 1 } L ν — tree structure; • φ ( j ) ν — indicator of j -th leaf; • V ( f ) — empirical error of a model f relative to the best possible f (cid:48) ∈ F ; • k ν ( · , · ) — single tree kernel; • K ( · , · ) — stationary kernel of the gradient boosting; • p ( ·| f, β ) — distribution of trees, f ∈ F ; • π ( · ) =
Revised: • X ⊂ R d — feature space; • d — dimension of feature vectors; • Y ⊂ R — target space; • ρ — distribution of features; • N — number of samples; • z = ( x N , y N ) — dataset; • V — set of all possible tree structures; • L ν : V → N — number of leaves for ν ∈ V ; • D ( ν, r ) — score used to choose a split (3); • S — indices of all possible splits; • n — number of borders in our implementation of SampleTree; • m — depth of the tree in our implementation of SampleTree; • β — random strength; • (cid:15) — learning rate; • λ — regularization; • F — space of all possible ensembles of trees from V ; • φ ν : X → { 0 , 1 } L ν — tree structure; • φ ( j ) ν — indicator of j -th leaf; • V ( f ) — empirical error of a model f relative to the best possible f (cid:48) ∈ F ; • k ν ( · , · ) — single tree kernel; • K ( · , · ) — stationary kernel of the gradient boosting; • p ( ·| f, β ) — distribution of trees, f ∈ F ; • π ( · ) =

Original: lim β →∞ p ( ·| f, β ) = p ( ·| f ∗ , β ) — stationary distribution of trees; • σ — kernel scale.
Revised: lim β →∞ p ( ·| f, β ) = p ( ·| f ∗ , β ) — stationary distribution of trees; • σ — kernel scale.

Original: The following diagram illustrates the kernels introduced in our paper:
Revised: The following diagram illustrates the kernels introduced in our paper:

Original: Weak learner’ kernel k
Revised: Weak learner’ kernel k

Original: ν τ : X × X
Revised: ν τ : X × X

Original: → R + Σ ν τ : F → F Σ f τ : F → F
Revised: → R + Σ [Equation] : F → F

Original: Iteration kernel K f τ : X × X
Revised: Iteration kernel K f τ : X × X

Original: → R +
Revised: → R +

Original: Stationary Kernel K : X × X
Revised: Stationary Kernel K : X × X

Original: → R +
Revised: [Equation] R +

Original: H = (cid:0) F , (cid:104)· , ·(cid:105) H (cid:1)
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: B A DDITIONAL RELATED WORK
Revised: B A DDITIONAL RELATED WORK

Original: Let us brieﬂy discuss some additional related work.
Revised: Let us brieﬂy discuss some additional related work.

Original: Mondrian forest method (Balog et al., 2016) and Generalized Random Forests (Athey et al., 2019), besides having links to the kernel methods, in fact, have the underlying limiting RKHS space that is much smaller than the space of all possible ensembles built on the same weak learners due to the independence of the trees that are added to the ensemble.
Revised: Mondrian forest method (Balog et al., 2016) and Generalized Random Forests (Athey et al., 2019), besides having links to the kernel methods, in fact, have the underlying limiting RKHS space that is much smaller than the space of all possible ensembles built on the same weak learners due to the independence of the trees that are added to the ensemble.

Original: Therefore, there is an issue of high bias when comparing plain gradient boosting with the plain random forest method.
Revised: Therefore, there is an issue of high bias when comparing plain gradient boosting with the plain random forest method.

Original: Also, these two methods are built from scratch to obtain an RKHS interpretation while we provide a link between the existing standard gradient boosting approaches to the kernel methods, i.e., we do not create a novel gradient boosting algorithm but rather show that the existing ones already have such a link to derive convergence rates and to exploit such linkage to obtain formal Gaussian process interpretation of the gradient boosting learning to get uncertainty estimates using well-established gradient boosting libraries.
Revised: Also, these two methods are built from scratch to obtain an RKHS interpretation while we provide a link between the existing standard gradient boosting approaches to the kernel methods, i.e., we do not create a novel gradient boosting algorithm but rather show that the existing ones already have such a link to derive convergence rates and to exploit such linkage to obtain formal Gaussian process interpretation of the gradient boosting learning to get uncertainty estimates using well-established gradient boosting libraries.

Original: Let us mention that there are approaches that study kernels induced by tree ensembles through the perspective of Neural Tangent Kernel (Kanoh & Sugiyama, 2022), though this analysis is not applicable for classical gradient boosting, while ours is. 
Revised: Let us mention that there are approaches that study kernels induced by tree ensembles through the perspective of Neural Tangent Kernel (Kanoh & Sugiyama, 2022), though this analysis is not applicable for classical gradient boosting, while ours is. Let us also brieﬂy discuss the papers on Neural Tangent Kernel, e.g., Jacot et al.

Original: Let us also brieﬂy discuss the papers on Neural Tangent Kernel, e.g., (Jacot et al., 2018; Li & Liang, 2018; Allen-Zhu et al., 2019; Du et al., 2018), that study deep learning convergence through the perspective of kernel methods.
Revised: (2018); Li & Liang (2018); Allen-Zhu et al. Du et al. (2019), that study deep learning convergence through the perspective of kernel methods.

Original: Though such works share similarities with what we do, there are fundamental differences.
Revised: Though such works share similarities with what we do, there are fundamental differences.

Original: First, our work is not in the over-parametrization regime, i.e., our kernel method correspondence works for tree ensembles with ﬁxed parameters, but the correspondence is achieved as the number of iterations goes to inﬁnity.
Revised: First, our work is not in the over-parametrization regime, i.e., our kernel method correspondence works for tree ensembles with ﬁxed parameters, but the correspondence is achieved as the number of iterations goes to inﬁnity.

Original: It is worth noting that the kernel method perspective on deep learning basically establishes that each trained deep learning model is a sample from Gaussian Process posterior (Lee et al., 2020; 2018; Yang, 2019; Cho & Saul, 2009), i.e., is sample-then-optimize.
Revised: It is worth noting that the kernel method perspective on deep learning basically establishes that each trained deep learning model is a sample from Gaussian Process posterior (Lee et al., 2020; 2018; Yang, 2019; Cho & Saul, 2009), i.e., is sample-then-optimize.

Original: For boosting, we achieve this only by introducing Algorithm 4 relying on
Revised: For boosting, we achieve this only by introducing Algorithm 4 relying on

Original: Algorithm 3, which in its essence, is random initialization for gradient boosting.
Revised: Algorithm 3, which in its essence, is random initialization for gradient boosting.

Original: The classic gradient boosting 2 can be considered as the mean value of the Gaussian Process, which has no analogs in the world of deep learning, and to achieve convergence to posterior mean there, one needs to average among many trained models.
Revised: The classic gradient boosting 2 can be considered as the mean value of the Gaussian Process, which has no analogs in the world of deep learning, and to achieve convergence to posterior mean there, one needs to average among many trained models.

Original: This can be considered as an advantage of gradient boosting over deep learning that we derive in our paper.
Revised: This can be considered as an advantage of gradient boosting over deep learning that we derive in our paper.

Original: C C ONVEX OPTIMIZATION IN FUNCTIONAL SPACES
Revised: C C ONVEX OPTIMIZATION IN FUNCTIONAL SPACES

Original: In this section, we formulate basic deﬁnitions of differentiability in functional spaces and the theorem on the convergence of gradient descent in functional spaces.
Revised: In this section, we formulate basic deﬁnitions of differentiability in functional spaces and the theorem on the convergence of gradient descent in functional spaces.

Original: For the proof of the theorem and further details on convex optimization in functional space, the reader can consult Luenberger (1969).
Revised: For the proof of the theorem and further details on convex optimization in functional space, the reader can consult Luenberger (1969).

Original: We consider H to be a Hilbert space with some scalar product (cid:104)· , ·(cid:105) H .
Revised: We consider H to be a Hilbert space with some scalar product (cid:104)· , ·(cid:105) H .

Original: Deﬁnition C.1.
Revised: Deﬁnition C.1.

Original: We say that F : H → R is Fr´echet differentiable if for any f ∈ H there exists a bounded linear functional L f : H → R such that ∀ h ∈ H 
Revised: We say that F : H → R is Fr´echet differentiable if for any f ∈ H there exists a bounded linear functional L f : H → R such that ∀ h ∈ H F ( f

Original: [Equation]
Revised: [Equation]

Original: The value of L f : H → R is denoted by D f F ( f ) and is called a Fr ´ echet differential of F at point f .
Revised: The value of L f : H → R is denoted by D f F ( f ) and is called a Fr ´ echet differential of F at point f .

Original: So, Fr ´ echet differential is a functional D f F : H → B ( H , R ) , where B ( X, Y ) denotes a normed space of linear bounded functionals from X to Y .
Revised: So, Fr ´ echet differential is a functional D f F : H → B ( H , R ) , where B ( X, Y ) denotes a normed space of linear bounded functionals from X to Y .

Original: Deﬁnition C.2.
Revised: Deﬁnition C.2.

Original: Let F : H → R be Fr´echet differentiable with a Fr´echet differential D f F ( f ) that is a bounded linear functional.
Revised: Let F : H → R be Fr´echet differentiable with a Fr´echet differential D f F ( f ) that is a bounded linear functional.

Original: Then, by the Riesz theorem there exists a unique h f such that (cid:0)
Revised: Then, by the Riesz theorem there exists a unique h f such that (cid:0)

Original: D f F ( f ) (cid:1)
Revised: D f F ( f ) (cid:1)

Original: [ h ] = (cid:104)
Revised: [ h ] = (cid:104)

Original: h f , h (cid:105) H ∀ h ∈ H .
Revised: h f , h (cid:105) H ∀ h ∈ H .

Original: We call such element a gradient of F in H at f ∈ H and denote it by ∇ H F ( f ) =
Revised: We call such element a gradient of F in H at f ∈ H and denote it by ∇ H F ( f ) =

Original: h f ∈ H .
Revised: h f ∈ H .

Original: Deﬁnition C.3.
Revised: Deﬁnition C.3.

Original: F : H → R is said to be twice Fr´echet differentiable if D f F is Fr´echet differentiable, where the deﬁnition of Fr´echet differential is analogous to Deﬁnition C.1 with the only difference that D f F takes values in B ( H , R ) .
Revised: F : H → R is said to be twice Fr´echet differentiable if D f F is Fr´echet differentiable, where the deﬁnition of Fr´echet differential is analogous to Deﬁnition C.1 with the only difference that D f F takes values in B ( H , R ) .

Original: The second Fr´echet differential is denoted by D 2 f F : H → B ( H , B
Revised: The second Fr´echet differential is denoted by D 2 f F : H → B ( H , B

Original: ( H , R )) .
Revised: ( H , R )) .

Original: As there is an isomorphism between B ( H , B ( H , R )) and
Revised: As there is an isomorphism between B ( H , B ( H , R )) and

Original: B ( H × H , R ) , we can consider the second Fr´echet differential to take values in B ( H × H , R ) .
Revised: B ( H × H , R ) , we can consider the second Fr´echet differential to take values in B ( H × H , R ) .

Original: Henceforth, we will not differentiate between B ( H , B ( H , R )) and B ( H × H , R ) .
Revised: Henceforth, we will not differentiate between B ( H , B ( H , R )) and B ( H × H , R ) .

Original: Deﬁnition C.4.
Revised: Deﬁnition C.4.

Original: A linear operator P : H × H → R is said to be semi-positive deﬁnite (denoted by
Revised: A linear operator P : H × H → R is said to be semi-positive deﬁnite (denoted by

Original: P (cid:23) 0 ) if ∀ f ∈ H we have P ( f, f ) ≥ 0 .
Revised: P (cid:23) 0 ) if ∀ f ∈ H we have P ( f, f ) ≥ 0 .

Original: P is said to be positive deﬁnite ( P (cid:31) 0 ) if ∀ f ∈ H \ { 0 } we have P ( f, f ) > 0 .
Revised: P is said to be positive deﬁnite ( P (cid:31) 0 ) if ∀ f ∈ H \ { 0 } we have P ( f, f ) > 0 .

Original: Deﬁnition C.5.
Revised: Deﬁnition C.5.

Original: Given two linear operators P, G : H × H
Revised: Given two linear operators P, G : H × H

Original: → R we write P (cid:23) G if P − G (cid:23) 0 and
Revised: → R we write P (cid:23) G if P − G (cid:23) 0 and

Original: P (cid:31) G if P − G (cid:31) 0 .
Revised: P (cid:31) G if P − G (cid:31) 0 .

Original: Let I ∈ B
Revised: Let I ∈ B

Original: ( H × H , R ) be a linear operator deﬁned as I ( g, h ) =
Revised: ( H × H , R ) be a linear operator deﬁned as I ( g, h ) =

Original: ( g, h ) H .
Revised: ( g, h ) H .

Original: Theorem C.6.
Revised: Theorem C.6.

Original: Let F be bounded below and twice Fr´echet differentiable functional on a Hilbert space H .
Revised: Let F be bounded below and twice Fr´echet differentiable functional on a Hilbert space H .

Original: Assume that D 2 f F ( f ) satisﬁes 0 ≺ mI (cid:22) D 2 f F ( f ) (cid:22) µI .
Revised: Assume that D 2 f F ( f ) satisﬁes 0 ≺ mI (cid:22) D 2 f F ( f ) (cid:22) µI .

Original: Then the gradient descent scheme:
Revised: Then the gradient descent scheme:

Original: [Equation]
Revised: [Equation]

Original: converges to f ∗ that minimizes
Revised: converges to f ∗ that minimizes

Original: F.
Revised: F.

Original: Proof.
Revised: Proof.

Original: For the proof see Luenberger (1969).
Revised: For the proof see Luenberger (1969).

Original: D K ERNEL RIDGE REGRESSION AND RKHS
Revised: D K ERNEL RIDGE REGRESSION AND RKHS

Original: Deﬁnition D.1.
Revised: Deﬁnition D.1.

Original: K :
Revised: K :

Original: X × X
Revised: X × X

Original: → R is called a kernel function if it is positive semi-deﬁnite, i.e., ∀ N ∈ N + ∀ x N ∈ X N : K ( x N , x N ) (cid:23) 0 .
Revised: → R is called a kernel function if it is positive semi-deﬁnite, i.e., ∀ N ∈ N + ∀ x N ∈ X N : K ( x N , x N ) (cid:23) 0 .

Original: Deﬁnition D.2.
Revised: Deﬁnition D.2.

Original: For any kernel function we can deﬁne a Reproducing Kernel Hilbert Space (RKHS)
Revised: For any kernel function we can deﬁne a Reproducing Kernel Hilbert Space (RKHS)

Original: [Equation]
Revised: [Equation]

Original: with a scalar product such that
Revised: with a scalar product such that

Original: [Equation]
Revised: [Equation]

Original: Consider the following Kernel Ridge Regression problem:
Revised: Consider the following Kernel Ridge Regression problem:

Original: [Equation]
Revised: [Equation]

Original: and the following Kernel Ridgeless Regression problem:
Revised: and the following Kernel Ridgeless Regression problem:

Original: [Equation]
Revised: [Equation]

Original: Lemma D.3.
Revised: Lemma D.3.

Original: min H ( K ) V ( f, λ ) has the only solution
Revised: min H ( K ) V ( f, λ ) has the only solution

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, let us show that f λ ∗ ∈ span (cid:8) K ( · , x i ) (cid:9) .
Revised: First, let us show that f λ ∗ ∈ span (cid:8) K ( · , x i ) (cid:9) .

Original: Let H ( K ) =
Revised: Let H ( K ) =

Original: span (cid:8) K ( · , x i ) (cid:9) ⊕ span (cid:8) K ( · , x i ) (cid:9) ⊥ and consider the projector P : H ( K ) → H ( K ) onto the space span (cid:8) K ( · , x i ) (cid:9) .
Revised: span (cid:8) K ( · , x i ) (cid:9) ⊕ span (cid:8) K ( · , x i ) (cid:9) ⊥ and consider the projector P : H ( K ) → H ( K ) onto the space span (cid:8) K ( · , x i ) (cid:9) .

Original: It is easy to show that P ( f )( x N ) =
Revised: It is easy to show that P ( f )( x N ) =

Original: f
Revised: f

Original: ( x N ) for any f ∈ H ( K ) .
Revised: ( x N ) for any f ∈ H ( K ) .

Original: Indeed, ( f − P ( f ))
Revised: Indeed, ( f − P ( f ))

Original: [ x N ] = (cid:104)
Revised: [ x N ] = (cid:104)

Original: f − P ( f ) , K ( · , x N ) (cid:105) = 0 .
Revised: f − P ( f ) , K ( · , x N ) (cid:105) = 0 .

Original: If f λ ∗ does not lie in span (cid:8) K ( · , x i ) (cid:9) , then (cid:107) f λ ∗ (cid:107) H ( K ) >
Revised: If f λ ∗ does not lie in span (cid:8) K ( · , x i ) (cid:9) , then (cid:107) f λ ∗ (cid:107) H ( K ) >

Original: (cid:107) P ( f λ ∗ ) (cid:107) H ( K ) and V ( P ( f λ ∗ ) , λ ) <
Revised: (cid:107) P ( f λ ∗ ) (cid:107) H ( K ) and V ( P ( f λ ∗ ) , λ ) <

Original: V ( f λ ∗ , λ ) .
Revised: V ( f λ ∗ , λ ) .

Original: We get a contradiction with the minimality of f λ ∗ .
Revised: We get a contradiction with the minimality of f λ ∗ .

Original: Now, let us prove the existence of f λ ∗ .
Revised: Now, let us prove the existence of f λ ∗ .

Original: Consider f = K ( · , x N ) c , where c ∈ R N .
Revised: Consider f = K ( · , x N ) c , where c ∈ R N .

Original: Then, we ﬁnd the optimal c by taking a derivative of V ( f, λ ) with respect to c and equating it to zero:
Revised: Then, we ﬁnd the optimal c by taking a derivative of V ( f, λ ) with respect to c and equating it to zero:

Original: [Equation]
Revised: [Equation]

Original: Then, c v = ( K ( x N , x N ) + λI ) − 1 ( y N + v ) , where v ∈ ker K ( x N , x N ) .
Revised: Then, c v = ( K ( x N , x N ) + λI ) − 1 ( y N + v ) , where v ∈ ker K ( x N , x N ) .

Original: Note that all K ( · , x N ) c v are equal.
Revised: Note that all K ( · , x N ) c v are equal.

Original: Then, we have the only solution of the KRR problem:
Revised: Then, we have the only solution of the KRR problem:

Original: [Equation]
Revised: [Equation]

Original: Lemma D.4.
Revised: Lemma D.4.

Original: min H ( K ) V ( f ) has the only solution in span (cid:8) K ( · , x i ) (cid:9) and it is the solution of minimum RKHS norm:
Revised: min H ( K ) V ( f ) has the only solution in span (cid:8) K ( · , x i ) (cid:9) and it is the solution of minimum RKHS norm:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider f = K ( · , x N ) c , where c ∈ R N .
Revised: Consider f = K ( · , x N ) c , where c ∈ R N .

Original: Now consider V ( f ) and differentiate it with respect to c .
Revised: Now consider V ( f ) and differentiate it with respect to c .

Original: If we equate the derivative to zero, we get:
Revised: If we equate the derivative to zero, we get:

Original: [Equation]
Revised: [Equation]

Original: Then, K ( x N , x N ) c − ( y N + v ) = 0 for some v ∈ ker K ( x N , x N ) .
Revised: Then, K ( x N , x N ) c − ( y N + v ) = 0 for some v ∈ ker K ( x N , x N ) .

Original: Note that y N +ker K ( x N , x N ) ∩
Revised: Note that y N +ker K ( x N , x N ) ∩

Original: Im K ( x N , x N ) (cid:54) = ∅ .
Revised: Im K ( x N , x N ) (cid:54) = ∅ .

Original: Then, for any v such that y N + v ∈
Revised: Then, for any v such that y N + v ∈

Original: Im K ( x N , x N ) there exists a solution c v =
Revised: Im K ( x N , x N ) there exists a solution c v =

Original: K ( x N , x N ) † ( y N + v ) + ker K ( x N , x N ) .
Revised: K ( x N , x N ) † ( y N + v ) + ker K ( x N , x N ) .

Original: This follows from the fact that K ( x N , x N ) K ( x N , x N ) † is an orthoprojector onto Im K ( x N , x N ) .
Revised: This follows from the fact that K ( x N , x N ) K ( x N , x N ) † is an orthoprojector onto Im K ( x N , x N ) .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Then, the existence and uniqueness of f ∗ follow.
Revised: Then, the existence and uniqueness of f ∗ follow.

Original: Now, consider a linear space F ⊂ L 2 ( ρ ) of all possible ensembles of trees from V :
Revised: Now, consider a linear space F ⊂ L 2 ( ρ ) of all possible ensembles of trees from V :

Original: [Equation]
Revised: [Equation]

Original: Deﬁne the unique function:
Revised: Deﬁne the unique function:

Original: [Equation]
Revised: [Equation]

Original: Then following two lemmas hold:
Revised: Then following two lemmas hold:

Original: Lemma D.5 (Lemma 3.7 in the main text) .
Revised: Lemma D.5.

Original: [Equation]
Revised: (cid:104)

Original:  − [Equation] 0 for any f ∈ F .
Revised: y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N = 0 for any f ∈ F .

Original: Proof.
Revised: Proof.

Original: Assume that (cid:104) y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N (cid:54) = 0 for some f ∈ F .
Revised: Assume that (cid:104) y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N (cid:54) = 0 for some f ∈ F .

Original: Then, for some f ∈ F , (cid:104) y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N > 0 .
Revised: Then, for some f ∈ F , (cid:104) y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N > 0 .

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: for small enough α > 0 , which contradicts with the deﬁnition of f ∗ .
Revised: for small enough α > 0 , which contradicts with the deﬁnition of f ∗ .

Original: Lemma D.6 (Lemma 3.8 in the main text) . V
Revised: Lemma D.6.

Original: [Equation]
Revised: [Equation]

Original:  f, λ [Equation] N [Equation]
Revised: where C λ = inf f ∈ F L ( f, λ ) − inf f ∈ F L ( f ) ≥ 0 .

Original: + C .

Original: Proof.
Revised: Proof.

Original: We need to prove it only for V ( f ) withour regularization as for regularized it follows immidiately (note that f ∗ is minimizer of not regularized objective).
Revised: We need to prove it only for V ( f ) without regularization as for regularized it follows immediately (note that f ∗ is minimizer of not regularized objective).

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Now, let us prove that
Revised: Now, let us prove that

Original: [Equation]
Revised: [Equation]

Revised: N = 0 .

Original: Indeed,
Revised: Indeed,

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows from the previous lemma
Revised: where the last equality follows from the previous lemma

Original: [Equation]
Revised: [Equation]

Revised: Lemma D.

Revised: [Equation]

Revised: Proof.

Revised: f λ ∗ is optimum for V ( f, λ ) .

Revised: Then Frechet derivative at f λ ∗ equals 0:

Revised: [Equation]

Revised: Consider then writing:

Revised: [Equation]

Revised: Explicit formula for Frechet Derivative of V ( f, λ ) you can ﬁnd in Appendix E.

Original: E G AUSSIAN P ROCESS INFERENCE
Revised: E G AUSSIAN P ROCESS INFERENCE

Original: In this section, we prove Lemma 2.1 from Section 2.3 of the main text.
Revised: In this section [Equation] we prove Lemma 2.1 from Section 2.3 of the main text.

Original: Firstly, consider the Kernel Ridge Regression problem loss functional:
Revised: Firstly, consider the following regularized error functional:

Revised: [Equation]

Revised: With this functional we can consider the following optimization problem:

Original: [Equation]
Revised: [Equation]

Revised: which is called as Kernel Ridge Regression.

Original: We will show that it satisﬁes the conditions needed for Theorem C.6.
Revised: We will show that this functional satisﬁes the conditions needed for Theorem C.6.

Original: We will also deduce the formula of the gradient of V in order to show that gradient descent takes the form (2).
Revised: We will also deduce the formula of the gradient of V in order to show that gradient descent takes the form (2).

Original: Lemma E.1.
Revised: Lemma E.1.

Original: V ( f, λ ) is Fr´echet differentiable with the differential given by:
Revised: V ( f, λ ) is Fr´echet differentiable with the differential given by:

Original: [Equation]
Revised: [Equation]

Original: where ev x i : H ( K ) → R is a bounded linear functional such that ev x i ( f )
Revised: where ev x i : H ( K ) → R is a bounded linear functional such that ev x i ( f )

Original: = f
Revised: = f

Original: ( x i ) =
Revised: ( x i ) =

Original: ( f, K ( x i , · ))
Revised: ( f, K ( x i , · ))

Original: H ( K ) .
Revised: H ( K ) .

Original: Proof.
Revised: Proof.

Original: As Fr ´ echet differential is linear, we only need to ﬁnd Fr ´ echet differential for ( f ( x i ) − y i ) 2 .
Revised: As Fr´echet differential is linear, we only need to ﬁnd Fr´echet differential for ( f ( x i ) − y i ) 2 .

Original: Note that ( f ( x i ) − y i ) 2 is a composition of two functions:
Revised: Note that ( f ( x i ) − y i ) 2 is a composition of two functions:

Original: [Equation]
Revised: [Equation]

Original: The differential of the composition can be found as:
Revised: The differential of the composition can be found as:

Original: [Equation]
Revised: [Equation]

Original: where D f F ( f ) =
Revised: where D f F ( f ) =

Original: ev x i because
Revised: ev x i because

Original: [Equation]
Revised: [Equation]

Original: Lemma E.2.
Revised: Lemma E.2.

Original: The gradient of V ( f, λ ) , Riesz representative of the functional above, is given by:
Revised: The gradient of V ( f, λ ) , Riesz representative of the functional above, is given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Follows from the previous lemma.
Revised: Follows from the previous lemma.

Original: Lemma E.3.
Revised: Lemma E.3.

Original: V ( f, λ ) is twice Fr´echet differentiable with the differential given by:
Revised: V ( f, λ ) is twice Fr´echet differentiable with the differential given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Due to the linearity of Fr´echet differential and lemma E.1 we need to ﬁnd only Fr´echet differential for ( f ( x i ) − y i ) ev x i .
Revised: Due to the linearity of Fr´echet differential and lemma E.1 we need to ﬁnd only Fr´echet differential for ( f ( x i ) − y i ) ev x i .

Original: Consider S ( f ) =
Revised: Consider S ( f ) =

Original: ( f ( x i ) − y i ) ev x i .
Revised: ( f ( x i ) − y i ) ev x i .

Original: Then we need to ﬁnd V f ∈ B ( H ( K ) , B ( H ( K ) , R ))
Revised: Then we need to ﬁnd V f ∈ B ( H ( K ) , B ( H ( K ) , R ))

Original: such that
Revised: such that

Original: S ( f + h ) =
Revised: S ( f + h ) =

Original: S ( f ) + V f
Revised: S ( f ) + V f

Original: [ h ] + o ( (cid:107) h (cid:107) ) .
Revised: [ h ] + o ( (cid:107) h (cid:107) ) .

Original: It is easy to show
Revised: It is easy to show

Original: that h (cid:55)→ h
Revised: that h (cid:55)→ h

Original: ( x i ) ev x i ∈ B
Revised: ( x i ) ev x i ∈ B

Original: ( H ( K ) , B ( H ( K ) , R )) and S ( f + h )
Revised: ( H ( K ) , B ( H ( K ) , R )) and S ( f + h )

Original: =
Revised: =

Original: S ( f )+ h ( x i ) ev x i .
Revised: S ( f )+ h ( x i ) ev x i .

Original: Thus, we get that D f S ( f )
Revised: Thus, we get that D f S ( f )

Original: [ h ] =
Revised: [ h ] =

Original: h
Revised: h

Original: ( x i ) ev x i .
Revised: ( x i ) ev x i .

Original: From this the statement of the lemma follows.
Revised: From this the statement of the lemma follows.

Original: Given all the above lemmas, as a corollary of Theorem C.6, we have the following.
Revised: Given all the above lemmas, as a corollary of Theorem C.6, we have the following.

Original: Corollary E.4.
Revised: Corollary E.4.

Original: Gradient descent, deﬁned by the following iterative scheme
Revised: Gradient descent, deﬁned by the following iterative scheme

Original: [Equation]
Revised: [Equation]

Original: converges to the optimum of V ( f, λ ) .
Revised: converges to the optimum of V ( f, λ ) .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By lemma E.2, our update rule has the form
Revised: By lemma E.2, our update rule has the form

Original: [Equation]
Revised: [Equation]

Original: Then, we will ﬁnd m, µ such that 0 ≺ mI (cid:22) D 2 f V ( f, λ ) (cid:22) µI .
Revised: Then, we will ﬁnd m, µ such that 0 ≺ mI (cid:22) D 2 f V ( f, λ ) (cid:22) µI .

Original: By lemma E.3,
Revised: By lemma E.3,

Original: [Equation]
Revised: [Equation]

Original: Then, we can take m = λ N .
Revised: Then, we can take m = λN .

Original: Let us also write
Revised: Let us also write

Original: [Equation]
Revised: [Equation]

Original: Then, we can take µ = λN + 1 N max x ∈X K
Revised: Then, we can take µ = λN + 1 N max x ∈X K

Original: ( x, x ) .
Revised: ( x, x ) .

Original: By theorem C.6 and lemma D.3 the corollary follows.
Revised: By theorem C.6 and lemma D.3 the corollary follows.

Original: Lemma E.5.
Revised: Lemma E.5.

Original: Consider the gradient descent:
Revised: Consider the gradient descent:

Original: [Equation]
Revised: [Equation]

Original: and the following randomization scheme:
Revised: and the following randomization scheme:

Original: sample f init ∼ GP ( 0 L 2 ( ρ ) , σ 2 K + [Equation] 2 Id L 
Revised: sample f init ∼ GP ( 0 L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 ); 2.

Original: [Equation]

Original: ; 2. set new labels y
Revised: set new labels y

Original: newN =
Revised: newN =

Original: y N − f init ( x N ); 3.
Revised: y N − f init ( x N ); 3.

Original: ﬁt GD f τ ( · ) on y
Revised: ﬁt GD f τ ( · ) on y

Original: newN assuming f 0 ( · ) =
Revised: newN assuming f 0 ( · ) =

Original: L 2 ( ρ ) ; 4.
Revised: L 2 ( ρ ) ; 4.

Original: output ˆ f
Revised: output ˆ f

Original: ( · ) =
Revised: ( · ) =

Original: f init ( · ) + f ∞ ( · ) as ﬁnal model .
Revised: f init ( · ) + f ∞ ( · ) as ﬁnal model .

Original: Then [Equation] ˆ f from the scheme above follows the Gaussian Process posterior with the following mean and covariance:
Revised: Then, ˆ f from the scheme above follows the Gaussian Process posterior with the following mean and covariance:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Let us ﬁnd the distribution of ˆ f at x ∈ R n .
Revised: Let us ﬁnd the distribution of ˆ f at x ∈ R n .

Original: It can be easily seen that:
Revised: It can be easily seen that:

Original: [Equation]
Revised: [Equation]

Original: N (cid:17) − 1 y N .

Original: Let us now calculate covariance:
Revised: Let us now calculate covariance:

Original: [Equation]
Revised: [Equation]

Original: , which is exactly what we need.
Revised: which is exactly what we need.

Original: F D ISTRIBUTION OF TREES
Revised: F D ISTRIBUTION OF TREES

Original: Lemma F.1 (Lemma 3.2 in the main text) .
Revised: Lemma F.1 (Lemma 3.2 in the main text) .

Original: [Equation]
Revised: [Equation]

Original: where the sum is over all permutations
Revised: where the sum is over all permutations

Original: ς ∈ P m , ν ς,i =
Revised: ς ∈ P m , ν ς,i =

Original: ( s ς (1) , . . .
Revised: ( s ς (1) , . . .

Original: , s ς ( i ) ) , and ν = ( s 1 , . . .
Revised: , s ς ( i ) ) , and ν = ( s 1 , . . .

Original: , s m ) .
Revised: , s m ) .

Original: Proof.
Revised: Proof.

Original: Let us ﬁx some permutation
Revised: Let us ﬁx some permutation

Original: ς ∈ P m .
Revised: ς ∈ P m .

Original: W.l.o.g.
Revised: W.l.o.g.

Original: , let ς = id P m , i.e. ς ( i ) =
Revised: , let ς = id P m , i.e. ς ( i ) =

Original: i ∀ i .
Revised: i ∀ i .

Original: It remains to derive the formula for the ﬁxed permutation.
Revised: It remains to derive the formula for the ﬁxed permutation.

Original: The probability of adding the next split given the previously build tree is:
Revised: The probability of adding the next split given the previously build tree is:

Original: [Equation]
Revised: [Equation]

Original: which comes from (4) and the Gumbel-SoftMax trick.
Revised: which comes from (4) and the Gumbel-SoftMax trick.

Original: Then, we decompose the probability P ( ν ) of a tree as:
Revised: Then, we decompose the probability P ( ν ) of a tree as:

Original: [Equation]
Revised: [Equation]

Original: and so for the ﬁxed permutation we have
Revised: and so for the ﬁxed permutation we have

Original: [Equation]
Revised: [Equation]

Original: Then we sum over all permutations and the lemma follows.
Revised: Then we sum over all permutations and the lemma follows.

Original: Now, let us deﬁne the following value indicating how different are the distribution of trees for f and f ∗ :
Revised: Now, let us deﬁne the following value indicating how different are the distribution of trees for f and f ∗ :

Original: [Equation]
Revised: [Equation]

Original: Lemma F.2.
Revised: Lemma F.2.

Original: The following bound relates the distributions.
Revised: The following bound relates the distributions.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider π = p ( ·| f ∗ , β ) and the following expression P ( ν, ς ) :
Revised: Consider π = p ( ·| f ∗ , β ) and the following expression P ( ν, ς ) :

Original: [Equation]
Revised: [Equation]

Original: Then, 
Revised: Then, (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where in second inequality we used D ( ν, r ) ≤ 2 V ( f ) which straightly follows from the deﬁnition.
Revised: where in second inequality we used D ( ν, r ) ≤ 2 V ( f ) which straightly follows from the deﬁnition.

Original: By noting that the probabilities remain the same if we shift D ( · , r ) ← D ( · , r ) − 2 V ( f ) which becomes everywhere non-positive and allows us to do the above trick once more but in reverse manner: if we formally replace the D with such modiﬁed function and repeat the steps with reversing the inequalities which is needed since the new function is everywhere negative then the lemma follows.
Revised: By noting that the probabilities remain the same if we shift D ( · , r ) ← D ( · , r ) − 2 V ( f ) which becomes everywhere non-positive and allows us to do the above trick once more but in reverse manner: if we formally replace the D with such modiﬁed function and repeat the steps with reversing the inequalities which is needed since the new function is everywhere negative then the lemma follows.

Original: [Equation]
Revised: [Equation]

Original: G P ROOF OF T HEOREM 3. G. RKHS STRUCTURE
Revised: G P ROOF OF T HEOREM 3. G. RKHS STRUCTURE

Original: In section 3.4 we deﬁned RKHS structure on F as:
Revised: In section 3.4 we deﬁned RKHS structure on F as:

Original: [Equation]
Revised: [Equation]

Original: and we introduced the kernels k ν , K f , K π .
Revised: and we introduced the kernels k ν , K f , K π .

Original: Let us also deﬁne a kernel K p 
Revised: Let us also deﬁne a kernel K p ( · , · )

Original: = (cid:82) X k ν  p (d ν ) for arbitrary distribution p on V .
Revised: = (cid:80) ν ∈V k ν ( · , · ) p ( ν ) for arbitrary distribution p on V .

Original: This way, taking p as δ ν ( · ) , p ( ν | f, β ) , π ( · ) we get K p equal to k ν , K f , K , respectively.
Revised: This way, taking p as δ ν ( · ) , p ( ν | f, β ) , π ( · ) we get K p equal to k ν , K f , K , respectively.

Original: For each kernel, we deﬁne the operator associated with it denoted similarly:
Revised: For each kernel, we deﬁne the operator associated with it denoted similarly:

Original: [Equation]
Revised: [Equation]

Original: Lemma G.1.
Revised: Lemma G.1.

Original: Consider two positive semideﬁnite operators on a ﬁnite dimensional vector space V :
Revised: Consider two positive semideﬁnite operators on a ﬁnite dimensional vector space V :

Original: A : V → V and B : V → V such that A (cid:23) B .
Revised: A : V → V and B : V → V such that A (cid:23) B .

Original: Then, Im A ≥ Im B .
Revised: Then, Im A ≥ Im B .

Original: Lemma G.2.
Revised: Lemma G.2.

Original: K p : F → F is invertible for p non-vanishing on V .
Revised: K p : F → F is invertible for p non-vanishing on V .

Original: Proof.
Revised: Proof.

Original: Note that Im k ν = span
Revised: Note that Im k ν = span

Original: φ jν | j = 1 , . .
Revised: φ jν | j = 1 , . .

Original: , L ν } and p ( ν ) k ν (cid:22) K p .
Revised: , L ν } and p ( ν ) k ν (cid:22) K p .

Original: Then, Im K p =
Revised: Then, Im K p =

Original: F follows from lemma G.1.
Revised: F follows from lemma G.1.

Original: Thus, K p is invertible.
Revised: Thus, K p is invertible.

Original: Lemma G.3.
Revised: Lemma G.3.

Original: F = span (cid:8) K p ( · , x ) | x ∈ X (cid:9) for p non-vanishing on V .
Revised: F = span (cid:8) K p ( · , x ) | x ∈ X (cid:9) for p non-vanishing on V .

Original: Proof.
Revised: Proof.

Original: From lemma G.2, Im K p = F .
Revised: From lemma G.2, Im K p = F .

Original: From the deﬁnition of the operator, Im K p ⊂ span (cid:8) K p ( · , x ) | x ∈ X (cid:9) .
Revised: From the deﬁnition of the operator, Im K p ⊂ span (cid:8) K p ( · , x ) | x ∈ X (cid:9) .

Original: Then, the lemma follows.
Revised: Then, the lemma follows.

Original: Lemma G.4.
Revised: Lemma G.4.

Original: For non-vanishing distribution p on V(cid:104)K p f, g
Revised: For non-vanishing distribution p on V

Original: [Equation]
Revised: [Equation]

Original: ( K p ) =

Original: (cid:104) f, g (cid:105)

Original: L 2 ( ρ ) .

Original: Proof.
Revised: Proof.

Original: It is sufﬁcient to check on the basis.
Revised: It is sufﬁcient to check on the basis.

Original: So, we take g = K p ( · , x ) .
Revised: So, we take g = K p ( · , x ) .

Original: Then,
Revised: Then,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the second equality holds by the deﬁnition of the operator K p .
Revised: where the second equality holds by the deﬁnition of the operator K p .

Original: For a weak learner ν , we deﬁne a covariance operator:
Revised: For a weak learner ν , we deﬁne a covariance operator:

Original: [Equation]
Revised: [Equation]

Original: Also, for an arbitrary probability distribution p over V , we denote Σ p = (cid:82) V Σ ν p (d ν ) .
Revised: Also, for an arbitrary probability distribution p over V , we denote Σ p = (cid:80) ν ∈V Σ ν p ( ν ) .

Original: These operators are typically referred to as covariance operators.
Revised: These operators are typically referred to as covariance operators.

Original: Let us formulate and prove several lemmas about the RKHS structure and operators Σ , Σ f , Σ ν .
Revised: Let us formulate and prove several lemmas about the RKHS structure and operators Σ , Σ f , Σ ν .

Original: Lemma G.5.
Revised: Lemma G.5.

Original: (Courant-Fischer)
Revised: (Courant-Fischer)

Original: Let A be an n × n real symmetric matrix and λ 1 ≤ . . .
Revised: Let A be an n × n real symmetric matrix and λ 1 ≤ . . .

Original: ≤ λ n its eigenvalues.
Revised: ≤ λ n its eigenvalues.

Original: Then, 
Revised: Then, λ

Original: [Equation]
Revised: [Equation]

Original: where R A ( x ) = 
Revised: where R A ( x ) = (cid:104)

Original: ( Ax,x ) ( x,x ) is the Rayleigh-Ritz quotient.
Revised: Ax,x (cid:105) (cid:104) x,x (cid:105) is the Rayleigh-Ritz quotient.

Original: Lemma G.6.
Revised: Lemma G.6.

Original: ρ ( k ν ( x N , x N ))
Revised: ρ ( k ν ( x N , x N ))

Original: = (cid:107) k
Revised: = (cid:107) k

Original: ν
Revised: ν

Original: ( x N , x N ) (cid:107) =
Revised: ( x N , x N ) (cid:107) =

Original: N for any ν ∈ V .
Revised: N for any ν ∈ V .

Original: Proof.
Revised: Proof.

Original: It is easy to see that
Revised: It is easy to see that

Original: [Equation]
Revised: [Equation]

Original: where 1 n × n is a matrix of size n × n consisting of ones.
Revised: where 1 n × n is a matrix of size n × n consisting of ones.

Original: Then, we note that (cid:107) 1 n × n (cid:107) = n and now the statement of the lemma follows.
Revised: Then, we note that (cid:107) 1 n × n (cid:107) = n and now the statement of the lemma follows.

Original: Lemma G.7.
Revised: Lemma G.7.

Original: (Covariation majorization)
Revised: (Covariation majorization)

Original: The following operator inequality holds for probability distributions p, p (cid:48) over V , where p is arbitrary and p (cid:48) does not vanish at any ν ∈ V :
Revised: The following operator inequality holds for probability distributions p, p (cid:48) over V , where p is arbitrary and p (cid:48) does not vanish at any ν ∈ V :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider the following operators:
Revised: Consider the following operators:

Original: [Equation]
Revised: [Equation]

Original: ( · , x N ) v. Then, Σ p = 1 N BA and K N = 1 N K p ( x N , x N ) = 1 N AB .
Revised: Then, Σ p = 1 N BA and K N = 1 N K p ( x N , x N ) = 1 N AB .

Original: As AB and BA have the same spectra, we further study the spectrum of K N .
Revised: As AB and BA have the same spectra, we further study the spectrum of K N .

Original: But K N = 1 N K p ( x N , x N ) = (cid:82) V 1 N k ν ( x N , x N ) p (d ν ) and λ max ( K N ) ≤ 1 follows from lemma
Revised: But K N = 1 N K p ( x N , x N ) = (cid:80) ν ∈V 1 N k ν ( x N , x N ) p ( ν ) and λ max ( K N ) ≤ 1 follows from lemma

Original: G. 6 .
Revised: G. 6 .

Original: Then, λ max (Σ p ) ≤ 1 follows.
Revised: Then, λ max (Σ p ) ≤ 1 follows.

Original: Now we need to show that λ min (Σ p (cid:48) ) ≥ 1 N .
Revised: Now we need to show that λ min (Σ p (cid:48) ) ≥ 1 N .

Original: Consider the following formula:
Revised: Consider the following formula:

Original: [Equation]
Revised: [Equation]

Original: where ( a ⊗ H ( K p (cid:48) )
Revised: where ( a ⊗ H ( K p (cid:48) )

Original: b )
Revised: b )

Original: [ c ] = (cid:104) b, c (cid:105) H  K p (cid:48)  a .
Revised: [ c ] = (cid:104) b, c (cid:105) H ( K p (cid:48) ) a .

Original: If a = b and (cid:107) a (cid:107) H ( K p (cid:48) )
Revised: If a = b and (cid:107) a (cid:107) H ( K p (cid:48) )

Original: = 1 , then 1 and 0 are the only eigenvalues of a ⊗ H ( K p (cid:48) ) a .
Revised: = 1 , then 1 and 0 are the only eigenvalues of a ⊗ H ( K p (cid:48) ) a .

Original: Denote by S = span {K p (cid:48) ( · , x i ) | i = 1 , . . .
Revised: Denote by S = span {K p (cid:48) ( · , x i ) | i = 1 , . . .

Original: , N } ⊂ H ( K p (cid:48) ) and m = dimS, n = dim H ( K p (cid:48) ) .
Revised: , N } ⊂ H ( K p (cid:48) ) and m = dimS, n = dim H ( K p (cid:48) ) .

Original: Then,λ min (Σ
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where R Σ p (cid:48)
Revised: where R Σ p (cid:48)

Original: ( x ) = (Σ p (cid:48)
Revised: ( x ) = (Σ p (cid:48)

Original: x,x ) H ( K p(cid:48) )
Revised: x,x ) H ( K p

Revised: [Equation]

Original:  ( x,x ) H ( K p (cid:48) ) .
Revised: ) ( x,x ) H ( K p (cid:48) ) .

Original: As dimU = n − m + 1 , then U ∩ S (cid:54) = ∅ .
Revised: As dimU = n − m + 1 , then U ∩ S (cid:54) = ∅ .

Original: Suppose K p (cid:48) ( · , x i ) ∈
Revised: Suppose K p (cid:48) ( · , x i ) ∈

Original: U ∩ S , then
Revised: U ∩ S , then

Original: [Equation]
Revised: [Equation]

Original: where (*) is fulﬁlled as a ⊗ H ( K p (cid:48) ) a is a positive semideﬁnite operator and the last inequality follows from K p (cid:48)
Revised: where (*) is fulﬁlled as a ⊗ H ( K p (cid:48) ) a is a positive semideﬁnite operator and the last inequality follows from K p (cid:48)

Original: ( x, x ) ≥ 1 ∀ x ∈ X .
Revised: ( x, x ) ≥ 1 ∀ x ∈ X .

Revised: G. N ORM MAJORIZATION

Revised: Following lemmas relate norms L 2 , H , R N 9 with respect to each other.

Revised: Indeed, by these lemmas we can consider bound (cid:107) · (cid:107) L 2 (cid:46) (cid:107) · (cid:107) H ≤ (cid:107) · (cid:107) R N .

Revised: Further, in the main theorems we will use these relations extensively.

Original: Corollary G.8.
Revised: Corollary G.8.

Original: [Equation] for f ∈ span {K ( · , x i ) | i = 1 , . . .
Revised: (cid:107) f ( x N ) (cid:107) ≥ (cid:107) f (cid:107) H for f ∈ span {K ( · , x i ) | i = 1 , . . .

Original: , N } .
Revised: , N } .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: as Σ (cid:23) 1 N I on span {K ( · , x i ) | i = 1 , . . .
Revised: as Σ (cid:23) 1 N I on span {K ( · , x i ) | i = 1 , . . .

Original: , N } .
Revised: , N } .

Original: Lemma G.9.
Revised: Lemma G.9.

Original: λ max ( K ) ≤ max x ∈X K ( x, x ) .
Revised: λ max ( K ) ≤ max x ∈X K ( x, x ) .

Original: Proof.
Revised: Proof.

Original: Consider K as an operator on ( F , L 2 ( ρ )) .
Revised: Consider K as an operator on ( F , L 2 ( ρ )) .

Original: We will prove that (cid:107)K(cid:107) B (( F ,L 2 ( ρ ))) ≤ max x ∈X K ( x, x ) and the lemma will follow.
Revised: We will prove that (cid:107)K(cid:107) B (( F ,L 2 ( ρ ))) ≤ max x ∈X K ( x, x ) and the lemma will follow.

Original: Consider the inequality
Revised: Consider the inequality

Original: [Equation]
Revised: [Equation]

Original: f [Equation] Then, (cid:107)K [ f ] (cid:107) L 2 ≤ max x ∈X (cid:107)K x (cid:107) L 2 (cid:107) f (cid:107) L 2 .
Revised: Then, (cid:107)K [ f ] (cid:107) L 2 ≤ max x ∈X (cid:107)K x (cid:107) L 2 (cid:107) f (cid:107) L 2 .

Original: Note also that K ( x, x (cid:48) )
Revised: Note also that K ( x, x (cid:48) )

Original: ≤ min( K ( x, x ) , K ( x (cid:48) , x (cid:48) )) which can be easily seen from the deﬁnition.
Revised: ≤ min( K ( x, x ) , K ( x (cid:48) , x (cid:48) )) which can be easily seen from the deﬁnition.

Original: Then, (cid:107)K x (cid:107) L 2 ≤ K ( x, x ) and the lemma follows.
Revised: Then, (cid:107)K x (cid:107) L 2 ≤ K ( x, x ) and the lemma follows.

Original: Corollary G.10.
Revised: Corollary G.10.

Original: (Expected squared norm majorization by RKHS norm)
Revised: (Expected squared norm majorization by RKHS norm)

Original: The following bound holds ∀ f ∈ H :
Revised: The following bound holds ∀ f ∈ H :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Then, from the previous lemma the bound holds.
Revised: Then, from the previous lemma the bound holds.

Revised: G. S YMMETRY OF OPERATORS

Revised: In this section we establish symmetry of various operators with respect to norms L 2 , H , R N .

Revised: These results mainly are required to claim that spectral radii of these operators coincide with its operator norms in various spaces: B ( F , L 2 ) , B ( H ) , B ( span {K ( · , x i ) | i = 1 , . . .

Revised: , N } , (cid:107) · (cid:107) R N ) .

Revised: Though, we use symmetry of operators not only this way.

Original: Lemma G.11.
Revised: Lemma G.11.

Revised: (Universal symmetry of covariance operators in H ) Operator Σ p for any p is symmetric w.r.t dot product of H ( K p (cid:48) ) for any non-singular p (cid:48) .

Revised: Proof.

Revised: First

Revised: [Equation]

Revised: let’s prove it for non-singular distribution p .

Original: For any ν ∈ V , we have k ν ( · , x N )[ y N − f ∗ ( x N )]
Revised: To see that we consider the following quantity: [Equation] Then we use the following trick (cid:104)K p [Equation]

Revised: It allows us to rewrite:

Revised: [Equation]

Revised: From which it immediately follows:

Revised: [Equation]

Revised: Which shows that Σ p is indeed symmetric w.r.t.

Revised: dot product of H ( K p (cid:48) ) .

Revised: Finally, we can use continuity argument, which we allowed to use due to intrinsic ﬁnite dimension, to conclude that symmetry must hold for arbitrary distributions, in particular for p = δ ν which corresponds to Σ ν .

Revised: Lemma G.12.

Revised: (Universal symmetry of covariance operators in L 2 ) Operator Σ p for any p is symmetric w.r.t dot product of L 2 .

Revised: Proof.

Revised: We consider similarly the following quantity:

Revised: [Equation]

Revised: Then we use the following trick (cid:104)K p ( · , x i )

Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: L 2 .

Revised: It allows us to rewrite:

Revised: [Equation]

Revised: From which it immediately follows:

Revised: [Equation]

Revised: Which shows that Σ p is indeed symmetric w.r.t.

Revised: dot product of L 2 .

Revised: Lemma G.13.

Revised: (Universal symmetry of kernel operators in H ) Operator K p for any p is symmetric w.r.t dot product of H .

Revised: Proof.

Revised: We consider decomposing K p as:

Revised: [Equation]

Revised: Then we use the following trick f ( x ) = (cid:104)K ( · , x ) , f (cid:105) H .

Revised: It allows us to rewrite:

Revised: [Equation]

Revised: From which it immediately follows:

Revised: [Equation]

Revised: Which shows that K p is indeed symmetric w.r.t.

Revised: dot product of H since both K ( · , x ) ⊗ H K ( · , y ) and

Revised: K ( · , y ) ⊗ H K ( · , x ) are present with the same weight K p ( x, y ) ρ (d x ) ρ (d y )

Revised: =

Revised: K p ( y, x ) ρ (d y ) ρ (d x ) .

Revised: G. I TERATIONS OF G RADIENT B OOSTING

Revised: Lemma G.14.

Revised: For any ν ∈ V , we have k ν ( · , x N )[ y N − f ∗ ( x N )]

Original: = 0 .
Revised: = 0 .

Original: Proof.
Revised: Proof.

Original: Follows from Lemma 3.7.
Revised: Follows from Lemma D.5.

Original: Lemma G.12 (Lemma 3.9 in the main text) .
Revised: Lemma G.15 (Lemma 3.7 in the main text) .

Original: Iterations f
Revised: Iterations f

Original: τ of Gradient Boosting (Algorithm 2) can be written in the form:
Revised: τ of Gradient Boosting (Algorithm 2) can be written in the form:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to Algorithm 2:
Revised: According to Algorithm 2:

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Now note that k ν τ ( · , x i ) =
Revised: Now note that k ν τ ( · , x i ) =

Original: ω jν τ φ jν τ ( · ) , where j is such that φ jν τ ( x i ) = 1 .
Revised: ω jν τ φ jν τ ( · ) , where j is such that φ jν τ ( x i ) = 1 .

Original: From this the lemma follows.
Revised: From this the lemma follows.

Revised: From lemmas G.15, D.4 it is quite easy to show that f τ , f ∗ ∈ span {K ( · , x i ) | i = 1 , . . .

Revised: , N } .

Revised: Then, hereafter we can use corollary G.8 to bound H norm with R N norm.

Original: Lemma G.13.
Revised: Lemma G.16.

Revised: The iterations of gradient boosting can be represented as:

Revised: [Equation]

Revised: where D : F → F is bounded linear operator deﬁned as Riesz representative with respect to Lscalar product of such bilinear function 1 N f ( x N ) T h ( x N )

Revised: = (cid:104) Df, h (cid:105) L 2 .

Revised: Similar decomposition holds for ∇ H V ( f, λ ) =

Revised: K D [ f − f ∗ ] + λN f .

Revised: Proof.

Revised: First observe that Ef τ +1 | f τ = f τ − (cid:15) ∇ V ( f τ , λ ) where gradient here is taken with respect to H ( K f τ ) .

Revised: Keep in mind that in the deﬁnition of V ( f τ , λ ) , norm in the regularizer term is taken with respect to H ( K f τ ) in stead of H ( K ) .

Revised: This way we need only to prove ∇ H V ( f, λ ) =

Revised: K D [ f − f ∗ ] + λN f .

Revised: Consider Frechet differential D f V ( f )

Revised: [ h ] = 1 N ( f ( x N ) − f ∗ ( x N ))

Revised: T h ( x N ) =

Revised: (cid:104) D

Revised: [ f − f ∗ ] , h (cid:105)

Revised: L 2 then by lemma G.4 we deduce

Revised: [Equation]

Revised: which implies ∇ V ( f )

Revised: =

Revised: K D [ f − f ∗ ] and lemma follows.

Revised: G. M AIN LEMMAS

Revised: Lemma G.17.

Original: Let A, B ∈ B
Revised: Let A, B ∈ B

Original: ( H , H ) be two PSD operators such that ξB − A and ξA − B are PSD for some ξ ∈ (1 , ∞ ) .
Revised: ( H , H ) be two PSD operators such that ξB − A and ξA − B are PSD for some ξ ∈ (1 , ∞ ) .

Original: Let g, h ∈ H be two arbitrary vectors and λ ∈ R ++ be a constant.
Revised: Let g, h ∈ H be two arbitrary vectors and λ ∈ R ++ be a constant.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider the following equality:
Revised: Consider the following equality:

Original: [Equation]
Revised: [Equation]

Original: which is basically the classical decomposition of the dot product (cid:104)
Revised: which is basically the classical decomposition of the dot product (cid:104)

Original: x, y [Equation]
Revised: x, y (cid:105)

Original: = [Equation] 2 + [Equation]{+12 (cid:0) (cid:107) x (cid:107) 
+}
Revised: = {+12 (cid:0) (cid:107) x (cid:107) 
+}

Original:  − [Equation]
Revised: + (cid:107) y (cid:107) 2 − (cid:107) x

Original: x − y (cid:107) 2 (cid:1) .
Revised: − y (cid:107) 2 (cid:1) .

Original: Then, we note that (cid:0) 1 − ξ − 2 (cid:1) B − [Equation] B − 
Revised: Then, we note that (cid:0) 1 − ξ − 2 (cid:1) B − (cid:0) B − ξ − 1 A (cid:1) =

Original: [Equation] − 1 A (cid:1) = ξ − 2 ( ξA − B ) is PSD by assumption, which implies:
Revised: ξ − 2 ( ξA − B ) is PSD by assumption and since (cid:0) B − ξ − 1 A (cid:1) is PSD it implies that (cid:0) 1 − ξ − 2 (cid:1) B ≥ (cid:0) B − ξ − 1 A (cid:1) , which implies:

Original: [Equation]
Revised: [Equation]

Revised: (cid:13) 2 H (cid:17) .

Original: Finally, note that
Revised: Finally, note that

Original: [Equation]
Revised: [Equation]

Original: ≤ ξ 2 − 1 .

Original: Then, the result directly follows from the following equality:
Revised: Then, the result directly follows from the following equality:

Original: [Equation]
Revised: [Equation]

Original: Denote κ ( A, B ) [Equation] H [Equation]
Revised: Denote κ ( A, B ) = (cid:107) Id H − BA − 1 (cid:107) B

Original: [Equation]
Revised: ( H , H ) =

Original: (cid:107) ( B − A 
Revised: (cid:107) ( B − A )

Original: [Equation]

Original: A − [Equation] (cid:107) B ( H 
Revised: A − 1 (cid:107) B ( H , H ) .

Original: [Equation]

Original: H

Original: [Equation]

Original: Lemma G.14.
Revised: Lemma G.18.

Original: If ξ − 1 K (cid:22) K f (cid:22) ξ K for ξ > 1 , then κ ( K , K f ) ≤ ξ − 1 .
Revised: If ξ − 1 K (cid:22) K f (cid:22) ξ K for ξ > 1 , then κ ( K , K f ) ≤ ξ − 1 .

Original: Proof.
Revised: Proof.

Original: First, we note that both operators are symmetric semi-positive deﬁnite in L 2 .
Revised: First, we note that both operators are symmetric semi-positive deﬁnite in L 2 .

Original: Now, let us look at the Rayleigh quotient:
Revised: Now, let us look at the Rayleigh quotient:

Original: [Equation]
Revised: [Equation]

Original: In the last equality we used fact that K is symmetric positive deﬁnite and therefore K 12 is too and hence we can substitute f ← K 12 f and use the explicit formula for the dot product in H via product in L 2 .
Revised: In the last equality we used fact that K is symmetric positive deﬁnite and therefore K 1 2 is too and hence we can substitute f ← K 12 f and use the explicit formula for the dot product in H via product in L 2 .

Original: Now we observe that
Revised: Now we observe that

Original: [Equation]
Revised: [Equation]

Original: which implies that the spectral radius ρ ( K − 12 ( K − K f ) K − 12 ) is bounded by ξ − 1 .
Revised: which implies that the spectral radius ρ ( K − 12 ( K − K f ) K − 12 ) is bounded by ξ − 1 .

Original: Therefore, we obtain:
Revised: Therefore, we obtain:

Original: [Equation]
Revised: [Equation]

Original: Lemma G.15.
Revised: Lemma G.19.

Original: The iterations of gradient boosting can be represented as:

Original: [Equation]

Original: where D : F → F is bounded and universal for f τ , i.e. Σ f τ = K f τ D ∀ τ .

Original: Similar decomposition holds for ∇ H V ( f, λ ) = K D [ f − f ∗ ] + λ N f . Proof. First observe that iterations are basically ∇ H ( K fτ ) V ( f τ , λ ) assuming that we replace H with
Revised: Let

Original: H ( K f τ ) in regularizer term. Therefore, we need to prove only ∇ H V ( f, λ ) = K D [ f − f ∗ ]+ λN f . To prove this observe that there is deﬁned linear
Revised: A

Original: Frechet differential D f V ( f )[ h ] = 1 N (cid:80) Ni =1 (cid:0) f ( x i ) − f ∗ ( x i ) (cid:1) h ( x i ) : F → R which is bilinear functional by f − f ∗ and h .
Revised: [Equation]

Original: Since F is ﬁnite dimensional then there is linear representation of the mapping in L 2 product restricted on F , let us denote it

Original: D : F → F . Now we have D f V ( f ) [ h ] = (cid:104) D [ f − f ∗ ] , h (cid:105)
Revised: B ∈ B ( H , H ) .

Original: L 2 .

Original: Now we transform it to D f V ( f )

Original: [ h ] = (cid:104)K − 1 K D [ f − f ∗ ] , h (cid:105)

Original: L 2 = (cid:104)K D [ f − f ∗ ] , h (cid:105) H which implies that ∇ H V ( f ) =

Original: K D [ f − f ∗ ] .

Original: Lemma G.16.

Original: Consider the setting of Lemma G.13 without the requirement B − A being PSD.

Original: Then the inequality holds:
Revised: Then the inequality holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us rewrite the left part as
Revised: Let us rewrite the left part as

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: Then we use the identity (cid:104) a, b (cid:105) =

Original: [Equation] [Table] [Equation] [Table]
Revised: 12 (cid:0) (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 − (cid:107) a − b (cid:107) 2 (cid:1) .

Original: [Equation] [Table] [Equation] [Table]
Revised: Also we use

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: to obtain

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: 2 H

Original: [Table]
Revised: Using (cid:107) a + b (cid:107) 2 ≤ 2( (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 ) we proceed

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: The following lemma holds.
Revised: The following lemma holds.

Original: Lemma G.17.
Revised: Lemma G.20.

Original: If ( λN + 1) (cid:15) < 1 and f 0 = 0 H , then ∀ τ the following holds almost surely
Revised: If ( λN + 1) (cid:15) < 1 and f 0 = 0 H , then ∀ τ the following holds almost surely

Original: [Equation]
Revised: [Equation]

Original: for norms (cid:107) · (cid:107) L 2 , (cid:107) · (cid:107) H , and (cid:107) · (cid:107) R N .
Revised: for norms (cid:107) · (cid:107) L 2 , (cid:107) · (cid:107) H , and (cid:107) · (cid:107) R N .

Original: Proof.
Revised: Proof.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Now observe that S :
Revised: Now observe that S :

Original: = (cid:16) Id L 2 − λ(cid:15)N
Revised: = (cid:16) Id L 2 − λ(cid:15)N

Original: Id L 2 − (cid:15) Σ
Revised: Id L 2 − (cid:15) Σ

Original: ν τ (cid:17) is symmetric with eigenvalues 0 <
Revised: ν τ (cid:17) is symmetric with eigenvalues 0 <

Original: λ i ( S ) ≤ 1 − λ(cid:15)N , therefore its operator norm in B ( L 2 ) , B ( H ) , and R N × N is less than 1 − λ(cid:15)N .
Revised: λ i ( S ) ≤ 1 − λ(cid:15)N , therefore its operator norm in B ( L 2 ) , B ( H ) , and R N × N is less than 1 − λ(cid:15)N .

Original: Taking the norm of left and right sides and using the sub-additivity of the norm, we obtain:
Revised: Taking the norm of left and right sides and using the sub-additivity of the norm, we obtain:

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) f 0 − f ∗ (cid:107) =
Revised: Since (cid:107) f 0 − f ∗ (cid:107) =

Original: (cid:107) f ∗ (cid:107) that recurrent relation inductively yields the statement of the lemma.
Revised: (cid:107) f ∗ (cid:107) that recurrent relation inductively yields the statement of the lemma.

Original: Corollary G.18.
Revised: Corollary G.21.

Original: Under the same conditions, (cid:107) f τ (cid:107) ≤ 2 (cid:107) f ∗ (cid:107) and V ( f τ , 0) ≤ R 2 almost surely.
Revised: Under the same conditions, (cid:107) f τ (cid:107) ≤ 2 (cid:107) f ∗ (cid:107) .

Original: G. M AIN THEOREM
Revised: G. M AIN THEOREMS

Original: Let us denote R1 = (cid:107) f ∗ (cid:107) H .
Revised: Let us denote R = (cid:107) f ∗ (cid:107) R N .

Original: We argue that it is a constant value since the kernel H and f ∗ are convergent as N → ∞ which makes it bounded by some constant with probability arbitrary close to one.
Revised: We argue that it is a constant value since the kernel H and f ∗ are convergent as N → ∞ which makes it bounded by some constant with probability arbitrary close to one.

Revised: By lemma F.2 Γ β ( f ) ≤ e 2 mV ( f ) β .

Revised: Then Γ β ( f τ ) ≤ e 2 m 12 N (cid:107) fτ − f ∗(cid:107) 2 R N β ≤ e mR 2 Nβ

Original: Moreover, denote M β = e2 mR 2 β > 1 .
Revised: and we denote M β = e mR 2 Nβ > 1 .

Original: Theorem G.19.
Revised: Theorem G.22.

Original: Consider an arbitrary (cid:15) , 0 < (cid:15) ( λN + 1) < 1 and (1+ M β λ ) 4 M β N ≥ (cid:15) .
Revised: Consider an arbitrary (cid:15) , 0 < (cid:15) ( λ N + 1) < 1 and (1+ M β λ ) 4 M β N ≥ (cid:15) .

Original: The following inequality holds:
Revised: The following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To prove it we will consider bounding V ( f ) ≤ V ( f, M β λ ) + constwith M β = e 2 mR 2 β > 0 .
Revised: To prove it we will consider bounding V ( f ) ≤ V ( f, M β λ ) + const .

Original: The reason is that it will allow us to invoke Lemma G.13.
Revised: The reason is that it will allow us to invoke Lemma G.17.

Original: After that by using strong convexity we obtain bound on EV ( f τ , M β ) and then bound on EV ( f τ ) will follow straightforwardly.
Revised: After that by using strong convexity we obtain bound on EV ( f τ , M β ) and then bound on EV ( f τ ) will follow straightforwardly

Original: To produce result for V ( f τ , M β λ ) we consider expanding V ( f τ +1 , M β λ ) by substituting formula for f τ +1 and ﬁrst dealing with the term V ( f τ +1 ) :

Original: [Equation]
Revised: [Equation]

Original: where in the last we used 12 D [Equation] V 
Revised: To produce result for V ( f τ , M β λ ) we consider expanding V ( f τ +1 , M β λ ) by substituting formula for f τ +1 and ﬁrst dealing with the term V ( f τ +1 ) :

Original: [Equation]
Revised: [Equation]

Original: N ) (cid:107) 2 R N by using inequality (cid:107)a + b (cid:107) 2 ≤ [Equation] + [Equation]
Revised: where in the last we used inequality (cid:107)

Original: b

Original: [Equation]
Revised: [Equation]

Original: 2 and lemma G.6 which allows to bound (cid:107) Σ [Equation] (cid:107) R N ≤(cid:107)
Revised: 2 and lemma G.7 which allows to bound (cid:107) Σ ν τ ( f τ − f ∗ ) (cid:107) R N ≤

Original: [Equation]
Revised: [Equation]

Original:  f [Equation] − f ∗ ) (cid:107) R N .
Revised: ( f τ − f ∗ ) (cid:107) R N .

Original: Then, we analyze the regularization:
Revised: Then, we analyze the regularization:

Original: [Equation]
Revised: [Equation]

Original: where in  ﬁrst inequality we used (cid:107) Σ ν τ [ fτ − f ∗ ] (cid:107) H ≤ (cid:107)
Revised: where in the ﬁrst inequality we used (cid:107) Σ ν τ [ f

Original: [Equation]
Revised: [Equation]

Original: − f ∗  (cid:107) H which is due to lemma G.7 Summing up expectations of those two expressions, using G.10 we obtain: 
Revised: − f ∗ ] (cid:107) H ≤ (cid:107) f τ − f ∗ (cid:107) H which is due to lemma G.7 Summing up expectations of those two expressions we obtain: [Equation] Here we used C λ

Original: [Equation]
Revised: [Equation]

Original:  Then by applying Lemma G.15 for Σ f τ = K f τ D and then by applying Lemma G.13 with ξ = M β , A = K , B = K f τ , g = D
Revised: 2 H ≤ λ 2 N R 2 . Then by applying Lemma G.16 for Σ f τ = K f τ D and then by applying Lemma G.17 with ξ = M β , A = K , B = K f τ , g = D

Original: [ f τ − f ∗ ] and h = f τ , we obtain the following bound:
Revised: [ f τ − f ∗ ] and h = f τ , we obtain the following bound:

Original: [Equation]
Revised: [Equation]

Original: Then by using Polyak-Łojasiewicz inequality 12 (cid:107)∇ V (cid:107) H ≥ µV for µ -strongly convex function V (restricted on image of Σ ) with µ ≥ 1+ M β λ N >
Revised: Then by using Polyak-Łojasiewicz inequality 12 (cid:107)∇ V (cid:107) H ≥ µV for µ -strongly convex function V (restricted on span {K ( · , x i ) | i = 1 , . .

Original: [Equation] which is due to corollary G. [Equation] we obtain:
Revised: . , N } ) with µ ≥ 1+ M β λ N > 0 , which is due to corollary G.8, we obtain:

Original: [Equation]
Revised: [Equation]

Original: Substituting it into bound on V ( f τ +1 , M β λ ) gives:
Revised: Substituting it into bound on V ( f τ +1 , M β λ ) gives:

Original: [Equation]
Revised: [Equation]

Original: Which yields
Revised: Which yields

Original: [Equation]
Revised: [Equation]

Revised: where we used bound V ( f 0 , M β λ )

Revised: =

Revised: V (0 , M β λ ) =

Revised: 12 N (cid:107) f ∗ ( x N ) (cid:107) 2 R N − C M β λ ≤

Revised: R

Original:  Next we use the following inequality:
Revised: N . Next we use the following inequality:

Original: [Equation]
Revised: [Equation]

Original: Which ﬁnally gives us bound on E V ( f T ) :
Revised: Which ﬁnally gives us bound on E V ( f T ) :

Original: [Equation]
Revised: [Equation]

Original: Theorem G.20.
Revised: Theorem G.23.

Original: (Theorem 3.10 in the main text) Let C
Revised: (Theorem 3.8 in the main text) Let C

Original: =8 λ 2 (cid:15)M β [Equation] [Equation]
Revised: =

Original: M[Equation] M β 
Revised: M β λ (cid:16) 12 N +

Original: [Equation]
Revised: [Equation]

Original: β [Equation] 2 + 8 λ 2 N 2 ) + [Equation]
Revised: R 2 .

Original: M 2 β − 1) λN (cid:1) + λN R 21 (cid:17) R 21 . Assume that 0 < (cid:15) ( λN + 1) < 1 and (1+ M β λ ) 4 M β N ≥ (cid:15) , (1+ λ ) 8 N ≥ (cid:15) 
Revised: Assume that 0 < (cid:15) ( λN + 1) < 1 and (1+ M β λ ) 4 M β N ≥ (cid:15) , (1+ λ ) 8 N ≥ (cid:15) , e 4 mCβ ≤ 54 10 and deﬁne T 1 = (cid:104)

Original: [Equation] N + λN 2 mM β C β < 1+ λ 2 N and deﬁne T 1 = (cid:104) 2 M β N  (1+ M β λ )) (cid:15) log R 2 C (cid:105) + 1 .
Revised: 2 M β N (cid:15) (1+ M β λ ) log R 2 2 CN (cid:105) + 1 .

Original: Then ∀ T ≥ T 1 it holds that
Revised: Then ∀ T ≥ T 1 it holds that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: First we apply previous theorem to obtain bound on V ( f τ ) which we will use to claim that kernels K f τ and K are close to each other in expectation.
Revised: First we apply previous theorem to obtain bound on V ( f τ ) which we will use to claim that kernels K f τ and K are close to each other in expectation.

Original: If we take
Revised: If we take

Original: [Equation]
Revised: [Equation]

Original: then the following inequality holds ∀ τ ≥ T 1 :
Revised: then the following inequality holds ∀ τ ≥ T 1 :

Original: [Equation]
Revised: [Equation]

Original: Overall, the proof is similar to the previous theorembut with some modiﬁcations. Instead of Lemma G.13, we will use Lemma G.16. The only problem would arise in the multiplier E (cid:16) 12 − e
Revised: Then analogously with previous theorem we estimate:

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: Further we bound − (cid:15) E (cid:104)K f τ D

Original: [Equation] H V ( f τ , λ ) (cid:107) 2 H .
Revised: [ f τ − f ∗ ] + λN f τ , K D [ f τ − f ∗ ] + λN f τ (cid:105) by Lemma G.19, instead of

Original: We can bound it as follows
Revised: Lemma G.17, which we used in previous theorem:

Original: [Equation]
Revised: [Equation]

Original: After using ( e z − 1) 2 ≤ e 2 z − 1 ≤ 2 M 2 β z for z = 2 mV ( f ) β satisfying e z ≤ M β , we obtain the bound:
Revised: Substituting this in the formula we yield:

Original: [Equation]
Revised: [Equation]

Original: Iterating the bound yields
Revised: Iterating the bound yields

Original: [Equation]
Revised: [Equation]

Original: Corollary G.21.
Revised: Corollary G.24.

Original: (Convergence to the solution of the KRR / Convergence to the Gaussian Process posterior mean function).
Revised: (Convergence to the solution of the KRR / Convergence to the Gaussian Process posterior mean function).

Original: Under the assumptions of both previous theorems we have that:
Revised: Under the assumptions of both previous theorems we have that:

Original: [Equation]
Revised: [Equation]

Revised: Proof.

Revised: By lemma D.7 V ( f, λ ) =

Revised: 12 N (cid:107) f ( x N ) − f λ ∗ ( x N ) (cid:107) 2 R N + λ 2 N (cid:107) f − f λ ∗ (cid:107) 2 H .

Original: {+Then by previous theorem we get bound on 12 N E (cid:107) f T ( x N ) − f λ ∗ ( x N ) (cid:107) 2 R N and by lemmas G.10 and G.
+}
Revised: {+Then by previous theorem we get bound on 12 N E (cid:107) f T ( x N ) − f λ ∗ ( x N ) (cid:107) 2 R N and by lemmas G.10 and G.
+}

Revised: we majorize our L 2 norm by R N norm.

Revised: Lemma then follows.

Original: Lemma G.22 (Lemma 4.1in the main text) .
Revised: Lemma G.25 (Lemma 4.1in the main text) .

Original: The following convergence holds almost surely in x ∈
Revised: The following convergence holds almost surely in x ∈

Original: X :
Revised: X :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From (5), we have that the covariance of h T is K independently from T .
Revised: From (5), we have that the covariance of h T is K independently from T .

Original: Thus, it remains to show that the limit is Gaussian almost surely which essentially holds due to the central limit theorem almost surely in x ∈ X :
Revised: Thus, it remains to show that the limit is Gaussian almost surely which essentially holds due to the central limit theorem almost surely in x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: where each individual tree h T,i is centered i.i.d.
Revised: where each individual tree h T,i is centered i.i.d.

Original: (with the same distribution as h 1 ).
Revised: (with the same distribution as h 1 ).

Original: H I MPLEMENTATION DETAILS
Revised: H I MPLEMENTATION DETAILS

Original: In the experiments, we ﬁx σ = 10 − 2 (scale of the kernel) and δ = 10 − 4 (scale of noise), which theoretically can be taken arbitrarily.
Revised: In the experiments, we ﬁx σ = 10 − 2 (scale of the kernel) and δ = 10 − 4 (scale of noise), which theoretically can be taken arbitrarily.

Original: As a hyperparameter (that is estimated on the validation set), we consider β ∈ { 10 − 2 , 10 − 1 , 1 } .
Revised: As a hyperparameter (that is estimated on the validation set), we consider β ∈ { 10 − 2 , 10 − 1 , 1 } .

Original: We use the standard CatBoost library and add the Gumbel noise term in selecting the trees for the “L2” scoring function, which is implemented in CatBoost out of the box but is not used by SGB and SGLB since it is not the default one for the library.
Revised: We use the standard CatBoost library and add the Gumbel noise term in selecting the trees for the “L2” scoring function, which is implemented in CatBoost out of the box but is not used by SGB and SGLB since it is not the default one for the library.

Original: Moreover, we do not consider subsampling of the data (as SGLB does also), and differently from SGB and
Revised: Moreover, we do not consider subsampling of the data (as SGLB does also), and differently from SGB and

Original: SGLB, we disable the “boost-from-average” option.
Revised: SGLB, we disable the “boost-from-average” option.

Original: Finally, we set l 2 − leaf − reg value to 0 , as
Revised: Finally, we set l 2 − leaf − reg value to 0 , as

Original: SGLB does.
Revised: SGLB does.