Original: G RADIENT B OOSTING P ERFORMS G AUSSIAN P ROCESS I NFERENCE
Revised: G RADIENT B OOSTING P ERFORMS G FAUSSIAN P ROCESS I NFERENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper shows that gradient boosting based on symmetric decision trees can be equivalently reformulated as a kernel method that converges to the solution of a certain Kernel Ridge Regression problem.
Revised: This paper shows that gradient boosting based on symmetric decision trees can be equivalently reformulated as a kernel method that converges to the solution of a certain Kernel Ridge Regression problem.

Original: Thus, we obtain the convergence to a
Revised: Thus, we obtain the convergence to a

Original: Gaussian Process’ posterior mean, which, in turn, allows us to easily transform gradient boosting into a sampler from the posterior to provide better knowledge uncertainty estimates through Monte-Carlo estimation of the posterior variance.
Revised: Gaussian Process’ posterior mean, which, in turn, allows us to easily transform gradient boosting into a sampler from the posterior to provide better knowledge uncertainty estimates through Monte-Carlo estimation of the posterior variance.

Original: We show that the proposed sampler allows for better knowledge uncertainty estimates leading to improved out-of-domain detection
Revised: We show that the proposed sampler allows for better knowledge uncertainty estimates leading to improved out-of-domain detection

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Gradient boosting (Friedman, 2001) is a classic machine learning algorithm successfully used for web search, recommendation systems, weather forecasting, and other problems (Roe et al., 2005;
Revised: Gradient boosting (Friedman, 2001) is a classic machine learning algorithm successfully used for web search, recommendation systems, weather forecasting, and other problems (Roe et al., 2005;

Original: Caruana & Niculescu-Mizil, 2006; Richardson et al., 2007; Wu et al., 2010; Burges, 2010; Zhang &
Revised: Caruana & Niculescu-Mizil, 2006; Richardson et al., 2007; Wu et al., 2010; Burges, 2010; Zhang &

Original: Haghani, 2015).
Revised: Haghani, 2015).

Original: In a nutshell, gradient boosting methods iteratively combine simple models (usually decision trees), minimizing a given loss function.
Revised: In a nutshell, gradient boosting methods iteratively combine simple models (usually decision trees), minimizing a given loss function.

Original: Despite the recent success of neural approaches in various areas, gradient-boosted decision trees (GBDT) are still state-of-the-art algorithms for tabular datasets containing heterogeneous features (Gorishniy et al., 2021; Katzir et al., 2021).
Revised: Despite the recent success of neural approaches in various areas, gradient-boosted decision trees (GBDT) are still state-of-the-art algorithms for tabular datasets containing heterogeneous features (Gorishniy et al., 2021; Katzir et al., 2021).

Original: This paper aims at a better theoretical understanding of GBDT methods for regression problems assuming the widely used RMSE loss function.
Revised: This paper aims at a better theoretical understanding of GBDT methods for regression problems assuming the widely used RMSE loss function.

Original: First, we show that the gradient boosting with regularization can be reformulated as an optimization problem in some Reproducing Kernel Hilbert
Revised: First, we show that the gradient boosting with regularization can be reformulated as an optimization problem in some Reproducing Kernel Hilbert

Original: Space (RKHS) with implicitly deﬁned kernel structure.
Revised: Space (RKHS) with implicitly deﬁned kernel structure.

Original: After obtaining that connection between
Revised: After obtaining that connection between

Original: GBDT and kernel methods, we introduce a technique for sampling from prior Gaussian process distribution with the same kernel that deﬁnes RKHS so that the ﬁnal output would converge to a sample from the Gaussian process posterior.
Revised: GBDT and kernel methods, we introduce a technique for sampling from prior Gaussian process distribution with the same kernel that deﬁnes RKHS so that the ﬁnal output would converge to a sample from the Gaussian process posterior.

Original: Without this technique, we can view the output of
Revised: Without this technique, we can view the output of

Original: GBDT as the mean function of the Gaussian process.
Revised: GBDT as the mean function of the Gaussian process.

Original: Importantly, our theoretical analysis assumes the regularized gradient boosting procedure (Algorithm 2) without any simpliﬁcations — we only need decision trees to be symmetric (oblivious) and properly randomized (Algorithm 1).
Revised: Importantly, our theoretical analysis assumes the regularized gradient boosting procedure (Algorithm 2) without any simpliﬁcations — we only need decision trees to be symmetric (oblivious) and properly randomized (Algorithm 1).

Original: These assumptions are non-restrictive and are satisﬁed in some popular gradient boosting implementations, e.g., CatBoost (Prokhorenkova et al., 2018).
Revised: These assumptions are non-restrictive and are satisﬁed in some popular gradient boosting implementations, e.g., CatBoost (Prokhorenkova et al., 2018).

Original: Our experiments conﬁrm that the proposed sampler from the Gaussian process posterior outperforms the previous approaches (Malinin et al., 2021) and gives better knowledge uncertainty estimates and improved out-of-domain detection.
Revised: Our experiments conﬁrm that the proposed sampler from the Gaussian process posterior outperforms the previous approaches (Malinin et al., 2021) and gives better knowledge uncertainty estimates and improved out-of-domain detection.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Assume that we are given a distribution (cid:37) over X × Y , where X ⊂ R d is called a feature space and
Revised: Assume that we are given a distribution (cid:37) over X × Y , where X ⊂ R d is called a feature space and

Original: Y ⊂ R — a target space.
Revised: Y ⊂ R — a target space.

Original: Further assume that we are given a dataset z = { ( x i , y i ) } N i =1 ⊂ X × Y of size N ≥ 1 sampled i.i.d.
Revised: Further assume that we are given a dataset z = { ( x i , y i ) } N i =1 ⊂ X × Y of size N ≥ 1 sampled i.i.d.

Original: from (cid:37) .
Revised: from (cid:37) .

Original: Let us denote by ρ (d x )
Revised: Let us denote by ρ (d x )

Original: = (cid:82) Y d (cid:37) (d x, d y ) .
Revised: = (cid:82) Y d (cid:37) (d x, d y ) .

Original: W.l.o.g.
Revised: W.l.o.g.

Original: , we also assume that X = supp ρ = (cid:8) x ∈ R d : ∀ (cid:15) > 0 , ρ ( { x (cid:48)
Revised: , we also assume that X = supp ρ = (cid:8) x ∈ R d : ∀ (cid:15) > 0 , ρ ( { x (cid:48)

Original: ∈ R
Revised: ∈ R

Original: d : (cid:107) x − x (cid:48) (cid:107) R d
Revised: d : (cid:107) x − x (cid:48) (cid:107) R d

Original: <
Revised: <

Original: (cid:15) } ) > 0 (cid:9) which is a closed subset of R d .
Revised: (cid:15) } ) > 0 (cid:9) which is a closed subset of R d .

Original: Moreover, for technical reasons, we assume that 12 N (cid:80) Ni =

Revised: Moreover, for technical reasons, we assume that 12 N (cid:80) Ni =

Original: y 2 i ≤ R 2 for some constant R > 0 almost surely, which can always be enforced by clipping.
Revised: y 2 i ≤ R 2 for some constant R > 0 almost surely, which can always be enforced by clipping.

Original: Throughout the paper, we also denote by x N and y N the matrix of all feature vectors and the vector of targets.
Revised: Throughout the paper, we also denote by x N and y N the matrix of all feature vectors and the vector of targets.

Original: 2. G RADIENT BOOSTED DECISION TREES
Revised: 2. G RADIENT BOOSTED DECISION TREES

Original: Given a loss function L : R 2 → R , a classic gradient boosting algorithm (Friedman, 2001) iteratively combines weak learners (usually decision trees) to reduce the average loss over the train set z :
Revised: Given a loss function L : R 2 → R , a classic gradient boosting algorithm (Friedman, 2001) iteratively combines weak learners (usually decision trees) to reduce the average loss over the train set z :

Original: L ( f ) =
Revised: L ( f ) =

Original: E z
Revised: E z

Original: [ L ( f ( x ) , y )] .
Revised: [ L ( f ( x ) , y )] .

Original: At each iteration τ , the model is updated as: f τ ( x ) =
Revised: At each iteration τ , the model is updated as: f τ ( x ) =

Original: f
Revised: f

Original: τ − 1 ( x ) + (cid:15)w τ ( x ) , where w τ ( · ) ∈ W is a weak learner chosen from some family of functions W , and (cid:15) is a learning rate.
Revised: τ − 1 ( x ) + (cid:15)w τ ( x ) , where w τ ( · ) ∈ W is a weak learner chosen from some family of functions W , and (cid:15) is a learning rate.

Original: The weak learner w τ is usually chosen to approximate the negative gradient of the loss function
Revised: The weak learner w τ is usually chosen to approximate the negative gradient of the loss function

Original: [Equation]
Revised: [Equation]

Original: The family W usually consists of decision trees.
Revised: The family W usually consists of decision trees.

Original: In this case, the algorithm is called GBDT (Gradient
Revised: In this case, the algorithm is called GBDT (Gradient

Original: Boosted Decision Trees).
Revised: Boosted Decision Trees).

Original: A decision tree is a model that recursively partitions the feature space into disjoint regions called leaves.
Revised: A decision tree is a model that recursively partitions the feature space into disjoint regions called leaves.

Original: Each leaf R j of the tree is assigned to a value, which is the estimated response y in the corresponding region.
Revised: Each leaf R j of the tree is assigned to a value, which is the estimated response y in the corresponding region.

Original: Thus, we can write w ( x ) =
Revised: Thus, we can write w ( x ) =

Original: (cid:80) dj
Revised: (cid:80) dj

Original: =
Revised: =

Original: θ j 1 { x ∈ R j } , so the decision tree is a linear function of the leaf values θ j .
Revised: θ j 1 { x ∈ R j } , so the decision tree is a linear function of the leaf values θ j .

Original: A recent paper Ustimenko & Prokhorenkova (2021) proposes a modiﬁcation of classic stochastic gradient boosting (SGB) called
Revised: A recent paper Ustimenko & Prokhorenkova (2021) proposes a modiﬁcation of classic stochastic gradient boosting (SGB) called

Original: Stochastic Gradient Langevin Boosting (SGLB).
Revised: Stochastic Gradient Langevin Boosting (SGLB).

Original: SGLB combines gradient boosting with stochastic gradient Langevin dynamics to achieve global convergence even for non-convex loss functions.
Revised: SGLB combines gradient boosting with stochastic gradient Langevin dynamics to achieve global convergence even for non-convex loss functions.

Original: The algorithm has two modiﬁcations compared to the classic boosting.
Revised: The algorithm has two modiﬁcations compared to the classic boosting.

Original: First, Gaussian noise (parametrized by a temperature parameter) is injected into the gradients, and the temperature regulates the level of exploration.
Revised: First, Gaussian noise (parametrized by a temperature parameter) is injected into the gradients, and the temperature regulates the level of exploration.

Original: Second, shrinkage is added to the boosting process: at each iteration, the currently built model is multiplied by a constant smaller than one.
Revised: Second, shrinkage is added to the boosting process: at each iteration, the currently built model is multiplied by a constant smaller than one.

Original: As a result, the obtained algorithm provably converges to some stationary distribution (invariant measure) concentrated near the global optimum of the loss function.
Revised: As a result, the obtained algorithm provably converges to some stationary distribution (invariant measure) concentrated near the global optimum of the loss function.

Original: We mention this method because it samples from similar distribution as our method but with a different kernel.
Revised: We mention this method because it samples from similar distribution as our method but with a different kernel.

Original: 2. E STIMATING UNCERTAINTY
Revised: 2. E STIMATING UNCERTAINTY

Original: In addition to the predictive quality, it is often important to detect when the system is uncertain and can be mistaken.
Revised: In addition to the predictive quality, it is often important to detect when the system is uncertain and can be mistaken.

Original: For this, different measures of uncertainty can be used.
Revised: For this, different measures of uncertainty can be used.

Original: There are two main sources of uncertainty: data uncertainty (a.k.a.
Revised: There are two main sources of uncertainty: data uncertainty (a.k.a.

Original: aleatoric uncertainty) and knowledge uncertainty (a.k.a.
Revised: aleatoric uncertainty) and knowledge uncertainty (a.k.a.

Original: epistemic uncertainty).
Revised: epistemic uncertainty).

Original: Data uncertainty arises due to the inherent complexity of the data, such as additive noise or overlapping classes.
Revised: Data uncertainty arises due to the inherent complexity of the data, such as additive noise or overlapping classes.

Original: For instance, if the target is distributed as y | x ∼
Revised: For instance, if the target is distributed as y | x ∼

Original: N ( f ( x ) , σ 2 ( x ) , then σ ( x ) reﬂects the level of data uncertainty.
Revised: N ( f ( x ) , σ 2 ( x ) , then σ ( x ) reﬂects the level of data uncertainty.

Original: This uncertainty can be assessed if the model is probabilistic.
Revised: This uncertainty can be assessed if the model is probabilistic.

Original: Knowledge uncertainty arises when the model gets input from a region either sparsely covered by or far from the training data.
Revised: Knowledge uncertainty arises when the model gets input from a region either sparsely covered by or far from the training data.

Original: Since the model does not have enough data in this region, it will likely make a mistake.
Revised: Since the model does not have enough data in this region, it will likely make a mistake.

Original: A standard approach to estimating knowledge uncertainty is based on ensembles (Gal, 2016; Malinin, 2019).
Revised: A standard approach to estimating knowledge uncertainty is based on ensembles (Gal, 2016; Malinin, 2019).

Original: Assume that we have trained an ensemble of several independent models.
Revised: Assume that we have trained an ensemble of several independent models.

Original: If all the models understand an input (low knowledge uncertainty), they will give similar predictions.
Revised: If all the models understand an input (low knowledge uncertainty), they will give similar predictions.

Original: However, for out-of-domain examples (high knowledge uncertainty), the models are likely to provide diverse predictions.
Revised: However, for out-of-domain examples (high knowledge uncertainty), the models are likely to provide diverse predictions.

Original: For regression tasks, one can obtain knowledge uncertainty by measuring the variance of the predictions provided by multiple models (Malinin, 2019).
Revised: For regression tasks, one can obtain knowledge uncertainty by measuring the variance of the predictions provided by multiple models (Malinin, 2019).

Original: Such ensemble-based approaches are standard for neural networks (Lakshminarayanan et al., 2017).
Revised: Such ensemble-based approaches are standard for neural networks (Lakshminarayanan et al., 2017).

Original: Recently, ensembles were also tested for GBDT models (Malinin et al., 2021).
Revised: Recently, ensembles were also tested for GBDT models (Malinin et al., 2021).

Original: The authors consider two ways of generating ensembles: ensembles of independent SGB models and ensembles of independent SGLB models.
Revised: The authors consider two ways of generating ensembles: ensembles of independent SGB models and ensembles of independent SGLB models.

Original: While empirically methods are very similar, SGLB has better theoretical properties: the convergence of parameters to the stationary distribution allows one to sample models from a particular posterior distribution.
Revised: While empirically methods are very similar, SGLB has better theoretical properties: the convergence of parameters to the stationary distribution allows one to sample models from a particular posterior distribution.

Revised: One drawback of SGLB is that its convergence rate is unknown, as the proof is asymptotic.

Revised: However, the convergence rate can be upper bounded by that of

Revised: Stochastic Gradient Langevin Dynamics for log-concave functions, e.g., (Zou et al., 2020), which is not dimension-free.

Revised: In contrast, our rate is dimension-free and scales linearly with inverse precision.

Original: 2. G AUSSIAN PROCESS INFERENCE
Revised: 2. G AUSSIAN PROCESS INFERENCE

Original: In this section, we brieﬂy describe the basics of Bayesian inference with Gaussian processes that is closely related to our analysis.
Revised: In this section, we brieﬂy describe the basics of Bayesian inference with Gaussian processes that is closely related to our analysis.

Original: A random variable f with values in L 2 ( ρ ) is said to be a Gaussian process f ∼ GP (cid:0) f 0 , σ 2 K + δ 2 Id L 2 (cid:1) with covariance deﬁned via a kernel K ( x, x (cid:48) )
Revised: A random variable f with values in L 2 ( ρ ) is said to be a Gaussian pro- cess f ∼ GP (cid:0) f 0 , σ 2 K + δ 2 Id L 2 (cid:1) with covariance deﬁned via a kernel K ( x, x (cid:48) )

Original: =
Revised: =

Original: cov (cid:0) f
Revised: cov (cid:0) f

Original: ( x ) , f ( x (cid:48) ) (cid:1) and mean value f 0 ∈
Revised: ( x ) , f ( x (cid:48) ) (cid:1) and mean value f 0 ∈

Original: L 2 ( ρ ) iff ∀ g ∈
Revised: L 2 ( ρ ) iff ∀ g ∈

Original: L 2 ( ρ ) we have 
Revised: L 2 ( ρ ) we have (cid:90)

Original: [Equation]
Revised: [Equation]

Original: A typical Gaussian Process setup is to assume that the target y | x is distributed as GP (cid:0) 

Revised: A typical Gaussian Process setup is to assume that the target y | x is distributed as GP (cid:0)

Original: L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 (cid:1) for some kernel function 1 K ( x, x 
Revised: L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 (cid:1) for some kernel function 1 K ( x, x (cid:48) ) with scales σ > 0 and δ > 0 :

Original: [Equation]
Revised: [Equation]

Original: ) with scales σ > 0 and δ > 0 :

Original: [Equation]
Revised: [Equation]

Original: The posterior distribution f ( x ) | x, x N , y N is again a Gaussian Process GP ( f ∗ , σ 2 (cid:101) K
Revised: The posterior distribution f ( x ) | x, x N , y N is again a Gaussian Process GP ( f ∗ , σ 2 (cid:101) K

Original: + δ 2 Id L 2 ) with mean and covariance given by (see Rasmussen & Williams (2006)):
Revised: + δ 2 Id L 2 ) with mean and covariance given by (see Rasmussen & Williams (2006)):

Original: [Equation]
Revised: [Equation]

Original:  To estimate the posterior mean f  ∗ ( x ) = (cid:82) R fp ( f | x, x N , y N ) d f , we can use the maximum a posteriori probability estimate (MAP) — the minimal norm solution for the following KernelRidge
Revised: with λ = δ 2 σ 2 . To estimate the posterior mean f λ ∗ ( x ) = (cid:82) R fp ( f | x, x N , y N ) d f , we can use the maximum a posteriori probability estimate (MAP) — the only solution for the following Kernel

Original:  Regression (KRR) problem:
Revised: Ridge Regression (KRR) problem:

Original: [Equation]
Revised: [Equation]

Original: where H = span (cid:8) K ( · , x ) (cid:12)(cid:12) x ∈ X (cid:9) ⊂
Revised: where H = span (cid:8) K ( · , x ) (cid:12)(cid:12) x ∈ X (cid:9) ⊂

Original: L 2 ( ρ ) is the reproducing kernel Hilbert space (RKHS) for the kernel K ( · , · ) and λ = δ 2 σ 2 .
Revised: L 2 ( ρ ) is the reproducing kernel Hilbert space (RKHS) for the kernel K ( · , · ) and L ( f ) → min f ∈H means that we are seeking minimizers of this function L ( f ) .

Original: We refer to Appendix D for the details on KRR and RKHS.
Revised: We refer to Appendix D for the details on KRR and RKHS.

Original: To solve the KRR problem, one can apply the gradient descent (GD) method in the functional space:
Revised: To solve the KRR problem, one can apply the gradient descent (GD) method in the functional space:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) > 0 is a learning rate.
Revised: where (cid:15) > 0 is a learning rate.

Original: Since this objective is strongly convex due to the regularization, the gradient descent rapidly converges to the unique optimum:
Revised: Since this objective is strongly convex due to the regularization, the gradient descent rapidly converges to the unique optimum:

Original: [Equation]
Revised: [Equation]

Original: see Appendices C and E for the details.
Revised: see Appendices C and E for the details.

Original: Gradient descent guides f τ to the posterior mean f  ∗ of the Gaussian Process with kernel σ 2 K + δ 2 Id L 2 .
Revised: Gradient descent guides f τ to the posterior mean f λ ∗ of the Gaussian Process with kernel σ 2 K + δ 2 Id L 2 .

Original: To obtain the posterior variance estimate (cid:101) K ( x, x ) for any x , one can use sampling and introduce a source of randomness in the above iterative scheme as follows:
Revised: To obtain the posterior variance estimate (cid:101) K ( x, x ) for any x , one can use sampling and introduce a source of randomness in the above iterative scheme as follows:

Original: sample f init ∼ GP ( 0 L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 ); 2.
Revised: sample f init ∼ GP ( 0 L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 ); 2.

Original: set new labels y
Revised: set new labels y

Original: newN =
Revised: newN =

Original: y N − f init ( x N ); 3.
Revised: y N − f init ( x N ); 3.

Original: ﬁt GD f τ ( · ) on y
Revised: ﬁt GD f τ ( · ) on y

Original: newN assuming F 0 ( · ) =
Revised: newN assuming F 0 ( · ) =

Original: L 2 ( ρ ) ; 4.
Revised: L 2 ( ρ ) ; 4.

Original: output f init ( · ) + f τ ( · ) as ﬁnal model .
Revised: output f init ( · ) + f τ ( · ) as ﬁnal model .

Original: This method is known as Sample-then-Optimize (Matthews et al., 2017) and is widely adopted for
Revised: This method is known as Sample-then-Optimize (Matthews et al., 2017) and is widely adopted for

Original: Bayesian inference.
Revised: Bayesian inference.

Original: As τ → ∞ , we get f init + f ∞ distributed as a Gaussian Process posterior with the desired mean and variance.
Revised: As τ → ∞ , we get f init + f ∞ distributed as a Gaussian Process posterior with the desired mean and variance.

Original: Formally, the following result holds.
Revised: Formally, the following result holds:

Original: Lemma 2.1.
Revised: Lemma 2.1.

Original: f init + f ∞ follows the Gaussian Process posterior GP ( f λ ∗ , σ 2 (cid:101) K
Revised: f init + f ∞ follows the Gaussian Process posterior GP ( f λ ∗ , σ 2 (cid:101) K

Original: + δ 2 Id L 2 ) with:
Revised: + δ 2 Id L 2 ) with:

Original: [Equation]
Revised: [Equation]

Original:  E VOLUTION OF GBDT IN RKHS 3. P RELIMINARIES
Revised:  E VOLUTION OF GBDT IN RKHS 3. P RELIMINARIES

Original: In our analysis, we assume that we are given a ﬁnite set V of weak learners used for the gradient boosting.
Revised: In our analysis, we assume that we are given a ﬁnite set V of weak learners used for the gradient boosting.

Original: Each ν corresponds to a decision tree that deﬁnes a partition of the feature space into disjoint regions (leaves).
Revised: Each ν corresponds to a decision tree that deﬁnes a partition of the feature space into disjoint regions (leaves).

Original: For each ν ∈ V , we denote the number of leaves in the tree by L ν ≥ 1 .
Revised: For each ν ∈ V , we denote the number of leaves in the tree by L ν ≥ 1 .

Original: Also, let φ ν : X → { 0 , 1 } L ν be a mapping that maps x to the vector indicating its leaf index in the tree ν .
Revised: Also, let φ ν : X → { 0 , 1 } L ν be a mapping that maps x to the vector indicating its leaf index in the tree ν .

Original: This mapping deﬁnes a decomposition of X into the disjoint union: X = ∪ L ν j =1 (cid:8) x ∈
Revised: This mapping deﬁnes a decomposition of X into the disjoint union: X = ∪ L ν j =1 (cid:8) x ∈

Original: X (cid:12)(cid:12) φ ( j ) ν
Revised: X (cid:12)(cid:12) φ ( j ) ν

Original: ( x ) = 1 (cid:9) .
Revised: ( x ) = 1 (cid:9) .

Original: Having φ ν , we deﬁne a weak learner associated with it as x (cid:55)→ (cid:104) θ, φ ν ( x ) (cid:105) R Lν for any choice of θ ∈ R L ν which we refer to as ‘leaf values’.
Revised: Having φ ν , we deﬁne a weak learner associated with it as x (cid:55)→ (cid:104) θ, φ ν ( x ) (cid:105) R Lν for any choice of θ ∈ R L ν which we refer to as ‘leaf values’.

Original: In other words, θ corresponds to predictions assigned to each region of the space deﬁned by ν .
Revised: In other words, θ corresponds to predictions assigned to each region of the space deﬁned by ν .

Original: Let us deﬁne a linear space F ⊂ L 2 ( ρ ) of all possible ensembles of trees from V :
Revised: Let us deﬁne a linear space F ⊂ L 2 ( ρ ) of all possible ensembles of trees from V :

Original: [Equation]
Revised: [Equation]

Original: We note that the space F can be data-dependent since V may depend on z , but we omit this dependence in the notation for simplicity.
Revised: We note that the space F can be data-dependent since V may depend on z , but we omit this dependence in the notation for simplicity.

Original: Note that we do not take the closure w.r.t.
Revised: Note that we do not take the closure w.r.t.

Original: the topology of L 2 ( ρ ) since we assume that V is ﬁnite and therefore F is ﬁnite-dimensional and thus topologically closed.
Revised: the topology of L 2 ( ρ ) since we assume that V is ﬁnite and therefore F is ﬁnite-dimensional and thus topologically closed.

Original: 3. GBDT ALGORITHM UNDER CONSIDERATION
Revised: 3. GBDT ALGORITHM UNDER CONSIDERATION

Original: Our theoretical analysis holds for classic GBDT algorithms discussed in Section 2.1 equipped with regularization from Ustimenko & Prokhorenkova (2021).
Revised: Our theoretical analysis holds for classic GBDT algorithms discussed in Section 2.1 equipped with regularization from Ustimenko & Prokhorenkova (2021).

Original: The only requirement we need is that the procedure of choosing each new tree has to be properly randomized.
Revised: The only requirement we need is that the procedure of choosing each new tree has to be properly randomized.

Original: Let us discuss a tree selection algorithm that we assume in our analysis.
Revised: Let us discuss a tree selection algorithm that we assume in our analysis.

Original: Each new tree approximates the gradients of the loss function with respect to the current predictions of the model.
Revised: Each new tree approximates the gradients of the loss function with respect to the current predictions of the model.

Original: Since we consider the RMSE loss function, the gradients are proportional to the residuals r j =
Revised: Since we consider the RMSE loss function, the gradients are proportional to the residuals r j =

Original: y
Revised: y

Original: j − f ( x j ) , where f is the currently built model.
Revised: j − f ( x j ) , where f is the currently built model.

Original: The tree structure is deﬁned by the features and the corresponding thresholds used to split the space.
Revised: The tree structure is deﬁned by the features and the corresponding thresholds used to split the space.

Original: The analysis in this paper assumes the SampleTree procedure (see Algorithm 1), which is a classic approach equipped with proper randomization.
Revised: The analysis in this paper assumes the SampleTree procedure (see Algorithm 1), which is a classic approach equipped with proper randomization.

Original: SampleTree builds an oblivious decision tree (Prokhorenkova et al., 2018), i.e., all nodes at a given level share the same splitting criterion (feature and threshold).
Revised: SampleTree builds an oblivious decision tree (Prokhorenkova et al., 2018), i.e., all nodes at a given level share the same splitting criterion (feature and threshold).

Original: To limit the number of candidate splits, each feature is quantized into n + 1 bins.
Revised: To limit the number of candidate splits, each feature is quantized into n + 1 bins.

Original: In other words, for each feature, we have n thresholds that can be chosen arbitrarily.
Revised: In other words, for each feature, we have n thresholds that can be chosen arbitrarily.

Original: The maximum tree depth is limited by m .
Revised: The maximum tree depth is limited by m .

Original: Recall that we denote the set of all possible tree structures by V .
Revised: Recall that we denote the set of all possible tree structures by V .

Original: We build the tree in a top-down greedy manner.
Revised: We build the tree in a top-down greedy manner.

Original: At each step, we choose one split among all the remaining candidates based on the following score deﬁned for ν ∈ V and residuals r :
Revised: At each step, we choose one split among all the remaining candidates based on the following score deﬁned for ν ∈ V and residuals r :

Original: [Equation]
Revised: [Equation]

Original: In classic gradient boosting, one builds a tree recursively by choosing such split s that maximizes the score D (( ν i , s ) , r ) (Ibragimov & Gusev, 2019).
Revised: In classic gradient boosting, one builds a tree recursively by choosing such split s that maximizes the score D (( ν i , s ) , r ) (Ibragimov & Gusev, 2019).

Original: 5 Random noise is often added to the scores to improve generalization.
Revised: 5 Random noise is often added to the scores to improve generalization.

Original: In SampleTree , we choose a split that maximizes
Revised: In SampleTree , we choose a split that maximizes

Original: D (cid:0) ( ν i , s ) , r (cid:1) + ε, where ε ∼ Gumbel(0 , β ) .
Revised: D (cid:0) ( ν i , s ) , r (cid:1) + ε, where ε ∼ Gumbel(0 , β ) .

Original: [Equation]
Revised: [Equation]

Original: Algorithm input: residuals r =
Revised: Algorithm input: residuals r =

Original: ( r i ) Ni = output: oblivious tree structure ν ∈ V hyper-parameters: number of feature splits n , max.
Revised: ( r i ) Ni = output: oblivious tree structure ν ∈ V hyper-parameters: number of feature splits n , max.

Original: tree depth m , random strength β ∈ [0 , ∞ ) deﬁnitions:
Revised: tree depth m , random strength β ∈ [0 , ∞ ) deﬁnitions:

Original: S = (cid:8) ( j, k ) (cid:12)(cid:12) j ∈ { 1 , . . .
Revised: S = (cid:8) ( j, k ) (cid:12)(cid:12) j ∈ { 1 , . . .

Original: , d } , k ∈ { 1 , . .
Revised: , d } , k ∈ { 1 , . .

Original: . , n } (cid:9) —
Revised: . , n } (cid:9) —

Original: indices of all possible splits instructions: initialize i = 0 , ν 0 = ∅ , S (0) =
Revised: indices of all possible splits instructions: initialize i = 0 , ν 0 = ∅ , S (0) =

Original: S repeat sample ( u i ( s )) s ∈S ( i ) ∼ U (cid:0) [0 , 1] nd − i (cid:1) choose next split as { s i +1 }
Revised: S repeat sample ( u i ( s )) s ∈S ( i ) ∼ U (cid:0) [0 , 1] nd − i (cid:1) choose next split as { s i +1 }

Original: = arg max s ∈S ( i ) (cid:16) D (cid:0) ( ν i , s ) , r (cid:1) − β log( − log u
Revised: = arg max s ∈S ( i ) (cid:16) D (cid:0) ( ν i , s ) , r (cid:1) − β log( − log u

Original: i ( s )) (cid:17) update tree: ν i +1 =
Revised: i ( s )) (cid:17) update tree: ν i +1 =

Original: ( ν i , s i +1 ) update candidate splits: S ( i +1)
Revised: ( ν i , s i +1 ) update candidate splits: S ( i +1)

Original: =
Revised: =

Original: S ( i ) \{ s i +1 } i
Revised: S ( i ) \{ s i +1 } i

Original: =
Revised: =

Original: i +until i ≥ m or S ( i ) =
Revised: i +until i ≥ m or S ( i ) =

Original: ∅ return:
Revised: ∅ return:

Original: ν i
Revised: ν i

Original: Algorithm
Revised: Algorithm

Original: ; (cid:15), input: dataset z = ( x N , y N ) hyper-parameters: learning rate (cid:15) > 0 , regularization λ > 0 , iterations of boosting T , parameters of SampleTree m, n, β instructions: initialize τ = 0 , f 0 ( · ) = 

Revised: ; (cid:15), input: dataset z = ( x N , y N ) hyper-parameters: learning rate (cid:15) > 0 , regularization λ > 0 , iterations of boosting T , parameters of SampleTree m, n, β instructions: initialize τ = 0 , f 0 ( · ) =

Original: L 2 ( ρ ) repeat r τ = y N − f τ ( x N ) — compute residuals ν τ = SampleTree ( r τ ; m, n, β ) — construct a tree θ τ = (cid:16) (cid:80) Ni =1 φ ( j ) ντ ( x i )
Revised: L 2 ( ρ ) repeat r τ = y N − f τ ( x N ) — compute residuals ν τ = SampleTree ( r τ ; m, n, β ) — construct a tree θ τ = (cid:16) (cid:80) Ni =1 φ ( j ) ντ ( x i )

Original: r
Revised: r

Original: ( i ) τ (cid:80)
Revised: ( i ) τ (cid:80)

Original: Ni =1 φ ( j ) ντ ( x i ) (cid:17)
Revised: Ni =1 φ ( j ) ντ ( x i ) (cid:17)

Original: L ντ j =1 — set values in leaves f τ +1 ( · ) =
Revised: L ντ j =1 — set values in leaves f τ +1 ( · ) =

Original: (1 − λ(cid:15) N ) f τ ( · ) + (cid:15) (cid:68) φ ν τ ( · ) , θ τ (cid:69) R Lντ — update model τ = τ +until τ ≥ T return: f T ( · )
Revised: (1 − λ(cid:15) N ) f τ ( · ) + (cid:15) (cid:68) φ ν τ ( · ) , θ τ (cid:69) R Lντ — update model τ = τ +until τ ≥ T return: f T ( · )

Original: Here β is random strength: β = 0 gives the standard greedy approach, while β → ∞ gives the uniform distribution among all possible split candidates.
Revised: Here β is random strength: β = 0 gives the standard greedy approach, while β → ∞ gives the uniform distribution among all possible split candidates.

Original: To sum up, SampleTree is a classic oblivious tree construction but with added random noise.
Revised: To sum up, SampleTree is a classic oblivious tree construction but with added random noise.

Original: We do this to make the distribution of trees regular in a certain sense: roughly speaking, the distributions should stabilize with iterations by converging to some ﬁxed distribution.
Revised: We do this to make the distribution of trees regular in a certain sense: roughly speaking, the distributions should stabilize with iterations by converging to some ﬁxed distribution.

Original: Given the algorithm SampleTree , we describe the gradient boosting procedure assumed in our analysis in Algorithm 2.
Revised: Given the algorithm SampleTree , we describe the gradient boosting procedure assumed in our analysis in Algorithm 2.

Original: It is a classic GBDT algorithm
Revised: It is a classic GBDT algorithm

Original: but with the update rule f τ +1 ( · ) = (1 − λ(cid:15)/N ) f τ ( · ) + (cid:15)w τ ( x ) .
Revised: but with the update rule f τ +1 ( · ) = (1 − λ(cid:15)/N ) f τ ( · ) + (cid:15)w τ ( x ) .

Original: In other words, we shrink the model at each iteration, which serves as regularization (Ustimenko & Prokhorenkova, 2021).
Revised: In other words, we shrink the model at each iteration, which serves as regularization (Ustimenko & Prokhorenkova, 2021).

Original: Such shrinkage is available, e.g., in the
Revised: Such shrinkage is available, e.g., in the

Original: CatBoost library.
Revised: CatBoost library.

Original: 3. D ISTRIBUTION OF TREES
Revised: 3. D ISTRIBUTION OF TREES

Original: The SampleTree algorithm induces a local family of distributions p ( ·| f, β ) for each f ∈ F : p (d ν | f, β )
Revised: The SampleTree algorithm induces a local family of distributions p ( ·| f, β ) for each f ∈ F : p (d ν | f, β )

Original: =
Revised: =

Original: P (cid:16)(cid:110) SampleTree (cid:0) y N − f ( x N ); m, n, β (cid:1) ∈
Revised: P (cid:16)(cid:110) SampleTree (cid:0) y N − f ( x N ); m, n, β (cid:1) ∈

Original: d ν (cid:111)(cid:17) .
Revised: d ν (cid:111)(cid:17) .

Original: Remark 3.1 .
Revised: Remark 3.1 .

Original: Lemma 3.7 ensure that such distribution coincides with the one where we use f ∗ ( x N ) instead of y N .
Revised: Lemma 3.7 ensure that such distribution coincides with the one where we use f ∗ ( x N ) instead of y N .

Original: This is due to the fact that D (cid:0) ν, y N − f ( x N ) (cid:1) =
Revised: This is due to the fact that D (cid:0) ν, y N − f ( x N ) (cid:1) =

Original: D (cid:0) ν, f ∗ − f ( x N ) (cid:1) ∀ ν ∈ V , f ∈ F .
Revised: D (cid:0) ν, f ∗ − f ( x N ) (cid:1) ∀ ν ∈ V , f ∈ F .

Original: The following lemma describes the distribution p (d ν | f, β ) , see Appendix F for the proof.
Revised: The following lemma describes the distribution p (d ν | f, β ) , see Appendix F for the proof.

Original: Note that for oblivious decision trees, changing the order of splits does not affect the obtained partition.
Revised: Note that for oblivious decision trees, changing the order of splits does not affect the obtained partition.

Original: Hence, we assume that each tree is deﬁned by an unordered set of splits.
Revised: Hence, we assume that each tree is deﬁned by an unordered set of splits.

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: (Probability of a tree)
Revised: (Probability of a tree)

Original: [Equation]
Revised: [Equation]

Original: where the sum is over all permutations
Revised: where the sum is over all permutations

Original: ς ∈ P m , ν ς,i =
Revised: ς ∈ P m , ν ς,i =

Original: ( s ς (1) , . . .
Revised: ( s ς (1) , . . .

Original: , s ς ( i ) ) , and ν = ( s 1 , . . .
Revised: , s ς ( i ) ) , and ν = ( s 1 , . . .

Original: , s m ) .
Revised: , s m ) .

Original: Let us deﬁne the stationary distribution of trees as π ( · ) =
Revised: Let us deﬁne the stationary distribution of trees as π ( · ) =

Original: lim β →∞ p ( ·| f, β ) .
Revised: lim β →∞ p ( ·| f, β ) .

Original: It follows from Remark 3.1 that we also have π ( · )
Revised: It follows from Remark 3.1 that we also have π ( · )

Original: =
Revised: =

Original: p ( ·| f ∗ , β ) .
Revised: p ( ·| f ∗ , β ) .

Original: Corollary 3.3.
Revised: Corollary 3.3.

Original: (Stationary distribution is the uniform distribution over tree structures)
Revised: (Stationary distribution is the uniform distribution over tree structures)

Original: We have π (d ν ) =
Revised: We have π (d ν ) =

Original: (cid:12)(cid:12)
Revised: (cid:12)(cid:12)

Original: d ν (cid:12)(cid:12)(cid:14)(cid:0) nd m (cid:1) , where (cid:0) nd m (cid:1) = ( nd )!
Revised: d ν (cid:12)(cid:12)(cid:14)(cid:0) nd m (cid:1) , where (cid:0) nd m (cid:1) = ( nd )!

Original: ( nd − m )!
Revised: ( nd − m )!

Original: m ! .
Revised: m ! .

Original: 3. RKHS S TRUCTURE
Revised: 3. RKHS S TRUCTURE

Original: In this section, we describe the evolution of GBDT in a certain Reproducing Kernel Hilbert Space (RKHS).
Revised: In this section, we describe the evolution of GBDT in a certain Reproducing Kernel Hilbert Space (RKHS).

Original: Even though, the problem is ﬁnite dimensional, treating it as functional regression is more beneﬁcial as dimension of the ensembles space growth rapidly and therefore we want to obtain dimension-free constants which is impossible if we treat it as ﬁnite dimensional optimization problem.
Revised: Even though the problem is ﬁnite dimensional, treating it as functional regression is more beneﬁcial as dimension of the ensembles space grows rapidly and therefore we want to obtain dimension-free constants which is impossible if we treat it as ﬁnite dimensional optimization problem.

Original: Let us start with deﬁning necessary kernels.
Revised: Let us start with deﬁning necessary kernels.

Revised: For convenience, we also provide a diagram illustrating the introduced kernels and relations between them in Appendix A.

Original: Deﬁnition 3.4.
Revised: Deﬁnition 3.4.

Original: A weak learner’s kernel k ν ( · , · ) is a kernel function associated with a tree structure ν ∈ V which can be deﬁned as:
Revised: A weak learner’s kernel k ν ( · , · ) is a kernel function associated with a tree structure ν ∈ V which can be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: This weak learner’s kernel is a building block for any other possible kernel in boosting and is used to deﬁne the iterations of the boosting algorithm analytically.
Revised: This weak learner’s kernel is a building block for any other possible kernel in boosting and is used to deﬁne the iterations of the boosting algorithm analytically.

Original: Deﬁnition 3.5.
Revised: Deﬁnition 3.5.

Original: We also deﬁne a greedy kernel of the gradient boosting algorithm as follows:
Revised: We also deﬁne a greedy kernel of the gradient boosting algorithm as follows:

Original: [Equation]
Revised: [Equation]

Original: This greedy kernel is a kernel that guides the GBDT iterations, i.e., we can think of each iteration as SGD with a kernel from 3.5, and 3.4 is used as a stochastic gradient estimator of the Fr´echet derivative in RKHS deﬁned by the kernel from 3.5.
Revised: This greedy kernel is a kernel that guides the GBDT iterations, i.e., we can think of each iteration as SGD with a kernel from 3.5, and 3.4 is used as a stochastic gradient estimator of the Fr´echet derivative in RKHS deﬁned by the kernel from 3.5.

Original: Deﬁnition 3.6.
Revised: Deﬁnition 3.6.

Original: Finally, there is a stationary kernel K ( x, x (cid:48) )
Revised: Finally, there is a stationary kernel K ( x, x (cid:48) )

Original: that is independent from f :
Revised: that is independent from f :

Original: [Equation]
Revised: [Equation]

Original: which we call a prior kernel of the gradient boosting.
Revised: which we call a prior kernel of the gradient boosting.

Original: This kernel deﬁnes the limiting solution since the gradient projection on RKHS converges to zero, and thus 3.5 converges to 3.6.
Revised: This kernel deﬁnes the limiting solution since the gradient projection on RKHS converges to zero, and thus 3.5 converges to 3.6.

Revised: Notice that F = span (cid:8) K ( · , x ) | x ∈ X (cid:9) .

Original: Having the space of functions F , we deﬁne a reproducing kernel Hilbert space (RKHS) structure H = (cid:0) F , (cid:104)· , ·(cid:105) H (cid:1) on it using a scalar product deﬁned as
Revised: Having the space of functions F , we deﬁne a reproducing kernel Hilbert space (RKHS) structure H = (cid:0) F , (cid:104)· , ·(cid:105) H (cid:1) on it using a scalar product deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where K 12 is the unique symmetric positive deﬁnite (SPD) square root operator such that (cid:16) K 12 (cid:17) 2 =

Original: K assuming that it acts as K [ g ] = (cid:82) X K ( · , x ) g ( x ) ρ (d x ) ∈

Original: L 2 ( ρ ) for any g ∈ L 2 ( ρ ) .

Original: Now, let us deﬁne the empirical error of a model f :
Revised: Now, let us deﬁne the empirical error of a model f :

Original: [Equation]
Revised: [Equation]

Original: Let us also deﬁne the following unique functions:
Revised: Let us also deﬁne the following unique functions:

Original: [Equation]
Revised: [Equation]

Original: And f ∗ =
Revised: And f ∗ =

Original: lim λ → 0 f λ ∗ ∈ arg min f ∈F V ( f ) where V ( f )
Revised: lim λ → 0 f λ ∗ ∈ arg min f ∈F V ( f ) where V ( f )

Original: =
Revised: =

Original: V ( f, 0) .
Revised: V ( f, 0) .

Original: It is known that such f ∗ exists and is unique since the set of all solutions is convex, and therefore there is a unique minimizer of the norm (cid:107) · (cid:107) H .
Revised: It is known that such f ∗ exists and is unique since the set of all solutions is convex, and therefore there is a unique minimizer of the norm (cid:107) · (cid:107) H .

Original: Finally, the following lemmas will be helpful for our analysis.
Revised: Finally, the following lemmas will be helpful for our analysis.

Original: In particular, we use them to analyze the distribution of trees in Section 3.3 and to derive the formula of the GBDT iteration in terms of kernels in Lemma 3.9.
Revised: In particular, we use them to analyze the distribution of trees in Section 3.3 and to derive the formula of the GBDT iteration in terms of kernels in Lemma 3.9.

Original: See Appendix D for the proofs.
Revised: See Appendix D for the proofs.

Original: Lemma 3.7.
Revised: Lemma 3.7.

Original: (cid:104)
Revised: (cid:104)

Original: y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N = 0 for any f ∈ F .
Revised: y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N = 0 for any f ∈ F .

Original: Lemma 3.8.
Revised: Lemma 3.8.

Original: V ( f, λ ) = 12 N (cid:107) f [Equation] ( x N [Equation]
Revised: V ( f, λ ) = 12 N (cid:107) f ∗ ( x N ) − f ( x N ) (cid:107) 2 R N + λ 2 N (cid:107) f

Original: ( x

Original: [Equation]

Original: [Equation] 2 R N + λ 2 N (cid:107)f [Equation] 2 H + const .
Revised: (cid:107) 2 H + const .

Original: Lemma 3.9.
Revised: Lemma 3.9.

Original: Iterations f
Revised: Iterations f

Original: τ of Gradient Boosting (Algorithm 2) can be written in the form:
Revised: τ of Gradient Boosting (Algorithm 2) can be written in the form:

Original: [Equation]
Revised: [Equation]

Original:  K ERNEL G RADIENT B OOSTING 4. K ERNEL R IDGE R EGRESSION
Revised: 3. K ERNEL G RADIENT B OOSTING C ONVERGENCE TO KRR

Original: Consider the sequence { f τ } τ ∈ N generated by the gradient boosting algorithm.
Revised: Consider the sequence { f τ } τ ∈ N generated by the gradient boosting algorithm.

Original: Its evolution is described by Lemma 3.9.
Revised: Its evolution is described by Lemma 3.9.

Original: The following theorem estimates the expected (w.r.t.
Revised: The following theorem estimates the expected (w.r.t.

Original: the randomness of tree selection) empirical error of f T relative to the best possible ensemble.
Revised: the randomness of tree selection) empirical error of f T relative to the best possible ensemble.

Original: The  proof can be found in Appendix G [Equation] Theorem
Revised: The full statement of the theorem and its proof can be found in Appendix G.

Original: [Equation]

Original: Let C = [Equation] Assume that (1+ λ ) 8 N [Equation]
Revised: Theorem 3.10.

Original: < [Equation] and deﬁne T 1 = (cid:104) [Equation]
Revised: Assume that β, T 1 are sufﬁciently large and (cid:15) is sufﬁciently small (see Appendix G).

Original: +

Original: [Equation]

Original: (cid:105)

Original: [-+ 
 then-] ∀ T ≥ T 1 it holds that
Revised: [-+ 
 then-]Then, ∀ T ≥ T 1 ,

Original: [Equation]
Revised: [Equation]

Original: Corollary 4.2.
Revised: Corollary 3.11.

Original: (Convergence to the solution of the KRR / Convergence to the Gaussian Process posterior mean function).
Revised: (Convergence to the solution of the KRR problem)

Original: Under  assumptions of the previous theorem we have:
Revised: Under the assumptions of the previous theorem, we have the following dimension-free bound:

Original: [Equation]
Revised: [Equation]

Original: We need this corollary to be sure that the Monte-Carlo sampling would approximately yield correct means and variances of the Gaussian process. We note that the estimate is pretty conservative.
Revised: This bound is dimension-free thanks to functional treatment and exponentially decaying to small value with iterations and therefore justiﬁes the observed rapid convergence of gradient boosting algorithms in practice even though dimension of space H is enormous.  G AUSSIAN INFERENCE

Original: First, it relies on the estimate λ min ( ∇ 2 H V ( f )) ≥ 1 N (when restricted on its image) which in fact in  the N → ∞ limit converges to λ min (lim N →∞ K ) > 0 independent from N which makes the bound of order O ( λ  2 βN + (cid:15) ) and requirement on 0 < (cid:15) ≤ O (1) for any T ≥ T 1 and T 1 = O (cid:16) − 1 (cid:15) log O ( (cid:15) + λ  2 βN ) (cid:17) which is linear convergence. Moreover, the condition N + λN
Revised: So far, the main result of the paper proved in Section 3.5 shows that Algorithm 2 solves the Kernel Ridge Regression problem which can be interpreted as learning Gaussian Process posterior mean f λ ∗ under the assumption that f ∼ GP (0 , σ 2 K + δ 2 Id L 2 ) where λ = σ 2 δ 2 .

Original: [Equation] N translates into β ≥ O (1) > 0 . If we do not use liming argument and use the bound as is, then it can be made of order ε by setting β ≥ λ 2 Nε and (cid:15) ≤ εN which imply T 1 = O (cid:16) N 2 ε log O ( 1 ε ) (cid:17) and
Revised: I.e. Algorithm 2 do not give us posterior variance, only mean but as we mentioned earlier we can estimate posterior variance through Monte-Carlo sampling in sample-then-optimize way.

Original: T − T 1 = O (cid:16) N 2 ε log O ( λε ) (cid:17) which yields that T = O (cid:16) N 2ε log O ( 1 ε ) (cid:17) .
Revised: For that we need to somehow sample from prior distribution GP (0 , σ 2 K + δ 2 Id L 2 ) itself.

Original: 4. P RIOR S AMPLING
Revised: 4. P RIOR S AMPLING

Original: We introduce Algorithm 3 for sampling from the prior distribution.
Revised: We introduce Algorithm 3 for sampling from the prior distribution.

Original: SamplePrior generates an ensemble of random trees (with random splits and random values in leaves).
Revised: SamplePrior generates an ensemble of random trees (with random splits and random values in leaves).

Original: Note that while being random, the tree structure depends on the dataset features x N since candidate splits are based on x N .
Revised: Note that while being random, the tree structure depends on the dataset features x N since candidate splits are based on x N .

Original: We ﬁrst note that the process h T ( · ) is centered with covariance operator K :
Revised: We ﬁrst note that the process h T ( · ) is centered with covariance operator K :

Original: [Equation]
Revised: [Equation]

Original: Then, we show that h T ( · ) converges to the
Revised: Then, we show that h T ( · ) converges to the

Original: Gaussian Process in the limit.
Revised: Gaussian Process in the limit.

Original: Lemma 4.3.
Revised: Lemma 4.1.

Original: The following convergence holds almost surely in x ∈ X :
Revised: The following convergence holds almost surely in x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: Algorithm hyper-parameters: number of iterations T , parameters of SampleTree m, n instructions: initialize τ = 0 , h 0 ( x ) =repeat ν 
Revised: Algorithm hyper number of iterations T , parameters of SampleTree m, n instructions: initialize τ = 0 , h 0 ( x ) =repeat ν τ = SampleTree ( 0 R N ; m, n, 1) — sample random tree θ τ ∼ N (cid:16) 0 R Lντ , diag (cid:16) N max { N ( j ) ντ , 1 } : j ∈ { 1 , . . . , L ν τ } (cid:17)(cid:17) — generate random values in leaves h τ

Original: [Equation]
Revised: [Equation]

Original: SampleTree (

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: N ; m, n, 1) — sample random tree
Revised: 1 √ T (cid:10) φ ν τ ( · ) , θ τ (cid:11) R Lντ — update model

Original: [Equation]
Revised: [Equation]

Original: — generate random values in leaves

Original:  [Equation]
Revised: = [Equation]

Original: update model τ = τ +until τ ≥ T return: h T ( · )
Revised: +until τ ≥ T return: h T ( · )

Original: (a) D 0.0 0.2 0.4 0.6 0.8 1.0 0.

Original: (b) (cid:101) µ 0.0 0.2 0.4 0.6 0.8 1.0 0.

Original: 642 (c) log (cid:101)

Original: σ

Original: 4. P OSTERIOR S AMPLING
Revised: 4. P OSTERIOR S AMPLING

Original: Now we are ready to introduce Algorithmfor sampling from the posterior.
Revised: Now we are ready to introduce Algorithm 4 for sampling from the posterior.

Original: The procedure is simple: we ﬁrst perform T 0 iterations of SamplePrior to obtain a function h T 0 ( · ) and then we train a standard GBDT model f T 1 ( · ) approximating y N − σh T 0 ( x N )+ N ( 0 N , δ 2 I N ) .
Revised: The procedure is simple: we ﬁrst perform T 0 iterations of SamplePrior to obtain a function h T 0 ( · ) and then we train a standard GBDT model f T 1 ( · ) approximating y N − σh T 0 ( x N ) + N ( 0 N , δ 2 I N ) .

Original: Our ﬁnal model is σh T 0 ( · ) + f T 1 ( · ) . 
Revised: Our ﬁnal model is σh T 0 ( · ) + f T 1 ( · ) . [Figure]

Original: We further refer to this procedure as SamplePosterior or KGB (Kernel Gradient Boosting) for brevity.
Revised: We further refer to this procedure as SamplePosterior or KGB (Kernel Gradient Boosting) for brevity.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: input: dataset z = ( x N , y N )

Original: -parameters iteration T 1 , SamplePrior iterations T 0 , parameters of SampleTree m, n, β , kernel scale σ > 0 (default σ = 1 ), noise scale δ > 0 (default: δ = 0 .

Original: 01 ) instructions: h T 0 ( · ) =

Original: [-SamplePrior( T 0 , m, n ) y newN = y N − σh T 0 ( x N ) + N ( 0 N , δ 
 I N ) f T 1 ( · ) = TrainGBDT (cid:0) ( x N , y newN ); (cid:15), T 1 , m, n, β, δ 2 σ 2 (cid:1) return: σh T 0 ( · ) + f T 1 ( · )-] where the ﬁrst limit is with respect to the point-wise convergence of stochastic processes and the second one with respect to L 2 ( ρ ) convergence.
Revised: [-SamplePrior( T 0 , m, n ) y newN = y N − σh T 0 ( x N ) + N ( 0 N , δ 
 I N ) f T 1 ( · ) = TrainGBDT (cid:0) ( x N , y newN ); (cid:15), T 1 , m, n, β, δ 2 σ 2 (cid:1) return: σh T 0 ( · ) + f T 1 ( · )-] where the ﬁrst limit is with respect to the point-wise convergence of stochastic processes and the second one with respect to L 2 ( ρ ) convergence.

Original: The following theorem shows that KGB indeed samples from the desired posterior.
Revised: The following theorem shows that KGB indeed samples from the desired posterior.

Original: The proof directly follows from Lemmas 4.3 and 2.1.
Revised: The proof directly follows from Lemmas 4.1 and 2.1.

Revised: input: dataset z = ( x N , y N ) hyper-parameters: learning rate (cid:15) > 0 , boosting iteration T 1 , SamplePrior iterations T 0 , parameters of SampleTree m, n, β , kernel scale σ > 0 (default σ = 1 ), noise scale δ > 0 (default: δ = 0 .

Revised: 01 ) instructions:

Original: Theorem 4.4.
Revised: [Equation] [Equation] , m, n, β, [Equation]

Revised: 2 σ

Revised: [Equation]

Revised: (cid:1) return: σh T 0 ( · ) + f T 1 ( · )

Revised: Theorem 4.2.

Original: In the limit, the output of Algorithm 4 follows the Gaussian  posterior:
Revised: In the limit, the output of Algorithm 4 follows the Gaussian process posterior:

Original: [Equation]
Revised: [Equation]

Original: with mean (cid:101) f
Revised: with mean (cid:101) f

Original: ( x ) =
Revised: ( x ) =

Original: K ( x, x N ) (cid:0) K ( x N , x N ) + λI N (cid:1) − 1 y N and covariance (cid:101) K ( x, x )
Revised: K ( x, x N ) (cid:0) K ( x N , x N ) + λI N (cid:1) − 1 y N and covariance (cid:101) K ( x, x )

Original: = δ 2 + σ 2 (cid:0) K
Revised: = δ 2 + σ 2 (cid:0) K

Original: ( x, x ) − K ( x, x N ) (cid:0) K ( x N , x N ) + λI N (cid:1) − 1 K ( x N , x ) (cid:1) .
Revised: ( x, x ) − K ( x, x N ) (cid:0) K ( x N , x N ) + λI N (cid:1) − 1 K ( x N , x ) (cid:1) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section empirically evaluates the proposed KGB algorithm and shows that it indeed allows for better knowledge uncertainty estimates.
Revised: This section empirically evaluates the proposed KGB algorithm and shows that it indeed allows for better knowledge uncertainty estimates.

Original: Synthetic experiment To illustrate the KGB algorithm in a controllable setting, we ﬁrst conduct a synthetic experiment.
Revised: Synthetic experiment To illustrate the KGB algorithm in a controllable setting, we ﬁrst conduct a synthetic experiment.

Original: For this, we deﬁned the feature distribution as uniform over D =
Revised: For this, we deﬁned the feature distribution as uniform over D =

Original: { ( x, y ) ∈ [0 , 1] 2 : 110 ≤ ( x − 12 ) 2 − ( y − 12 ) 2 ≤ 14 ∧ ( x ≤ 25 ∨ x ≥ 35 ) ∧ ( y ≤ 25 ∨ y ≥ 35 ) } .
Revised: { ( x, y ) ∈ [0 , 1] 2 : 110 ≤ ( x − 12 ) 2 − ( y − 12 ) 2 ≤ 14 ∧ ( x ≤ 25 ∨ x ≥ 35 ) ∧ ( y ≤ 25 ∨ y ≥ 35 ) } .

Original: We sample 10K points from U ([0 , 1] 2 ) and take into the train set only those that fall into D .
Revised: We sample 10K points from U ([0 , 1] 2 ) and take into the train set only those that fall into D .

Original: The target is deﬁned as f ( x, y ) =
Revised: The target is deﬁned as f ( x, y ) =

Original: x + y .
Revised: x + y .

Original: Figure 1(a) illustrates the training dataset colored with the target values.
Revised: Figure 1(a) illustrates the training dataset colored with the target values.

Original: For evaluation, we take the same 10K points without restricting them to D .
Revised: For evaluation, we take the same 10K points without restricting them to D .

Original: For KGB, we ﬁx (cid:15) = 0 .
Revised: For KGB, we ﬁx (cid:15) = 0 .

Original: 3 , T 0 = 100 , T 1 = 900 , σ = 10 − 2 , δ = 10 − 4 β = 0 .
Revised: 3 , T 0 = 100 , T 1 = 900 , σ = 10 − 2 , δ = 10 − 4 β = 0 .

Original: 1 , m = 4 , n = 64 , and sampled 100 KGB models.
Revised: 1 , m = 4 , n = 64 , and sampled 100 KGB models.

Original: Figure 1(b) shows the estimated by Monte-Carlo posterior mean (cid:101) µ .
Revised: Figure 1(b) shows the estimated by Monte-Carlo posterior mean (cid:101) µ .

Original: On
Revised: On

Original: Figure 1(c), we show log (cid:101) σ , where (cid:101) σ 2 is the posterior variance estimated by Monte-Carlo.
Revised: Figure 1(c), we show log (cid:101) σ , where (cid:101) σ 2 is the posterior variance estimated by Monte-Carlo.

Original: One can see that the posterior variance is small in-domain and grows when we move outside the dataset D , as desired.[Table] [Table]
Revised: One can see that the posterior variance is small in-domain and grows when we move outside the dataset D , as desired.

Original: Experiment on real datasets Uncertainty estimates for GBDTs have been previously analyzed by Malinin et al.
Revised: Experiment on real datasets Uncertainty estimates for GBDTs have been previously analyzed by Malinin et al.

Original: Our experiments on real datasets closely follow their setup [Table] and we compare the proposed KGB with SGB, SGLB, and their ensembles.
Revised: Our experiments on real datasets closely follow their setup, and we compare the proposed KGB with SGB, SGLB, and their ensembles.

Original: For the experiments, we use several standard regression datasets (Gal & Ghahramani, 2016).
Revised: For the experiments, we use several standard regression datasets (Gal & Ghahramani, 2016).

Original: The implementation details can be found in Appendix H.
Revised: The implementation details can be found in Appendix H.

Original: The code of our experiments can be found on GitHub. 
Revised: The code of our experiments can be found on GitHub. [Table] [Table]

Original: We note that in our setup we cannot compute likelihoods as kernel K is deﬁned implicitly  and its evaluation requires summing up among all possible trees structures number of which grows as ( nd ) m which is unfeasible, not to mention  requirement to inverse the kernel which requires O ( N 2+ ω ) number of operations.
Revised: We note that in our setup, we cannot compute likelihoods as kernel K is deﬁned implicitly [Table] and its evaluation requires summing up among all possible trees structures number of which grows as ( nd ) m which is unfeasible, not to mention the requirement to inverse the kernel which requires O ( N 2+ ω ) operations which additionally rules out the applicability of classical Gaussian Processes methods with our kernel.

Original: Therefore,  typical Bayesian setup is not applicable andthat is why we resort to  uncertainty estimation setup described in Malinin et al.
Revised: Therefore, a typical Bayesian setup is not applicable, and we resort to the uncertainty estimation setup described in Malinin et al.

Revised: Also, the intractability of the kernel does not allow us to treat σ, δ in a fully Bayesian way, as it will require estimating the likelihood.

Revised: Therefore, we ﬁx them as constants, but we note that this will not affect the evaluation metrics for our setup as they are scale and translation invariant.

Original: First, we compare KGB with SGLB since they both sample from similar posterior distributions.
Revised: First, we compare KGB with SGLB since they both sample from similar posterior distributions.

Original: Thus, this comparison allows us to ﬁnd out which of the algorithms does a better sampling from the posterior and thus provides us with more reliable estimates of knowledge uncertainty.
Revised: Thus, this comparison allows us to ﬁnd out which of the algorithms does a better sampling from the posterior and thus provides us with more reliable estimates of knowledge uncertainty.

Original: Moreover, we consider the SGB approach as the most “straightforward” way to generate an ensemble of models.
Revised: Moreover, we consider the SGB approach as the most “straightforward” way to generate an ensemble of models.

Original: In Table 1, we compare the predictive performance of the methods.
Revised: In Table 1, we compare the predictive performance of the methods.

Original: Interestingly, we obtain improvements on almost all the datasets.
Revised: Interestingly, we obtain improvements on almost all the datasets.

Original: Here we perform cross-validation to estimate statistical signiﬁcance with paired t -test and highlight the approaches that are insigniﬁcantly different from the best one (p-value > 0 . 05 ).
Revised: Here we perform cross-validation to estimate statistical signiﬁcance with paired t -test and highlight the approaches that are insigniﬁcantly different from the best one (p-value > 0 . 05 ).

Original: Then, we check whether uncertainty measured as the variance of the model’s predictions can be used to detect errors and out-of-domain inputs.
Revised: Then, we check whether uncertainty measured as the variance of the model’s predictions can be used to detect errors and out-of-domain inputs.

Original: Detecting errors can be evaluated via the Prediction-Rejection Ratio (PRR) (Malinin, 2019; Malinin et al., 2020).
Revised: Detecting errors can be evaluated via the Prediction-Rejection Ratio (PRR) (Malinin, 2019; Malinin et al., 2020).

Original: PRR measures how well uncertainty estimates correlate with errors and rank-order them.
Revised: PRR measures how well uncertainty estimates correlate with errors and rank-order them.

Original: Out-of-domain (OOD) detection is usually assessed via the area under the ROC curve (AUC-ROC) for the binary task of detecting whether a sample is OOD (Hendrycks & Gimpel, 2016).
Revised: Out-of-domain (OOD) detection is usually assessed via the area under the ROC curve (AUC-ROC) for the binary task of detecting whether a sample is OOD (Hendrycks & Gimpel, 2016).

Original: For this, one needs an OOD test set.
Revised: For this, one needs an OOD test set.

Original: We use the same OOD test sets (sampled from other datasets) as Malinin et al.
Revised: We use the same OOD test sets (sampled from other datasets) as Malinin et al.

Original: The results of this experiment are given in Table 2.
Revised: The results of this experiment are given in Table 2.

Original: We can see that the proposed method signiﬁcantly outperforms the baselines for out-of-domain detection.
Revised: We can see that the proposed method signiﬁcantly outperforms the baselines for out-of-domain detection.

Original: These improvements can be explained by the theoretical soundness of KGB: convergence properties are theoretically grounded and non-asymptotic.
Revised: These improvements can be explained by the theoretical soundness of KGB: convergence properties are theoretically grounded and non-asymptotic.

Original: In contrast, for SGB, there are no general results applicable in our setting, while for SGLB the guarantees are asymptotic.
Revised: In contrast, for SGB, there are no general results applicable in our setting, while for SGLB the guarantees are asymptotic.

Original: In summary, these results show that our approach is superior to SGB and SGLB, achieving smaller values of RMSE and having better knowledge uncertainty estimates.
Revised: In summary, these results show that our approach is superior to SGB and SGLB, achieving smaller values of RMSE and having better knowledge uncertainty estimates.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper theoretically analyses the classic gradient boosting algorithm.
Revised: This paper theoretically analyses the classic gradient boosting algorithm.

Original: In particular, we show that
Revised: In particular, we show that

Original: GBDT converges to the solution of a certain Kernel Ridge Regression problem.
Revised: GBDT converges to the solution of a certain Kernel Ridge Regression problem.

Original: We also introduce a simple modiﬁcation of the classic algorithm allowing one to sample from the Gaussian posterior.
Revised: We also introduce a simple modiﬁcation of the classic algorithm allowing one to sample from the Gaussian posterior.

Original: The proposed method gives much better knowledge uncertainty estimates than the existing approaches.
Revised: The proposed method gives much better knowledge uncertainty estimates than the existing approaches.

Original: We highlight the following important directions for future research.
Revised: We highlight the following important directions for future research.

Original: First, to explore how one can control the kernel and use it for better knowledge uncertainty estimates.
Revised: First, to explore how one can control the kernel and use it for better knowledge uncertainty estimates.

Original: Also, we do not analyze generalization in the current work, which is another important research topic.
Revised: Also, we do not analyze generalization in the current work, which is another important research topic.

Original: Finally, we need to establish universal approximation property which further justiﬁes need for functional formalism.
Revised: Finally, we need to establish universal approximation property which further justiﬁes need for functional formalism.

Original: A N OTATION USED IN THE PAPER
Revised: A N OTATION USED IN THE PAPER

Original: For convenience, let us list some frequently used notation:
Revised: For convenience, let us list some frequently used notation:

Original: • X ⊂ R d — feature space; • d — dimension of feature vectors; • Y ⊂ R — target space; • ρ — distribution of features; • N — number of samples; • z = ( x N , y N ) — dataset; • V — set of all possible tree structures; • L ν : V → N — number of leaves for ν ∈ V ; • D ( ν, r ) — score used to choose a split (3); • S — indices of all possible splits; • n — number of borders in our implementation of SampleTree; • m — depth of the tree in our implementation of SampleTree; • β — random strength; • (cid:15) — learning rate; • λ — regularization; • F — space of all possible ensembles of trees from V ; • φ ν : X → { 0 , 1 } L ν — tree structure; • φ ( j ) ν — indicator of j -th leaf; • V ( f ) — empirical error of a model f relative to the best possible f (cid:48) ∈ F ; • k ν ( · , · ) — single tree kernel; • K ( · , · ) — stationary kernel of the gradient boosting; • p ( ·| f, β ) — distribution of trees, f ∈ F ; • π ( · ) = 
Revised: • X ⊂ R d — feature space; • d — dimension of feature vectors; • Y ⊂ R — target space; • ρ — distribution of features; • N — number of samples; • z = ( x N , y N ) — dataset; • V — set of all possible tree structures; • L ν : V → N — number of leaves for ν ∈ V ; • D ( ν, r ) — score used to choose a split (3); • S — indices of all possible splits; • n — number of borders in our implementation of SampleTree; • m — depth of the tree in our implementation of SampleTree; • β — random strength; • (cid:15) — learning rate; • λ — regularization; • F — space of all possible ensembles of trees from V ; • φ ν : X → { 0 , 1 } L ν — tree structure; • φ ( j ) ν — indicator of j -th leaf; • V ( f ) — empirical error of a model f relative to the best possible f (cid:48) ∈ F ; • k ν ( · , · ) — single tree kernel; • K ( · , · ) — stationary kernel of the gradient boosting; • p ( ·| f, β ) — distribution of trees, f ∈ F ; • π ( · ) = lim β →∞ p ( ·| f, β ) = p ( ·| f ∗ , β ) — stationary distribution of trees; • σ — kernel scale. The following diagram illustrates the kernels introduced in our paper:

Original: lim β →∞ p ( ·| f, β ) = p ( ·| f ∗ , β ) — stationary distribution of trees; • σ — kernel scale.
Revised: Weak learner’ kernel k ν τ : X × X → R + Σ ν τ : F → F Σ f τ : F → F Iteration kernel K f τ : X × X

Original: B A DDITIONAL RELATED WORK
Revised: → R + Stationary Kernel K : X × X → R + H = (cid:0) F , (cid:104)· , ·(cid:105) H (cid:1)

Revised: [Table]

Revised: B A DDITIONAL RELATED WORK

Original: Let us brieﬂy discuss some additional related work.
Revised: Let us brieﬂy discuss some additional related work.

Original: Mondrian forest method Balog et al. and Generalized Random Forests Athey et al. (2019) besides having links to the kernel methods, in fact, have the underlying limiting RKHS space that is much smaller than the space of all possible ensembles built on the same weak learners due to the independence of the trees that are added to the ensemble.
Revised: Mondrian forest method (Balog et al., 2016) and Generalized Random Forests (Athey et al., 2019), besides having links to the kernel methods, in fact, have the underlying limiting RKHS space that is much smaller than the space of all possible ensembles built on the same weak learners due to the independence of the trees that are added to the ensemble.

Original: Therefore, there is an issue of high bias when comparing plain gradient boosting with the plain random forest method.
Revised: Therefore, there is an issue of high bias when comparing plain gradient boosting with the plain random forest method.

Original: Also, these two methods are built from scratch to obtain an RKHS interpretation while we provide a link between the existing standard gradient boosting approaches to the kernel methods, i.e., we do not create a novel gradient boosting algorithm but rather show that the existing ones already have such a link to derive convergence rates and to exploit such linkage to obtain formal Gaussian process interpretation of the gradient boosting learning to get uncertainty estimates using well-established gradient boosting libraries.
Revised: Also, these two methods are built from scratch to obtain an RKHS interpretation while we provide a link between the existing standard gradient boosting approaches to the kernel methods, i.e., we do not create a novel gradient boosting algorithm but rather show that the existing ones already have such a link to derive convergence rates and to exploit such linkage to obtain formal Gaussian process interpretation of the gradient boosting learning to get uncertainty estimates using well-established gradient boosting libraries.

Revised: Let us mention that there are approaches that study kernels induced by tree ensembles through the perspective of Neural Tangent Kernel (Kanoh & Sugiyama, 2022), though this analysis is not applicable for classical gradient boosting, while ours is.

Revised: Let us also brieﬂy discuss the papers on Neural Tangent Kernel, e.g., (Jacot et al., 2018; Li & Liang, 2018; Allen-Zhu et al., 2019; Du et al., 2018), that study deep learning convergence through the perspective of kernel methods.

Revised: Though such works share similarities with what we do, there are fundamental differences.

Original: C C ONVEX OPTIMIZATION IN FUNCTIONAL SPACES
Revised: First, our work is not in the over-parametrization regime, i.e., our kernel method correspondence works for tree ensembles with ﬁxed parameters, but the correspondence is achieved as the number of iterations goes to inﬁnity. It is worth noting that the kernel method perspective on deep learning basically establishes that each trained deep learning model is a sample from Gaussian Process posterior (Lee et al., 2020; 2018; Yang, 2019; Cho & Saul, 2009), i.e., is sample-then-optimize. For boosting, we achieve this only by introducing Algorithm 4 relying on Algorithm 3, which in its essence, is random initialization for gradient boosting.

Revised: The classic gradient boosting 2 can be considered as the mean value of the Gaussian Process, which has no analogs in the world of deep learning, and to achieve convergence to posterior mean there, one needs to average among many trained models.

Revised: This can be considered as an advantage of gradient boosting over deep learning that we derive in our paper.

Revised: C C ONVEX OPTIMIZATION IN FUNCTIONAL SPACES

Original: In this section, we formulate basic deﬁnitions of differentiability in functional spaces and the theorem on the convergence of gradient descent in functional spaces.
Revised: In this section, we formulate basic deﬁnitions of differentiability in functional spaces and the theorem on the convergence of gradient descent in functional spaces.

Original: For the proof of the theorem and further details on convex optimization in functional space, the reader can consult Luenberger (1969).
Revised: For the proof of the theorem and further details on convex optimization in functional space, the reader can consult Luenberger (1969).

Original: We consider H to be a Hilbert space with some scalar product (cid:104)· , ·(cid:105) H .
Revised: We consider H to be a Hilbert space with some scalar product (cid:104)· , ·(cid:105) H .

Original: Deﬁnition C.1.
Revised: Deﬁnition C.1.

Original: We say that F : H → R is Fr´echet differentiable if for any f ∈ H there exists a bounded linear functional L f : H → R such that ∀ h ∈ H
Revised: We say that F : H → R is Fr´echet differentiable if for any f ∈ H there exists a bounded linear functional L f : H → R such that ∀ h ∈ H

Original: [Equation]
Revised: [Equation]

Original: The value of L f : H → R is denoted by D f F ( f ) and is called a Fr´echet differential of F at point f .
Revised: The value of L f : H → R is denoted by D f F ( f ) and is called a Fr ´ echet differential of F at point f .

Original: So, Fr´echet differential is a functional D f F : H → B ( H , R ) , where B ( X, Y ) denotes a normed space of linear bounded functionals from X to Y .
Revised: So, Fr ´ echet differential is a functional D f F : H → B ( H , R ) , where B ( X, Y ) denotes a normed space of linear bounded functionals from X to Y .

Original: Deﬁnition C.2.
Revised: Deﬁnition C.2.

Original: Let F : H → R be Fr´echet differentiable with a Fr´echet differential D f F ( f ) that is a bounded linear functional.
Revised: Let F : H → R be Fr´echet differentiable with a Fr´echet differential D f F ( f ) that is a bounded linear functional.

Original: Then, by the Riesz theorem there exists a unique h f such that (cid:0)
Revised: Then, by the Riesz theorem there exists a unique h f such that (cid:0)

Original: D f F ( f ) (cid:1)
Revised: D f F ( f ) (cid:1)

Original: [ h ] = (cid:104)
Revised: [ h ] = (cid:104)

Original: h f , h (cid:105) H ∀ h ∈ H .
Revised: h f , h (cid:105) H ∀ h ∈ H .

Original: We call such element a gradient of F in H at f ∈ H and denote it by ∇ H F ( f ) =
Revised: We call such element a gradient of F in H at f ∈ H and denote it by ∇ H F ( f ) =

Original: h f ∈ H .
Revised: h f ∈ H .

Original: Deﬁnition C.3.
Revised: Deﬁnition C.3.

Original: F : H → R is said to be twice Fr´echet differentiable if D f F is Fr´echet differentiable, where the deﬁnition of Fr´echet differential is analogous to Deﬁnition C.1 with the only difference that D f F takes values in B ( H , R ) .
Revised: F : H → R is said to be twice Fr´echet differentiable if D f F is Fr´echet differentiable, where the deﬁnition of Fr´echet differential is analogous to Deﬁnition C.1 with the only difference that D f F takes values in B ( H , R ) .

Original: The second Fr´echet differential is denoted by D 2 f F : H → B ( H , B
Revised: The second Fr´echet differential is denoted by D 2 f F : H → B ( H , B

Original: ( H , R )) .
Revised: ( H , R )) .

Original: As there is an isomorphism between B ( H , B ( H , R )) and
Revised: As there is an isomorphism between B ( H , B ( H , R )) and

Original: B ( H × H , R ) , we can consider the second Fr´echet differential to take values in B ( H × H , R ) .
Revised: B ( H × H , R ) , we can consider the second Fr´echet differential to take values in B ( H × H , R ) .

Original: Henceforth, we will not differentiate between B ( H , B ( H , R )) and B ( H × H , R ) .
Revised: Henceforth, we will not differentiate between B ( H , B ( H , R )) and B ( H × H , R ) .

Original: Deﬁnition C.4.
Revised: Deﬁnition C.4.

Original: A linear operator P : H × H → R is said to be semi-positive deﬁnite (denoted by
Revised: A linear operator P : H × H → R is said to be semi-positive deﬁnite (denoted by

Original: P (cid:23) 0 ) if ∀ f ∈ H we have P ( f, f ) ≥ 0 .
Revised: P (cid:23) 0 ) if ∀ f ∈ H we have P ( f, f ) ≥ 0 .

Original: P is said to be positive deﬁnite ( P (cid:31) 0 ) if ∀ f ∈ H \ { 0 } : P ( f, f ) > 0 .
Revised: P is said to be positive deﬁnite ( P (cid:31) 0 ) if ∀ f ∈ H \ { 0 } we have P ( f, f ) > 0 .

Original: Deﬁnition C.5.
Revised: Deﬁnition C.5.

Original: Given two linear operators P, G : H × H
Revised: Given two linear operators P, G : H × H

Original: → R we write P (cid:23) G if P − G (cid:23) 0 and
Revised: → R we write P (cid:23) G if P − G (cid:23) 0 and

Original: P (cid:31) G if P − G (cid:31) 0 .
Revised: P (cid:31) G if P − G (cid:31) 0 .

Original: Let I ∈ B
Revised: Let I ∈ B

Original: ( H × H , R ) be a linear operator deﬁned as I ( g, h ) =
Revised: ( H × H , R ) be a linear operator deﬁned as I ( g, h ) =

Original: ( g, h ) H .
Revised: ( g, h ) H .

Original: Theorem C.6.
Revised: Theorem C.6.

Original: Let F be bounded below and twice Fr´echet differentiable functional on a Hilbert space H .
Revised: Let F be bounded below and twice Fr´echet differentiable functional on a Hilbert space H .

Original: Assume that D 2 f F ( f ) satisﬁes 0 ≺ mI (cid:22) D 2 f F ( f ) (cid:22) µI .
Revised: Assume that D 2 f F ( f ) satisﬁes 0 ≺ mI (cid:22) D 2 f F ( f ) (cid:22) µI .

Original: Then the gradient descent scheme:
Revised: Then the gradient descent scheme:

Original: [Equation]
Revised: [Equation]

Original: converges to f ∗ that minimizes
Revised: converges to f ∗ that minimizes

Original: F.
Revised: F.

Original: Proof.
Revised: Proof.

Original: For the proof see Luenberger (1969).
Revised: For the proof see Luenberger (1969).

Original: D K ERNEL RIDGE REGRESSION AND RKHS
Revised: D K ERNEL RIDGE REGRESSION AND RKHS

Original: Deﬁnition D.1.
Revised: Deﬁnition D.1.

Original: K :
Revised: K :

Original: X × X
Revised: X × X

Original: → R is called a kernel function if it is positive semi-deﬁnite, i.e., ∀ N ∈ N + ∀ x N ∈ R N : K ( x N , x N ) (cid:23) 0 .
Revised: → R is called a kernel function if it is positive semi-deﬁnite, i.e., ∀ N ∈ N + ∀ x N ∈ X N : K ( x N , x N ) (cid:23) 0 .

Original: Deﬁnition D.2.
Revised: Deﬁnition D.2.

Original: For any kernel function we can deﬁne a Reproducing Kernel Hilbert Space (RKHS)
Revised: For any kernel function we can deﬁne a Reproducing Kernel Hilbert Space (RKHS)

Original: [Equation]
Revised: [Equation]

Original: with a scalar product such that
Revised: with a scalar product such that

Original: [Equation]
Revised: [Equation]

Original: Alternatively, we can deﬁne the scalar product as

Original: [Equation]

Original: where K 12 corresponds to the unique SPD square root operator such that (cid:16) K 12 (cid:17) 2 = K assuming that it acts as K

Original: [ g ] = (cid:82) X K ( · , x ) g ( x ) ρ (d x ) ∈

Original: L 2 ( ρ ) for any g ∈ L 2 ( ρ ) .

Original: Both deﬁne the same scalar product.

Original: Consider the following Kernel Ridge Regression problem:
Revised: Consider the following Kernel Ridge Regression problem:

Original: [Equation]
Revised: [Equation]

Original: and the following Kernel Ridgeless Regression problem:
Revised: and the following Kernel Ridgeless Regression problem:

Original: [Equation]
Revised: [Equation]

Original: Lemma D.3.
Revised: Lemma D.3.

Original: min H ( K ) V ( f, λ ) has the only solution
Revised: min H ( K ) V ( f, λ ) has the only solution

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, let us show that f λ ∗ ∈ span [Equation]
Revised: First, let us show that f λ ∗ ∈ span (cid:8) K ( · , x i ) (cid:9) .

Original: Let H ( K ) =
Revised: Let H ( K ) =

Original: span (cid:8) K ( · , x i ) (cid:9) ⊕ span (cid:8) K ( · , x i ) (cid:9) ⊥ and consider the projector P : H ( K ) → H ( K ) onto the space span (cid:8) K ( · , x i ) (cid:9) .
Revised: span (cid:8) K ( · , x i ) (cid:9) ⊕ span (cid:8) K ( · , x i ) (cid:9) ⊥ and consider the projector P : H ( K ) → H ( K ) onto the space span (cid:8) K ( · , x i ) (cid:9) .

Original: It is easy to show that P ( f )( x N ) =
Revised: It is easy to show that P ( f )( x N ) =

Original: f
Revised: f

Original: ( x N ) for any f ∈ H ( K ) .
Revised: ( x N ) for any f ∈ H ( K ) .

Original: Indeed, ( [Equation]
Revised: Indeed, ( f − P ( f ))

Original:  = 0 .
Revised: [ x N ] = (cid:104) f − P ( f ) , K ( · , x N ) (cid:105) = 0 .

Original: If f λ ∗ does not lie in span (cid:8) K ( · , x i ) (cid:9) , then (cid:107) f λ ∗ (cid:107) H ( K ) >
Revised: If f λ ∗ does not lie in span (cid:8) K ( · , x i ) (cid:9) , then (cid:107) f λ ∗ (cid:107) H ( K ) >

Original: (cid:107) P ( f λ ∗ ) (cid:107) H ( K ) and L ( P ( f λ ∗ ) , λ ) <
Revised: (cid:107) P ( f λ ∗ ) (cid:107) H ( K ) and V ( P ( f λ ∗ ) , λ ) <

Original: L ( f λ ∗ , λ ) .
Revised: V ( f λ ∗ , λ ) .

Original: We get a contradiction with the minimality of f λ ∗ .
Revised: We get a contradiction with the minimality of f λ ∗ .

Original: Now, let us prove the existence of f λ ∗ .
Revised: Now, let us prove the existence of f λ ∗ .

Original: Consider f = K ( · , x N ) c , where c ∈ R N .
Revised: Consider f = K ( · , x N ) c , where c ∈ R N .

Original: Then, we ﬁnd the optimal c by taking a derivative of V ( f, λ ) with respect to c and equating it to zero:
Revised: Then, we ﬁnd the optimal c by taking a derivative of V ( f, λ ) with respect to c and equating it to zero:

Original: [Equation]
Revised: [Equation]

Original: Then, [Equation] where v ∈ ker K ( x N , x N ) .
Revised: Then, c v = ( K ( x N , x N ) + λI ) − 1 ( y N + v ) , where v ∈ ker K ( x N , x N ) .

Original: Note that all K ( · , x N ) c v are equal.
Revised: Note that all K ( · , x N ) c v are equal.

Original: Then, we have the only solution of the KRR problem:
Revised: Then, we have the only solution of the KRR problem:

Original: [Equation]
Revised: [Equation]

Original: Lemma D.4.
Revised: Lemma D.4.

Original: min H ( K ) V ( f ) has the only solution in span (cid:8) K ( · , x i ) (cid:9) and it is the solution of minimum RKHS norm:
Revised: min H ( K ) V ( f ) has the only solution in span (cid:8) K ( · , x i ) (cid:9) and it is the solution of minimum RKHS norm:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider f = K ( · , x N ) c , where c ∈ R N .
Revised: Consider f = K ( · , x N ) c , where c ∈ R N .

Original: Now consider L ( f ) and differentiate it with respect to c .
Revised: Now consider V ( f ) and differentiate it with respect to c .

Original: If we equate the derivative to zero, we get:
Revised: If we equate the derivative to zero, we get:

Original: [Equation]
Revised: [Equation]

Original: Then, K ( x N , x N ) c − ( y N + v ) = 0 for some v ∈ ker K ( x N , x N ) .
Revised: Then, K ( x N , x N ) c − ( y N + v ) = 0 for some v ∈ ker K ( x N , x N ) .

Original: Note that y N +ker K ( x N , x N ) ∩
Revised: Note that y N +ker K ( x N , x N ) ∩

Original: Im K ( x N , x N ) (cid:54) = ∅ .
Revised: Im K ( x N , x N ) (cid:54) = ∅ .

Original: Then, for any v such that y N + v ∈
Revised: Then, for any v such that y N + v ∈

Original: Im K ( x N , x N ) there exist a solution c v =
Revised: Im K ( x N , x N ) there exists a solution c v =

Original: K ( x N , x N ) † ( y N + v ) + ker K ( x N , x N ) .
Revised: K ( x N , x N ) † ( y N + v ) + ker K ( x N , x N ) .

Original: This follows from the fact that K ( x N , x N ) K ( x N , x N ) † is an orthoprojector onto Im K ( x N , x N ) .
Revised: This follows from the fact that K ( x N , x N ) K ( x N , x N ) † is an orthoprojector onto Im K ( x N , x N ) .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Then, the existence and uniqueness of f ∗ follow.
Revised: Then, the existence and uniqueness of f ∗ follow.

Original: Now, consider a linear space F ⊂ L 2 ( ρ ) of all possible ensembles of trees from V :
Revised: Now, consider a linear space F ⊂ L 2 ( ρ ) of all possible ensembles of trees from V :

Original: [Equation]

Original: Consider such funtional on space F :

Original: [Equation]
Revised: [Equation]

Original: and deﬁne unique function:
Revised: Deﬁne the unique function:

Original: [Equation]
Revised: [Equation]

Original: Then following two lemmas hold.
Revised: Then following two lemmas hold:

Original: Lemma D.5 (Lemma 3.7 in the main text) .
Revised: Lemma D.5 (Lemma 3.7 in the main text) .

Original: (cid:104)
Revised: [Equation]

Original: y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N = 0 for any f ∈ F .
Revised: − [Equation] 0 for any f ∈ F .

Original: Proof.
Revised: Proof.

Original: Assume that (cid:104) y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N (cid:54) = 0 for some f ∈ F .
Revised: Assume that (cid:104) y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N (cid:54) = 0 for some f ∈ F .

Original: Then, for some f ∈ F , (cid:104) y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N > 0 .
Revised: Then, for some f ∈ F , (cid:104) y N − f ∗ ( x N ) , f ( x N ) (cid:105) R N > 0 .

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: for small enough α > 0 , which contradicts with the deﬁnition of f ∗ .
Revised: for small enough α > 0 , which contradicts with the deﬁnition of f ∗ .

Original: Lemma D.6 (Lemma 3.8 in the main 
Revised: Lemma D.6 (Lemma 3.8 in the main text) . V

Revised: [Equation]

Revised: f, λ

Original: [Equation]
Revised: [Equation]

Revised: N

Revised: [Equation]

Revised: + C .

Original: Proof.
Revised: Proof.

Original: We need to prove it only for V ( f ) withour regularization as for regularized it follows immidiately (note that f ∗ is minimizer of not regularized objective).
Revised: We need to prove it only for V ( f ) withour regularization as for regularized it follows immidiately (note that f ∗ is minimizer of not regularized objective).

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Now, let us prove that
Revised: Now, let us prove that

Original: [Equation]
Revised: [Equation]

Original: Indeed,
Revised: Indeed,

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows from the previous lemma
Revised: where the last equality follows from the previous lemma

Original: [Equation]
Revised: [Equation]

Original: E G AUSSIAN P ROCESS INFERENCE
Revised: E G AUSSIAN P ROCESS INFERENCE

Original: In this section [Equation] we prove Lemma 2.1 from Section 2.3 of the main text.
Revised: In this section, we prove Lemma 2.1 from Section 2.3 of the main text.

Original: Firstly [Equation] consider the Kernel Ridge Regression problem loss functional:
Revised: Firstly, consider the Kernel Ridge Regression problem loss functional:

Original: [Equation]
Revised: [Equation]

Original: We will show that it satisﬁes the conditions needed for Theorem C.6.
Revised: We will show that it satisﬁes the conditions needed for Theorem C.6.

Original: We will also deduce the formula of the gradient of L in order to show that gradient descent takes the form (2) [Equation]
Revised: We will also deduce the formula of the gradient of V in order to show that gradient descent takes the form (2).

Original: Lemma E.1.
Revised: Lemma E.1.

Original: L ( f ) is Fr´echet differentiable with the differential given by:
Revised: V ( f, λ ) is Fr´echet differentiable with the differential given by:

Original: [Equation]
Revised: [Equation]

Original: where ev x i : H ( K ) → R is a bounded linear functional such that ev x i ( f )
Revised: where ev x i : H ( K ) → R is a bounded linear functional such that ev x i ( f )

Original: = f
Revised: = f

Original: ( x i ) =
Revised: ( x i ) =

Original: ( f, K ( x i , · ))
Revised: ( f, K ( x i , · ))

Original: H ( K ) .
Revised: H ( K ) .

Original: Proof.
Revised: Proof.

Original: As Fr´echet differential is linear, we only need to ﬁnd Fr´echet differential for ( f ( x i ) − y i ) 2 .
Revised: As Fr ´ echet differential is linear, we only need to ﬁnd Fr ´ echet differential for ( f ( x i ) − y i ) 2 .

Original: Note that ( f ( x i ) − y i ) 2 is a composition of two functions:
Revised: Note that ( f ( x i ) − y i ) 2 is a composition of two functions:

Original: [Equation]
Revised: [Equation]

Original: The differential of the composition can be found as:
Revised: The differential of the composition can be found as:

Original: [Equation]
Revised: [Equation]

Original: where D f F ( f ) =
Revised: where D f F ( f ) =

Original: ev x i because
Revised: ev x i because

Original: [Equation]
Revised: [Equation]

Original: Lemma E.2.
Revised: Lemma E.2.

Original: The gradient of V ( f ) , Riesz representative of the functional above, is given by:
Revised: The gradient of V ( f, λ ) , Riesz representative of the functional above, is given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Trivially follows from the previous lemma.
Revised: Follows from the previous lemma.

Original: Lemma E.3.
Revised: Lemma E.3.

Original: V  is twice Fr´echet differentiable with the differential given by:
Revised: V ( f, λ ) is twice Fr´echet differentiable with the differential given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Due to the linearity of Fr ´ echet differential and lemma E.1 we need to ﬁnd only Fr ´ echet differential for ( f ( x i ) − y i ) ev x i .
Revised: Due to the linearity of Fr´echet differential and lemma E.1 we need to ﬁnd only Fr´echet differential for ( f ( x i ) − y i ) ev x i .

Original: Consider S ( f ) =
Revised: Consider S ( f ) =

Original: ( f ( x i ) − y i ) ev x i .
Revised: ( f ( x i ) − y i ) ev x i .

Original: Then we need to ﬁnd V f ∈ B ( H ( K ) , B ( H ( K ) , R ))
Revised: Then we need to ﬁnd V f ∈ B ( H ( K ) , B ( H ( K ) , R ))

Original: such that
Revised: such that

Original: S ( f + h ) =
Revised: S ( f + h ) =

Original: S ( f ) + V f
Revised: S ( f ) + V f

Original: [ h ] + o ( (cid:107) h (cid:107) ) .
Revised: [ h ] + o ( (cid:107) h (cid:107) ) .

Original: It is easy to show
Revised: It is easy to show

Original: that h (cid:55)→ h
Revised: that h (cid:55)→ h

Original: ( x i ) ev x i ∈ B
Revised: ( x i ) ev x i ∈ B

Original: ( H ( K ) , B ( H  , R )) and S ( f + h )=
Revised: ( H ( K ) , B ( H ( K ) , R )) and S ( f + h )

Revised: =

Original: S ( f ) + h ( x i ) ev x i .
Revised: S ( f )+ h ( x i ) ev x i .

Original: Thus, we get that D f S ( f )
Revised: Thus, we get that D f S ( f )

Original: [ h ] =
Revised: [ h ] =

Original: h
Revised: h

Original: ( x i ) ev x i .
Revised: ( x i ) ev x i .

Original: From this the statement of the lemma follows.
Revised: From this the statement of the lemma follows.

Original: Given all the above lemmas, as a corollary of Theorem C.6, we have the following.
Revised: Given all the above lemmas, as a corollary of Theorem C.6, we have the following.

Original: Corollary E.4.
Revised: Corollary E.4.

Original: Gradient descent, deﬁned by the following iterative scheme:
Revised: Gradient descent, deﬁned by the following iterative scheme

Original: [Equation]
Revised: [Equation]

Original: converges to the optimum of V ( f, λ ) .
Revised: converges to the optimum of V ( f, λ ) .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As mentioned in Section 2.3, the solution of the Kernel Ridgeless Regression problem coincides with a posterior of a Gaussian.
Revised: By lemma E.2, our update rule has the form [Equation] Then, we will ﬁnd m, µ such that 0 ≺ mI (cid:22) D 2 f V ( f, λ ) (cid:22) µI . By lemma E.3,

Revised: [Equation]

Revised: Then, we can take m = λ N .

Revised: Let us also write

Original: Since gradient descent converges to this solution and thus to the posterior of a Gaussian, equation (6) follows.
Revised: [Equation] Then, we can take µ = λN + 1 N max x ∈X K ( x, x ) . By theorem C.6 and lemma D.3 the corollary follows.

Original: Lemma E.5.
Revised: Lemma E.5.

Original: Consider the gradient descent:
Revised: Consider the gradient descent:

Original: [Equation]
Revised: [Equation]

Original: and the following randomization scheme:
Revised: and the following randomization scheme:

Original: sample f init ∼ GP ( 0 L 2 ( ρ ) , σ 2 K + δ 2 Id L 2 ); 2.
Revised: sample f init ∼ GP ( 0 L 2 ( ρ ) , σ 2 K + [Equation] 2 Id L [Equation]

Original:  set new labels y
Revised: ; 2. set new labels y

Original: newN =
Revised: newN =

Original: y N − f init ( x N ); 3.
Revised: y N − f init ( x N ); 3.

Original: ﬁt GD f τ ( · ) on y
Revised: ﬁt GD f τ ( · ) on y

Original: newN assuming f 0 ( · ) =
Revised: newN assuming f 0 ( · ) =

Original: L 2 ( ρ ) ; 4.
Revised: L 2 ( ρ ) ; 4.

Original: output ˆ f
Revised: output ˆ f

Original: ( · ) =
Revised: ( · ) =

Original: f init ( · ) + f ∞ ( · ) as ﬁnal model .
Revised: f init ( · ) + f ∞ ( · ) as ﬁnal model .

Original: Then, ˆ f from the scheme above follows the Gaussian Process posterior with the following mean and covariance:
Revised: Then [Equation] ˆ f from the scheme above follows the Gaussian Process posterior with the following mean and covariance:

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Let us ﬁnd the distribution of ˆ f at x ∈ R n .
Revised: Let us ﬁnd the distribution of ˆ f at x ∈ R n .

Original: It can be easily seen that:
Revised: It can be easily seen that:

Original: [Equation]
Revised: [Equation]

Revised: N (cid:17) − 1 y N .

Original: Let us now calculate covariance:
Revised: Let us now calculate covariance:

Original: [Equation]
Revised: [Equation]

Original: init ( x N ) (cid:1) · (cid:0) f init ( x ) − K ( x, x N ) (cid:16) K ( x N , x N ) + λI N (cid:17) − 1 f init ( x N ) (cid:1) T =

Original: E f init ( x ) f init ( x ) T − E f init ( x ) f init ( x N ) T (cid:16) K ( x N , x N ) + λI N (cid:17) − 1 K ( x N , x ) − K ( x, x N ) (cid:16) K ( x N , x N ) + λI N (cid:17) − 1 E f init ( x N ) f init ( x ) T + K

Original: ( x, x N ) (cid:16) K ( x N , x N ) + λI N (cid:17) − 1 E f init ( x N ) f init ( x N ) T (cid:16) K ( x N , x N ) + λI N (cid:17) − 1 K ( x N , x ) =

Original: δ 2 + σ 2 (cid:0) K ( x, x ) − 2 K ( x, x N ) (cid:16) K ( x N , x N ) + λI N (cid:17) − 1 K ( x N , x ) + K ( x, x N ) (cid:16) K ( x N , x N ) + λI N (cid:17) − 1 K ( x N , x N ) (cid:16) K ( x N , x N ) + λI N (cid:17) − 1 K ( x N , x ) (cid:1) = δ 2 + σ 2 (cid:0) K ( x, x ) − K ( x, x N ) (cid:16) K ( x N , x N ) + λI N (cid:17) − 1 K ( x N , x ) (cid:1) , which is exactly what we need.
Revised: , which is exactly what we need.

Original: F D ISTRIBUTION OF TREES
Revised: F D ISTRIBUTION OF TREES

Original: Lemma F.1 (Lemma 3.2 in the main text) .
Revised: Lemma F.1 (Lemma 3.2 in the main text) .

Original: [Equation]
Revised: [Equation]

Original: ∈P m m (cid:89) i =

Original: e D ( νς,i,r ) β (cid:80) s ∈S\

Original: ν ς,i − 1 e D (( νς,i − 1 ,s ) ,r ) β , where the sum is over all permutations ς ∈ P m , ν ς,i =
Revised: where the sum is over all permutations ς ∈ P m , ν ς,i =

Original: ( s ς (1) , . . .
Revised: ( s ς (1) , . . .

Original: , s ς ( i ) ) , and ν = ( s 1 , . . .
Revised: , s ς ( i ) ) , and ν = ( s 1 , . . .

Original: , s m ) .
Revised: , s m ) .

Original: Proof.
Revised: Proof.

Original: Let us ﬁx some permutation
Revised: Let us ﬁx some permutation

Original: ς ∈ P m .
Revised: ς ∈ P m .

Original: W.l.o.g.
Revised: W.l.o.g.

Original: , let ς = id P m , i.e. ς ( i ) =
Revised: , let ς = id P m , i.e. ς ( i ) =

Original: i ∀ i .
Revised: i ∀ i .

Original: It remains to derive the formula for the ﬁxed permutation.
Revised: It remains to derive the formula for the ﬁxed permutation.

Original: The probability of adding the next split given the previously build tree is:
Revised: The probability of adding the next split given the previously build tree is:

Original: [Equation]
Revised: [Equation]

Original: which comes from (4) and the Gumbel-SoftMax trick.
Revised: which comes from (4) and the Gumbel-SoftMax trick.

Original: Then, we decompose the probability P ( ν ) of a tree as:
Revised: Then, we decompose the probability P ( ν ) of a tree as:

Original: [Equation]
Revised: [Equation]

Original: and so for the ﬁxed permutation we have
Revised: and so for the ﬁxed permutation we have

Original: [Equation]
Revised: [Equation]

Original: Then we sum over all permutations and the lemma follows.
Revised: Then we sum over all permutations and the lemma follows.

Original: Now, let us deﬁne the following value indicating how different are the distribution of trees for f and f ∗ :
Revised: Now, let us deﬁne the following value indicating how different are the distribution of trees for f and f ∗ :

Original: [Equation]
Revised: [Equation]

Original: Lemma F.2.
Revised: Lemma F.2.

Original: The following bound relates the distributions.
Revised: The following bound relates the distributions.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider π = p ( ·| f ∗ , β ) and the following expression P ( ν, ς 
Revised: Consider π = p ( ·| f ∗ , β ) and the following expression P ( ν, ς ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation] Then,(cid:88)
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where in second inequality we used D ( ν, r ) ≤ 2 V ( f ) which straightly follows from the deﬁnition.
Revised: where in second inequality we used D ( ν, r ) ≤ 2 V ( f ) which straightly follows from the deﬁnition.

Original: By noting that the probabilities remain the same if we shift D ( · , r ) ← D ( · , r ) − 2 V ( f ) which becomes everywhere non-positive and allows us to do the above trick once more but in reverse manner: if we formally replace the D with such modiﬁed function and repeat the steps with reversing the inequalities which is needed since the new function is everywhere negative then the lemma follows.
Revised: By noting that the probabilities remain the same if we shift D ( · , r ) ← D ( · , r ) − 2 V ( f ) which becomes everywhere non-positive and allows us to do the above trick once more but in reverse manner: if we formally replace the D with such modiﬁed function and repeat the steps with reversing the inequalities which is needed since the new function is everywhere negative then the lemma follows.

Original: [Equation]
Revised: [Equation]

Original: G P ROOF OF T HEOREM 4. G. RKHS STRUCTURE
Revised: G P ROOF OF T HEOREM 3. G. RKHS STRUCTURE

Revised: In section 3.4 we deﬁned RKHS structure on F as:

Revised: [Equation]

Revised: and we introduced the kernels k ν , K f , K π .

Revised: Let us also deﬁne a kernel K p

Original: For a weak learner ν  ,  we deﬁne a covariance operator:
Revised: = (cid:82) X k ν p (d ν ) for arbitrary distribution p on V . This way, taking p as δ ν ( · ) , p ( ν | f, β ) , π ( · ) we get K p equal to k ν , K f , K , respectively. For each kernel, we deﬁne the operator associated with it denoted similarly: [Equation]

Revised: Lemma G.1.

Revised: Consider two positive semideﬁnite operators on a ﬁnite dimensional vector space V :

Revised: A : V → V and B : V → V such that A (cid:23) B .

Revised: Then, Im A ≥ Im B .

Revised: Lemma G.2.

Revised: K p : F → F is invertible for p non-vanishing on V .

Revised: Proof.

Revised: Note that Im k ν = span

Revised: φ jν | j = 1 , . .

Revised: , L ν } and p ( ν ) k ν (cid:22) K p .

Revised: Then, Im K p =

Revised: F follows from lemma G.1.

Revised: Thus, K p is invertible.

Revised: Lemma G.3.

Revised: F = span (cid:8) K p ( · , x ) | x ∈ X (cid:9) for p non-vanishing on V .

Revised: Proof.

Revised: From lemma G.2, Im K p = F .

Revised: From the deﬁnition of the operator, Im K p ⊂ span (cid:8) K p ( · , x ) | x ∈ X (cid:9) .

Revised: Then, the lemma follows.

Revised: Lemma G.4.

Revised: For non-vanishing distribution p on V (cid:104)K p f, g

Revised: [Equation]

Revised: ( K p ) =

Revised: (cid:104) f, g (cid:105)

Revised: L 2 ( ρ ) .

Revised: Proof.

Revised: It is sufﬁcient to check on the basis.

Revised: So, we take g = K p ( · , x ) .

Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Using this operator, we deﬁne Σ f = (cid:82) V Σ ν p (d ν | f, β ) and Σ
Revised: [Equation] where the second equality holds by the deﬁnition of the operator K p . For a weak learner ν , we deﬁne a covariance operator:

Original:  = (cid:82) V Σ ν π (d ν ) .
Revised: [Equation] Also, for an arbitrary probability distribution p over V , we denote Σ p = (cid:82) V Σ ν p (d ν ) .

Original: These operators are typically referred to as covariance operators.
Revised: These operators are typically referred to as covariance operators.

Original: Here we formulate and prove several lemmas about the RKHS structure and operators Σ , Σ f , Σ ν .
Revised: Let us formulate and prove several lemmas about the RKHS structure and operators Σ , Σ f , Σ ν .

Original: Lemma G.1.
Revised: Lemma G.5.

Original: The following formula holds ∀ h, g ∈ H :
Revised: (Courant-Fischer) Let A be an n × n real symmetric matrix and λ 1 ≤ . . . ≤ λ n its eigenvalues. Then,

Original: [Equation]
Revised: [Equation]

Revised: where R A ( x ) =

Revised: ( Ax,x ) ( x,x ) is the Rayleigh-Ritz quotient.

Original: Lemma G.2.
Revised: Lemma G.6.

Original: (Covariation majorization)
Revised: ρ ( k ν ( x N , x N )) = (cid:107) k

Original: The following operator inequality holds ∀ ν ∈ V and ∀ f ∈ H : [Equation]
Revised: ν ( x N , x N ) (cid:107) = N for any ν ∈ V .

Original: Proof.
Revised: Proof.

Original: We will show that λ max (Σ) ≤ 1 where π in the deﬁnition of Σ can be chosen arbitrary.
Revised: It is easy to see that [Equation] where 1 n × n is a matrix of size n × n consisting of ones.

Original: This way the statement would follow for Σ , Σ f , Σ ν by taking π to be equal p ( · | f ∗ , β ) , p ( · | f, β ) , or δ ν ( · ) respectively.
Revised: Then, we note that (cid:107) 1 n × n (cid:107) = n and now the statement of the lemma follows. Lemma G.7.

Original:  The  operator Σ is dual to K N = 1 N K ( x N , x N ) since we can deﬁne the operator
Revised: (Covariation majorization) The following operator inequality holds for probability distributions p, p (cid:48) over V , where p is arbitrary and p (cid:48) does not vanish at any ν ∈ V :

Original: [Equation]
Revised: [Equation]

Original: so that Σ = 1 N S ∗ N S N and K N = 1 N S N S ∗ N and, thus, has the same spectra. Therefore, instead of Σ , we can study the maximal eigenvalue of K N but the latter admits another decomposition
Revised: Proof. Consider the following operators:

Original: [Equation]
Revised: [Equation]

Original: where Φ ν = φ ν ( x N  ) ∈ R L ν × N  .
Revised: ( · , x N ) v. Then, Σ p = 1 N BA and K N = 1 N K p ( x N , x N ) = 1 N AB .

Original: The key observation is that the sub-expectation expression Φ Tν (cid:110) Φ ν Φ Tν (cid:111) † Φ ν deﬁnes a matrix with maximal eigenvalue bounded by 1 since the former is an orthoprojector which is also shown in Ustimenko & Prokhorenkova (2021).
Revised: As AB and BA have the same spectra, we further study the spectrum of K N . But K N = 1 N K p ( x N , x N ) = (cid:82) V 1 N k ν ( x N , x N ) p (d ν ) and λ max ( K N ) ≤ 1 follows from lemma

Original: Therefore, the expectation also inherits this bound on the maximal eigenvalue which with symmetry of K N gives us the desired inequality.
Revised: G. 6 .

Original: Now we need to show that λ min (Σ f ) ≥ 1 N . Similarly to the previous proof, we need to show that λ min (Σ f ∗ ) ≥ 1 Nand then to replace π ← p ( ·| f, β ) .
Revised: Then, λ max (Σ p ) ≤ 1 follows. Now we need to show that λ min (Σ p (cid:48) ) ≥ 1 N .

Original: Consider the following formula:
Revised: Consider the following formula:

Original: [Equation]
Revised: [Equation]

Original: where ( a ⊗ H  b )
Revised: where ( a ⊗ H ( K p (cid:48) ) b )

Original: [ c ] = (cid:104) b, c (cid:105) H  a .
Revised: [ c ] = (cid:104) b, c (cid:105) H K p (cid:48) a .

Original: If a = b and (cid:107) a (cid:107) H  = 1 , then 1 and 0 are the only eigenvalues of a ⊗ H  a .
Revised: If a = b and (cid:107) a (cid:107) H ( K p (cid:48) ) = 1 , then 1 and 0 are the only eigenvalues of a ⊗ H ( K p (cid:48) ) a .

Original: Then by induction assuming that we already summed up some of such rank one operators one of two scenarios can happen if we add one more rank one operator: either the rank does not increase or the rank increases by one.
Revised: Denote by S = span {K p (cid:48) ( · , x i ) | i = 1 , . . . , N } ⊂ H ( K p (cid:48) ) and m = dimS, n = dim H ( K p (cid:48) ) . Then, λ min (Σ [Equation]

Original: In the ﬁrst case we obtain the operator that is greater in the operator sense then the previously obtained one whilst both of them share the same linear space as image and coimage which due to positive deﬁniteness of both of them on the space can happen only if all eigenvalues of the new one greater or equal compared to the ones of the previous one.
Revised: where R Σ p (cid:48) ( x ) = (Σ p (cid:48) x,x ) H ( K p (cid:48) ) ( x,x ) H ( K p (cid:48) ) .

Original: In the second scenario the newly added rank one operator adds new eigenvalue 1 due to symmetry of the operators.
Revised: As dimU = n − m + 1 , then U ∩ S (cid:54) = ∅ . Suppose K p (cid:48) ( · , x i ) ∈ U ∩ S , then

Original: Finally, by noting that inf x ∈ X K  ( x, x ) ≥ 1 , we obtain the bound.
Revised: [Equation] where (*) is fulﬁlled as a ⊗ H ( K p (cid:48) ) a is a positive semideﬁnite operator and the last inequality follows from K p (cid:48) ( x, x ) ≥ 1 ∀ x ∈ X .

Original: Corollary G.3.
Revised: Corollary G.8.

Original: (Expected squared norm majorization by RKHS norm)

Original: The following bound holds ∀ f ∈ H :

Original: [Equation]
Revised: [Equation]

Revised: for f ∈ span {K ( · , x i ) | i = 1 , . . .

Revised: , N } .

Original: Proof.
Revised: Proof.

Original: From λ max (Σ) ≤ 1 it follows that λ max ( K )  = 
Revised: [Equation] as Σ (cid:23) 1 N I on span {K ( · , x i ) | i = 1 , . . .

Original: lim N →∞ λ max (Σ) ≤ 1 .
Revised: , N } . Lemma G.9. λ max ( K ) ≤ max x ∈X K ( x, x ) .

Original: Let K − 1 be taken as the unique inverse K − 1 on F and for arbitrary f ∈ L 2 ( ρ ) we consider L 2 ( ρ )
Revised: Proof.

Original: = F ⊕ F ⊥ which implies f = f 1 + f 2 , f 1 ∈ F , f 2 ∈ F ⊥ ∀ f ∈ L 2 ( ρ ) and, thus, allows us to deﬁne K − 1 f = K − 1 f 1 ∈
Revised: Consider K as an operator on ( F , L 2 ( ρ )) . We will prove that (cid:107)K(cid:107) B (( F ,L 2 ( ρ ))) ≤ max x ∈X K ( x, x ) and the lemma will follow.

Original: F . Such deﬁnition of the inverse implies that λ min ( K − 1 ) =
Revised: Consider the inequality

Original: [Equation]
Revised: [Equation]

Original: K ) ≥ 1 which when substituted back into the product
Revised: f

Original: [Equation]
Revised: [Equation]

Original: implies the bound.
Revised: Then, (cid:107)K [ f ] (cid:107) L 2 ≤ max x ∈X (cid:107)K x (cid:107) L 2 (cid:107) f (cid:107) L 2 .

Original: Lemma G.4. (cid:107) k
Revised: Note also that K ( x, x (cid:48) )

Original: ν
Revised: ≤ min( K ( x, x ) , K ( x (cid:48) , x (cid:48) )) which can be easily seen from the deﬁnition.

Original: ( x N , xN ) (cid:107)  ≤ N for any ν ∈ V .
Revised: Then, (cid:107)K x (cid:107) L 2 ≤ K ( x, x ) and the lemma follows.

Original: Proof. Consider S 1 , ..., S L ν — unions such that S i
Revised: Corollary G.10.

Original: = { x j | φ ( i ) ν ( x j ) = 1 } . Then, it is easy to see that
Revised: (Expected squared norm majorization by RKHS norm) The following bound holds ∀ f ∈ H :

Original: [Equation]
Revised: [Equation]

Original: where 1 n × n is a matrix of size n × n consisting of ones. Then, we note that (cid:107) 1 n × n (cid:107) = n and now the statement of the lemma follows.
Revised: Proof.

Original:  Lemma G.5.
Revised: We have [Equation] Then, from the previous lemma the bound holds. Lemma G.11.

Original: For any ν ∈ V , we have k ν ( · , x N )[ y N − f ∗ ( x N )]
Revised: For any ν ∈ V , we have k ν ( · , x N )[ y N − f ∗ ( x N )]

Original: = 0 .
Revised: = 0 .

Original: Proof.
Revised: Proof.

Original: Follows from Lemma 3.7.
Revised: Follows from Lemma 3.7.

Original: Lemma G.6 (Lemma 3.9 in the main text) .
Revised: Lemma G.12 (Lemma 3.9 in the main text) .

Original: Iterations f
Revised: Iterations f

Original: τ of Gradient Boosting (Algorithm 2) can be written in the form:
Revised: τ of Gradient Boosting (Algorithm 2) can be written in the form:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to Algorithm 2:
Revised: According to Algorithm 2:

Original: [Equation]
Revised: [Equation]

Original: θ

Original: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Now note that k ν τ ( · , x i ) =
Revised: Now note that k ν τ ( · , x i ) =

Original: ω jν τ φ jν τ ( · ) , where j is such that φ jν τ ( x i ) = 1 .
Revised: ω jν τ φ jν τ ( · ) , where j is such that φ jν τ ( x i ) = 1 .

Original: From this the lemma follows.
Revised: From this the lemma follows.

Original: Lemma G.7.
Revised: Lemma G.13.

Original: Let A, B ∈ B
Revised: Let A, B ∈ B

Original: ( H , H ) be two PSD operators such that ξB − A and ξA − B are PSD for some ξ ∈ (1 , ∞ ) .
Revised: ( H , H ) be two PSD operators such that ξB − A and ξA − B are PSD for some ξ ∈ (1 , ∞ ) .

Original: Let g, h ∈ H  two arbitrary vectors and λ ∈ R ++ be a constant.
Revised: Let g, h ∈ H be two arbitrary vectors and λ ∈ R ++ be a constant.

Original: Then:
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider the following equity:
Revised: Consider the following equality:

Original: [Equation]
Revised: [Equation]

Original: which is basically  classical decomposition of the dot product (cid:104)
Revised: which is basically the classical decomposition of the dot product (cid:104)

Revised: x, y

Original: [Equation]
Revised: [Equation]

Original:  2 (cid:1) .
Revised: = [Equation] 2 + [Equation]

Original: Then we note that (cid:0) 1 − ξ − 2 (cid:1) B −(cid:0) B − ξ − 1 A (cid:1) =
Revised: − [Equation] x − y (cid:107) 2 (cid:1) . Then, we note that (cid:0) 1 − ξ − 2 (cid:1) B −

Revised: [Equation]

Original:  ξ − 2 ( ξA − B ) is PSD by assumption which implies:
Revised: B − [Equation] − 1 A (cid:1) = ξ − 2 ( ξA − B ) is PSD by assumption, which implies:

Original: [Equation]
Revised: [Equation]

Original: Finally, noting that
Revised: Finally, note that

Original: [Equation]
Revised: [Equation]

Original: ≤ ξ 2 − 1  the result directly follows from equity:
Revised: ≤ ξ 2 − 1 . Then, the result directly follows from the following equality:

Original: [Equation]
Revised: [Equation]

Revised: Denote κ ( A, B )

Original: [Table]
Revised: [Equation]

Original: Denote κ (A, B 
Revised: H [Equation] [Equation] (cid:107) ( B − A

Revised: [Equation]

Revised: A −

Original: [Equation]
Revised: [Equation]

Revised: (cid:107) B ( H

Original:  Lemma G.8.
Revised: [Equation] H [Equation] Lemma G.14.

Original: If ξ − 1 K (cid:22) K f (cid:22) ξ K for ξ > 1  then κ ( K , K f ) ≤ ξ − 1 .
Revised: If ξ − 1 K (cid:22) K f (cid:22) ξ K for ξ > 1 , then κ ( K , K f ) ≤ ξ − 1 .

Original: Proof.
Revised: Proof.

Original: First we note that both operators are symmetric semi-positive deﬁnite in L 2 .
Revised: First, we note that both operators are symmetric semi-positive deﬁnite in L 2 .

Original: Now let us look at  Rayleigh quotient:
Revised: Now, let us look at the Rayleigh quotient:

Original: [Equation]
Revised: [Equation]

Original: In the last equality we used fact that K is symmetric positive deﬁnite and therefore K 12  too and hence we can substitute f ← K 12 f and use  explicit formula for the dot product in H via product in L 2 .
Revised: In the last equality we used fact that K is symmetric positive deﬁnite and therefore K 12 is too and hence we can substitute f ← K 12 f and use the explicit formula for the dot product in H via product in L 2 .

Original: Now we observe that
Revised: Now we observe that

Original: [Equation]
Revised: [Equation]

Original: which implies that  spectral radius ρ ( K − 12 ( K−K f ) K − 12 ) is bounded by ξ − 1 .
Revised: which implies that the spectral radius ρ ( K − 12 ( K − K f ) K − 12 ) is bounded by ξ − 1 .

Original: Therefore we obtain:
Revised: Therefore, we obtain:

Original: [Equation]
Revised: [Equation]

Original: Lemma G.9.
Revised: Lemma G.15.

Original: Iterations of gradient boosting can be represented as:
Revised: The iterations of gradient boosting can be represented as:

Original: [Equation]
Revised: [Equation]

Original: where D : F → F is bounded and universal for f τ , i.e. Σ f τ = K f τ D ∀ τ .
Revised: where D : F → F is bounded and universal for f τ , i.e. Σ f τ = K f τ D ∀ τ .

Original: Similar decomposition holds for ∇ H V ( f, λ ) =
Revised: Similar decomposition holds for ∇ H V ( f, λ ) =

Original: K D [ f − f ∗ ] + λN f .
Revised: K D [ f − f ∗ ] + λ N f .

Original: Proof.
Revised: Proof.

Original: First observe that iterations are basically ∇ H ( K fτ ) V ( f τ , λ ) assuming that we replace H with
Revised: First observe that iterations are basically ∇ H ( K fτ ) V ( f τ , λ ) assuming that we replace H with

Original: H ( K f τ ) in regularizer term.
Revised: H ( K f τ ) in regularizer term.

Original: Therefore, we need to prove only ∇ H V ( f, λ ) =
Revised: Therefore, we need to prove only ∇ H V ( f, λ ) =

Original: K D [ f − f ∗ ]+ λN f .
Revised: K D [ f − f ∗ ]+ λN f .

Original: To prove this observe that there is deﬁned linear
Revised: To prove this observe that there is deﬁned linear

Original: Frechet differential D f V ( f )[ h ] = 1 N (cid:80) Ni =1 (cid:0) f
Revised: Frechet differential D f V ( f )[ h ] = 1 N (cid:80) Ni =1 (cid:0) f

Original: ( x i ) − f ∗ ( x i ) (cid:1) h
Revised: ( x i ) − f ∗ ( x i ) (cid:1) h

Original: ( x i ) :
Revised: ( x i ) :

Original: F → R which is bilinear functional by f − f ∗ and h .
Revised: F → R which is bilinear functional by f − f ∗ and h .

Original: Since F is ﬁnite dimensional then there is linear representation of the mapping in L 2 product restricted on F , let us denote it
Revised: Since F is ﬁnite dimensional then there is linear representation of the mapping in L 2 product restricted on F , let us denote it

Original: D : F → F .
Revised: D : F → F .

Original: Now we have D f V ( f )
Revised: Now we have D f V ( f )

Original: [ h ] = (cid:104) D
Revised: [ h ] = (cid:104) D

Original: [ f − f ∗ ] , h (cid:105)
Revised: [ f − f ∗ ] , h (cid:105)

Original: L 2 .
Revised: L 2 .

Original: Now we transform it to D f V ( f )
Revised: Now we transform it to D f V ( f )

Original: [ h ] = (cid:104)K − 1 K D [ f − f ∗ ] , h (cid:105)
Revised: [ h ] = (cid:104)K − 1 K D [ f − f ∗ ] , h (cid:105)

Original: L 2 = (cid:104)K D [ f − f ∗ ] , h (cid:105) H which implies that ∇ H V ( f ) =
Revised: L 2 = (cid:104)K D [ f − f ∗ ] , h (cid:105) H which implies that ∇ H V ( f ) =

Original: K D [ f − f ∗ ] .
Revised: K D [ f − f ∗ ] .

Original: Lemma G.10.
Revised: Lemma G.16.

Original: Consider  setting of  theprevious lemma with requirement B − A being SPD being lifted.
Revised: Consider the setting of Lemma G.13 without the requirement B − A being PSD.

Original: Then the inequality holds:
Revised: Then the inequality holds:

Original: [Equation]
Revised: [Equation]

Original: ] + λh, B [ g ] + λh (cid:105) H ≥ (cid:0) 1 2 − κ ( A, B ) (cid:1)(cid:13)(cid:13)

Original: A [ g ] + λh (cid:13)(cid:13) 2 H − κ 2 ( A, B ) λ 2 (cid:107) h (cid:107) 2 H .

Original: Proof.
Revised: Proof.

Original: Consider rewriting the left part as (cid:104)
Revised: Let us rewrite the left part as [Equation]

Original: A [ g ]+ λh, B
Revised: [Table] [Equation]

Original: [ g ]+ λh (cid:105) H = (cid:104) A [ g ]+ λh, (cid:0) Id L 2 +( BA − 1 − Id L 2 ) (cid:1)
Revised: [Table]

Original: ( A [ g ]+ λh )+ λ
Revised: [Equation] [Table]

Original: [-(Id L 2 − BA − 1 ) h (cid:105) H ≥ ≥ (cid:107) A [ g ]+ λh (cid:107) 2 H −(cid:
 A [ g ]+ λh, (Id L 2 − BA − 1 )-]
Revised: [-(Id L 2 − BA − 1 ) h (cid:105) H ≥ ≥ (cid:107) A [ g ]+ λh (cid:107) 2 H −(cid:
 A [ g ]+ λh, (Id L 2 − BA − 1 )-][Equation]

Original: ( A [ g ]+ λh ) (cid:105) H + (cid:104) A [ g ]+ λh, λ
Revised: [Table] [Equation] [Table]

Original: (Id L 2 − BA − 1 ) h (cid:105) H ≥ ≥ (3 2 − κ ( A, B )) (cid:107) A [ g ]+ λh (cid:107) 2 H + λ 2 2 (cid:107) (Id L 2 − BA − 1 ) h (cid:107) 2 H − 1 2 (cid:107) (
Revised: [Equation]

Original: A [ g ]+ λh ) − λ
Revised: [Table] [Equation]

Original: (Id L 2 − BA − 1 ) h (cid:107) 2 H ≥ ≥ (32 − κ ( A, B ))
Revised: [Table]

Original: (cid:107)
Revised: [Equation]

Original: A [ g ] + λh (cid:107) 2 H − (cid:107) ( A [ g ] + λh ) (cid:107) 2 H − λ 2 (cid:107) (Id L 2 − BA − 1 ) h (cid:107) 2 H ≥ ≥ (cid:0)2 − κ ( A, B ) (cid:1)(cid:13)(cid:13) A [ g ] + λh (cid:13)(cid:13) 2 H
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: The following lemma holds.
Revised: The following lemma holds.

Original: Lemma G.11.
Revised: Lemma G.17.

Original: If ( λN + 1) (cid:15) < 1 and f 0 = [-
-] H  then ∀ τ the following holds almost surely
Revised: If ( λN + 1) (cid:15) < 1 and f 0 = [-
-]0 H , then ∀ τ the following holds almost surely

Original: [Equation]
Revised: [Equation]

Original: for norms (cid:107) · (cid:107) L 2 , (cid:107) · (cid:107) H  and (cid:107) · (cid:107) R N .
Revised: for norms (cid:107) · (cid:107) L 2 , (cid:107) · (cid:107) H , and (cid:107) · (cid:107) R N .

Original: Proof.
Revised: Proof.

Original: Consider writing
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Now observe that S :
Revised: Now observe that S :

Original: = (cid:16) Id L 2 − λ(cid:15)N
Revised: = (cid:16) Id L 2 − λ(cid:15)N

Original: Id L 2 − (cid:15) Σ
Revised: Id L 2 − (cid:15) Σ

Original: ν τ (cid:17) is symmetric with eigenvalues 0 <
Revised: ν τ (cid:17) is symmetric with eigenvalues 0 <

Original: λ i ( S ) ≤ 1 − λ(cid:15)N  therefore its operator norm in B ( L 2 ) , B ( H )  and R N × N is lesser than 1 − λ(cid:15)N .
Revised: λ i ( S ) ≤ 1 − λ(cid:15)N , therefore its operator norm in B ( L 2 ) , B ( H ) , and R N × N is less than 1 − λ(cid:15)N .

Original: Taking  norm from left and right side and using  sub-additivity of norm we obtain:
Revised: Taking the norm of left and right sides and using the sub-additivity of the norm, we obtain:

Original: [Equation]
Revised: [Equation]

Original: Since (cid:107) f 0 − f ∗ (cid:107) =
Revised: Since (cid:107) f 0 − f ∗ (cid:107) =

Original: (cid:107) f ∗ (cid:107) that recurrent relation inductively yields  statement of the lemma.
Revised: (cid:107) f ∗ (cid:107) that recurrent relation inductively yields the statement of the lemma.

Original: Corollary G.12.
Revised: Corollary G.18.

Original: Under the same conditions, (cid:107) f τ (cid:107) ≤ 2 (cid:107) f ∗ (cid:107) and V ( f τ , 0) ≤ R 2 almost surely.
Revised: Under the same conditions, (cid:107) f τ (cid:107) ≤ 2 (cid:107) f ∗ (cid:107) and V ( f τ , 0) ≤ R 2 almost surely.

Original: G. M AIN THEOREM
Revised: G. M AIN THEOREM

Original: Let us denote R 1 = (cid:107) f ∗ (cid:107) H .
Revised: Let us denote R 1 = (cid:107) f ∗ (cid:107) H .

Original: We argue that it is  constant value since  kernel H and f ∗ are convergent as N → ∞ which makes it bounded by some constant with probability arbitrary close to one.
Revised: We argue that it is a constant value since the kernel H and f ∗ are convergent as N → ∞ which makes it bounded by some constant with probability arbitrary close to one.

Original: Moreover, denote M β = e 2 mR 2 β > 1 .
Revised: Moreover, denote M β = e 2 mR 2 β > 1 .

Original: Theorem G.13.
Revised: Theorem G.19.

Original: Consider an arbitrary (cid:15) , 0 < (cid:15) ( λN + 1) < 1 and (1+ M β λ ) 4 M β N ≥ (cid:15) .
Revised: Consider an arbitrary (cid:15) , 0 < (cid:15) ( λN + 1) < 1 and (1+ M β λ ) 4 M β N ≥ (cid:15) .

Original: The following inequality holds:
Revised: The following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Revised: To prove it we will consider bounding V ( f ) ≤ V ( f, M β λ ) with M β = e 2 mR 2 β > 0 .

Revised: The reason is that it will allow us to invoke Lemma G.13.

Revised: After then by using strong convexity we obtain bound on V ( f τ , M β ) .

Original: First we deal with the term V ( f τ +1 ) :
Revised: After which we can produce bound on V ( f ) as min V ( f, M β λ ) ≤ V ( f ∗ ) + M β λ 2 N (cid:107) f ∗ (cid:107) 2 H . To produce result for V ( f τ , M β λ ) we consider expanding V ( f τ +1 , M β λ ) by substituting formula for f τ +1 and ﬁrst dealing with the term V ( f τ +1 ) :

Original: [Equation]
Revised: [Equation]

Revised: where in the last we used 12 D 2 f V ( f

Revised: [Equation]

Revised: [ f

Revised: [Equation]

Revised: +1 −

Revised: [Equation]

Revised: τ

Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: +1 − f τ ] ≤

Revised: [Equation]

Revised: 2 V

Revised: [Equation]

Revised: f τ ) + λ 2 (cid:15) 2 N 3 (cid:107) f τ ( x N ) (cid:107) 2 R N by using inequality (cid:107) a + b (cid:107) 2 ≤ 2 (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 and fact that Σ ν τ acts on R N as projector Ustimenko & Prokhorenkova (2021) which allows to bound (cid:107) Σ ν τ ( f τ − f ∗ ) (cid:107) R N ≤

Revised: [Equation]

Revised: f τ − f ∗ ) (cid:107) R N .

Original: Then with regularization:
Revised: Then, we analyze the regularization:

Revised: [Equation]

Revised: Summing up expectations of those two expressions, using G.10 we obtain:

Original: [Equation]
Revised: [Equation]

Original: Summing up those two expressions, applying Lemma G.7 and noting that Σis non-degenerate if we restrict on its image (which we can since f τ lies there) by using Polyak-Łojasiewicz inequality 12 (cid:107)∇ V (cid:107) H ≥ µV for µ -strongly convex function V (restricted on image of Σ ) with µ ≥ 1+ M β λ N >we obtain the following bound:
Revised: Then by applying Lemma G.15 for Σ f τ = K f τ D and then by applying Lemma G.13 with ξ = M β , A = K , B = K f τ , g = D [ f τ − f ∗ ] and h = f τ , we obtain the following bound: [Equation] Then by using Polyak-Łojasiewicz inequality 12 (cid:107)∇ V (cid:107) H ≥ µV for µ -strongly convex function V (restricted on image of Σ ) with µ ≥ 1+ M β λ N > 0 , which is due to corollary G.8, we obtain:

Revised: [Equation]

Revised: [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Revised: M

Revised: [Equation]

Revised: λ )

Revised: [Equation]

Revised: (cid:15)M

Revised: [Equation]

Revised: ( M

Revised: [Equation]

Revised: − 1)

Revised: λ

Original:  R21 .
Revised: [Equation] N [Equation] R

Revised: Substituting it into bound on V ( f τ +1 , M β λ ) gives:

Revised: [Equation]

Original: Which yields
Revised: Which yields

Original: [Equation]
Revised: [Equation]

Revised: Next we use the following inequality:

Revised: [Equation]

Original: Which ﬁnally translates into bound 
Revised: Which ﬁnally gives us bound on E V ( f T ) :

Original: [Equation]
Revised: [Equation]

Original: Theorem G.14.
Revised: Theorem G.20.

Original: (Theorem 4.1 in the main text) Let C=
Revised: (Theorem 3.10 in the main text) Let C

Original: [Equation] + M 
Revised: = 8 λ 2 (cid:15)M β (1+ M β λ

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: [Equation] β 
Revised: 2 + (cid:16) 8 M 2 β (cid:15) (1 + λ 2 N 2 ) +

Original: [Equation] M 2 β − 1) + 3 (cid:1)(cid:17) R 21 .
Revised: M β λ 2 N (cid:0) 8 M β ( M 2 β − 1) + 3 (cid:1)(cid:17) R 21 .

Original: Assume that 0 < (cid:15) ( λN + 1) < 1 and (1+ M β λ ) 4 M β N ≥ (cid:15) , (1+ λ ) 8 N ≥ (cid:15) ,
Revised: Assume that 0 < (cid:15) ( λN + 1) < 1 and (1+ M β λ ) 4 M β N ≥ (cid:15) , (1+ λ ) 8 N ≥ (cid:15) ,

Original: N +
Revised: N +

Original: [Equation]
Revised: [Equation]

Original: 2 mM β C β < 1+ λ 2 N and deﬁne T 1 = (cid:104) 2 M β N (1+ M β λ ))
Revised: 2 mM β C β < 1+ λ 2 N and deﬁne T 1 = (cid:104) 2 M β N (1+ M β λ ))

Original: (cid:15) log R 2 C (cid:105) + 1 then ∀ T ≥ T 1 it holds that
Revised: (cid:15) log R 2 C (cid:105) + 1 . Then ∀ T ≥ T 1 it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: If we take T 1 = (cid:104) [Equation] + [Equation]
Revised: First we apply previous theorem to obtain bound on V ( f τ ) which we will use to claim that kernels K f τ and K are close to each other in expectation.

Original: λ [Equation] N
Revised: If we take

Original: [Equation]
Revised: [Equation]

Original: M β [Equation] M 2 β − 1) + 3 (cid:1)(cid:17) R 21 R 2 (cid:17) − 1 (cid:105) + 1 then the following inequality holds ∀ τ ≥ T 1 :
Revised: then the following inequality holds ∀ τ ≥ T 1 :

Original: [Equation]
Revised: [Equation]

Original: Overall, the proof is similar to the previous theorem but with modiﬁcations, instead of
Revised: Overall, the proof is similar to the previous theorem but with some modiﬁcations.

Original:  Lemma G.7 we will use Lemma G.10.
Revised: Instead of Lemma G.13, we will use Lemma G.16.

Original: The only problem would arise in  multiplier E (cid:16)12 − e
Revised: The only problem would arise in the multiplier E (cid:16)

Original: [Equation]
Revised: [Equation]

Original: H V [Equation] f τ , λ )
Revised: H V ( f τ , λ )

Original: [Equation]
Revised: [Equation]

Original:  H .
Revised: 2 H .

Original: We can bound 
Revised: We can bound it as follows

Original: [Equation]
Revised: [Equation]

Original: After using ( e z − 1) 2 ≤ e 2 z − 1 ≤ 2 M 2 β z for z = 2 mV ( f ) β satisfying e z ≤ M β  we obtain the bound:
Revised: After using ( e z − 1) 2 ≤ e 2 z − 1 ≤ 2 M 2 β z for z = 2 mV ( f ) β satisfying e z ≤ M β , we obtain the bound:

Original: [Equation]
Revised: [Equation]

Original: Iterating the bound yields
Revised: Iterating the bound yields

Original: [Equation]
Revised: [Equation]

Original: Corollary G.15.
Revised: Corollary G.21.

Original: (Convergence to the solution of the KRR / Convergence to the Gaussian Process posterior mean function).
Revised: (Convergence to the solution of the KRR / Convergence to the Gaussian Process posterior mean function).

Original: Under  assumptions of both previous theorems we have that:
Revised: Under the assumptions of both previous theorems we have that:

Original: [Equation]
Revised: [Equation]

Original: Lemma G.16 (Lemma 4.3in the main text) .
Revised: Lemma G.22 (Lemma 4.1in the main text) .

Original: The following convergence holds almost surely in x ∈
Revised: The following convergence holds almost surely in x ∈

Original: X :
Revised: X :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From (5), we have that the covariance of h T is K independently from T .
Revised: From (5), we have that the covariance of h T is K independently from T .

Original: Thus, it remains to show that the limit is Gaussian almost surely which essentially holds due to the central limit theorem almost surely in x ∈ X :
Revised: Thus, it remains to show that the limit is Gaussian almost surely which essentially holds due to the central limit theorem almost surely in x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: where each individual tree h T,i is centered i.i.d.
Revised: where each individual tree h T,i is centered i.i.d.

Original: (with the same distribution as h 1 ).
Revised: (with the same distribution as h 1 ).

Original: H I MPLEMENTATION DETAILS
Revised: H I MPLEMENTATION DETAILS

Original: In the experiments, we ﬁx σ = 10 − 2 (scale of the kernel) and δ = 10 − 4 (scale of noise), which theoretically can be taken arbitrarily.
Revised: In the experiments, we ﬁx σ = 10 − 2 (scale of the kernel) and δ = 10 − 4 (scale of noise), which theoretically can be taken arbitrarily.

Original: As a hyperparameter (that is estimated on the validation set), we consider β ∈ { 10 − 2 , 10 − 1 , 1 } .
Revised: As a hyperparameter (that is estimated on the validation set), we consider β ∈ { 10 − 2 , 10 − 1 , 1 } .

Original: We use the standard CatBoost library and add the Gumbel noise term in selecting the trees for the “L2” scoring function, which is implemented in CatBoost out of the box but is not used by SGB and SGLB since it is not the default one for the library.
Revised: We use the standard CatBoost library and add the Gumbel noise term in selecting the trees for the “L2” scoring function, which is implemented in CatBoost out of the box but is not used by SGB and SGLB since it is not the default one for the library.

Original: Moreover, we do not consider subsampling of the data (as SGLB does also), and differently from SGB and
Revised: Moreover, we do not consider subsampling of the data (as SGLB does also), and differently from SGB and

Original: SGLB, we disable the “boost-from-average” option.
Revised: SGLB, we disable the “boost-from-average” option.

Original: Finally, we set l 2 − leaf − reg value to 0 , as
Revised: Finally, we set l 2 − leaf − reg value to 0 , as

Original: SGLB does.
Revised: SGLB does.