Original: U NCERTAINTY IN G RADIENT B OOSTING VIA E NSEMBLES
Revised: U NCERTAINTY IN G RADIENT B OOSTING VIA E NSEMBLES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: For many practical, high-risk applications, it is essential to quantify uncertainty in a model’s predictions to avoid costly mistakes.
Revised: For many practical, high-risk applications, it is essential to quantify uncertainty in a model’s predictions to avoid costly mistakes.

Original: While predictive uncertainty is widely studied for neural networks, the topic seems to be under-explored for models based on gradient boosting.
Revised: While predictive uncertainty is widely studied for neural networks, the topic seems to be under-explored for models based on gradient boosting.

Original: However, gradient boosting often achieves state-ofthe-art results on tabular data.
Revised: However, gradient boosting often achieves stateof-the-art results on tabular data.

Original: This work examines a probabilistic ensemblebased framework for deriving uncertainty estimates in the predictions of gradient boosting classiﬁcation and regression models.
Revised: This work examines a probabilistic ensemblebased framework for deriving uncertainty estimates in the predictions of gradient boosting classiﬁcation and regression models.

Original: We conducted experiments on a range of synthetic and real datasets and investigated the applicability of ensemble approaches to gradient boosting models that are themselves ensembles of decision trees.
Revised: We conducted experiments on a range of synthetic and real datasets and investigated the applicability of ensemble approaches to gradient boosting models that are themselves ensembles of decision trees.

Original: Our analysis shows that ensembles of gradient boosting models successfully detect anomaly inputs while having limited ability to improve the predicted total uncertainty.
Revised: Our analysis shows that ensembles of gradient boosting models successfully detect anomalous inputs while having limited ability to improve the predicted total uncertainty.

Original: Importantly, we also propose a concept of a virtual ensemble to get the beneﬁts of an ensemble via only one gradient boosting model, which signiﬁcantly reduces complexity.
Revised: Importantly, we also propose a concept of a virtual ensemble to get the beneﬁts of an ensemble via only one gradient boosting model, which signiﬁcantly reduces complexity.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Gradient boosting (Friedman, 2001) is a widely used machine learning algorithm that achieves state-of-the-art results on tasks containing heterogeneous features, complex dependencies, and noisy data: web search, recommendation systems, weather forecasting, and many others (Burges, 2010; 
Revised: Gradient boosting (Friedman, 2001) is a widely used machine learning algorithm that achieves stateof-the-art results on tasks containing heterogeneous features, complex dependencies and noisy data: web search, recommendation systems, weather forecasting, and many others (Burges, 2010; Caruana & Niculescu-Mizil, 2006;

Original: Caruana & Niculescu-Mizil, 2006; Richardson et al., 2007; Roe et al., 2005; Wu et al., 2010; Zhang & Haghani, 2015).
Revised: Richardson et al., 2007; Roe et al., 2005; Wu et al., 2010; Zhang & Haghani, 2015).

Original: Gradient boosting based on decision trees (GBDT) underlies such well-known libraries like XGBoost, LightGBM, and CatBoost.
Revised: Gradient boosting based on decision trees (GBDT) underlies such well-known libraries like XGBoost, LightGBM, and CatBoost.

Original: In this paper, we investigate the estimation of predictive uncertainty in GBDT models.
Revised: In this paper, we investigate the estimation of predictive uncertainty in GBDT models.

Original: Uncertainty estimation is crucial for avoiding costly mistakes in high-risk applications, such as autonomous driving, medical diagnostics, and ﬁnancial forecasting.
Revised: Uncertainty estimation is crucial for avoiding costly mistakes in high-risk applications, such as autonomous driving, medical diagnostics, and ﬁnancial forecasting.

Original: For example, in self-driving cars, it is necessary to know when the AI-pilot is conﬁdent in its ability to drive and when it is not to avoid a fatal collision.
Revised: For example, in self-driving cars, it is necessary to know when the AI-pilot is conﬁdent in its ability to drive and when it is not to avoid a fatal collision.

Original: In ﬁnancial forecasting and medical diagnostics, mistakes on the part of an AI forecasting or diagnostic system could either lead to large ﬁnancial or reputational loss or to the loss of life.
Revised: In ﬁnancial forecasting and medical diagnostics, mistakes on the part of an AI forecasting or diagnostic system could either lead to large ﬁnancial or reputational loss or to the loss of life.

Original: Crucially, both ﬁnancial and medical data are often represented in heterogeneous tabular form — data on which GBDTs are typically applied, highlighting the relevance of our work on obtaining uncertainty estimates for GBDT models.
Revised: Crucially, both ﬁnancial and medical data are often represented in heterogeneous tabular form — data on which GBDTs are typically applied, highlighting the relevance of our work on obtaining uncertainty estimates for GBDT models.

Original: Approximate Bayesian approaches for uncertainty estimation have been extensively studied for neural network models (Gal, 2016; Malinin, 2019).
Revised: Approximate Bayesian approaches for uncertainty estimation have been extensively studied for neural network models (Gal, 2016; Malinin, 2019).

Original: Bayesian methods for tree-based models (Chipman et al., 2010; Linero, 2017) have also been widely studied in the literature.
Revised: Bayesian methods for tree-based models (Chipman et al., 2010; Linero, 2017) have also been widely studied in the literature.

Original: However, this research did not explicitly focus on studying uncertainty estimation and its applications.
Revised: However, this research did not explicitly focus on studying uncertainty estimation and its applications.

Original: Some related work was done by Coulston et al.
Revised: Some related work was done by Coulston et al.

Original: ; Shaker & Hüllermeier (2020), who examined quantifying predictive uncertainty for random forests.
Revised: ; Shaker & Hüllermeier (2020), who examined quantifying predictive uncertainty for random forests.

Original: However, the area has been otherwise relatively underexplored, especially for GBDT models that are widely used in practice and known to outperform other approaches based on tree ensembles.
Revised: However, the area has been otherwise relatively under-explored, especially for GBDT models that are widely used in practice and known to outperform other approaches based on tree ensembles.

Original: While for classiﬁcation problems GDBT models already return a distribution over class labels, for regression tasks they typically yield only point predictions.
Revised: While for classiﬁcation problems GDBT models already return a distribution over class labels, for regression tasks they typically yield only point predictions.

Original: Recently, this problem was addressed in the NGBoost algorithm (Duan et al., 2020), where a GBDT model is trained to return the mean and variance of a normal distribution over the target variable y for a given feature vector.
Revised: Recently, this problem was addressed in the NGBoost algorithm (Duan et al., 2020), where a GBDT model is trained to return the mean and variance of a normal distribution over the target variable y for a given feature vector.

Original: However, such models only capture data uncertainty (Gal, 2016; Malinin, 2019), also known as aleatoric uncertainty , which arises due to inherent class overlap or noise in the data.
Revised: However, such models only capture data uncertainty (Gal, 2016; Malinin, 2019), also known as aleatoric uncertainty , which arises due to inherent class overlap or noise in the data.

Original: However, this does not quantify uncertainty due to the model’s inherent lack of knowledge about inputs from regions either far from the training data or sparsely covered by it, known as knowledge uncertainty , or epistemic uncertainty (Gal, 2016; Malinin, 2019).
Revised: However, this does not quantify uncertainty due to the model’s inherent lack of knowledge about inputs from regions either far from the training data or sparsely covered by it, known as knowledge uncertainty , or epistemic uncertainty (Gal, 2016; Malinin, 2019).

Original: One class of approaches for capturing knowledge uncertainty are Bayesian ensemble methods, which have recently become popular for estimating predictive uncertainty in neural networks (Depeweg et al., 2017; Gal & Ghahramani, 2016; 
Revised: One class of approaches for capturing knowledge uncertainty are Bayesian ensemble methods, which have recently become popular for estimating predictive uncertainty in neural networks (Depeweg et al., 2017; Gal & Ghahramani, 2016; Kendall et al., 2017;

Original: Kendall et al., 2017; Lakshminarayanan et al., 2017; Maddox et al., 2019; Smith & Gal, 2018).
Revised: Lakshminarayanan et al., 2017; Maddox et al., 2019; Smith & Gal, 2018).

Original: A key feature of ensemble approaches is that they allow overall uncertainty to be decomposed into data uncertainty and knowledge uncertainty within an interpretable probabilistic framework (Depeweg et al., 2017; Gal, 2016; Malinin, 2019).
Revised: A key feature of ensemble approaches is that they allow overall uncertainty to be decomposed into data uncertainty and knowledge uncertainty within an interpretable probabilistic framework (Depeweg et al., 2017; Gal, 2016; Malinin, 2019).

Original: Ensembles are also known to yield improvements in predictive performance.
Revised: Ensembles are also known to yield improvements in predictive performance.

Original: This work examines ensemble-based uncertainty-estimation for GBDT models.
Revised: This work examines ensemble-based uncertainty-estimation for GBDT models.

Original: The contributions are as follows.
Revised: The contributions are as follows.

Original: First, we consider generating ensembles using both classical Stochastic Gradient Boosting (SGB) as well as the recently proposed Stochastic Gradient Langevin Boosting (SGLB) (Ustimenko & Prokhorenkova, 2020).
Revised: First, we consider generating ensembles using both classical Stochastic Gradient Boosting (SGB) as well as the recently proposed Stochastic Gradient Langevin Boosting (SGLB) (Ustimenko & Prokhorenkova, 2020).

Original: Importantly, SGLB allows us to guarantee that the models are asymptotically sampled from a true Bayesian posterior.
Revised: Importantly, SGLB allows us to guarantee that the models are asymptotically sampled from a true Bayesian posterior.

Original: Second, we show that using SGLB we can construct a virtual ensemble using only one gradient boosting model, signiﬁcantly reducing the computational complexity.
Revised: Second, we show that using SGLB we can construct a virtual ensemble using only one gradient boosting model, signiﬁcantly reducing the computational complexity.

Original: Third, to understand the attributes of using ensembles-based uncertainty estimation in GBDT models, we conduct extensive analysis on several synthetic datasets.
Revised: Third, to understand the attributes of using ensembles-based uncertainty estimation in GBDT models, we conduct extensive analysis on several synthetic datasets.

Original: Finally, we evaluate the proposed approach on a range of real regression and classiﬁcation datasets.
Revised: Finally, we evaluate the proposed approach on a range of real regression and classiﬁcation datasets.

Original: Our results show that this approach successfully enables the detection of anomalous out-of-domain inputs.
Revised: Our results show that this approach successfully enables the detection of anomalous out-of-domain inputs.

Original: Importantly, our solution is easy to combine with any implementation of GBDT.
Revised: Importantly, our solution is easy to combine with any implementation of GBDT.

Revised: Our methods have been implemented within the open-source CatBoost library.

Revised: The code of our experiments is publicly available at https://github.com/yandex-research/GBDT-uncertainty .

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Uncertainty Estimation via Bayesian Ensembles
Revised: Uncertainty Estimation via Bayesian Ensembles

Original: In this work we consider uncertainty estimation within the standard Bayesian ensemble-based framework (Gal, 2016;
Revised: In this work we consider uncertainty estimation within the standard Bayesian ensemble-based framework (Gal, 2016;

Original: Malinin, 2019).
Revised: Malinin, 2019).

Original: Here, model parameters θ are considered random variables and a prior p ( θ ) is placed over them to compute a posterior p ( θ |D ) via Bayes’ rule:
Revised: Here, model parameters (cid:18) are considered random variables and a prior p ( (cid:18) ) is placed over them to compute a posterior p ( (cid:18) jD ) via Bayes’ rule:

Original: [Equation]
Revised: [Equation]

Original: where D = { x ( i ) , y ( i ) } Ni =1 is the training dataset.
Revised: where D = f x ( i ) ; y ( i ) g Ni =1 is the training dataset.

Original: Each set of parameters can be considered a hypothesis or explanation about how the world works.
Revised: Each set of parameters can be considered a hypothesis or explanation about how the world works.

Original: Samples from the posterior should yield explanations consistent with the observations of the world contained within the training data D .
Revised: Samples from the posterior should yield explanations consistent with the observations of the world contained within the training data D .

Original: However, on data far from D each set of parameters can yield different predictions.
Revised: However, on data far from D each set of parameters can yield different predictions.

Original: Therefore, estimates of knowledge uncertainty can be obtained by examining the diversity of predictions.
Revised: Therefore, estimates of knowledge uncertainty can be obtained by examining the diversity of predictions.

Original: Consider an ensemble of probabilistic models { P ( y | x ; θ ( m ) ) 
Revised: Consider an ensemble of probabilistic models f P ( y j x ; (cid:18) ( m ) ) g

Original: } Mm =1 sampled from the posterior p ( θ |D ) .
Revised: Mm =1 sampled from the posterior p ( (cid:18) jD ) .

Original: Each model P ( y | x , θ ( m ) ) yields a different estimate of data uncertainty , represented by the entropy of its predictive distribution (Malinin, 2019).
Revised: Each model P ( y j x ; (cid:18) ( m ) ) yields a different estimate of data uncertainty , represented by the entropy of its predictive distribution (Malinin, 2019).

Original: Uncertainty in predictions due to knowledge uncertainty is expressed as the level of spread, or “disagreement”, of models in the ensemble (Malinin, 2019).
Revised: Uncertainty in predictions due to knowledge uncertainty is expressed as the level of spread, or “disagreement”, of models in the ensemble (Malinin, 2019).

Original: Note that exact Bayesian inference is often intractable, and it is common to consider either an explicit or implicit approximation q ( θ ) to the true posterior p ( θ |D ) .
Revised: Note that exact Bayesian inference is often intractable, and it is common to consider either an explicit or implicit approximation q ( (cid:18) ) to the true posterior p ( (cid:18) jD ) .

Original: While a range of approximations has been explored for neural network models (Gal & Ghahramani, 2016;Lakshminarayanan et al., 2017;
Revised: While a range of approximations has been explored for neural network models (Gal & Ghahramani, 2016;

Original:  Maddox et al., 2019) 1 , to the best of our knowledge, limited work has explored Bayesian inference for gradient-boosted trees.
Revised: Lakshminarayanan et al., 2017; Maddox et al., 2019) 1 , to the best of our knowledge, limited work has explored Bayesian inference for gradient-boosted trees.

Original: Given p ( θ |D ) , the predictive posterior of the ensemble is obtained by taking the expectation with respect to the models in the ensemble:
Revised: Given p ( (cid:18) jD ) , the predictive posterior of the ensemble is obtained by taking the expectation with respect to the models in the ensemble:

Original: [Equation]
Revised: [Equation]

Original: The entropy of the predictive posterior estimates total uncertainty in predictions:
Revised: The entropy of the predictive posterior estimates total uncertainty in predictions:

Original: [Equation]
Revised: [Equation]

Original: Total uncertainty is due to both data uncertainty and knowledge uncertainty .
Revised: Total uncertainty is due to both data uncertainty and knowledge uncertainty .

Original: However, in applications like active learning (Kirsch et al., 2019) and out-of-domain detection it is desirable to estimate knowledge uncertainty separately.
Revised: However, in applications like active learning (Kirsch et al., 2019) and out-of-domain detection it is desirable to estimate knowledge uncertainty separately.

Original: The sources of uncertainty can be decomposed by considering the mutual information between the parameters θ and the prediction y
Revised: The sources of uncertainty can be decomposed by considering the mutual information between the parameters (cid:18) and the prediction y

Original: (Depeweg et al., 2017):
Revised: (Depeweg et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: This is expressed as the difference between the entropy of the predictive posterior, a measure of total uncertainty , and the expected entropy of each model in the ensemble, a measure of expected data uncertainty .
Revised: This is expressed as the difference between the entropy of the predictive posterior, a measure of total uncertainty , and the expected entropy of each model in the ensemble, a measure of expected data uncertainty .

Original: Their difference is a measure of ensemble diversity and estimates knowledge uncertainty .
Revised: Their difference is a measure of ensemble diversity and estimates knowledge uncertainty .

Original: Unfortunately, when considering ensembles of probabilistic regression models { p ( y | x ; θ ( m ) ) } Mm = over continuous-valued target y ∈ R , it is no longer possible to obtain tractable estimates of the (differential) entropy of the predictive posterior, and, by extension, mutual information.
Revised: Unfortunately, when considering ensembles of probabilistic regression models f p ( y j x ; (cid:18) ( m ) ) g Mm = over continuous-valued target y 2 R , it is no longer possible to obtain tractable estimates of the (differential) entropy of the predictive posterior, and, by extension, mutual information.

Original: In this cases uncertainty estimates can instead derived via the law of total variation:
Revised: In this cases uncertainty estimates can instead derived via the law of total variation:

Original: [Equation]
Revised: [Equation]

Revised: Expected Data

Revised: [Equation]

Original: This is conceptually similar to the decomposition (4) obtained via mutual information.
Revised: This is conceptually similar to the decomposition (4) obtained via mutual information.

Original: For an ensemble of probabilistic regression models which parameterize the normal distribution, and where each models yields a mean and standard-deviation, the total variance can be computed as follows: 
Revised: For an ensemble of probabilistic regression models which parameterize the normal distribution, and where each models yields a mean and standard-deviation, the total variance can be computed as follows: [Equation] Uncertainty [Equation]

Revised: Data

Original: [Equation]
Revised: [Equation]

Original: However, while these measures are tractable, they are based on only ﬁrst and second moments, and may therefore miss high-order details in the uncertainty.
Revised: However, while these measures are tractable, they are based on only ﬁrst and second moments, and may therefore miss high-order details in the uncertainty.

Original: They are also not scale-invariant, which can cause issues is the scale of prediction on in-domain and out-of-domain data is very different.
Revised: They are also not scale-invariant, which can cause issues is the scale of prediction on in-domain and out-of-domain data is very different.

Original: Gradient boosting is a powerful machine learning technique especially useful on tasks containing heterogeneous features.
Revised: Gradient boosting is a powerful machine learning technique especially useful on tasks containing heterogeneous features.

Original: It iteratively combines weak models, such as decision trees, to obtain more accurate predictions.
Revised: It iteratively combines weak models, such as decision trees, to obtain more accurate predictions.

Original: Formally, given a dataset D and a loss function L : R 2 → R , the gradient boosting algorithm (Friedman, 2001) iteratively constructs a model F : X → R to minimize the empirical risk L ( F |D ) =
Revised: Formally, given a dataset D and a loss function L : R 2 ! R , the gradient boosting algorithm (Friedman, 2001) iteratively constructs a model F : X ! R to minimize the empirical risk L ( F jD ) =

Original: E D [ L ( F ( x ) , y )] .
Revised: E D [ L ( F ( x ) ; y )] .

Original: At each iteration t the model is updated as:
Revised: At each iteration t the model is updated as:

Original: [Equation]
Revised: [Equation]

Original: where F ( t − 1) is a model constructed at the previous iteration, h ( t ) ( x ) ∈ H is a weak learner chosen from some family of functions H , and (cid:15) is learning rate.
Revised: where F ( t (cid:0) 1) is a model constructed at the previous iteration, h ( t ) ( x ) 2 H is a weak learner chosen from some family of functions H , and ϵ is learning rate.

Original: The weak learner h ( t ) is usually chosen to approximate the negative gradient − g ( t ) ( x , y ) : 
Revised: The weak learner h ( t ) is usually chosen to approximate the negative gradient (cid:0) g ( t ) ( x ; y ) : = (cid:0) @

Original: [Equation]
Revised: [Equation]

Original: − ∂L ( y

Original: [Equation]
Revised: [Equation]

Original: ∂s

Original: [Equation]
Revised: [Equation]

Original: A weak learner h ( t ) is associated with parameters φ ( t ) ∈ R d .
Revised: A weak learner h ( t ) is associated with parameters ϕ ( t ) 2 R d .

Original: We write h ( t ) ( x , φ ( t ) ) to reﬂect this dependence.
Revised: We write h ( t ) ( x ; ϕ ( t ) ) to reﬂect this dependence.

Original: The set of weak learners H often consists of shallow decision trees, which are models that recursively partition the feature space into disjoint regions called leaves.
Revised: The set of weak learners H often consists of shallow decision trees, which are models that recursively partition the feature space into disjoint regions called leaves.

Original: Each leaf R j of the tree is assigned to a value, which is the estimated response y in the corresponding region.
Revised: Each leaf R j of the tree is assigned to a value, which is the estimated response y in the corresponding region.

Original: We can write h ( x , φ ( t ) )=
Revised: We can write h ( x ; ϕ ( t ) )

Original: (cid:80) dj
Revised: =

Original:  =1 φ ( t ) j 1 { x ∈ R j } , so the decision tree is a linear function of φ ( t ) .
Revised: ∑ dj =1 ϕ ( t ) j 1 f x 2 R j g , so the decision tree is a linear function of ϕ ( t ) .

Original: The ﬁnal GBDT model F is a sum of decision trees (7) and the parameters of the full model are denoted by θ .
Revised: The ﬁnal GBDT model F is a sum of decision trees (7) and the parameters of the full model are denoted by (cid:18) .

Original: For classiﬁcation tasks, a model yields estimates data uncertainty if it is trained via negative loglikelihood and provides a distribution over class labels.
Revised: For classiﬁcation tasks, a model yields estimates data uncertainty if it is trained via negative loglikelihood and provides a distribution over class labels.

Original: However, classic GBDT regression models yield point predictions, and there has been little research devoted to estimating predictive uncertainty.
Revised: However, classic GBDT regression models yield point predictions, and there has been little research devoted to estimating predictive uncertainty.

Original: Recently, this issue was addressed by Duan et al.
Revised: Recently, this issue was addressed by Duan et al.

Original: (2020) via an algorithm called NGBoost (Natural
Revised: (2020) via an algorithm called NGBoost (Natural

Original: Gradient Boosting), which allows estimating data uncertainty .
Revised: Gradient Boosting), which allows estimating data uncertainty .

Original: NGBoost simultaneously estimates the parameters of a conditional distribution p ( y | x , θ ) over the target y given the features x , by optimizing a proper scoring rule.
Revised: NGBoost simultaneously estimates the parameters of a conditional distribution p ( y j x ; (cid:18) ) over the target y given the features x , by optimizing a proper scoring rule.

Original: Typically, a normal distribution over y is assumed and negative log-likelihood is taken as a scoring rule.
Revised: Typically, a normal distribution over y is assumed and negative log-likelihood is taken as a scoring rule.

Original: Formally, given an input x , the model F predicts two parameters of normal distribution - the mean µ and the logarithm of the standard deviation log σ .
Revised: Formally, given an input x , the model F predicts two parameters of normal distribution - the mean (cid:22) and the logarithm of the standard deviation log (cid:27) .

Original: The loss function is the expected negative log-likelihood:
Revised: The loss function is the expected negative log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: Note that θ denotes the concatenation of two parameter vectors used to predict µ and log σ .
Revised: Note that (cid:18) denotes the concatenation of two parameter vectors used to predict (cid:22) and log (cid:27) .

Original:  G ENERATING ENSEMBLES OF GDBT MODELS
Revised:  G ENERATING ENSEMBLES OF GDBT MODELS

Original: As discussed in Section 2, knowledge uncertainty can be estimated by consideringthe an ensemble of models { p ( y | x ; θ ( m ) )
Revised: As discussed in Section 2, knowledge uncertainty can be estimated by considering an ensemble of models f p ( y j x ; (cid:18) ( m ) )

Original: } M m =1 sampled from the posterior p ( θ |D ) .
Revised: g Mm =1 sampled from the posterior p ( (cid:18) jD ) .

Original: The level of diversity or “disagreement” between the models is an estimate of knowledge uncertainty .
Revised: The level of diversity or “disagreement” between the models is an estimate of knowledge uncertainty .

Original: In this section we consider three approaches to generating an ensemble of GBDT models.
Revised: In this section, we consider three approaches to generating an ensemble of GBDT models.

Original: We emphasize that this section discusses ensembles of GBDT models , where a each GBDT model is itself an ensemble of trees .
Revised: We emphasize that this section discusses ensembles of GBDT models , where a each GBDT model is itself an ensemble of trees .

Original: SGB ensembles One way to generate an ensemble is to consider several independent models generated via Stochastic Gradient Boosting (SGB).
Revised: SGB ensembles One way to generate an ensemble is to consider several independent models generated via Stochastic Gradient Boosting (SGB).

Original: Stochasticity is added to GBDT models via random subsampling of the data at every iteration (Friedman, 2002).
Revised: Stochasticity is added to GBDT models via random subsampling of the data at every iteration (Friedman, 2002).

Original: Speciﬁcally, at each iteration of (8) we select a subset of training objects D (cid:48) (via bootstrap or uniformly without replacement), which is smaller than the original training dataset D , and use D (cid:48) to ﬁt the next tree instead of D .
Revised: Speciﬁcally, at each iteration of (8) we select a subset of training objects D ′ (via bootstrap or uniformly without replacement), which is smaller than the original training dataset D , and use D ′ to ﬁt the next tree instead of D .

Original: The fraction of chosen objects is called sample rate .
Revised: The fraction of chosen objects is called sample rate .

Original: This implicitly injects noise into the learning process, effectively inducing a distribution q ( θ ) over such models.
Revised: This implicitly injects noise into the learning process, effectively inducing a distribution q ( (cid:18) ) over such models.

Original: Thus, an SGB ensemble is an ensemble of independent models { θ ( m ) }
Revised: Thus, an SGB ensemble is an ensemble of independent models f (cid:18) ( m ) g

Original: Mm =1 built according to SGB with different random seeds for sub-sampling data.
Revised: Mm =1 built according to SGB with different random seeds for subsampling data.

Original: Unfortunately, there are no guarantees on how well the distribution q ( θ ) estimates the true posterior p ( θ |D ) .
Revised: Unfortunately, there are no guarantees on how well the distribution q ( (cid:18) ) estimates the true posterior p ( (cid:18) jD ) .

Original: SGLB ensembles Remarkably, there is a way to sample GBDT models from the true posterior p ( θ |D ) via a recently proposed Stochastic Gradient Langevin Boosting (SGLB) algorithm (Ustimenko & Prokhorenkova, 2020).
Revised: SGLB ensembles Remarkably, there is a way to sample GBDT models from the true posterior p ( (cid:18) jD ) via a recently proposed Stochastic Gradient Langevin Boosting (SGLB) algorithm (Ustimenko & Prokhorenkova, 2020).

Original: SGLB combines gradient boosting with stochastic gradient Langevin dynamics (Raginsky et al., 2017) in order to achieve convergence to the global optimum even for non-convex loss functions.
Revised: SGLB combines gradient boosting with stochastic gradient Langevin dynamics (Raginsky et al., 2017) in order to achieve convergence to the global optimum even for non-convex loss functions.

Original: The algorithm has two differences compared with SGB.
Revised: The algorithm has two differences compared with SGB.

Original: First, Gaussian noise is explicitly injected into the gradients, so (8) is replaced by:
Revised: First, Gaussian noise is explicitly injected into the gradients, so (8) is replaced by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Figure]

Original: where β is the inverse diffusion temperature and I |D| is an identity matrix.
Revised: where (cid:12) is the inverse diffusion temperature and I jDj is an identity matrix.

Original: This random noise ν helps to explore the solution space in order to ﬁnd the global optimum and the diffusion temperature controls the level of exploration.
Revised: This random noise (cid:23) helps to explore the solution space in order to ﬁnd the global optimum and the diffusion temperature controls the level of exploration.

Original: Second, the update (7) is modiﬁed as:
Revised: Second, the update (7) is modiﬁed as:

Original: [Equation]
Revised: [Equation]

Original: where γ is regularization parameter.
Revised: where (cid:13) is regularization parameter.

Original: If the number of all possible trees is ﬁnite (a natural assumption given that the training dataset is ﬁnite), then the SGLB parameters θ ( t ) at each iteration form a
Revised: If the number of all possible trees is ﬁnite (a natural assumption given that the training dataset is ﬁnite), then the SGLB parameters (cid:18) ( t ) at each iteration form a

Original: Markov chain that weakly converges to the stationary distribution, also called the invariant measure:
Revised: Markov chain that weakly converges to the stationary distribution, also called the invariant measure:

Original: [Equation]
Revised: [Equation]

Original: where Γ = Γ T > 0 is an implicitly deﬁned regularization matrix which depends on a particular tree construction algorithm (Ustimenko & Prokhorenkova, 2020).
Revised: where (cid:0) = (cid:0) T > 0 is an implicitly deﬁned regularization matrix which depends on a particular tree construction algorithm (Ustimenko & Prokhorenkova, 2020).

Original: While Ustimenko & Prokhorenkova (2020) used the weak convergence to (13) to prove the global convergence, we apply this to enable sampling from the true posterior.
Revised: While Ustimenko & Prokhorenkova (2020) used the weak convergence to (13) to prove the global convergence, we apply this to enable sampling from the true posterior.

Original: For this purpose, we set [Figure]
Revised: For this purpose, we set (cid:12) =

Original: β = |D| and γ = 1 2 |D| .
Revised: jDj and (cid:13) = 1 2 jDj .

Original: For the negative log-likelihood loss function (10) the invariant measure (13) can be expressed as:
Revised: For the negative log-likelihood loss function (10) the invariant measure (13) can be expressed as:

Original: [Equation]
Revised: [Equation]

Original:  which is proportional to the true posterior distribution p ( θ |D ) under Gaussian prior p ( θ )
Revised: / p ( Dj (cid:18) ) p ( (cid:18) ) [Equation] (14) which is proportional to the true posterior distribution p ( (cid:18) jD ) under Gaussian prior p ( (cid:18) )

Original: =
Revised: =

Original: N (0 , Γ) .
Revised: N (0 ; (cid:0)) .

Original: Thus, an SGLB ensemble is an ensemble of independent models { θ ( m ) }
Revised: Thus, an SGLB ensemble is an ensemble of independent models f (cid:18) ( m ) g

Original: Mm =1 generated according to the SGLB algorithm using different random seeds.
Revised: Mm =1 generated according to the SGLB algorithm using different random seeds.

Original: In this case, asymptotically, models are sampled from the true posterior p ( θ |D ) .
Revised: In this case, asymptotically, models are sampled from the true posterior p ( (cid:18) jD ) .

Original: Virtual SGLB ensembles While SGB and SGLB yield ensembles of independent models,
Revised: Virtual SGLB ensembles While SGB and SGLB yield ensembles of independent models,

Original: their time and space complexity is M times larger than that of a single model, which is a signiﬁcant overhead.
Revised: their time and space complexity is M times larger than that of a single model, which is a signiﬁcant overhead.

Original: Consequently, generating an ensemble requires either signiﬁcantly increasing complexity or sacriﬁcing the quality by reducing the number of training iterations.
Revised: Consequently, generating an ensemble requires either signiﬁcantly increasing complexity or sacriﬁcing the quality by reducing the number of training iterations.

Original: To address this, we introduce the concept of a virtual ensemble that enables generating an ensemble using only one model.
Revised: To address this, we introduce the concept of a virtual ensemble that enables generating an ensemble using only one model.

Original: This is possible since a GBDT model is itself an ensemble of trees .
Revised: This is possible since a GBDT model is itself an ensemble of trees .

Original: However, in contrast to random forests formed by independent trees (Shaker & Hüllermeier, 2020), the sequential nature of GBDT models implies that all trees are dependent and individual trees cannot be considered as separate models.
Revised: However, in contrast to random forests formed by independent trees (Shaker & Hüllermeier, 2020), the sequential nature of GBDT models implies that all trees are dependent and individual trees cannot be considered as separate models.

Original: Hence, we use “truncated” sub-models of a single GBDT model as elements of an ensemble, as illustrated in Figure 1.
Revised: Hence, we use “truncated” sub-models of a single GBDT model as elements of an ensemble, as illustrated in Figure 1.

Original: Notably, a virtual ensemble can be obtained using any already constructed
Revised: Notably, a virtual ensemble can be obtained using any already constructed

Original: GBDT model.
Revised: GBDT model.

Original: Below we formally describe this procedure applied to SGLB models since in this case we can guarantee asymptotically sampling from the true posterior p ( θ |D ) .
Revised: Below we formally describe this procedure applied to SGLB models since in this case we can guarantee asymptotically sampling from the true posterior p ( (cid:18) jD ) .

Original: Each “truncated” model is described by the vector of parameters θ ( t ) .
Revised: Each “truncated” model is described by the vector of parameters (cid:18) ( t ) .

Original: As the parameters θ ( t ) at each iteration of the SGLB algorithm form a Markov chain that weakly convergences to the stationary distribution (14), we can consider using them as an ensemble of models.
Revised: As the parameters (cid:18) ( t ) at each iteration of the SGLB algorithm form a Markov chain that weakly convergences to the stationary distribution (14), we can consider using them as an ensemble of models.

Original: However, unlike parameters taken from different SGLB trajectories, these will have a high degree of correlation, which adversely affects the ensemble’s quality.
Revised: However, unlike parameters taken from different SGLB trajectories, these will have a high degree of correlation, which adversely affects the ensemble’s quality.

Original: This problem can be overcome by retaining only every K -th set of parameters.
Revised: This problem can be overcome by retaining only every K -th set of parameters.

Original: Formally, ﬁx K ≥ 1 and consider a set of models Θ T,K = { θ ( Kt ) , (cid:2) T 2 K (cid:3) ≤ t ≤ (cid:2) TK (cid:3) } , i.e., we add to Θ T,K every K -th model obtained while constructing one SGLB model using T iterations of gradient boosting.
Revised: Formally, ﬁx K (cid:21) 1 and consider a set of models (cid:2) T;K = f (cid:18) ( Kt ) ; [ T 2 K ] (cid:20) t (cid:20) [ T K ] g , i.e., we add to (cid:2) T;K every K -th model obtained while constructing one SGLB model using T iterations of gradient boosting.

Original: Choosing larger values of K allows us to reduce the correlation between samples from the SGLB Markov chain.
Revised: Choosing larger values of K allows us to reduce the correlation between samples from the SGLB Markov chain.

Original: Furthermore, we do not include to the ensemble the models θ ( t ) with t < T/ 2 as (14) holds only asymptotically.
Revised: Furthermore, we do not include to the ensemble the models (cid:18) ( t ) with t < T= 2 as (14) holds only asymptotically.

Original: The set of M = (cid:2) T 2 K (cid:3) models Θ T,K is called a virtual ensemble .
Revised: The set of M = [ T 2 K ] models (cid:2) T;K is called a virtual ensemble .

Original: Note that virtual ensembles behave similarly to true ensembles in the limit (for large K and T ). 
Revised: Note that virtual ensembles behave similarly to true ensembles in the limit (for large K and T ). [Figure] [Figure]

Revised: Uncertainty 0 1 2 3 4 5 6 70246 0.005 0.

Original:  Importantly, we can compute the prediction of Θ T,K with the same computation time as one θ ( T ) .
Revised: [Figure] Importantly, we can compute the prediction of (cid:2) T;K with the same computation time as one (cid:18) ( T ) .

Original: Indeed, when computing the prediction of one model, we have to sum up the predictions made by individual trees.
Revised: Indeed, when computing the prediction of one model, we have to sum up the predictions made by individual trees.

Original: To get the virtual ensemble, we only have to store the partial sums.
Revised: To get the virtual ensemble, we only have to store the partial sums.

Original: For SGLB we also have to account for regularization  (12),see Appendix A for a particular formula.
Revised: For SGLB, we also have to account for regularization (12). Formally, according to (12), for SGLB we have (cid:18) ( T ) =

Revised: ∑ Ti =1 ϵ (1 (cid:0) (cid:13)ϵ ) T (cid:0)

Revised: i ϕ ( i ) , where (1 (cid:0) (cid:13)ϵ ) T (cid:0) i appears due to shrinkage.

Revised: While computing (cid:18) ( T ) we store the partial sums (cid:18) ( T ) (cid:20) t

Revised: =

Original: A NALYSIS ON S YNTHETIC D ATA
Revised: ∑ ti =1 ϵ (1 (cid:0) (cid:13)ϵ ) T (cid:0) i ϕ ( i ) . Then, any model (cid:18) ( t ) from (cid:2) T;K can easily be obtained from the stored values: [Equation]

Revised: i

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original:  In this section, we analyze how ensemble algorithms discussed in Section 3 perform on synthetic data.
Revised: [Table] [Equation]  A NALYSIS ON S YNTHETIC D ATA In this section, we analyze how ensemble algorithms discussed in Section 3 perform on synthetic data.

Original: The aim is to understand the attributes of ensembles of GBDT models for estimating data and knowledge uncertainty in a controllable setting.
Revised: The aim is to understand the attributes of ensembles of GBDT models for estimating data and knowledge uncertainty in a controllable setting.

Original: GBDT models are usually applied to tabular data, where features are often categorical.
Revised: GBDT models are usually applied to tabular data, where features are often categorical.

Original: Hence, we ﬁrst generate a dataset with each example described by two categorical features x 1 , x 2 with 9 values each, resulting in 81 possible combinations.
Revised: Hence, we ﬁrst generate a dataset with each example described by two categorical features x 1 ; x 2 with 9 values each, resulting in 81 possible combinations.

Original: The target depends on the features as y = a ( x 1 , x 2 )+ ε ( x 1 , x 2 ) , where ε ( x 1 , x 2 ) ∼ N (0 , b ( x 1 , x 2 )) and a ( x 1 , x 2 ) , b ( x 1 , x 2 ) are some deterministic functions.
Revised: The target depends on the features as y = a ( x 1 ; x 2 ) + " ( x 1 ; x 2 ) , where " ( x 1 ; x 2 ) (cid:24) N (0 ; b ( x 1 ; x 2 )) and a ( x 1 ; x 2 ) ; b ( x 1 ; x 2 ) are some deterministic functions.

Original: The values for a ( x 1 , x 2 ) are randomly generated and the  values for b ( x 1 , x 2 ) are shown on Figure 2(a).
Revised: The values for a ( x 1 ; x 2 ) are randomly generated according to the uniform distribution over [0 ; 1] . The values for b ( x 1 ; x 2 ) are shown on Figure 2(a).

Original: We generate a heart-shaped dataset with this distribution: inside the heart (white region on Figure 2(a))
Revised: We generate a heartshaped dataset with this distribution: inside the heart (white region on Figure 2(a))

Original: there are no training points, for the other cells we have 1000 examples.
Revised: there are no training points, for the other cells we have 1000 examples per cell.

Original: [Figure] We train an ensemble of 10 SGLB models  and observe the following effects.
Revised: We train an ensemble of 10 SGLB models (each model consists of 1000 trees) and observe the following effects.

Original: First, Figure 2(b) shows  that the models correctly capture the data uncertainty in all cells containing training examples.
Revised: First, Figure 2(b) shows the total uncertainty estimated with SGLB ensemble and we see that the models correctly capture this uncertainty in all cells containing training examples.

Original: At the same time, arbitrary values can be predicted inside the heart, as no training data constrain the models’ behavior there.
Revised: At the same time, arbitrary values can be predicted inside the heart, as no training data constrain the models’ behavior there.

Original: Second, Figure 2(c) shows that we can detect regions that are out-of-domain and are not covered by  training data via estimates of knowledge uncertainty.
Revised: Second, Figure 2(c) shows that estimates of knowledge uncertainty allow us to detect regions that are out-of-domain and are not covered by the training data.

Original: Notably, the separation is perfect, as there is no trace of the original heart border.
Revised: Notably, the separation is perfect, as there is no trace of the original heart border.

Original: To further analyze ensembles of GBDT models, we apply them to a  classiﬁcation task with continuousvalued features.
Revised: To further analyze ensembles of GBDT models, we apply them to a two-dimensional classiﬁcation task with continuous-valued features.

Original: This setting is much harder for gradient boosting trees. 3 We consider a 3-class spiral dataset shown on Figure 3(a).
Revised: We consider a 3-class spiral dataset shown on Figure 3(a); and this setting is much harder for gradient boosted trees.

Original:  Figure 3(b) shows the total uncertainty estimated with SGLB ensemble, while Figure 3(c) demonstrates knowledge uncertainty.
Revised: 3 Figure 3(b) shows the total uncertainty estimated with SGLB ensemble, while Figure 3(c) demonstrates knowledge uncertainty .

Original: We observe several effects.
Revised: We observe several effects.

Original: First, total uncertainty correctly detects class boundaries and ‘sectors’ of input space outside the training dataset.
Revised: First, total uncertainty correctly detects class boundaries and ‘sectors’ of input space outside the training dataset.

Original: Second, looking at these ‘sectors’ of high uncertainty, we can better understand how GBDT ensembles work: as decision trees are discriminative functions (Bishop, 2006), if features have values outside the training domain, then the prediction is the same as for the “closest” elements in the dataset.
Revised: Second, looking at these ‘sectors’ of high uncertainty, we can better understand how GBDT ensembles work: as decision trees are discriminative functions (Bishop, 2006), if features have values outside the training domain, then the prediction is the same as for the “closest” elements in the dataset.

Original: In other words, the models’ behavior on the boundary of the dataset is further extended to the outer regions.
Revised: In other words, the models’ behavior on the boundary of the dataset is further extended to the outer regions.

Original: Third, estimates of knowledge uncertainty allow  discrimination between out-of-domain regions and class boundaries.
Revised: Third, estimates of knowledge uncertainty allow [Figure] [Figure] discrimination between out-of-domain regions and class boundaries.

Original: However, we still can see traces of the class boundaries in Figure 3(c).
Revised: However, we still can see traces of the class boundaries in Figure 3(c).

Original: A possible reason is the fact that for real-valued features, near the class borders, the splitting values may vary across all models in the ensemble, resulting in non-zero estimates of knowledge uncertainty due to decision-boundary ‘jitter’.
Revised: A possible reason is the fact that for real-valued features, near the class borders, the splitting values may vary across all models in the ensemble, resulting in nonzero estimates of knowledge uncertainty due to decision-boundary ‘jitter’.

Original: [Figure]

Original: On both “heart” and “spiral” datasets, we observed that the absolute values of knowledge uncertainty are much smaller than of data uncertainty and therefore contribute very little to total uncertainty .
Revised: On both “heart” and “spiral” datasets, we observed that the absolute values of knowledge uncertainty are much smaller than of data uncertainty and therefore contribute very little to total uncertainty .

Original: Thus, we expect that while knowledge uncertainty is especially useful for detecting anomalous inputs, the proposed approaches will contribute little to error detection on top of estimates of data uncertainty provided by single models.
Revised: Thus, we expect that while knowledge uncertainty is especially useful for detecting anomalous inputs, the proposed approaches will contribute little to error detection on top of estimates of data uncertainty provided by single models.

Original: Finally, on Figure 4, we compare the performance of ‘true’ SGLB ensembles with the virtual SGLB ensembles (vSGBL) on both the “heart” and “spiral” datasets.
Revised: Finally, on Figure 4, we compare the performance of ‘true’ SGLB ensembles with the virtual SGLB ensembles (vSGBL) on both the “heart” and “spiral” datasets.

Original: The virtual ensemble is ten times cheaper to train and infer, but the ensemble members are strongly correlated.
Revised: The virtual ensemble is ten times cheaper to train and infer, but the ensemble members are strongly correlated.

Original: We observe that on the “heart” dataset, vSGLB perfectly detects regions not covered by training data.
Revised: We observe that on the “heart” dataset, vSGLB perfectly detects regions not covered by training data.

Original: However, the absolute values of knowledge uncertaintyin this region are much smaller than for SGLB, which can be explained by the correlations.
Revised: However, the absolute values of knowledge uncertainty are much smaller than for SGLB, which can be explained by the correlations.

Original: The “spiral” dataset is more challenging for both SGLB and vSGLB.
Revised: The “spiral” dataset is more challenging for both SGLB and vSGLB.

Original: While having qualitatively similar behavior, virtual ensembles struggle to detect out-of-domain regions and separate them from class boundaries.
Revised: While having qualitatively similar behavior, virtual ensembles struggle to detect out-of-domain regions and separate them from class boundaries.

Original: In all cases, the absolute values of knowledge uncertainty are far lower than for ‘true’ SGLB ensembles.
Revised: In all cases, the absolute values of knowledge uncertainty are far lower than for ‘true’ SGLB ensembles.

Original: This shows that while vSGLB yields very cheap estimates of knowledge uncertainty by exploiting the ‘ensemble of trees’ structure of GBDT models, the quality of these estimates is inferior to ensembles of independent models.[Figure]
Revised: This shows that while vSGLB yields very cheap estimates of knowledge uncertainty by exploiting the ‘ensemble of trees’ structure of GBDT models, the quality of these estimates is inferior to ensembles of independent models.

Original:  E XPERIMENTS ON CLASSIFICATION AND REGRESSION DATASETS
Revised:  E XPERIMENTS ON CLASSIFICATION AND REGRESSION DATASETS

Original: In this section, we evaluate the performance of ensembles of GBDT models on a range of classiﬁcation and regression tasks, focusing on their ability to detect errors and out-of-domain inputs.
Revised: In this section, we evaluate the performance of ensembles of GBDT models on a range of classiﬁcation and regression tasks, focusing on their ability to detect errors and out-of-domain inputs.

Original: Experimental setup Our implementation of all GBDT models is based on the CatBoost library that is known to achieve state-of-the-art results in a variety of tasks (Prokhorenkova et al., 2018).
Revised: Experimental setup Our implementation of all GBDT models is based on the CatBoost library that is known to achieve state-of-the-art results in a variety of tasks (Prokhorenkova et al., 2018).

Original: Classiﬁcation models yield a probability distribution over binary class labels, while the regression models yield the mean and variance of the normal distribution, as discussed in Section 2.
Revised: Classiﬁcation models yield a probability distribution over binary class labels, while the regression models yield the mean and variance of the normal distribution, as discussed in Section 2.

Original: All models are trained by optimizing the negative log-likelihood.
Revised: All models are trained by optimizing the negative log-likelihood.

Original: 4 We consider SGB and SGBL single models as the baselines and examine all ensemble methods deﬁned in Section 3.
Revised: 4 We consider SGB and SGBL single models as the baselines and examine all ensemble methods deﬁned in Section 3.

Original: Ensembles of SGB and
Revised: Ensembles of SGB and

Original: SGLB models consist of 10 independent (with different seeds) models with 1K trees each.
Revised: SGLB models consist of 10 independent (with different seeds) models with 1000 trees each.

Original: The virtual ensemble vSGLB is obtained from one model with 1K trees, where each 50th model from the interval [500 , 1000] is added to the ensemble.
Revised: The virtual ensemble vSGLB is obtained from one model with 1000 trees, where each 50th model from the interval [500 ; 1000] is added to the ensemble.

Original: Thus, vSGLB has the same computational and space complexity as just one SGB or SGLB model.
Revised: Thus, vSGLB has the same computational and space complexity as just one SGB or SGLB model.

Original: Hyper-parameters are tuned by random-search, for details see Appendix B.2.
Revised: Hyper-parameters are tuned by grid search, for details see Appendix A.2.

Original: We compare the algorithms on several classiﬁcation and regression tasks (Gal & Ghahramani, 2016;
Revised: We compare the algorithms on several classiﬁcation and regression tasks (Gal & Ghahramani, 2016;

Original: Prokhorenkova et al., 2018), the description of which is available in Appendix B.3.
Revised: Prokhorenkova et al., 2018), the description of which is available in Appendix A.3.

Original: While not being the focus of the current research, Random Forest  models are naturally suitable for ensemble approaches.
Revised: While not being the focus of the current research, Random Forest (RF) models are naturally suitable for ensemble approaches.

Original: Hence, we conduct additional experiments on comparing our results with these models in Appendix D.
Revised: Hence, we conduct additional experiments and analyze the performance of ensemble approaches applied to RF models in Appendix C.

Original: Detection of errors and anomalous inputs We analyze whether measures of total and knowledge uncertainty can be used to detect errors and out-of-domain inputs.
Revised: of errors and anomalous inputs We analyze whether measures of total and knowledge uncertainty can be used to detect errors and out-of-domain inputs.

Original: Error detection can be evaluated via the Prediction-Rejection Ratio (PRR) (Malinin, 2019;
Revised: Error detection can be evaluated via the Prediction-Rejection Ratio (PRR) (Malinin, 2019;

Original: Malinin et al., 2020), which measures how well uncertainty estimates correlate with errors and rank-order them.
Revised: Malinin et al., 2020), which measures how well uncertainty estimates correlate with errors and rank-order them.

Original: The best value is 100, random is 
Revised: The best value is 100, random is 0.

Original: Out-of-domain (OOD) detection is assessed via area under the ROC curve (AUR-ROC) (Hendrycks & Gimpel, 2016).
Revised: Out-of-domain (OOD) detection is assessed via area under the ROC curve (AUCROC) (Hendrycks & Gimpel, 2016).

Original: For OOD detection, we need an OOD test-set.
Revised: For OOD detection, we need an OOD test-set.

Original: However, obtaining ‘real’ OOD examples for the datasets considered in this work is challenging, so we instead create synthetic OOD data as follows.
Revised: However, obtaining ‘real’ OOD examples for the datasets considered in this work is challenging, so we instead create synthetic OOD data as follows.

Original: For each dataset, we take its test set as the in-domain examples and sample an OOD dataset of the same size from the Naval dataset to get out-of-domain (OOD) ones.
Revised: For each dataset, we take its test set as the in-domain examples and sample an OOD dataset of the same size from the Year MSD dataset to get out-of-domain (OOD) ones.

Original: For the 
Revised: The only two exceptions are Click and Year MSD, for which we sample OOD data from the Relative location of CT slices on axial axis Data Set (Graf et al., 2011). All numerical features in

Original: Naval dataset itself, we sample OOD from the horizontally stacked Kin8nm and Protein datasets.
Revised: OOD data are normalized by the per-column mean and variance obtained on the in-domain training data.

Original: Categorical features are sampled from a uniform distribution.
Revised: For categorical features, we sample a random category uniformly at random from the set of all feature’s categories.

Original: Total and knowledge uncertainty are estimated via entropy of the predictive posterior (3) and mutual information (4) for classiﬁcation models and via total variance and variance of the mean (5) for regression ones.
Revised: Total and knowledge uncertainty are estimated via entropy of the predictive posterior (3) and mutual information (4) for classiﬁcation models and via total variance and variance of the mean (5) for regression ones.

Original: Test errors can occur due to both noise and lack of knowledge, so we expect that ranking elements by total uncertainty would give better values of PRR.
Revised: Test errors can occur due to both noise and lack of knowledge, so we expect that ranking elements by total uncertainty would give better values of PRR.

Original: Table 1 shows that measures of total uncertainty consistently yield better PPR results across all datasets.
Revised: Table 1 shows that measures of total uncertainty consistently yield better PPR results across all datasets.

Original: This is consistent with results obtained for ensembles of neural network models (Lakshminarayanan et al., 2017; Malinin, 2019; Malinin & Gales, 2019;
Revised: This is consistent with results obtained for ensembles of neural network models (Lakshminarayanan et al., 2017; Malinin, 2019; Malinin & Gales, 2019;

Original: Malinin et al., 2020).
Revised: Malinin et al., 2020).

Original: However, ensembles do not outperform single models.
Revised: However, ensembles do not outperform single models.

Original: We believe this occurs for two reasons.
Revised: We believe this occurs for two reasons.

Original: First, due to the additive nature of boosting, GDBT models are already ensembles.
Revised: First, due to the additive nature of boosting, GDBT models are already ensembles.

Original: Second, as we have discussed in Section 4, for GBDT models, estimates of knowledge uncertainty obtained via the approaches considered here contribute little to estimates of total uncertainty .[Table] 39 KU — —   — — Appetency
Revised: Second, as we have discussed in Section 4, for GBDT models, estimates of knowledge uncertainty obtained via the approaches considered here contribute little to estimates of total uncertainty .

Original: TU         17 KU — —   — —

Original: Churn

Original: TU         94 KU — —   — —

Original: Upselling

Original: TU         48 KU — —   — —

Original: Internet

Original: TU         71 KU — —   — —

Original: Kick

Original: TU         58 KU — —   — —

Original: Dataset

Original: Regression % PRR ( ↑ )

Original: Regression % AUC-

Original: [Table]

Original: ( ↑ )

Original: Protein

Original: TU         91 KU — —   — —

Original: Concrete

Original: TU         78 KU — —   — —

Original: Naval-p

Original: TV         86 KU — —   — —

Original: Energy

Original: TU         60 KU — —   — —

Original: Power-p

Original: TU         47 KU — —   — —

Original: Year TU [Table] In contrast, Table 1 shows that measures of knowledge uncertainty yield superior OOD detection performance compared to total uncertainty in terms of AUC-ROC, which is consistent with results for non-GBDT models (Malinin, 2019; Malinin & Gales, 2019;
Revised: In contrast, Table 1 shows that measures of knowledge uncertainty yield superior OOD detection performance compared to total uncertainty in terms of AUC-ROC, which is consistent with results for non-GBDT models (Malinin, 2019; Malinin & Gales, 2019;

Original: Malinin et al., 2020).
Revised: Malinin et al., 2020).

Original: The results also show that SGB and SGLB ensembles performed almost equally well, with SGLB having marginally better results.
Revised: The results also show that SGB and SGLB ensembles performed almost equally well.

Original: At the same time, virtual ensembling (vSGLB) performed consistently worse (with one exception) than SGB/SGLB ensembles, which is explained by the presence of strong correlations between the models in a virtual ensemble.
Revised: At the same time, virtual ensembling (vSGLB) performed consistently worse (with one exception) than SGB/SGLB ensembles, which is explained by the presence of strong correlations between the models in a virtual ensemble.

Original: However, inall classiﬁcation tasks, estimates of knowledge uncertainty provided by vSGLB nevertheless outperform uncertainty estimates derived from single SGB and SGLB models.
Revised: However, in classiﬁcation tasks, estimates of knowledge uncertainty provided by vSGLB nevertheless outperform uncertainty estimates derived from single SGB and SGLB models.

Original: This shows thatfor classiﬁcation tasks, useful measures of knowledge uncertainty can be derived from a single SGLB model by interpreting it as a virtual ensemble at no additional computational or memory cost . 
Revised: This shows that useful measures of knowledge uncertainty can be derived from a single SGLB model by interpreting it as a virtual ensemble at no additional computational or memory cost . For vSGLB, the difference between classiﬁcation and regression tasks can be explained by the presence or absence of categorical features. In our preliminary experiments on synthetic data, we noticed that categorical features may have a noticeable affect on the diversity of vSGLB models, and our classiﬁcation datasests contain categorical features. [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work examined principled, ensemble-based uncertainty-estimation for GBDT models.
Revised: This work examined principled, ensemble-based uncertainty-estimation for GBDT models.

Original: Two main approaches to generating ensembles of GDBT models, where each model is itself an ensemble of trees, were considered — Stochastic Gradient Boosting (SGB) and 
Revised: Two main approaches to generating ensembles of GDBT models, where each model is itself an ensemble of trees, were considered — Stochastic Gradient Boosting (SGB) and Stochastic Gradient Langevin

Original: Stochastic Gradient Langevin Boosting (SGLB).
Revised: Boosting (SGLB).

Original: Based on SGLB, we propose constructing a virtual ensemble (vSGLD) by exploiting the ‘ensemble-of-trees’ nature of GBDT models.
Revised: Based on SGLB, we propose constructing a virtual ensemble (vSGLB) by exploiting the ‘ensemble-of-trees’ nature of GBDT models.

Original: Properties of the estimates of total , data , and knowledge uncertainty derived from these ensembles were ﬁrst analyzed on synthetic data.
Revised: Properties of the estimates of total , data , and knowledge uncertainty derived from these ensembles were ﬁrst analyzed on synthetic data.

Original: It was shown that the proposed approach can successfully detect anomalous inputs and is especially successful on tabular data.
Revised: It was shown that the proposed approach can successfully detect anomalous inputs and is especially successful on tabular data.

Original: On continuous data, detecting knowledge uncertainty is still possible, but it becomes harder to differentiate it with data uncertainty due to decision-boundary ‘jitter’.
Revised: On continuous data, detecting knowledge uncertainty is still possible, but it becomes harder to differentiate it with data uncertainty due to decision-boundary ‘jitter’.

Original: Further experiments on a wide range of classiﬁcation and regression datasets showed that while ensembles of GDBT models do not offer much advantage in terms of error detection, as each model is already an ensemble of trees, they do yield useful measures of knowledge uncertainty , which enables out-of-domain detection in both regression and classiﬁcation tasks.
Revised: Further experiments on a wide range of classiﬁcation and regression datasets showed that while ensembles of GDBT models do not offer much advantage in terms of error detection, as each model is already an ensemble of trees, they do yield useful measures of knowledge uncertainty , which enables outof-domain detection in both regression and classiﬁcation tasks.

Original: Notably, measures of knowledge uncertainty , which can only be obtained via ensembles, achieve far better OOD detection performance than measures of total uncertainty .
Revised: Notably, measures of knowledge uncertainty , which can only be obtained via ensembles, achieve far better OOD detection performance than measures of total uncertainty .

Original: It is also shown that while there is little practical difference between SGB and SGLB ensembles, vSGLB performs noticeably worse.
Revised: It is also shown that while there is little practical difference between SGB and SGLB ensembles, vSGLB performs noticeably worse.

Original: However, for classiﬁcation tasks, vSGLB still yields useful measures of knowledge uncertainty at the computational time and space complexity of a single SGLB model.
Revised: However, for classiﬁcation tasks containing categorical features, vSGLB still yields useful measures of knowledge uncertainty at the computational time and space complexity of a single SGLB model.

Original: Thus, vSGLB allows us to derive the beneﬁts of an ensemble at no additional computational and memory cost. A I NFERENCE IN VIRTUAL SGLB
Revised: Thus, vSGLB allows us to derive the beneﬁts of an ensemble at no additional computational and memory cost

Original: We can compute Θ T,K with the same computation time as one θ ( T ) .

Original: According to (12), for SGLB we have θ ( T ) = (cid:80) Ti =1 (cid:15) (1 − γ(cid:15) ) T − i φ ( i ) , where (1 − γ(cid:15) )
Revised: A CKNOWLEDGMENTS

Original: T − i appears due to shrinkage.

Original: While computing θ ( T ) we store the partial sums θ ( T ) ≤ t = (cid:80) t i =1 (cid:15) (1 − γ(cid:15) ) T − i φ ( i ) . Then, any model θ ( t ) from Θ T,K can easily be obtained from the stored values: [Equation]
Revised: We would like to thank Ekaterina Ermishkina and Stanislav Kirillov for implementing the proposed methods within the CatBoost library.

Original: B E XPERIMENTAL SETUP B. O UR IMPLEMENTATION OF DATA UNCERTAINTY
Revised: A E XPERIMENTAL SETUP A. O UR IMPLEMENTATION OF DATA UNCERTAINTY

Original: As discussed in Section 2.2 of the main text, for regression we simultaneously predict the parameters µ and log σ of the Normal distribution.
Revised: As discussed in Section 2.2 of the main text [Table] for regression we simultaneously predict the parameters (cid:22) and log (cid:27) of the Normal distribution.

Original: Similarly to NGBoost, we use the natural gradients.
Revised: Similarly to NGBoost [Table] we use the natural gradients.

Original: For our loss and parameterization, the natural gradient is:
Revised: For our loss and parameterization, the natural gradient is:

Original: [Equation]
Revised: [Equation]

Revised: At each step of the gradient boosting procedure

Original: [Equation]
Revised: [Table]

Original: At each step of the gradient boosting procedure, we construct one tree predicting boththe components of g  t  , similarly to the MultiRMSE regime of CatBoost.
Revised: we construct one tree predicting both components of g ( t ) , similarly to the MultiRMSE regime of CatBoost.

Original: Recall that for classiﬁcation we optimize the logistic loss.
Revised: Recall that for classiﬁcation we optimize the logistic loss.

Original: In Table [Table] we compare our implementation with NGBoots Duan et al.
Revised: In Table 2, we compare our implementation with NGBoots (Duan et al., 2020) and Deep Ensembles (Lakshminarayanan et al., 2017) on regression datasets.

Original: and
Revised: For our implementation, we consider

Original: Deep Ensembles Lakshminarayanan et al. (2017) on regression datasets.
Revised: SGB with ﬁxed sample rate (0.5) and perform parameter tuning as described below.

Original: The best results are highlighted
Revised: The best results are highlighted.

Original: [Table]

Original: [Table]

Original: B. P ARAMETER TUNING
Revised: A. P ARAMETER TUNING

Original: For all approaches, we use random search to tune learning-rateparameter in { 0 . 1 , 0 . 01 , 0 . 001 } , tree depth in { 3 , 4 , 5 , 6 } , subsample rate in { 0 . 25 , 0 . 5 , 0 . 75 } . 
Revised: For all approaches, we use grid search to tune learning-rate in f 0 : 001 ; 0 : 01 ; 0 : 1 g , tree depth in f 3 ; 4 ; 5 ; 6 g . We ﬁx subsample to 0.5 for SGB and to 1 for SGLB.

Revised: This is done to avoid joint randomization effects of SGB sampling and SGLB noise in gradients.

Original: For SGLB-based approaches, we set diffusion-temperature = N and model-shrink-rate = 12 N .
Revised: We also set diffusion-temperature = N and model-shrink-rate = 12 N for SGLB.

Original: B. D ATASETS
Revised: A. D ATASETS

Original: The datasets are described in Table 3.
Revised: The datasets are described in Table 3.

Revised: For regression, we use standard train/validation/test splits (UCI).

Revised: For classiﬁcation, we split the datasets in proportion 65/15/20 in train, validation, and test sets.

Revised: For more details, see our GitHub repository.

Original: B. S TATISTICAL SIGNIFICANCE
Revised: A. S TATISTICAL SIGNIFICANCE

Original: For regression, weuse standard train/validation/test splits UCI and perform cross-validation to estimate statistical signiﬁcance with paired t -test.
Revised: For regression, we perform cross-validation to estimate statistical signiﬁcance with paired t -test.

Original: In the corresponding tables, we highlight the approaches that are insigniﬁcantly different from the best one (p-value > 0 . 05 ).
Revised: In the corresponding tables, we highlight the approaches that are insigniﬁcantly different from the best one (p-value > 0 : 05 ).

Original: For classiﬁcation (and the Years regression dataset) we split the datasets in proportion 65/15/ in train, validation, and test sets.
Revised: For classiﬁcation (and Year MSD), we split the datasets in proportion 65/15/20 into the train, validation, and test sets.

Original: In this case, we measure statistical signiﬁcance forNLL and error/RMSE on the test test and in the corresponding tables the approaches that are insigniﬁcantly different from the best one are highlighted.
Revised: In this case, we measure statistical signiﬁcance for and error/ on the [Table] [Table] Appetency

Original: For PRR and AUC-ROC results (for classiﬁcation and
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Upselling

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Dataset

Revised: [Table]

Revised: BostonH 2.

Revised: Concrete 3.

Revised: 5.37 Energy 1.

Revised: 0.64 Kin8nm -0.

Revised: 0.15 Naval-p -5.

Revised: 0.00 Power-p 2.

Revised: 3.64 Protein 2.

Revised: 4.02 Wine-qu 0.

Revised: [Table]

Revised: [Table]

Revised: test set.

Revised: In the corresponding tables, the approaches that are insigniﬁcantly different from the best one are highlighted.

Revised: For PRR and AUC-ROC (for classiﬁcation and Year MSD)

Original: Years) we highlight the best value.
Revised: [Table] we highlight the best value.

Original: C A DDITIONAL EXPERIMENTAL RESULTS
Revised: B A DDITIONAL EXPERIMENTAL RESULTS

Original: In Table 4, we compare ensemble approaches with single models in terms of NLL and error-rate for classiﬁcation and in terms of NLL and RMSE for regression tasks.
Revised: In Table [Table] we compare ensemble approaches with single models in terms of NLL and error rate for classiﬁcation and in terms of NLL and RMSE for regression tasks.

Original: Results for NLL demonstrate an advantage of ensembling approaches compared to single models.
Revised: Results for NLL demonstrate an advantage of ensembling approaches compared to single models.

Original: However, in many cases the difference is not signiﬁcant, which can be explained by the additive nature of boosting: averaging several tree ensembles gives another (larger) tree ensemble.
Revised: However, in some cases the difference is not signiﬁcant, which can be explained by the additive nature of boosting: averaging several tree ensembles gives another (larger) tree ensemble.

Original: Thus, improved NLL can result from the increased complexity of ensemble models.
Revised: Thus, improved NLL can result from the increased complexity of ensemble models.

Original: We can make similar conclusion from the results for
Revised: We can make similar conclusion from the results for

Original: RMSE and error rate: there is ensembling yields an advantage, but it is not so signiﬁcant.

Original: Surprisingly, for NLL optimization virtual ensembling performed better than single models indicating that SGLB can beneﬁt from using a proper re-weighting of trees which simulates an ensemble.

Original: Tables 5 extends the corresponding table from the main text to all considered datasets.

Original: [Table]

Original: [Table]

Original: Upselling 0.

Original: Internet 0.

Original: Kick 0.

Original: Dataset

Original: Regression NLL [Table] [Table] BostonH 2.
Revised: RMSE and error rate.

Original: 3.40 Kin8nm -0.

Original: Protein 2.

Original: 4.08 Concrete 3.

Original: 5.63 Naval-p -5.

Original: 0.00 Wine-qu 0.

Original: Energy 0.

Original: Power-p 2.

Original: [Table]

Original: [Table]

Original: D C OMPARISON WITH R ANDOM F OREST
Revised: C C OMPARISON WITH R ANDOM F ORESTS

Original: In our paper
Revised: In our paper

Original: [Table]
Revised: [Table]

Original: we speciﬁcally focus on uncertainty estimation in Gradient Boosted Decision Tree (GBDT) models.
Revised: we speciﬁcally focus on uncertainty estimation in Gradient Boosted Decision Tree (GBDT) models.

Original: However, some related work was done for quantifying uncertainty in random forests (Coulston et al., 2016; Shaker & Hüllermeier, 2020), which are also ensembles of decision trees.
Revised: However, some related work was done for quantifying uncertainty in random forests (Coulston et al., 2016; Shaker & Hüllermeier, 2020), which are also ensembles of decision trees.

Original: Thus, for completeness, we additionally compare our uncertainty estimation results on GBDTs to random forests and ensembles of random forests.
Revised: Thus, for completeness, we also analyze how ensemble approaches perform in combination with random forests.

Original: In these experiments, we used the scikit-learn implementation of random forests (Pedregosa et al., 2011).
Revised: In these experiments, we used the scikit-learn implementation of random forests (Pedregosa et al., 2011).

Revised: We limit the maximum depth to 10 and keep all other parameters default.

Revised: For categorical features, we use leave-one-out encoding.

Original: Unlike GBDT, where trees are added to correct the previous model’s mistakes, random forests (RF) consist of decision trees that are independently trained on bootstrapped sub-samples of the dataset.
Revised: Unlike GBDT, where trees are added to correct the previous model’s mistakes, random forests (RF) consist of decision trees that are independently trained on bootstrapped sub-samples of the dataset.

Original: Hence,in contrast to GBDT, each individual decision tree model in a random forest can be considered an element of an ensemble for the purpose ofestimating knowledge uncertainty .
Revised: Hence, for the purpose of knowledge uncertainty we can divide RF into several independent parts, each consisting of several trees.

Original: Drawing a parallel to virtual SGLB, we call this approach vRF (virtual RF) since it allows estimating knowledge uncertainty using only one trained random forest model.
Revised: Drawing a parallel to virtual SGLB, we call this approach vRF (virtual RF) since it allows estimating knowledge uncertainty using only one trained random forest model.

Revised: In our experiments with vRF, we divide one RF model into 10 independent parts each consisting of 100 trees.

Original: Similarly, one can also construct an ensemble of several independently trained random forest models, which is expected to be a stronger baseline.
Revised: Similarly, one can also construct an ensemble of several independently trained random forest models, which is expected to be a stronger baseline.

Original: However, we expect a small difference between vRF and an ensemble of random forests, as there are, a-priori, no correlations between trees both in a single model and across multiple RF models.
Revised: However, we expect a small difference between vRF and an ensemble of random forests, as there are, a-priori, no correlations between trees both in a single model and across multiple RF models.

Original: In Tables 6 and 7, we compare the predictive performance of random forests (both individual models and explicit ensembles of multiple models) to SGLB individual and ensemble models on classiﬁcation and regression tasks.
Revised: In Tables 5 and 6, we compare the predictive performance of random forests (both individual models and explicit ensembles of multiple models) to SGLB individual and ensemble models on classiﬁcation and regression tasks.

Original: The results show that generally, though with exceptions, GBDT models outperform random forest models in terms of classiﬁcation error rate and NLL.Similarly, for regressions tasks, GBDT models outperform random forests in terms of RMSE. Note that unlike GBDT models, where trees’ maximum depth is limited to 6 in our experiments, the maximum depth for RF is unlimited.
Revised: The results show that generally, though with exceptions, GBDT models outperform random forest models in terms of classiﬁcation error rate and NLL.

Original: It is known that RFs require deeper trees to achieve performance comparable to

Original: GBDTs, but this makes these models more complex in terms of memory and inference time (for the same number of trees).

Original: Note that we cannot calculate NLL for RF regression models as they are not naturally probabilistic (do not yield a predicted variance).
Revised: Note that we cannot calculate NLL for RF regression models as they are not naturally probabilistic (do not yield a predicted variance).

Original: As a result, they are unable to estimate data uncertainty , and therefore we can only obtain estimates of knowledge uncertainty .
Revised: As a result, they are unable to estimate data uncertainty , and therefore we can only obtain estimates of knowledge uncertainty .

Original: [Table] [Table] Table 8 shows that RF models are generally inferior in their estimates of uncertainty relative to
Revised: Table 7 shows that RF models are generally inferior in their estimates of uncertainty relative to

Original: GBDT models both in terms of error detection (PRR) and in terms of out-of-domain input detection (ROC-AUC).
Revised: GBDT models both in terms of error detection (PRR) and in terms of out-of-domain input detec- [Table] [Table] tion (ROC-AUC).

Original: One reason for this may be that bootstrap simply fails to generate sufﬁcient diversity in fully independent models.
Revised: However, the performance is often comparable. Notably [Table]

Original: Furthermore, error detection performance, in terms of PRR, is especially poor for  RF regression models.
Revised: as we expected [Table] for OOD detection vRF and RF give similar results.

Original: This is because we are only able to obtain estimates of knowledge uncertainty , which tend to yield worse performance than estimates of total uncertainty .
Revised: Thus, we conclude that for random forests, virtual ensembles can be safely used instead of true ones

Original: [Table]
Revised: [Table]