SketchBoost: Fast Gradient Boosted Decision Tree for Multioutput Problems
Anonymous Author(s)
AffiliationAddress email
Abstract
Gradient Boosted Decision Tree (GBDT) is a widely-used machine learning algorithm that has been shown to achieve state-of-the-art results on many standarddata science problems.
We are interested in its application to multioutput problemswhen the output is highly multidimensional.
Although there are highly effective
GBDT implementations, their scalability to such problems is still unsatisfactory.
In this paper, we propose novel methods aiming to accelerate the training processof GBDT in the multioutput scenario.
The idea behind these methods lies inthe approximate computation of a scoring function used to find the best split ofdecision trees.
These methods are integrated into our easily customizable GPUimplementation of GBDT in Python which we call SketchBoost.
Our numericalstudy demonstrates that SketchBoost speeds up the training process of GBDT byup to over 40 times while achieving comparable or even better performance
Introduction
Gradient Boosted Decision Tree (GBDT) is one of the most powerful methods for solving predictionproblems in both classification and regression domains.
It is a dominant tool today in applicationdomains where tabular data is abundant, for example, in e-commerce, financial, and retail industries.
GBDT has contributed to a large amount of top solutions in benchmark competitions such as Kaggle.
This makes GBDT a fundamental component in the modern data scientist’s toolkit.
The main focus of this paper is the scalability of
GBDT to multioutput problems.
Such problems include multiclass classification (a classification taskwith more than two mutually exclusive classes),multilabel classification (a classification task withmore than two classes that are not mutually exclusive), and multioutput regression (a regressiontask with a multivariate response variable).
Theseproblems arise in various areas such as Finance[Obermann and Waack, 2016], Multivariate Time
Series Forecasting [Zhai, Yao, and Zhou, 2020], Recommender Systems [Jahrer, Töscher, and Legenstein, 2010], and others.
There are several extremely efficient, open-source,and production-ready implementations of gradientboosting such as XGBoost
[Chen and Guestrin,2016], LightGBM [Ke, Meng, Finley, Wang, Chen, Ma, Ye, and Liu, 2017], and CatBoost
[Prokhorenkova, Gusev, Vorobev, Dorogush, and Gulin, 2018].
Even for them, learning a GBDT model for moderately large datasets can require muchtime.
Furthermore, this time also grows with the output size of a model.
Figure 1 demonstrates howrapidly the training time of CatBoost and XGBoost grows with the output dimension.
Consequently,the number of possible applications of GBDT in the multioutput regime is very limited. GBDT
is a boosting based algorithm that ensembles decision trees as “base learners”.
At eachboosting step , a newly added tree improves the ensemble by minimizing the error of an already built
composition.
There are two possible strategies on how to use GBDT to handle a multioutput problem
• One-versus-all strategy.
Here, at each boosting step, a single decision tree is built for every output.
Consequently, every output is handled separately.
XGBoost and LightGBM use this strategy.
• Single-tree strategy.
Here, at each boosting step, a single multivariate decision tree is built for all
outputs.
Consequently, all outputs are handled together.
CatBoost uses this strategy.
The computational complexity of both strategies is proportional to the number of outputs.
Specifically,the one-versus-all strategy requires fitting a separate decision tree for each single output at eachboosting step.
The single-tree strategy requires scanning all the output dimensions (a) to estimatethe information gain during the search of the best tree structure and (b) to compute leaf values ofa decision tree with a given structure (see details in Section 2).
A straightforward idea to reducethe training time of single-tree GBDT is to exclude some of the outputs during the search of thetree structure which is the most time-consuming step of GBDT.
However, this turns out to be ratherchallenging since it is unclear what outputs contribute the most to the information gain.
In this paper,we address this problem and propose novel methods for fast scoring of multivariate decision treeswhich show a significant decrease in computational overhead without compromising the performanceof the final model.
Related work.
Many suggestions have been made to speed up the training process of GBDT.
Thefirst group of methods aims at reducing the number of data instances.
For example, Stochastic
Gradient Boosting [Friedman, 2002] chooses a random subset of samples.
LightGBM [Ke et al.
,2017] uses gradient-based one-side sampling (GOSS) which keeps the data instances with largegradients and randomly drops the instances with small gradients.
Minimal Variance Sampling(MVS)
[Ibragimov and Gusev, 2019] randomly chooses the data instances to maximize the estimationaccuracy of split scoring.
Further, the second group of methods is based on reducing the number offeatures.
One can similarly choose a random subset of features or use principle component analysisor projection pursuit to exclude weak features; see, for example, [Jimenez and Landgrebe, 1999,
Zhou, 2012, Appel et al., 2013].
LightGBM [Ke et al., 2017] uses exclusive feature bundling (EFB)where sparse features are greedily bundled together.
CatBoost [Prokhorenkova et al., 2018] replacescategorical features with numerical ones using a special algorithm based on target statistics.
Finally,the last group of methods is applied to speed up the evaluation of split candidates during the splitsearch.
The pre-sorted algorithm [Mehta et al., 1996] enumerates all possible split points on thepre-sorted feature values.
The histogram-based algorithm [Alsabti et al., 1998, Jin and Agrawal,2003b, Li et al., 2007], instead of finding the split points on the sorted feature values, bucketscontinuous feature values into discrete bins and uses these bins to construct feature histograms.
Thehistogram-based algorithm often is more efficient in both memory consumption and training speed.
To the best of our knowledge, there are only two works devoted to the efficient training of GBDTin the multioutput scenario.
Namely, Si, Zhang, Keerthi, Mahajan, Dhillon, and Hsieh [2017] and
Zhang and Jung [2021] consider models with sparse output and discuss how to utilize this sparsityto enforce the leaf values be also sparse.
Their modifications of GBDT are called GBDT-Sparseand GBDT-MO (sparse) correspondingly.
We, in turn, approach the problem of fast GBDT trainingin the multioutput regime from a different perspective.
Namely, instead of employing the modelsparsity, we, loosely speaking, approximate the scoring function used to find the best tree structureusing the most essential outputs while keeping other boosting steps without change.
The methodswe suggest are completely different from the ones mentioned above and can be applied to modelswith both dense and sparse outputs.
We emphasize that we are not focused on Extreme Multilabel
Classification problems, where output is high-dimensional and sparse as in Bhatia et al.
Mostof such problems arise in domains such as NLP, and in real-life cases, are not solved by the tree-basedmethods.
Our goal is to speed up GBDT on datasets without sparsity, and hence the reasonableoutput size is limited by a few thousand due to the model size and memory limits.
However, ourmethods can be easily combined with the sparsity utilization used in GBDT-Sparse and GBDT-MO (sparse).
Specifically, one can use our fast split search method, and then, when the tree structure isbuilt, compute the optimal leaf values with sparsity constraint (to keep only s outputs with the largestabsolute values and set other outputs to 0 ).
Contributions.
The contributions of this work can be summarized as follows
• We propose and theoretically justify three novel methods to speed up GBDT on multioutput tasks.
They are based on approximate computation of a scoring function used to find the best split ofmultivariate decision trees while keeping other boosting steps without change.
These methods aregeneric and can be easily integrated into any GBDT realization that uses the single-tree strategy.•
We implemented the proposed methods together with a basic version of the GBDT algorithm in Python.
We will refer to it as the SketchBoost.
The Python GBDT implementation seems tobe of independent interest since it does not use low-level programming languages and is easilycustomizable.
Although it is written in Python, it is fast since it works on GPU.
• We present an empirical study using public datasets that demonstrate that SketchBoost achievescomparable or even better performance compared to the existing state-of-the-art boosting toolkitsbut in remarkably less time.
Paper Organization.
First, we review the GBDT algorithm in Section 2.
Next, we propose methodsleading to a noticeable reduction in the training time of GBDT on multioutput tasks in Section 3.
Weillustrate the performance of these methods on real-world datasets in Section 4.Proofs and experimentdetails are postponed to the Supplementary Material.
Preliminaries
Let { ( x i , y i ) } ni
=1 be a dataset with n samples, where x i ∈ R
m is an m dimensional input and y i ∈ R dis a d dimensional output.
Let also F be a class of base learners, that is, functions f : R m → R d .
In Gradient Boosting, the idea of which goes back to Schapire [1990], Freund [1995], Freund and
Schapire [1997], the model F T uses T ∈ N base learners f ∈ F and is trained in an additive andgreedy manner.
Namely, at the t -th iteration, a newly added base learner f improves the quality of analready built model F t − 1 by minimization of some specified loss function l : R d × R d → R ,
[Equation]
This optimization problem is usually approached by the Newton method using the second-orderapproximation of the loss function
[Equation]
where we omitted a term independent of f ; here Ω( f ) is a regularization term, usually added to buildnon-complex models, and
[Equation]
Due to the complexity of optimization over a large set of base learners F , the problem (1) is solvedtypically in a greedy fashion which leads us to an approximate minimizer f t .
Finally, the model F t isupdated by applying a learning rate ε > 0 typically treated as a hyperparameter: F t = F t − 1 + εf t .
GBDT uses decision trees as the base learners F ; see the seminal paper of Friedman [2001].
Adecision tree is a model built by a recursive partition of the feature space into several disjoint regions.
Each final leaf is assigned to a value, which is a response of the tree in the given region.
Based onthis construction mechanism, a decision tree f can be expressed as
[Equation]
where [ predicate ] denotes the indicator function, J is the number of leaves, R j is the j -th leaf, andv j ∈ R d is the value of j -th leaf.
The problem of learning f t can be naturally divided into two separateproblems: (1) finding the best tree structure (dividing the feature space into J areas R 1 , . . .
, R J ), and(2) fitting a decision tree with a given structure (computing leaf values v 1 , . .
. , v J ).
Finding the leaf values.
Since decision trees take constant values at each leaf, for a decision tree f twith leaves R 1 , . .
, R J , we can optimize the objective function from (1) for each leaf R j separately,
[Equation]
where we employ l 2 regularization on leaf values with a parameter λ > 0 ; here I denotes the identitymatrix and ∥ · ∥ denotes the Euclidean norm.
It is worth mentioning that if the loss function l is separable with respect to different outputs, all
Hessians H 1 , . . .
, H n are diagonal.
If it is not the case, it is a common practice to purposely simplifythem to this extent in order to avoid time-consuming matrix inversion.
It is done so in most of thesingle-tree GBDT algorithms (for example, CatBoost, GBDT-Sparse, and GBDT-MO).
We will alsofollow this idea in our work.
For diagonal Hessians, the optimal leaf values can be rewritten as
[Equation]
where g i =   g
[Equation]
di   and H i =
  h 1 i . . .
[Equation] [Equation]
[Equation] [Equation]
di  
[Equation]
Finding the tree structure.
Substituting the leaf values from (3) back into the objective function,and omitting insignificant terms, we obtain
[Equation]
where
[Equation]
The function S ( · ) will be referred to as the scoring function.
To find the best tree structure, we use agreedy algorithm that starts from a single leaf and iteratively adds branches to the tree.
At a generalstep, we want to split one of existing leaves.
To do this, we iterate through all leaves, features, andthresholds for each feature (they are usually determined by the histogram-based algorithm).
For allleaves R and all possible splits for R , say R left and R
right , we compute the impurity score given by
S ( R left ) + S ( R right ) .
The best split is considered the one which achieves the largest impurity score.
This is equivalent to maximization of the information gain which is usually defined as the differencebetween values of the loss function before and after the split, that is,
[Equation]
Similar to the previous step, some simplifications can be made to speed up computation of the scoringfunction which is done a tremendous number of times.
For instance, GBDT-Sparse does not use thesecond-order information at all (Hessians are simplified to identity matrices).
In the multioutputregime of CatBoost, the second-order derivatives are left out during the split search and are used onlyto compute leaf values.
GBDT-MO uses the second-order derivatives in both steps but it increases thecomputational complexity twice (histograms for both gradients and Hessians need to be accumulated).
Sketched Split Scoring
In this section, we propose three novel methods to speed up the split search for multivariate decisiontrees.
These methods can achieve a good balance between reducing the computational complexity inthe output dimension and keeping the accuracy for learned decision trees.
They are generic and canbe used together with the methods mentioned in the Related work section that aim at reducing thenumber of sample instances, features, or split candidates.
Moreover, the proposed methods are easyto implement upon modern boosting frameworks such as XGBoost, LightGBM, and CatBoost.
As it was mentioned before, there are two “best practices” to speed up the training of a GBDT modelon multioutput tasks: (a) to totally ignore the second-order derivatives during the split search and(b) to use only the main diagonal of the second-order derivatives to compute the leaf values.
It is doneso, for example, in Catboost, one of the few boosting toolkits that use the single-tree strategy andachieve state-of-the-art results on multioutput problems.
We will also develop our work on this basis.
The proposed methods are applied at each boosting step before the search for the best tree structureand after first- and second-order derivatives (see (2)) are computed.
The key idea of the proposed methods is to reduce the number of gradient values used in the split search so that the scoring function
S from (4) or, equivalently, the information gain will not change much.
Specifically, the scoringfunction without the second-order information can be rewritten as
[Equation]
where G =   gg. .
g d... g 1 n g 2 n . . .
g dn   and v R =   [ x 1 ∈ R ] ...
[Equation]
n ∈ R ]   . Here G ∈ R n × d is the gradient matrix and v R is the indicator vector of the leaf R (its i -th coordinate isequal to
if x i ∈ R and 0 otherwise).
Note that we added the subscript to S to indicate its dependenceon the gradient matrix G .
To reduce the complexity of computing S G in d , we approximate it with
S G k for some other matrix G k ∈ R n × k with k ≪ d .
We will refer to G k as the sketch matrix and tok as the reduced dimension or sketching parameter.
We emphasize that G k is assumed to be usedonly in building histograms and finding the tree structure.
After this, the optimal leaf values of a treeare assumed to be computed fairly using the full gradient matrix G .
Further we discuss three novel methods to construct reasonably good sketches G k — Top Outputs,
Random Sampling, and Random Projections.
These methods are motivated by the minimization ofthe approximation error given by
[Equation]
Here the supremum is taken over all possible leaves R .
The reason for this choice is that wewant the proposed approximation to be universal for all splits we will possibly iterate over.
Inthe Supplementary Material, we show that the proposed methods lead to a nearly-optimal upperbounds on the proposed error.
Since the corresponding optimization problem is an instance of Integer
Programming problem, methods leading to the optimal upper bounds can be obtained only by bruteforce, which is not an option in our case.
More details will be given in the Supplementary Material.
3.1 Top Outputs
The key idea of Top Outputs is rather straightforward: to choose the columns of G with the largest
Euclidian norm.
Namely, by a slight abuse of notation, let us denote the columns of G by g 1 , . .
. , g d .
Let also i 1 , . .
, i d be the indexes which sort the columns of G in descending order by their norm,that is, ∥ g i 1 ∥ ≥ ∥ g i 2 ∥ ≥ . . .
≥
∥ g i d ∥ .
Now the full gradient matrix and its sketch can be written as
[Equation]
The parameter k here can be chosen adaptively to the norms of g 1 , . .
. , g d .
We have not consideredthis generalization here since, in our view, it will greatly complicate the algorithm.
Moreover, theadaptive choice of k may result in large values for this parameter and hence less gain in training time.
It is worth pointing out that Top Outputs is akin to the Gradient-based One-Side Sampling (GOSS),which is successively used in LightGBM; see [Ke et al., 2017].
In GOSS, data instances withsmall gradients are excluded to speed up the split search.
Similarly, Top Outputs excludes outputcomponents based on gradient values.
This method has one major drawback.
This method chooses top k output dimensions which maynot vary much from step to step.
For instance, if several columns have large norms and others havemedium norms, Top Outputs may completely ignore the latter columns during the split search.
Belowwe consider another method that deals with this problem by introducing the randomness in the choiceof output dimensions.
3.2 Random Sampling
The probabilistic approach for algebraic computations, sometimes called the “Monte-Carlo method”,is ubiquitous; we refer the reader to the monographs of Robert and Casella [2005],Mahoney [2011],and Woodruff [2014].
We consider its application to the fast split search.
The key idea of Random Sampling is to randomly sample the columns of G with probabilitiesproportional to their norms.
Namely, we define the sampling probabilities by
[Equation]
These probabilities are known to be optimal for random sampling since they minimize the varianceof the resulting estimate; see, for example, [Robert and Casella, 2005].
Further, let i 1 , . .
, i kbe independent and identically distributed random variables taking values j with probabilities p j ,j = 1 , . . .
, k .
These random variables represent indexes of the chosen columns of G .
Finally, weconsider the following sketch
[Equation]
The additional column normalization by 1 / √ kp i is needed for unbiasedness of the resulting estimate.
There is a close affinity between Importance Sampling and Minimal Variance Sampling (MVS) of
Ibragimov and Gusev [2019].
MVS decreases the number of sample instances in the split search bymaximizing the estimation accuracy of split scoring.
Our idea is the same with the only differencethat it is applied to output dimensions rather than sample instances.
Random Sampling works well especially in the extreme cases as those mentioned above.
For example,if several outputs have large weights and others have medium weights, Random Sampling will notignore the latter outputs due to randomness.
Or, if the number of outputs with large weights is largerthan k , Random Sampling will choose different output dimensions at different steps.
As a result, thecorresponding base learners will also be quite different, which usually leads to a better generalizationability of the ensemble; see Breiman [1996].
3.3 Random Projections
In the previous section, the sketch G k was constructed by sampling columns from G according tosome probability distribution.
This process can be viewed as multiplication of G by a matrix Π ,
G k = G Π , where Π ∈ R d × k has independent columns, and each column is all zero except for a 1 ina random location.
In Random Projections, we consider sampling matrices Π , every entry of whichis an independently sampled random variable.
This results in using random linear combinations ofcolumns of G as columns of G k .
This approach is based on the Johnson-Lindenstrauss (JL) lemma; see the seminal paper of Johnsonand Lindenstrauss [1984].
They showed that projections Π from d dimensions onto a randomlychosen k -dimensional subspace do not distort the pairwise distances too much.
Indyk and Motwani[1998] proved that to obtain the same guarantee, one can independently sample every entry of Πusing the normal distribution.
In fact, this is true for many other distributions; see, for example,
Achlioptas [2003].
Since there was no significant difference between distributions in our numericalexperiments, we decided to focus on the normal distribution.
In Random Projections, we consider the following sketch
[Equation]
where Π ∈ R d × k is a random matrix filled with independently sampled N (0 , k − 1 ) entries.
In the
Supplementary Material, we discuss why this choice leads to a nearly-optimal solution to the problemwe consider and why the property of preserving the pairwise distances matters here.
Random Projections has the same merits as Random Sampling since it is also a random approach.
Besides that, the sketch matrix G k here uses gradient information of all possible outputs since eachcolumn of G k is a linear combination of columns of G .
3.4 Complexity analysis.
Most of the GBDT frameworks use histogram-based algorithm to speed up split finding; see [Alsabti,
Ranka, and Singh, 1998], [Jin and Agrawal, 2003a], and [Li, Wu, and Burges, 2008].
Instead offinding the split points on all possible feature values, histogram-based algorithm buckets feature valuesinto discrete bins and uses these bins to construct feature histograms during training.
Let us say thatthe number of possible splits per feature is limited to h ≪ n (usually h ≤ 256 to store the histogrambin index using a single byte).
It is shown in [Ke et al., 2017] that in the case of a single output,splitting a leaf R with n R samples requires O ( mn R ) operations for histogram building and O ( hm )operations for split finding.
As a result, if the actual tree construction is performed using a depth-firstsearch algorithm, the complexity of building a complete tree of depth D is O
( Dnm + 2 D hm ) .
In the multioutput scenario, this complexity increases by d times: splitting a leaf R with n R samples costs
O ( mn R d + hmd ) and depth-wise tree construction costs O
( Dmnd + 2 D hmd ) .
The methods wepropose reduce the impact of d to k with k ≪ d .
They require a preprocessing step which can be done,depending on the method, in O ( ndk ) or O ( nd ) operations.
As a result, the complexity of buildinga complete tree of depth D using the depth-first search can be reduced from O ( Dmnd + 2 D hmd )to O ( nd + Dmnk + 2 D hmk ) .
Taking into account that n , m , and d can be extremely large, thesemethods may lead to a significant improvement in the training time.
Numerical Experiments
In this section, we numerically compare (a) the proposed methods from Section 3 to speed up GBDTin the multioutput regime and (b) existing state-of-art boosting toolkits supporting multioutput tasks.
Data.
The experiments were conducted on 9 real-world publicly available datasets from Kaggle,
OpenML, and Mulan 1 for multiclass ( 4 datasets) and multilabel ( 3 datasets) classification andmultitask regression ( 2 datasets).
The associated details are given in the Supplementary Material.
SketchBoost.
We implemented a version of the GBDT algorithm in Python that works only on GPUand uses Python GPU libraries such as CuPy and Numba.
It follows the classic scheme described in[Chen and Guestrin, 2016].
Implementation details are given in the Supplementary Material.
Togetherwith all the experimental results and evaluation code, it is available at link 2 .
Using SketchBoost, we evaluate the following thee algorithms for fast split search: Top Outputs(Section 3.1),
Random Sampling (Section 3.2), and Random Projections (Section 3.3).
The basicimplementation of SketchBoost without sketching strategies will be reffered as SketchBoost Full .
Baselines.
We compare SketchBoost with CatBoost (supports multiclass and, starting from September 2021, multilabel classification tasks; uses the single-tree strategy), and XGBoost (supports multiclass, but does not support multilabel classification tasks; uses the one-vs-all strategy).
There are tworeasons why we have chosen these frameworks.
First, they are commonly used among practitionersand represent two different approaches to multiouput tasks (one-vs-all and single-tree).
Second, theycan be efficiently trained on GPU on multiclass classification tasks, which allows us to compare theirtraining time with GPU-based SketchBoost.
Experiment Design.
If there is no official train/test split, we randomly split the data into trainingand test sets with ratio 80% - 20% .
Then each algorithm is trained with 5-fold cross-validation (thetrain folds are used to fit a model and the validation fold is used for early stopping).
We evaluateall the obtained models on the test set and get 5 scores for each model.
The overall performance ofalgorithms is computed as an average score.
We use the cross-entropy for classification and RMSE forregression as a performance measure, but, for the sake of completeness, we also report the accuracyscore for classification and R-squared score for regression tasks in the Supplementary Material.
For Catboost and XGBoost, we did hyperparameter optimization using the Optuna framework[Akiba, Sano, Yanase, Ohta, and Koyama, 2019].
In our implementation, we have used the samehyperparameters as for CatBoost (to speed up the experiment; we do not expect that hyperparameterswill vary much since we use the same single-tree approach).
The sketch size k was iterated throughthe grid { 1 , 2 , 5 , 10 , 20 } (or through the grid values which are less than the output dimension).
Moreinformation on experiment design is given in the Supplementary Material.
Results.
The final test scores are summarized in Table 1.
Experiments show that, in general,
SketchBoost, together with the sketching strategies, obtains results comparable or even better to thecompeting boosting frameworks.
Promisingly, there is always a sketching strategy which outperforms
SketchBoost Full.
Random Projections achieve the best scores, but Random Sampling also performsquite well.
The deterministic Top Outputs strategy scores less than other baselines everywhere.
It isalso seen that the one-vs-all strategy implemented in XGBoost leads to a worse generalization abilitythan the single-tree strategy on most multiclass datasets.
The dependence of test scores on the sketch size k for four datasets is shown in Figure 2 (for otherdatasets see the Supplementary Material).
It confirms the idea that, in general, the larger values k
[Table]
(XGBoost supports only multiclass classification tasks and hence has missing values for other tasks
[Table] [Table]
[Figure]
we take, the better performance we obtain.
Moreover, our study shows that there is a wide range ofvalues of k for which sketching strategies work well.
For most datasets, k ≤ 10 is enough to obtain aresult similar or even better to SketchBoost Full or other baselines.
An interesting observation is that Random Projections perform well even with k = 1 .
An intuitive explanation why reducing outputdimension may increase the ensemble quality is that building a tree using all outputs often leadsto bad split choices for some particular outputs.
Sketching strategies use small groups of outputs,which leads to a better tree structures for these outputs and more diverse ensemble overall.
In thisconnection, the optimal k strongly depends on the relations between the outputs in a given dataset.
SketchBoost does a good job in reducing the training time.
In Table 2 we compare SketchBoostwith CatBoost and XGBoost on GPU on the multiclass classification tasks.
One can see that thetraining time of GBDT significantly increases with the size of the output.
If a dataset is small, as, forexample, the Otto dataset ( 9 classes, 61 K rows), our Python implementation is slightly slower thanthe efficient CatBoost GPU implementation written on low-level programming languages.
But forthe Dionis dataset ( 355 classes, 416 K rows), our implementation together with a sketching strategybecomes 40 times faster than XGBoost or CatBoost without sacrificing performance.
For other tasks time comparison between SketchBoost and other baselines is not fair (since they can be trained onlyin CPU mode).
Nevertheless, we can compare time for sketching strategies with full version of SketchBoost.
The proposed sketching algorithms are always faster, and for some datasets we see thea significant speed up.
Finally, the learning curves for validation errors on some datasets are given in Figure 3.
In general,it shows that small values of k result in a slower error decay at early iterations. But if k is properlydefined, the validation error of an algorithm with sketching is comparable to the error of the fullalgorithm, and hence both algorithms need approximately the same number of steps to convergence.
This means that the sketching strategies do not result in more complex models and do not significantlyaffect the model size or inference time.
Detailed information on the number of steps to convergencefor all strategies and baselines is given in the Supplementary Material.
Comparison with GBDT-MO.
We compare SketchBoost with GBDT-MO Full and GBDT-MO(sparse) from Zhang and Jung [2021] (GBDT-Sparse from Si et al.
[2017] does not have an opensource implementation).
As sketching strategies, we consider here only Random Sampling and
Random Projection.
As the baseline, we consider CatBoost on CPU (to make it comparable to
GBDT-MO which works only on CPU).
The datasets to compare and the best hyperparameters setswere taken from the original paper.
For other experimental details please see the Supplementary
Material.
Summary results are presented in Table 3 and Table 4.
SketchBoost with sketching strategiesoutperforms other algorithms on most datasets in terms of accuracy.
GBDT-MO (sparse) is everywhereslower than GBDT-MO Full.
Furthermore, its training time is comparable to CatBoost.
Here timecomparison with SketchBoost is not fair because of the GPU training, but, as it is shown, it is ordersof magnitude faster.
Detailed results for this experiment are given in the Supplementary Material.
and RMSE for regression) ±
their standard deviation
[Table] [Table]
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See the Supplementary Material
(b) Did you include complete proofs of all theoretical results?
[Yes] See the Supplementary Material
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See theSupplementary Material
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See the Supplementary Material
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4 and the Supplementary Material
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the Supplementary Material
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See the Supplementary
Material
(b) Did you mention the license of the assets?
[Yes] We use open source assets
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See the Supplementary Material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We use open source assets
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]