MABSplit: Faster Forest Training Using Multi-Armed Bandits
Abstract
Ensemble learning methods, such as random forest, are some of the most widelyused machine learning models, especially in domains that necessitate interpretability.
We present an algorithm that accelerates the training of random forest andother popular tree-based learning methods.
At the core of our algorithm is a noveland fast node-splitting subroutine, dubbed MABSplit , used to efficiently find splitpoints for constructing decision trees.
Our algorithm borrows techniques fromthe multi-armed bandit literature to judiciously determine how to allocate computational power across the set of features.
We provide theoretical guarantees the
MABSplit improves the computational complexity from linear to logarithmic in thenumber of data points.
Even on small datasets such as MNIST, our algorithm leadsto 7x faster training (an 85% reduction in training time) without any decrease intest accuracy.
We demonstrate similar speedups when our subroutine is used acrossa variety of forest-based variants, such as Extremely Random Forests and Random Patches.
We also show our algorithm can be used in both classification and regression tasks.
Finally, MABSplit outperforms existing methods in test performanceand feature importance calculation under a fixed computational budget.
Introduction
Random forest (RF) is a supervised learning technique that is widely used for classification andregression tasks [20, 10].
In RF, an ensemble of decision trees (DTs) is trained for the same predictiontask.
Typically, bootstrapping [9] is used to generate multiple samples from a set of datapoints, eachof which is used to train a separate DT.
Moreover, in the process of training each DT, a random subsetof the features is considered at each candidate split.
The prediction of the different DTs is then beaggregated to provide a single prediction.
Common aggregation schemes include majority voting inclassification tasks and averaging in regression tasks.
By aggregating the prediction of the different DTs in the ensemble, RFs tend to be more robust against noise and overfitting [37] and are capable ofcapturing multiple patterns in the data, as opposed to a single DT [40].
RF has gained tremendous popularity due to its flexibility, usefulness in multi-class classificationand regression tasks, high performance across a broad range of data types, and its relatively lowcomputational complexity [42, 38, 7, 26].
Furthermore, RF inherits the interpretability of decisiontrees, as the prediction of each individual DT can be explained through a sequence of binary decisions.
RFs have been successfully applied in contexts as varied as the prediction of legal court decisions[23], solar radiation analysis [7], and the Higgs Boson classification problem [2].
As we move intothe era of big data, a simple and flexible machine learning technique such as RF is expected to play akey role in processing large datasets – e.g., from the biological sciences, atmospheric sciences, andsocial networks – and providing accurate and interpretable predictions.
The need for training prediction models on massive datasets and doing so on lightweight hardware(such as smartphones and Internet-of-Things devices) requires the development of new algorithmsthat can deliver faster performance without sacrificing results [42].
For that reason, recent work hasproposed ways to accelerate the training of RFs, both at the algorithmic level and at the hardwarelevel.
At the algorithmic level, the focus is on the fast construction of each individual decision tree.
A decision tree is built by identifying a feature f and a threshold t that best split the data pointsaccording to the class labels.
The data points are split based on f and t , and the process is repeatedfor each resulting subset.
The key computational bottleneck is the subroutine required for identifyingthe pair ( f, t ) that provides the best split for a set of N data points, which typically requires O ( N )computation per split.
Recent proposals include computing (or estimating) f and t from a subsampleof the data points and a subset of the features, or quantizing the feature values.
The latter techniquecreates a histogram of values of each feature across the datapoints and restricts t to be at the edges ofhistogram bins.
In Subsection 1.1, we discuss these approaches in more detail.
While existing approaches provide significant speedups in the training of RFs, they are often basedon a pre-defined strategy in terms of the number of data points and number of features to consider ateach split.
As such, they are not able to adjust to specifics of the data distribution or obtain speedupsfrom adaptivity.
Moreover, when comparing different candidate features f for a split, all features aretreated equally and the quality of their split is computed based on the same number of data points.
Intuitively, this is wasteful as features that are not informative for the prediction task can be identifiedbased on a very small number of data points.
Alternatively, an adaptive scheme could better allocatecomputational resources towards a promising set of candidate features and achieve a better tradeoffbetween computational cost and accuracy.
In this work, we propose MABSplit, a fast subroutine for the node splitting problem, which adaptivelyrefines the estimate of the “quality” of each feature-threshold pair ( f, t ) as a candidate split.
In thisway, bad splitting candidates can be discarded early, which can lead to significant computationalsavings.
The main idea behind our algorithm is to cast the node-splitting task as a multi-armed banditproblem [25, 1, 4, 39], where each pair ( f, t ) is a distinct arm.
The unknown parameter of each arm,µ ft , corresponds to the quality of the split of the data points X
= { x 1 , . .
, x N } based on featuref and threshold t .
The split quality is measured in terms of how much the split would reduce labelimpurity, for a chosen label impurity measure such as Gini impurity.
An arm ( f, t ) can be “pulled”by computing the reduction of label impurity on a small subset of data points from X .
This allows usto compute an estimate ˆ µ ft and a corresponding confidence interval, which can be used in a batchedvariant of the Upper Confidence Bound (UCB) and successive elimination algorithms [25, 44] toidentify the best arm ( f, t ) .
We demonstrate the benefits of MABSplit on a variety of datasets, for both classification andregression tasks.
Even on small datasets such as MNIST, our algorithm leads to 7x faster training (an85% reduction in training time) without any decrease in test accuracy over an exact implementation of RF that searches for the optimal ( f, t ) pair via brute-force computation.
Additionally, we demonstratesimilar speedups when using MABSplit across a variety of forest-based variants, such as Extremely
Random Forests and Random Patches.
1.1 Related work
Random Forests were originally proposed by Ho [20].
In its basic formulation, a RF involvesconstructing n tree DTs, each based on a bootstrap sample of N data points and a random subset of thefeatures (known as random subspacing [21]).
More recently, the need for training RFs on large-scaledatasets has prompted the development of several techniques to accelerate the training process, bothat the software and at the hardware levels.
Software acceleration of RF: Most of the software/algorithmic acceleration techniques
are basedon accelerating the training of each individual DT.
FastForest [42] accelerates the node-splittingtask using three ideas: subsampling a pre-specified number of datapoints (subbagging) withoutreplacement, subsampling a pre-specified number of features dependent on the current number of datapoints (Dynamic Restricted Subspacing), and dividing values of a given feature into T bins where Tdepends on the number of data points at the node (Logarithmic Split-Point Sampling, inspired by thesingle-tree SPAARC algorithm
MABSplit is inspired by the ideas in FastForest, but does notrequire the number of datapoints or features to be pre-specified as these are determined adaptively.
The idea of using adaptivity to identify the best split has appeared in a few previous works.
Forexample, Very Fast Decision Tree (VFDT) [13] and Extremely Fast Decision Tree (EFDT) [29] areincremental decision tree learning algorithms in which trees can be updated in streaming settings.
Acceleration at the node-splitting task is achieved by adaptively selecting a subset of data pointssufficient to distinguish the best and second best features.
This is similar to our approach, but thesampled data points are evaluated for all features, unlike in MABSplit, which adaptively discardsunpromising feature splits.
The F-forest [16] algorithm also applies adaptivity and utilizes an upperbound on the impurity reduction of a split in order to discard candidate splits early and save oncomputation.
This is similar in spirit to the goal of MABSplit, but is based on a deterministic upperbound on the impurity reduction (as opposed to our statistical estimate) and is linear in N per splitcomputation, even when considering a fixed number of splitting thresholds per feature.
Other variations of RF have been proposed to improve training time.
Random Patches [28] buildstrees based on a subset of data points and a subset of features (which is fixed for each entire DT,unlike in RF).
Extra-Trees (ETs) [17] draws a random subset of K feature at each node and, for eachone, selects a number R of random splits, and then selects among these KR candidate splits the onethat yields the largest impurity reduction.
Other recent work attempts to accelerate RF training byidentifying the optimal number of decision trees needed in the forest [31], a form of hyperparametertuning.
The MABSplit subroutine can also be incorporated into these methods.
Hardware acceleration of RF: The training of RFs can also be significantly accelerated throughthe use of specialized hardware.
For instance, the implementation of RF available in Weka [14]allows trees to be trained at different cores and reduces forest build time.
A GPU-based parallelimplementation of RF has also been proposed in [18].
These solutions require specialized hardware(e.g.
, GPU-based PC video cards) and are inappropriate everyday users locally executing data-miningtasks on standard PCs or smartphones.
As such, it is also desirable to develop techniques to improveclassification accuracy and processing speed on standard platforms.
Preliminaries
We consider N data points { ( x i , y i ) } Ni =1 such that each x i is a M -dimensional feature vector and y iis its label.
Following standard RF literature, we consider flexible feature types such as numericalor categorical; RFs also deal naturally with missing values [22].
We also consider both categoricaldata labels for classification and numerical data labels for regression.
With a slight abuse of notation,we use X to mean either the set of indices { i } or the values { ( x i , y i ) } , with the meaning clear fromcontext.
A random forest contains n tree decision trees, each constructed using a set of bootstrapped N samples(sample with replacement) and a random set of features (typically √ M [8]) at each node for a giventree; it outputs the majority vote across trees for the classification task and average prediction acrossall trees for the regression task [20, 10].
We focus on the top-down approach of constructing DTs bychoosing a variable at each step that best splits the set of labels at a given node [37].
Specifically,consider a node with n data points X and m features F .
Let X L ,ft and X R ,ft be the left and rightchild subsets of X , split according to feature f at threshold t .
The approach finds the split that bestreduces the label impurity; i.e., it finds
[Equation]
where I ( X ) measures labels impurity of { y i } i ∈X and T f is the set of feasible thresholds for featuref .
Common impurity measures include Gini impurity or entropy for classification, and mean-squarederror (MSE) for regression [11]:
[Equation]
where K is the total number of classes, p k = 1 n (cid:80) i ∈X I ( y i = k ) is the proportion of class k in X ,and ¯ y = 1 n (cid:80) i ∈X y i is the average response value.
We note that our proposed algorithm, MABSplit,does not assume a particular structure of I ( · ) .
While the conventional RF considers all ( n − 1) possible splits at a feature f , in this work we focuson the histogram-based variation which chooses the threshold from a set of predefined values T f , e.g.,|T f | equally-spaced histogram bin edges; this variation is substantially more computationally efficient,offers comparable accuracy, and has been the approach of most state-of-the-art implementations[36, 12, 24, 43].
MABSplit
Without loss of generality, consider the classification task with K classes.
RF constructs n tree DTs,each of which requires recursively partitioning data points at a given node across a set of √ M features.
Consider a node with n data points X , m features F , and T possible thresholds for each feature T f .
We note that n and m at the given node may or may not be the same as N and M of the entire dataset.
A naive algorithm finds the best feature-threshold pairs in Eq.
(1) by evaluating the label impurityreduction for each feature-threshold over all n data points, incurring computational cost linear in n .
We present MABSplit, which can substantially reduce this complexity to sublinear in n .
Note that Eq.(1) is equivalent to
[Equation]
and so we focus on solving Eq.
Let µ ft = |X L ,ft | n
I ( X L ,ft [Equation] |X R
[Equation] ft | [Equation] I ( X R ,ft ) be the optimizationobjective for feature-threshold pair ( f, t ) .
Computing µ ft exactly is at least O ( n ) 1 , but we caninstead estimate µ ft with fewer computations by drawing n ′ <
n independent samples uniformlywith replacement from X .
As shown in Subsec.
3.1, it is possible to construct a point estimate ˆ µ ft ( n ′ )and a (1 − δ ) confidence interval (CI) C ft ( n ′ , δ ) for the parameter µ ft , where n ′ and δ determineestimation accuracy.
The width of CIs generally scales with (cid:113) log1 /δ n ′ .
To estimate the solution to
Problem (3) with high confidence, we can then choose to sample data points for different featuresto different degrees of accuracy.
Intuitively, promising features with small values of µ ft should beestimated with high accuracy, while less promising ones can be discarded without being evaluated ontoo many data points.
The specific adaptive estimation procedure is described in Algorithm 1.
It can be viewed as a batchedversion of the conventional UCB algorithm [25, 44] combined with successive elimination [15], andis straightforward to implement.
Algorithm 1 uses the set S solution to track all potential solutions to
Problem (3); S solution is initialized as the set of all feature-threshold pairs { ( f, t ) } .
Then, for eachpotential solution ( f, t ) ∈ S solution , Algorithm 1 maintains its mean objective estimate ˆ µ ft and (1 − δ )
CI C ft .
In each iteration, a new batch of data points X batch is evaluated for all potential solutions in S solution ,allowing us to update the histogram for each feature f and make the estimate ˆ µ ft more accurate.
Based on the current estimate, if a candidate’s lower confidence bound ˆ µ ft − C ft is greater thanthe upper confidence bound of the most promising candidate min f,t (ˆ µ ft + C ft ) , we remove it from
S solution .
This process continues until there is only one candidate in S solution or until we have sampledmore than n data points.
In the latter case, we know that the difference between the remainingcandidates in S solution is so subtle that an exact computation is warranted.
We then compute thosecandidates’ objectives exactly and return the best candidates in the set.
3.1 Point estimate and confidence interval for impurity metrics
We consider two widely used impurity metrics, Gini impurity and entropy, although mean estimatesand confidence intervals for other metrics can be derived similarly (more details are provided in
Appendix 3).
We consider point estimates and confidence intervals of µ ft based on a set of n ′ points,{ ( X i , Y i ) } n ′ i =1 , sampled independently and with replacement.
Let p L ,k := 1 n (cid:80) ni
=
I ( x if < t, y i = k ) and p R ,k := 1 n {+(cid:80) ni =
 I ( x if ≥ t, y i = k ) denote theproportion of the full n data points in class k and each of the two subsets (“left” and “right”),+}
1: S solution ← { ( f, t ) , ∀ f ∈ F , ∀ t ∈ T f } ▷ Set of potential solutions to Problem (3) 2: n used ←▷ Number of data points sampled 3: For all ( f, t ) ∈ S solution , set ˆ µ ft ← ∞ , C ft ← ∞ ▷ Initialize mean and CI for each arm 4: for all f ∈ F do 5:
Create empty histogram, h f , with equally spaced bins 6: while n used < n and |S solution | > 1 do 7:
Draw a batch sample of size B with replacement X batch ⊂ X 8: for all unique f in S solution do 9: for all x in X batch do 10:
Insert x f into histogram h
f
▷ Each insertion is O (1) 11: for all ( f, t ) ∈ S solution do 12:
Update ˆ µ ft and C ft based on histogram
h f
▷ Update running mean and CI in O ( T ) 13:
S solution ← { ( f, t ) : ˆ µ ft − C ft ≤ min f,t (ˆ µ ft + C ft ) } ▷ Retain only promising points 14: n used ← n used + B 15: if |S solution | = 1 then 16: return ( f ∗ , t ∗ ) ∈ S solution 17: else 18:
Compute µ ft exactly for all ( f, t ) ∈ S solution 19: return ( f ∗ , t ∗ ) =
arg min ( f,t ) ∈S solution µ ft respectively.
Notice that(cid:88)
k p L ,k = |X L ,ft | n and (cid:88) k
p R ,k = |X R ,ft | n .
Furthermore, let ˆ p L ,k := 1 n ′ (cid:80) n ′ i
=1 I ( X if < t, Y i = k ) and ˆ p R ,k := 1 n ′ (cid:80) n ′ i =
I
( X if ≥ t, Y i =
k )denote the empirical estimates of p L ,k and p R ,
k based on the n ′ subsamples.
Then, { ˆ p L ,k , ˆ p R ,k } Kk =1jointly follow a multinomial distribution (since for each random data point ( X i , Y i ) , exactly one ofthese variables is incremented) satisfying
[Equation]
Using Eqs.
(2) and (4), and the definition of µ ft after Eq.
(3), we can write
[Equation]
where we can replace the true parameters { p L ,k , p R ,k } Kk =1 with the empirical parameters{ ˆ p L ,k , ˆ p R ,k } Kk =1 in the equations above to produce the point estimate ˆ µ ft .
In Algorithm 1, aftereach batch of B data points is obtained, we update ˆ p L ,k and ˆ p R ,k , which can then be used to updateour point estimate ˆ µ ft and the corresponding CIs; the CIs of ˆ µ ft are based on standard error derivedusing the delta method [41].
As in standard applications of the delta method, these estimates ˆ µ ft areasymptotically unbiased and the confidence intervals are asymptotically valid.
Appendix 3 providesfurther details including derivation of CIs, discussion of convergence properties, and results for othermetrics such as MSE.
3.2 Algorithmic and implementation details
We considered sampling with replacement Algorithm 1 primarily for the ease of theoretical analysis.
In practice, we found sampling without replacement was more computationally efficient and did notsignificantly change the results, and was used in the actual implementation.
For DTs, in classification tasks, we allow individual DTs to provide soft votes and average theirpredicted class probabilities to determine the forest’s predicted class label, following existing approaches.
This is in contrast with hard votes, in which each DT is only permitted to produce its bestlabel and the forest’s prediction is determined by majority voting.
For the fixed-budget experimentsdescribed in 5, we terminate tree construction and do not split the next node if doing so would violateour budget constraints.
Finally, we note that a head-to-head comparison with common implementations of forest-basedalgorithms, such as Weka’s [14] or sklearn’s
[33], would be unfair due to their extensive hardwareand language-level optimizations in those implementations.
As such, we reimplemented thesebaselines in Python and focused on algorithmic improvements.
We verified the quality of our ownimplementations agreed with that of sklearn in the appendices.
The only difference between ourmodel and the baselines is the call to the node splitting subroutine (MABSplit in our model, andthe exact, brute-force solver for the baseline models); thus, any improvements in runtime are due toimprovements in the node-splitting algorithm.
This is verified by profiling the implementations andmeasuring the relative time spent in the node-splitting subroutine versus total runtime (see Appendix5).
This allows us to focus on algorithmic improvements as opposed implementation-specificoptimizations.
Our implementation of these baselines may be of independent interest.
Analysis of the Algorithm
In this section, we prove that MABSplit returns the optimal feature-threshold pair for a node splitwith high probability.
Furthermore, we provide bounds on computational complexity of MABSplit,leading to the logarithmic dependency on the number of data points n .
Consider a node with n data points X , m features F , and T possible thresholds for each feature( |T f | = T for all f ).
Suppose ( f ∗ , t ∗ ) =
arg min f ∈F ,t ∈T f µ ft is the optimal feature-threshold pairat which to split the node.
For any other feature-threshold pair ( f, t ) , define ∆ ft := µ ft − µ f ∗ t ∗ .
To state the following results, we will assume that, for a fixed feature-threshold pair ( f, t ) and n ′randomly sampled data point, the (1 − δ ) confidence interval scales as C ft ( n ′ , δ )
=
O ( (cid:113) log1 /δ
n ′ ) .
This is the case for the confidence intervals of Gini impurity and entropy in Subsec.
Withthis assumption, we state the following theorem:
Theorem 1.
Assume ∃ c 0 > 0 s.t. ∀
δ > 0 , n ′ > 0 , C ft ( n ′ , δ ) <
c 0 (cid:113) log1 /δ n ′ .
For δ = 1 n 2 mT , withprobability at least 1 − 1 n , Algorithm 1 returns the correct solution to (3) .
Furthermore, Algorithm 1uses a total of M computations, where
[Equation]
Theorem 1 is proven in Appendix 1.
Intuitively, Theorem 1 states that with high probability, MABSplitreturns the optimal feature-threshold pair at which to split the node.
The instance-wise bound Eq.
(7)suggests the computation cost of a feature-threshold pair ( f, t ) , i.e., min (cid:104) 4 c 20 ∆ 2 ft log( n 2 mT ) + B, 2 n (cid:105) ,depends on ∆ ft , which measures how close its optimization parameter µ ft is to µ f ∗ t ∗ .
Mostreasonably different features f ̸ = f ∗ will have a large ∆ ft and incur a O (log( n 2 mT ))
computationthat is sublinear in n .
In general, MABSplit takes only O (log( n 2 mT )) computations per feature-threshold pair if thereis reasonable heterogeneity among them.
As proven in Appendix 2 in [4], this is the case under awide range of distributional assumptions on the µ ft ’s, e.g., when the µ ft ’s follow a sub-Gaussiandistribution across the pairs ( f, t ) .
These assumptions ensure that MABSplit has an overall complexityof O ( mT log( n 2 mT )) , which is sublinear in the number of data points n .
However, we note that MABSplit may take O ( n ) computation per feature-threshold pair when most feature-threshold pairsare equally good, for example in highly symmetric datasets where all splits reduce the impurityequally.
Other recent work provides further discussion on the conversion between an instance-wisebound like Equation (7) and an instance-independent bound such as O ( mT log n ) [4, 44, 6, 39, 3, 5].
Finally, we note that δ is a hyperparameter governing the error rate.
It is possible to prove resultsanalogous to Theorem 1 for arbitrary δ .
Classification: MNIST
[Table]
Experimental Results
We demonstrate the advantages of MABSplit in two settings.
In the first setting, the baseline modelswith and without MABSplit are trained to completion, and training time and test performances aremeasured.
In the second setting, we consider training each forest with a fixed computational budgetand study effect of MABSplit on test performance.
Datasets: We run experiments on public datasets to validate the behavior of MABSplit, in bothclassification and regression.
For classification, we use the MNIST dataset [27] consisting of60,000 training and 10,000 test images of handwritten digits, where each black-and-white image isrepresented as a 784-dimensional vector.
For regression, we generate 200,000 datapoints with 50features, 6 of which are correlated with the targets and 44 of which are pure noise, using sklearn ’smake_regression function [32]; 160,000 datapoints are used for training and the remaining 40,000for test.
We call this dataset the Random Linear Model dataset.
The two datasets are available under MIT and BSD licenses, respectively.
Baselines: We compare the histogrammed versions of three baselines with and without MABSplit:
Random Forest (RF), Extremely Randomized Forests (ExtraTrees) [17], and Random Patches (RP)[28].
In Random Forests, each tree fits a bootstrap sample of N datapoints and only considersrandom subset of √ M features at every node split.
Extremely Randomized Forests are identical to
Random Forests except for two differences.
First, in regression problems, all features are consideredat every node split (in classification problems, we still use only √ M features).
Second, the histogramedges of a feature are randomly chosen from a uniform distribution over that feature’s minimum andmaximum value.
In classification problems, each histogram has √ M bins and in regression problems,each histogram has M bins.
These conventions follow standard implementations [32].
Note thatin Extremely Randomized Forests, the bins in a feature’s histogram need not be equally spaced.
In
Random Patches, the dataset is reduced to α n of its original datapoints and α f of its original features,where α n and α f are constants.
The reduced dataset is fixed for the training of the entire forest.
Fullsettings for all experiments are given in Appendix 6.
5.1 Wall clock time comparisons
In the first setting, we compare baseline models with and without MABSplit in terms of wall-clocktime.
Tables 1 and 2 show that MABSplit provides similar test accuracy but leads to faster trainingthan the usual training algorithm for node-splitting for almost all baselines in both classificationand regression tasks.
Across various tasks, MABSplit leads to 2x-7x faster training, a reductionof training time of 50-85%.
As the only difference between successive rows is the node-splittingsubroutine. Thus, these benefits are wholly attributable to MABSplit(see Appendix 5). Regression [Table]
MABSplit accelerates these models by approximately 2x (an 50% reduction in training time) while achieving comparable results.
We omit the number of histogram insertions in favor of wall-clock time for simplicity; unlike in classification, the different baseline regression models have widely varying histogram bin counts.
Since the histogram insertion complexity scales as O (log T ) , the comparison across models is not fair.
[Table] 5.2 Fixed budget comparisons
In the second setting, we consider training a random forest with a fixed computational budget.
Asbefore, insertion into a histogram is taken to be an O (1) operation.
This is justified if the histogram’sthresholds are evenly spaced, in which the correct bin in which to insert a value can be indexeddirectly.
If the bins are unevenly spaced, we may perform binary searches to locate the correct bin,which is O (log T ) and does not depend on n , or cache the results of these binary searches for anevenly-spaced grid across the range of the given feature’s value.
Intuitively, the MABSplit algorithm allows for splitting a given node with less data point queries andhistogram insertions than the brute force solution.
As such, when the computational budget is fixed,forests trained with MABSplit should be able to split more nodes and therefore train more trees thanforests trained with the brute force solver.
Prior work suggests that increasing the number of trees ina forest improves generalization performance by reducing variance at the cost of slightly increasedbias [19].
Tables 3 and 4 demonstrates the generalization performance of different models as the computationalbudget is held constant for classification and regression, respectively.
When using MABSplit, thetrained forests consist of more trees and demonstrate better generalization performance across allmodels.
We also apply MABSplit to compute feature importances under a fixed budget.
We follow thecommonly used approaches of computing the feature importances of multiple forests and thenmeasuring the stability of feature selection across forests [30, 34]; see Table 5. [Table] [Table] + MABSplit +
[Table] [Table]
+ [Table] [Table]
+ [Table] +
[Table] [Table] + MABSplit + [Table]
Discussions and Conclusions
[Table] In this work, we presented a novel algorithm, MABSplit, for determining the optimal feature andcorresponding feature threshold at which to split a node in tree-based learning models.
Unlike priormodels such as Random Patches, in which the hyperparameters α n and α f must be chosen manually,
MABSplit requires no tuning and only queries as much data as is needed.
Indeed, robustness tochoice of hyperparameters is one of primary appeals of algorithms like RF [35].
MABSplit avoids the expensive O ( n log n ) sort used in many existing baselines and can be used inconjunction with existing language- and hardware-specific optimizations.
MABSplit can also beused with other methods such as logarithmic split-point sampling and boosting [42].
In boosting, MABSplit has the potential advantage of only needing to update target variables on-the-fly as needed,as opposed to current boosting approaches which update target variables for the entire dataset ateach iteration.
Additionally, MABSplit may permit easier parallelization due to lower memoryrequirements than existing algorithms, which may enable greater use in edge computing.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] We have all reviewed the claims in the abstract and
introduction and confirmed that they reflect the paper’s contributions and scope.
(b) Did you describe the limitations of your work?
[Yes] We discuss the assumptions
necessary for MABSplit to work in Section 4 and also discuss failure modes
when
these assumptions are violated.
We also describe further room to improve our work in
Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss
potential negative societal impacts in the Future Work section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We have all reviewed the ethics review guidelines and ensured that
our
paper conforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] The statement of Theorem {+
+} contains a full description of assumptions, which is further expounded upon before and after the theorem statement.
(b) Did you include complete proofs of all theoretical results?
[Yes] The proof of Theorem 1 is provided in Appendix 13.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
[Yes] Yes, full specifications of all experiments are provided in Section 5 and Appendix 6, and full,
documented source code is provided to reproduce our results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Yes, full specifications of all experiments are provided in Section5 and Appendix 6, and full, documented source code is provided to reproduce ourresults.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Every numerical measurement in Tables 1, 2, 3, and 4contains confidence intervals
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Full specifications of computing
resources are provided in
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, the only existing assets we used were publicly available datasets, which we cited in Section
(b) Did you mention the license of the assets?
[Yes] We explicitly mention the licenses in
Section 5.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Yes, the only new assets created are in the form of code, which is entirely submitted in
the supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] Yes, both datasets we use are publicly available and we mention
the licenses in Section 5.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
The two datasets are the MNIST dataset and a
mathematical dataset; neither have any personally identifiable information or offensive
content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
We did not use crowdsourcing or conduct research with humansubjects.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
We did not use crowdsourcing or conduct
research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
We did not use crowdsourcing or conduct
research with human subjects.
[-All experiments were run on 
-]