A Scalable Deterministic Global Optimization Algorithm for Training Optimal Decision Tree
Abstract
The training of optimal decision tree via mixed-integer programming (MIP) has attracted much attention in recent literature.
However, for large datasets, state-ofthe-art approaches struggle to solve the optimal decision tree training problems to a provable global optimal solution within a reasonable time.
In this paper, we reformulate the optimal decision tree training problem as a two-stage optimization problem and propose a tailored reduced-space branch and bound algorithm to train optimal decision tree for the classification tasks with continuous features.
We present several structure-exploiting lower and upper bounding methods.
The computation of bounds can be decomposed into the solution of many small-scale subproblems and can be naturally parallelized.
With these bounding methods, we prove that our algorithm can converge by branching only on variables representing the optimal decision tree structure, which is invariant to the size of datasets.
Moreover, we propose a novel sample reduction method that can predetermine the cost of part of samples at each BB node.
Combining the sample reduction method with the parallelized bounding strategies, our algorithm can be extremely scalable.
Our algorithm can find global optimal solutions on dataset with over 245,000 samples (1000 cores, less than 1% optimality gap, within 2 hours).
We test 21 real-world datasets from UCI Repository.
The results reveal that for datasets with over 7, samples, our algorithm can, on average, improve the training accuracy by 3.6% and testing accuracy by 2.8%, compared to the current state-of-the-art.
Introduction
Decision tree, as a typical supervised learning algorithm with strong interpretability, has been applied in a wide range of fields.
As a rule-based inductive model, a decision tree concludes a series of decision rules to navigate tasks such as classification or regression.
The target of the decision tree is to achieve a low prediction error while maintaining the low complexity of the decision tree structure.
Therefore, modelling a decision tree can be treated as an optimization problem that minimizes the training error and the tree complexity.
However, even constructing an optimal decision tree with binary class is NP-complete [Laurent and Rivest, 1976].
Since the first establishment of the decision tree algorithm by [Morgan and Sonquist, 1963], many heuristic algorithms, such as ID3 [Quinlan, 1986],
C4.5 [Quinlan, 1996] and CART [Breiman et al., 1984], are proposed to achieve the optimization target.
However, these heuristic algorithms inevitably fall into sub-optimal solutions.
A recent study based on a comprehensive benchmark shows that global solutions lead to average absolute improvements in test accuracy of 1 - 5% [Bertsimas and Dunn, 2017].
Several formulations and algorithms have been proposed to learn optimal decision trees through mixed integer programming (MIP)
[Verwer and Zhang, 2017, 2019, Aghaei et al., 2020, Günlük et al., 2021], itemset mining [Nijssen and Fromont, 2007, Aglin et al., 2020], tailed branch-and-bound with dynamic programming [Hu et al., 2019, Lin et al., 2020] or SAT [Narodytska et al., 2018, Avellaneda, 2020, Ordyniak and Szeider, 2021, Hu et al., 2020, Shati et al., 2021].
However, a majority of these works can only address binary features.
Binary encoding (converting continuous features into binary features) may lead to an enormous feature space, and may also lose information on the relationship between values of numerical features.
Besides the works using binary-encoded features, the other research direction that focuses on directly addressing numerical features through MIP is rarely explored.
Bertsimas and Dunn [2017] propose a seminal work on modelling the optimal classification tree (OCT) as a MIP problem, which directly address continuous features.
This approach avoids the drawback of binary encoding and can be extended naturally to the multivariate (oblique) decision tree problem [Bertsimas and Dunn, 2019].
Recently, Zhu et al.
[2020] present an SVM-based MIP formulation for learning optimal multivariate decision trees.
This work also presents a sample selection method to select a subset of samples and use it to train the decision tree.
Although the authors claim that the sample selection method can handle large datasets, the decision tree learned on the subset is not guaranteed to be the optimal one of the original dataset.
The above MIP based approaches all rely on off-the-shelf MIP solvers such as CPLEX [Cplex, 2020] and Gurobi [Optimization, 2020], and have reported directly addressing datasets with up to 7,000 samples.
Bertsimas and Dunn [2017] mentioned in their article that "datasets with tens of thousands of points were tested, but the time required for our methods to generate highquality solutions was prohibitive".
The main obstacle preventing the solvable problem from scaling to large datasets is that all these solvers implement a classical branch-and-bound (BB) scheme, which needs to branch on all binary variables to guarantee convergence, the dimension of which expands linearly with the size of the dataset.
In this paper, we propose a scalable deterministic global optimization algorithm for training optimal decision tree on classification task with numerical features.
The first contribution of our work is that we reformulate the optimal decision tree training as a two-stage optimization problem.
The stochastic programming community has made great progress in exploiting the structure of this type of problems [Karuppiah and Grossmann, 2008, Li and Grossmann, 2019].
Recently, Cao and Zavala [2019] prove that for a two-stage problem with continuous second stage variables, branching only on the first stage variables can guarantee the convergence under mild assumptions.
Many machine learning problems can be modeled as a two-stage problem and can apply this reduced-space BB method [Hua et al., 2021, Shi et al., 2022].
Although for the decision tree training problem, there are both binary and continuous variables in the second stage (i.e., sample specific variables, such as the predicted class of a sample), we prove that the convergence of our branch-and-bound algorithm can still be guaranteed by only branching on the first-stage variables (i.e., variables that represents the structure of the decision tree), the dimension of which is independent of the number of samples.
Secondly, we provide several bounding strategies to construct the lower and upper bound of each BB node.
Our new
MIP formulation consists of a decomposable structure that can be directly parallelized when solving the lower bounding problem.
We also present a tight and efficient MIP formula to obtain a lower bound quickly.
Thirdly, we develop a sample reduction method that can predetermine the costs of some samples before the lower bound calculation at each BB node.
Such a method can be combined with the lower bound calculation to further speed up the solution process, especially for large-scale datasets.
The ablation test (Appendix D) performed on datasets reveals a consistant effectiveness on improving the computation speed of the BB process.
Our experiments show that some BB nodes (e.g., when training the dataset B ANKNOTE ) can even have more than 99% of samples to be determined before computing the lower bound.
With these strategies, our tailored reduced-space BB algorithm can scale well on large datasets.
For large datasets, the optimal solution found by our algorithm can on average reduce the optimal cost by 12 .
5% and improve the training accuracy by 3 . 6% and testing accuracy by 2 . 8% , compared to the current state-of-the-art [Bertsimas and Dunn, 2019].
Remarkably, our work is the first that successfully trains a decision tree to global optimum on a dataset with 245,000 samples (1000 cores, less than 1% optimality gap, within 2 hours). We also improve the test accuracy by 1.7% on a dataset with almost one million samples (928,991 samples, 1000 cores, under 56 .
2% optimality gap within 4 hours), compared with CART and OCT.
We highlight that we also report the optimality gap as a criteria to evaluate the goodness of the optimal solution, which is not presented in the majority of MIP based works.
Bertsimas and Dunn [2019] states that the current MIP solvers struggle to provide a reasonable optimality gap for their OCT problem.
It is also aligned with our numerical analysis that OCT still retains over 90% optimality gap on average for large datasets.
At the same time, our algorithm can provide a practical gap on most of the dataset.
Although OCT could find a better solution than CART at the early stage of the solving process, our results reveal that an even better solution could be discovered later and can result in better testing accuracy on large datasets.
In the following discussion of the paper, to avoid the ambiguity, we denote the node for the decision tree as decision node and leaf node and the node for the branch-and-bound algorithm as BB node .
Optimal Decision Tree with Decomposible Structure
In this work, we focus on training an optimal decision tree for the task of classification.
Most part of the formulation presented here follows the work of Bertsimas and Dunn [2019].
However, in our formulation, we do not consider the constraint about the minimum number of samples required in the leaf node.
Let S
= { 1 , · · · , n } be the sample set and K = { 1 , · · · , K } be the class set.
For a given data set X
= { x s | x s ∈ R P , s ∈ S} with corresponding label set Y =
{ y s | y s ∈ { 0 , 1 } K , s ∈ S} , where P is the number of features and K is the number of class.
Initially, we scale each feature of the dataset to the range between 0 and 1.
We seek to find an optimal decision tree model
F : R P −→ { 0 , 1 } K with maximum depth D , such that:
[Equation]
is minimized.
Here, E ( · ) measures the misclassification error, R ( · ) is the regularization function that measure the complexity of the optimal decision tree model and λ is the complexity parameter.
We denote t ∈ { 1 , · · · , T } as the node of the tree, where T = 2 D +1 − 1 is the size (total number of nodes) of the tree.
Following the definition of Bertsimas and Dunn [2017], let p ( t ) =
⌊ t/ 2 ⌋ be the parent decision node of t , A L ( t ) as the set of ancestors of t whose left branch is followed on the path from root node to node t , and A R ( t ) as the set of ancestors of t whose right branch is followed on the path from root node to node t .
The tree node consists of two sets: decision node with t ∈ T D = { 1 , · · · , ⌊ T/ 2 ⌋} and leaf node with t ∈ T L = {⌊ T/ 2 ⌋ + 1 , · · · , T } .
The training of optimal decision tree can be treated as an optimization problem (Problem 2):
[Equation]
d t ≤ d p ( t ) , ∀ t ∈ T D (2i) 0 ≤ L s ≤(2j) a jt , d t ∈ { 0 , 1 } , 0 ≤ b t ≤ 1 ∀ t ∈ T D , j ∈ { 1 , · · · , P } (2k)
z st , c kt ∈ { 0 , 1 } , ∀ t ∈ T L (2l)
[Equation]
Here, variable a, b, c, d describe the structure of decision tree.
Specifically, variable d t on node t ∈ T D determines whether a decision node splits or not.
We track such split with variable a t =
[ a 1 t , · · · , a Pt ] T ∈ { 0 , 1 } P and b t ∈ [0 , 1] .
The prediction of each leaf node is controlled by the class indicator c kt ∈ { 0 , 1 } , ∀ t ∈ T L , k ∈ K .
Variable z st , ∀ t ∈ T L , s ∈ S represents whether sample s falls into leaf t .
Variable L s ∈ [0 , 1] represents the loss of sample s .
y sk ∈ { 0 , 1 } is the element of vector y s which represents whether sample s is labeled k .
ˆ L is the parameter to normalize the misclassification with the baseline accuracy.
ϵ , ϵ max , and ϵ min are constants to maintain the calculation stability of problem 2.
Their values are generated following Bertsimas and Dunn [2019].
A detailed explanation of constraints can be found in Appendix A at supplementary materials.
Reduced-space Branch-and-Bound Algorithm
In this section, we model Problem 2 as a two-stage optimization problem and propose a reduced-space branch-and-bound (BB) algorithm with properly designed lower and upper bounds.
Algorithm(presented in Appendix B) elaborates the details of our algorithm.
The algorithm starts from the initial feasible region M 0 .
It narrows the gap between the lower and upper bound by partitioning
M 0 recursively into smaller regions and pruning any regions that are verified to contain no optimal solution.
In each iteration with particular sub-region M , lower bound β ( M ) and upper bound α ( M ) are computed.
If the gap between β and α is smaller than ϵ , Algorithm 1 terminates.
Otherwise M is partitioned into two disjoint sub-region with some branching strategies explained in Appendix B.2.
A critical feature of our algorithm is that it can ensure the convergence by only branching on the first stage variables (i.e. a, b, c, d ).
Cao and Zavala [2019] prove that for a two-stage optimization problem, if the second stage variables are continuous, then the BB algorithm can converge by branching only on the first stage variables.
Although the decision tree training problem is different since we have both continuous and binary second stage variables, we can still prove its convergence.
Theorem 1.
Algorithm 1 converges to the global optimum, in the sense that
[Equation]
The proof of Theorem 1 is presented in Appendix B.1.
3.1 Two Stage Optimization Problem
Note that variables a, b, c, d describe the structure of decision tree and are the same for all sample, while variables z and L are sample-specific variables and describe the allocation and cost of a specific sample.
Therefore, Problem 2 can be reformulated as a two-stage optimization problem in the following form:
[Equation]
where we denote m
=
( a, b, c, d ) as all first-stage variables and Q s ( · ) represents the optimal value of the second-stage problem:
[Equation]
The closed set M 0 :=
[ m l , m u ] denotes the region of the first stage variables.
( · ) l , ( · ) u represent the lower and upper bound of each variable.
In each BB node with a specific partition M ⊆ M 0 , we solve the problem f ( M )
=
min m ∈ M (cid:80) s ∈S Q s ( m ) .
3.2 Lower Bounding Problem
Basic lower bounding problem: the problem we solve at each BB node has an implicit constraint.
That is, all samples share the same first stage variables (i.e., tree structure).
This is called nonanticipativity constraints in the stochastic programming community.
We can obtain a basic lower bound by relaxing this constraint and solve the resulting problem:
[Equation]
such a problem can be easily decomposed into n subproblems β s ( M ) :=
min m ∈ M Q s ( m ) with β ( M ) :
= (cid:80) s ∈S β s ( M ) .
The optimal value of β s can be calculated through Algorithm 2 (presented in Appendix C) by enumerating all possible leaf nodes that sample s can fall into, without the need to solve any optimization problems explicitly.
The complexity of Algorithm 2 is O ( |T D | + |T L | ) .
Since the depth of the tree is typically small for the sake of interpretability, the speed of Algorithm 2 can be executed much faster than other methods.
Theorem 2.
With given bound M =
[ m l , m u ] , Algorithm 2 finds all possible leaf nodes that sample s can fall into and gives the global optimal value of β s ( M ) .
The proof of Theorem 2 is in Appendix C.1.
Relaxed MIP problem: One byproduct of Algorithm 2 is T z s ⊆ T L , the set of all possible leaf nodes that sample s can fall into.
Figure 1 in Section 4 provides an example on the determination of T z s .
We have z st = 0 , ∀ t ̸∈ T z s .
For each sample, basic lower bounding method computes the lower bound by comparing the label y sk
with the prediction c kt of each leaf node in t ∈ T z s .
Note here basic lower bounding method relaxes the non-anticipativity constraints on c and allow different values of c for each sample.
We can enforce this constraint to generate a tighter lower bound by solving the following MIP problem:
[Equation]
where d l is the lower bound of variable d .
Since the only number of integer variables in Problemis c , the dimension of which is typically small and independent of the number of samples, solving
Problem 7 is much easier than the original problem.
Because Problem 7 is based on the T z s calculated from the basic lower bounding method but enforces non-anticipativity constraints on c , we have:
Proposition 3.
β [Equation] M ) [Equation]
Notice that when the variable c is fixed ( c l = c u in M ) at a BB node, β ( M ) =
β R ( M ) holds and we do not need to compute β R ( M ) .
Group decomposition : for the basic lower bounding method, we treat each sample as one subproblem.
One natural extension is to assign multiple samples to a subproblem.
Specifically, we define the group set G = { 1 , · · · , G } , and we partition S into G groups {S 1 , · · · , S G | G (cid:83) g =1 S g = S and
S i ∩ S g = ∅ , ∀ i, g ∈ G , i ̸
= g } , then we can obtain a lower bound from:
[Equation]
which can be decomposed into G subproblems with β G ( M )
= (cid:80) g ∈G β Gg ( M ) and β Gg ( M ) :
= min m g ∈ M Q g ( m g ) .
Here each subproblem g can be obtained by replacing s ∈ S in the original problem with s ∈ S g .
The non-anticipativity constraints is still enforced within each subproblem (i.e. a, b, c, d are the same for all samples within the group), while the non-anticipativity constraints between groups are relaxed.
Hence, the β G provides a tighter lower bound than the basic lower bound β .
Proposition 4.
[Equation]
The fewer groups we split, the tighter is the lower bound.
However, the solution time of each subproblem is also longer.
Therefore, there is a trade-off between the efficiency and tightness when setting the number of groups.
Besides this, the way of assigning samples to groups also influences the quality of lower bounds.
Finding the best grouping scheme is itself NP-hard.
In our implementation, we adopted the heuristic results (e.g. CART) to navigate the assignment of groups.
First, we divide all samples into raw groups based on leaves they fall under from the heuristic solution.
Second, in each raw group, a k -means clustering is launched.
Finally, the samples for each cluster at each raw group are evenly assigned to G groups.
In this way, samples are approximately evenly distributed among different groups.
We keep the group assignment the same for all BB nodes.
3.3 Upper Bounding Problem
The upper bound can be easily obtained by choosing any arbitrary first stage solution ˆ m ∈ M .
Let α ( M ) be the upper bounding problem, we have α ( M ) =
(cid:80) s ∈S Q s ( ˆ m ) .
The calculation of α ( M ) is the same as the evaluation of a decision tree ˆ m , which is computationally cheap.
In our implementation, at the root BB node, we use CART to provide a feasible solution ˆ m for the initial upper bound.
We also propose two heuristics in each descendent BB node to enable a fast search for an optimal solution.
First, the optimal solution to each subproblem directly provides a feasible solution.
We calculate the corresponding cost and pick the most optimal one to be the upper bound.
This method does not require additional computational time to generate ˆ m .
However, it could be affected by the group assignment.
The second method applies bootstrapping to randomly generate a small subset of samples and train a decision tree for the subset.
This "bootstrapped" subproblem provides a feasible solution and introduces more possibilities for finding a better upper bound.
Sample Reduction
During the calculation of the basic lower bound, Algorithm 2 reduces the possible leaf nodes that sample s can fall into from T L to T z s .
Comparing the label y sk
with the range of c of each leaf node in t ∈ T z s , the loss of some samples can be determined.
Specifically, suppose k is the label of sample s (i.e. y sk = 1 ).
Equation 9 checks the loss of sample s under M .
[Equation]
Figure 1 provides an example to illustrate the determination of T z s and the loss determination of a sample s under some conditions of c l , c u .
If the loss of sample s is determined at the current BB node, we no longer need to take this sample into consideration in the lower bound calculation, resulting in sample reduction.
Moreover, determined loss holds for all descendent BB nodes.
Therefore, the determination of some sample’s loss can significantly reduce the computational load, especially when the level of the search tree goes deep.
With sample reduction, many of these samples can determine their loss without being involved in the solution of any optimization problem.
Indeed, for dataset
S KIN -S EGMENTATION (one of the largest test dataset in our experiment with n = 245 , 057 ), on average, 63 .
6% of the samples can be determined among all BB nodes.
For some BB nodes, 99 . 6% of samples can have their loss determined.
Such a strategy significantly improves node exploration efficiency.
We notice that a similar "data selection" method is proposed in [Zhu et al., 2020].
A major difference between their method and our "cost determination" method is that the solution found from the "selected" data subset can not guarantee the global optimum of the original dataset.
In contrast, in our method, the loss of removed samples is not deleted but pre-determined and added to the final total loss calculation.
Hence, the optimal solution we obtain is the same as the optimal solution of the original problem.
Improvements on the calculation of lower bound: define the sample set with determined loss as S dt and those with undetermined loss as S ud , such that |S dt | + |S ud | = n .
Here, we analyze the effects of sample reduction combined with the reach-leaf determination ( T z s ) on three lower bound strategies.
Initially, the calculation of the basic lower bounding problem requires running Algorithmfor n times.
With sample reduction, the basic lower bounding problem (Problem 6) can be rewritten as:
[Equation]
Here, L s , ∀ s ∈ S dt is already determined at the parent BB node via the sample reduction.
Thus, the computing load of Problem 6 can be reduced from O ( n ) to O
( |S ud | ) .
Since M of a descendent BB node is a subset of its parent BB node, T z s can also be inherited from the parent BB node, and the calculation of T z s only need to be executed on samples in S ud .
Similarly, for the lower bound from relaxed MIP, the number of samples involved in the optimization is reduced from n to | S ud | .
Finally, for the group decomposition based lower bounding strategy, the number of samples involved in each subproblem is reduced from |S g | to |S ud ∩ S g | .
The detailed formulations of these lower bounding methods after sample reduction can be found in Appendix D. [Equation] [Figure] [Equation]
[Table]
[Figure]
Sample 1 : since x 1 [1] ≥ b u 1 , x 1 can only fall into the right descendent of the root decision node, and as x 1 [1] ∈ [ b l 3 , b u 3 ] , leaf node 3 and 4 can be reached by sample 1 and thus T z 1 = { 3 , 4 } . Since c l 2 t = c u 2 t = 1 , ∀ t ∈ T z 1 (colored in blue), which is identical to y 12 , we have L 1 = 0 .
Sample 2 : since x 2 [1] ∈ [ b l 1 , b u 1 ] , x 2 can fall into both descendent of the root decision node, However, as x 2 [1] ≥ b u 2 , leaf node 1 can not be reached by sample 2 , and thus T z 2 = { 2 , 3 , 4 } .
Since y 2 is inconsistent with all the [ c lt , c ut ] , ∀ t ∈ T z 2 , we determine that L 2 = 1 .
Sample 3 : since x 3 [1] ∈ [ b lt , b ut ] , ∀ t ∈ T D , all leaf node can be reached by sample 3, T z 3 = { 1 , 2 , 3 , 4 } .
Sample 3 can have several possible loss values if fall into different leaf node in T z 3 .
Here, variable c is fixed in all leaves except leaf 1 (colored in red).
Therefore, we are unable to determine L 3 .
Computational Experiments
We evaluate the performance of our algorithm (denoted as RS-OCT) on 21 real-world datasets from
UCI machine learning repository [Lichman et al., 2013].
The algorithm is implemented in Julia 1.7.0.
Our implementation code is available at: https://github.com/YankaiGroup/optimal_ decision_tree .
Comparison is made against the CART algorithm
[Breiman et al., 1984] and Optimal Classification Tree (OCT) method [Bertsimas and Dunn, 2017].
A comparison with the performance of FlowOCT [Aghaei et al., 2020] is also presented in Appendix E.1.
To avoid the effects of hyper-parameter choice, the penalty parameter λ is set to 0 .
for all three algorithms.
The split of training and testing data follows the work of Bertsimas and Dunn [2017], that is, 75% of the entire dataset is selected as the training set while the rest 25% is set as the testing set.
All datasets are split by the stratifiedobs function from Julia Package MLDataUtils , under random seed 1, so that the distribution of class is guaranteed to be the same in both training and testing set.
RS-OCT and OCT are executed until either they converge to an optimality gap within 1% or the runtime exceeds a limit of 4 hours.
The result of CART is obtained through the Julia Package DecisionTree with version v0.10.12 .
The MIP problem of OCT and the subproblem of RS-OCT (for grouping based lower bound) are solved by the state-of-art global optimizer CPLEX 20.1.
[Cplex, 2020].
Both OCT and RS-OCT are provided with the same CART solution as warm-start.
A preliminary parallel version of our algorithm is implemented by assigning subproblems in the lower bound calculation to multiple CPU cores.
The size of each subproblem for group decomposition is set as max(2 K, 2 D n ( ct + P +2)
− 2 D
( P +2+ K ) ) , where ct (default: 150) is a constant that controls the average group-size.
The serial and parallel experiments are executed on a high-performance computing cluster (40 Intel "Skylake" at 2.4 GHz with 202G RAM per computing node).
Five criteria (upper bound, training accuracy, testing accuracy, optimality gap, and time) are used to compare the performance of the algorithms.
Here, the upper bound (UB) represents the best feasible solution we found until the termination of each algorithm.
The optimality gap is calculated as UB − LB UB × 100% .
It measures the closeness from the best-searched solution (i.e. UB) to its possible global optimum (i.e. lower bound).
As a unique property of deterministic global optimization algorithms, heuristic algorithms like CART cannot provide such criteria.
Our work focuses on improving the solution process of the optimal decision tree problem.
Therefore, to best represent the solving performance, we use the same value of λ , so that both our method and OCT are solving the same optimization problem, and we can have a fair comparison of the optimality gap, solution time, and best achieved upper bound on this specific problem formulation.
However, to further demonstrate the performance of RS-OCT in practical, compared to other state-of-the-art algorithms, we also perform experiments with hyperparameter tuning on depth 2 and 3.
The results are presented in Appendix E.1.
The effectiveness of sample reduction is validated via the ablation test and the result is shown in Appendix E.2.
Small and medium-scale datasets We perform experiments on 15 datasets ranging fromto 5,473 samples.
Table 1 presents the result of using three algorithms.
In terms of UB, RS-OCT can find a better (or equally better) solution with the lowest cost for almost all datasets, compared to CART and OCT.
RS-OCT also terminates with a smaller optimality gap for all datasets.
It is remarkable that for datasets such as G LASS , B ANKNOTE , and W ALL - FOLLOWING , RS-OCT can achieve an optimality gap within 1% while OCT still has a large gap over 98% after 4 hours of running.
Our result indicates that although OCT could find a better solution than CART, it is still hard to claim that such a solution is the optimal one.
In contrast, RS-OCT can find an even better solution more efficiently and prove the optimality within a reasonable time.
Due to the better cost found by
RS-OCT, the training accuracy of all datasets in Table 1 improves by 2 .
9% on average compared to CART and 0 .
8% on average compared to OCT.
In terms of testing accuracy, since we fixed the penalty parameter λ for all datasets, the result on smaller datasets ( n < 1000 ) is unsatisfactory probably due to an overfitting issue.
In total, the testing accuracy is improved by 2 . 2% and 1 . 0% , on average, compared to CART and OCT, respectively.
For medium-scale datasets ( n ≥ 1000 ), the testing accuracy improves by 3 .
3% on average compared to CART and 1 . 4% on average compared to
OCT.
Large-scale datasets We test our algorithm on several large datasets ( n ≥ 7 , 000 ), with a variety of dimensions (3-16) and number of classes (2-12), as shown in Table 2.
RS-OCT is executed in parallel with 1000 CPU cores.
Remarkably, RS-OCT outperforms OCT and CART on all criteria for all datasets, including the HT-S ENSOR which has almost one million samples.
In terms of upper bound, we notice that for datasets with a size over 40 , 000 samples, OCT can not provide a better solution than the warm-start given by CART.
Nevertheless, RS-OCT discovers better solutions in terms of UB than both CART ( 12 . 5% improvement on average) and OCT ( 11 .
7% improvement on average) on all large datasets.
In terms of optimality gap, RS-OCT converges to an optimality gap within 1% on P ENDIGITS , S HUTTLE , and S KIN -S EGMENTATION .
It verifies that the best solutions found for these datasets are merely at most 1% close to the actual global optimal solution.
It is the first time an optimal decision tree (without binary encoding) is found for a dataset with almost one million samples with a meaningful optimality gap ( 56 . 2% ) in four hours of runtime.
Remarkably, our algorithm can also solve for a dataset with over 200 , 000 samples to a practical optimality gap ( ≤ 1% ) in less than two hours.
Given the more optimal solution found by RS-OCT, its training accuracy is on average boosted by 3 .
7% and 3 .
6% , compared to CART and OCT, respectively.
The testing accuracy is also boosted by 3 . 3% and 2 .
8% on CART and OCT, respectively.
Notably, compared with the results from Table 1, we find that the performance of RS-OCT is improving with the increment of the data size. [Table]
Conclusion
This paper proposed a tailed reduced-space branch and bound (BB) algorithm to train optimal decision tree for the classification tasks.
Our BB algorithm can converge to a global ϵ -optimal solution by only branching on variables describing the tree structure [Table] which is invariant to the number of samples.
Combined with the sample reduction and paralleled lower bound calculation, our algorithm can perform well on all datasets in terms of UB, optimality gap and training accuracy.
Our algorithm also improves the testing accuracy on all large datasets ( n ≥ 7 , 000 )
[Table] Impact and Future work: We hope that the MIP-based scalable framework proposed in this paper can help address the needs of optimal decision tree for large-scale data training in high-stake domains such as crime analysis [Zhuang et al., 2017] [Table] or medical decision [Bertsimas et al., 2018].
We look forward to integrating our algorithms with other functionality (e [Table] g., fairness [Aghaei et al., 2019]) to provide handy tools for wider applications.
Limitation: Similar to other works, our algorithm only tests optimal decision tree problems with small depth (e.g. D = 2 , 3 ).
Larger D slows down the algorithm’s efficiency.
Proper bound tightening methods will be designed in the future to solve this problem.
Bertsimas and Dunn [2017] do not report results on datasets of
[Table]
Acknowledgments and Disclosure of Funding
Y.C. acknowledges funding from the discovery program of the Natural Science and Engineering Research Council of Canada under grant RGPIN-2019-05499.
The authors also gratefully acknowledge the computing resources provided by SciNet (www.scinethpc.ca) and Digital Research Alliance of Canada (www.alliancecan.ca)
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work is focus on developing novel training method on optimal decision tree (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] all proofs are in appendix file from the supplementary materials 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] Since our algorithm is a deterministic global optimization algorithm, whatever random seed provide, the algorithm will always converge to the same global optimum (with given optimality gap) (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] OCT is coded by ourselves and the code is provided in the supplementary materials and we also provide the result reported in their original paper for reference.
(b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We are using data from UCI Machine Learning Repository (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] We are using data from UCI Machine Learning Repository 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
Our work didn’t use crowdsourcing or conducted research with human subjects (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
Our work didn’t use crowdsourcing or conducted research with human subjects (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Our work didn’t use crowdsourcing or conducted research with human subjects
Supplementary Materials
A Constraint Explanation for Problem
Constraint 2b checks the loss of each sample and the derivation is shown as follows.
Let a latent variable L st to be the loss of sample s at leaf node t , as y sk and c kt are both binary, we can set:
[Equation]
It is easy to see that 0 ≤ L st ≤ 1 , as the loss of each sample s at leaf t can only have one mis-match on class label.
We introduce variable L s ∈ [0 , 1] to indicate the loss of sample s in its best allocated leaf.
To guarantee that the best leaf node for sample s is chosen, we conclude the following big-M constraints:
[Equation]
As L s − L st will no exceed 1 , combined with Equation 11, we have:
[Equation]
Constraint 2c ensures that only one label should be assigned to a particular leaf.
Constraint 2d to 2i are mainly adopted from Bertsimas and Dunn [2019] on Chapter 8.2.
Here we briefly explain the meaning and derivations of these constraints.
Constraint 2d guarantees that a sample can only be assigned to one leaf node.
Constraint 2e and 2f are set to enforce the split of the tree with given training data.
They are derived as follow:
First, to enforce the splits required by the structure of the tree when assigning points to leaves, we set constraints:
[Equation]
Here ϵ is added to avoid the strict inequality and can be expressed as the smallest non-zero distance between adjacent values of j th feature:
[Equation]
where x ( i ) j is the i th largest value in the j th feature.
To specify the M 1 and M 2 , let ϵ min =
min j { ϵ j } and ϵ max = max j { ϵ j } , we notice that the largest value for a Tm ( x s + ϵ ) − b m is 1 + ϵ max .
Thus, M 1 can be set to 1 + ϵ max .
Similarly, The largest value of a Tm x s − b m is 1 , so we can set M 2 = 1 .
Therefore, we reformulate Equation 14 and 15 as follow:
[Equation]
Constraint 2g and 2h are set to ensure that if there’s no split on t ∈ T D , then all samples at this node will go to the right branch of this decision node and send down to the same right most leaf node (along with Constraint 2e and 2f).
Constraint 2i enforces the hierarchical structure of the tree.
It guarantees a parent node must have a split if its child node has a split.
B Branch-and Bound Algorithm for Optimal Decision Tree Problem
Algorithm 1 depicts the details of the Branch-and-bound scheme for training the optimal decision tree.
Based on Proposition 3 and 4, relaxed MIP and group decomposition provide tighter bounds than the basic lower bound.
Therefore, using the basic lower and upper bound is adequate to deduct the convergence proof of Algorithm 1.
The critical property of our BB based optimal decision tree training algorithm is that it can guarantee the convergence by only branching on the space of first stage variables m =
( a, b, c, d ) (i.e. the variables describing the tree structure).
Algorithm 1 can be treated as a rooted BB tree.
It starts from a root BB node indexed at levelwith the original variable space M 0 .
Let δ ( M ) =
|| m u − m l || ∞ be the diameter of the set M .
We denote M i q as the BB node at level q which is explored at iteration i q .
A node M i q
+1 is a descendent node that connected to its parent node M i q , with M i q +1 ⊂
M
i q .
The descendent node is at level q + 1 and is explored at iteration i q
+1 .
We denote { M i q } as the sequence of the partition element that represents a path of the BB tree from the root node to the node M i q at the level q .
Since the search space is narrowing along the path, the sequence { β i } is monotonically increasing, while { α i } is monotonically decreasing.
If lim i →∞ α
i =
lim
i →∞ β i = f , we say the BB algorithm converges.
Once the algorithm is convergent, it produces a global ϵ -optimal solution in a finite number of steps.
B.1 Proof of Theorem 1 (Convergence Analysis)
The proof follows the work of Cao and Zavala [2019] which proves the convergence of the BB algorithm by only branching on first stage variables.
In their case, both first stage and second stage variables are continuous variables.
They also assume that Q ( . ) is a continous function.
However, we prove that for optimal decision tree problem, which has mixed-integer first and second stage variables and also discontinous Q ( . ) function, such convergence can still hold.
We also adopt some basic results from the seminal work in the Chapter IV of Horst and Tuy [2013] and modified definitions and theories accordingly for the optimal decision tree problem.
Specifically, based on Corollary IV.1 and Definition IV.10 from Horst and Tuy [2013], it is easy to see that Algorithm 1 creates exhaustive subdivisions .
That is, for all decreasing sub-sequences { M i q } , lim q →∞ δ ( M i q ) = 0 .
To prove that the lower bound β converges to f , we have to show that the lower bounding operation of Algorithm 1 is strongly consistent and its selection operation is bound improving .
From Definition IV.6 and Theorem IV.3 of Horst and Tuy [2013], it is trivial that the selection operation in Algorithm 1 is bound improving , since it always selects the node with the actual attained lower bound for further partition.
In Lemma 3 of Cao and Zavala [2019], they prove that the basic lower bounding operation is strongly consistent if the first stage variables are continuous.
In our case, by giving priority to the first stage integer variables, after finite steps, all integer variables are determined and then the prove can be down following Cao and Zavala [2019].
Lemma 5.
Given an exhaustive subdivision on m , Algorithm 1 satisfies lim i →∞ β i = f .
Proof.
Since Algorithm 1 uses a strongly consistent lower bounding operation and its selection operation is bound improving , from Theorem IV.3 in Horst and Tuy [2013], the result is trivial.
Then, we have to prove the convergence of the upper bounds α through the following lemma.
Lemma 6.
Given an exhaustive subdivision on m , Algorithm 1 generates a sequence { α i } such that lim
i →∞ α i
= f .
Proof.
Since a, c, d are binary variables and these variables are given priority in the branching strategy, the value of a, c, d will eventually be fixed after finite step of subdivision.
Suppose the optimal limit points of them are a ∗ , c ∗ , d ∗ .
Therefore, we can have a i 0 , such that m ′ =
( a ∗ , b, c ∗ , d ∗ ) , ∀ m ′ ∈
M i , ∀ i ≥ i 0 with fixed a ∗ , c ∗ , d ∗ .
Let m ∗ ∈ M 0 as an optimal solution of Problem 2, with m ∗ =
( a ∗ , b ∗ , c ∗ , d ∗ ) .
Without loss of generality, we assume at a decision node t , the j th feature is selected for splitting (i.e. a ∗ jt = 1 ).
Suppose the sorted value of j th feature from small to large is { x 1 ,j , · · · , x s,j , · · · , x n,j } .
Assume x s,j is an optimal solution of b t .
It is easy to see that all values within the range [ x s,j , x s +1 ,j ) gives the same optimal value.
Define b ∗ t as the midpoint of the range.
Based on the definition of ϵ in Problem 2 (See Chapter 8.2 of Bertsimas and Dunn [2019]), we are guarantee to have the same optimal value on any b such that b t ∈ [ b ∗ t − 0 .
5 ϵ min , b ∗ t + 0 .
5 ϵ min ] , ∀ t ∈
T D .
We denote B ( m ∗ ) =
{ ( a ∗ , b, c ∗ , d ∗ )
| b t ∈ [ b ∗ t − 0 .
5 ϵ min , b ∗ t + 0 .
5 ϵ min ] , ∀ t ∈ T D } .
For every point m ∈ B ( m ∗ ) , we have Q ( m ) =
Q ( m ∗ ) holds.
Since the subdivision is exhaustive, after a finite number of iterations i ′ such
that
i ′ ≥ i 0 , we have, either the partition considered satisfies M i ′
⊆ B ( m ∗ ) , or the partition M i ′ which contain the solution m ∗ is pruned.
For the first case, since M i ′ ⊆ B ( m ∗ ) , we have Q ( m ) =
Q ( m ∗ ) , ∀ m ∈ M i ′ .
Then we have that α i ′ =
Q ( m ∗ )
= f .
In the second one, since M i ′ is pruned, then we have α i ′ ≤ β ( M i ′ ) .
Because m ∗ ∈ M i ′ , we also have β ( M i ′ ) ≤ Q ( m ∗ ) .
Hence, α i ′ =
Q ( m ∗ )
= f .
Therefore, lim i →∞ α i =
f holds.
Combing Lemma 5 and 6, Theorem 1 can be obtained.
B.2 Branching Strategy for Algorithm
The branching strategy for Algorithm 1 is designed based on the variables’ effect on the structure of the optimal decision tree.
Since d indicates whether each decision node is splitting or not, it directly determines the skeleton of the tree structure.
Thus, in implementation, the variables of d on each decision node are first used for branching.
We devise the following heuristics for the other three decision variables ( a, b, c ) to select the branching variable:
First, we generate a threshold τ = 1 − 12 ||
b u − b l || ∞ .
Suppose τ is smaller than a randomly generated number between 0 and 1.
In that case, we branch on variable a , which determines the splitting feature of the decision node (If a is all determined, then branching on variable c ).
When τ is larger than the randomly generated number, we branch on variable b (continuous) at the midpoint of its range.
The interior order of each type of variables is based on picking variable with smaller decision(leaf) node index (e.g. a j,t will be chosen over a j,t +1 , if they both are not determined.)
C Basic Lower Bounding Problem Calculation
Algorithm 2 explicit the way of finding optimal value of the subproblem β s .
It also provides a byproduct T z s , which indicates the leaf node that sample s can reach under region M .
T z s can inherit successively to the descendent BB node and directly reduce the number of variables on z for the lower bound calculation.
Moreover, T z s can also enhance the effect of sample reduction, which further improves the performance of the BB algorithm.
Input: x s , y s , m l = ( a l , b l , c l , d l ) , m u = ( a u , b u , c u , d u ) , D, n, ˆ L, λ 1 Set DT root node
(decision node) t 0 = 1 , node set T ← { t 0 } ,
Leaf loss set L = ∅ , reach leaf node set T z s = ∅ , T D = { 1 , · · · , 2 D − 1 } , T L = { 1 , · · · , 2 D } ; 2 while T ̸ = ∅ do t = min { t | t ∈ T } ;
T ← T \ { t } ; if t ∈ T D then if d lt = d ut =
then while t / ∈ T L do t ← 2 t + end
T ← T ∪ { t } ; else
F t = { j | ( a ljt = a ujt = 1) ∨ ( a ljt ̸
= a ujt ) , ∀ j ∈ { 1 , · · · , P }} ; for Each split feature f ∈ F t do if x sf < b lt then
T ← T ∪ { 2 t } ; else if x sf ≥ b ut then
T ← T ∪ { 2 t + 1 } ; else
T ← T ∪ { 2 t, 2 t + 1 } ; /* x sf ∈ [ b lt , b ut ) */ end end end else
T z s
← T z s ∪ { t } ; if y sk ∈ [ c lkt , c ukt ] , ∀ k ∈ { 1 , · · · , K } then
L ← L ∪ { 0 } ; else
L ← L ∪ { 1 } ; end end 31 end 32 return T z s , L = 1ˆ L min { l | l ∈ L } + λn (cid:80)
[Equation]
C.1 Proof of Theorem
In this part we prove Theorem 2 that Algorithm 2 can find all leafs that sample s can fall into under
M and can obtain the global optimum for the lower bound subproblem β s ( M ) .
Proof.
We proof by induction on maximum depth D :
Base case: Suppose maximum depth D
= 0 , then there’s only one leaf node t = 1 for the tree model,
T z = { t } under M .
The statement is true.
Induction: Suppose the statement is true for the decision tree problem with maximum depth D ≥under any specific M , then for decision tree problem with maximum depth D + 1 , since D ≥ 0 , the first node is a decision node and line 6-22 guarantee to put all its possible descendent nodes into the node set T according to bound M at decision node t .
Since each possible descendent node can be regarded as a root node of decision tree model with depth D , then the decision tree problem rooted with the descendent nodes can find all possible leaf nodes that sample s can fall into with given bound M =
[ m l , m u ] , according to the assumption.
Thus the statement is proved true on decision tree problem with maximum depth D + 1 .
Therefore, since Algorithm 2 exhaust all possible loss of sample s and pick the minimum one,
Algorithm 2 will find the global optimum of subproblem β s ( M ) :
= min m ∈ M Q s ( m ) .
D Additional Analysis for Sample Reduction
In this section
[Equation]
we provide the new formula for two tighter lower bound strategies after sample reduction.
Relaxed MIP
With the assistance of S dt we can further simplify Problem 7 as:
[Equation]
Here, L s ′ are determined according to sample reduction.
Compared to Problem 7, the number of variable L s is reduced from n to | S ud | .
The solving efficiency of the second lower bound strategy can be further enhanced.
Group decomposition
The computing load of the subproblem formed by each group can also be reduced through sample reduction.
Specifically, we can have the following equation for each subproblem β Gg :
[Equation]
Here, we can see that the number of decision variables is also vastly reduced.
The number of constraints related to samples is significantly reduced accordingly under the scheme of sample reduction.
In the case that |T z s | = 1 for some s , the corresponding constraint 20e- 20g are eliminated on these s , since here z s is fully determined.
E Additional Experiments E.1 Results with Hyperparameter Tuning
We implement the experiments with hyperparameter tuning and compared the performance of RSOCT with CART, OCT [Bertsimas and Dunn, 2017], and FlowOCT [Aghaei et al., 2020].
Since
FlowOCT is designed for datasets with binary features, all datasets are transformed to binary with one-hot encoding before feeding to FlowOCT.
The code of FlowOCT is implemented in Python † and use Gurobi as the MIP solver.
Table 3 and 4 present the results of training optimal classification trees on all datasets with hyperparameters selected through validation for all four methods.
For OCT and our method, we performed hyperparameter tuning following the instruction in [Bertsimas and Dunn, 2017] to select the continuous regularization parameter λ .
For CART, we first train a full decision tree with the maximum depth and then we prune the tree by tuning the pruning-purity parameter from the Julia package DecisionTree.jl .
For FlowOCT, we followed the instruction in [Aghaei et al., 2020] and modify its regularization parameter λ by selecting values in the set { 0 , 0 . 1 , · · · , 0 . 9 } .
From the result, we can see that even after hyperparameter tuning, RS-OCT still competitive on most of datasets, compared to other methods.
Particularly, for large datasets (i.e., n ≥ 7 , 000 ), when maximum depth of tree is two, RS-OCT can improve the training accuracy of 2.7% and 2.5% on average, compared to CART and OCT, respectively.
The testing accuracy (out of sample accuracy) can be improved by 2.8% and 2.5% on average, compared to CART and OCT, respectively.
E.2 Analysis and Ablation Test for Sample Reduction
We provide the analysis and ablation test of the sample reduction on small and median dataset to demonstrate its effectiveness.
First, four indicators are presented to help evaluate the effect of sample reduction on each dataset.
The N ODES represents how many BB nodes are explored during the running of the BB algorithm.
The M AX -L EVEL represents the maximum depth of the BB nodes explored by the BB algorithm.
The M AX -R ATE and A VG -R ATE are the indicators of the maximum and average percent of samples whose loss are pre-determined before the calculation of each node.
We did not put the M IN -R ATE here since, in the root decision node of the tree, no sample can be determined, and thus the M IN -R ATE is always zero.
The numerical results (Table 5) reveal that the effect of sample reduction will primarily reduce the computation load when the search tree of the BB algorithm goes deep.
Specifically, for datasets (S EEDS , G LASS , B ODY , B ANKNOTE , W ALL -F OLLOWING ) that can converge to a small optimality gap (e.g. ≤ 1 ), the sample reduction devote a significant effort on the calculation process in terms of
Max-rate and Avg-Rate.
Almost all of these datasets can, on maximum, have the loss determined for over 90% of samples and maintain a relatively high average rate of determined samples among all explored nodes.
For the indicator of Nodes and Max-Level, we discover that the exploration of more nodes could also harness the effect of sample reduction even if the node’s level is not deep, especially when the data size is large.
Next, we perform the ablation test for sample reduction.
To make the comparison more precise, we only test on datasets where the average percentage of determined samples is above zero.
Tablepresents the test result and reveals that, with sample reduction, all datasets can either converge to a smaller optimality gap within the same time or use less time to reach <1% optimality gap.
Therefore, the sample reduction can have a recognizable effect on improving the calculation efficiency of each
BB node.
One limitation of the sample reduction is that it could have reduced influence on datasets with high feature dimensions (e.g. Both two O ZONE datasets have 72 features).
The reason could be probably due to the large search space of the decision variables (i.e. The number of first-stage variable is ( P + 2)
|T D | + K |T L | , which is proportional to P and K ).
Even if the bounds of decision variables is narrowed through the BB procedure, the feasible space under a BB node is still too large to help determine the loss of any samples.
[Table] [Table]
[Table] [Table]