Introduction. Decision trees have gained increasing popularity these years due to their effectiveness in solving classification and regression problems and their capability to explain prediction results. Learning an optimal decision tree with a predefined depth is NP-hard. Hence, greedy based heuristics such as CART and ID3 have been widely used to construct sub-optimal trees. Recent years have seen an increasing number of work that employ various Mathematical Optimization (MO) methods to build better quality decision trees. A limitation of the state-of-the-art Mathematical Optimization formulations, i.e., [1, 2] , for this problem is that they create constraints and variables for every row in the training data. Consequently, the solving time of finding decision trees increases dramatically with the problem size. We formulate the problem of constructing the optimal decision tree of a given depth as an binary linear program. We call our method BinOCT, a Binary encoding for constructing Optimal Classification Trees. Our novel formulation models the selection of decision threshold via a binary search procedure encoded using a type of big-M constraints. This requires a very small number of binary decision variables and is therefore able to find good quality solutions within limited time. Noteworthy is that the number of decision variables is largely independent of the number of training data rows: it only depends logarithmically on the number of unique feature values. We show using experiments that BinOCT outperforms existing MO based approaches on a variety of data sets in terms of accuracy and computation time.
Recently, Verwer and Zhang REF formulate the optimal decision tree learning problem of a given depth as a binary linear program.
Learning Optimal Classification Trees Using a Binary Linear Program Formulation
BNAIC/BENELEARN
Computer Science Mathematics