We describe an experimental study of pruning methods for decision tree classi ers in two learning situations: minimizing loss and probability estimation. In addition to the two most common methods for error minimization, CART's cost-complexity pruning and C4.5's errorbased pruning, we study the extension of cost-complexity pruning to loss and two pruning variants based on Laplace corrections. We perform an empirical comparison of these methods and evaluate them with respect to the following three criteria: loss, mean-squared-error (MSE), and log-loss. We provide a bias-variance decomposition of the MSE to show how pruning a ects the bias and variance. We found that applying the Laplace correction to estimate the probability distributions at the leaves was bene cial to all pruning methods, both for loss minimization and for estimating probabilities. Unlike in error minimization, and somewhat surprisingly, performing no pruning led to results that were on par with other methods in terms of the evaluation criteria. The main advantage of pruning was in the reduction of the decision tree size, sometimes by a factor of 10. While no method dominated others on all datasets, even for the same domain di erent pruning mechanisms are better for different loss matrices. We show this last result using Receiver Operating Characteristics (ROC) curves.
An experimental study for cost complexity pruning and C4.5's error-based pruning that concentrated on pruning with loss minimization and probability estimation instead of error minimization was conducted in REF .
Pruning decision trees with misclassification costs
ECML
Computer Science Mathematics