Paper: GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data (ICLR-2024)

Reviewer comment (Experiments):
Instead of using exclusively binary class benchmark datasets, could the authors also present comparative results for multi-label classification problems?

Author response:
> Instead of using exclusively binary class benchmark datasets, could the authors also present comparative results for multi-label classification problems?

**TL;DR: GRANDE outperforms existing methods on several multi-class datasets**

In our paper, we focused on binary classification tasks to present a comprehensive evaluation within the limited scope. Yet, we acknowledge the importance of demonstrating that our method is applicable beyond binary classification. Therefore, we conducted additional experiments on several openml multi-class tasks (we used the multi-class datasets from `[1]`) detailed below. We report the results both without HPO and with HPO (30 trials for all methods, constrained by the limited time available during the rebuttal period).

Similar to binary classification, GRANDE outperforms SOTA methods on several datasets for multi-class tasks without additional adjustments of our method. Accordingly, the results for multi-class classification are in line with the results on binary classification tasks presented in the paper. We also included the results in the appendix of our revised version.

HPO (30 trials):
|  | GRANDE | XGBoost | CatBoost | NODE |
| -------- | -------- | -------- | -------- | -------- |
| GesturePhaseSegmentationProcessed      |    0.455 | **0.643** |      0.638 |  0.247 |
| artificial-characters                  |    0.726 | **0.905** |      0.867 |  0.614 |
| audiology                              |    0.848 | 0.85  |      0.861 |  **0.89**  |
| balance-scale                          |    **0.78**  | 0.681 |      0.776 |  0.616 |
| cnae-9                                 |    0.94  | 0.919 |      0.939 |  **0.942** |
| jungle_chess_2pcs_raw_endgame_complete |    **0.842** | 0.832 |      0.827 |  0.807 |
| mfeat-fourier                          |    **0.859** | 0.831 |      0.847 |  0.859 |
| mfeat-zernike                          |    **0.823** | 0.778 |      0.783 |  0.811 |
| one-hundred-plants-texture             |    0.818 | 0.741 |      **0.852** |  0.753 |
| splice                                 |    **0.964** | 0.957 |      0.947 |  0.96  |
| vehicle                                |    0.817 | 0.809 |      **0.819** |  0.816 |
| **Normalized Mean**                             |    **0.751** | 0.338 |      0.661 |  0.484 |
| **Mean Reciprocal Rank (MRR)**                                         |    **0.674** | 0.432 |      0.508 |  0.47  |
| **Count**                                       |    **5**     | 2     |      2     |  2     |


Default Parameters (summarized):

|  | GRANDE | XGBoost  | CatBoost | NODE |
| -------- | -------- | -------- | -------- | -------- |
| Normalized Mean | **0.653**     | 0.324    | 0.523         | 0.425     |
| Mean Reciprocal Rank (MRR)     | **0.674**     | 0.379    | 0.508         | 0.523     |
| Wins |    **5**     | 1     | 2     | 3     |

Reviewer comment (Experiments):
It would be valuable if the authors could include comparisons with other tree ensemble methods, such as random forest and extra trees.

Author response:
> It would be valuable if the authors could include comparisons with other tree ensemble methods, such as random forest and extra trees.

We included random forests and extra trees to our evaluation. The results are presented below. Due to lack of space, we were not able to adjust the results table in the main part of our paper, but included them to the Appendix C (Table 13-15). The results are consistent with the claims in our paper: GRANDE achieves superior results both with default parameters and after HPO.

HPO (250 trials):
|  | GRANDE | XGBoost  | CatBoost | NODE | ExtraTree | RandomForest |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Normalized Mean |    **0.817**     | 0.518     | 0.705     | 0.382     | 0.385     | 0.404     |
| Mean Reciprocal Rank (MRR)     | **0.668**     | 0.365     | 0.541     | 0.352     | 0.251     | 0.275     |
| Wins |    **9**     | 2     | 5     | 2     | 0     | 1     |

Default:
|  | GRANDE | XGBoost  | CatBoost | NODE | ExtraTree | RandomForest |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Normalized Mean                |    **0.652**     | 0.607     | 0.587    | 0.352     | 0.380     | 0.471     |
| Mean Reciprocal Rank (MRR)     | **0.595**     | 0.450     | 0.416     | 0.339     | 0.283     | 0.371     |
| Wins                           |    **9**     | 3     | 2     | 2     | 0     | 3     |

Reviewer comment (Evaluation):
Regarding the interpretability advantages of instance-wise weighting, could the authors provide a more comprehensive analysis? For instance, could they share statistics like the average node count of the highest-weighted estimators across all datasets? Furthermore, could the authors offer any theoretical insights into the benefits of instance-wise weighting?

Author response:
> Regarding the interpretability advantages of instance-wise weighting, could the authors provide a more comprehensive analysis? For instance, could they share statistics like the average node count of the highest-weighted estimators across all datasets? 

**TL;DR: We provided additional statistics (node count, estimator frequency, skewness, kurtosis, etc.) supporting the claims from our paper.**

[Also see response to `JdQZ`]

Within our paper, we showed in an ablation study that our instance-wise weighting has a positive impact on the performance of GRANDE. In addition, we showcased a real-world scenario where the weighting significantly increases local interpretability by enabling the model to learn representations for simple and complex rules. We agree with the reviewer that a more comprehensive analysis of the weighting would be beneficial and therefore provide additional statistics. We summarize the main results below and provide detailed statistics for each dataset, along with an additional discussion in Appendix D.

The following statistics were obtained by calculating the post-softmax weights for each sample and averaging the values over all samples. In addition, we calculated the same statistics for the top 5% of samples with the highest weights. This provides additional insights as we argue that unique local interactions might only exist for a subset of the samples.


| statistic | average value|
| -------- | -------- |
| **average internal node count of highest weighted estimator**     | 2.918 |
| **average leaf node count of highest weighted estimator**     | 3.918  |
| **fraction of samples for highest weighted estimator**    | 0.325  |
| **fraction of samples for highest weighted estimator top 5% samples**     | 0.631  |
| **number of highest weighted estimators**    | 44.221  |
| **number of highest weighted estimators top 5% samples**    | 5.347  |

We can observe that on average, the highest weighted estimator comprises a small number of ~3 internal and ~4 leaf nodes, allowing for easy interpretation. Furthermore, on average, the highest weighted estimator is the same for ~33% of the data instances with a total of ~44 different estimators having the highest weight for at least one instance. If we further inspect the top 5% of instances with the highest weight for a single estimator, we can observe that the same estimator has the highest weight for ~63% of these instances and only ~5 different estimators achieve the highest weight for at least one instance. This indicates the presence of a small number of local experts for most datasets, having high weights for certain instances where the local rules can be applied.


In the subsequent analysis, we evaluate the distribution of the weights in more detail, focusing on skewness as a measure of distribution asymmetry and kurtosis as a measure of tailedness. In general, it stands out that both values increase significantly when considering only the top 5% of samples with the highest weight for a single estimator. This again indicates the presence of local expert estimators for a subset of the data, where unique local interactions were identified. As this is very dataset-specific, we will give a more detailed analysis in the following. The exact values for each dataset are summarized in the Appendix D.


| statistic | average value|
| -------- | -------- |
| **skewness (absolute)**     | 0.933  |
| **skewness top 5% (absolute)**     | 1.400  |
| **(excess) kurtosis**     | 1.291  |
| **(excess) kurtosis top 5%**     | 3.630  |

Reviewer comment (Experiments):
My major concern is about hyper-parameters tuning (section C of appendix): I understand that compute resources should be spared, but it seems unfair to optimize the number of trees for GRANDE but not for XGBoost and CatBoost, especially given the fact that XGBoost and CatBoost are the cheapest algorithms to train.

Author response:
**TL;DR: Tuning the number of estimators for XGBoost and CatBoost has only minor impact on the performance and the results remain unchanged.**

We understand the concerns. The decision to use early stopping instead of tuning the number of estimators was made in line with related work `[1]` to increase search efficiency (not only based on the runtime). Yet, we conducted an additional experiment where we also tuned the number of estimators. Overall, this did not significantly impact the performance of tree-based methods: The average performance slightly decreased when tuning the estimators for both XGBoost (by 0.0010) and CatBoost (by 0.0003). Consequently, for both methods, performance slightly increased in 10 out of 19 datasets and decreased in the remaining 9. In the following, we summarized the comparison of all methods when tuning the number of estimators. The complete results can be found in the appendix of the revised paper (Table 21 and Table 22). These additional results are consistent with the claims in our paper.



|  | GRANDE | XGBoost | CatBoost | NODE |
| -------- | -------- | -------- |-------- | -------- |
| **Mean Reciprocal Rank (MRR) (new, with tuning estimators)**     | **0.684**     | 0.434     | 0.566     | 0.399     |
| **Mean Reciprocal Rank (MRR) (old, without tuning estimators)**     | **0.702**     | 0.417     | 0.570     | 0.395     |
| **Normalized Mean (new, with tuning estimators)**     | **0.773**     | 0.436     | 0.629     | 0.285     |
| **Normalized Mean (old, without tuning estimators)**      | **0.776**     | 0.483     | 0.671     | 0.327     |
| **Wins (new, with tuning estimators)** |    **9**     | 2     | 6     | 2     |
| **Wins (old, without tuning estimators)** |    **9**     | 4     | 4     | 2     |

Reviewer comment (Evaluation):
The results of GRANDE are good on several datasets, but become less impressive when the number of features is high.

Author response:
In general, our method excels in handling datasets with a small to medium number of features. For very high-dimensional datasets, learning GRANDE becomes more challenging (as the matrices $I \in \mathbb{R}^{2^d-1 \times n}$ and $T \in \mathbb{R}^{2^d-1 \times n}$ scale with the number of features). This challenge is addressed by using a feature subset for each estimator, supported by our instance-wise weighting, which allows assigning higher weights to estimators focusing on more important features. Although GRANDE showed comparatively modest results on the *madelon* dataset (500 features) and the *Bioresponse* dataset (1,776 features), it outperformed other methods on the SpeedDating dataset (256 features).

Reviewer comment (Theory):
The 2^d term in the sums suggests that the depth is a real limitation of the method.

Author response:
Theoretically, depth is a limitation of the method. However, in practice, GRANDE can be efficiently trained up to a depth of 10. Preliminary experiments indicated that a depth of 6 is sufficient to achieve SOTA performance, as increasing the depth further does not significantly impact performance.

Reviewer comment (Evaluation):
Fact is that gradient descent is much slower than greedy optimization.

Author response:
We agree that a gradient-based optimization is slower than a greedy optimization, as we discussed in Section 4.2. However, using gradient-based optimization with GRANDE significantly enhances performance on several datasets while typically maintaining a reasonable runtime of less than 60 seconds for most datasets. Furthermore, gradient-based optimization offers additional flexibility (integration of custom loss functions and compatibility with deep learning architectures, for example, multimodal learning) compared to greedy methods.

Reviewer comment (Novelty):
The related-works part seems a bit short given the huge literature on ensemble trees.

Author response:
We acknowledge that due to the limited scope, the related work section is comparatively brief. In the revised version of our paper, we have included additional references to gradient-based trees `[2,3]`, greedy tree ensembles `[4,5]` and tree weighting schemes `[6,7]`. Furthermore, we conducted additional experiments including further greedy tree ensemble methods (RandomForest and ExtraTree) `[SMN1]` and SAINT as new DL benchmark `[Duhp]`. For a more detailed discussion of existing gradient-based tree ensemble methods, please refer to our response to reviewer `Duhp`.

Reviewer comment (Writing):
Page 4 (section 3.3): the W matrix is in R^{E\times 2^d}, not R^E \times R^{2^d}.

Author response:
Thank you for this notification. We have made the necessary adjustments in the revised version of our paper.

Reviewer comment (Reproducibility):
Regarding the code I tried unsuccessfully to run the notebooks: two packages were missing in the installation script: "chardet" and "cchardet", even with this that fixed there is a remaining issue with the "TabSurvey" sub-package which seems to require a specific environment to run (It fails on "from models import" statements).

Author response:
Unfortunately, we were not able to reproduce the error. The error seems to come from the TabSurvey package, which we used to access the implementation of NODE. However, it should be possible to run the experiments without NODE by omitting the corresponding import statement and setting the variable in our configuration to false (`config['benchmarks']['NODE'] = false`).

Reviewer comment (Presentation):
Page 7: the Phishing Website study should appear as a subsection.

Author response:
We adjusted this in the revised paper version.

Reviewer comment (Novelty):
Your weighing scheme seems interesting but does it really require gradient descent? Could you take a fixed forest, add weights to the leaves and optimize these weights afterwards? I am thinking of a paper like (Cui et al. 2023) https://arxiv.org/pdf/2304.13761.pdf.

Author response:
Taking a fixed forest and then optimizing the leaf weights, akin to the approach of Cui et al. (2023), is indeed a viable strategy and can be viewed as a variant of the method described in our paper. However, this would separate the process of constructing the trees from learning the weighting scheme.

In GRANDE's method, gradient descent is used to optimize the entire model structure, including the tree construction and the weights, in an end-to-end manner. This contrasts with using a predefined forest and optimizing only the leaf weights.

The key difference here is that in GRANDE, the tree structure and the weights are optimized together, therefore allowing individual trees to learn unique local interactions (as highlighted in the case study) during the training. This is only possible if the weighting is incorporated in the training procedure and not when learning the weights post-hoc. We included a short discussion in the section on out instance-wise weighting in the revised version of our paper.

Reviewer comment (Experiments):
Did you consider hybridizing greedy and gradient approaches?

Author response:
In this paper, we did not explore hybridizing greedy and gradient-based approaches. However, this presents an interesting research direction for future work.

Reviewer comment (Experiments):
This paper lacks further analysis for instance-wise weighting. Because the final results are weighted by Softmax, the prediction of each tree is not separate now. If we cut off one tree, the contributions of the other tree are also changed. This is different from XGB and NODE, but the authors did not point out it.

Author response:
> Because the final results are weighted by Softmax, the prediction of each tree is not separate now. If we cut off one tree, the contributions of the other tree are also changed. This is different from XGB and NODE, but the authors did not point out it.

Thank you for pointing this out. Indeed, when employing a softmax function to calculate the weights for GRANDE, the individual tree predictions are not separated. Therefore, the contributions of individual trees change. This occurs not only in scenarios such as removing one tree from the ensemble but also during instance-wise weighting: Assume we have two data instances that are very similar and the prediction of our ensemble differs only in one tree. For this particular tree, depending on the different paths taken by the samples, we select two distinct logits. Consequently, when calculating the softmax to generate the weights, the contribution of each tree changes.

This aspect is crucial in enabling the learning of "local experts" within GRANDE. If the path capturing a unique local interaction is traversed by a sample, the corresponding tree should have a very high weight, resulting in reduced weights for the remaining trees.

We acknowledge that this aspect was not adequately addressed in the current version of the paper, and have therefore provided clarification in the revised version (Section 3.3).

Reviewer comment (Evaluation):
It is better to analyze the distribution of instance weights. For example: a) Is it long-tailed? b) Are some trees very important for most of the samples?

Author response:
> Moreover, it is better to analysis the distribution of instance weights. 
For example:
a) Is it long-tailed?
b) Are some trees very important for most of the samples?

**TL;DR: (a) In most cases, the distributions are long-tailed (and heavy-tailed). (b) On average, for 38% of the samples, the same tree is the most important one, but this is very dataset- and instance-specific (see discussion below).**

[Also see response to `SMN1`]

Within our paper, we showed in an ablation study that our instance-wise weighting has a positive impact on the performance of GRANDE. In addition, we showcased a real-world scenario where the weighting significantly increases local interpretability by enabling the model to learn representations for simple and complex rules. We agree with the reviewer that a more comprehensive analysis of the weighting would be beneficial and therefore provide additional statistics. We summarize the main results below and provide detailed statistics for each dataset, along with an additional discussion in Appendix D.

The following statistics were obtained by calculating the post-softmax weights for each sample and averaging the values over all samples. In addition, we calculated the same statistics for the top 5% of samples with the highest weights. This provides additional insights as we argue that unique local interactions might only exist for a subset of the samples.


| statistic | average value|
| -------- | -------- |
| **average internal node count of highest weighted estimator**     | 2.918 |
| **average leaf node count of highest weighted estimator**     | 3.918  |
| **fraction of samples for highest weighted estimator**    | 0.325  |
| **fraction of samples for highest weighted estimator top 5% samples**     | 0.631  |
| **number of highest weighted estimators**    | 44.221  |
| **number of highest weighted estimators top 5% samples**    | 5.347  |

We can observe that on average, the highest weighted estimator comprises a small number of ~3 internal and ~4 leaf nodes, allowing for easy interpretation. Furthermore, on average, the highest weighted estimator is the same for ~33% of the data instances with a total of ~44 different estimators having the highest weight for at least one instance. If we further inspect the top 5% of instances with the highest weight for a single estimator, we can observe that the same estimator has the highest weight for ~63% of these instances and only ~5 different estimators achieve the highest weight for at least one instance. This indicates the presence of a small number of local experts for most datasets, having high weights for certain instances where the local rules can be applied.

In the subsequent analysis, we evaluate the distribution of the weights in more detail, focusing on skewness as a measure of distribution asymmetry and kurtosis as a measure of tailedness. In general, it stands out that both values increase significantly when considering only the top 5% of samples with the highest weight for a single estimator. This again indicates the presence of local expert estimators for a subset of the data, where unique local interactions were identified. As this is very dataset-specific, we will give a more detailed analysis in the following. The exact values for each dataset are summarized in the Appendix D.


| statistic | average value|
| -------- | -------- |
| **skewness (absolute)**     | 0.933  |
| **skewness top 5% (absolute)**     | 1.400  |
| **(excess) kurtosis**     | 1.291  |
| **(excess) kurtosis top 5%**     | 3.630  |

1. **Skewness:**

    
    In total, for 16/19 datasets the weights exhibit either a skewed (10) or very skewed (6) distribution, i.e. are long-tailed **(a)**. In general, right-skewed distributions are more desired as they indicate a small number of trees having high weights (which we consider as "local experts"), while the majority of estimators have small weights. Considering the top 5% of samples with the highest weights of a single estimator, we can see that the number of very skewed distributions increases from 6 to 13, indicating samples that can be assigned to a class more easily based on a small number of trees (similar to the example in the case study).
    
    
    | skewness | very skewed; (-∞, -1.0) or (1.0, ∞) | skewed; [-1.0, -0.5) or (0.5, 1.0] | slight skew; [-0.5, 0.5] | sum
    | -------- | -------- | -------- | -------- | -------- |
    | **left skew**     | 0     | 1     | 1     | **2**     |
    | **right skew**     | 6     | 9     | 2     | **17**     |
    | **sum**     | **6**     | **10**     | **3**     |      |

    | skewness top 5% | very skewed; (-∞, -1.0) or (1.0, ∞) | skewed; [-1.0, -0.5) or (0.5, 1.0] | slight skew; [-0.5, 0.5] | sum
    | -------- | -------- | -------- | -------- | -------- |
    | **left skew**     | 0     | 1     | 1     | **2**     |
    | **right skew**     | 13     | 1     | 3     | **17**     |
    | **sum**     | **13**     | **2**     | **4**     |      |

2. **(Excess) Kurtosis:**

    
    We can observe that 11/19 distributions are leptokurtic, i.e. have heavy tails, with 4 considered as very leptokurtic. In general, distributions with heavy tails are interesting, as this indicates outliers (estimators with significantly higher / lower weights). Trees comprising significantly higher weights can be considered as local experts that have learned unique local interactions. Again when considering the top 5% of samples with the highest weights of a single estimator, the number of leptokurtic distributions increases from 11 to 15 with the number of very leptokurtic distributions increasing from 4 to 8. Again, this indicates that there exists samples that can be assigned to a class more easily with simple rules (from "local experts") in many datasets.
    
    | excess kurtosis | count | count top 5% |
    | -------- | -------- | -------- | 
    | **very leptokurtic (3.0, ∞)**    | 4     | 8 |
    | **moderatly leptokurtic (0.5, 3.0]**    |   7   | 6 |
    | **mesokurtic [-0.5, 0.5]**    | 5     | 3 |
    | **platykurtic (-∞, -0.5)**   |   3   | 2 |

    
Overall, the additional statistics are in line with the discussion in the paper. For several datasets, the distribution of weight is long- and/or heavy-tailed indicating the presence of "local expert" estimators. However, the advantage of learning local expert estimators is dataset-dependent and not feasible in all scenarios. This is also confirmed by our results as there are datasets where the distribution is symmetric and mesokurtic. This is valid for instance for numerai28.6, which is a very complex dataset without many simple rules (as indicated by the poor performance of all methods).

This additional analysis has been included in Appendix D. However, it should be noted that a more in-depth analysis is necessary to confirm these findings, which we consider beyond the scope of this paper.

Reviewer comment (Experiments):
Too many hyperparameters, i.e. 11 are tuned for the proposed method. While the number of hyperparameters is no more than 4 for compared methods. We have reason to doubt that the performance gain is obtained by finer hyperparameter tuning. Could you reduce your number of hyperparameters to 4 and retrain your model for comparison?

Author response:
> Too many hyperparameters, i.e. 11 are tuned for the proposed method. While the number of hyperparameters is no more than 4 for compared methods. We have reason to doubt that the performance gain is obtained by finer hyperparameter tuning. Could you reduce your number of hyperparameters to 4 and retrain your model for comparison?

**TL,DR: When reducing the number of hyperparameters or the number of trials, GRANDE still achieves SOTA performance**

We understand the concerns regarding the number of hyperparameters. We already tried to account for this with the default hyperparameter experiment included in the original version of the paper (Table 3 and Table 8), demonstrating that GRANDE achieved superior results even without hyperparameter optimization. Acknowledging the importance of this topic, we have included two additional experiments focusing on hyperparameter optimization:
1. **Reduced number of trials to 30:** We reduced the number of trials in the HPO from originally 250 trials to 30 for all methods (similar to `[1]`) to compare the performance with a less extensive search. We can observe that GRANDE has the greatest benefit when increasing the number of trials. Yet, even with only 30 trials, GRANDE achieves the highest mean reciprocal rank (MRR) and normalized mean, as well as the most wins. The detailed results on all datasets along with a discussion are included in the appendix of the revised paper version and summarized in the following table:

    |  | GRANDE | XGBoost  | CatBoost | NODE |
    | -------- | -------- | -------- | -------- | -------- |
    | Normalized Mean                           |    **0.694** | 0.430 |      0.676  |  0.336 |
    | Mean Reciprocal Rank (MRR)                                       |    **0.636**  | 0.434 |      0.579 |  0.434 |
    | Count                                     |    **8**     | 3     |      5     |  3     |


2. **Reduce grid for GRANDE to 4 parameters:** Following the reviewer's suggestion, we reduced the number of hyperparameters for GRANDE to four (one learning_rate, dropout, loss, and cosine_decay) and conducted an additional HPO with 250 trials. Unfortunately, we could not complete the search for all datasets during the rebuttal period (*nomao* and *adult* datasets are pending and will be included in the camera-ready version). Reducing the grid results in a slightly worse performance for GRANDE. Especially, optimizing separate learning rates for different components (split values, split indices, leaf probabilities, and leaf weights) using a larger grid did further enhances GRANDE's performance. Yet, the results are in line with the claims from the paper and GRANDE still has the highest normalized mean, MRR and number of wins.


    |  | GRANDE | XGBoost | CatBoost | NODE |
    | -------- | -------- | -------- | -------- | -------- |
    | Normalized Mean                           |    **0.736** | 0.428 |      0.710 |  0.364 |
    | Mean Reciprocal Rank (MRR)                                       |    **0.652** | 0.358 |      0.608 |  0.466 |
    | Count                                     |    **7**     | 0     |      6     |  4     |


`[1]` McElfresh, Duncan C., et al. ‘When Do Neural Nets Outperform Boosted Trees on Tabular Data?’ Thirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023, https://openreview.net/forum?id=CjVdXey4zT.