Paper: End-to-End Training of  Unsupervised Trees: KAURI and DOUGLAS (ICLR-2024)

Reviewer comment (Novelty):
Several successful experimental cases are presented, yet the paper lacks theoretical backing. Although the proposed method is straightforward, it doesn't appear to offer a high degree of novelty. Consequently, the research's significance remains unclear.

Author response:
The global motivation for the study was to introduce a complete framework on tree architecture and GEMINI training. The link with kernel KMeans is here to serve as a theoretical justification for the expected performances of one specific case of this framework: KAURI.

We believe that there exists indeed a clear difference between the end-to-end training approach of such a tree compared to twofold methods, especially with Table 4 (now 5) displaying a shallower structure on average, Figure 2 highlighting how we deal better with non-axis aligned decision boundary beyond Appendix E (now F).

Still, none of our experiments showed KAURI as significantly underperforming compared to kernel KMeans. In addition, the normalised KMeans scores suggested by reviewer #2 (dexj) that we added to the paper remain better or close on average.

Reviewer comment (Reproducibility):
The DOUGLAS experiment is said to be limited by memory constraints, but there is no information about the specific memory requirements or the machine resources used.

Author response:
To begin with, the memory complexity of DOUGLAS is that of DNDTs. It grows in $\mathcal{O}((T+1)^d)$ with $d$ the number of features and $T$ the number of cuts, e.g. in section 5.1: 20 features cut in 2 is approx. 2^20 parameters. Then, the autodiff framework and the number of samples make this number grow linearly. Consequently, with the double encoding which is required for the Wasserstein GEMINI, the model associated to the car evaluation dataset would already take multiple GBs which saturated our GeForce RTX 2080 (8GB).

Reviewer comment (Evaluation):
Performance metrics are mentioned, but the paper does not provide data on computational time and memory usage.

Author response:
The time performances of the model's complete training could range from a couple of seconds for the Haberman survival (fewest features and 2 clusters) to half an hour for the Avila dataset (20,000 samples and 12 clusters). The number of clusters intervenes on the complexity of the GEMINI computation by a $K^2$ factor.

For KAURI, beyond the complexity per split which could be at worst of $\mathcal{O}(n^2[(L+d)(n+K)+dL]+L^2(d+K))$ as reported in App. C.1.4 worst for $n$ samples, $L$ leaves, $d$ features and $K$ clusters, practical numbers would be roughly less than a second for the iris dataset, and close to a few minutes for the Avila dataset.

Reviewer comment (Reproducibility):
In the Appendix, it's stated that the handling of categorical variables varies depending on the dataset. However, information is only provided for the US congressional votes dataset, affecting the experiment's reproducibility.

Author response:
The other method used for the categorical variables in other datasets is one hot-encoding. While this was briefly mentioned in Table 2 for the car dataset, we clarified it in Appendix D (now E).

Reviewer comment (Theory):
The paper does not provide sufficient motivation for the use of a generalised mutual information (GEMINI) objective for clustering. While the original paper (Ohl et al., 2022) shows good results on unsupervised neural network training, it is still not clear what makes this objective well-suited for clustering problems, particularly with trees.

Author response:
To elaborate on the choice of this objective (GEMINI), there exist very few clustering objectives that can train discriminative models, i.e. without parametric assumptions on the data distribution $p(x)$. Most of these objectives are related to mutual information. Yet, Ohl et al. highlighted the superiority of GEMINI to MI in discriminative clustering. Moreover, the connection of this objective function to KMeans makes it a sensible objective for clustering. These reasons motivate the choice of GEMINI for training unsupervised trees.

Reviewer comment (Novelty):
The paper has quite limited novelty. It adapts a recent clustering objective into the traditional CART-type greedy recursive partitioning algorithm to learn the axis-aligned tree (KAURI algorithm). And similarly with DOUGLAS algorithm, which just uses the existing differentiable deep neural decision trees.

Author response:
We would like to emphasize that we showed here novel types of splits that are not part of the initial CART algorithm, e.g. completely reallocating samples to other or new clusters. We extended GEMINI to train non-differentiable models, contrary to its initial design by Ohl et al. To the best of our knowledge, we are the first to optimise the KMeans objective within a decision tree without prior warm starts, explaining better or similar performances on average.

Reviewer comment (Experiments):
The datasets used in experiments seem to be quite small. The largest contains 20k points in 10 dimensions. Having a dataset of at least MNIST-level size can help to show its scalability.

Author response:
We already had an additional dataset (Poker hand dataset) containing approx 1 million entries. We originally discarded the dataset because all entries were 0 of score for ARI, but this experiment clearly shows that KAURI can scale to large datasets. We could not add DOUGLAS for this specific dataset due to the limitations of the Wasserstein GEMINI. We restored this dataset in the results, but discarded its ARI score because all models obtained 0.  Still, we would like to draw attention to App. C.1.4 where we report a complexity of $\mathcal{O}(n^2[(L+d)(n+K)+dL]+L^2(d+K))$ at worst for $n$ samples, $L$ leaves, $d$ features and $K$ clusters when choosing the optimal split. 

You may see the performance in the global response. As a follow-up on the performance metrics, we added the table with normalised KMeans score on the same benchmark as Table 3. The performances are written in the new Table 4.

Specifically, we followed the suggestion of Reviewer #3 (4Lqz) regarding the placement of section 2.3 and relegated it to the appendix for the table above. This helped the incorporation of proposed related works by Reviewer #1 (xtwY) in the coherence of the section 2.

Reviewer comment (Evaluation):
Adjusted rand index measure used to compare the clustering performance is questionable. As far as I understand, adjusted rand index uses ground-truth class labels but clustering is an unsupervised problem. Reporting both the k-means objective and GEMINI objective might help as they are the objective function being optimized.

Author response:
We already had an additional dataset (Poker hand dataset) containing approx 1 million entries. We originally discarded the dataset because all entries were 0 of score for ARI, but this experiment clearly shows that KAURI can scale to large datasets. We could not add DOUGLAS for this specific dataset due to the limitations of the Wasserstein GEMINI. We restored this dataset in the results, but discarded its ARI score because all models obtained 0.  Still, we would like to draw attention to App. C.1.4 where we report a complexity of $\mathcal{O}(n^2[(L+d)(n+K)+dL]+L^2(d+K))$ at worst for $n$ samples, $L$ leaves, $d$ features and $K$ clusters when choosing the optimal split. 

You may see the performance in the global response. As a follow-up on the performance metrics, we added the table with normalised KMeans score on the same benchmark as Table 3. The performances are written in the new Table 4.

Specifically, we followed the suggestion of Reviewer #3 (4Lqz) regarding the placement of section 2.3 and relegated it to the appendix for the table above. This helped the incorporation of proposed related works by Reviewer #1 (xtwY) in the coherence of the section 2.

Reviewer comment (Experiments):
Why is this particular neural decision tree used for the DOUGLAS algorithm? How interpretable are these trees? Why not just regular soft decision tree?

Author response:
Assuming you refer to *Soft Decision Trees O. Irsoy, O. T. Yildiz, E. Alpaydin ICPR 21*, the differentiable neural tree allows us to have a ready-made full structure when training with the Wasserstein GEMINI. Specifically, we found that this objective is expensive to compute and the optimisation of one branch parameter after another like for soft decision trees combined with the combinatorial question of which cluster to affect each new branch would cost even more than the proposal of DNDTs architectures.

The interpretability of DNDTs lies in their soft-binning functions which translate to affectation rules: certain combinations of bins lead to a specific class/cluster. That is why Yang et al., the authors, introduced the notion of *active cut points* that we used in section 5.3: these are the features for which the bins from the soft-thresholding actually leverage a partition of the data, rather than putting everything in a single bin.