Paper: Expected Probabilistic Hierarchies (ICLR-2024)

Reviewer comment (Novelty):
There is a concern about the novelty contribution when the paper is compared with [1], as the overall strategy and presentation closely resemble the scheme of [1].

Author response:
**Comment:** Similarity to FPH.  
**Response:** While our work builds upon the work of Zügner et al., it addresses multiple drawbacks of FPH. We prove that Soft-Das is a lower bound of the discrete Dasgupta cost (Sec. 4.2) and present a simple example where it fails to find the minimizing hierarchy. Based on these observations, we propose our cost functions, the expected metrics (Sec. 4.1). We motivate them by showing that the optimal hierarchies align with their discrete counterparts, unlike Soft-Das (Sec. 4.2). Furthermore, we introduce an unbiased subgraph sampling scheme, allowing us to employ both, FPH and EPH, to vector datasets. In an exhaustive empirical evaluation, we demonstrate the superiority of EPH on graph and vector datasets.

Reviewer comment (Experiments):
The example $K_4$ graph illustrates the advantage of EPH over FPH, but it would be more persuasive to provide an example graph whose edge weights are not the same. In the context of hierarchical clustering, an unweighted graph has no real hierarchy, so an instance with non-uniform weights would strengthen the argument.

Author response:
**Comment:** Weighted example where FPH fails.  
**Response:** We decided to show the unweighted $K_4$ graph as an example since it is the most simple case. We included another weighted example with four nodes in Appendix A.7. As in the previous case, EPH finds the minimizing hierarchy, while FPH fails to do so.

Reviewer comment (Experiments):
It is unclear how the quality changes with changes in the number of samples used.

Author response:
**Comment:** How does the quality change with changes in the number of samples used?  
**Response:** We show the normalized Dasgupta cost across a different number of sampled hierarchies and edges in Fig.3. As expected, the Dasgupta cost decreases when the number of samples increases. For the edges, we can already observe an improvement towards the average linkage algorithm when sampling $\sqrt{n}$ many edges. For the number of sampled hierarchies, we observe that on most of the datasets, 20 samples are sufficient. However, even when using only one sample, EPH outperforms FPH, the second-best method:


| Method | Brain | Citeseer | CoraML | Genes | OpenFlight | PolBlogs | WikiPhysics |
| ------ | ----- | -------- | ------ | ----- | ---------- | -------- | ----------- |
| FPH    | 466.94|  77.16   | 257.42 | 183.63| 355.61     |   262.48 |    538.47   |
| EPH    | 437.12|  74.35   | 244.29 | 182.09| 332.03     |  253.39  |    494.23   |

Reviewer comment (Experiments):
Please include some more details around how much the results vary with the number of samples used.

Author response:
**Comment:** How does the quality change with changes in the number of samples used?  
**Response:** We show the normalized Dasgupta cost across a different number of sampled hierarchies and edges in Fig.3. As expected, the Dasgupta cost decreases when the number of samples increases. For the edges, we can already observe an improvement towards the average linkage algorithm when sampling $\sqrt{n}$ many edges. For the number of sampled hierarchies, we observe that on most of the datasets, 20 samples are sufficient. However, even when using only one sample, EPH outperforms FPH, the second-best method:


| Method | Brain | Citeseer | CoraML | Genes | OpenFlight | PolBlogs | WikiPhysics |
| ------ | ----- | -------- | ------ | ----- | ---------- | -------- | ----------- |
| FPH    | 466.94|  77.16   | 257.42 | 183.63| 355.61     |   262.48 |    538.47   |
| EPH    | 437.12|  74.35   | 244.29 | 182.09| 332.03     |  253.39  |    494.23   |

Reviewer comment (Evaluation):
Please include speed comparisons.

Author response:
**Comment:** How much more expensive is this process compared to the baseline?  
**Response:** We include the runtimes for the experiments in Tab. 18 and Tab. 19 in the Appendix. While EPH is slower than the heuristic-based baselines, it performs similarly to the optimization-based baselines HypHC and FPH. EPH has a shorter runtime than HypHC on the graphs and large vector datasets. Furthermore, it matches the runtime of FPH for large vector datasets, as the number of sampled hierarchies is set to 1.

Reviewer comment (Evaluation):
I think that the paper could have benefited from discussion of how the proposed cost functions relate to down stream tasks of clustering (e.g., evaluation against target labels, target hierarchies, etc.) and how continuous cost functions compare to discrete ones in this setting.

Author response:
**Comment:** How do the proposed cost functions relate to downstream tasks of clustering, and how do the continuous cost functions relate to the discrete ones?  
**Response:** To analyze the relationship between the different cost functions and downstream tasks, we performed an external evaluation, as detailed in Appendix B.2. Here, we flatten the hierarchies derived from the different methods and compare the resulting clusters with the available ground-truth labels (see Table 12). 

To perform an external evaluation on graph datasets, we use two synthetic HSBMs. Our goal was to answer two questions: First, how well do the inferred hierarchies align with the ground-truth hierarchies? And second, how well do the distances correlate between graphs and hierarchies? To address the first question, we calculated the Normalized Mutual information (NMI) across the different levels of the hierarchies. For the second question, we compute the Cophenetic correlation based on the shortest path and Euclidean distances of DeepWalk embeddings (see Table 4). Exp-Das and Exp-TSD almost perfectly recover the first three levels of the ground-truth hierarchy and achieve a notably high cophenetic correlation comparable to the ground truth. In both cases, Exp-TSD achieves slightly better scores, implying a better capability to recover the ground truth.

Reviewer comment (Evaluation):
Apologies, if I have missed something, are the trees produced by EPH binary? Do you convert them into binary trees for Dasgupta cost evaluation?

Author response:
**Comment:** Are the trees produced by EPH binary?  
**Response:** EPH allows non-binary hierarchies, providing more flexibility. While the Dasgupta cost favors binary hierarchies, TSD does not. We can observe these in our qualitative evaluation in Fig. 4. While the hierarchies inferred using Exp-TSD have multiple branches, the hierarchies inferred by Exp-Das tend to be binary.