Paper: Average Sensitivity of Hierarchical Clustering (ICLR-2024)

Reviewer comment (Novelty):
Applications of methods from differential privacy, i.e., exponential mechanism, may not be as surprising as initially seemed, due to connections to the notion of worst-case sensitivity in differential privacy

Author response:
Differential privacy (DP) corresponds to measuring the sensitivity using TV distance with $\max\_{x \in X}$ instead of $\frac{1}{n} \sum\_{x \in X}$ in Eq. (1), i.e., DP is more pesimistic than Average Sensitivity (AS). Because of this relation, one can derive a trivial bound like $\mathrm{AS} \le \mathrm{DP} \cdot n$. In general, ensuring the stability with respect to DP inccurs worse complexity.

Reviewer comment (Theory):
The main approximation guarantee for the stable-on-average hierarchical clustering algorithm seems to be in terms of the expected size of the sparsest cut, rather than the expected cost of the clustering

Author response:
As you noted, we will be able to obtain some approximation guarantees for useless algorithms (such as random cut). The challenge is to provide both the approixamtion and stability guarantess for useful algorithms that can produce high-quality clustering, which is unfurtunately beyond the scope of the current paper.

Reviewer comment (Experiments):
Other hierarchical clustering objectives such as the Moseley-Wang or Cohen-Addad objectives are not considered

Author response:
Moseley-Wang is essentially the same as Dasguta score, while Cohen-Addad is a generalization of Dasguta score. The question when using Cohen-Addad is on the choice of the function $g$, where Dasguta score corresponds to a specific choice of $g$. We there think the use of Moseley-Wang or Cohen-Addad do not add much on our results.

Reviewer comment (Theory):
What can be said about the approximation guarantees of the main algorithm in terms of the Dasgupta objective? Can anything be said about the approximation guarantees of the algorithm given an $\alpha$-approximation algorithm for sparsest cut?

Author response:
Theorem 12 of (Dasgupta, 2016) showed the $O(\alpha \log n)$ approximation.

Reviewer comment (Evaluation):
The experimental analysis appears to be insufficient. Firstly, the experiments are conducted on three benchmark datasets and a single set of real-world data, which results in a relatively small dataset size. Secondly, common external clustering performance evaluation metrics, such as NMI, ARI, and ACC, are not employed in the experiments. Lastly, there is no clear discernible consistent trend evident from Figure 2 and the related figures in the supplementary materials.

Author response:
The purpose of hierarchical clustering is to visualize the relationships between the data points. Visualization of thousands of data points are generally hard for users to inspect. Because of this reason, hierarchical clustering is better suited for the cases where the number of data points are not too large.

We used the metrics tailored specifically for hierarchical clustering, which are Dasgupta score, Dendrogram purity, and Cophenetic correlation. We believe these are appropriate choices for assessing the proposed method, rather than using generic metircs.

We nevertheless agree with the importance of evaluating the proposed method using several different metrics. However, we have to say that it is not possible to evaluate the method with (possibly) tens or hundreds of metrics. We therefore believe that the exhaustive evaluation is not the duty. Our responsibility is to relase the code so that any users with interest in alternative metrics can perform evaluations by modifying the provided code.

We do not say that there is a discernible consistent trend in the results. Our bound $O(\lambda Dn)$ is the worst-case upper bound. It is sometimes the case that the worst-case bound and practical performances have some gaps. Our results indicate that using a slightly larger $\lambda$ can result in better stability in some cases.

Reviewer comment (Theory):
Why is sensitivity crucial for hierarchical clustering?

Author response:
Hierarchical clustering is used to understand the data visually. If a cluster structure changes drastically, the users will be confused whether their data understanding is truly reliable or only a noisy artifact induced by the unstable clustering algorithm. The stability is essential for reliable data understading.

Reviewer comment (Theory):
SHC has time complexity $O(D n^3)$. This makes it impractical to moderate to large size data sets, which is perhaps the reason why experiments are conducted on mostly small data sets. I would like to see a more efficient algorithm, or at least an approximate one with proven theoretical bound.

Author response:
We would like to remind that the typical agglomerative methods also run in $O(n^3)$ time. Because small $D$ is typicall chosen, the $O(Dn^3)$ time of the propsoed method is comparable with the existing methods. We also observed in the experiments that the proposed methos can run faster than the agglomerative methods for $D \le 10$.

Reviewer comment (Theory):
The average sensitivity of SHC is $O(\lambda D n)$, which can be made small by setting $\lambda \ll 1$. In the experiments, this is not always the case, for instance in Section 8. This signifies that SHC in practice doesn't really have a low average sensitivity.

Author response:
Our bound $O(\lambda Dn)$ is the worst-case upper bound. It is sometimes the case that the worst-case bound and practical performances have some gaps. Our results indicate that using a slightly larger $\lambda$ can result in better stability in some cases.

Reviewer comment (Experiments):
The data sets chosen in the experiments are way too small. This makes the results not so convincing.

Author response:
The purpose of hierarchical clustering is to visualize the relationships between the data points. Visualization of thousands of data points are generally hard for users to inspect. Because of this reason, hierarchical clustering is better suited for the cases where the number of data points are not too large.

Reviewer comment (Theory):
For the first point, it would help to see some notion of lower bound to better understand the complexity of the problem.

Author response:
Unfortunately, we do not have a rigid answer to this point. The complexity analysis requires an approximation guarantee for the sparsest cut problem. However, the known approximation algorithm are computationally quite demanding even though they are polynomial time, which can result in $O(D \cdot \mathrm{poly}(n))$ time.

Reviewer comment (Theory):
The proposed algorithm makes use of the exponential mechanism from differential privacy. How does average sensitivity relate to differential privacy? Does one notion imply the other or vice versa? Both seem to capture the intuitive idea that the output distribution for similar inputs should be similar.

Author response:
Differential privacy (DP) corresponds to measuring the sensitivity using TV distance with $\max\_{x \in X}$ instead of $\frac{1}{n} \sum\_{x \in X}$ in Eq. (1), i.e., DP is more pesimistic than Average Sensitivity (AS). Because of this relation, one can derive a trivial bound like $\mathrm{AS} \le \mathrm{DP} \cdot n$. In general, ensuring the stability with respect to DP inccurs worse complexity.

Reviewer comment (Evaluation):
Can the quality of the proposed algorithm be analyzed in terms of some cost or utility function (e.g., Dasgupta's cost function or the utility function due to Moseley and Wang)?

Author response:
For the analysis, we need an approximation guarantee for the sparsest cut problem. The typical approximation algorithms are LP (Leighton-Rao) and SDP (Arora-Rao-Vazirani). However, the rounding operation in LP will not be stable, and SDP is generally too slow. In the current state, we do not have a good way to ensure the approximation and stability guarantees in the same time.