Paper: Clustering with Geometric Modularity (ICLR-2024)

Reviewer comment (Writing):
The connection of proposed geometric modularity and density-based clustering is not clearly stated.

Author response:
**Q. Connection between geometric modularity and density-based clustering.**

Thanks for raising this point. We will clarify it in the final version. The main reason for introducing this notion in metric spaces is that input to density-based clustering algorithms is often points in a vector space. We focus on DBSCAN because (i) we were able to give a theoretical explanation regarding the effectiveness of using geometric modularity, (ii) the empirical performance is also very good. DBSCAN is also one of the most widely used clustering algorithms. For $k$-means, etc. it is not clear how we would tune the hyperparameter $\rho$ as we mention in our reply to Reviewer WBYe. Exploring further applications of this notion is an interesting direction for future work.

Reviewer comment (Reproducibility):
Some part of experiments are not described clearly enough to reproduce the results.

Author response:
**Q. Experiments for reproducibility.**

Thanks for the suggestion. We will add an Appendix with full details of all the parameters. We will also make the code available before publication.

Reviewer comment (Experiments):
The connection of proposed geometric modularity and density-based clustering is not directly explained. From the definition of geometric modularity, it can be used for tuning hyper-parameters for any clustering method (the title also implies this) and is not directly related to DBSCAN. But the authors only evaluated it with DBSCAN. The authors are recommended to explain more about this, why not other clustering methods. Maybe it is due to empirical observations, then it would be nice to give evaluations of using geometric modularity on other clustering algorithms.

Author response:
**Q. Connection between geometric modularity and density-based clustering.**

Thanks for raising this point. We will clarify it in the final version. The main reason for introducing this notion in metric spaces is that input to density-based clustering algorithms is often points in a vector space. We focus on DBSCAN because (i) we were able to give a theoretical explanation regarding the effectiveness of using geometric modularity, (ii) the empirical performance is also very good. DBSCAN is also one of the most widely used clustering algorithms. For $k$-means, etc. it is not clear how we would tune the hyperparameter $\rho$ as we mention in our reply to Reviewer WBYe. Exploring further applications of this notion is an interesting direction for future work.

Reviewer comment (Reproducibility):
The method described in 4.1 is not clear enough for the others to reproduce the results. For instance, the searching ranging of \epsilon, granularity of \rho and detailed description of local search. BTW, will the authors open source the code to improve the reproducibility?

Author response:
**Q. Experiments for reproducibility.**

Thanks for the suggestion. We will add an Appendix with full details of all the parameters. We will also make the code available before publication.

Reviewer comment (Experiments):
The authors are recommended to compare with HDBSCAN, which is known to be robust to hyper-parameters. And its implementation is also available in sklearn.

Author response:
**Q. HDBSCAN.**

We will report this experiments in the final paper. And also post results on this discussion thread as soon as they are available.

Reviewer comment (Presentation):
Section 4.1 starts with "Our algorithm is outlined in Section 2". However I am not able to find a procedure such as Alg 1. It is not a big issue but for better clarity I think there should be one.

Author response:
We thought that the algorithm as described in the text was sufficient, but thanks for pointing this out, we will add it in the new version.

Reviewer comment (Novelty):
Geometric modularity is used frequently for evaluation if there's no label at all (in contrast to pseudo-unsupervised), it is already a routine to examine clustering results. Therefore introducing an algorithm simply maximizing it does not shed new light. This also leads to a weak theoretical analysis which is basically proving what is assumed. I understand the authors want a "reasonable" datasets. But the point of beyond-worst-case analysis, per my understanding, is to avoid pathological instances but not to design properties of datasets tailored to what we want to prove. In definition 3.1, for example if one item is removed or relaxed, the proof breaks down.

Author response:
We are very confused by the statement that geometric modularity is frequently used. We are not aware of any work mentioning “geometric modularity”. We acknowledge that there are various works on graph modularity but one of the novelties of this work is to introduce a notion of modularity for (pseudo)metric spaces.  

It seems that there is a confusion here: the fact that the input exhibits clusters does not mean that the algorithm can find them. The assumptions used here only state that there is an underlying clustering and not that this clustering is tailored to the algorithm since similar assumptions have appeared in the literature before. In fact, such an analysis is lacking for most algorithms and, for example, DBSCAN with poor parameter choices would fail to recover the clusters.

Reviewer comment (Novelty):
I agree with the claim that "The unsupervised geometric modularity tracks a supervised measure AMI". But the positive relationship between these two measures is also known to the community, especially in the lens of density-based clustering. I am not sure if any previous work formally states this, but in this paper it is still shown by experiments. But anyway I can stay open to discussion.

Author response:
We are not aware of any work connecting geometric modularity with AMI (also because we are not aware of any other work mentioning the quantity “geometric modularity” that we introduce here). If the reviewer could point us to these works, we could reference them in our paper.

Reviewer comment (Theory):
If I am understanding correctly, the motivation is to get rid of the parameter choice on $\varepsilon$, then what is the point if we have to choose $\rho$ again? Indeed, theorem 3.2 gives a range, but it is subject to other parameters given by the dataset, and according to section 4, it is selected in [0.5, 1].

Author response:
The main point here is that there is no such thing as getting rid of parameters. The only way to do this is to set them to default values and show that they are good, or to show that they can be auto-tuned using a data-dependent fashion. We take the latter approach. We believe it is not straightforward to choose $\epsilon$ in a data-dependent fashion, but picking $\rho$ is easier as outlined in Section 4. As $\rho$ can be auto-tuned, once $\rho$ is fixed we use geometric modularity can be used to pick $\epsilon$.

Reviewer comment (Experiments):
The experiments include quite a number of datasets but not enough clustering methods. For example, SOM, HDBSCAN, or even $k$-means with the best $k$.

Author response:
Thanks for the suggestion we will add experiments with  HDBSCAN. For $k$-means we are less sure what the reviewer has in mind. Selecting $k$ is indeed the part hard of the problem (in practice) and so for a fair comparison that will require us comparing heuristics that are very different from those for density based clustering, which is our focus in this paper.

We will post results with HDBSCAN on this discussion forum as soon as they are available.

Reviewer comment (Novelty):
I wonder if computing the modularity in linear time is one of the contributions of this work? If not, is it folklore (just a practical heuristic) or proposed by previous work?

Author response:
Graph Modularity would require a computational time proportional to the number of edges of the graph (which is at least as large as the number of vertices). Our new geometric modularity framework takes a more global vision of the dataset (by looking at all pairwise distances) and that can be done in time linear in the number of data elements. This is indeed one of the contributions, and while the proof itself is simple, we believe this is what makes the approach attractive in practice.

Reviewer comment (Experiments):
I do not see a straightforward break-down if we apply the geometric modularity to other clustering methods, even $k$-means/median/center. Is there any foreseeable problem?

Author response:
This is indeed an interesting research direction to explore. However, as mentioned above picking $k$ itself is not usually an easy problem, and unlike in the case of density-based algorithms it is not clear how we would pick $\rho$. Although Geometric modularity with a fixed value of $\rho$, e.g. 0.8 does work well most of the time, to obtain better results, we do need to be able to tune $\rho$.

Reviewer comment (Novelty):
The proposed geometric modularity seems identical to Newman's weighted modularity [2004, https://journals.aps.org/pre/abstract/10.1103/PhysRevE.70.056131], but this link to a well-established variant has not been made by this paper.

Author response:
Geometric modularity is a modularity framework for Euclidean spaces, it deals with distances and not similarity edges. This is a fundamental difference between the two definitions. In particular, the definition in [Newman](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.70.056131) would have limited meaning in Euclidean space and would not have the properties that we show for our definition. We note that we have cited Newman’s paper on graph modularity and we will add a citation to this one (thanks for the pointer).

Reviewer comment (Evaluation):
The claim that DPC did not result 'in any meaningful results' is not supported by facts.

Author response:
DPC always reports one cluster in the experiments we have performed (and simply doesn't run in reasonable time for larger datasets), the bound on AMI follows directly. Although it is possible that hyperparameters of DPC could be tuned, as our key contribution is better auto-tuning of DBSCAN hyperparameters, we believe it is fair that we compare to other available methods out of the box that also claim to autotune hyperparameters.

Reviewer comment (Experiments):
The empirical results lack context. That is, the paper does not compare geometric modularity to other `competing' ("internal"/"unsupervised") clustering-quality measures for tuning DBSCAN, including the classical Silhouette coefficient, Davies–Bouldin index, or Dunn index (under CV or model selection criteria).

Author response:
The fact that the geometric modularity (for squared Euclidean distance) can be computed in linear time is actually key to the implementation. Computing other measures (as the one suggested below) requires a time quadratic in the number of input points, and so are not scalable. The geometric modularity measure is the only measure among the ones listed above that is scalable.

Reviewer comment (Writing):
The writing is unclear, not technical, and includes weasel-words, hyperbole, and unspecific adjectives such as "few papers", "some", "tracks AMI incredibly well", "better output quality", or "not much has been done".

Author response:
We apologize if the writing appeared as lacking rigor. We are working on an improved version that we will soon post. AMI is one of the most standard ways of measuring cluster quality — proposed as an improvement to NMI which often favors clusters with outliers.  See for example [[1]](https://proceedings.mlr.press/v139/gosgens21a.html).

Reviewer comment (Evaluation):
The paper does not properly motivate the usage of AMI well.

Author response:
AMI is one of the most standard ways of measuring cluster quality — proposed as an improvement to NMI which often favors clusters with outliers.  See for example [[1]](https://proceedings.mlr.press/v139/gosgens21a.html).

Reviewer comment (Reproducibility):
The paper does not properly describe their local-search post-processing heuristic, which had a profound impact on the performance, thus inhibiting the reproducibility.

Author response:
Thank you for your comment, we will add this to the new version we will post. But in short, it is the standard local search approach, tentatively move one point from one cluster (picked at random) to another (random) cluster - accept the move if the geometric modularity increases, reject otherwise. Stop if no accepting move is found for a certain preset number of attempts.

Reviewer comment (Theory):
I don't understand the formal argument and the implication on the identifiability of the optimal solution, of your suggestion to smooth-over the hard-to-optimize (rugged?) solution landscape using isotonic regression. Could you expand on this?

Author response:
The only place we suggest using isotonic regression is to autoselect $\rho$. While we expect the #clusters obtained to be a decreasing function of $\rho$, this is not a formal statement as DBSCAN clusters may exhibit different behavior from expectation (this was not the case in our experiments). Since the way we select $\rho$ is to pick the most stable part of the curve for #clusters v $\rho$ we recommend making the curve monotonic (if needed to remove outliers) before picking $\rho$. We will clarify this in the Appendix in the revised version of the paper with some plots and more explanation.

Reviewer comment (Experiments):
Given that the modularity ifself can be used for community detection in network data, and various approaches exist for clustering by maximzing the modularity, such as those mentioned in [1]. I think it is natural to ask if we can also apply similar approahces to the geometric modularity here, without using any auxiliiary tools such as DB-SCAN. If there are any critical challanges e.g. computational complexity, it might be worthy to point these out, and such challenges can be also good motivations for instead applying geometric modularity to DB-SCAN. Currently, it seems like there is a lack of such discussion.

Author response:
Thanks for pointing this out. Optimizing geometric modularity directly is an interesting open problem. Optimizing directly is NP-hard (same reason as the original modularity) and we do not think that classic approaches such as local search would lead to any formal result in this space. We focused on a new algorithm based on DBSCAN because it had interesting theoretical properties (as shown in our paper) and also because the experimental results were quite interesting. We will expand the discussion in the final version of the paper.

Reviewer comment (Novelty):
I was a little bit confused about the originarity of the geometric modularity, as it is defined very similar to Eq.(1.1) in [2] which is also mentioned by the authors. Therefore, I think in prior to introducing the geometric modularity, it might be better to give a more detailed introducing fisrt on the modularity on unweighted graph, such as Eq.(1) in [1], then its generalized version on weighted graph, e.g. Eq.(1.1) in [2]. As a result, the geometric modularity can be better motivated and their differences (I think the biggest difference is on the extra parameter $\rho$) is more clearly demonstrated.

Author response:
Thanks for pointing this out. A key difference between the two modularities is that one deals with distances (our case) and the other with similarity. Although the definition of geometric modularity itself does not require $d$ to be a metric, the theoretical analysis we present does as it uses the triangle inequality (our theoretical analysis also works with somewhat weaker variants of triangle inequality, but some constraints are required). This is why we cannot directly adopt a version of modularity defined over weighted graphs; though the definition itself is very similar and is indeed inspired by work in graph modularity.

Reviewer comment (Experiments):
In the experimental results, the performance of OPTICS is dramatically poor compared to the DB-SCAN with geometric modularity. However, given that the high similarity between OPTICS and DB-SCAN, I would not expect such a huge performance gap exists, and thus I am very curious on the root causes, which could fall onto the following two categories from my opinions:  
1. The methodology of OPTICS essentially fails in these datasets, no matter how we choose the hyper parameters  
2. The result of OPTICS is sensitive to the hyper-parameters, and the parameters chosen in the experiments is still far from the optimal one.

Author response:
Although OPTICS does give a way to automatically perform hyperparameter selection on DBSCAN, it does not in fact consider all possible (or even a large range of possible) values of $\epsilon$ and then perform hyperparameter autotuning. Instead, it uses a clever heuristic to order the points according to distances, and looks for steep changes in these distances as a means to identify when a new cluster has been discovered. This parameter $\xi$ is what controls this and its effect is also presented in our additional experiments. This approach works quite well in very low dimensions, but starts to suffer in even modest dimensions.