Paper: Village-Net clustering: A novel unsupervised manifold clustering method (ICLR-2024)

Reviewer comment (Reproducibility):
It is not clear whether the method estimates the final number of clusters or the number of clusters is given as input to the method.

Author response:
Our method estimates the number of clusters.

Reviewer comment (Experiments):
Experimental part is weak and could be improved (see questions below).

Author response:
We address this below.

Reviewer comment (Evaluation):
If the method estimates the number of clusters, then results should be provided on how close are the estimations to the actual values.

Author response:
We agree and have demonstrated this using a number of metrics, specifically NMI, adjusted RAND, among others.

Reviewer comment (Experiments):
Since the similarity matrix is computed, typical agglomerative approaches (e.g., single linkage) or spectral clustering methods could have been used instead of the WLCF algorithm. What is the advantage of WLCF?

Author response:
WLCF is shown to be better than state of the art algorithms like Louvain's algorithm. Plus unlike spectral clustering, it does not need the number of communities as an input, instead the number of communities is given as an output.

Reviewer comment (Theory):
A short presentation of the WLCF method should be included. Does this method contain hyperparameters? What is the theoretical complexity (not the empirical complexity) of the method?

Author response:
In the revised version, a more detailed explanation of WLCF is included. The theoretical complexity of WLCF has not been calculated in the paper.

Reviewer comment (Experiments):
The number of attributes in Pendigits dataset is not 64 but 16.

Author response:
We used the sklearn version of digits. It has 64 (8X8) parameters

Reviewer comment (Experiments):
Experiments using additional datasets could have been included to draw more reliable conclusions on the performance of the method.

Author response:
The revised version contains this.

Reviewer comment (Novelty):
The novelty of this work is relatively limited, and it seems that only the construction of a village network is proposed in this paper.

Author response:
We have expanded the discussion to include the broader contribution of our paper.

Reviewer comment (Experiments):
The comparison algorithms used in the experimental part are few, and all of them are early clustering algorithms, so the comparison experiments with the clustering algorithms in recent years should be added.

Author response:
We have significantly expanded the list of comparison algorithms in the revised version of the paper.

Reviewer comment (Experiments):
According to the part of the effect of hyperparameters in this paper, the algorithm is affected by k and r, but there is no strategy to analyze the selection of hyperparameters, and r is fixed at 20 in the experiment part, and comparative experiments for hyperparameter analysis should be added.

Author response:
We have included an extensive analysis of the effects of the hyperparameters.

Reviewer comment (Experiments):
Compared with recent clustering algorithms, is the performance of the proposed algorithm still better?

Author response:
We have provided a better comparison of our method in the revised version. However because of lack of publicly available code, we found it hard to compare our algorithm to the recent one.

Reviewer comment (Theory):
Can the convergence of the proposed algorithm be proved theoretically?

Author response:
We haven’t found a method to do that yet. In the future, we plan to work on it.

Reviewer comment (Evaluation):
When evaluating the performance of the algorithm, Table 2 only takes NMI as an evaluation index. Can the author verify the effectiveness of the proposed algorithm under more evaluation indexes?

Author response:
We have expanded the list of evaluation metrics for making a more persuasive case.

Reviewer comment (Experiments):
This paper uses the k-means algorithm for initialization. If the number of class clusters in the data set is large, such as greater than 100 or 1000, is this algorithm applicable?

Author response:
Our algorithm can work well as long as we have the condition m<<k<<N where m is the number of class clusters, k is the number of villages and N is the number of datapoints.

Reviewer comment (Novelty):
The contribution of the paper is not so clear. The idea of over-clustering + merging is not new, which can be find in many previous methods, like BIRCH.

Author response:
We have expanded the discussion to include the broader contribution of our paper. We have also included a comparison to other methods, such as BIRCH.

Reviewer comment (Novelty):
The proposed method is a combination of different techniques, but it is not clear why this combination is unique and which specific unsolved problem it can handle.

Author response:
We have explicitly provided a more detailed explanation of what makes Village-Net innovative, emphasizing its fast clustering capabilities for complex data.

Reviewer comment (Experiments):
The comparison methods are weak, only K-means and DBSCAN.

Author response:
We have added more comparison methods that are unsupervised and computationally feasible.

Reviewer comment (Experiments):
The performance of the proposed method strongly depend on K-means and hyper-parameters.

Author response:
While it's acknowledged that the VillageNet clustering method depends on the initial performance of K-Means clustering, we emphasize that our primary goal is not to cluster the data effectively into k villages. Instead, we use K-Means to form a network of villages, leveraging the information from this clustering to achieve the final results. Specifically, choosing k such that k≫m ensures meaningful clustering. We also include a detailed analysis of the key hyperparameters.

Reviewer comment (Novelty):
Where is the title of unsupervised manifold clustering reflected in the manuscript? The authors should have explicitly described what is innovative about Village-Net.

Author response:
We have explicitly provided a more detailed explanation of what makes Village-Net innovative, emphasizing its fast clustering capabilities for complex data.

Reviewer comment (Experiments):
The authors should consider collecting more publicly available data to confirm the validity of their Village-Net Clustering.

Author response:
We acknowledge the importance of publicly-available data. As such, we have added a few public datasets and artificial datasets as benchmarks to our analysis.

Reviewer comment (Experiments):
The authors did not compare their method with latest state-of-the-art methods. They may need to compare Village-Net with other unsupervised manifold clustering methods.

Author response:
We agree that we should have compared our methods with the latest state-of-the-art methods. We have run new experiments with other unsupervised clustering techniques in order to fully compare our method with others.

Reviewer comment (Novelty):
Where is the title of unsupervised manifold clustering reflected in the manuscript? The authors should have explicitly described what is innovative about Village-Net.

Author response:
We have explicitly provided a more detailed explanation of what makes Village-Net innovative, emphasizing its fast clustering capabilities for complex data.

Reviewer comment (Experiments):
The authors should have discussed the effect on the model of the choice of the hyperparameter r. r is a key parameter in Village-Net, and the clustering effect is highly dependent on r. However, the manuscript does not provide any details on how to choose r.

Author response:
In the new version, we've included a detailed analysis of the hyperparameter "r" to provide a better understanding of its impact on the algorithm and clustering results.

Reviewer comment (Experiments):
The authors should consider collecting more publicly available data to confirm the validity of the Village-Net Clustering.

Author response:
We acknowledge the importance of publicly-available data. As such, we have added a few public datasets and artificial datasets as benchmarks to our analysis.

Reviewer comment (Evaluation):
The authors mention Village-Net clustering outperformed other algorithms on the FMNIST and Letters and is the second best in all the other datasets. The manuscript only use NMI to evaluate the clustering, and also need to consider more evaluation metrics on the performance comparison is more persuasive.

Author response:
In response to your suggestion, we've expanded the set of evaluation metrics in the revised version to provide a more comprehensive assessment of the proposed algorithm's performance.

Reviewer comment (Experiments):
The authors may need to compare Village-Net with other unsupervised manifold clustering methods.

Author response:
We agree that we should have compared our methods with the latest state-of-the-art methods. We have run new experiments with other unsupervised clustering techniques in order to fully compare our method with others.

Reviewer comment (Experiments):
How to improve the model generalization ability? For different datasets, the hyperparameter k needs to be adjusted individually to obtain superior results, indicating poor model generalization ability.

Author response:
In general we expect stable results across a range of hyperparameter k.

Reviewer comment (Reproducibility):
How to divide a network of villages into disjoint village communities using WLCF? Please describe the WLCF algorithm in detail.

Author response:
We've enhanced the clarity of the description of WLCF in response to your feedback.

Reviewer comment (Writing):
Please check that punctuation is used correctly in the manuscript. For example, Page 1 line 1 ”Village-Net Clustering,” should be changed to “Village-Net Clustering”.

Author response:
We believe these revisions address the concerns raised, providing a more robust and comprehensive presentation of our work. We appreciate your insightful comments and hope the revised manuscript meets your expectations.

Reviewer comment (Writing):
Page 4 line 9, “T-SNE” should be changed to “t-SNE”.

Author response:
We believe these revisions address the concerns raised, providing a more robust and comprehensive presentation of our work. We appreciate your insightful comments and hope the revised manuscript meets your expectations.

Reviewer comment (Writing):
Page 5 line 12, “O(N ∗ k)” should be changed to “O(N ∗ k)”.

Author response:
We believe these revisions address the concerns raised, providing a more robust and comprehensive presentation of our work. We appreciate your insightful comments and hope the revised manuscript meets your expectations.

Reviewer comment (Experiments):
Lack of More Experimental Validation: The algorithm was only compared with a few traditional clustering algorithms, and the experimental results indicate that the proposed algorithm performs well only on two datasets. Therefore, it is difficult to demonstrate that the proposed algorithm exhibits competitive performance.

Author response:
We've performed additional experiments and comparisons with more clustering algorithms in the revised version. However, obtaining comparisons with certain new clustering algorithms is challenging due to unavailability or impractical implementation on large datasets.

Reviewer comment (Evaluation):
The evaluation metrics for the experimental results are too narrow. The author solely relies on NMI as the sole measure.

Author response:
In response to your suggestion, we've expanded the set of evaluation metrics in the revised version to provide a more comprehensive assessment of the proposed algorithm's performance.

Reviewer comment (Experiments):
The experimental section lacks an analysis of the hyperparameter "r".

Author response:
In the new version, we've included a detailed analysis of the hyperparameter "r" to provide a better understanding of its impact on the algorithm and clustering results.

Reviewer comment (Experiments):
WLCF is one of the crucial steps in the algorithm. However, the author's description of WLCF is not sufficiently clear. Additionally, during the experimental process, I would like to see some ablation experiments to verify the impact of each step in the algorithm on the final clustering results.

Author response:
We've enhanced the clarity of the description of WLCF in response to your feedback. Additionally, we acknowledge the importance of ablation experiments and will consider incorporating them in future work.

Reviewer comment (Experiments):
I suggest that the author perform visual analysis on some artificial non-convex datasets (such as Two Moon, Flower, etc.) and visualize each step of the algorithm to illustrate the effectiveness of the proposed algorithm.

Author response:
To illustrate the effectiveness of the proposed algorithm, we've included visualizations of key steps on an artificial dataset in Fig 2.