Paper: LMCC-MBC: Metric-Constrained Model-Based Clustering with Wasserstein-2 Distance of Gaussian Markov Random Fields (ICLR-2024)

Reviewer comment (Theory):
In the contribution section: the authors indicate that they present a solid mathematical proof of the soundness of their generalized semivariogram. But there isn’t any proof in this paper and later they say “The soundness of this generalized definition can be easily verified on real-world datasets”.

Author response:
**A1:** Briefly speaking, the expected Wasserstein-2 distance for a given metric distance bin is part of the computation for determining the threshold of goodness-of-fit tests, and we formally define it as Model-based semivariogram in Section 4.1 since it has similar form to the classical semivariograms. A detailed discussion and derivation can be found in Appendix 7.4. *

Reviewer comment (Theory):
Same problem for the fourth contribution. It would be better to at least have a complexity analysis for your algorithm to say that your algorithm solves the problem efficiently with “solid theoretical proofs”.

Author response:
**A2:** Please kindly refer to our general response Q3 as well as Appendix 7.4 for a detailed discussion and additional experiments.

Reviewer comment (Theory):
I think the authors need to justify the use of GMRF in the model since they are fitting a GMRF for each sample. It would make more sense to fit one GMRF for one cluster as different data clusters will naturally have different variable dependency structures.

Author response:
**A3:** Please see Part 2 of our general response as well as Appendix 7.4 about the main contribution of our work: a novel family of clustering objectives -- i.e., instead of optimizing the total data likelihood, we propose to minimize the number of failed goodness-of-fit tests, which compare pairs of data points and their underlying models in the clustering.

Fitting one model (e.g., GMRF) for one clustering is commonly used in existing model-based clustering algorithms such as TICC and STICC, which optimize the total data likelihood. They require a non-trivial  estimation-maximization (EM) procedure which leads to several drawbacks in efficiency and modeling:. 
Firstly, the EM step is very time-consuming and takes many iterations to converge. Secondly, the clustering result relies heavily on the initial assignment, and the optimization is subject to local optima. Thirdly, tuning the hyper-parameters (e.g., number of clusters) requires expensive retraining. Finally, data likelihood objectives have a lack of flexibility when modeling constraints such as spatial metric autocorrelation as a generative process. These problems can be avoided by using a bottom-up model-based clustering algorithm, i.e., data points merge into clusters according to their underlying model similarity (like DBSCAN). 

In order to enable a bottom-up model-based clustering, we need to learn a GMRF for each data sample instead of learning one GMRF for one cluster. This allows us to perform the goodness-of-fit test (i.e., test whether two samples come from a statistically identical distribution) between any given two data samples in order to determine whether we can merge these two data samples into one cluster.

In short, we estimate one GMRF for one data sample instead of one cluster to enable a  bottom-up DBSCAN-like model-based clustering rather than the traditional kMeans-like practice. 

For Q3, the GMRF is learned from the neighborhood of a sample, i.e., what the hyperparameter $n$ of "number of neighbors" controls. We usually choose $n>30$ so that the estimation is statistical significant. This is a common practice in model-based clustering. For example, TICC and STICC both used the neighborhood to estimate underlying models for each sample. The difference is that they additionally estimate a common underlying model for each cluster.

I wish this addresses your concerns.

Reviewer comment (Experiments):
One of the advantages of the proposed method they can deal with arbitrary dimensions of constraint space instead of just 1-D and 2-D. Did you try it with a 3-D dataset?

Author response:
**A4:** This is a very good point. We have found 3-D datasets about point cloud-based scene segmentation and are working on the experiments. Due to time limits of rebuttal, we can not present the results in the rebuttal, but we will add this result to the camera-ready version.

Reviewer comment (Reproducibility):
In section 5.1, the authors claim that the proposed method has only three hyperparameters to tune. What about the tuning parameter for Graphical Lasso?

Author response:
**A5:** We grid-searched the hyperparameters alpha, tolerance and maximum number of iterations for Graphical Lasso. The hyperaramters we use are alpha=0.01, tolerance=1e-4 and maximum iterartion=1000. Theoretically, as long as Graphical Lasso converges, it yields very stable covariance estimation. Thus there is no need to tune these hyperparameters on the entire LMCC-MBC algorithm. We simply find out hyperparameters that enables Graphical Lasso to converge on a subset of the dataset and fix them throughout the experiments.

Reviewer comment (Theory):
As with many clustering algorithms, there could be a theoretical comparison analysis of the space and time complexities for LMCC-MBC and competing SOTA techniques.

Author response:
**Q1: Theoretical comparison analysis of the space and time complexities for LMCC-MBC and competing SOTA techniques**

**A1:** Please refer to our general response Q3 as well as Appendix 7.5 for a detailed discussion and additional experiments.

Reviewer comment (Theory):
Regarding the following claim:  “In fact, Wasserstein-2 distance is the only feasible choice of model distance that theoretically guarantees the generalized model-based semivariogram is compatible with the classic definition. Consequently, GMRF is chosen because it is the most computationally efficient model parameterization under Wasserstein-2 distance. The following section will prove this in detail.” It does not sound to me that there is a proof here. The usage of Wasserstein-2 distance and GMRF is justified, yes, but I don't have the feeling that this proves that these are the only options as stated.

Author response:
**Q4: The proof of the usage of Wasserstein-2 distance and GMRF**

**A4:** Please refer to our general response Q2 for the reason for our choice of Wasserstein-2 distance, GMRF, Graphical Lasso, and DBSCAN. We also provide the theoretical foundation of the reason why we use Wasserstein-2 distance (see our general response Q1). Essentially, in order to enable a bottom-up model-based clustering, we need to formulate the loss function (see Equation 4 ) in terms of only pairwise computations between data points. The goodness-of-fit test (i.e., test if two samples come from a statistically identical distribution) statistic perfectly matches our needs. The square of Wasserstein-2 distance (Panaretos & Zemel, 2019) satisfies our requirements since it metricizes weak convergence.

Reviewer comment (Experiments):
No experimental study of the effects of the number of neighbors and the shift parameter on accuracy.

Author response:
**Q2: No experimental study of the effects of the number of neighbors and the shift parameter on accuracy.**

**A2:** Please refer to our general response Q4 and Appendix 7.6 for the added ablation studies on these two hyperparameters.

Reviewer comment (Writing):
Minor typo — missing space after “unknown” in intro p.1.

Author response:
**Q6: Typos.**
**A6:** We are sorry about these typos. We have corrected the typos in our updated version, highlighted in red.

Reviewer comment (Writing):
Minor typo — "requirment", p.7.

Author response:
**Q6: Typos.**
**A6:** We are sorry about these typos. We have corrected the typos in our updated version, highlighted in red.

Reviewer comment (Writing):
Minor typo — "natrually", p.7 before eq. 11.

Author response:
**Q6: Typos.**
**A6:** We are sorry about these typos. We have corrected the typos in our updated version, highlighted in red.

Reviewer comment (Writing):
Minor typo — "fittiing" p.8 in Algorithm 1.

Author response:
**Q6: Typos.**
**A6:** We are sorry about these typos. We have corrected the typos in our updated version, highlighted in red.

Reviewer comment (Writing):
Minor typo — "hyperparamters" p.16.

Author response:
**Q6: Typos.**
**A6:** We are sorry about these typos. We have corrected the typos in our updated version, highlighted in red.

Reviewer comment (Writing):
Minor typo — missing space after “our baselines(Kang et al, 2022) p.16.

Author response:
**Q6: Typos.**
**A6:** We are sorry about these typos. We have corrected the typos in our updated version, highlighted in red.

Reviewer comment (Writing):
Minor typo — "unlined" instead of "underlined" in p.17.

Author response:
**Q6: Typos.**
**A6:** We are sorry about these typos. We have corrected the typos in our updated version, highlighted in red.

Reviewer comment (Theory):
The effectiveness of modification from Eq. (5) to (6) is empirical and lacks a theoretical guarantee.

Author response:
**A1:** Thank you for raising this important question! We provide more detailed reasoning in the general rebuttal Q1 and Appendix 7.4. Briefly speaking, the expected Wasserstein-2 distance for a given metric distance bin is part of the computation for determining the threshold of goodness-of-fit tests, and we formally define it as Model-based semivariogram in Section 4.1 since it has similar form to the classical semivariograms.

Reviewer comment (Novelty):
The proposed method just combines several existing components. Even though the authors argue that the chosen combination is not heuristic, and provides some special theoretical properties of them, the overall technical contribution looks less significant to me.

Author response:
**A2:** Thank you for  raising this important question regarding our work! Please kindly refer to the general rebuttal Q2 for a discussion about our main contribution, which we briefly summarize here:
The most important contribution of our work  is  a novel family of clustering objectives -- i.e., instead of optimizing the total data likelihood, we propose to minimize the number of failed goodness-of-fit tests, which compare pairs of data points and their underlying models in the clustering. Compared to total data likelihood based objectives, this novel approach enjoys several benefits in efficiency and modeling (see details in Appendix 7.4).

The choice of specific implementations, i.e. Graphical Lasso and DBSCAN, is not important. Using other covariance estimation and distance-based clustering algorithms yields comparable results. We refer to Appendix 7.7.1 (Table 3) and 7.7.2 (Table 4) for the ablation studies on this. That demonstrates we are not mere heuristic combinations of existing algorithms.

Reviewer comment (Experiments):
The experiments are weak. Only an overview of clustering performance is provided. Ablation studies of the design choices are lacking, so these claims are not well-supported. Comparisons with strong baselines are also lacking.

Author response:
**A3:** Thanks for your suggestion. We updated our paper and added an ablation study on the influence of hyperparameters. Please kindly refer to the general rebuttal Q4 and Appendix 7.6. In general, the conclusion is that both the number-of-neighbor hyperparameter and the shift hyperparameter have nearly convex responses. That means we can easily tune them by simple hierarchical grid search.

Additionally, Table 1 in our paper does not only report the overall clustering performance, but also compares the effectiveness of introducing spatial information (i.e., we compared performances w/wo spatial distance). We can see from the results that introducing metric distance into the clustering objective significantly improves the clustering accuracy.
We also added ablations studies on the choice of covariance matrix estimation methods (*Graphical Lasso* v.s. *Minimum Covariance Determinant* v.s. *Shrunk Covariance*) and the choice of distance-based clustering (*DBSCAN* v.s. *HDBSCAN* v.s. *OPTICS*). Please refer to our general response Q2.3, Appendix 7.7.1 and Appendix 7.7.2.

Reviewer comment (Novelty):
Despite the clarification that the major contribution is minimizing the number of failed goodness-of-fit tests rather than optimizing the total data likelihood, the components are still taken as-is despite being well-crafted, so the original concern about limited technical contribution remains.

Author response:
**A2:** Thank you for  raising this important question regarding our work! Please kindly refer to the general rebuttal Q2 for a discussion about our main contribution, which we briefly summarize here:
The most important contribution of our work  is  a novel family of clustering objectives -- i.e., instead of optimizing the total data likelihood, we propose to minimize the number of failed goodness-of-fit tests, which compare pairs of data points and their underlying models in the clustering. Compared to total data likelihood based objectives, this novel approach enjoys several benefits in efficiency and modeling (see details in Appendix 7.4).

The choice of specific implementations, i.e. Graphical Lasso and DBSCAN, is not important. Using other covariance estimation and distance-based clustering algorithms yields comparable results. We refer to Appendix 7.7.1 (Table 3) and 7.7.2 (Table 4) for the ablation studies on this. That demonstrates we are not mere heuristic combinations of existing algorithms.

Reviewer comment (Experiments):
From a general perspective, TICC was published in 2017 and hence does not appear new; STICC was published in a geoscience journal rather than ML/DL journals/conferences and has been cited only 7 times as of now, raising concerns about the strength of the chosen baselines.

Author response:
**A4:** To the best of our knowledge, in temporal and spatial clustering, TICC and STICC are the strongest baselines for metric constrained clustering. If  you can point us to other baselines we are happy to do more experimental comparisons.