Paper: An Enhanced Gromov-Wasserstein Barycenter Method for Graph-based Clustering (ICLR-2024)

Reviewer comment (Theory):
Some parts in Section 3.3 are not clear and should be clarified: (i) srGW to Identity: ‘This results in each cluster containing an equal number of data points.’ It clearly does not seem to be the case. (ii) Relation to NMF: does it really coincide with the srGW diagonal barycenter problem or rather with the complete barycenter problem? *[Addressed by authors]*

Author response:
5.1 The objective function of srGW is equivalent to a standard Min-Cut function augmented by an exclusive lasso regularization term with a coefficient of $\frac{1}{2}$. The minimum value of this regularization term is achieved when the sizes of all classes are equal. You can find the proof in our Appendix B.1  for details.

5.2 According to our proof in Appendix B.1, page 21, for any square matrix, Weighted Symmetric NMF is equivalent to the complete Monge-type GW barycenter problem, which means the feasible set is constrained to be mappings represented by rectangular permutation matrices.

Reviewer comment (Writing):
Challenges in Readability: In certain instances, the meaning of the notation, although not formally introduced, can be grasped from the context (e.g., $mathbb{I}_K$ denoting the identity matrix with $K$ rows). However, there are situations where the notation becomes confusing, posing a challenge to the paper's readability. For example, in the discussion of the "Monge's type Gromov-Wasserstein barycenter" in Section 3, the optimal mapping matrix is denoted as $MGW(G, G′)$, but this notation is also used as the objective in the minimization equation (7). The concept of minimizing a matrix raises confusion. In addition, the symbols $\pi$ and $\Gamma$ are interchangeably used to refer to the same object. For instance, three lines before equation (7), it states $\nu=\pi^T 1_N$, but in this context, as far as I did not misunderstand it, we are assuming a hard clustering mapping and therefore $\Gamma$ should be the appropriate symbol. Furthermore, though it is not crucial, adding the labels to the x-axis and y-axis of the plots would also ease the readability of the figures.

Author response:
We appreciate your insightful observations and constructive criticism.
1. Thank the reviewer for point out those typos, and in response, these have been incorporated in the revised paper. Specifically, (i) MGW represents the Monge-type GW distance induced by the optimal mapping $\Gamma$. It also depends on the target graph $G^{'}$, which is composed of a matrix $D^{'}$ and node distribution $\nu$. Our objective is to find the optimal graph $G^{'}(D^{'},\nu)$. (ii) Figure 3  illustrates the AMI results of SpecGWL, srGW, and EGWB across various heat kernel times (t) for the WIKI dataset.  This figure serves as an illustrative representation of our motivation for delving into graph-based hard clustering.

Reviewer comment (Reproducibility):
The proposed method initialization depends on the results of other methods such as GWL and SpecGWL.

Author response:
We appreciate your insightful observations and constructive criticism.
2. In fact, we thoroughly elaborate on this experiment in the appendix. It is crucial to recognize that this challenge poses a non-convex optimization problem. Consequently, selecting a  well-chosen initial value undoubtedly improves the optimality of the final convergence point to a certain extent.  
    Both GWL and SpecGWL provide a transport plan $ \pi $ in the joint distribution space. Using the combinations of these plans as initial points is a natural idea. Moreover, if a uniform distribution is directly employed as the initial value, theoretically, it is equivalent to SpecGWL in the early stages of the algorithm.

Reviewer comment (Experiments):
Initialization dependency: The paper mentions using a linear combination of GWL, SpecGWL, and joint distribution results as initial values for EGWB. However, it's unclear how the algorithm relies on this initialization. Could a less informed start, like the uniform distribution, yield comparable results or does one need to start from a relatively good initialization?

Author response:
We appreciate your insightful observations and constructive criticism.
2. In fact, we thoroughly elaborate on this experiment in the appendix. It is crucial to recognize that this challenge poses a non-convex optimization problem. Consequently, selecting a  well-chosen initial value undoubtedly improves the optimality of the final convergence point to a certain extent.  
    Both GWL and SpecGWL provide a transport plan $ \pi $ in the joint distribution space. Using the combinations of these plans as initial points is a natural idea. Moreover, if a uniform distribution is directly employed as the initial value, theoretically, it is equivalent to SpecGWL in the early stages of the algorithm.

Reviewer comment (Evaluation):
Computational cost: While the paper outlines the computational cost per iteration, the average number of iterations required for convergence remains undisclosed. Additionally, considering the initialization dependency, it's crucial to know the overall time needed to run EGWB, especially if solutions for GWL and SpecGWL must be computed beforehand.

Author response:
We appreciate your insightful observations and constructive criticism.
3. The theoretical prediction of average numbers of iterations is still open for such non-convex optimization problems. However, in experiments, we observe that during the inner optimization process for $\pi$, approximately three hundred iterations are required to reduce the relative error, denoted as $ \|\frac{\pi^{t+1} - \pi^{t}}{\pi^{t}}\|_F $, from $ O(1)  $ to $ O(1 \times 10^{-4}) $.

Reviewer comment (Evaluation):
Synthetic data: Figure 4 exhibits superior results for GWL and SpecGWL compared to Figure 2. Moreover, in Figure 4 GWL out performs SpecGWL given the true cluster size distribution. Is there any reason why is that the case? I am actually surprised that, for these apparently simple problems GWL and SpecGWL fail to retrieve the right clustering. Understanding the specific reasons for their failure is beneficial in order to comprehend why EGWB, in contrast, succeeds.

Author response:
We appreciate your insightful observations and constructive criticism.
4. Indeed, in our experiments, SpecGWL achieved a smaller GW distance compared to GWL. The crucial point is that both methods employ a fixed $G^{'}(D',\nu)$ and then learn the transport map, which makes the clustering results less convincing. In contrast, our EGWB treats it as a variable for optimization, introducing more degrees of freedom that are determined by the data, naturally yielding superior results.

Reviewer comment (Reproducibility):
How to initialize $D’(0)$? How to set the value of $K$?

Author response:
The  $D^{'}(0)$ can be readily derived from equation (10) of our paper, provided one thoroughly peruses our document. Anyway, we can explain it more in detail here: 
    When we have an initial guess $\pi(0)$, we employ $$\frac{\pi^{\top}_{:,j}(0)D\pi_{:,j}(0)}{\pi^{\top}_{:,j}(0)\pi_{:,j}(0)}$$ as the diagonal of  $D^{'}(0)$.  The selection of $K$ follows standard procedures as outlined in any clustering textbook. Consequently, in our paper, we treat $K$ as a predetermined parameter, ensuring a fair comparison across all methods.

Reviewer comment (Writing):
It seems the authors confuse the task of graph partitioning and that of graph clustering. They muddle up partitioning and clustering throughout this paper. I think the experiments in section 4.2 are more likely to be a graph partitioning task, rather than graph clustering, as is claimed by the authors. Please use one of the two definitions consistently in the paper.

Author response:
The main focus of our paper is the clustering based on graph structure. This is clearly stated in section 3.1, where our aim is to solve the hard clustering problems via graph partitioning. The point of doing clustering via graph partitioning has already been made in the literature prior to our paper, see [a][b].

Reviewer comment (Evaluation):
In the subsection of Results and Discussion, the authors say they employ five metrics, however, I only find AMI. If they take the results reported in appendix into account, then should clarify this in the main context.

Author response:
Once more, it appears that the reviewer has not meticulously perused our paper. We have meticulously presented comprehensive numerical results in the appendices, encompassing indices such as ARI, V-measure, and FMI. Nevertheless, to further address this concern, we will augment the main text with a clarifying sentence.

Reviewer comment (Presentation):
What do the two axes in Fig.3 represent? It should be labeled in the figure.

Author response:
Figure 3  illustrates the AMI results of SpecGWL, srGW, and EGWB across various heat kernel times (t) for the WIKI dataset.  This figure serves as an illustrative representation of our motivation for delving into graph-based hard clustering.   We still thank the reviewer for pointing out this ambiguity, and in response, corrected labels have been incorporated in the revised paper.

Reviewer comment (Writing):
There are typos and careless statements and the authors need to polish this paper carefully. For example, 1) page 1, it should be $G_1(D_1,P_1)$ in the third row from the bottom, 2) page 2, the second paragraph, the second point of the limitations has grammatical mistakes, 3) What is EGWB an abbreviation for? The author put forward EGWB without any explanations in page 6.

Author response:
Thank the reviewer for point out those typos. They have been corrected.