Paper: Graph Clustering with Masked AutoEncoders (ICLR-2024)

Reviewer comment (Evaluation):
More result analysis should be included in Sec. 4.4.

Author response:
Additional result analyses have been incorporated into the original text. In Section 4.4, our intention was to demonstrate the superiority of the GCMA method and the impact of integrating AE encoding, namely the GCMA-A model. Therefore, visual results for only two datasets are presented along with brief explanations. From another perspective, Section 4.4 serves as a visual representation of Table 3. It can be observed from the graphs that the separation effect between clusters is worse for GCMA-A, proving the effectiveness of integrating AE encoding. The integrated graph embedding makes full use of node information to correct the clustering results.

Reviewer comment (Experiments):
There are some missing prior works about graph clustering in 2022-2023, e.g., [1], which should be included in performance comparison.

Author response:
We did not have the methodology of the literature [1] when we started writing this paper, so we added additional experiments. The experimental dataset is cite. However, it is clear that this is a parametric class method that requires the correct number of clusters k to be entered in advance for optimal performance.

|         | R2FGC | GCMA   |
|---------|--------|--------|
| Shot    | -   | 10/10  |
| NMI     | 45.39  | 44.30  |
| ARI     | 47.07  | 46.07  |

Reviewer comment (Novelty):
It seems that your self-optimization modules is similar to Deep Embedding Clustering, which should be discussed.

Author response:
The self-optimization method we applied indeed stems from deep clustering research, as seen in references [2] and [3]. This method has become nearly a universal approach within the clustering domain for enhancing the quality of autoencoder encodings. It achieves this by compelling better representations through self-supervised training. Subsequent ablation experiments substantiate this point, and for this aspect, we conducted more in-depth experimental research, specifically investigating the impact of the number q on the performance of the self-supervised module (refer to Table 4). The results of these experiments consistently indicate that our application of the optimal number q for the self-supervised optimization module yields the best performance.

Reviewer comment (Novelty):
The novelty contribution is incremental. This paper applies graph MAE for graph clustering. However, MAE is well-known and graph MAE has been used for various graph tasks, the idea is not novel.

Author response:
Certainly, as the reviewer pointed out, graph masking methods have become increasingly prevalent. However, there hasn't been a well-performing non-parametric method for graph clustering. In earlier works like MGAE, edge masking was applied to the graph, followed by decoding through cross-correlation. Subsequent methods such as MaskGAE utilized single-edge masking and path masking. More recent approaches like GiGaMAE and GraphMAE introduced feature masking and re-masking to enhance training data for better results. However, current research predominantly focuses on the study of graph masking encoding performance, with downstream tasks mainly oriented towards link prediction and node classification. There is a lack of a dedicated deep learning model applied to clustering tasks. The only existing small-scale downstream experiment on deep clustering with graph masking is found in GiGaMAE, but this model did not incorporate specific improvements for clustering, resulting in relatively poor experimental performance. In light of these research developments, GCMA innovatively combines graph masking technology with clustering. We seamlessly integrate graph masking technology, utilizing single-edge, node, and re-masking, and incorporating AE encoding. We introduce a multi-objective decoder specifically tailored for clustering tasks. In terms of graph encoding technology, our differentiator lies in the modifications made specifically for clustering tasks. We inherit multi-objective reconstruction and the second CFSFDP decoder, achieving better clustering performance while keeping training costs low.

Reviewer comment (Writing):
The method description is not entirely clear. As a major difference from the parametric baselines, automatically determine k is a claimed advantage, but how it can be done is not clear from the current writing.

Author response:
Next, we provide a detailed explanation of the process for predicting the clustering number, k. In the main text, especially in Figure 2, we have depicted the relevant flowchart. The algorithm operates based on two fundamental assumptions. First, data points near the cluster center have lower density, and second, data points are farther away from other centers with higher density. Sample point density is defined as $\rho$, and the minimum distance between a data point and points with higher density is defined as $\delta$. Among all data points, there are points with a product $\gamma_i$ that is larger, representing points with higher density $\rho$. These points are the centers of clusters. After determining several centers, the remaining data points are assigned in order of density, belonging to the point with higher density closest to them. This way, we can automatically obtain the clustering number, k. We integrate this process into our end-to-end GCMA model, optimizing the distance between sample points and cluster centers, further enhancing the algorithm's performance.

Reviewer comment (Experiments):
Empirical evaluation is not sufficient. Only citation graphs are considered. Does the proposed method also work for other types of graphs?

Author response:
Regarding the consideration for dataset selection, our primary concern is the distinction between graph and non-graph data. This allows for better validation of our model's generalization capability. For graph data, most clustering algorithms primarily utilize referenced graph datasets, such as in the literature SDFN, DFCN, S3GC. Rarely do we see other types of graph data in related research.

Reviewer comment (Experiments):
For Table 3, are the ground truth k used as input for the parametric algorithms?

Author response:
For other baselines in Table 3, we provided the correct k value as input. Otherwise, it would be unfair to compare with other parameter-based methods. As mentioned in our Figure 1, the overall clustering performance of parameter-based methods would significantly decline.

Reviewer comment (Evaluation):
How do the non-parametric methods perform in terms of those metrics in Table 3, e.g., ACC, NMI, and ARI?

Author response:
We did not find a non-parametric method that is highly suitable for graph data. Therefore, we had to use non-parametric methods from Table 2 as baselines for performance comparison. However, we acknowledge that this might be unfair to methods not specifically designed for graph data, as they may struggle to showcase their performance on non-specialized data. Here are the additional experimental results we have supplemented. We choose the Cora dataset.

|       | DeepDPM | DED    |
|-------|---------|--------|
| ACC   | 11.23   | 4.26   |
| NMI   | 15.91   | 3.32   |
| ARI   | 8.10   | 2.13   |

Reviewer comment (Reproducibility):
In section 3.2.2, how exactly was k generated? The paper says "Thus the clustering center can be clearly separated from the rest of the data points to get the best k value". The authors may illustrate more.

Author response:
As mentioned in our response in section "W2," the algorithmic process of CFSFDP allows us to obtain points with larger $\gamma_i$ values, which we consider as cluster centers, thus determining the number of clusters, k. It is essential to highlight the improvements we introduced in our work. First, in our enhancements, we employed a Gaussian kernel to determine the truncation distance. Second, the original CFSFDP, being an independent algorithm, was integrated into the end-to-end GCMA, where we designed a unified loss function for optimization. These improvements position CFSFDP as the second decoding module within GCMA.

Reviewer comment (Novelty):
The whole method is based on GraphMAE (or a modified version). The only significant difference is “CFSFDP DECODER“, which is not new and the authors also point out the citation.

Author response:
Firstly, we want to clarify the distinctions between our approach and GraphMAE. GraphMAE primarily masks nodes and their corresponding row vectors, replacing the decoder with a GNN layer (GAT/GIN) instead of an MLP. In contrast, our method masks weights, including nodes and edges, and employs a multi-objective decoding approach to address decoding challenges.

Regarding the previously proposed CFSFDP method, it had three main issues. Firstly, the algorithm's truncation distance, dc, originally required human experience for determination. In our enhancement, we utilize a Gaussian kernel, enabling automatic determination of the truncation distance. Secondly, the original algorithm necessitated the semi-automatic input of the cluster number, k, based on observed built-in parameters. We, however, directly obtain cluster centers by calculating parameters γi and point density. Lastly, the original CFSFDP operated as an independent algorithm, but we integrated it into the end-to-end GCMA, designing a unified optimized loss function. These improvements position CFSFDP as the second decoding module in GCMA. Consequently, our method operates as a non-parametric approach, eliminating the need for pre-inputting the cluster number k. The original method fails to achieve comparable experimental results, and notably, few researchers in the field of graph clustering have employed this method.

Reviewer comment (Novelty):
The novelty is also an issue. The method involves self-supervised learning on graphs combined with a specific loss on clustering, but there is no experiment comparing SSL on graphs with the proposed method.

Author response:
Additionally, concerning your remark about not conducting comparative experiments between SSL and the proposed method, this decision stems from the fact that, as a deep clustering model, the SSL module has become a standard component, as seen in AGC and EGAE within our model. Its coercive learning approach has been validated to optimize embeddings to some extent. Therefore, our comparative experiments focused solely on parameter selection within SSL, as detailed in Table 4.

Reviewer comment (Writing):
**Writing Quality**: The writing quality of this article needs improvement. The language used in the article feels difficult to understand. Additionally, the quality of some of the figures, such as Figure 1, could be enhanced.

Author response:
Thank you very much for your evaluation of our work. We will now provide detailed responses to the raised questions.

We have made several modifications to the original text, improving the grammar and sentences to enhance the overall readability of the article. For instance, in the introduction of the first chapter, we strengthened the logical flow to make the development of graph masking technology more comprehensible. Regarding Figure 1, we redesigned the information within the image. We added a grid to the background and clarified the numerical values of each point, allowing readers to clearly observe the overall trend.

Reviewer comment (Writing):
**Method is not clearly presented**: The introduction of the proposed method in this article is quite unclear. For instance, in Section 3.2.2, the authors abruptly introduce a new concept, CFSFDP, making it challenging for readers unfamiliar with this concept to grasp the purpose of this section. It seems that the authors did not effectively emphasize the main focus of this paper. In my view, since Masked Graph Autoencoders are already established content, the author's primary contribution should lie in the clustering part. However, the authors dedicate a significant amount of content to the introduction of Graph masking and AutoEncoder, which, in my opinion, is unnecessary. While the authors provide Algorithm 1 as a summary of the entire model, this algorithm appears overly concise. Furthermore, in Equation (3), the authors propose using mutual information to calculate the loss. However, I fail to discern the connection between Equation (3) and Equation (4) with mutual information.

Author response:
Our approach is a non-parametric clustering analysis method that primarily employs a graph masking autoencoder. It is crucial to emphasize two key components of our model, GCMA: 1) the masking method and 2) the CFSFDP algorithm. First, introducing the graph masking autoencoder is necessary as it serves as the foundation for the entire GCMA framework, upon which the functionalities of other modules depend. A comprehensive framework introduction is essential for the logical coherence of the entire article. Additionally, compared to graph masking technology, the content related to CFSFDP is more straightforward. Therefore, we did not extensively introduce its original content in terms of length but relied more on the overall flowcharts in the article. We provided more details about the improvements we made to CFSFDP, specifically how to achieve clustering end-to-end and determine the graph clustering number k more accurately. Here, we supplement the content related to the CFSFDP algorithm.

The basic assumptions of CFSFDP are twofold: firstly, data points near the cluster centers have lower density, and secondly, data points are farther away from centers with higher density. Sample point density is defined as rho, and the minimum distance between a data point and a point with higher density is defined as $\delta$. Among all data points, there is a type of point with $\delta$ large $\rho$, meaning that their product $\gamma_i$ is larger. Obviously, these points are the centers of the clusters. After determining several centers, the remaining data points belong to the point with higher density nearest to them in sequence. This way, we can automatically obtain the number of clusters, k. In Algorithm 1 of the original text, we have supplemented explanations in this regard.

For the second question, we proposed a reconstruction loss algorithm based on Mutual Information (MI) to effectively learn knowledge from multiple collaborative targets. In Formula (3), we explain the computation of the reconstruction loss between two objectives. A straightforward design is to maximize the similarity between the projected representation $S_n$ and the target representation $T_n$. However, this approach learns from each target separately and fails to capture shared information between them, which may include critical common knowledge that the model should emphasize. To overcome this issue, we propose using MI to quantify shared knowledge, specifically the reconstruction process from the masked encoding $S_n$ to the decoded node representation $V_n$, presented in the general form of Formula (3). In Formula (4), we provide detailed explanations of the specific calculation method for Formula (3).

Reviewer comment (Theory):
**Rationality for the designs**: Some of the designs proposed in this paper appear to rely heavily on intuition, and certain claims lack robust supporting evidence. For example, in Section 3.1.1, it is not clear why masking both modal information at the same time is considered harmful. In Section 3.1.2, the reasoning behind why a graph autoencoder would overemphasize proximity information needs further explanation. In 3.2.1, the motivation behind adding a mask to Z_m is not well-justified. I hope the authors can provide their own analysis rather than simply following others' claims.

Author response:
First and foremost, it is crucial to clarify that the viewpoints presented in the article are derived from prior relevant research and our experimental results. The perspectives in sections 3.1.1 and 3.1.2 are based on references [1] and [2], respectively. These sections were briefly mentioned in Chapter 1 without elaboration, as they are not the focus of this section, to avoid unnecessary repetition. Therefore, in these subsections, we only provide brief mentions. A more detailed explanation is warranted, as we believe in the viewpoint presented in 3.1.1. This is because, when both node features and edges of the same node are masked, we lack direct information about this node and can only obtain its information from the hidden representations of other nodes. This poses a greater obstacle to the originally intended reconstruction decoding. Without alternative decoding methods, it would have a detrimental impact on the model's performance. To validate this point, we conducted experiments. Specifically, we changed the decoding method to reconstruct the graph structure traditionally. This approach did not consider the harmful effects of simultaneously blocking two modal information types. From the experimental results, it is evident that the clustering performance of this reconstruction method significantly deteriorated. We experimented with the Cora dataset.GCMA-r stands for using the traditional reconstruction graph structure as the decoder, and SHOT stands for the number of correct k-values predicted.

|         | GCMA-r | GCMA   |
|---------|--------|--------|
| Shot    | 7/10   | 10/10  |
| NMI     | 45.00  | 59.16  |
| ARI     | 36.00  | 55.41  |

Regarding the point mentioned in Section 3.1.2 about GAE overly emphasizing proximity information, this is because previous GAE clustering methods mostly used the link reconstruction graph method to construct loss. In each node's hidden representation, there is information about other neighboring nodes, once again emphasizing the impact of proximity information on overall reconstruction.

As for the point in Section 3.2.1 about re-masking Zm to obtain more compressed representations, this is done because a single masked encoding may not provide comprehensive information for multi-target reconstruction. As the masks are random, using different masked compression representations for the reconstruction of different targets allows our model to learn more diverse modal knowledge content. Additional experiments have been added to demonstrate a significant decrease in experimental performance if we only use a single masked embedding. GCMA-one represents the result under just one mask, and shot represents the number of times the correct k-value was predicted.

|         | GCMA-one | GCMA   |
|---------|--------|--------|
| Shot    | 8/10   | 10/10  |
| NMI     | 57.39  | 59.16  |
| ARI     | 54.10  | 55.41  |

Finally, thank you again for taking the time to review my paper and I look forward to hearing from you again.

[1] Jintang Li, et al.What’s behind the mask: Understanding masked graph modeling for graph autoencoders.

[2] Yucheng Shi, et al.Gigamae: Generalizable graph masked autoencoder via collaborative latent space reconstruction.

Reviewer comment (Novelty):
The technique novelty is limited. As they claimed in the introduction, the main contribution of this paper is the usage of graph masking autoencoder for clustering analysis. However, this technique has been well studied in self-supervised learning on graphs/pretrain models on graphs. It is not clear why the model can improve the generalization ability. Another important technique, density-based clustering algorithm, has also been well-studied in both graph and non-graph domains.

Author response:
Thank you very much for your evaluation of our work. We will now provide detailed responses to the raised questions.

Our primary contribution lies in the application of graph mask autoencoders for clustering analysis. However, we have not identified any existing non-parametric methods utilizing graph mask autoencoders for clustering analysis. Most clustering models employing self-supervised learning on graphs utilize single-class-based autoencoders, as documented in references [1] and [2]. These references respectively employ GAT and GCN as encoders to construct autoencoder-based deep clustering models. Additionally, certain methods leverage fusion encoders, combining GNN networks and DNN networks, as illustrated in reference [3]. This alignment is also evident in our Table 3, where our chosen baselines are consistent with these methods.

Presently, various masking strategies have evolved in graph mask techniques. The earliest approach, as demonstrated in MGAE, involves initially applying edge masks to the graph followed by decoding through cross-linking. Subsequent methods such as MaskGAE utilize single-edge masks and path masks. More recent models like GiGaMAE and GraphMAE introduce feature masks and heavy masks to enhance training data and achieve superior results. However, current research primarily focuses on investigating the encoding performance of graph mask techniques, with most studies concentrating on downstream tasks such as link prediction and node classification. Notably, there is a dearth of deep learning models specifically applied to clustering tasks. The available data on graph mask methods for deep clustering is limited to a small downstream experiment in GiGaMAE. Unfortunately, this model has not made specific improvements for clustering and exhibits suboptimal experimental performance. Its performance on the Cora dataset is as follows.

|         | GiGaMAE | GCMA   |
|---------|--------|--------|
| Shot    | -   | 10/10  |
| NMI     | 56.39  | 59.16  |
| ARI     |50.02  | 55.41  |

As a deep graph clustering model, we innovatively integrate graph mask technology with clustering. Our end-to-end integration of graph mask techniques, utilizing single-edge, node, and heavy masks while incorporating AE encoding, involves setting up a multi-target decoder specifically tailored for clustering tasks. In terms of graph encoding techniques, our distinctiveness lies in the modifications made for clustering tasks. We have incorporated multi-target reconstruction and a second CFSFDP decoder, achieving superior clustering performance while keeping training costs economical. For a specialized deep clustering model, our approach marks the first application of graph mask technology, resulting in more compact graph embeddings.

Regarding the density-based clustering algorithm CFSFDP, despite its earlier proposal, it faces three primary challenges. First, the original algorithm's truncation distance traditionally required manual determination based on empirical experience. Our enhancement addresses this by employing a Gaussian kernel, enabling automatic determination of the truncation distance. Second, the original algorithm necessitates the manual input of the cluster number (k) through observation of internal parameters, constituting a semi-automatic determination of k. In contrast, we directly derive cluster centers by computing parameters $\gamma_i$ and point density. Finally, the original CFSFDP is a standalone algorithm, whereas we seamlessly integrate it into the end-to-end GCMA framework and design a unified loss function for optimization. These improvements position CFSFDP as the second decoder module in GCMA. Consequently, our approach, as a non-parametric method, eliminates the need for pre-inputting the cluster number k. The original method fails to yield results as robust as our improved algorithm. It is noteworthy that in the field of graph clustering, few researchers have explored the application of this method.

[1] Chun Wang, et al.Attributed graph clustering: A deep attentional embedding approach.

[2] Hong Yuan，et al.. Embedding graph auto-encoder for graph clustering

[3] Deyu Bo, et al.. Structural deep clustering network.

Reviewer comment (Evaluation):
Some important claims are not well verified. It claims that the model has better generalization ability. They conduct experiments on noisy/incomplete datasets to verify these claims. This is confusing. It is more like robustness instead of generalization. It is not clear why the model has better interpretability.

Author response:
**Weaknesses:2**

Generalization ability reflects the algorithm's adaptability to new samples. We contend that true generalization is demonstrated when a method applied to a specific type of dataset can be used on other datasets of the same type without requiring specific modifications and achieves comparable experimental results. Robustness, as you mentioned, is manifested in the performance on noisy or incomplete data. In literature [4] and [5], relevant studies have shown that graph mask technology enhances generalization ability. In our method, the graph mask encoder is inspired by these findings and adopts a similar structure. The introduction of mask strategies enhances the model's learning capability, theoretically improving generalization ability. The specific reason lies in masking and heavy masking experimental data, which can yield more compressed representation data for the model to learn. This underscores the advantage of mask technology, namely, avoiding excessive reliance on neighborhood structural information and balancing the weights between different pieces of information. Our experiments in Table 8 indicate that the GCMA graph clustering method can also outperform other baselines on non-graph-structured data. Detailed theory and experiments affirm the interpretability of GCMA.

[4] Jintang Li, et al.What’s behind the mask: Understanding masked graph modeling for
graph autoencoders.

[5] Yucheng Shi, et al.Gigamae: Generalizable graph masked autoencoder via collaborative latent space reconstruction.

Reviewer comment (Evaluation):
Some experiments are confusing. For example, "but both NMI and ARI values are significantly decreased. This means that the interpretability and generalization performance of the results decreased." How to infer interpretability and generalization from NMI and ARI are not clear. From table 3, GCMA is better than GCMA-A in COra. However, the visualization results in Figure 5 show that GCMA-A is better. There is no explanation or description of that.

Author response:
**Weaknesses:3**

Firstly, addressing the first question, the principles governing the computation of NMI and ARI determine their ability to represent the quality of clustering, thereby indicating the strength of generalization. For commonly used clustering evaluation metrics like NMI and ARI, larger values imply better clustering performance. For instance, NMI is essentially based on the calculation of mutual information, with its components representing the probability of a sample belonging to a certain cluster. The subsequent calculations are derived from maximum likelihood estimation of probabilities. When NMI and ARI decrease, it indicates more evenly distributed allocation probabilities among clusters, making it challenging to achieve an ideal result, i.e., a decrease in clustering algorithm performance. This decrease is due to a change in dataset type, signifying a decline in generalization ability, and the interpretability achieved on the previous dataset cannot be extended to the current dataset.

Moving on to the second question, the experimental design in Table 3 aims to validate the clustering performance of GCMA. The rationale behind the ablation experiments in Table 5 is to visualize the impact on GCMA. As seen in Table 3, GCMA performs better, especially in terms of accuracy (ACC). The visualization results in Figure 5 convey a similar message. While the clusters of GCMA-A appear more separated, the absence of the fused AE encoding leads to a decline in its performance. Thus, visually more separated clusters in GCMA-A represent misclustering of some points. The descriptions in these two sections collectively demonstrate the effectiveness of AE fused encoding for GCMA.

Reviewer comment (Presentation):
The presentation is messy, especially table 4 to table 8.

Author response:
**Weaknesses:4**

Regarding expression, the constraints of small-scale ablation experiments limited our presentation methods. However, for clarity, we aligned tables of different sizes in the original text and merged Tables 4-6 into a single large table to present them neatly.

Once again, thank you for taking the time to review my paper. I look forward to your further feedback.


[4] Jintang Li, et al.What’s behind the mask: Understanding masked graph modeling for
graph autoencoders.

[5] Yucheng Shi, et al.Gigamae: Generalizable graph masked autoencoder via collaborative latent space reconstruction.