Paper: StructComp: Substituting propagation with Structural Compression in Training Graph Contrastive Learning (ICLR-2024)

Reviewer comment (Theory):
The applicability of StructComp seems to be limited. Theoretically, StructComp has to rely on the approximation of the diffusion matrix for a specific graph and GNN model, as demonstrated in Theorem 4.1. How well can StructComp approximate more complicated graphs such as heterophilous graphs, and more complicated while commonly used GNNs such as GraphSage, GIN, GAT, or even more interesting variants such as PNA?

Author response:
**Q1:Theoretically, StructComp has to rely on the approximation of the diffusion matrix for a specific graph and GNN model, as demonstrated in Theorem 4.1. How well StructComp can approximate to more complicated graphs such as heterophilous graphs, and more complicated while commonly used GNNs such as GraphSage, GIN, GAT, or even more interesting variants such as PNA?**



In order to verify the approximation quality to the diffusion matrix of StructComp, we test the performance on a deep GNN architecture called SSGC [1]. We transferred the trained parameters of StructComp to the SSGC encoder for inference. For full graph training in GCL, both the training and inference stages were performed using the SSGC encoder. Table 1 shows our experimental results, indicating that even with a deeper and more complicated encoder, StructComp still achieved outstanding performance.



Table 1: The results of GCLs with SSGC encoders over 50 random splits.

| Method                        | Cora         | Citeseer     | Pubmed       |
| ----------------------------- | ------------ | ------------ | ------------ |
| SCE                           | 81.8$\pm$0.9 | 72.0$\pm$0.9 | 78.4$\pm$2.8 |
| SCE$_{\text{StructComp}}$     | 82.0$\pm$0.8 | 71.7$\pm$0.9 | 77.8$\pm$2.9 |
| COLES                         | 81.8$\pm$0.9 | 71.3$\pm$1.1 | 74.8$\pm$3.4 |
| COLES$_{\text{StructComp}}$   | 82.0$\pm$0.8 | 71.6$\pm$1.0 | 75.6$\pm$3.0 |
| GRACE                         | 80.2$\pm$0.8 | 70.7$\pm$1.0 | 77.3$\pm$2.7 |
| GRACE$_{\text{StructComp}}$   | 81.1$\pm$0.8 | 71.0$\pm$1.0 | 78.2$\pm$1.3 |
| CCA-SSG                       | 82.1$\pm$0.9 | 71.9$\pm$0.9 | 78.2$\pm$2.8 |
| CCA-SSG$_{\text{StructComp}}$ | 82.6$\pm$0.7 | 71.7$\pm$0.9 | 79.4$\pm$2.6 |



To the best of our knowledge, current mainstream GCL models, whether single-view models such as SCE, COLES, GLEN [2], or multi-view models such as DGI, GRACE, CCA-SSG, GGD, and their variants, all use SGC or GCN encoders. These unsupervised GCL models can achieve performance surpassing supervised GNNs (such as GAT, SAGE) with simple encoders on many datasets. Few GCL models have focused on whether using different encoders can achieve better performance. Moreover, GCL training is more complicated than supervised GNNs, so introducing complex encoders may make the models more difficult to train. Considering this, we did not extensively explore how using other encoders would affect StructComp. We believe this is a good question, and we will investigate it further in future work.

On the other hand, we have not implemented StructComp on heterophilous graphs , as the chosen basic GCL methods - SCE, COLES, GRACE, and CCA-SSG - are not suitable for heterophilous graph. In our future work, we will explore applying StructComp to more complex graph data structures.

Reviewer comment (Experiments):
The applicability of StructComp seems to be limited. Empirically, what is the exact setting of StructComp for node classification? Can StructComp be applied to both transductive and inductive node classification?

Author response:
**Q2:Empirically, what is the exact setting of StructComp for node classification? Can StructComp be applied to both transductive and inductive node classification?**

Our StructComp can also be used to handle inductive node classification tasks. We provide additional experiments on inductive node classification in Table 2. Clearly, the GCL models trained with StructComp also perform exceptionally well on inductive node classification tasks.

Table 2: The results on two inductive datasets. OOM means Out of Memory on GPU.

| Method                        | Flickr Acc | Flickr Time | Flickr Mem | Reddit Acc | Reddit Time | Reddit Mem |
| ----------------------------- | ---------- | ----------- | ---------- | ---------- | ----------- | ---------- |
| SCE                           | 50.6       | 0.55        | 8427       | -          | -           | OOM        |
| SCE$_{\text{StructComp}}$     | 51.6       | 0.003       | 43         | 94.4       | 0.017       | 1068       |
| COLES                         | 50.3       | 0.83        | 9270       | -          | -           | OOM        |
| COLES$_{\text{StructComp}}$   | 50.7       | 0.003       | 48         | 94.2       | 0.024       | 1175       |
| GRACE                         | -          | -           | OOM        | -          | -           | OOM        |
| GRACE$_{\text{StructComp}}$   | 51.5       | 0.010       | 221        | 94.3       | 0.079       | 8683       |
| CCA-SSG                       | 51.6       | 0.125       | 1672       | 94.9       | 0.21        | 5157       |
| CCA-SSG$_{\text{StructComp}}$ | 51.8       | 0.007       | 99         | 95.2       | 0.56        | 457        |

Reviewer comment (Experiments):
The applicability of StructComp seems to be limited. How well can StructComp approximate different augmentations in GCL?

Author response:
**Q3: How well can StructComp approximate different augmentations in GCL?**

For multi-view GCLs, we designed a new data augmentation method, Dropmember, for our StructComp. It is important to note that Dropmember is not designed to approximate other graph data augmentation methods, but rather because previous graph data augmentation methods cannot be used under the StructComp framework. According to the analysis in DropMessage [3], common graph data augmentations can be unified under a single framework, and we have proved that our Dropmember falls within this framework as well.

Reviewer comment (Experiments):
Several claims have not been verified. The paper claims that StructComp can work for large scale graphs, while the benchmarked datasets are rather small or medium scale. Although it’s claimed that ogbn-products and arxiv are large datasets, they are indeed medium scale datasets according to OGB (https://ogb.stanford.edu/docs/nodeprop/). To support the claim, it’s expected to evaluate StructComp on large datasets such as papers100M, reddit, or OGBG-LSC datasets.

Author response:
**Q4:The paper claims that StructComp can work for large scale graphs, while the benchmarked datasets are rather small or medium scale. Although it’s claimed that ogbn-products and arxiv are large datasets, while they are indeed medium scale datasets according to OGB. To support the claim, it’s expected to evaluate StructComp in large datasets such as papers100M, reddit, or OGBG-LSC datasets.**

We have conducted extra experiments on the ogbn-papers100M dataset according to your suggestion. We use StructComp to train four representative GCL models. Here, we compressed ogbn-papers100M into a feature matrix $X_c \in R^{5000*128}$ and trained GCL using StructComp. The table also presents the results of GGD trained with ClusterGCN. Although GGD is specifically designed for training large graphs, when dealing with datasets of the scale of ogbn-papers100M, it still requires graph sampling to construct subgraphs and train GGD on a large number of subgraphs. In contrast, our StructComp only requires training a simple and small-scale MLP, resulting in significantly lower resource consumption compared to GGD+ClusterGCN.



Table 3: The accuracy, training time per epoch and memory usage on the Ogbn-papers100M dataset.

| Method                        | Acc          | Time  | Mem   |
| ----------------------------- | ------------ | ----- | ----- |
| GGD                           | 63.5$\pm$0.5 | 1.6h  | 4.3GB |
| SCE$_{\text{StructComp}}$     | 63.6$\pm$0.4 | 0.18s | 0.1GB |
| COLES$_{\text{StructComp}}$   | 63.6$\pm$0.4 | 0.16s | 0.3GB |
| GRACE$_{\text{StructComp}}$   | 64.0$\pm$0.3 | 0.44s | 0.9GB |
| CCA-SSG$_{\text{StructComp}}$ | 63.5$\pm$0.2 | 0.18s | 0.1GB |

Reviewer comment (Experiments):
Several claims have not been verified. The paper also claims that StructComp has better robustness and stability with the additional regularization, while no evidence has been provided.

Author response:
**Q5:The paper also claims that StructComp has better robustness and stability with the additional regularization, while no evidence’s been found.**

We have conducted extra experiments to study the robustness of StructComp. We randomly add 10\% of noisy edges into three datasets and perform the node classification task. On the original dataset, the models trained with StructComp showed performance improvements of **0.36**, **0.40**, **1.30** and **1.87**, respectively, compared to the models trained with full graphs. With noisy perturbation, the models trained with StructComp showed performance improvements of **0.80**, **1.27**, **2.47**, and **1.87**, respectively, compared to full graph training. This indicates that GCL models trained with StructComp exhibit better robustness.



Table 4: The results over 50 random splits  on the perturbed datasets.

| Method                        | Cora         | CiteSeer     | PubMed       |
| ----------------------------- | ------------ | ------------ | ------------ |
| SCE                           | 78.8$\pm$1.2 | 69.7$\pm$1.0 | 73.4$\pm$2.2 |
| SCE$_{\text{StructComp}}$     | 79.3$\pm$0.9 | 69.3$\pm$0.9 | 75.7$\pm$2.8 |
| COLES                         | 78.7$\pm$1.2 | 68.0$\pm$1.0 | 66.5$\pm$1.8 |
| COLES$_{\text{StructComp}}$   | 79.0$\pm$1.0 | 68.3$\pm$0.9 | 69.7$\pm$2.6 |
| GRACE                         | 77.6$\pm$1.1 | 64.1$\pm$1.4 | 64.5$\pm$1.7 |
| GRACE$_{\text{StructComp}}$   | 78.3$\pm$0.8 | 69.1$\pm$0.9 | 66.2$\pm$2.4 |
| CCA-SSG                       | 75.5$\pm$1.3 | 69.1$\pm$1.2 | 73.5$\pm$2.2 |
| CCA-SSG$_{\text{StructComp}}$ | 78.2$\pm$0.7 | 69.2$\pm$0.8 | 76.3$\pm$2.5 |

Reviewer comment (Novelty):
Some related works have not been compared or discussed. Some GCL works have not been discussed in the paper, for example, [1,2,3].

Author response:
**Q6:Some GCL works have not been discussed in the paper, for example, [1,2,3].**

It should be noted that the goal of these studies and our work are different. The aim of SPGCL is to handle homophilic graphs and heterophilic graphs simultaneously. BlockGCL attempts to explore the application of deep GNN encoder in the GCL field. Contrast-Reg is a novel regularization method which is motivated by the analysis of expected calibration error. On the other hand, StructComp is a framework designed to scale up the training of GCL models: it aims to efficiently train common GCL models without performance drop. It is not a new GCL model that aims to achieve  SOTA performance compared to existing GCL models. So our work is orthogonal to these three previous works. In fact, StructComp can be used as the training method for SP-GCL, BlockGCL and Contrast-Reg. In future work, we will further investigate how to train these recent graph contrastive learning methods using StructComp.



In terms of scalability, which is the main goal of our work, we have conducted extra experiments to compare SP-GCL, BlockGCL, Contrast-Reg and StructComp-trained baseline. The results confirm that the aforementioned three GCL models are not designed for reducing training costs. 



Table 5: The results of StructComp-trained GCLs and some GCL baselines over 50 random splits. For SP-GCL, we are unable to get the classification accuracy on CiteSeer since it does not take isolated nodes as input.

| Method                                                       | Cora Acc     | Cora Time | Cora Mem | CiteSeer Acc | CiteSeer Time | CiteSeer Mem | PubMed Acc   | PubMed Time | PubMed Mem |
| ------------------------------------------------------------ | ------------ | --------- | -------- | ------------ | ------------- | ------------ | ------------ | ----------- | ---------- |
| BlockGCL                                                     | 78.1$\pm$2.0 | 0.026     | 180      | 64.5$\pm$2.0 | 0.023         | 329          | 74.7$\pm$3.1 | 0.037       | 986        |
| SP-GCL                                                       | 81.4$\pm$1.2 | 0.016     | 247      | -            | 0.021         | 319          | 74.8$\pm$3.2 | 0.041       | 1420       |
| Contrast-Reg                                                 | 79.2$\pm$1.3 | 0.048     | 355      | 69.8$\pm$1.6 | 0.097         | 602          | 72.4$\pm$3.5 | 0.334       | 11655      |
| SCE$_{\text{StructComp}}$                                    | 81.6$\pm$0.9 | 0.002     | 23       | 71.5$\pm$1.0 | 0.002         | 59           | 77.2$\pm$2.9 | 0.003       | 54         |
| COLES$_{\text{StructComp}}$                                  | 81.8$\pm$0.8 | 0.002     | 24       | 71.6$\pm$0.9 | 0.003         | 60           | 75.3$\pm$3.1 | 0.003       | 61         |
| GRACE$_{\text{StructComp}}$                                  | 79.7$\pm$0.9 | 0.009     | 37       | 70.5$\pm$1.0 | 0.009         | 72           | 77.2$\pm$1.4 | 0.009       | 194        |
| CCA-SSG$_{\text{StructComp}}$ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp; | 82.3$\pm$0.8 | 0.006     | 38       | 71.6$\pm$0.9 | 0.005         | 71           | 78.3$\pm$2.5 | 0.006       | 85         |

Reviewer comment (Experiments):
Some related works have not been compared or discussed. Why not compare efficient GCL baselines such as CCA-SSG and GGD discussed in the paper?

Author response:
**Q7:Why not comparing efficient GCL baselines such as CCA-SSG, and GGD discussed in the paper?**

*We have compared the performance of CCA-SSG with CCA-SSG trained using StructComp in the experimental section of the original submission. The specific results can be found in table 1 and table 2 of section 6.* Clearly, the performance of CCA-SSG trained with StructComp far surpasses the original CCA-SSG. Moreover, we add a comparison between StructComp and GGD in the new submission, the specific results are shown in the table below. The performance and resource consumption of various GCL models trained with StructComp are superior to GGD.

Table 6: The results of StructComp-trained GCLs and GGD.

| Method                        | Cora Acc     | Cora Time | Cora Mem | CiteSeer Acc | CiteSeer Time | CiteSeer Mem | PubMed Acc   | PubMed Time | PubMed Mem |
| ----------------------------- | ------------ | --------- | -------- | ------------ | ------------- | ------------ | ------------ | ----------- | ---------- |
| GGD                           | 79.9$\pm$1.7 | 0.013     | 118      | 71.3$\pm$0.7 | 0.018         | 281          | 74.0$\pm$2.4 | 0.015       | 311        |
| SCE$_{\text{StructComp}}$     | 81.6$\pm$0.9 | 0.002     | 23       | 71.5$\pm$1.0 | 0.002         | 59           | 77.2$\pm$2.9 | 0.003       | 54         |
| COLES$_{\text{StructComp}}$   | 81.8$\pm$0.8 | 0.002     | 24       | 71.6$\pm$0.9 | 0.003         | 60           | 75.3$\pm$3.1 | 0.003       | 61         |
| GRACE$_{\text{StructComp}}$   | 79.7$\pm$0.9 | 0.009     | 37       | 70.5$\pm$1.0 | 0.009         | 72           | 77.2$\pm$1.4 | 0.009       | 194        |
| CCA-SSG$_{\text{StructComp}}$ | 82.3$\pm$0.8 | 0.006     | 38       | 71.6$\pm$0.9 | 0.005         | 71           | 78.3$\pm$2.5 | 0.006       | 85         |

Reviewer comment (Evaluation):
How are the time and memory costs computed? Do they count in the preprocessing steps?

Author response:
**Q8:How the time and memory cost are computed? Do they count in the preprocessing steps?**

We compute the time and memory cost in the same way as previous work on scalable GNNs: the time is the training time per epoch and memory is the peak GPU memory cost during training. The preprocessing time for our StructComp is strictly less than common scalable methods (e.g., ClusterGCN, GraphSaint and GraphAutoScale), since it only performs METIS once (like ClusterGCN) and does not need to do random sampling in each epoch. METIS is highly scalable and even papers100M (with 100M nodes) can be processed within an hour on a commercial CPU. Thus for big graphs, the prepocessing time is dominated by the training time, and more importantly, it only needs to be done once and the partition result can be shared by all experiments.

Reviewer comment (Theory):
Lack of discussion of graph partition: The paper lacks a comprehensive discussion on graph partitioning. Given that the efficacy of the method hinges on graph partitioning—a classic NP-hard problem—a detailed exploration of its impact on the proposed method is warranted. A cursory introduction does not suffice.

Author response:
**Q1:Lack of discussion of graph partition.**

Our main contribution is a novel scalable framework for training GCLs and empirically show that even with off-the-shelf partition algorithms, our framework achieves remarkable speedup. We believe that no need for a specially designed partition algorithm is actually a big advantage. 

We agree that different graph partition algorithms will have an impact on the performance of StructComp. We have conducted extra experiments on graph partition. In Table 1, we demonstrate the effects of three algorithms, algebraic JC, variation neighborhoods, and affinity GS, on the performance of StructComp. These three graph coarsening algorithms are widely used in scalable GNNs, from which we can obtain the specific graph partition matrix P. The experimental results suggest that different graph partition methods has little impact on StructComp on these datasets. 

Table 1: The results of different graph partition methods.

| Method                            | Cora         | CiteSeer     | PubMed       |
| --------------------------------- | ------------ | ------------ | ------------ |
| VN+SCE$_{\text{StructComp}}$      | 81.3$\pm$0.8 | 71.5$\pm$1.0 | 77.5$\pm$2.7 |
| JC+SCE$_{\text{StructComp}}$      | 81.2$\pm$0.9 | 71.5$\pm$1.1 | 77.3$\pm$2.7 |
| GS+SCE$_{\text{StructComp}}$      | 81.5$\pm$0.8 | 71.4$\pm$1.0 | 77.4$\pm$3.0 |
| METIS+SCE$_{\text{StructComp}}$   | 81.6$\pm$0.9 | 71.5$\pm$1.0 | 77.2$\pm$2.9 |
| VN+COLES$_{\text{StructComp}}$    | 81.4$\pm$0.9 | 71.6$\pm$0.9 | 75.5$\pm$3.0 |
| JC+COLES$_{\text{StructComp}}$    | 81.4$\pm$0.9 | 71.5$\pm$1.0 | 75.3$\pm$3.0 |
| GS+COLES$_{\text{StructComp}}$    | 81.8$\pm$0.8 | 71.6$\pm$1.0 | 75.5$\pm$3.2 |
| METIS+COLES$_{\text{StructComp}}$ | 81.8$\pm$0.8 | 71.6$\pm$0.9 | 75.3$\pm$3.1 |

Graph partition is a common technique for scalable supervised GNN (e.g., ClusterGCN[1], GAS[2]), thus we have not introduced its details in our paper. To our knowledge, Metis is the mainstream graph partitioning method for scalable supervised GNNs. Previous work has not deeply investigated the impact of graph partitioning methods on scalable GNNs. We believe that this is a valuable research topic and we notice that there is an active submission at ICLR 2024 regarding this issue on scalable supervised GNNs .

Reviewer comment (Theory):
Inadequate theoretical provements: The theoretical justifications provided are somewhat limited. The authors' attempt to establish the equivalence between the compressed loss and the original loss is based solely on the ER model, which may not be representative of real-world datasets.

Author response:
**Q2:Inadequate theoretical provements: The theoretical justifications provided are somewhat limited.**

We understand your concern that the ER model may seem a strong assumption and possibly not entirely reflective of real-world scenarios. However, random graphs like the ER graph and the CSBM is widely adopted for GNN analysis(e.g., [3-8]). In our paper, we choose ER over CSBM since they have no significant difference in unsupervised GCL. We will specify the data distribution we used for analysis in the abstract and introduction. We would also like to point out that strong assumptions are common in the analysis of neural networks, e.g., infinite width in NTK [9], shallow layers [10], removing non-linearity [11], and strong data distribution assumptions [3-8]. 

According to the suggestion of the reviewer, we give an extra analysis on arbitrary graphs. For non-random graphs, the approximation gap of losses is simply bounded by the Eq 4. Suppose the loss $\mathcal{L}$ is $L$-Lipschitz continuous,
\begin{equation}
    \begin{split}
        |\mathcal{L}(P^\dagger P^TXW)-\mathcal{L}(\hat{A}^kXW)|\leq L \underbrace{\Vert P^\dagger P^T - \hat{A}^k\Vert}_\text{Eq 4.}  \Vert X\Vert  \Vert  W\Vert .
    \end{split}
\end{equation}

And for a spectral contrastive loss $\mathcal{L}'$ , assume the graph partition are even, we have:
\begin{equation}
\mathcal{L}'(P^TXW)=-\frac{2}{n}\sum_{i=1}^n e^T_{1,i}e_{2,i}+\frac{1}{n^2}\sum_{i=1}^n \sum_{j=1}^n (e^T_{1,i} e_{2,j})^2
        =-\frac{2}{n}\sum_{k=1}^{n'}\sum_{i\in S_k}e^T_{1,i}e_{2,i}+\frac{1}{n^2}\sum_{i=1}^n \sum_{l=1}^{n'} \sum_{j\in S_l} (e^T_{1,i} e_{2,j})^2
\end{equation}
\begin{equation}
        =-\frac{2}{n'}\sum_{k=1}^{n'}E^T_{1,i}E_{2,i}+\frac{1}{nn'}\sum_{i=1}^n \sum_{l=1}^{n'}  (e^T_{1,i} E_{2,j})^2
        =-\frac{2}{n'}\sum_{k=1}^{n'}E^T_{1,i}E_{2,i}+\frac{1}{nn'}\sum_{k=1}^{n'}\sum_{i\in S_k} \sum_{l=1}^{n'}  (e^T_{1,i} E_{2,j})^2
\end{equation}
\begin{equation}
        =-\frac{2}{n'}\sum_{k=1}^{n'}E^T_{1,i}E_{2,i}+\frac{1}{n'^2}\sum_{k=1}^{n'}\sum_{l=1}^{n'}  (E^T_{1,i} E_{2,j})^2=\mathcal{L}'(P^\dagger P^TXW),
\end{equation}

where $e_{1,i}$ denotes the representations of a recovered node and $E^T_{1,i}$ denotes the representations of a compressed node. The above analysis shows that our approximation is reasonable for fixed graphs.

Empirically, we want to highlight *Figure 3 in the original submission* as it greatly validates our approximation on real-world datasets. We have trained two encoders $U$ and $W$ with the compressed loss $\mathcal{L}(X_c; U)$ and the original loss $\mathcal{L}(A,X;W)$, respectively. And we plot the trends of the origianl loss on $U$ and $W$ (i.e., $\mathcal{L}(A,X;U)$ and $\mathcal{L}(A,X;W)$). The figure clearly shows that vanilla training method and StructComp yields extremely similar trends, which justifies our approximation.

Reviewer comment (Writing):
Adding more discussions about graph partition: The authors should delve deeper into the topic of graph partitioning, as highlighted in the first weakness.

Author response:
**Q1:Lack of discussion of graph partition.**

Our main contribution is a novel scalable framework for training GCLs and empirically show that even with off-the-shelf partition algorithms, our framework achieves remarkable speedup. We believe that no need for a specially designed partition algorithm is actually a big advantage. 

We agree that different graph partition algorithms will have an impact on the performance of StructComp. We have conducted extra experiments on graph partition. In Table 1, we demonstrate the effects of three algorithms, algebraic JC, variation neighborhoods, and affinity GS, on the performance of StructComp. These three graph coarsening algorithms are widely used in scalable GNNs, from which we can obtain the specific graph partition matrix P. The experimental results suggest that different graph partition methods has little impact on StructComp on these datasets. 

Table 1: The results of different graph partition methods.

| Method                            | Cora         | CiteSeer     | PubMed       |
| --------------------------------- | ------------ | ------------ | ------------ |
| VN+SCE$_{\text{StructComp}}$      | 81.3$\pm$0.8 | 71.5$\pm$1.0 | 77.5$\pm$2.7 |
| JC+SCE$_{\text{StructComp}}$      | 81.2$\pm$0.9 | 71.5$\pm$1.1 | 77.3$\pm$2.7 |
| GS+SCE$_{\text{StructComp}}$      | 81.5$\pm$0.8 | 71.4$\pm$1.0 | 77.4$\pm$3.0 |
| METIS+SCE$_{\text{StructComp}}$   | 81.6$\pm$0.9 | 71.5$\pm$1.0 | 77.2$\pm$2.9 |
| VN+COLES$_{\text{StructComp}}$    | 81.4$\pm$0.9 | 71.6$\pm$0.9 | 75.5$\pm$3.0 |
| JC+COLES$_{\text{StructComp}}$    | 81.4$\pm$0.9 | 71.5$\pm$1.0 | 75.3$\pm$3.0 |
| GS+COLES$_{\text{StructComp}}$    | 81.8$\pm$0.8 | 71.6$\pm$1.0 | 75.5$\pm$3.2 |
| METIS+COLES$_{\text{StructComp}}$ | 81.8$\pm$0.8 | 71.6$\pm$0.9 | 75.3$\pm$3.1 |

Graph partition is a common technique for scalable supervised GNN (e.g., ClusterGCN[1], GAS[2]), thus we have not introduced its details in our paper. To our knowledge, Metis is the mainstream graph partitioning method for scalable supervised GNNs. Previous work has not deeply investigated the impact of graph partitioning methods on scalable GNNs. We believe that this is a valuable research topic and we notice that there is an active submission at ICLR 2024 regarding this issue on scalable supervised GNNs .

Reviewer comment (Theory):
Considering not over-claiming your work: It's crucial to avoid overstating the contributions. While the authors assert that they have provided theoretical proof, the strong assumptions (like the ER model) limit its applicability. It might be prudent to either temper such claims in the abstract and introduction or offer more exhaustive proof. In essence, while I acknowledge the novelty and results presented in this paper, I urge the authors to provide a more in-depth rationale behind their impressive outcomes. Without this, the paper leans more toward a technical report than a comprehensive research paper.

Author response:
**Q2:Inadequate theoretical provements: The theoretical justifications provided are somewhat limited.**

We understand your concern that the ER model may seem a strong assumption and possibly not entirely reflective of real-world scenarios. However, random graphs like the ER graph and the CSBM is widely adopted for GNN analysis(e.g., [3-8]). In our paper, we choose ER over CSBM since they have no significant difference in unsupervised GCL. We will specify the data distribution we used for analysis in the abstract and introduction. We would also like to point out that strong assumptions are common in the analysis of neural networks, e.g., infinite width in NTK [9], shallow layers [10], removing non-linearity [11], and strong data distribution assumptions [3-8]. 

According to the suggestion of the reviewer, we give an extra analysis on arbitrary graphs. For non-random graphs, the approximation gap of losses is simply bounded by the Eq 4. Suppose the loss $\mathcal{L}$ is $L$-Lipschitz continuous,
\begin{equation}
    \begin{split}
        |\mathcal{L}(P^\dagger P^TXW)-\mathcal{L}(\hat{A}^kXW)|\leq L \underbrace{\Vert P^\dagger P^T - \hat{A}^k\Vert}_\text{Eq 4.}  \Vert X\Vert  \Vert  W\Vert .
    \end{split}
\end{equation}

And for a spectral contrastive loss $\mathcal{L}'$ , assume the graph partition are even, we have:
\begin{equation}
\mathcal{L}'(P^TXW)=-\frac{2}{n}\sum_{i=1}^n e^T_{1,i}e_{2,i}+\frac{1}{n^2}\sum_{i=1}^n \sum_{j=1}^n (e^T_{1,i} e_{2,j})^2
        =-\frac{2}{n}\sum_{k=1}^{n'}\sum_{i\in S_k}e^T_{1,i}e_{2,i}+\frac{1}{n^2}\sum_{i=1}^n \sum_{l=1}^{n'} \sum_{j\in S_l} (e^T_{1,i} e_{2,j})^2
\end{equation}
\begin{equation}
        =-\frac{2}{n'}\sum_{k=1}^{n'}E^T_{1,i}E_{2,i}+\frac{1}{nn'}\sum_{i=1}^n \sum_{l=1}^{n'}  (e^T_{1,i} E_{2,j})^2
        =-\frac{2}{n'}\sum_{k=1}^{n'}E^T_{1,i}E_{2,i}+\frac{1}{nn'}\sum_{k=1}^{n'}\sum_{i\in S_k} \sum_{l=1}^{n'}  (e^T_{1,i} E_{2,j})^2
\end{equation}
\begin{equation}
        =-\frac{2}{n'}\sum_{k=1}^{n'}E^T_{1,i}E_{2,i}+\frac{1}{n'^2}\sum_{k=1}^{n'}\sum_{l=1}^{n'}  (E^T_{1,i} E_{2,j})^2=\mathcal{L}'(P^\dagger P^TXW),
\end{equation}

where $e_{1,i}$ denotes the representations of a recovered node and $E^T_{1,i}$ denotes the representations of a compressed node. The above analysis shows that our approximation is reasonable for fixed graphs.

Empirically, we want to highlight *Figure 3 in the original submission* as it greatly validates our approximation on real-world datasets. We have trained two encoders $U$ and $W$ with the compressed loss $\mathcal{L}(X_c; U)$ and the original loss $\mathcal{L}(A,X;W)$, respectively. And we plot the trends of the origianl loss on $U$ and $W$ (i.e., $\mathcal{L}(A,X;U)$ and $\mathcal{L}(A,X;W)$). The figure clearly shows that vanilla training method and StructComp yields extremely similar trends, which justifies our approximation.

Reviewer comment (Theory):
The theoritical results only hold in linear-GNN, which over-simplifies the problem. It is well know that deep neural network behave different from linear model in contrastive learning [1]. Without consider non-linearity, the problem in Eq. 4 is simply matrix decomposition problem (e.g., [2] section 3).

Author response:
**Q1: How theoretical results could be generalized to non-linear models?**

Thanks for the insightful question. We explain how to extend the results to non-linear deep models below. 

Eq. (4) provides the motivation of structural compression on a linear GNN (which can also be considered as an approximation to one layer in a multi-layer non-linear GNN). The analysis can be extended to non-linear deep GNNs. For instance, given a two-layer non-linear GCN $\sigma(\\hat{A}\sigma(\\hat{A}XW_1)W_2)$, we first approximate $\hat{A}$ by $P^&dagger; P^T$, then the whole GCN can be approximated as

$$
\begin{align*}
    \sigma(P^&dagger; P^T\sigma(P^&dagger; P^TXW_1)W_2)&=\sigma(P^&dagger; P^TP^&dagger;\sigma(P^TXW_1)W_2)\\
    &=P^&dagger;\sigma(\sigma(P^TXW_1)W_2).
\end{align*}
$$

The first equality holds because $P^&dagger;$ is a partition matrix and the last equality follows from the fact that $P^TP^&dagger; = I$. Therefore, our analysis provides theoretical justifications of using StructComp as a substitute for non-linear deep GNNs. We will add this extended analysis to the revision. Note that previous studies on scalable methods for training GNNs, such as GraphSaint [1], FastGCN [2], Adapt [3], GRADE [4] also rely on various types of approximation to the propagation matrix, however, their analyses only focus on the approximation quality to a single layer or only work for linear GNNs. On the contrary, our analysis of StructComp can be easily extended to non-linear deep GNNs, which is another advantage of our framework.

Reviewer comment (Experiments):
Experiment datasets are too small (even arxiv dataset is small)... please try some large-scale graph datasets (e.g., Yelp, Reddit datasets that previously GraphSaint paper) to validate the effectiveness. Especially when this paper is focussing on improving the scalability issue.

Author response:
**Q2:Experiment datasets are too small (even arxiv dataset is small)... please try some large-scale graph datasets (e.g., Yelp, Reddit datasets that previously GraphSaint paper) to validate the effectiveness.**

The ogbn-products dataset used in our experiments has **2,449,029** nodes, which is much larger than both Yelp and Reddit you suggested. We have also conducted extra experiments on Flickr and Reddit to show the performance of StructComp under inductive setting. The experimental results are as follows. StructComp shows remarkable scalability on these datasets. 

Table 1: The results on two inductive datasets. OOM means Out of Memory on GPU.

| Method                        | Flickr Acc | Flickr Time | Flickr Mem | Reddit Acc | Reddit Time | Reddit Mem |
| ----------------------------- | ---------- | ----------- | ---------- | ---------- | ----------- | ---------- |
| SCE                           | 50.6       | 0.55        | 8427       | -          | -           | OOM        |
| SCE$_{\text{StructComp}}$     | 51.6       | 0.003       | 43         | 94.4       | 0.017       | 1068       |
| COLES                         | 50.3       | 0.83        | 9270       | -          | -           | OOM        |
| COLES$_{\text{StructComp}}$   | 50.7       | 0.003       | 48         | 94.2       | 0.024       | 1175       |
| GRACE                         | -          | -           | OOM        | -          | -           | OOM        |
| GRACE$_{\text{StructComp}}$   | 51.5       | 0.010       | 221        | 94.3       | 0.079       | 8683       |
| CCA-SSG                       | 51.6       | 0.125       | 1672       | 94.9       | 0.21        | 5157       |
| CCA-SSG$_{\text{StructComp}}$ | 51.8       | 0.007       | 99         | 95.2       | 0.56        | 457        |

According to the suggestions of other reviewers, the experiments on the ogbn-papers100M (which has **111,059,956** nodes) are conducted as well. The experimental results are shown in Table 2. Here, we compressed ogbn-papers100M into a feature matrix $X_c \in R^{5000*128}$ and trained GCL using StructComp. Table 2 also presents the results of GGD trained with ClusterGCN. Although GGD is specifically designed for training large graphs, when dealing with datasets of the scale of ogbn-papers100M, it still requires graph sampling to construct subgraphs and train GGD on a large number of subgraphs. In contrast, our StructComp only requires training a simple and small-scale MLP, resulting in significantly lower resource consumption compared to GGD+ClusterGCN.

Table 2: The accuracy, training time per epoch and memory usage on the Ogbn-papers100M dataset.

| Method                        | Acc          | Time  | Mem   |
| ----------------------------- | ------------ | ----- | ----- |
| GGD+ClusterGCN                | 63.5$\pm$0.5 | 1.6h  | 4.3GB |
| SCE$_{\text{StructComp}}$     | 63.6$\pm$0.4 | 0.18s | 0.1GB |
| COLES$_{\text{StructComp}}$   | 63.6$\pm$0.4 | 0.16s | 0.3GB |
| GRACE$_{\text{StructComp}}$   | 64.0$\pm$0.3 | 0.44s | 0.9GB |
| CCA-SSG$_{\text{StructComp}}$ | 63.5$\pm$0.2 | 0.18s | 0.1GB |

Reviewer comment (Experiments):
Repeat experiment multiple times instead of just once. For example Figure 4.

Author response:
**Q3:Repeat experiment multiple times instead of just once. For example Figure 4.**

We emphasize that all of the experiments in the original submission are results of multiple repetitions. On Cora, Citeseer, PubMed, Computers, and Photo, we repeated the experiment 50 times. On Arxiv and Products, we repeated the experiment 5 times. Figure 4 shows the average accuracy of 50 repetitions.

Reviewer comment (Experiments):
Evaluating on larger datasets like papers100M and/or OGBG-LSC datasets would better support scalability claims, since the experimented datasets are rather small or medium scale.

Author response:
**Q1:Evaluating on larger datasets like papers100M and or OGBG-LSC datasets would better support scalability claims, since the experimented datasets are rather small or medium scale.**

We have conducted extra experiments on the ogbn-papers100M dataset according to your suggestion. We use StructComp to train four representative GCL models. Here, we compressed ogbn-papers100M into a feature matrix $X_c \in R^{5000*128}$ and trained GCL using StructComp. The table also presents the results of GGD trained with ClusterGCN. Although GGD is specifically designed for training large graphs, when dealing with datasets of the scale of ogbn-papers100M, it still requires graph sampling to construct subgraphs and train GGD on a large number of subgraphs. In contrast, our StructComp only requires training a simple and small-scale MLP, resulting in significantly lower resource consumption compared to GGD+ClusterGCN.



Table 1: The accuracy, training time per epoch and memory usage on the Ogbn-papers100M dataset.

| Method                        | Acc          | Time  | Mem   |
| ----------------------------- | ------------ | ----- | ----- |
| GGD                           | 63.5$\pm$0.5 | 1.6h  | 4.3GB |
| SCE$_{\text{StructComp}}$     | 63.6$\pm$0.4 | 0.18s | 0.1GB |
| COLES$_{\text{StructComp}}$   | 63.6$\pm$0.4 | 0.16s | 0.3GB |
| GRACE$_{\text{StructComp}}$   | 64.0$\pm$0.3 | 0.44s | 0.9GB |
| CCA-SSG$_{\text{StructComp}}$ | 63.5$\pm$0.2 | 0.18s | 0.1GB |

Reviewer comment (Experiments):
It would be great to verify the model stability/robustness with the proposed regularization, since it is claimed in the presentation.

Author response:
**Q2:Would be great to verify the model stability/robustness with the proposed regularization, since it is claimed in the presentation.**

We have conducted extra experiments to study the robustness of StructComp. We randomly add 10\% of noisy edges into three datasets and perform the node classification task. On the original dataset, the models trained with StructComp showed performance improvements of **0.36**, **0.40**, **1.30** and **1.87**, respectively, compared to the models trained with full graphs. With noisy perturbation, the models trained with StructComp showed performance improvements of **0.80**, **1.27**, **2.47**, and **1.87**, respectively, compared to full graph training. This indicates that GCL models trained with StructComp exhibit better robustness.



Table 2: The results over 50 random splits  on the perturbed datasets.

| Method                        | Cora         | CiteSeer     | PubMed       |
| ----------------------------- | ------------ | ------------ | ------------ |
| SCE                           | 78.8$\pm$1.2 | 69.7$\pm$1.0 | 73.4$\pm$2.2 |
| SCE$_{\text{StructComp}}$     | 79.3$\pm$0.9 | 69.3$\pm$0.9 | 75.7$\pm$2.8 |
| COLES                         | 78.7$\pm$1.2 | 68.0$\pm$1.0 | 66.5$\pm$1.8 |
| COLES$_{\text{StructComp}}$   | 79.0$\pm$1.0 | 68.3$\pm$0.9 | 69.7$\pm$2.6 |
| GRACE                         | 77.6$\pm$1.1 | 64.1$\pm$1.4 | 64.5$\pm$1.7 |
| GRACE$_{\text{StructComp}}$   | 78.3$\pm$0.8 | 69.1$\pm$0.9 | 66.2$\pm$2.4 |
| CCA-SSG                       | 75.5$\pm$1.3 | 69.1$\pm$1.2 | 73.5$\pm$2.2 |
| CCA-SSG$_{\text{StructComp}}$ | 78.2$\pm$0.7 | 69.2$\pm$0.8 | 76.3$\pm$2.5 |

Reviewer comment (Experiments):
It would be great to discuss the approximation quality to the diffusion matrix of StructComp for more complicated graphs and other model architectures (like GAT, GraphSAGE).

Author response:
**Q3:Would be great to discuss the approximation quality to the diffusion matrix of StructComp for more complicated graphs and other models architectures (like GAT, GraphSAGE).**

In order to verify the approximation quality to the diffusion matrix of StructComp, we test the performance on a deep GNN architecture called SSGC [1]. We transferred the trained parameters of StructComp to the SSGC encoder for inference. For full graph training in GCL, both the training and inference stages were performed using the SSGC encoder. Table 3 shows our experimental results, indicating that even with a deeper and more complicated encoder, StructComp still achieved outstanding performance. To the best of our knowledge, current mainstream GCL models, whether single-view models such as SCE, COLES, GLEN [2], or multi-view models such as DGI, GRACE, CCA-SSG, GGD, and their variants, all use SGC or GCN encoders. These unsupervised GCL models can achieve performance surpassing supervised GNNs (such as GAT, SAGE) with simple encoders on many datasets. Few GCL models have focused on whether using different encoders can achieve better performance. Moreover, GCL training is more complicated than supervised GNNs, so introducing complex encoders may make the models more difficult to train. Considering this, we did not extensively explore how using other encoders would affect StructComp. We believe this is a good question, and we will investigate it further in future work.



Table 3: The results of GCLs with SSGC encoders over 50 random splits.

| Method                        | Cora         | Citeseer     | Pubmed       |
| ----------------------------- | ------------ | ------------ | ------------ |
| SCE                           | 81.8$\pm$0.9 | 72.0$\pm$0.9 | 78.4$\pm$2.8 |
| SCE$_{\text{StructComp}}$     | 82.0$\pm$0.8 | 71.7$\pm$0.9 | 77.8$\pm$2.9 |
| COLES                         | 81.8$\pm$0.9 | 71.3$\pm$1.1 | 74.8$\pm$3.4 |
| COLES$_{\text{StructComp}}$   | 82.0$\pm$0.8 | 71.6$\pm$1.0 | 75.6$\pm$3.0 |
| GRACE                         | 80.2$\pm$0.8 | 70.7$\pm$1.0 | 77.3$\pm$2.7 |
| GRACE$_{\text{StructComp}}$   | 81.1$\pm$0.8 | 71.0$\pm$1.0 | 78.2$\pm$1.3 |
| CCA-SSG                       | 82.1$\pm$0.9 | 71.9$\pm$0.9 | 78.2$\pm$2.8 |
| CCA-SSG$_{\text{StructComp}}$ | 82.6$\pm$0.7 | 71.7$\pm$0.9 | 79.4$\pm$2.6 |

Reviewer comment (Experiments):
There is a lack of comparisons with certain related works, such as recent graph contrastive learning methods [1-2].  

[1] Wang, H., Zhang, J., Zhu, Q., & Huang, W. (2022). Can Single-Pass Contrastive Learning Work for Both Homophilic and Heterophilic Graph?. arXiv preprint arXiv:2211.10890.  

[2] Li, J., Sun, W., Wu, R., Zhu, Y., Chen, L., & Zheng, Z. (2023). Scaling Up, Scaling Deep: Blockwise Graph Contrastive Learning. arXiv preprint arXiv:2306.02117.

Author response:
**Q4:There is a lack of comparisons with certain related works, such as recent graph contrastive learning methods [1-2].**

It should be noted that the goal of these studies and our work are different. The aim of SPGCL is to handle homophilic graphs and heterophilic graphs simultaneously. BlockGCL attempts to explore the application of deep GNN encoder in the GCL field. On the other hand, StructComp is a framework designed to scale up the training of GCL models: it aims to efficiently train common GCL models without performance drop. It is not a new GCL model that aims to achieve  SOTA performance compared to existing GCL models. So our work is orthogonal to these two previous works. In fact, StructComp can be used as the training method for both SP-GCL and BlockGCL. In future work, we will further investigate how to train these recent graph contrastive learning methods using StructComp.

In terms of scalability, which is the main goal of our work, we have conducted extra experiments to compare SP-GCL, BlockGCL and StructComp-trained baseline. The results confirm that the aforementioned two GCL models are not designed for reducing training costs. 



Table 4: The results of StructComp-trained GCLs and some GCL baselines over 50 random splits. For SP-GCL, we are unable to get the classification accuracy on CiteSeer since it does not take isolated nodes as input.

| Method                                                       | Cora Acc     | Cora Time | Cora Mem | CiteSeer Acc | CiteSeer Time | CiteSeer Mem | PubMed Acc   | PubMed Time | PubMed Mem |
| ------------------------------------------------------------ | ------------ | --------- | -------- | ------------ | ------------- | ------------ | ------------ | ----------- | ---------- |
| BlockGCL                                                     | 78.1$\pm$2.0 | 0.026     | 180      | 64.5$\pm$2.0 | 0.023         | 329          | 74.7$\pm$3.1 | 0.037       | 986        |
| SP-GCL                                                       | 81.4$\pm$1.2 | 0.016     | 247      | -            | 0.021         | 319          | 74.8$\pm$3.2 | 0.041       | 1420       |
| SCE$_{\text{StructComp}}$                                    | 81.6$\pm$0.9 | 0.002     | 23       | 71.5$\pm$1.0 | 0.002         | 59           | 77.2$\pm$2.9 | 0.003       | 54         |
| COLES$_{\text{StructComp}}$                                  | 81.8$\pm$0.8 | 0.002     | 24       | 71.6$\pm$0.9 | 0.003         | 60           | 75.3$\pm$3.1 | 0.003       | 61         |
| GRACE$_{\text{StructComp}}$                                  | 79.7$\pm$0.9 | 0.009     | 37       | 70.5$\pm$1.0 | 0.009         | 72           | 77.2$\pm$1.4 | 0.009       | 194        |
| CCA-SSG$_{\text{StructComp}}$ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp; | 82.3$\pm$0.8 | 0.006     | 38       | 71.6$\pm$0.9 | 0.005         | 71           | 78.3$\pm$2.5 | 0.006       | 85         |



------------------------------------------------------------

The discussion of all the aforementioned issues will be added into the revised version of our paper. We appreciate your insightful feedback once again.



[1]H. Zhu, and P. Koniusz. Simple spectral graph convolution. ICLR 2020.

[2]H. Zhu, and P. Koniusz. Generalized Laplacian Eigenmaps. Neurips 2022.