Paper: Graph Inference Acceleration by Bridging GNNs and MLPs with Self-Supervised Learning (ICLR-2024)

Reviewer comment (Theory):
In this paper, the structure-aware ability of SSL-GM is supported by the objective $\mathcal{L}_{cont}$. However, its success depends on the quality of the learned representations from the GNN, rather than solely on the capabilities of the student MLP. In other words, $\mathcal{L}{cont}$ enforces the student MLP to output representations similar to the aggregated representations from the GNN but does not inherently empower the student with structure-aware abilities. For unseen nodes during training, the student MLP may encounter challenges in learning structure-aware representations without the supervision provided by the GNN.

Author response:
Thank you for your insightful comment regarding the structure-aware capabilities of our SSL-GM model. We appreciate the opportunity to clarify how our model addresses the limitations of traditional GNN-MLP methods and its independence from GNN supervision.

**Addressing Limitations of GNN-MLP Methods.** Our primary goal with SSL-GM is to overcome the challenges faced by existing GNN-MLP models that depend heavily on knowledge distillation from teacher GNNs to student MLPs. We found that the performance of student MLPs in these models is tightly bound to the quality of pre-trained GNNs, leading to limited generalizability, especially for unseen nodes. traditional methods, SSL-GM injects high-order information into MLPs through self-supervised learning (SSL), negating the need for a well pre-trained GNN. This approach ensures that the GNN representations serve as high-order approximations of the MLP representations.

**Inherent High-Order Information in MLP Representations.** By maximizing the mutual information between MLP representations and the approximated GNN representations, we enable the MLP to inherently contain high-order structural information. This mechanism is key to ensuring the structure-aware capabilities of the student MLP independently of the GNN. To validate this capability, we conducted experiments in inductive and cold-start settings, assessing the model's effectiveness on unseen nodes. The results, as presented in Tables 2 and 3 of our paper, demonstrate the robustness of SSL-GM in these challenging scenarios.

**Inductive and Cold-Start Settings.** The inductive and cold-start experiments provide concrete evidence of SSL-GM's ability to generalize well to unseen nodes, a critical measure of its structure-aware capabilities. These settings were specifically chosen to test the model performance in scenarios where traditional GNN-MLP methods typically struggle.

Reviewer comment (Evaluation):
It's worth considering whether addressing the representation shift issue is necessary. The absence of the reconstruction strategy only leads to a slight decrease in performance (as shown in Table 1, "w/o Rec." column), indicating that this strategy may not be indispensable in some cases.

Author response:
Thank you for your observation regarding the impact of the reconstruction strategy on our SSL-GM model's performance. We appreciate this opportunity to discuss the conditions under which this strategy is particularly impactful.

We concur that the significance of the representation shift issue is closely tied to the size and complexity of the graph. In large-scale graphs with extensive and complex structures, addressing representation shift becomes increasingly important. As illustrated in our experiments, particularly on datasets like Arxiv, the absence of the reconstruction regularizer led to a noticeable performance drop (nearly 1 percentage point). This indicates that for large-scale graphs, the reconstruction strategy plays a crucial role in maintaining the integrity and quality of representations.

Our findings highlight the need for a strategic use of the reconstruction strategy in SSL-GM, applying it judiciously based on the characteristics of the graph at hand. For larger graphs, the strategy is indispensable for maintaining high-quality representations and overall performance. We plan to continue exploring the dynamics of representation shift across different graph sizes and types, further refining our understanding of when and how to effectively implement reconstruction strategies in our model.

Reviewer comment (Theory):
Theorem 1 is possibly wrong. I check the proof in the appendix, which basically connects each term in (17) to each term in (13) and leads to two problems. (1) The authors seem that minimizing each term in (17) also minimizes the corresponding term in (13). Pls prove these conclusions rigorously instead of only using intuitively explanations. (2) Even if the first point holds, (13) minimizes the summarization of 4 terms, and thus the minimizer may not be the minimizers of the 4 terms. Thus, there is no guarantee that (13) and (16) will have the same minimizer. Nowadays, many machine learning papers have theorems but my opinion is that theorems should be rigorous. Moreover, I failed to follow the mutual information part of Section 5, especially how the authors transform different models into mutual information forms. If this can be done, pls conduct the mathematical transformations in rigorous ways.

Author response:
Thank you for highlighting the need for a more rigorous explanation of the connections between the terms in equations (17) and (13). We appreciate the opportunity to delve deeper into the mathematical underpinnings of our model. First of all, we want to emphasize that we consider Eq. (13) only as an instantiation of Eq. (17). Here we will reinterpret the proof process. 

**Information Bottleneck Perspective.** We start by reinterpreting equation (17) from the perspective of the information bottleneck principle. The equation aims to jointly optimize four terms, each representing a different aspect of the information bottleneck:

$$
\mathbf{T}^* = argmin_{\mathbf{H}, \mathbf{Z}}  \lambda H(\mathbf{H} | \mathcal{G}_\mathcal{I})  + \lambda H(\mathbf{Z} | \mathbf{H}, {\mathcal{G}}_I) + H(\mathbf{Z}) + H(\mathbf{H} | \mathbf{Z})
$$

where $\mathbf{T} = (\mathbf{H}, \mathbf{Z})$, $\mathbf{H}$ and $\mathbf{Z}$ are the representations learned by MLPs and GNNs, respectively. 

**Modeling the Terms.** We model the first term by minimizing the distance between $\mathbf{H}$ and $\mathbf{F}$, represented as $\| \mathbf{H} - \mathbf{F}^* \|$. For the second term, we minimize the distance between $\mathbf{F}$ and $\mathbf{Z}$, corresponding to $\| \mathbf{Z} - \mathbf{F}^* \|$. The third term, $H(\mathbf{H} | \mathbf{Z})$, is approached by optimizing the covariance between $\mathbf{Z}$ and $\mathbf{H}$, expressed as $\sum_i \text{Cov}(\mathbf{H}_i, \mathbf{Z}_i)$. To minimize the last term, $H(\mathbf{Z})$, we make $\mathbf{Z}$ invariant to a specific variable, using $\mathbf{Z}$ to reconstruct $\mathbf{X}$. 

Combining these four terms, with some transformation, we derive our final optimization objective:
$$
\mathbb{E}_{\hat{\mathbf{A}}, \hat{\mathbf{X}}} \left[ \| \hat{\mathbf{H}} - \mathbf{F}^* \|^2 + \| \hat{\mathbf{Z}} - \mathbf{F}^* \|^2 +  \| \mathcal{D}(\hat{\mathbf{Z}}) - \mathbf{X} \|^2  | \hat{\mathbf{A}}, \hat{\mathbf{X}}  \right] - 2  {\mathbb{E}} \left[ \sum_i \text{Cov}(\hat{\mathbf{H}}_i, \hat{\mathbf{Z}}_i) |  \hat{\mathbf{A}}, \hat{\mathbf{X}}, \mathbf{F}^* \right]
$$

**Clarification on the Connection Between Eq. (13) and Eq. (17).** It is important to note that a direct, rigorous connection between the terms in equations (17) and (13) is not necessary. Our final objective serves as a practical implementation of the information bottleneck concept, tailored to the specific context of our model.

In summary, our approach to optimizing equation (17) is grounded in the principles of the information bottleneck, with each term contributing to the overarching goal of learning robust and informative representations. We believe this explanation provides a more rigorous understanding of our method's mathematical foundation.    


Thank you for your critical observation regarding the relationship between the minimizers of equations (13) and (17). We appreciate this opportunity to clarify our approach and address the need for theoretical rigor.

**Differentiation Between Eq. (13) and (17).** As discussed, equation (13) represents an implementation of the broader conceptual framework outlined in equation (17). Our approach to minimizing equation (17) through equation (13) is a practical instantiation rather than a direct equivalence. We acknowledge that the minimizer of equation (13) may not necessarily be the same as the individual minimizers of the four terms in equation (17). The composite nature of equation (13) means that its minimization involves a balance among these terms, which may not align perfectly with their individual minimization.

**Future Research Directions.** This realization forms the basis for our future research. We aim to develop more refined implementations that bring the minimization of equation (13) closer to the ideal minimization of equation (17). This will involve exploring methods with lower upper bounds and potentially tighter convergence to the theoretical optimum. We also recognize the importance of theoretical rigor in our assertions and theorems. Moving forward, we will focus on strengthening the mathematical foundations of our model, ensuring that our theoretical claims are robustly supported by rigorous analysis.


**Remark.** The current implementation, as outlined in equation (13), represents the first step in an ongoing process to optimize equation (17). We are committed to continuous improvement and refinement of our methods. Your feedback is invaluable in guiding our efforts towards more rigorous and effective model development. It helps us identify areas for theoretical and practical enhancement in our research.

Thank you for pointing out the need for a more rigorous and detailed mathematical explanation of the transformations into mutual information forms for various models. We appreciate this opportunity to delve into the specifics of these transformations.

**Transformation Process.** Based on the findings in [1], optimizing the cross-entropy $\text{CE}(\mathbf{X}, \mathbf{Y})$ between $\mathbf{X}$ and $\mathbf{Y}$ corresponds to maximizing the mutual information $\text{MI}(\mathbf{X}, \mathbf{Y}) = \sum_{i\in \mathcal{V}} I(\mathbf{y}_i, \mathbf{x}_i)$ between $\mathbf{X}$ and $\mathbf{Y}$. 

[1] Malik Boudiaf, Jerome Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In ECCV, 2020.

**Model-Specific Transformation.** The specific transformation is summarized in the following table. 

|               | Objective                                                                      |
|---------------|------------------------------------------------------------------------------------|
| MLP           | $\sum_{i \in \mathcal{V}} I(\mathbf{y}_i, \mathbf{x}_i)$                           |
| GNN           | $\sum_{i \in \mathcal{V}} I(\mathbf{y}_i; \mathcal{S}_i)$                          |
| GLNN          | $\sum_{i \in \mathcal{V}} I(\mathbf{x}_i; \mathbf{y}_i \| \mathcal{S}_i)$           |
| NOSMOG & GENN | $\sum_{i \in \mathcal{V}} I(\mathbf{x}_i; \mathbf{y}_i \| \mathcal{S}_i) + I(\mathbf{y}_i; \mathbf{A}^{[i]})$    |
| SSL-GM (ours) | $\sum_{i \in \mathcal{V}} I(\mathbf{y}_i; \mathbf{x}_i \| \mathcal{S}_i) + I(\mathbf{x}_i; \mathcal{S}_i)$ |

- **MLPs.** For MLPs optimizing the cross-entropy between node features and labels, the objective transforms directly into $\sum_{i\in \mathcal{V}} I(\mathbf{y}_i, \mathbf{x}_i)$. 

- **GNNs.** GNNs consider both structural information and node features. We define the computational graph surrounding node $i$ as $S_i = (\mathbf{X}^{[i]}, \mathbf{A}^{[i]})$, where $\mathbf{X}^{[i]}$ is the node features of neighborhoods of target node $i$ and $\mathbf{A}^{[i]}$ is the corresponding subgraph structure, transforming the GNN learning objective into $\sum_{i \in \mathcal{V}} I(\mathbf{y}_i; \mathcal{S}_i)$. 

- **GLNN.** For GLNN that uses MLPs to mimic the predictions of GNNs, we denote the outputs of GNNs as $y_i | S_i$ and thus convert the learning objective $\sum_{i \in \mathcal{V}} I(\mathbf{x}_i; \mathbf{y}_i | \mathcal{S}_i)$.

- **NOSMOG and GENN.** The following works of GLNN further add structural information in the learning objective, denoted as $\sum_{i \in \mathcal{V}} I(\mathbf{x}_i; \mathbf{y}_i | \mathcal{S}_i) + I(\mathbf{y}_i; \mathbf{A}^{[i]})$. 
        
- **SSL-GM (Ours).** Our proposed SSL-GM applies contrastive learning to maximize the mutual information between the node features and the high-order representations first and then use the learned representations to do classification, denoted as $\sum_{i \in \mathcal{V}} I(\mathbf{y}_i; \mathbf{x}_i | \mathcal{S}_i) + I(\mathbf{x}_i; \mathcal{S}_i)$. 

**Detailed Explanation.** In the context of MLPs, the focus is on maximizing the mutual information solely between the labels and node features. This approach, however, does not account for structural information inherent in the graph data, potentially limiting the depth of understanding that can be derived from the features alone. GNNs extend this concept by maximizing the mutual information between the labels and the corresponding subgraphs. This approach captures both semantic (feature-based) and structural (graph topology) information, offering a more holistic representation of the graph data. GLNNs further enhance this framework by maximizing the mutual information between node features and the soft labels generated by pre-trained GNNs. This approach leverages high-level abstract information retained in GNN outputs, enriching the feature representation with insights gleaned from the graph structure. NOSMOG and GENN incorporate an additional layer of complexity by maximizing mutual information between the subgraph of the target node and its labels. This addition emphasizes fine-grained structural information, further enriching the model understanding of the graph's intricacies.

In our SSL-GM approach, the second term of mutual information maximization is directly optimized. We posit that when this second term is maximized, it effectively transforms the first term into $\sum_{i \in \mathcal{V}} I(y_i; x_i | S_i) = \sum_{i \in \mathcal{V}} I(y_i; S_i)$. This transformation aligns our objective with that of traditional GNNs, focusing on maximizing the mutual information between labels and subgraphs, thereby capturing both semantic and structural elements of the graph.

In summary, each model adopts a unique approach to maximizing mutual information, progressively incorporating more complex layers of graph information. Our SSL-GM method synthesizes these approaches, aiming to capture a comprehensive representation of both node features and graph structure.

Reviewer comment (Novelty):
It is unclear what are the challenges of using self-supervised learning (SSL) to train MLP and what are the new designs of the paper. SSL is widely used for graph learning, and the author should be very specific in the challenges of using it to train MLP. Currently, descriptions of the weak points of existing works, e.g., “insufficient exploration of structural information when inferring unseen nodes”, “cannot fully model the structural information” are rather vague. In section 3, the authors propose several techniques and loss terms, e.g., alignment loss, data augmentation, and reconstruction loss. These are not new for SSL, which are also evidenced by the citations provided by the authors. The question is that what are the new things proposed by the authors. The paper will be stronger if the authors can connect the challenges and the proposed new techniques.

Author response:
Thank you for your insightful comment. We appreciate the opportunity to clarify the innovative aspects of our work, particularly in addressing the limitations of existing GNN-MLP methods in inductive, cold-start, and label sparsity settings.


- **Insight on Generalization Issues.** Our core contribution does not lie in new model architectures and new objectives. Instead, our work identifies the lack of generalization in current methods as a key issue, stemming from the limitations of knowledge distillation (KD) that aligns GNN and MLP outputs only at the label level. We propose that aligning these models in the representation space using self-supervised learning (SSL) can yield more generalizable representations. This insight represents a significant shift from existing practices.
        
- **Challenges in SSL Alignment.** Different from KD methods, aligning GNNs and MLPs under SSL is challenging due to the absence of a pre-trained GNN. To this end, we convert the objective to jointly train GNN and MLP by aligning the 0-hop representations (MLP output) with high-order representations (GNN output). Our approach is applicable with various objectives like Bootstrap, InfoNCE, or MSE. This flexibility in alignment strategy is a novel contribution to the field.

- **Addressing Representation Inconsistency and Model Collapse.** We observed inconsistency in representations between GNNs and MLPs, potentially leading to model collapse (Appendix D.3). Our solution involves using MLP outputs to approximate GNN outputs, a strategy akin to SGC but applied to a different problem context. 

- **Graph Augmentation and Reconstruction Regularizer.** We further incorporate data augmentation to enhance the quality of the learned representations. However, given that structural permutation in augmentation can significantly shift GNN representations, we introduce a reconstruction regularizer to mitigate representation shift. This addition is critical in ensuring the stability and reliability of the learned representations.

Reviewer comment (Experiments):
Experiments can be improved. (1) Pls provide the time for model training, which is also an important practical consideration. (2) Pls run the experiments on large practical datasets, e.g., Papers100M, MAG240M, and IGB.

Author response:
Thank you for your valuable feedback. In response to your comment, we have presented additional experimental results in Appendix C of our paper to further validate the efficacy of our SSL-GM model in inference acceleration.

- **Accuracy vs Efficiency.** We present a comprehensive comparison between our method and other acceleration methods. The results showcase the inference time and accuracy for various methods, including SGC, APPNP, Quantization (QSAGE), Pruning (PSAGE), and Neighbor Sample on Flickr and OGB-Arxiv. We have intentionally omitted GNN-MLP methods that were compared in Figure 3 for brevity. Please note that the time consumption of SAGE and BGRL are the same since the learning process of SSL does not affect inference. 

- **Different Settings.** To ensure a thorough evaluation, we chose three datasets across all node classification settings, including transductive, inductive, and cold-start. Our results, as shown in the additional tables, clearly indicate that SSL-GM outperforms existing methods in terms of efficiency and accuracy. 

[Experimental results are following]

| Datasets |  | SAGE      | BGRL  | SGC                  | APPNP                | QSAGE                | PSAGE                | Neighbor Sample      | **SSL-GM** |
|------------|------------------|-----------|-------|----------------------|----------------------|----------------------|----------------------|----------------------|----------------------------|
| Flickr     | Time (ms) | 80.7  | 80.7 (1.00$\times$)  | 76.9 (1.05$\times$)  | 78.1 (1.03$\times$)  | 70.6 (1.14$\times$)  | 67.4 (1.20$\times$)  | 25.5 (3.16$\times$)        | **0.9 (89.67$\times$)** |
|                              | Acc (\%)  | 47.17 | 49.12                | 47.35                | 47.53                | 47.22                | 47.25                | 47.01                      | **49.27**                |
| Arxiv       | Time (ms) | 314.7 | 314.7 (1.00$\times$) | 265.9 (1.18$\times$) | 284.1 (1.11$\times$) | 289.5 (1.09$\times$) | 297.5 (1.06$\times$) | 78.3 (4.02$\times$)        | **2.5 (125.88$\times$)** |
|                              | Acc (\%)  | 68.52 | 69.29                | 68.93                | 69.10                | 68.48                | 68.55                | 68.35                      | **70.23**                |



|           |        | Trans      | Trans          | Ind        | Ind            | Cold-start | Cold-start     |
|-----------|--------|------------|----------------|------------|----------------|------------|----------------|
| Dataset   | Models | Time (ms)  | Acc (\%)       | Time (ms)  | Acc (\%)       | Time (ms)  | Acc (\%)       |
| Pubmed    | SAGE   | 73         | 85.94          | 15         | 85.04          | 15         | 77.98          |
|           | SGC    | 64         | 85.28          | 14         | 85.22          | 13         | 76.10          |
|           | NOSMOG | 5          | 86.18          | 3          | 83.84          | 3          | 81.48          |
|           | SSL-GM | **3** | **86.99** | **3** | **86.47** | **3** | **86.44** |
| Amazon-CS | SAGE   | 103        | 88.88          | 31         | 87.24          | 25         | 61.01          |
|           | SGC    | 89         | **89.31** | 26         | 87.12          | 24         | 63.08          |
|           | NOSMOG | 5          | 87.64          | 4          | 86.61          | 4          | 81.95          |
|           | SSL-GM | **4** | 88.46          | **3** | **87.65** | **3** | **87.58** |
| Arxiv     | SAGE   | 485        | **72.05** | 315        | 68.52          | 305        | 43.47          |
|           | SGC    | 410        | 69.95          | 266        | 68.93          | 250        | 42.08          |
|           | NOSMOG | 6          | 70.84          | 4          | 69.10          | 4          | 61.64          |
|           | SSL-GM | **4** | 71.12          | **3** | **70.23** | **3** | **66.13** |
    

Thank you for your valuable suggestion to extend our experiments to include large-scale practical datasets like Papers100M, MAG240M, and IGB. Although these datasets are seldomly used in graph learning experiments, we appreciate your emphasis on the importance of testing our model in diverse and challenging real-world scenarios. 

We fully acknowledge the significance of evaluating our model on these large datasets. Such an expansion in our experimental scope would undoubtedly provide deeper insights into the scalability and robustness of our model. We are committed to incorporating this suggestion into our future research plans. Conducting experiments on these larger datasets will be a key focus in our subsequent work, allowing us to further validate and refine our model.

Reviewer comment (Experiments):
The experimental results only contain the performance over node classification and graph classification. Is it possible to evaluate the proposed method over link prediction?

Author response:
Thank you for your insightful question regarding the application of our model to link prediction tasks. We appreciate this opportunity to discuss the adaptability of our method to different graph analysis tasks.

**Applying the Model to Link Prediction.** Our model adopts self-supervised learning (SSL) approach that is inherently flexible and can be applied to link prediction without requiring significant modifications to the graph training process. To represent links, we suggest concatenating the corresponding node representations. This concatenated representation will then serve as the input to the logistic regression model for predicting the presence or absence of a link. For the inference phase in link prediction, we propose adopting a linear protocol where the parameters of the encoder are frozen. This means that during inference, only the logistic regression model, which is used for predicting links, will be trained. The link prediction task can be approached in a standard manner, involving the sampling of some edges from the original graph as positive examples and performing negative sampling for negative edges.

**Future Work.** While our current experimental results do not include link prediction, we acknowledge the importance of this task in graph analysis. Therefore, we are considering extending our experimental evaluation to include link prediction, providing a comprehensive view of the applicability across various graph-related tasks.

Reviewer comment (Novelty):
It seems that this paper only borrows some ideas from contrastive learning. Based on contrastive learning, this paper develops a new objective function that can be used to solve the model generalization problem. Therefore, could the authors highlight some contributions here? I think it is a good paper but the contribution of this paper is a little bit marginal. If the authors are able to emphasize their contributions here, I am willing to improve my rate.

Author response:
Thank you for your insightful comment. We appreciate the opportunity to clarify the innovative aspects of our work, particularly in addressing the limitations of existing GNN-MLP methods in inductive, cold-start, and label sparsity settings.

- **Insight on Generalization Issues.** Our core contribution does not lie in new model architectures and new objectives. Instead, our work identifies the lack of generalization in current methods as a key issue, stemming from the limitations of knowledge distillation (KD) that aligns GNN and MLP outputs only at the label level. We propose that aligning these models in the representation space using self-supervised learning (SSL) can yield more generalizable representations. This insight represents a significant shift from existing practices.
        
- **Challenges in SSL Alignment.** Different from KD methods, aligning GNNs and MLPs under SSL is challenging due to the absence of a pre-trained GNN. To this end, we convert the objective to jointly train GNN and MLP by aligning the 0-hop representations (MLP output) with high-order representations (GNN output). Our approach is applicable with various objectives like Bootstrap, InfoNCE, or MSE. This flexibility in alignment strategy is a novel contribution to the field.

- **Addressing Representation Inconsistency and Model Collapse.** We observed inconsistency in representations between GNNs and MLPs, potentially leading to model collapse (Appendix D.3). Our solution involves using MLP outputs to approximate GNN outputs, a strategy akin to SGC but applied to a different problem context. 

- **Graph Augmentation and Reconstruction Regularizer.** We further incorporate data augmentation to enhance the quality of the learned representations. However, given that structural permutation in augmentation can significantly shift GNN representations, we introduce a reconstruction regularizer to mitigate representation shift. This addition is critical in ensuring the stability and reliability of the learned representations.

Reviewer comment (Novelty):
Although a significant number of experiments were conducted, the novelty and contribution of the proposed method remain limited.

Author response:
Thank you for your insightful comment. We appreciate the opportunity to clarify the innovative aspects of our work, particularly in addressing the limitations of existing GNN-MLP methods in inductive, cold-start, and label sparsity settings.

- **Insight on Generalization Issues.** Our core contribution does not lie in new model architectures and new objectives. Instead, our work identifies the lack of generalization in current methods as a key issue, stemming from the limitations of knowledge distillation (KD) that aligns GNN and MLP outputs only at the label level. We propose that aligning these models in the representation space using self-supervised learning (SSL) can yield more generalizable representations. This insight represents a significant shift from existing practices.
        
- **Challenges in SSL Alignment.** Different from KD methods, aligning GNNs and MLPs under SSL is challenging due to the absence of a pre-trained GNN. To this end, we convert the objective to jointly train GNN and MLP by aligning the 0-hop representations (MLP output) with high-order representations (GNN output). Our approach is applicable with various objectives like Bootstrap, InfoNCE, or MSE. This flexibility in alignment strategy is a novel contribution to the field.

- **Addressing Representation Inconsistency and Model Collapse.** We observed inconsistency in representations between GNNs and MLPs, potentially leading to model collapse (Appendix D.3). Our solution involves using MLP outputs to approximate GNN outputs, a strategy akin to SGC but applied to a different problem context. 

- **Graph Augmentation and Reconstruction Regularizer.** We further incorporate data augmentation to enhance the quality of the learned representations. However, given that structural permutation in augmentation can significantly shift GNN representations, we introduce a reconstruction regularizer to mitigate representation shift. This addition is critical in ensuring the stability and reliability of the learned representations.

Reviewer comment (Theory):
I cannot understand Formula 4. The author injects randomness by perturbing the structure and features of the original graph, with the expectation that the MLP encoder can capture invariant key features. However, Formula 4 is perplexing. The first part of the formula computes the mutual information between G_1 and G_2, but is it merely a distinction of whether the perturbed structure is included? Why does this part enable the encoder to obtain high-quality representations? It seems more like an optimization of the random augmentation methods q_e and q_f, but the author did not clarify whether they are learnable.

Author response:
We apologize for any confusion caused by Equation 4 and appreciate the opportunity to clarify its meaning and significance in our paper.

**Recap.** The augmentation process is aligned with the objective of optimal contrastive learning, which aims to train an augmentation-invariant encoder. This is encapsulated in Equation 4, where $\mathcal{E}^* = argmin_{\mathcal{E}} I(\mathcal{G}_1, \mathcal{G}_2) - I(\hat{\mathbf{H}}, \hat{\mathbf{Z}})$. In this equation, $\mathcal{G}_1 = (\emptyset, \hat{\mathbf{X}})$ and $\mathcal{G}_2 = (\hat{\mathbf{A}}, \hat{\mathbf{X}})$ represent two augmented views of the graph - $\mathcal{G}_1$ as the augmented node-set without edges and $\mathcal{G}_2$ as the augmented graph. $\hat{\mathbf{H}}$ and $\hat{\mathbf{Z}}$ are the representations encoded by MLPs and GNNs on these augmented graphs, respectively.

**Detailed Explanation:**
    
**Random Augmentation.** The random augmentation is applied in each epoch to create $\mathcal{G}_1$ and $\mathcal{G}_2$. The first part of the equation computes the mutual information between these two augmented views.

**Learning Objective.** The equation aims to identify an encoder that can capture all information in the augmented views, regardless of the specific augmentations applied. While achieving this ideal is challenging, the goal is for the encoder to learn the most essential information common to both views, a principle known as mutual information maximization.

**Significance of Perturbation.** The perturbation in the structure and features of the original graph (as represented in $\mathcal{G}_1$ and $\mathcal{G}_2$) is crucial for this process. It ensures that the encoder focuses on invariant key features across different augmentations, enhancing the robustness and generalizability of the learned representations.

We hope this detailed explanation clarifies the purpose and mechanism of Equation 4 in our methodology. Our aim is to ensure the encoder captures essential, invariant features from augmented graph views, reinforcing the model generalization.

Thank you for your question seeking further clarification on the quality of representations obtained from our encoder. We are happy to provide more insight into this aspect.

As outlined in our discussion on Equation 4, our encoder is designed to capture the most essential information present in the graph. This is achieved through the process of mutual information maximization between the augmented graph views. We posit that this 'essential information', which remains consistent across various augmentations, is inherently discriminative. It represents the fundamental characteristics of the graph that are crucial for tasks such as node classification or link prediction. Therefore, by training the encoder to focus on such invariant features, we enhance the discriminative power of the learned representations, which can enhance the quality of representations. 
    
We appreciate your inquiry and the opportunity to clarify the role of the augmentation methods $q_e$ and $q_f$ in our model. As we have discussed, our primary objective is to train an encoder that is invariant to the augmentations applied. This means the encoder should consistently capture the essential features of the graph regardless of the specific augmentations. In line with this objective, we have designed $q_e$ and $q_f$ to be non-trainable. We believe that making these augmentation methods trainable could potentially lead to overfitting to specific augmentation patterns, thereby defeating the purpose of achieving augmentation invariance.

Reviewer comment (Experiments):
Given the method proposed in the paper, I believe it would be interesting to include GNN methods like SGC and APPNP in the experiments concerning acceleration effectiveness.

Author response:
Thank you for your valuable feedback. In response to your comment, we have presented additional experimental results in Appendix C of our paper to further validate the efficacy of our SSL-GM model in inference acceleration.

- **Accuracy vs Efficiency.** We present a comprehensive comparison between our method and other acceleration methods. The results showcase the inference time and accuracy for various methods, including SGC, APPNP, Quantization (QSAGE), Pruning (PSAGE), and Neighbor Sample on Flickr and OGB-Arxiv. We have intentionally omitted GNN-MLP methods that were compared in Figure 3 for brevity. Please note that the time consumption of SAGE and BGRL are the same since the learning process of SSL does not affect inference. 

- **Different Settings.** To ensure a thorough evaluation, we chose three datasets across all node classification settings, including transductive, inductive, and cold-start. Our results, as shown in the additional tables, clearly indicate that SSL-GM outperforms existing methods in terms of efficiency and accuracy.

| Datasets |  | SAGE      | BGRL  | SGC                  | APPNP                | QSAGE                | PSAGE                | Neighbor Sample      | **SSL-GM** |
|------------|------------------|-----------|-------|----------------------|----------------------|----------------------|----------------------|----------------------|----------------------------|
| Flickr     | Time (ms) | 80.7  | 80.7 (1.00$\times$)  | 76.9 (1.05$\times$)  | 78.1 (1.03$\times$)  | 70.6 (1.14$\times$)  | 67.4 (1.20$\times$)  | 25.5 (3.16$\times$)        | **0.9 (89.67$\times$)** |
|                              | Acc (\%)  | 47.17 | 49.12                | 47.35                | 47.53                | 47.22                | 47.25                | 47.01                      | **49.27**                |
| Arxiv       | Time (ms) | 314.7 | 314.7 (1.00$\times$) | 265.9 (1.18$\times$) | 284.1 (1.11$\times$) | 289.5 (1.09$\times$) | 297.5 (1.06$\times$) | 78.3 (4.02$\times$)        | **2.5 (125.88$\times$)** |
|                              | Acc (\%)  | 68.52 | 69.29                | 68.93                | 69.10                | 68.48                | 68.55                | 68.35                      | **70.23**                |



|           |        | Trans      | Trans          | Ind        | Ind            | Cold-start | Cold-start     |
|-----------|--------|------------|----------------|------------|----------------|------------|----------------|
| Dataset   | Models | Time (ms)  | Acc (\%)       | Time (ms)  | Acc (\%)       | Time (ms)  | Acc (\%)       |
| Pubmed    | SAGE   | 73         | 85.94          | 15         | 85.04          | 15         | 77.98          |
|           | SGC    | 64         | 85.28          | 14         | 85.22          | 13         | 76.10          |
|           | NOSMOG | 5          | 86.18          | 3          | 83.84          | 3          | 81.48          |
|           | SSL-GM | **3** | **86.99** | **3** | **86.47** | **3** | **86.44** |
| Amazon-CS | SAGE   | 103        | 88.88          | 31         | 87.24          | 25         | 61.01          |
|           | SGC    | 89         | **89.31** | 26         | 87.12          | 24         | 63.08          |
|           | NOSMOG | 5          | 87.64          | 4          | 86.61          | 4          | 81.95          |
|           | SSL-GM | **4** | 88.46          | **3** | **87.65** | **3** | **87.58** |
| Arxiv     | SAGE   | 485        | **72.05** | 315        | 68.52          | 305        | 43.47          |
|           | SGC    | 410        | 69.95          | 266        | 68.93          | 250        | 42.08          |
|           | NOSMOG | 6          | 70.84          | 4          | 69.10          | 4          | 61.64          |
|           | SSL-GM | **4** | 71.12          | **3** | **70.23** | **3** | **66.13** |

Reviewer comment (Experiments):
In section 4.3, there is only 'Figure 3' to demonstrate the capability of SSL-GM for inference acceleration. However, it is necessary to provide detailed experimental results regarding accuracy and inference time. These results should be obtained from a wider range of datasets and classification settings.

Author response:
Thank you for your valuable feedback. In response to your comment, we have presented additional experimental results in Appendix C of our paper to further validate the efficacy of our SSL-GM model in inference acceleration.

- **Accuracy vs Efficiency.** We present a comprehensive comparison between our method and other acceleration methods. The results showcase the inference time and accuracy for various methods, including SGC, APPNP, Quantization (QSAGE), Pruning (PSAGE), and Neighbor Sample on Flickr and OGB-Arxiv. We have intentionally omitted GNN-MLP methods that were compared in Figure 3 for brevity. Please note that the time consumption of SAGE and BGRL are the same since the learning process of SSL does not affect inference. 

- **Different Settings.** To ensure a thorough evaluation, we chose three datasets across all node classification settings, including transductive, inductive, and cold-start. Our results, as shown in the additional tables, clearly indicate that SSL-GM outperforms existing methods in terms of efficiency and accuracy. 

[We list the experimental results in the next part. ]


| Datasets |  | SAGE      | BGRL  | SGC                  | APPNP                | QSAGE                | PSAGE                | Neighbor Sample      | **SSL-GM** |
|------------|------------------|------------|-------------|----------------------|----------------------|----------------------|----------------------|----------------------|----------------------------|
| Flickr     | Time (ms) | 80.7  | 80.7 (1.00$\times$)  | 76.9 (1.05$\times$)  | 78.1 (1.03$\times$)  | 70.6 (1.14$\times$)  | 67.4 (1.20$\times$)  | 25.5 (3.16$\times$)        | **0.9 (89.67$\times$)** |
|                              | Acc (\%)  | 47.17 | 49.12                | 47.35                | 47.53                | 47.22                | 47.25                | 47.01                      | **49.27**                |
| Arxiv       | Time (ms) | 314.7 | 314.7 (1.00$\times$) | 265.9 (1.18$\times$) | 284.1 (1.11$\times$) | 289.5 (1.09$\times$) | 297.5 (1.06$\times$) | 78.3 (4.02$\times$)        | **2.5 (125.88$\times$)** |
|                              | Acc (\%)  | 68.52 | 69.29                | 68.93                | 69.10                | 68.48                | 68.55                | 68.35                      | **70.23**                |



|           |        | Trans      | Trans          | Ind        | Ind            | Cold-start | Cold-start     |
|-----------|--------|------------|----------------|------------|----------------|------------|----------------|
| Dataset   | Models | Time (ms)  | Acc (\%)       | Time (ms)  | Acc (\%)       | Time (ms)  | Acc (\%)       |
| Pubmed    | SAGE   | 73         | 85.94          | 15         | 85.04          | 15         | 77.98          |
|           | SGC    | 64         | 85.28          | 14         | 85.22          | 13         | 76.10          |
|           | NOSMOG | 5          | 86.18          | 3          | 83.84          | 3          | 81.48          |
|           | SSL-GM | **3** | **86.99** | **3** | **86.47** | **3** | **86.44** |
| Amazon-CS | SAGE   | 103        | 88.88          | 31         | 87.24          | 25         | 61.01          |
|           | SGC    | 89         | **89.31** | 26         | 87.12          | 24         | 63.08          |
|           | NOSMOG | 5          | 87.64          | 4          | 86.61          | 4          | 81.95          |
|           | SSL-GM | **4** | 88.46          | **3** | **87.65** | **3** | **87.58** |
| Arxiv     | SAGE   | 485        | **72.05** | 315        | 68.52          | 305        | 43.47          |
|           | SGC    | 410        | 69.95          | 266        | 68.93          | 250        | 42.08          |
|           | NOSMOG | 6          | 70.84          | 4          | 69.10          | 4          | 61.64          |
|           | SSL-GM | **4** | 71.12          | **3** | **70.23** | **3** | **66.13** |

Reviewer comment (Writing):
In section 3.3, it would be better to introduce representation shift in detail and explain how reconstruction helps in mitigating representation shift.

Author response:
Thank you for pointing out the need for a more detailed discussion on representation shift and its mitigation. We have revised Section 3.3 to address this.

**Defining Representation Shift.** We define representation shift as the alteration in learned representations due to data augmentations. While feature augmentation does not significantly alter the distribution of learned representations, structural augmentation can have a profound impact. This shift, particularly pronounced in structural augmentation, can significantly alter the local structure of the target node. For instance, as depicted in Fig. 2, simple edge permutation can dramatically change the 2-hop neighborhoods of a target node, leading to a substantial shift in the representations.
        
**Mitigation Strategy - Reconstruction Regularizer.** To counter this problem, we hypothesize that if GNN representations can preserve more localized information, the impact of representation shift can be minimized. Based on this, we introduce a reconstruction regularizer. This regularizer aims to reconstruct the raw node features based on the GNN representations, thereby mitigating the representation shift. The underlying idea is to anchor the learned representations closer to the original feature space, reducing the impact of structural changes.
        
**Effectiveness of Our Approach.** The reconstruction regularizer thus serves as a stabilizing factor, ensuring that the GNN representations remain informative and reliable despite augmentations. This approach is innovative in how it leverages the reconstruction concept to maintain the integrity of learned representations in the face of potential shifts.

Reviewer comment (Novelty):
Although this work is a combination of existing methods, it is fascinating to introduce this model from a higher level rather than loss function level.

Author response:
Thank you for your insightful suggestion to present our SSL-GM model from a more holistic perspective. We appreciate this opportunity to highlight the integrative approach and overarching objectives of our model, beyond the specifics of the loss function.

**Holistic Approach.** In developing SSL-GM, our focus extends beyond the choice of loss functions to a comprehensive consideration of the model architecture and its ability to generalize effectively. This broader view is essential for understanding the innovative aspects of our approach. A key component of our model is the use of contrastive learning between GNNs and MLPs. This strategy is designed to infuse MLPs with generalizable high-order representations, enhancing their capability to capture complex graph structures and features.

**Strategic Components.** To avoid model collapse, a common challenge when directly minimizing the distance between GNNs and MLPs, we introduce non-parametric aggregation. This component ensures consistency and stability in the relationship between GNNs and MLPs. Further strengthening our model, we apply data augmentation techniques and leverage a reconstruction term. These additions are geared towards improving the model capability and preventing potential representation shifts. Importantly, the choices of augmentations and loss functions are flexible, allowing for adaptability to different graph scenarios.

**Emphasizing Generalization and Adaptability.** The central theme of SSL-GM is to enhance the model generalization capabilities, enabling it to perform effectively across a variety of graph-related tasks and settings. The flexible nature of key components, including the choice of loss functions and augmentations, reflects our commitment to creating a versatile and adaptable model that can address diverse challenges in graph model acceleration.

Reviewer comment (Writing):
In B.3, learning rates is misspelled where ‘5e4’ appears twice.

Author response:
Thank you for bringing this typographical error to our attention. We sincerely apologize for the oversight and appreciate your meticulous reading of our manuscript. We have reviewed Section B.3 and corrected the error where the learning rate '5e4' is incorrectly mentioned twice.