Paper: An Instance-Level Framework for Multi-tasking Graph Self-Supervised Learning (ICLR-2024)

Reviewer comment (Experiments):
The main concern is the experiments in this paper are only related to the classification task in a multi-tasking setting, while the proposed method is task-irrelevant. Although the some vision tasks are involved in the appendix, the graph-related tasks, such as node clustering and link prediction, should be investigated.

Author response:
**A1**: We have added performance comparisons of MGSSL, AutoSSL, and ParetoGNN on various graph-related downstream tasks (graph regression, node clustering, link prediction) in **Tables A2/A3/A4** and **Lines 642-654** in **Appendix A.10**. These experiments demonstrate the applicability of MGSSL to various downstream tasks and its superiority over AutoSSL and ParetoGNN.

Reviewer comment (Reproducibility):
How do you perform vision task in Appendix 10? Please provide more details for the graph construction. Maybe you can discuss the performance difference between different methods to build the graph.

Author response:
**A4**: To adapt MGSSL to vision tasks, we conduct graph construction by taking images as nodes and connecting the k-Nearest Neighbors (kNN) of each image to build edges. As you suggested, we have also provided the results of constructing the graph by thresholding in **Table A5** of **Appendix A.10**, where two images with cosine similarity greater than 0.7 will be connected by an edge. The results show that constructing the graph by kNN outperforms thresholding, and we speculate that this is because kNN guarantees the balance of node degrees in the constructed graph and prevents the over-squeezing problem that is common in graph learning. We have added the above explanation to **Lines 659-669** of the revised manuscript.

Reviewer comment (Evaluation):
Is there more evidence for Figure 4? From Guidance 1, the integrated teacher probability is as close as possible to the true Bayesian probability. What is the specific reason for the failure of the other two schemes?

Author response:
**A5**: Among the three schemes in Figure 4, only "Adaptive (ours)" follows Guideline 1, while "Averaged" and "Weighted" are two heuristic (vanilla) schemes that have nothing to do with Guideline 1. The reasons for the failure of "Averaged" and "Weighted" are that they cannot differentiate important teachers from irrelevant ones, and may mislead the student in the presence of low-quality teachers, as the manuscript described. By contrast, "Adaptive (ours)" follows Guideline 1, which enables the integrated teacher probability to be as close as possible to the true Bayesian probability by adaptive teacher weighting. To make it clear, we have added the above explanation to **Lines 186-193** of the revised manuscript.

Reviewer comment (Evaluation):
The authors say that they integrate knowledge from multiple teacher models to the student models. I wonder what knowledge is integrated into the student models. I think the knowledge may be the common knowledge from the multiple teacher models. The authors may present more analysis.

Author response:
We follow previous work [1] in decomposing knowledge into high- and low- frequency components, which are measured by cosine similarity and KL-divergence, respectively (see Appendix D of [1] for details on how to measure high/low frequency knowledge). We provide a comparison of high- and low-frequency knowledge for the five teacher and student models in **Figure A2** and **Lines 711-721** of **Appendix A.16**, from which it can be observed that (1) low-frequency knowledge (a.k.a., common knowledge) from multiple teachers can be well learned by the student, and (2) the student model differently learn high-frequency knowledge from each teacher.

[1] Wu L, Lin H, et al. Extracting Low-/High-Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework[J]. AAAI, 2023.

Reviewer comment (Writing):
I find the paper not easy to follow. The symbols maybe complex for me to understand. As a result, I suggest the authors to simplify and well structure the used symbols.

Author response:
Thanks for your constructive suggestions. We have simplified and reorganized the symbols used in this paper and provided a list of symbols in **Table A10** of **Appendix A.17**. We hope this helps you understand and appreciate our contributions. If anything is still unclear, please do not hesitate to contact us.

Reviewer comment (Evaluation):
An evaluation of the scalability and computational costs of the proposed framework, especially in large-scale settings, would be a valuable addition.

Author response:
The discussion and analysis on the computational complexity and overhead of MGSSL has been provided in **Lines 694-709**, **Appendix A.15** of the original manuscript. Did you miss this part of the experiment? Following your suggestion, we have added the computational cost of MGSSL on a larger ogbn-products dataset (consisting of 2,449,029 nodes and 61,859,140 edges) to demonstrate its excellent scalability in **Table A9** in the revised manuscript.

Reviewer comment (Experiments):
The author only compares their work with AutoSSL in Table 1. Since AutoSSL is a work from 2021, I suggest that the author should compare their work with more recent studies. This would further enhance the quality of the paper.

Author response:
One latest work, ParetoGNN, published in ICLR'2023, has been included in the comparison in **Table 1**.  Additionally, we provide more comparisons of MGSSL with AutoSSL and ParetoGNN on various graph-related tasks in **Tables A2/A3/A4** and **Lines 642-654** in **Appendix A.10**.

Reviewer comment (Experiments):
Experiments on other downstream tasks (e.g. node clustering, link prediction, and partition prediction as in AutoSSL and ParetoGNN) are missing.

Author response:
**A1**: We have added performance comparisons of MGSSL, AutoSSL, and ParetoGNN on various graph-related downstream tasks (graph regression, node clustering, link prediction) in **Tables A2/A3/A4** and **Lines 642-654** in **Appendix A.10**. These experiments demonstrate the applicability of MGSSL to various downstream tasks and its superiority over AutoSSL and ParetoGNN.

Reviewer comment (Novelty):
It seems better to add the discussion and comparison on related works about graph knowledge distillation [abc] and recent self-supervised graph learning [def].  
a. Quantifying the Knowledge in GNNs for Reliable Distillation into MLP, ICML2023.  
b. Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework, AAAI2023.  
c. Knowledge Distillation Improves Graph Structure Augmentation for Graph Neural Networks, NeurIPS2022.  
d. Decoupled Self-supervised Learning for Graphs, NeurIPS2022.  
e. Graph Self-Supervised Learning with Accurate Discrepancy Learning, NeurIPS2022.  
f. GraphMAE: Self-Supervised Masked Graph Autoencoders, KDD2022.

Author response:
**A2**: Thanks for your valuable suggestion. We have added the discussion on graph knowledge distillation and recent graph self-supervised learning in **Lines 591-602** and **Lines 626-640** of **Appendix A.9**, and the six papers you mentioned are all included, cited, and discussed. Moreover, we have included GraphMAE as an additional baseline in **Table 2**.

Reviewer comment (Writing):
Typo: In the line2 of page6, “…this paper proposes two…” is miswritten as “…this paper propose stwo…”.

Author response:
**A3**: All typos have been corrected in the revised manuscript.