Paper: Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks (ICLR-2024)

Reviewer comment (Writing):
The motivation of focusing on graph-structured data is unclear.

Author response:
The reason for your confusion may be because we have not added too many graph-specific priors to our method. This is due to the diversity of downstream tasks in the graph domain, where different downstream tasks contain different instance levels, e.g., graph classification and node classification. This leads to the fact that imposing too many restrictions on a certain method can make it difficult to handle diverse downstream tasks, since the tasks employed in the method already provide domain-specific a priori (e.g., node masking, edge prediction, etc.). For example, the previous work AutoSSL\[1], used graph homophily as a priori knowledge, which caused it to only be able to handle node-level tasks and not graph-level tasks. We note that this series of subsequent work (ParetoGNN\[2], AGSSL\[3]) follows the idea of not imposing a priori on methods. Our starting point is common, i.e., imposing too many a priori on combinatorial methods can limit the scenarios in which the methods can be used.

As with previous tasks, the reason we are a framework positioned on graph-structured data is that the phenomena we observe (sharp performance degradation due to conflicts among graph pre-training tasks, etc.) are graph-based. In addition, our method does show potential for extension to other domains, but we do not believe that this should be one of the drawbacks, and we will consider extensions of our method as meaningful follow-on work.



\[1]Wei Jin, Xiaorui Liu, Xiangyu Zhao, Yao Ma, Neil Shah and Jiliang Tang, Automated Self-Supervised Learning for Graphs, ICLR2022

\[2] Mingxuan Ju, Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye and Chuxu Zhang, Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization, ICLR 2023

\[3] Lirong Wu, Yufei Huang, Haitao Lin, Zicheng Liu, Tianyu Fan and Stan Z. Li. Automated graph self-supervised learning via multi-teacher knowledge distillation. arxiv2210.02099

Reviewer comment (Evaluation):
Compared to the baseline methods for multi-task learning on instance-level, the improvement from WAS is marginal. On the largest graphs in the experiment (ogbn-arxiv), it achieves worse performance compared to baseline method.

Author response:
We kindly point out that "the improvement from WAS is marginal" might be somewhat inappropriate.

Firstly, the major contribution of our method lies in its ability to **continually achieve performance gains from an expanding task pool**, a capability not possessed by previous methods. Besides this primary contribution, our method outperforms previous multi-task learning methods in terms of performance on seven node-level datasets, even when the task pool is kept the same. Additionally, our method is capable of handling **graph-level tasks**.

Reviewer comment (Theory):
Besides the empirical results, the theoretical analysis is still needed to answer why both selecting and weighing is needed, and why there is compatibility issue (i.e., why some tasks shouldn't be selected).

Author response:
We sincerely appreciate your suggestions for enhancing the clarity of our manuscript, which will make it stronger!  We have provided relevant analysis in Fig.1 of the Introduction, where Fig.1b illustrates the importance of a reasonable weighting scheme, and Fig.1c demonstrates the compatibility issues between different tasks. You may want us to include more detailed explanations regarding these issues. 

We will address your concerns in three steps: 1. Are different tasks relatively independent? 2. Why do compatibility issues arise? 3. Theoretically speaking, what are the advantages of incorporating a selection mechanism as opposed to simply using a weighting approach? For the figures and proofs mentioned in the following content, please refer to the Appendix G.

- **[Are tasks independent? Different teachers acquired different knowledge during pre-training, and this knowledge remains different after migration to downstream tasks]** In Fig. A2, we analyze the knowledge acquired by different teachers. Different pre-training methods, e.g., GraphCL performs contrastive learning between instances, while AM masks node attributes on a single instance, are able to acquire different knowledge. Fig. A2 visualizes the representations obtained by different teachers after training on the same pre-training dataset by T-sne. It can be seen that they are distributed in different spaces, which means that **they do not acquire the same knowledge.** We can conclude that **the knowledge they acquire is essentially independent**. 
- **[Why do compatibility issues arise? Different instances may require different knowledge.]** Then, for downstream tasks, what differences do different teachers' acquired knowledge make? **Fig. A3** visualizes the graphs from the test set of BACE, where blue dots represent molecular graphs that can be correctly predicted for all seven teachers, while red dots represent graphs that can only be correctly predicted by some of the seven teachers. The red dots make up the majority, which means that **different teachers will be able to predict different graphs.** Therefore, for a given instance, if  a teacher whose knowledge is not suitable for them is chosen, this may cause conflicts and could be worse than not having that teacher at all. This is the compatibility issue.
- **[What are the advantages of incorporating a selecting mechanism? Having only the weighting module can lead to an inability to improve performance.]** In the Appendix G, we provide a theoretical analysis of the performance without the selecting module (only the weighting module). Please refer to the Appendix G. Through this analysis, we can conclude that the selection module is a crucial factor for the model to achieve continuous performance improvement from an ever-expanding task pool.

More detailed content (figures and theoretical analysis) has been added to the **Appendix G**. Please refer to **Appendix G** for further information.

---

In light of these responses, we hope we have addressed your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please share and we will attend to these points.

Reviewer comment (Novelty):
The proposed method does not treat graph-structured data specially, nor is it tailored for graph modeling. Why is it positioned for pre-training on graphs, instead of for general pre-training?

Author response:
The reason for your confusion may be because we have not added too many graph-specific priors to our method. This is due to the diversity of downstream tasks in the graph domain, where different downstream tasks contain different instance levels, e.g., graph classification and node classification. This leads to the fact that imposing too many restrictions on a certain method can make it difficult to handle diverse downstream tasks, since the tasks employed in the method already provide domain-specific a priori (e.g., node masking, edge prediction, etc.). For example, the previous work AutoSSL\[1], used graph homophily as a priori knowledge, which caused it to only be able to handle node-level tasks and not graph-level tasks. We note that this series of subsequent work (ParetoGNN\[2], AGSSL\[3]) follows the idea of not imposing a priori on methods. Our starting point is common, i.e., imposing too many a priori on combinatorial methods can limit the scenarios in which the methods can be used.

As with previous tasks, the reason we are a framework positioned on graph-structured data is that the phenomena we observe (sharp performance degradation due to conflicts among graph pre-training tasks, etc.) are graph-based. In addition, our method does show potential for extension to other domains, but we do not believe that this should be one of the drawbacks, and we will consider extensions of our method as meaningful follow-on work.



\[1]Wei Jin, Xiaorui Liu, Xiangyu Zhao, Yao Ma, Neil Shah and Jiliang Tang, Automated Self-Supervised Learning for Graphs, ICLR2022

\[2] Mingxuan Ju, Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye and Chuxu Zhang, Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization, ICLR 2023

\[3] Lirong Wu, Yufei Huang, Haitao Lin, Zicheng Liu, Tianyu Fan and Stan Z. Li. Automated graph self-supervised learning via multi-teacher knowledge distillation. arxiv2210.02099

Reviewer comment (Evaluation):
Any in-depth understanding of the relationship for the selected tasks? Are they independent?

Author response:
We sincerely appreciate your suggestions for enhancing the clarity of our manuscript, which will make it stronger!  We have provided relevant analysis in Fig.1 of the Introduction, where Fig.1b illustrates the importance of a reasonable weighting scheme, and Fig.1c demonstrates the compatibility issues between different tasks. You may want us to include more detailed explanations regarding these issues. 

We will address your concerns in three steps: 1. Are different tasks relatively independent? 2. Why do compatibility issues arise? 3. Theoretically speaking, what are the advantages of incorporating a selection mechanism as opposed to simply using a weighting approach? For the figures and proofs mentioned in the following content, please refer to the Appendix G.

- **[Are tasks independent? Different teachers acquired different knowledge during pre-training, and this knowledge remains different after migration to downstream tasks]** In Fig. A2, we analyze the knowledge acquired by different teachers. Different pre-training methods, e.g., GraphCL performs contrastive learning between instances, while AM masks node attributes on a single instance, are able to acquire different knowledge. Fig. A2 visualizes the representations obtained by different teachers after training on the same pre-training dataset by T-sne. It can be seen that they are distributed in different spaces, which means that **they do not acquire the same knowledge.** We can conclude that **the knowledge they acquire is essentially independent**. 
- **[Why do compatibility issues arise? Different instances may require different knowledge.]** Then, for downstream tasks, what differences do different teachers' acquired knowledge make? **Fig. A3** visualizes the graphs from the test set of BACE, where blue dots represent molecular graphs that can be correctly predicted for all seven teachers, while red dots represent graphs that can only be correctly predicted by some of the seven teachers. The red dots make up the majority, which means that **different teachers will be able to predict different graphs.** Therefore, for a given instance, if  a teacher whose knowledge is not suitable for them is chosen, this may cause conflicts and could be worse than not having that teacher at all. This is the compatibility issue.
- **[What are the advantages of incorporating a selecting mechanism? Having only the weighting module can lead to an inability to improve performance.]** In the Appendix G, we provide a theoretical analysis of the performance without the selecting module (only the weighting module). Please refer to the Appendix G. Through this analysis, we can conclude that the selection module is a crucial factor for the model to achieve continuous performance improvement from an ever-expanding task pool.

More detailed content (figures and theoretical analysis) has been added to the **Appendix G**. Please refer to **Appendix G** for further information.

---

In light of these responses, we hope we have addressed your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please share and we will attend to these points.

Reviewer comment (Evaluation):
The complexity of the proposed has not been studied. Although the proposed method can achieve some performance improvement, it may take more training/inference time, due to combining multiple pre-training tasks. The authors need to have some analysis about the complexity of the proposed method.

Author response:
We have provided an analysis of additional parameters, extra storage space, additional time consumption, and extra time complexity in **Appendix C (complexity analysis)**, and mentioned this at the end of Sec. 4.2.3 (Line 267).

Reviewer comment (Experiments):
The authors also need to perform experiments to analyse how many tasks are selected for each instance on average.

Author response:
We sincerely appreciate your suggestions for enhancing the clarity of our manuscript, which will make it stronger! In the following Table, we count the average number of selected teachers *number*, and then count the *ratio of selecting Top-1/Top-[number] important teacher*.

From the table, we can draw the following conclusions:

1. teachers with the highest weights have a higher probability of being selected
2. the selection module does not necessarily always select the higher weight teachers

|                                                     | Cora | Citeseer | BACE | BBBP |
| :-------------------------------------------------- | :--- | :------- | :--- | :--- |
| *number*                                            | 2.7  | 3.9      | 2.9  | 3.1  |
| *ratio of selecting Top-1 important teacher*        | 0.63 | 0.71     | 0.68 | 0.55 |
| *ratio of selecting Top-[number] important teacher* | 0.26 | 0.34     | 0.41 | 0.10 |

This table is added to the updated **Appendix F.**

Reviewer comment (Experiments):
On average, how many tasks are usually selected for each instance?

Author response:
We sincerely appreciate your suggestions for enhancing the clarity of our manuscript, which will make it stronger! In the following Table, we count the average number of selected teachers *number*, and then count the *ratio of selecting Top-1/Top-[number] important teacher*.

From the table, we can draw the following conclusions:

1. teachers with the highest weights have a higher probability of being selected
2. the selection module does not necessarily always select the higher weight teachers

|                                                     | Cora | Citeseer | BACE | BBBP |
| :-------------------------------------------------- | :--- | :------- | :--- | :--- |
| *number*                                            | 2.7  | 3.9      | 2.9  | 3.1  |
| *ratio of selecting Top-1 important teacher*        | 0.63 | 0.71     | 0.68 | 0.55 |
| *ratio of selecting Top-[number] important teacher* | 0.26 | 0.34     | 0.41 | 0.10 |

This table is added to the updated **Appendix F.**

Reviewer comment (Experiments):
From Figure 6(d), we can observe that the following 4 tasks have the largest weights, i.e, EP, AM, IG, and CP. How about we only use these 4 tasks and learning the their weights for prediction?

Author response:
Figure 6(d) represents only the result of one instance. The tasks with higher weights in this instance may not necessarily have high weights in other instances. 

Using only these four tasks could potentially lead to a decline in performance. In the table below, "7-tasks" indicates that the task pool contains all 7 tasks, while "4-tasks" means the task pool includes only four tasks (EP, AM, IG, and CP).

|         | 7-tasks      | 4-tasks      | EP           | AM           | IG           | CP           |
| :------ | :----------- | :----------- | ------------ | ------------ | ------------ | ------------ |
| Toxcast | 63.7$\pm$1.5 | 62.8$\pm$1.1 | 59.0$\pm$0.9 | 62.6$\pm$0.1 | 60.7$\pm$0.9 | 61.7$\pm$0.7 |

With the reduction of tasks, there was a decrease in performance, but it still remained better than using a single task.

Reviewer comment (Theory):
The concept of instance is not well defined. Different pre-training tasks may have different definitions of instances fundamentally (eg. at a node-level, subgraph/graph-level, edge-level etc). Having a common and converged definition may not work well. Furthermore, downstream task could also have different definitions of instances.

Author response:
We kindly point out that your concerns are the same as those mentioned in our manuscript.

> " The concept of instance is not well defined. "

We provided an example of instances on page 1,Line 36-37 ("e.g., nodes in a social network or graphs in a molecular dataset").

> " Different pre-training tasks may have different definitions of instances fundamentally "

 We completely agree with your statement and we discussed this in Line 191-193 (Original text: "For example, GraphCL performs contrastive learning between instances, while AM masks node attributes on a single instance"). For a graph-level task, GraphCL defines an instance as a graph, whereas for AM, the instance is still a node.

It might be that our expression was not clear enough. We will modify it to "For example, GraphCL performs contrastive learning between graphs, while AM masks node attributes on a single graph".

> "downstream task could also have different definitions of instances."

We also fully agree with this. Therefore, we conducted experiments at both the node-level and graph-level to demonstrate the effectiveness of our method.

Reviewer comment (Novelty):
While 4 questions are clear in the identification of the 4 steps/issues to address, the solutions are quite standard. E.g. using Gumbel-Softmax sampling for Bernoulli distribution, the weighting scheme and the Siamese network architecture are all well known tools.

Author response:
We sincerely thank you for appreciating our work "4 questions are clear in the identification of the 4 steps/issues to address". 

The Gumbel-Softmax, weighing scheme, and the Siamese architecture are seen as a tool rather than a contribution.

Our contributions are:

1. examining the shortcomings of various existing multi-task graph learning methods (they mainly based on weighing-only method), i.e., not considering the compatibility issue.
2. distinguishing between the compatibility issue and the importance issue, and using decoupling frameworks to deal with these two issues.

Considering that we are already achieving what the framework was designed for (1. good performance 2. the ability to sustain performance growth) by using those classic components, we kindly point out that we think that there is no need to force modifications to the classic components just to appear that we are different.

Reviewer comment (Novelty):
The compatibility issue, or interferences among tasks have been observed in previous work in other areas or problem settings [a,b,c]. Some discussion on this aspect, and its particular challenges in graph context, would further strengthen the motivation of the paper. 

[a] Zeyuan Wang, Qiang Zhang, HU Shuang-Wei, Haoran Yu, Xurui Jin, Zhichen Gong, and Huajun Chen. 2022. Multi-level Protein Structure Pre-training via Prompt Learning. In The Eleventh International Conference on Learning Representations.  
[b] Sen Wu, Hongyang R Zhang, and Christopher Ré. 2020. Understanding and improving information transfer in multi-task learning. arXiv preprint arXiv:2005.00944 (2020).  
[c] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems 33 (2020), 5824–5836.

Author response:
We sincerely appreciate your suggestions for enhancing the motivation of our manuscript, which will make it stronger! Among the works you mentioned\[a]\[b]\[c], \[a] is specifically designed for transformers. However, the dominant architecture in the graph domain is still GNN, which limits its applicability. 

\[b] differs in focus from our method. We focus on extracting and combining knowledge from various pre-trained tasks to enhance the performance of downstream tasks, while \[b] is concerned with handling different downstream tasks when using a single model. \[c] combines tasks through gradient surgery, a method similar to AutoSSL and ParetoGNN mentioned in our paper. These approaches combine task losses through various weighting methods. Firstly, they are unable to provide a customized combination for each instance. Secondly, they overlook compatibility issues, making it difficult to achieve continual performance improvement as the task pool expands.  Moreover, these works do not address the concept of "compatibility". Their solutions differ from ours as their main focus remains on various weighting methods, overlooking the necessity of selection.

We have updated Sec.2 to include these more discussions.

---

[a] Zeyuan Wang, Qiang Zhang, HU Shuang-Wei, Haoran Yu, Xurui Jin, Zhichen Gong, and Huajun Chen. 2022. Multi-level Protein Structure Pre-training via Prompt Learning. In The Eleventh International Conference on Learning Representations. 

 [b] Sen Wu, Hongyang R Zhang, and Christopher Ré. 2020. Understanding and improving information transfer in multi-task learning. arXiv preprint arXiv:2005.00944 (2020).

 [c] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems 33 (2020), 5824–5836.

---

In light of these responses, we hope we have addressed your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please share and we will attend to these points.

Reviewer comment (Experiments):
The comparison model in A5 needs more description to distinguish it from A1, such as whether Random-Select and ALL use learned weights.

Author response:
We sincerely appreciate your suggestions for enhancing the readability and clarity of our manuscript, which will make it stronger! 

Both Random-Select and ALL use learned weights, but Random-Select randomly select teachers, and ALL select all the teachers. We add corresponding explanations in the updated manuscript to better clarify this point.