From: ICLR.cc/2023/Conference/Paper754/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper754/Authors
title: Response to Reviewer A89U
comment: Thank you for your time and effort in reviewing our work, and the valuable comments. We have responded to other reviewers' comments in detail, including the experimental settings and results. We hope to address your concerns. Looking forward to furthering discussion with you.

From: ICLR.cc/2023/Conference/Paper754/Authors
title: Response to Reviewer aLXm
comment: Thank you for your time and effort in reviewing our work, and the valuable comments. Point-to-point responses can be found below to each reviewer. We are also glad to improve our work to address any further concerns continually. Looking forward to furthering discussion with you.

**Q1: Datasets**

The implementation details of the three datasets satisfying the multi-task condition can be found in the supplementary material of [1] (see Algorithm 2, https://arxiv.org/abs/2012.06755). To avoid ambiguity, we give a detailed experimental setting.

In each run, we split the graphs into 70%, 10%, and 20% for training $B_{tra}$, validation $B_{val}$, and test $B_{test}$. For the node classification, we mimic the common semi-supervised setting, where feature vectors are available for all nodes, and only a small subset of nodes (30% in this paper) is labeled [2]. For the link prediction, in each graph, some randomly removed edges and unremoved edges are treated as positive examples. Couples of non-adjacent nodes are randomly sampled as negative examples. The steps are as follows:

for each graphs $g$ in datasets $B$ ($B_{tra}$, $B_{val}$, or $B_{test}$) do:

//Node Classification

num_labelled_nodes $\leftarrow$ $g$.num_nodes*30%;

$N$ $\leftarrow$ divide nodes per class, then iteratively randomly sample one node per class without replacement and add it to $N$ until $|N|$ = num_labelled_nodes;

$g$.labelled_nodes $\leftarrow$ $N$;

// Link Prediction

$E^N$ $\leftarrow$ randomly pick negative samples (edges that are not in the graph; possibly in the same number as the number of edges in the graph);

$E^P$ $\leftarrow$ randomly remove 20% of the edges in $g$;

 $g'$ $\leftarrow$ $g$ removed of $E^P$;

$g$.positive edges $\leftarrow$  $g'$.edges;

$g$.negative edges $\leftarrow$ $E^P$.

We will upload the revised version in the future (the manuscript cannot be edited in this phase). In addition, our tasks contain **both graph-level (graph classification) and node-level (node classification and link prediction) tasks**. Three standard datasets containing multiple graphs (ENZYMES, PROTEINS, and DHFR) are adopted. Therefore, our experimental setting is reasonable and rigorous. Our experiments fully verify the effectiveness of the proposed bi-level multi-objective optimization algorithm for automatic multi-task graph representation learning.

[1] Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach, IJCNN, 2022.

[2] Semi-Supervised Classification with Graph Convolutional Networks, ICLR, 2017.

**Q2: Standard Node Classification and Link Prediction Datasets**

Our tasks contain **both graph-level and node-level tasks**. Standard node classification and link prediction datasets, such as Cora, CiteSeer, and PubMed, often contain a single graph. Therefore, they do not satisfy the multi-task condition for graph-level tasks. In order to further dispel the reviewers' concerns, we added a set of comparative experiments for solving multiple node-level tasks (node classification NC, link prediction LP, and attribute inference AI) to illustrate the effectiveness of our method. Three widely used datasets (Cora, CiteSeer, and PubMed) are adopted. The comparison algorithms include a single-level multi-objective (multi-task handcrafted) baseline (JAME)[1]. In each run, we split the nodes into 60%, 20%, and 20% for training, validation, and testing. The other experimental settings are exactly the same as in [1]. The source code of JAME can be found in the original paper. The experimental results in the table below can still show the superiority of our method **when solving multiple node-level tasks**.

|   |  Cora-NC |  Cora-LP |  Cora-AI |  CiteSeer-NC |  CiteSeer-LP |  CiteSeer-AI |  PubMed-NC |  PubMed-LP |  PubMed-AI |
| :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
|JAME | 87.4  |98.8 | 96.4  |  78.8 | 98.1 | **97.3**  |  89.6 | 98.3 | 67.8|
|DMTGAS-EPO | **91.6** | **99.8** | **97.6**  |  **79.7** | **99.9** | 94.2  |  **91.1** | **98.9** | **89.8**|

[1] Multi-task Network Embedding with Adaptive Loss Weighting, ASONAM, 2020.

**Q3: Minor side point**

We will upload the revised version in the future (the manuscript cannot be edited in this phase).

From: ICLR.cc/2023/Conference/Paper754/Reviewer_A89U
title: Followup
comment: I carefully read author's response and the reviews/comments from other reviewers.

First, thanks of author's effort for proving more explanations of the model and conducting a few extra experiments.

Second, I agree with other reviewers (especially reviewer aLXm) that there are still some unsolved issues including the experimental settings and results aspects. More specifically, the response still not clearly, logically answered questions raised by reviewers (e.g., the questions raised by reviewer L5LN).

To this end, I hold negative opinion of this submission.

From: ICLR.cc/2023/Conference/Paper754/Reviewer_aLXm
title: followup
comment: Thanks for the detailed response! 

Regarding my previous question on the datasets, I have checked the newly added citation [1]. Nonetheless, even [1] did not clearly state how the node labels are generated and how the datasets were splitted. Hence, the experimental setting is questionable and not rigorous enough to convince readers on the effectiveness of the proposed method. I would suggest the authors to also consider standard node classification and link prediction datasets, and also explore other tasks (given this work is focusing on "multi-task") on those datasets. Hence, I think this paper still needs substantial update on the experiments, and I would keep my recommendation.

Minor side point: I would recommend the authors to use `\citep{}` instead of `\citet{}` for most inline citations.

[1] Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach, IJCNN, 2022.

From: ICLR.cc/2023/Conference/Paper754/Authors
title: We hope we clarified all the reviewer's concerns 
comment: We hope that our response was able to clarify all the concerns raised by the reviewer. We kindly ask the reviewer to let us know if they have some additional questions that they would like to be addressed.

We look forward to engaging in a fruitful discussion!

From: ICLR.cc/2023/Conference/Paper754/Authors
title: We hope we clarified all the reviewer's concerns 
comment: We hope that our response was able to clarify all the concerns raised by the reviewer. We kindly ask the reviewer to let us know if they have some additional questions that they would like to be addressed.

We look forward to engaging in a fruitful discussion!

From: ICLR.cc/2023/Conference/Paper754/Authors
title: We hope we clarified all the reviewer's concerns 
comment: We hope that our response was able to clarify all the concerns raised by the reviewer. We kindly ask the reviewer to let us know if they have some additional questions that they would like to be addressed.

We look forward to engaging in a fruitful discussion!

From: ICLR.cc/2023/Conference/Paper754/Authors
title: We hope we clarified all the reviewer's concerns 
comment: We hope that our response was able to clarify all the concerns raised by the reviewer. We kindly ask the reviewer to let us know if they have some additional questions that they would like to be addressed.

We look forward to engage in a fruitful discussion!

From: ICLR.cc/2023/Conference/Paper754/Authors
title: Response to Reviewer L5LN [2/2]
comment: **Q6: $h$ in the Figure 2**

**A6:** Thank you for raising this issue. $h_i^1$ and $h_i^2$ are the node representations of node $i$ at the 1th and 2th layers, respectively. The node representation of node $i$ in the $k$-th layer ($h_i^k$) is determined by all the node representations in the $k-1$-th layer ($h^{k-1}$), as shown in Eq. 4.

$ h_i^k=ACT^k \left( W^k \cdot AGG^k \left( \\{ h_j^{k-1},  \forall j \in N(i) \\} \right)  \right)$. 

To avoid ambiguity, we remove the subscript $i$ of $h$ in Figure 2. The revised figure means that the node representation $h^k$ in the $k$-th layer is determined by the node representation $h^{k-1}$ in the $k-1$-th layer.

**Q7: Technical Contributions to The Field of BL-MOP**

**A7:** We reorganized the introduction and related work to re-emphasize our major technical contributions to the field of bi-level multi-objective optimization (BL-MOP). The optimization difficulty of BL-MOP is caused by the fact that **each candidate architecture (UL variable) corresponds to a set of Pareto weights (LL variable)** (see Figure 1). The existing work on solving BL-MOPs mainly includes three methods: classical [1], evolution-based [2], and gradient-based [3]. Classical methods require some strong mathematical assumptions, which make them incapable of being directly applicable to many practical scenarios. For evolution-based methods, the high computational cost of maintaining two solution sets is unacceptable in the deep learning system. The gradient-based method directly approximates the LL-MOP as a single objective problem to guarantee that each architecture corresponds to an optimal weight. But this method may lead to inconsistencies in task trade-offs at the upper and lower levels. However, our proposed BL-MOP algorithm firstly guarantees **the consistency of task preference at the upper and lower levels and does not need to maintain multiple weights for each candidate architecture in the optimization process**, which cleverly avoids the above optimization difficulties. A more detailed literature review and motivational discussion can be found in Appendix A.1.

[1] Multiobjective Bilevel Optimization. Mathematical Programming, 2010.

[2] Minimizing Expected Deviation in Upper-level Outcomes Due to Lower-level Decision-making in Hierarchical Multi-objective Problems, IEEE TEC, 2022.

[3] Multi-objective Meta Learning, NeurIPS, 2021.

**Q8: More Experimental Results on DMTGAS and Ablated Studies**

**A8:** DMTGAS is proposed to solve multiple tasks simultaneously, and thus cannot learn one task and predict another task. For the ablated study, Table 5 lists the experimental results obtained by DMTGASIC and DMTGAS on all data sets (see Appendix 4 in detail). It can be seen that DMTGAS outperforms DMTGASIC on all tasks, which shows the importance of maintaining preference consistency of UL-MOP and LL-MOP. We can observe this phenomenon in all data sets.

**Q9: Reproducibility**

**A9:** Thank you for raising this issue. We have revised Appendix A.3 to better describe the baselines, and make it clear how the dataset is preprocessed for downstream tasks. In addition, we have uploaded the source code: https://github.com/DMTGAS/DMTGAS.

For the baseline settings, we give a brief introduction as follows:

"**To verify the efficacy of DMTGAS, we provide four types of baselines for experiments with respect to two levels - MTL and Graph NAS: single-task handcrafted baselines (single-level single-objective optimization), single-task Graph NAS baselines (bi-level single-objective optimization), multi-task handcrafted baselines (single-level multi-objective optimization), and multi-task Graph NAS baselines (bi-level multi-objective optimization).**"

From: ICLR.cc/2023/Conference/Paper754/Authors
title: Response to Reviewer L5LN [1/2]
comment: Thank you for your time and effort in reviewing our work, and the valuable comments. We are glad to know you find our problem setting important and experimental results promising. We address your concerns as follows.

**Q1: Buffelli and Vandin's Work**

**A1:** Buffelli et al. [1] proposed a handcrafted GNN architecture for multi-task learning (MTGNN) and optimized it with a meta-learning approach. To the best of our knowledge, this is the latest task-agnostic MTGNN architecture. In this paper, to illustrate the advantages of multi-task graph neural architecture search, we adopt this handcrafted MTGNN architecture and optimize it with four MOO strategies, including linear scalarization (LS), multiple-gradient descent algorithm (MGDA), exact Pareto optimal search (EPO), and conditioned one-shot multi-objective search (COSMOS). To avoid ambiguity, we modified the description of multi-task handcrafted baselines (See Appendix A.3).

[1] Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach, IJCNN, 2022.

**Q2: UL and LL Sub-problems in The Abstract**

**A2:** Thank you for your suggestion. We revised the abstract to highlight the UL and LL sub-problems, and the search space (Figure 2).

“ *For the first time, we propose a simple but effective differentiable multi-task GNN architecture search framework (DMTGAS) with convergence guarantees. Its search space contains aggregation and connection operations of shared layers. By introducing consistent task trade-offs for UL and LL sub-problems, DMTGAS only alternately searches a single architecture and its weights to minimize multiple task losses via the gradient-based multi-objective optimizer, which neatly overcomes the above optimization difficulties.* ”

**Q3: Results in Table 3 for DMTGAS**

**A3:** Thank you for your suggestion. Both Handcrafted-COSMOS and DMTGAS-COSMOS are multi-task baselines. Therefore, they train and infer a model to solve three tasks (graph classification, node classification, and link prediction), simultaneously. To avoid ambiguity, we modified the description of Table 3 to highlight our task setting.

“ *Table 3 reports the Hypervolume (HV) values obtained by Handcrafted-COSMOS and DMTGAS-COSMOS to solve GC, NC, and LP over 5 independent runs.* ”

**Q4: Same Task-specific Layers for Multi-task Baseline**

**A4:** This paper considers a multi-task learning paradigm with hard parameter sharing [1], which consists of the shared layer and the task-specific layer. From a multi-objective perspective, weight optimization of shared-layer requires a trade-off between tasks, while that of task-specific layers does not [2]-[3]. To verify the effectiveness of the proposed two-layer multi-objective optimization algorithm, the designed search space in this paper only contains shared layers. Therefore, all multi-task baselines (both handcrafted and Graph NAS) use the same task-specific layers to exclude the effect of task-specific layers on algorithm performance.

Furthermore, we have added a more detailed baseline description for better readability (see Appendix A.3).

“ **To verify the efficacy of DMTGAS, we provide four types of baselines for experiments with respect to two levels - MTL and Graph NAS: single-task handcrafted baselines (single-level single-objective optimization), single-task Graph NAS baselines (bi-level single-objective optimization), multi-task handcrafted baselines (single-level multi-objective optimization), and multi-task Graph NAS baselines (bi-level multi-objective optimization).** ”

[1] An Overview of Multi-task Learning in Deep Neural Networks, arXiv, 2017.
[2] Multi-Task Learning as Multi-Objective Optimization, NeurIPS. 2018.
[3] Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization, arXiv, 2022.

**Q5: Comparison of Multi-task and Single-task Baselines**

**A5:** From Table 2, none of the multi-task handcrafted baselines with different optimization strategies (Handcrafted-MGDA, Handcrafted-LS, Handcrafted-EPO, and Handcrafted-COSMOS) achieves vast superiority compared to the single-task handcrafted baselines (ST-GCN, ST-SAGE, ST-GAT, ST-GIN). This phenomenon can also be found in Table 1 in Buffelli and Vandin’s work. In the setting of handcrafted MTGNN models with limited representational capabilities, excessive parameter sharing (negative transfer) may lead to multi-task handcrafted baselines suffering from some sub-optimality when tasks conflict.

From Table 2, the multi-task Graph NAS methods (DMTGAS-EPO and DMTGAS-COSMOS) outperform the single-task Graph NAS baseline (ST-SANE) in most cases. This phenomenon shows that the negative transfer is alleviated in the setting of searched MTGNN models with stronger representational capabilities. In addition, it should be emphasized that multi-task baselines consume less computational cost (see Table 6) and can achieve excellent performance on a certain task that cannot be achieved by single-task baselines (see Table 7).

From: ICLR.cc/2023/Conference/Paper754/Authors
title: Response to Reviewer A89U
comment: Thank you very much for your time and effort in reviewing our work. We are very glad to know you find our problem-setting novel and the paper well-structured. We address your concerns as follows.

**Q1: Baselines**

**A1:** To verify the efficacy of DMTGAS, we provide four types of baselines for experiments with respect to two levels - MTL and Graph NAS: single-task handcrafted baselines (single-level single-objective optimization), single-task Graph NAS baselines (bi-level single-objective optimization), multi-task handcrafted baselines (single-level multi-objective optimization), and multi-task Graph NAS baselines (bi-level multi-objective optimization).
Our comparison algorithms contain state-of-the-art single-task Graph NAS baselines [1]-[2] and multi-task handcrafted baselines with MOO algorithms [3]-[4]. It is worth noting that literatures [1] and [2] adopt a similar differentiable optimization method, called ST-SANE in this paper, which is one of the state-of-the-art methods in the single-task Graph NAS paradigm. EPO [3] is a MOO method for MTL with strict convergence guarantees. COSMOS [4] is one of the most advanced MOO methods in practice. Therefore these two state-of-the-art MOO algorithms are used to optimize the handcrafted MTGNN model, which are regarded as multi-task handcrafted baselines with MOO algorithms.

In addition, a new multi-task self-supervised graph learning problem [5] is proposed during the review of our paper, which adopted a classic MOO algorithm, MGDA [6]. Therefore, we use MGDA to optimize the handcrafted MTGNN model as a new multi-task handcrafted baseline for comparison. The experimental results (see Table 2) again demonstrate the effectiveness of our method.

|Methods | ENZYMES-GC | ENZYMES-NC | ENZYMES-LP | PROTEINS-GC | PROTEINS-NC | PROTEINS-LP | DHFR-GC | DHFR-NC | DHFR-LP |
| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: |
|Handcrafted-MGDA| 51.7(0.035) |88.6(0.008) | 70.9(0.019) | 68.0(0.016)|82.3(0.006)|75.5(0.018)|69.7(0.030)|85.7(0.003)|97.1(0.006)|
|DMTGAS-EPO (Ours)|**56.7**(0.049) |**89.1**(0.012) |**95.8**(0.022) |**87.0**(0.004) |**82.7**(0.005) |**96.8**(0.043) |**83.3**(0.023) |**86.2**(0.004) |**99.9**(0.006)|
|DMTGAS-COSMOS (Ours) |**52.7**(0.040) |**89.2**(0.006) |**94.3**(0.006) |**88.9**(0.024) |81.7(0.013) |**96.2**(0.011) |**79.2**(0.027) |**94.5**(0.014) |**98.7**(0.009)|

[1] Designing the Topology of Graph Neural Networks A Novel Feature Fusion Perspective, WWW 2022.

[2] Search to Aggregate Neighborhood for Graph Neural Network, ICDE, 2021.

[3] Scalable Pareto Front Approximation for Deep Multi-Objective Learning, ICDM, 2021.

[4] Multi-Task Learning with User Preferences Gradient Descent with Controlled Ascent in Pareto Optimization, ICML, 2020.

[5] Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization, arXiv, 2022.

[6] Multiple-gradient Descent Algorithm (MGDA) for Multiobjective Optimization. Comptes Rendus Mathematique, 2012.

**Q2: Reproducibility**

**A2:** We have uploaded the source code: https://github.com/DMTGAS/DMTGAS.

From: ICLR.cc/2023/Conference/Paper754/Authors
title: Response to Reviewer N2H6
comment: Thank you for your time and effort in reviewing our work. We are glad to know you find our work is novel, well-written and easy to follow, and has comprehensive literature reviews for Graph MTL and Automatic GNN. We address your concerns as follows.

**Q1: Technical Contribution**

**A1:** The combination of MTL and automatic graph learning is not straightforward. Automatic multi-task graph representation learning is modeled as a two-level multi-objective optimization problem (BL-MOP). The optimization difficulty of BL-MOP is caused by the fact that **each candidate architecture (UL variable) corresponds to a set of Pareto weights (LL variable)** (see Figure 1). The existing work on solving BL-MOPs mainly includes three methods: classical [1], evolution-based [2], and gradient-based [3]. Classical methods require some strong mathematical assumptions, which make them incapable of being directly applicable to many practical scenarios. For evolution-based methods, the high computational cost of maintaining two solution sets is unacceptable in the deep learning system. The gradient-based method directly approximates the LL-MOP as a single objective problem to guarantee that each architecture corresponds to an optimal weight. But this method may lead to inconsistencies in task trade-offs at the upper and lower levels. However, our proposed BL-MOP algorithm firstly guarantees **the consistency of task preference at the upper and lower levels and does not need to maintain multiple weights for each candidate architecture in the optimization process**, which cleverly avoids the above optimization difficulties. A more detailed literature review and motivational discussion can be found in Appendix A.1. We reorganized the introduction and related work to re-emphasize our major technical contributions to the field of BL-MOP.

[1] Multiobjective Bilevel Optimization. Mathematical Programming, 2010.

[2] Minimizing Expected Deviation in Upper-level Outcomes Due to Lower-level Decision-making in Hierarchical Multi-objective Problems, IEEE TEC, 2022.

[3] Multi-objective Meta Learning, NeurIPS, 2021.

**Q2: Better Modeling of Downstream Tasks**

**A2:** This paper is the first attempt at multi-task automatic graph representation learning. It is desirable to design a task-agnostic bi-level multi-objective optimization method. There are no specific prior requirements for downstream tasks. But it is worth noting that modeling the relationships between downstream tasks is a direction worth exploring, which is added in future work. For example, for graph classification and node classification tasks, it is necessary to search for GNN architectures that can produce node embeddings that are beneficial for classification according to the properties of the graph.

**Q3: Single-task Graph NAS Baselines**

**A3:** Our baselines contain a state-of-the-art single-task Graph NAS (bi-level single-objective optimization) method: ST-SANE [1-2]. It is worth noting that literatures [1] and [2] adopt a similar differentiable optimization method. The experimental results (see Table 2) demonstrate the advantage of the proposed method.

[1] Designing the Topology of Graph Neural Networks A Novel Feature Fusion Perspective, WWW 2022.

[2] Search to Aggregate Neighborhood for Graph Neural Network, ICDE, 2021.

From: ICLR.cc/2023/Conference/Paper754/Authors
title: Response to Reviewer aLXm [3/3]
comment: **Q8: Novelty**

**A8:** We reorganized the introduction and related work to re-emphasize our major technical contributions to the field of bi-level multi-objective optimization (BL-MOP). The optimization difficulty of BL-MOP is caused by the fact that **each candidate architecture (UL variable) corresponds to a set of Pareto weights (LL variable)** (see Figure 1). The existing work on solving BL-MOPs mainly includes three methods: classical [1], evolution-based [2], and gradient-based [3]. Classical methods require some strong mathematical assumptions, which make them incapable of being directly applicable to many practical scenarios. For evolution-based methods, the high computational cost of maintaining two solution sets is unacceptable in the deep learning system. The gradient-based method directly approximates the LL-MOP as a single objective problem to guarantee that each architecture corresponds to an optimal weight. But this method may lead to inconsistencies in task trade-offs at the upper and lower levels. However, our proposed schemes **firstly guarantee the consistency of task preference at the upper and lower levels and do not need to maintain multiple weights for each candidate architecture in the optimization process**, which cleverly avoids the above optimization difficulties. A more detailed literature review and motivational discussion can be found in Appendix A.1.

[1] Multiobjective Bilevel Optimization. Mathematical Programming, 2010.

[2] Minimizing Expected Deviation in Upper-level Outcomes Due to Lower-level Decision-making in Hierarchical Multi-objective Problems, IEEE TEC, 2022.

[3] Multi-objective Meta Learning, NeurIPS, 2021.

**Q9: Reproducibility**

**A9:** Thank you for raising this issue. We have uploaded the source code: https://github.com/DMTGAS/DMTGAS.

From: ICLR.cc/2023/Conference/Paper754/Authors
title: Response to Reviewer aLXm [2/3]
comment: **Q6: Self-supervised GNNs**

**A6:** This paper is the first attempt at multi-task automatic graph representation learning. It is desirable to design a task-agnostic optimization method. There are no specific requirements for downstream tasks. Therefore, it is unreasonable to contrast with self-supervised methods that exploit prior knowledge of downstream tasks. But it is worth noting that automatic multi-task self-supervised graph representation learning is a direction worth exploring, which is added in future work. Next, we discuss two literatures [1]-[2] on self-supervised GNNs. AutoSSL [1] is one of the most famous self-supervised GNNs. ParetoGNN [2] is the latest self-supervised GNN, which was proposed during the review of our paper.

- In the AutoSSL [1], the problem of automated self-supervised task search is $\min _{\lambda_1, \cdots, \lambda_n} \mathcal{H} (f_\theta* (\mathcal{G}) ),\quad s.t. \theta^*=\arg \min_\theta \sum_\{i=1\}^n \lambda_i \ell_i (f_\theta(\mathcal{G}).$ $\mathcal{H}$ denotes the quality measure for the obtained node embeddings, which needs to be pre-defined according to the downstream task required to be solved. Therefore, AutoSSL cannot directly handle the three downstream tasks (node classification, graph classification, and link prediction) in our experiments simultaneously.
- In the ParetoGNN [2], they formulate the self-supervised graph learning as a multi-objective optimization: $\min _{\substack{\boldsymbol{\theta}_g \\ \boldsymbol{\theta}_1, \ldots, \boldsymbol{\theta}_K}} \mathcal{L}\left(\mathcal{G}, \boldsymbol{\theta}_g, \boldsymbol{\theta}_1, \ldots, \boldsymbol{\theta}_K\right)=\min _{\substack{\boldsymbol{\theta}_g, \ldots, \boldsymbol{\theta}_K}}\left(\mathcal{L}_1\left(\mathcal{G} ; \mathcal{T}_1, \boldsymbol{\theta}_g, \boldsymbol{\theta}_1\right), \ldots, \mathcal{L}_K\left(\mathcal{G} ; \mathcal{T}_K, \boldsymbol{\theta}_g, \boldsymbol{\theta}_K\right)\right)$. The most classic MOO algorithm, MGDA [3], is adopted to deal with this problem. Therefore, we use MGDA to optimize the handcrafted MTGNN model as a new multi-task handcrafted baseline for comparison. The experimental results (see Table 2) again demonstrate the effectiveness of our method.

|Methods | ENZYMES-GC | ENZYMES-NC | ENZYMES-LP | PROTEINS-GC | PROTEINS-NC | PROTEINS-LP | DHFR-GC | DHFR-NC | DHFR-LP |
| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: |
|Handcrafted-MGDA| 51.7(0.035) |88.6(0.008) | 70.9(0.019) | 68.0(0.016)|82.3(0.006)|75.5(0.018)|69.7(0.030)|85.7(0.003)|97.1(0.006)|
|DMTGAS-EPO (Ours)|**56.7**(0.049) |**89.1**(0.012) |**95.8**(0.022) |**87.0**(0.004) |**82.7**(0.005) |**96.8**(0.043) |**83.3**(0.023) |**86.2**(0.004) |**99.9**(0.006)|
|DMTGAS-COSMOS (Ours) |**52.7**(0.040) |**89.2**(0.006) |**94.3**(0.006) |**88.9**(0.024) |81.7(0.013) |**96.2**(0.011) |**79.2**(0.027) |**94.5**(0.014) |**98.7**(0.009)|

[1] Automated Self-Supervised Learning for Graphs, ICLR, 2022.

[2] Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization, arXiv, 2022.

[3] Multiple-gradient Descent Algorithm (MGDA) for Multiobjective Optimization. Comptes Rendus Mathematique, 2012.

**Q7: Dataset**

**A7:** Thank you for this suggestion. We add a reference [1], where three downstream tasks (graph classifications, node classification, and link prediction) are performed on these datasets. The implementation details of the three datasets satisfying the multi-task condition can be found in [1]. For better clarity, we have added a more detailed description of the experimental setup of node labels in Appendix A.3.

[1] Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach, IJCNN, 2022.

From: ICLR.cc/2023/Conference/Paper754/Authors
title: Response to Reviewer aLXm [1/3]
comment: Thank you for your time and effort in reviewing our work, and the valuable comments. We are glad to know you find our work interesting and our contributions significant. We address your concerns as follows.

**Q1: The Writing of This Paper**

**A1:** Thank you for this suggestion. We have carefully revised the paper to add more detailed claims, and will continually proofread the paper before the final publication.

**Q2: Claims About The Decoder**

**A2:** Thank you for raising this issue. We fully agree that GNNs do not need a “decoder”. “Graph representation learning models typically follow the same architectural pattern with an encoder-decoder structure.” Such claims are common in various existing literature about graph representation learning [1]-[3]. Most commonly adopted GNNs, e.g., GCN, GAT, PPNP, GCNII, etc., can produce embedding vectors (called encoders). The embedding vectors are input into task-specific layers (called decoders) to handle various downstream tasks. They are trained end-to-end. To avoid ambiguity, we have revised this claim.

[1] Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach, IJCNN, 2022.

[2] Machine Learning on Graphs: A Model and Comprehensive Taxonomy. JMLR, 2022.

[3] A Comprehensive Survey on Graph Neural Networks. IEEE TNNLS, 2020.

**Q3: Claims About The Task Conflicts in Citation “Buffelli & Vandin (2020)**

**A3:** Thank you for raising this issue. To avoid ambiguity, we removed the citation “Buffelli & Vandin (2020)”. As can be seen from Table 2, none of the multi-task handcrafted baselines with different MOO strategies (Handcrafted-MGDA, Handcrafted-LS, Handcrafted-EPO, and Handcrafted-COSMOS) achieves vast superiority compared to the single-task handcrafted baseline (ST-GCN, ST-SAGE, ST-GAT, and ST-GIN), which indicates a potential conflict between tasks in the setting of the handcrafted model. It is worth noting that this phenomenon can also be found in Table 2 of the citation “Buffelli & Vandin (2020)”.

**Q4: Claims About Technical Details**

**A4:** Thank you for this suggestion. We have reorganized the sections before the preliminary background knowledge to increase readability by modifying the technical details about the Pareto optimality and preference. It should be emphasized that the main technical contribution of this paper is to propose a bi-level multi-objective optimization (BL-MOP) algorithm with consistent preference vectors. Existing evolution-based and gradient-based works cannot directly deal with BL-MOPs in deep learning. We re-emphasize the contribution to the field of BL-MOPs (see Introduction and Appendix A.1).

**Q5: Claims About Eqs 2 and 3**

**A5:** We thank again the reviewer for the comments. We removed the redundant “=” in Eqs 2 and 3. $L(…)=(L_1(…), L_2(), …, L_m(…))$ is the general form of the MOO problem, which can be found in various literatures on MOO[1]-[2]. When the base optimizer of DMTGAS is Linear Scalarization, $L()$ is the weighted sum of all objectives, otherwise (EPO and COSMOS) it is not. So, we adopt the general form of the MOO problem in this paper.

[1] Multi-objective Optimization by Learning Space Partition, ICLR, 2022.

[2] Multi-Task Learning as Multi-Objective Optimization, NeurIPS, 2018.

**Q6: Baseline**

**A6:** To verify the efficacy of DMTGAS, we provide four types of baselines for experiments with respect to two levels - MTL and Graph NAS: single-task handcrafted baselines (single-level single-objective optimization), single-task Graph NAS baselines (bi-level single-objective optimization), multi-task handcrafted baselines (single-level multi-objective optimization), and multi-task Graph NAS baselines (bi-level multi-objective optimization).

Our comparison algorithms contain state-of-the-art single-task Graph NAS baselines [1]-[2] and multi-task handcrafted baselines with MOO algorithms [3]-[4]. It is worth noting that literatures [1] and [2] adopt a similar differentiable optimization method, called ST-SANE in this paper, which is one of the state-of-the-art methods in the single-task Graph NAS paradigm. EPO [3] is a MOO method for MTL with strict convergence guarantees. COSMOS [4] is one of the most advanced MOO methods in practice. Therefore, these two MOO algorithms are used to optimize the handcrafted MTGNN model, which are regarded as state-of-the-art multi-task handcrafted baselines with MOO algorithms.

[1] Designing the Topology of Graph Neural Networks A Novel Feature Fusion Perspective, WWW 2022.

[2] Search to Aggregate Neighborhood for Graph Neural Network, ICDE, 2021.

[3] Scalable Pareto Front Approximation for Deep Multi-Objective Learning, ICDM, 2021.

[4] Multi-Task Learning with User Preferences Gradient Descent with Controlled Ascent in Pareto Optimization, ICML, 2020.

From: ICLR.cc/2023/Conference/Paper754/Authors
title: Response Summary
comment: We sincerely thank all reviewers for their constructive comments and valuable suggestions. Following their suggestions, we have carefully revised our paper and the major revision can be summarized as follow:

- **Better Emphasis on Our Key Idea**: We have reorganized the Introduction and Related Work (see Appendix A.1) to re-emphasize our major technical contributions to the field of bi-level multi-objective optimization (BL-MOP). In addition, we have also revised the abstract to highlight the upper-level and lower-level sub-problems, and the search space.

- **Detailed Description for Dataset and Baseline**: We have revised Appendix A.3 to better describe the baselines, and make it clear how the dataset is preprocessed for downstream tasks.

- **More Experiments**: We have added a multi-task handcrafted baseline with the MGDA to again illustrate the effectiveness of the proposed method.

- **Potential Future Work**: We have revised the Conclusion to clearly discuss the potential future work, including self-supervised learning and task-tailored multi-task graph learning.

- **Better Clarity**: Many careful modifications have been made for better exposition and clarification.

- **Better Reproducibility**: We have uploaded the source code: https://github.com/DMTGAS/DMTGAS.

We combine the main paper and the appendix into a single file but keep the main paper 9-page following this year's requirement. We will adjust the content once an extra page is allowed. All major changes are highlighted in **red**.

Point-to-point responses can be found below to each reviewer. We are also glad to improve our work to address any further concerns continually.

Paper754 Authors

From: ICLR.cc/2023/Conference/Paper754/Reviewer_aLXm
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a multi-task learning framework for graph neural networks that not only simultaneously handles multiple tasks but also searches optimal model architectures. The proposed framework, namely DMTGAS, utilizes the exact Pareto optimization (EPO) of the existing literatures to fulfill these two purposes. The architecture search space includes node aggregators, layer aggregators, and skip connections; and the Pareto solution can be acquired both with or without the preference vector. Experiments on three tasks (i.e., graph classification, node classification, and link prediction) are conducted over three graph-level datasets. 
strength_and_weaknesses: S1.
This paper successfully migrates the idea of the exact Pareto optimization (EPO) from the machine learning community to the graph learning field. It utilizes this idea to simultaneously learn model architectures as well as their corresponding Pareto optimal parameters. 

S2.
Pareto learning over graphs is fairly un-explored, but it's a very interesting research topic.

W1.
The writing of this paper requires substantial improvement beyond typos and grammatical errors:  \
W1.1 In Introduction, the authors made several false claims or those that are too arbitrary. For example, “The decoder inputs the embedding…” does not hold for most of the commonly adopted GNNs. E.g., GCN, GAT, PPNP, GCNII and many more do not need such a “decoder”. \
W1.2. The authors exploit the citation “Buffelli & Vandin (2020)” to support too many claims that the original paper did not cover (e.g., task conflicts). \
W1.3. This paper mentioned a lot technical details about Pareto learning (e.g., Pareto optimality, preference vector, etc) before the preliminary background knowledge, which hurts the readability. \
W1.4. There are too many typos and mistakes in the equations that hinders the readers from understanding. For example, for Eqs 2 and 3,  there is nothing behind argmin in the second line; the comma looks like a superscript for the second line. The multiple used $\mathcal{L(\cdots)}=(\mathcal{L_1(\cdots)}, \mathcal{L_2(\cdots)}, \dots, \mathcal{L_m(\cdots)})$ is also kinda confusing. Is the RHS supposed to be a summation or something?

W2.
The evaluation misses two important perspectives: \
W2.1 [baseline]: The proposed framework focuses on both NAS (neural architecture search) and multi-task learning. Hence, the recent works on these two fields should be compared; otherwise it is difficult to tell the contributions of the performance improvement. Besides, self-supervised GNNs should also be compared, as they can also conduct multi-task downstream inference with the parameters of GNN encoders frozen. \
W2.2 [dataset]: The three datasets used for evaluation are used for graph classifications and no previous literature conduct the node classification and the link prediction tasks over these datasets. These two tasks are hardly applicable and legitimate on graph-level datasets and hence the experiments are not convincing. Moreover, this paper has poor descriptions upon these new settings and how the node labels are populated.

W3.
Though not purely focusing on multi-task learning on GNN, there is one very revenant paper called AutoSSL [Wei Jin, et al, ICLR 22] that also works on multi-task GNN (both training and inference), and the authors completely ignores this notable work. 

clarity,_quality,_novelty_and_reproducibility: Clarity: very poor. Please refer to my previous comments. 
 
Novelty and Quality: I think the work would have a good impact after substantial update.
 
Reproducibility: no code uploaded, cannot evaluate. 

summary_of_the_review: The idea of this paper is interesting. However I think the writing problems that are far beyond typos and grammatical errors hurt the quality of this paper. Besides, the evaluation is also problematic, which needs additional comparison experiments with NAS baselines and multi-task GNNs (AutoSSL). 

Based on my comments above, I think this paper requires substantial revision and is not ready to be published yet. Hence, I recommend rejection. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper754/Reviewer_N2H6
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper presents a NAS framework named DMTGAS for multi-task graph representation learning problems. The model is formulated as a bi-level multi-objective optimization problem to optimize the GNN architecture and its weights. The overall optimization for multi-task learning is guided by the multi-objective Pareto optimality with two strategies covering both situations w/ or w/o preference. The authors also provide theoretical analysis to prove the convergence of the method. The authors conduct experiments to demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Pros:  
1. Automatic multi-task graph representation learning seems to be a novel problem.  
2. The paper is clearly written and easy to follow.  
3. The literature reviews for Graph MTL and Automatic GNN are comprehensive.  

Cons:  
1. The technical contribution of the paper is limited since the combination of MTL and automated graph learning is straightforward. Specifically, the proposed method seems to simply replace the loss function in the classical graph NAS framework and adopts a general multi-objective optimization method. Thus, though I acknowledge that this proposed method is a valid solution, the technical novelty and potential influence on the community are limited.   
2. The proposed formulation for the multi-task learning problem simply combines all the objective functions, while neglecting more delicate relationships between tasks. Better modeling of tasks and even jointly with GNN architecture can further improve the proposed method.  
3. In experiments, the authors do not compare with the existing (single-task) graph NAS baselines. Comparing with them can further demonstrate the advantage of the proposed method.  

=== after rebuttal===
I have read the rebuttal and thank the authors for their efforts in improving the paper. Nevertheless, I think the quality of the paper does not meet the bar, as shared by other reviewers. All things considered, I will keep my score.

clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and reproducibility of the paper are reasonbly good. The novelty is limited (see Cons above). 
summary_of_the_review: See above
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper754/Reviewer_A89U
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposed a novel approach for multi-task graph representation learning. More specifically, graph representation learning aims to learn a dense representation of a given graph structured data. In addition, multi-task scenario aims to learn the graph representations which is able to be optimized to multiple tasks such as classification and prediction. However, directly/manually tune the network structure which is feasible for multiple tasks is challenging. While it is not well explored of GNN architecture searching for multi-task scenario. 

In this work, a few specific design optimization strategies are proposed to solve the task. The relaxation loss objective which balances the notes/layers aggregators, and skip connections are proposed. In addition, an optimization strategy is proposed to iteratively optimize the network structure. Experimental experiments demonstrate the effectiveness of the proposed model.

strength_and_weaknesses: Strength:

This paper well introduces the background of the multi-task graph representation learning approach in structure searching scenarios. The background is comprehensive which help readers clear understand the motivation of the task.

In addition, the model is clearly introduced with explanations about the optimization details as well as the motivation of each optimization steps.

Moreover, extra theoretical analysis is provided to show the bound of the performance.

Weaknesses:

The baselines evaluated in this paper is either the general GNN networks (e.g., GCN, GAT, GIN) or the handcrafted strategy which is proposed in 2020. The experimental results is not solid. An extra state-of-the-art method is highly recommended.

clarity,_quality,_novelty_and_reproducibility: This paper clearly introduces the proposed method. The paper provide a link which claims the code availability AFTER the paper is accepted.
summary_of_the_review: This paper proposed a novel approach for GNN architecture searching for multi-graph representation learning scenario. Explanations, experiments, and analysis are fully provided. To this end, I hold positive for this submission.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper754/Reviewer_L5LN
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes a differentiable multi-task learning approach for GNN. The contributions of the work are: automatic multi-task graph representation learning using bi-level multi-objective learning, using a single Pareto architecture and its weights instead of two solution sets, and introducing DMTGAS, a differentiable multi-task GNN architecture framework.
Authors evaluate the performance of three variations of DMTGAS (i.e. LS, EPO, COSMOS) on three real world graph data sets (i.e. ENZYMES, PROTEINS, and DHFR) under Node Classifier (NC), Link Prediction (LP), and Graph Classifier (GC) multi-task learning settings and compare it with a set of baseline methods. Experimental results show superior performance of the proposed method (with a few exceptions for NC) over most baseline methods and the hand-crafted alternatives. Results also show the benefit of DMTGAS when two tasks (out of GC, NC, LP) are used to train a model to predict the third task. In addition an ablation study on ENZYMES data set shows DMTGAS better performance compared to its alternative with inconsistent trade-offs. 
strength_and_weaknesses: Strength: 
- The paper addresses multi-task learning in GNN which is an active research area with solid significance.
- Experimental results look promising.

Weakness:
Please refer to comments on clarity, novelty and quality.
clarity,_quality,_novelty_and_reproducibility: Clarity:
There are a few areas that after reading the paper are still unclear to the reviewer.
- Can authors elaborate more on why they refer to Buffelli and Vandin's meta-task learning architecture for multi-task learning as manual/hand crafted?
- Can authors elaborate more on their focus on LL and UP tasks in the abstract section and how it connects to their proposed model illustration in Figure 2?
- It's not clear to the reviewer whether reported results in Table 3 for DMTGAS are generated by multi-task learning. If yes, which tasks are used to train the model versus inference? If not, it's not clear why DMTGAS outperforms single task learning alternatives. I can imagine authors might be reporting results from using embedding generated by multi-task learning used in simpler models but explanations are not clear enough.
- In section 4, authors state that "We use the same architecture of task-specific layers for all multi-task baselines to ensure a fair comparison". Can authors clarify why using the same task-specific layers for multi-task baseline is a fair comparison although one can assume without letting multi-task learning, learn these layers it won't achieve its highest performance.
- Can authors shed more light into the reason behind inferior performance of single task learning alternatives compared to the proposed approaches (in most cases) reported in Table 3, although when dealing with single task learning scenarios it's more likely that multi-task learning suffers from some sub-optimality due to forcing the model learning multiple-tasks simultaneously? e.g. refer to results reported in Table 1 in Buffelli and Vandin work.
- I might be misinterpreting but Looking at Figure 2, it looks like that for example, h_i^2 is being determined by h_i^1 only which maybe interpreted as node is representation is independent of it's neighbors (e.g. h_j, h_k, etc.), am I missing something?

Novelty:
- The contributions of the paper, i.e. BL-MOP, using a single Pareto architecture, and differentiable multi-task learning are not individually new. It would be beneficial to discuss what makes the combination of these techniques so successful.

Quality:
- Experimental results can be more compelling if more scenarios are covered. e.g. for multi-task learning (Figure 3) in addition to learning for two tasks and predicting for the third, authors could have reported learning for one task and predicting for another one as well. For the ablated study knowing the consistency of results over other data sets would help. Authors do not need to report additional data if space is a concern and can suffice by mentioning that results are consistent/inconsistent under those scenarios with what is covered in the paper. 

Reproducibility: 
- Due to the lack of clarity on a few aspects of the work, reproducibility is currently under question but hopefully this will be addressed by providing further clarification and publishing the source code in case the paper gets published.
summary_of_the_review: The paper addresses an important issue and experimental results look promising, however, certain aspects of the paper is not clear and requires further clarifications.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold