Original: P ARETO
Revised: P ARETO

Original: A UTOMATIC
Revised: A UTOMATIC

Original: M ULTI
Revised: M ULTI

Original: T ASK
Revised: -T ASK

Original: G RAPH
Revised: G RAPH

Original: R EPRE -
Revised: R EPRE -

Original: SENTATION L EARNING
Revised: SENTATION L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Various excellent graph representation learning models, such as graph neural networks (GNNs), can produce highly task-specific embeddings in an end-toend manner.
Revised: Various excellent graph representation learning models, such as graph neural networks (GNNs), can produce highly task-specific embeddings in an end-toend manner.

Original: Due to the low transferability of learned embeddings and limited representational capabilities of handcrafted models, existing efforts cannot efficiently handle multiple downstream tasks simultaneously, especially in resourceconstrained scenarios.
Revised: Due to the low transferability of learned embeddings and limited representational capabilities of handcrafted models, existing efforts cannot efficiently handle multiple downstream tasks simultaneously, especially in resourceconstrained scenarios.

Original: This paper first tries to automatically search for multi-task
Revised: This paper first tries to automatically search for multi-task

Original: GNN architectures to improve the generalization performance of GNNs through knowledge sharing across tasks.
Revised: GNN architectures to improve the generalization performance of GNNs through knowledge sharing across tasks.

Original: Because of possible task (objective) conflicts and complex dependencies of architectures and weights, the multi-task GNN architecture search is a bi-level multi-objective optimization problem (BL-MOP) to find a set of Pareto architectures and their Pareto weights, representing different trade-offs across tasks at upper and lower levels (UL and LL), respectively.
Revised: Because of possible task (objective) conflicts and complex dependencies of architectures and weights, the multi-task GNN architecture search is a bi-level multi-objective optimization problem (BL-MOP) to find a set of Pareto architectures and their Pareto weights, representing different trade-offs across tasks at upper and lower levels (UL and LL), respectively.

Original: The
Revised: The

Original: Pareto optimality of sub-problems results in each Pareto architecture corresponding to a set of Pareto weights, which is particularly challenging in deep learning with high training costs.
Revised: Pareto optimality of sub-problems results in each Pareto architecture corresponding to a set of Pareto weights, which is particularly challenging in deep learning with high training costs.

Original: For the first time, we propose a simple but effective differentiable multi-task GNN architecture search framework (DMTGAS) with convergence guarantees.
Revised: For the first time, we propose a simple but effective differentiable multi-task GNN architecture search framework (DMTGAS) with convergence guarantees.

Revised: Its search space contains aggregation and connection operations of shared layers.

Original: By introducing consistent task preferences for UL and
Revised: By introducing consistent task trade-offs for UL and

Original: LL, DMTGAS only alternately optimizes a single architecture and  weights  via the gradient-based multi-objective optimizer, which neatly overcomes the above optimization difficulties.
Revised: LL, DMTGAS only alternately searches a single architecture and its weights to minimize multiple task losses via the gradient-based multi-objective optimizer, which neatly overcomes the above optimization difficulties.

Original: Experimental results on several tasks in three real-world graph datasets demonstrate the superiority of the GNNs obtained by our proposal compared with existing handcrafted ones.
Revised: Experimental results on several tasks in three real-world graph datasets demonstrate the superiority of the GNNs obtained by our proposal compared with existing handcrafted ones.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph neural network (GNN) Xu et al.
Revised: Graph neural network (GNN) Xu et al.

Original: (2019) captures the graph information by massage passing with the neighbors on graph-structured data.
Revised: (2019) captures the graph information by massage passing with the neighbors on graph-structured data.

Original: It has been widely used in several graph machine learning tasks, including graph classification (GC), node classification (NC), and link prediction (LP) Cai et al.
Revised: It has been widely used in several graph machine learning tasks, including graph classification (GC), node classification (NC), and link prediction (LP) Cai et al.

Original: A typical GNN model follows an encoder-decoder neural module trained in an end-to-end fashion.
Revised: A typical graph representation learning model follows an encoder-decoder neural module trained in an end-to-end fashionChami et al.

Original: The encoder generates low-dimensional embedding vectors by aggregating node features and graph-structured information.
Revised: Most commonly adopted GNNs can generate low-dimensional embedding vectors by aggregating node features and graph-structured information.

Original: The decoder inputs the embedding vectors to the custom neural network to handle downstream graph machine learning tasks.
Revised: Then embedding vectors are fed into the custom neural network to handle downstream graph machine learning tasks.

Original: The model tends to get highly task-relevant embedding vectors.
Revised: The model tends to get highly task-relevant embedding vectors.

Original: However, the low transferability Buffelli & Vandin (2020) makes it unsuitable for multiple downstream tasks.
Revised: However, the low transferability Buffelli & Vandin (2022) makes it unsuitable for multiple downstream tasks.

Original: Since graph tasks often do not appear alone and there is exploitable knowledge between them, multi-task graph representation learning has received much attention in practical applications in recent years, such as biomedical relation extraction Li & Ji (2019) and physico-chemical ADMET endpoints modeling Montanari et al.
Revised: Since graph tasks often do not appear alone and there is exploitable knowledge between them, multi-task graph representation learning has received much attention in practical applications in recent years, such as biomedical relation extraction Li & Ji (2019) and physico-chemical ADMET endpoints modeling Montanari et al.

Original: In real-world scenarios, optimizing multiple tasks is difficult due to the limited representational capabilities of handcrafted models and task conflicts.
Revised: In real-world scenarios, optimizing multiple tasks is difficult due to the limited representational capabilities of handcrafted models and task conflicts.

Original: For the former, existing multi-task graph neural networks (MTGNNs) adopt a fixed architecture serving a few specific tasks, which lacks flexibility
Revised: For the former, existing multi-task graph neural networks (MTGNNs) adopt a fixed architecture serving a few specific tasks, which lacks flexibility

Original: Wang & Zhu (2021).
Revised: Wang & Zhu (2021).

Original: With the emergence of many techniques for graph representation learning, given a few new tasks on graphs, manually identifying the optimal MTGNN model from scratch requires a lot of expertise.
Revised: With the emergence of many techniques for graph representation learning, given a few new tasks on graphs, manually identifying the optimal MTGNN model from scratch requires a lot of expertise.

Original: For the latter, the existing MTGNN training strategy combines multiple objectives (task losses) into a single objective, called linear scalarization (LS).
Revised: For the latter, the existing MTGNN training strategy combines multiple objectives (task losses) into a single objective, called linear scalarization (LS).

Original: This strategy leadsto the inevitable lower performance when tasks conflict due to excessive parameter sharing between tasks Buffelli & Vandin (2020).
Revised: This strategy leads

Original: Since different conflicting tasks require trade-offs, multi-task graph
Revised: [Figure] [Figure]

Original: Case 1: Task preference r is known 
Revised: preference [Figure] known Case

Original: Case 2: Task preference r is unknown losslossloss
Revised: [Figure] [Figure] preference r is unknown losslossloss

Original: Network Weights
Revised: Network Weights

Original: Network Architecture lossr
Revised: Network Architecture lossr

Original: Outer loop lossloss
Revised: Outer loop lossloss

Original: Network Weights
Revised: Network Weights

Original: Network Architecture
Revised: Network Architecture

Original: Inner loop lossr~
Revised: Inner loop lossr~

Original: P r
Revised: P r

Original: Outer loop loss
Revised: Outer loop loss

Original: Sample
Revised: Sample

Original: Sample
Revised: Sample

Original: BL-MOP
Revised: BL-MOP

Original: PBL-MOP
Revised: PBL-MOP

Original: BL-PFLP
Revised: BL-PFLP

Original: Inner loop
Revised: [Figure]

Original: Network Weights
Revised: Network Weights

Original: Network Architecture losslosslossloss 1 Pareto Weight Front
Revised: Network Architecture loss [Figure] loss [Figure]

Revised: [Figure]

Original: Pareto Architecture Front
Revised: Pareto [Figure] Pareto [Figure]

Original: Sample
Revised: Sample

Original: UL-MOP
Revised: UL-MOP

Original: LL-MOP
Revised: LL-MOP

Original: Inner loop
Revised: Inner loop

Original: Outer loop
Revised: Outer loop

Original: DMTGAS
Revised: DMTGAS

Original: Input Task Task
Revised: [Figure]

Original: Supernet
Revised: Supernet

Original: Task
Revised: [Figure]

Original: Task
Revised: preference

Original: Supernet Task Task Supernet
Revised: [Figure]

Original: Task
Revised: Case

Original: Task Supernet Case 1: Task preference r is known
Revised: [Figure]

Original: Case 2: Task preference r is unknownlosslossloss
Revised: preference [Figure] unknown

Original: Network Weights
Revised: [Figure] [Figure]

Original: Network Architecture lossr 
Revised: Architecture lossr [Figure]

Original: Outer loop lossloss
Revised: lossloss

Original: Network Weights
Revised: Network Weights

Original: Network Architecture
Revised: Network Architecture

Original: Inner loop lossr~
Revised: Inner loop lossr~

Original: P r
Revised: P r

Original: Outer loop loss
Revised: Outer loop loss

Original: Sample
Revised: Sample

Original: Sample
Revised: Sample

Original: BL-MOP
Revised: BL-MOP

Original: PBL-MOP
Revised: PBL

Revised: [Figure]

Original: BL-PFLP
Revised: BL-PFLP

Original: Inner loop Network Weights Network Architecture losslosslossloss 1 Pareto Weight Front
Revised: [Figure]

Original: Pareto ArchitectureFront
Revised: Network Architecture

Original: Sample
Revised: [Figure]

Original: UL-MOP

Original: LL-MOP

Original: Inner loop

Original: Outer loop DMTGAS Input Task
Revised: to the inevitable lower performance when tasks conflict due to excessive parameter sharing between tasks.

Original: Task

Original: Supernet Task Task Supernet representation learning is inherently a multi-objective optimization (MOO) problem Sener & Koltun (2018).
Revised: Since different conflicting tasks require trade-offs, multi-task graph representation learning is inherently a multi-objective optimization (MOO) problem Sener & Koltun (2018).

Original: Therefore, it is necessary to automatically search for a MTGNN model that matches various user-desired task preferences from a multi-objective perspective.
Revised: Therefore, it is necessary to automatically search for a MTGNN model that matches various user-desired task preferences from a multi-objective perspective.

Original: This paper first investigates automatic multi-task graph representation learning to search for userdesired MTGNN models,which is naturally modeled as a bi-level multi-objective optimization problem (BL-MOP), where each task loss is an objective.
Revised: This paper first investigates automatic multi-task graph representation learning to search for userdesired MTGNN models, naturally modeled as a bi-level multi-objective optimization problem (BLMOP), where each task loss is an objective.

Original: The upper-level and lower-level sub-problems aim to obtain a set of Paretooptimal architectures and weights with different task trade-offs, respectively.
Revised: It aims to obtain a set of Pareto architectures and weights, representing different trade-offs across tasks at the upper level (UL) and lower level (LL), respectively.

Original: Due to the Pareto optimality (non-uniqueness of Pareto solutions) for sub-problems and the complicated dependency across architecture and weights, a candidate architecture is feasible if and only if its weight is Pareto optimal. Each Pareto architecture corresponding to a set of Pareto weights.
Revised: The optimization difficulty of BL-MOP is caused by the fact that each candidate architecture corresponds to a set of Pareto weights (see Figure 1).

Original: In general, a common method directly approximates the lower-level MOP (LL-MOP) as a single objective problemYe et al.
Revised: In general, a common method directly approximates the LL-MOP as a single objective problem to guarantee that each architecture corresponds to an optimal weightYe et al.

Original: ; Franceschi et al.
Revised: ; Franceschi et al.

Original: (2018), which may lead to inconsistencies in task preferences for inner and outer optimizations.
Revised: But this method may lead to inconsistencies in task trade-offs at the UL and LL.

Original: Furthermore, the  performanceof the method is severely affected by the manually tuned task loss weights.
Revised: Furthermore, the method’s performance is severely affected by the manually tuned task loss weights.

Original: In this paper, we propose a differentiable multi-task GNN architecture search framework (DMTGAS) with a convergence guarantee.
Revised: In this paper, we propose a differentiable multi-task GNN architecture search framework (DMTGAS) with a convergence guarantee.

Revised: In practice, the user may give the desired task trade-off (preference) r .

Original: Two novel optimization schemeswith or without preference r are presented to solve the BL-MOP, as shown in Figure 1. 1)  a preference-consistent scheme is proposed to obtain a single Pareto optimal architecture and weights.
Revised: Two novel optimization schemes are presented to solve the BL-MOP by involving r , as shown in Figure 1. 1) When r is unknown, a preference-consistent scheme is proposed to obtain a single Pareto optimal architecture and weights.

Original: It alternately optimizes a single MTGNN architecture and weights with the same task preferencer via preference-based MOO methods Mahapatra & Rajan (2020).
Revised: It alternately optimizes a single MTGNN architecture and weights with the same task preference via preference-based MOO methods Mahapatra & Rajan (2020).

Original: The architecture and weights of a single model are optimized alternately via Pareto-front learning methods Ruchte & Grabocka (2021).
Revised: When r is unknown, the architecture and weights of a single model are optimized alternately via Pareto-front learning methods Ruchte & Grabocka (2021).

Original: At training time, the model couples the sampled preference to the feature space for joint training.
Revised: At training time, the model couples the sampled preference to the feature space for joint training.

Original: At inference time, the model can generate the entire Pareto front given different preferences.
Revised: At inference time, the model can generate the entire Pareto architecture set given different preferences.

Original: These two schemes  do not need to maintain multiplemodel weightswith different task trade-offs for each  architecture in the optimization process, which cleverly avoids the above optimization difficulties.
Revised: Both schemes guarantee the consistency of preference and do not need to maintain multiple weights for each candidate architecture in the optimization process, which cleverly avoids the above optimization difficulties.

Original: In DMTGAS, a single MTGNN model is trained by the above schemes to obtain a Pareto optimal architecture and weights with the given r or the entire Pareto front.
Revised: In DMTGAS, a single MTGNN model is trained by the above schemes.

Original: All candidate architectures are sampled from a predefined search space (represented as a supernet).
Revised: All candidate architectures are sampled from a predefined search space (represented as a supernet).

Original: Through the continuous relaxation of the search space Huan et al.
Revised: Through the continuous relaxation of the search space Huan et al.

Original: (2021), the multiple objectives are converted to differentiable.
Revised: (2021), the multiple objectives are converted to differentiable.

Original: We evaluate DMTGAS and handcrafted methods on three real-world graph datasets (ENZYMES, PROTEINS, and DHFR), which are employed to perform multiple high-profile tasks, including node classification, graph classification, and link prediction.
Revised: We evaluate DMTGAS and handcrafted methods on three real-world graph datasets (ENZYMES, PROTEINS, and DHFR), which are employed to perform multiple high-profile tasks, including node classification, graph classification, and link prediction.

Original: Experimental results demonstrate the superiority of the MTGNN model obtained by DMTGAS over the handcrafted ones and that DMTGAS can provide a model that matches user task preferences.
Revised: Experimental results demonstrate the superiority of the MTGNN model obtained by DMTGAS over the handcrafted ones and that DMTGAS can provide a model that matches user task preferences.

Original: The main contributions of the work are summarized as follows: 1 1) To improve the representation ability of handcrafted GNN models through knowledge sharing among downstream tasks, we first propose the automatic multi-task graph representation learning, which is modeled as a BL-MOP.
Revised: The main contributions of the work are summarized as follows: 1 1) To improve the representation ability of handcrafted GNN models through knowledge sharing among downstream tasks, we first propose the automatic multi-task graph representation learning, which is modeled as a BL-MOP.

Original: 2) To overcome difficulties caused by eachPareto architecture corresponding to a Pareto weight set, two optimization strategies are presented by introducing  preference r , which only maintain one architecture and its weights instead of two solution sets.
Revised: 2) To overcome difficulties caused by each architecture corresponding to a Pareto weight set, two optimization strategies are presented by introducing consistent task preference r , which only maintain one architecture and its weights instead of two solution sets.

Original: 3) We develop a differentiable multi-task GNN architecture search framework, called DMTGAS, to obtain a Pareto optimal architecture and  weights with a user-desired task preference or entire Pareto front.
Revised: 3) We develop a differentiable multi-task GNN architecture search framework, called DMTGAS, to obtain entire Pareto front, or an architecture and its weights with a user-desired task preference.

Original: Furthermore, the convergence properties of DMTGAS are also proved.
Revised: Furthermore, the convergence properties of DMTGAS are also proved.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-Task Graph Representation Learning Most works on multi-task graph representation learning have focused on learning an embedding model for several applications on graphs Avelar et al.
Revised: Multi-Task Graph Representation Learning Most works on multi-task graph representation learning have focused on learning an embedding model for several applications on graphs Avelar et al.

Original: ; Holtz et al.
Revised: ; Holtz et al.

Original: ; Li & Ji (2019); Montanari et al.
Revised: ; Li & Ji (2019); Montanari et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: Xie et al.
Revised: Xie et al.

Original: (2020a;b); Buffelli & Vandin (2020); Sun et al.
Revised: (2020a;b); Buffelli & Vandin (2022); Sun et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: However, these applicationspecific handcrafted methods rely heavily on expert knowledge and ignore task conflicts.
Revised: However, these applicationspecific handcrafted methods rely heavily on expert knowledge and ignore task conflicts.

Original: We provide more reviews in the Appendix 1.
Revised: We provide more reviews in the Appendix 1.

Original: Automatic Graph Representation Learning This section mainly focuses on the differentiable neural architecture search (DNAS), which has gained popularity in recent yearsGao et al.
Revised: Automatic Graph Representation Learning This section mainly focuses on the differentiable neural architecture search (DNAS), which has gained popularity in recent yearsGao et al.

Original: Wei et al.
Revised: Wei et al.

Original: DNAS builds a single supernet that contains all operations Huan et al.
Revised: DNAS builds a single supernet that contains all operations Huan et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: ; Qin et al. (2021).
Revised: ; Qin et al. (2021).

Original: A candidate operation is treated as a probability distribution over all possible operations.
Revised: A candidate operation is treated as a probability distribution over all possible operations.

Original: The objective function of DNAS is differentiable through continuous relaxation of the search space.
Revised: The objective function of DNAS is differentiable through continuous relaxation of the search space.

Original: Thus, gradient-based methods can jointly optimize the architecture and weights.
Revised: Thus, gradient-based methods can jointly optimize the architecture and weights.

Original: Existing works mainly focus on the search space and search strategy to learn a graph representation model customized for a given task Wang & Zhu (2021).
Revised: Existing works mainly focus on the search space and search strategy to learn a graph representation model customized for a given task Wang & Zhu (2021).

Original: However, automatic learning methods for multiple tasks on graphs have not been noticed.
Revised: However, automatic learning methods for multiple tasks on graphs have not been noticed.

Original: We provide more reviews in the Appendix 1.
Revised: We provide more reviews in the Appendix 1.

Original: Gradient-Based Multi-Objective Optimization for Multi-Task Learning
Revised: Gradient-Based Multi-Objective Optimization for Multi-Task Learning

Original: Due to the inevitable objective conflicts when learning multiple machine learning tasks by a model simultaneously, gradient-based multi-objective optimization for multi-task learning has been widely studied.
Revised: Due to the inevitable objective conflicts when learning multiple machine learning tasks by a model simultaneously, gradient-based multi-objective optimization for multi-task learning has been widely studied.

Original: Multiobjective optimization (MOO) aims to find a set of Pareto solutions (called Pareto front in the objective space) with different objective trade-offs. 
Revised: Multiobjective optimization (MOO) aims to find a set of Pareto solutions (called Pareto front in the objective space) with different objective trade-offs. Sener et al.

Revised: Sener & Koltun (2018) firstly employed

Revised: MGDA to train a deep multi-task neural network to generate a Pareto stationary solution.

Original: A series of preference-based methods are proposed to generate a Pareto solution with a given task preference (task trade-off)
Revised: Then, a series of preference-based methods are proposed to generate a Pareto solution with a given task preference (task trade-off)

Original: Lin et al.
Revised: Lin et al.

Original: ; Mahapatra & Rajan (2020).
Revised: ; Mahapatra & Rajan (2020).

Original: For example, an exact Pareto optimal (EPO) search can find an exact Pareto optimal solution by combining gradient descent and controlled ascent Mahapatra & Rajan (2020).
Revised: For example, an exact Pareto optimal (EPO) search can find an exact Pareto optimal solution by combining gradient descent and controlled ascent Mahapatra & Rajan (2020).

Original: However, most of these methods need to learn a model from scratch separately for each Pareto solution, which is not desirable when dealing with deep multi-task learning.
Revised: However, most of these methods need to learn a model from scratch separately for each Pareto solution, which is not desirable when dealing with deep multi-task learning.

Original: Therefore, some Pareto-front learning methods Lin et al.
Revised: Therefore, some Pareto-front learning methods Lin et al.

Original: (2020); Navon et al.
Revised: Navon et al.

Original: (2021); Ruchte & Grabocka (2021) are presented to learn a model that at inference time can generate entire Pareto front.
Revised: (2021); Ruchte & Grabocka (2021) are presented to learn a model that at inference time can generate entire Pareto front.

Original: (called Pareto-front learning problems).
Revised: (called Pareto-front learning problems).

Original: One of the most common ideas is to design a hypernetwork to generate model weights with different trade-offs in an end-to-end trainingLin et al.
Revised: One of the most common ideas is to design a hypernetwork to generate model weights with different trade-offs in an end-to-end trainingLin et al.

Original: (2020); Navon et al.
Revised: (2020); Navon et al.

Original: Ruchte et al.
Revised: Ruchte et al.

Original: Ruchte & Grabocka (2021) proposed to augment the preference to the feature space to directly condition the model to avoid the extra overhead incurred in training the hyper-network (called COSMOS).
Revised: Ruchte & Grabocka (2021) proposed to augment the preference to the feature space to directly condition the model to avoid the extra overhead incurred in training the hyper-network (called COSMOS).

Original: However, the gradient-based BLMOP method has not been noticed.
Revised: However, the gradient-based BLMOP method has not been noticed.

Original:  A UTOMATIC M ULTI -T ASK G RAPH R EPRESENTATION L EARNING 3. P ROBLEM S TATEMENT
Revised:  A UTOMATIC M ULTI -T ASK G RAPH R EPRESENTATION L EARNING 3. P ROBLEM S TATEMENT

Original: In general, no one solution  can achieve the optimum of all objectives in a MOO problem.
Revised: In general, no one solution α ∗ can achieve the optimum of all objectives in a MOO problem.

Original: Thus, the MOO min α ∈A
Revised: Thus, the MOO min α ∈A

Original: L ( α ) =
Revised: L ( α ) =

Original: ( L 1 ( α ) , · · · , L m ( α )) aims to obtain a set of Pareto optimal solutions
Revised: ( L 1 ( α ) , · · · , L m ( α )) aims to obtain a set of Pareto optimal solutions

Original: A ∗ , which provide different optimal trade-offs among all objectives.
Revised: A ∗ , which provide different optimal trade-offs among m objectives.

Original: The following definitions are given.
Revised: The following definitions are given.

Original: Pareto doninance A solution α ∈ A is said to dominate another solution
Revised: Pareto doninance A solution α ∈ A is said to dominate another solution

Original: α ′ ∈
Revised: α ′ ∈

Original: A ( α ≺ α ′ )
Revised: A ( α ≺ α ′ )

Original: if and only if two conditions are met: (1) L i ( α ) ⩽ L i ( α ′ ) , ∀ i ∈ { 1 , . . .
Revised: if and only if two conditions are met: (1) L i ( α ) ⩽ L i ( α ′ ) , ∀ i ∈ { 1 , . . .

Original: , m } , (2) L j ( α ) <
Revised: , m } , (2) L j ( α ) <

Original: L j ( α ′ ) , ∃ j ∈ { 1 , . . .
Revised: L j ( α ′ ) , ∃ j ∈ { 1 , . . .

Original: , m } .
Revised: , m } .

Original: Pareto optimality A solution α ∈ A is said to Pareto optimal if it is not dominated by any other solution α ′ ∈ A .
Revised: Pareto optimality A solution α ∈ A is said to Pareto optimal if it is not dominated by any other solution α ′ ∈ A .

Original: The set of all Pareto optimal solutions is called the Pareto set A ∗ .
Revised: The set of all Pareto optimal solutions is called the Pareto set A ∗ .

Original: The Pareto front is M -dimensional manifold of the Pareto set in the loss space.
Revised: The Pareto front is M -dimensional manifold of the Pareto set in the loss space.

Original: Exact Pareto optimal with a preference vector r A Pareto optimal solution α ∗ ∈
Revised: Exact Pareto optimal with a preference vector r A Pareto optimal solution α ∗ ∈

Original: A is said to be exact Pareto optimal with a preference vector r =
Revised: A is said to be exact Pareto optimal with a preference vector r =

Original: ( r 1 , r 2 , . . .
Revised: ( r 1 , r 2 , . . .

Original: , r m ) if r 1 L 1 ( α ∗ ) =
Revised: , r m ) if r 1 L 1 ( α ∗ ) =

Original: r 2 L 2 ( α ∗ )
Revised: r 2 L 2 ( α ∗ )

Original: = . . .
Revised: = . . .

Original: =
Revised: =

Original: r m L m ( α ∗ ) .
Revised: r m L m ( α ∗ ) .

Original: It is worth noting that for any exact Pareto optimal solution, L ( α ∗ ) is the intersection of the preference vector r and the Pareto front.
Revised: It is worth noting that for any exact Pareto optimal solution, L ( α ∗ ) is the intersection of the preference vector r and the Pareto front.

Original: To address the challenge of automatically designing MTGNN models that matches user-desired task preferences, we model automatic multi-task graph representation learning as a BL-MOP.
Revised: To address the challenge of automatically designing MTGNN models that matches user-desired task preferences, we model automatic multi-task graph representation learning as a BL-MOP.

Original: Formally, let A and W be the search space of multi-task model architectures α and weights w .
Revised: Formally, let A and W be the search space of multi-task model architectures α and weights w .

Original: In general, given a training set D train := { ( x n , y n ) } NTn =1 , x n ∈ X , y n ∈ Y and a validation set D valid := { ( x n , y n ) } NVn =1 , x n ∈ X , y n ∈ Y drawn from a data sampling distribution P D , a BL-MOP can be formulated as follow:
Revised: In general, given a training set D train := { ( x n , y n ) } NTn =1 , x n ∈ X , y n ∈ Y and a validation set D valid := { ( x n , y n ) } NVn =1 , x n ∈ X , y n ∈ Y drawn from a data sampling distribution P D , a BL-MOP can be formulated as follow:

Original: [Equation]
Revised: [Equation]

Original: where function L : R p × R n → R m .
Revised: where function L : R p × R n → R m .

Original: p, n, m are the dimensions of architecture search space, weight search space, and loss space, respectively.
Revised: p, n, m are the dimensions of architecture search space, weight search space, and loss space, respectively.

Original: The upper level and lower level MOP (UL-MOP and LLMOP) aim to find Pareto optimal architecture set A ∗ and Pareto optimal weight set W ∗ , respectively.
Revised: The upper level and lower level MOP (UL-MOP and LLMOP) aim to find Pareto optimal architecture set A ∗ and Pareto optimal weight set W ∗ , respectively.

Original: Because a feasible architecture requires that its weights be Pareto optimal, the difficulty of BL-MOP lies in the customization of the interaction scheme of architecture and weights during the optimization process.
Revised: Because a feasible architecture requires that its weights be Pareto optimal, the difficulty of BL-MOP lies in the customization of the interaction scheme of architecture and weights during the optimization process.

Original: For each architecture, which exact Pareto optimal weight is chosen is challenging in deep learning.
Revised: For each architecture, which exact Pareto optimal weight is chosen is challenging in deep learning.

Original: In practical scenarios, users may give task preferences r .
Revised: In practical scenarios, users may give task preferences r .

Original: So the following two strategies for different scenarios are proposed to address the issue by maintaining only a single architecture and weights during the alternate optimization process, as shown in Figure 1.
Revised: So the following two strategies for different scenarios are proposed to address the issue by maintaining only a single architecture and weights during the alternate optimization process, as shown in Figure 1.

Original: r is known In the automatic multi-task graph representation learning, we expect to obtain the architecture and weights with the consistent preference r , so the problem (1) is transformed into finding the exact Pareto optimal architecture α ∗ and weights w ∗ with a given preference vector r ∈ [0 , 1] m , (cid:80) mj
Revised: r is known In the automatic multi-task graph representation learning, we expect to obtain the architecture and weights with the consistent preference r , so the problem (1) is transformed into finding the exact Pareto optimal architecture α ∗ and weights w ∗ with a given preference vector r ∈ [0 , 1] m , (cid:80) mj

Original: =1 r j = 1 , r j ≥ 0 .
Revised: =1 r j = 1 , r j ≥ 0 .

Original: The preference-specific BL-MOP is described as
Revised: The preference-specific BL-MOP is described as

Original: [Equation]
Revised: [Equation]

Original: where L ( w ∗ ( α ) , α, D valid , r ) =
Revised: where L ( w ∗ ( α ) , α, D valid , r ) =

Original: ( L 1 ( w ∗ ( α ) , α, D valid , r 1 ) , · · · , L m ( w ∗ ( α ) , α, D valid , r m )) and
Revised: ( L 1 ( w ∗ ( α ) , α, D valid , r 1 ) , · · · , L m ( w ∗ ( α ) , α, D valid , r m )) and

Original: L ( w, α, D train , r ) =
Revised: L ( w, α, D train , r ) =

Original: ( L 1 ( w, α, D train , r 1 ) , · · · , L m ( w, α, D train , r m )) .
Revised: ( L 1 ( w, α, D train , r 1 ) , · · · , L m ( w, α, D train , r m )) .

Original: In the iterative process of optimizing the problem (2), it is only necessary to maintain one architecture and weights with the consistent preference in real time.
Revised: In the iterative process of optimizing the problem (2), it is only necessary to maintain one architecture and weights with the consistent preference in real time.

Original: r is unknown When the task trade-offs are unknown, it is expensive to learn the architecture and weights of a separate model for each task preference r ∈ [0 , 1] m , (cid:80) mj
Revised: r is unknown When the task trade-offs are unknown, it is expensive to learn the architecture and weights of a separate model for each task preference r ∈ [0 , 1] m , (cid:80) mj

Original: =1 r j = 1 , r j ≥ 0 sampled from a probability distribution P r .
Revised: =1 r j = 1 , r j ≥ 0 sampled from a probability distribution P r .

Original: Therefore, task preference r are incorporated into the end-to-end training of architecture α ∗ and weights w ∗ of a single model f
Revised: Therefore, task preference r are incorporated into the end-to-end training of architecture α ∗ and weights w ∗ of a single model f

Original: ( x, r ; w, α ) to obtain the entire Pareto front at inference time, which is formulated as a bi-level Pareto learning problem: 
Revised: ( x, r ; w, α ) to obtain the entire Pareto front at inference time, which is formulated as a bi-level Pareto learning problem: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: L ( y, f ( x, r ; w ∗ ( α ) , α )) =
Revised: L ( y, f ( x, r ; w ∗ ( α ) , α )) =

Original: ( L 1 ( y, f ( x, r ; w ∗ ( α ) , α ))) , · · · , L m ( y, f ( x, r ; w ∗ ( α ) , α ))) and L ( y, f ( x, r ; w, α )) =
Revised: ( L 1 ( y, f ( x, r ; w ∗ ( α ) , α ))) , · · · , L m ( y, f ( x, r ; w ∗ ( α ) , α ))) and L ( y, f ( x, r ; w, α )) =

Original: ( L 1 ( y, f ( x, r ; w, α )) , · · · , L m ( y, f ( x, r ; w, α ))) .
Revised: ( L 1 ( y, f ( x, r ; w, α )) , · · · , L m ( y, f ( x, r ; w, α ))) .

Original: In the scheme, the inference condition of a model is transformed into f ( x, r ; w ∗ ( α ) , α ) :
Revised: In the scheme, the inference condition of a model is transformed into f ( x, r ; w ∗ ( α ) , α ) :

Original: X × R m × A → Y for the UL-MOP and f ( x, r ; w, α ) :
Revised: X × R m × A → Y for the UL-MOP and f ( x, r ; w, α ) :

Original: X × R m × W → Y for the LL-MOP.
Revised: X × R m × W → Y for the LL-MOP.

Original: Practically, the input x is concatenated with the r .
Revised: Practically, the input x is concatenated with the r .

Original: Architecture and weights of a model are trained on this joint feature space.
Revised: Architecture and weights of a model are trained on this joint feature space.

Original: Once we find the optimal solution α ∗ and w ∗ ( α ∗ ) for the problem (3), we can approximate the entire Pareto architecture front and Pareto weight front at inference time.
Revised: Once we find the optimal solution α ∗ and w ∗ ( α ∗ ) for the problem (3), we can approximate the entire Pareto architecture front and Pareto weight front at inference time.

Original: h iz i SK l ( hi 1 , hi 2 , …, h iK -1 , h iK ) h iK … …… … h ih i KNode classification ？
Revised: hz SK l ( h 1 , h 2 , …, h K -1 , h K ) h K … …… … hh KNode classification ？

Original: Task
Revised: Task

Original: Task
Revised: Task

Original: Task n Graph classification Link prediction n o  s o  l o 
Revised: Task n Graph classification Link prediction n o  s o  l o 

Original: Node Aggregators
Revised: Node Aggregators

Original: Skip Connections
Revised: Skip Connections

Original: Layer Aggregators
Revised: Layer Aggregators

Original: Graph Data

Original: [Figure]

Original: Task trade-off

Original: [Figure]
Revised: [Figure]

Original: Optimal

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: GCN, GIN, GENIEPATH, MLP, GAT,
Revised: GCN, GIN, GENIEPATH, MLP, GAT,

Original: GAT-SYM, GAT-COS, GAT-LINEAR, GAT-GEN-LINEAR,
Revised: GAT-SYM, GAT-COS, GAT-LINEAR, GAT-GEN-LINEAR,

Original: MEAN-SAGE, MAX-SAGE, SUM-SAGE
Revised: MEAN-SAGE, MAX-SAGE, SUM-SAGE

Original: [Table]
Revised: [Table]

Original: LSTM, CONCAT, MAX, MEAN, SUM
Revised: LSTM, CONCAT, MAX, MEAN, SUM

Original: Skip

Original: [Table]
Revised: [Table]

Original: ZERO, IDENTITY 3. D IFFERENTIABLE M ULT -T ASK GNN A RCHITECTURE S EARCH F RAMEWORK 
Revised: ZERO, IDENTITY 3. D IFFERENTIABLE M ULT -T ASK GNN A RCHITECTURE S EARCH F RAMEWORK Preliminaries:

Original: Preliminaries: GNN The message-passing framework of a typical K -layer GNN can be formulated as follows:
Revised: GNN The message-passing framework of a typical K -layer GNN can be formulated as follows:

Original: the node representation h ki of each node i in the k -th layer is updated as:
Revised: the node representation h ki of each node i in the k -th layer is updated as:

Original: [Equation]
Revised: [Equation]

Original: where ACT k , W ( k ) , and AGG ( k ) are the activation function, the learnable weight matrix, and the aggregation function in the k -th layer, respectively.
Revised: where ACT k , W ( k ) , and AGG ( k ) are the activation function, the learnable weight matrix, and the aggregation function in the k -th layer, respectively.

Original: N ( i ) is the neighborhood of node i .
Revised: N ( i ) is the neighborhood of node i .

Original: Considering the residual GNN model Xu et al.
Revised: Considering the residual GNN model Xu et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: the final node representation z i of the node i is calculated by the layer aggregator function AGG l and layer skip-connections SK l :
Revised: the final node representation z i of the node i is calculated by the layer aggregator function AGG l and layer skip-connections SK l :

Original: [Equation]
Revised: [Equation]

Original: For the graph-level representation, the node representation information from the entire graph G = { V, E } is aggregated by a readout function R:
Revised: For the graph-level representation, the node representation information from the entire graph G = { V, E } is aggregated by a readout function R:

Original: [Equation]
Revised: [Equation]

Original: Search Space
Revised: Search Space

Original: The search space of DMTGAS, represented as a supernet Huan et al.
Revised: The search space of DMTGAS, represented as a supernet Huan et al.

Original: (2021), is a directed acyclic graph (DAG) as shown in Figure 2.
Revised: (2021), is a directed acyclic graph (DAG) as shown in Figure 2.

Original: The source neural module is the input, the sink neural module is the output, and the edge ( a, b ) represents the operation o ab on the graph data (such as a GCN layer and a GIN layer).
Revised: The source neural module is the input, the sink neural module is the output, and the edge ( a, b ) represents the operation o ab on the graph data (such as a GCN layer and a GIN layer).

Original: Similar to the search space of many well-established single-task
Revised: Similar to the search space of many well-established single-task

Original: NAS Huan et al.
Revised: NAS Huan et al.

Original: ; Wei et al.
Revised: ; Wei et al.

Original: (2021; 2022), we focus on three important operation components in the micro and macro search space: node aggregators O n , layer aggregators O l , and layer skipconnections O s as shown in Table 1, which are based on classic model: MLP Wei et al.(2021), GIN
Revised: (2021; 2022), we focus on three important operation components of shared layers: node aggregators O n , layer aggregators O l , and layer skip-connections O s as shown in Table 1, which are based on classic model: MLP Wei et al.

Original:  Xu et al.
Revised: GIN Xu et al.

Original: (2019), GCN Kipf & Welling (2017), GAT Velickovic et al.  GeniePath Liu et al.
Revised: GCN Kipf & Welling (2017), GAT Velickovic et al. (2017), GeniePath Liu et al.

Original: (2019), and SAGE Hamilton et al.
Revised: (2019), and SAGE Hamilton et al.

Original: For a three-layer  architecture backbone, the size of the search space is 12 3 × 2 3 × 5 = 69120 .
Revised: For a three-layer shared-layer architecture (backbone), the size of the search space is 12 3 × 2 3 × 5 = 69120 .

Original: If the computational resources are sufficient, more operations can be expanded into the search space.
Revised: If the computational resources are sufficient, more operations can be expanded into the search space.

Original: Continuous Relaxation Following the design principles of representative differentiable NAS methods, DMTGAS are employed to solve problems (2) and (3), whose search space of UL-MOPs is continuous.
Revised: Continuous Relaxation Following the design principles of representative differentiable NAS methods, DMTGAS are employed to solve problems (2) and (3), whose search space of UL-MOPs is continuous.

Original: The discrete selection of an operation o ab ( x a ) in the supernet is relaxed to a softmax of all candidate operations in O ( chosen from O n , O l , and O s ) as:
Revised: The discrete selection of an operation o ab ( x a ) in the supernet is relaxed to a softmax of all candidate operations in O ( chosen from O n , O l , and O s ) as:

Original: [Equation]
Revised: [Equation]

Original: where α ab is trainable operation mixing vectors (called architecture parameters) to control the choice of operations, and x a is the output of node a in the supernet.
Revised: where α ab is trainable operation mixing vectors (called architecture parameters) to control the choice of operations, and x a is the output of node a in the supernet.

Original: In short, each operation is treated as a probability distribution over all candidate operations.
Revised: In short, each operation is treated as a probability distribution over all candidate operations.

Original: After obtaining architecture parameters, the input of each node b in the supernet is calculated by aggregating all input edges as x b = (cid:80) a o ab ( x a ) .
Revised: After obtaining architecture parameters, the input of each node b in the supernet is calculated by aggregating all input edges as x b = (cid:80) a o ab ( x a ) .

Original: In this way, the search space of UL-MOPs is parameterized by the architecture parameters α .
Revised: In this way, the search space of UL-MOPs is parameterized by the architecture parameters α .

Original: Optimization Strategy
Revised: Optimization Strategy

Original: The optimization strategy of DMTGAS is presented in Algorithm 1 to solve problems (2) and (3).
Revised: The optimization strategy of DMTGAS is presented in Algorithm 1 to solve problems (2) and (3).

Original: It is difficult to directly obtain the closed solution of the LL-MOP in problems (2) and (3) due to the complicated dependency across architecture and weights.
Revised: It is difficult to directly obtain the closed solution of the LL-MOP in problems (2) and (3) due to the complicated dependency across architecture and weights.

Original: A common method Beirami et al.
Revised: A common method Beirami et al.

Original: ; Ye et al.
Revised: ; Ye et al.

Original: ; Franceschi et al.
Revised: ; Franceschi et al.

Original: (2018) is to replace weight optimization with a dynamical system.
Revised: (2018) is to replace weight optimization with a dynamical system.

Original: So we consider approximate forms of these problems as:
Revised: So we consider approximate forms of these problems as:

Original: [Equation]
Revised: [Equation]

Original: (9) where w K ( α ) is the iterative solution of the LL-MOP.
Revised: where w K ( α ) is the iterative solution of the LL-MOP.

Original: For the LL-MOP in the formula (2), given an architecture α , a preference vector r , and numbers of iterations K , a sequence { w k ( α ) } Kk =1 can be obtained as:
Revised: For the LL-MOP in the formula (2), given an architecture α , a preference vector r , and numbers of iterations K , a sequence { w k ( α ) } Kk =1 can be obtained as:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: η is the step size.
Revised: η is the step size.

Original: And d k is a valid MOO gradient direction, which is a convex combination of the vanilla gradients.
Revised: And d k is a valid MOO gradient direction, which is a convex combination of the vanilla gradients.

Original: Parameter µ = ( µ 1 , · · · , µ m ) can be calculated by the preference-based MOO method (such as LS Navon et al.
Revised: Parameter µ = ( µ 1 , · · · , µ m ) can be calculated by the preference-based MOO method (such as LS Navon et al.

Original: (preference vector r as loss weights) and EPO Mahapatra & Rajan (2020)).
Revised: (preference vector r as loss weights) and EPO Mahapatra & Rajan (2020)).

Original: For the LL-MOP in the formula (3), given an architecture α , a preference vector r ∼ P r , and numbers of iterations K , a sequence { w k ( α ) } Kk =1 can be obtained as the formula (10), where
Revised: For the LL-MOP in the formula (3), given an architecture α , a preference vector r ∼ P r , and numbers of iterations K , a sequence { w k ( α ) } Kk =1 can be obtained as the formula (10), where

Original: [Equation]
Revised: [Equation]

Original: τ is the penalty parameter.
Revised: τ is the penalty parameter.

Original: d k is a valid MOO gradient direction that can be obtained by the Paretofront learning method.
Revised: d k is a valid MOO gradient direction that can be obtained by the Paretofront learning method.

Original: COSMOS Ruchte & Grabocka (2021) is adopted in the paper.
Revised: COSMOS Ruchte & Grabocka (2021) is adopted in the paper.

Original: In the same way, the UL-MOP in problems (8) and (9) can also be directly solved by the same MOO method.
Revised: In the same way, the UL-MOP in problems (8) and (9) can also be directly solved by the same MOO method.

Original: α can be updated by:
Revised: α can be updated by:

Original: [Equation]
Revised: [Equation]

Original: For the UL-MOP in the problem (8),
Revised: For the UL-MOP in the problem (8),

Original: [Equation]
Revised: [Equation]

Original:  where parameter ν = ( ν 1 , · · · , ν m ) can be calculated by the same preference-based MOO method.
Revised: [Equation] where parameter ν = ( ν 1 , · · · , ν m ) can be calculated by the same preference-based MOO method.

Original: For the UL-MOP in the problem (9),
Revised: For the UL-MOP in the problem (9),

Original: [Equation]
Revised: [Equation]

Original: where σ is the penalty parameter.
Revised: where σ is the penalty parameter.

Original: After the optimization, a operation o ab of the largest weight in the supernet is selected as the final discrete architecture α , i.e., o ab = argmax o ∈O α abo .
Revised: After the optimization, a operation o ab of the largest weight in the supernet is selected as the final discrete architecture α , i.e., o ab = argmax o ∈O α abo .

Original: Finally, discrete architecture α is retrained by the same MOO method to obtain weights with the high performance.
Revised: Finally, discrete architecture α is retrained by the same MOO method to obtain weights with the high performance.

Original: Require: T, K : Numbers of iterations, ξ, η : Step size, r : Preference vector ( r is known), τ, σ : Penalty parameter ( r is unknown).
Revised: Require: T, K : Numbers of iterations, ξ, η : Step size, r : Preference vector ( r is known), τ, σ : Penalty parameter ( r is unknown).

Original: Ensure: α, w : The obtained architecture and weights.
Revised: Ensure: α, w : The obtained architecture and weights.

Original: 1: Randomly initialize α 0 ; 2: for each t ∈ [0 , T − 1] do 3: if r is unknown then 4:
Revised: 1: Randomly initialize α 0 ; 2: for each t ∈ [0 , T − 1] do 3: if r is unknown then 4:

Original: Sample a preference vector r ∼ P r ; 5: end if 6:
Revised: Sample a preference vector r ∼ P r ; 5: end if 6:

Original: Randomly initialize w t 0 ( α t ) ; 7: for each k ∈ [0 , K − 1] do 8:
Revised: Randomly initialize w t 0 ( α t ) ; 7: for each k ∈ [0 , K − 1] do 8:

Original: Update w tk
Revised: Update w tk

Original: +

Revised: +

Original: ( α t ) by the formula (10); 9: end for 10:
Revised: ( α t ) by the formula (10); 9: end for 10:

Original: Update α t +1 by the formula (13); 11: end for 12: Derive the final discrete architecture α based on α T ; 13: Retrain on the architecture α by the same MOO method to obtain its weights w .
Revised: Update α t +1 by the formula (13); 11: end for 12: Derive the final discrete architecture α based on α T ; 13: Retrain on the architecture α by the same MOO method to obtain its weights w .

Original: 3. T HEORETICAL A NALYSIS
Revised: 3. T HEORETICAL A NALYSIS

Original: In this section, the convergence of Algorithm 1 for the problem (2) is presented (The proof for the problem (3) is similar).
Revised: In this section, the convergence of Algorithm 1 for the problem (2) is presented (The proof for the problem (3) is similar).

Original: The following assumptions are given first.
Revised: The following assumptions are given first.

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: It is assumed that: 1) A is a compact set; 2) argmin w ∈W
Revised: It is assumed that: 1) A is a compact set; 2) argmin w ∈W

Original: L ( w, α, D train , r ) is a singleton for every α ∈ A ; 3) { w k ( α ) }
Revised: L ( w, α, D train , r ) is a singleton for every α ∈ A ; 3) { w k ( α ) }

Original: Kk =1 is uniformly bounded on A 4) L ( w ( α ) , α, D valid , r ) is jointly continuous.
Revised: Kk =1 is uniformly bounded on A 4) L ( w ( α ) , α, D valid , r ) is jointly continuous.

Original: Assumption 3.2.
Revised: Assumption 3.2.

Original: It is assumed that: 1) L ( ∗ , α, D valid , r ) is uniformly Lipschitz continuous; 2) { w k ( α ) } Kk =1 converges uniformly to w ∗ ( α ) on A as K → + ∞ .
Revised: It is assumed that: 1) L ( ∗ , α, D valid , r ) is uniformly Lipschitz continuous; 2) { w k ( α ) } Kk =1 converges uniformly to w ∗ ( α ) on A as K → + ∞ .

Original: Assumption 3.1 2) shows that for each architecture α ∗ , the problem (2) (or (3)) only admits a unique optimal weight w ∗ .
Revised: Assumption 3.1 2) shows that for each architecture α ∗ , the problem (2) (or (3)) only admits a unique optimal weight w ∗ .

Original: In addition, Assumption (3.1) and Assumption (3.2) are natural and widely used to analyzed bi-level single-objective optimization problems (BLSOPs) Franceschi et al.
Revised: In addition, Assumption (3.1) and Assumption (3.2) are natural and widely used to analyzed bi-level single-objective optimization problems (BLSOPs) Franceschi et al.

Original: Grazzi et al.
Revised: Grazzi et al.

Original: ; Ye et al.
Revised: ; Ye et al.

Original: Under these assumptions, we have:
Revised: Under these assumptions, we have:

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Assuming Assumption (3.1) is satisfied, then L ( w ∗ ( α ) , α, D valid , r ) →
Revised: Assuming Assumption (3.1) is satisfied, then L ( w ∗ ( α ) , α, D valid , r ) →

Original: L ( w ∗ ( α ) , α, D valid , r ) for any convergent sequence α → α .
Revised: L ( w ∗ ( α ) , α, D valid , r ) for any convergent sequence α → α .

Original: Proof.
Revised: Proof.

Original: See the Appendix 2.
Revised: See the Appendix 2.

Original: Theorem 3.1 shows the existence of solutions and the convergence of the LL-MOP.
Revised: Theorem 3.1 shows the existence of solutions and the convergence of the LL-MOP.

Original: It is similar to that of the BL-SOPs Franceschi et al.
Revised: It is similar to that of the BL-SOPs Franceschi et al.

Original: ; Grazzi et al.
Revised: ; Grazzi et al.

Original: ; Ye et al.
Revised: ; Ye et al.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Assuming Assumption (3.1) and Assumption (3.2) are satisfied, then
Revised: Assuming Assumption (3.1) and Assumption (3.2) are satisfied, then

Original: L ( w K ( α ) , α, D valid , r ) continuously converges to L ( w ∗ ( α ) , α, D valid , r ) .
Revised: L ( w K ( α ) , α, D valid , r ) continuously converges to L ( w ∗ ( α ) , α, D valid , r ) .

Original: Proof.
Revised: Proof.

Original: See the Appendix 2.
Revised: See the Appendix 2.

Original: Theorem 3.2 shows the convergence of Algorithm 1.
Revised: Theorem 3.2 shows the convergence of Algorithm 1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Experimental Setting We evaluate DMTGAS with LS, EPO, and COSMOS (called DMTGASLS, DMTGAS-EPO, and DMTGAS-COSMOS) on three real-world graph datasets (ENZYMES,
Revised: Experimental Setting We evaluate DMTGAS with LS, EPO, and COSMOS (called DMTGASLS, DMTGAS-EPO, and DMTGAS-COSMOS) on three real-world graph datasets (ENZYMES,

Original: PROTEINS, and DHFR Morris et al.
Revised: PROTEINS, and DHFR Morris et al.

Original: that allow multi-task settings (graph classification (GC), node classification (NC), and link prediction (LP)).
Revised: that allow multi-task settings (graph classification (GC), node classification (NC), and link prediction (LP)).

Original: We consider the baselines: 1) Single Task: a single task baseline for the handcrafted model.
Revised: We consider the baselines: 1) Single Task: a single task baseline for the handcrafted model.

Original: It trains each task using the different backbone (GCN, SAGE, GAT, and GIN) and a task-specific layer; 2) Random Search: a simple baseline in
Revised: It trains each task using the different backbone (GCN, SAGE, GAT, and GIN) and a task-specific layer; 2) Random Search: a simple baseline in

Original: NAS.
Revised: NAS.

Original: It randomly samples architectures from the discrete search space; 3) ST-SANE: an advanced single task baseline in Graph NAS; 4) Handcrafted methods with  LS, EPO, and COSMOS: the stateof-the-art handcrafted MTGNN Buffelli & Vandin (2020) with different MOO strategies.
Revised: It randomly samples architectures from the discrete search space; 3) ST-SANE: an advanced single task baseline in Graph NAS; 4) Handcrafted methods with MGDA, LS, EPO, and COSMOS: the state-of-the-art handcrafted MTGNN Buffelli & Vandin (2022) with different MOO strategies.

Original: We use the same architecture of task-specific layers for all multi-task baselines to ensure a faircomparison.
Revised: We use the same architecture of task-specific layers for all multi-task baselines to ensure a fair

Original: For LS and EPO, a uniform preference vector r = (0 . 333 , 0 . 333 , 0 . 333) is given.
Revised: For LS and EPO, a uniform preference vector r = (0 . 333 , 0 . 333 , 0 . 333) is given.

Original: For COSMOS, the model is inferred 25 times with 25 equally distributed r to obtain the optimal performance on each task at inference time
Revised: For COSMOS, the model is inferred 25 times with 25 equally distributed r to obtain the optimal performance on each task at inference time

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: comparison.

Original: Complete experimental settings and more experimental results are presented in the Appendix 3 and 
Revised: Complete experimental settings and more experimental results are presented in the Appendix 3 and Appendix

Original: Appendix 4.
Revised: [Table]

Original: Comparison with Baselines We demonstrate the overall multi-task performance of all the comparison methods in terms of accuracy (%) for NC and GC and AUC (%) for LP over five independent runs as shown in Table 2.
Revised: Comparison with Baselines We demonstrate the overall multi-task performance of all the comparison methods in terms of accuracy (%) for NC and GC and AUC (%) for LP over five independent runs as shown in Table 2.

Original: The following five observations are presented: 1) DMTGASEPO and DMTGAS-COSMOS obtain higher values than the Handcrafted-EPO and HandcraftedCOSMOS, respectively, which illustrates our proposed DMTGAS can obtain a MTGNN model with stronger graph representation ability.
Revised: The following five observations are presented: 1) DMTGASEPO and DMTGAS-COSMOS obtain higher values than the Handcrafted-EPO and HandcraftedCOSMOS, respectively, which illustrates our proposed DMTGAS can obtain a MTGNN model with stronger graph representation ability.

Original: 2) Since tasks conflict with each other, none of the handcrafted
Revised: 2) Since tasks conflict with each other, none of the handcrafted

Original: MTGNN methods with different optimization strategies achieves vast superiority compared to the single-task baseline.
Revised: MTGNN methods with different optimization strategies achieves vast superiority compared to the single-task baseline.

Original: However, the accuracy of models obtained by DMTGAS-EPO and DMTGASCOSMOS exceeds that of the single-task baseline in most cases, thanks to the real-time trade-off among tasks during the alternate optimization process and knowledge sharing.
Revised: However, the accuracy of models obtained by DMTGAS-EPO and DMTGASCOSMOS exceeds that of the single-task baseline in most cases, thanks to the real-time trade-off among tasks during the alternate optimization process and knowledge sharing.

Original: Furthermore, it is worth noting that DMTGAS solves all tasks at once, whereas the single-task baseline is individually tailored for each model, requiring more computational cost (see the Appendix 4 for a more detailed explanation).
Revised: Furthermore, it is worth noting that DMTGAS solves all tasks at once, whereas the single-task baseline is individually tailored for each model, requiring more computational cost (see the Appendix 4 for a more detailed explanation).

Original: Compared to LS (simply weighting for all task losses), EPO and COSMOS (both handcrafted methods and DMTGAS) achieve better performance on all tasks by finding the search direction in which all task losses decrease, indicating that our proposal can better handle conflict among tasks.
Revised: Compared to LS (simply weighting for all task losses), EPO and COSMOS (both handcrafted methods and DMTGAS) achieve better performance on all tasks by finding the search direction in which all task losses decrease, indicating that our proposal can better handle conflict among tasks.

Original: 4) Since LS cannot guarantee to find a Pareto optimal solution with a given r Ruchte & Grabocka (2021), in the iterative optimization of DMTGAS-LS, it isn’t easy to maintain the architecture and parameters with consistent preferences.
Revised: 4) Since LS cannot guarantee to find a Pareto optimal solution with a given r Ruchte & Grabocka (2021), in the iterative optimization of DMTGAS-LS, it isn’t easy to maintain the architecture and parameters with consistent preferences.

Original: This phenomenon makes DMTGAS-LS unable to outperform Handcrafted-LS comprehensively.
Revised: This phenomenon makes DMTGAS-LS unable to outperform Handcrafted-LS comprehensively.

Original: Our method outperforms random search on all tasks, which indicates that differentiable search methods in the continuous search space are more likely to find excellent solutions.
Revised: Our method outperforms random search on all tasks, which indicates that differentiable search methods in the continuous search space are more likely to find excellent solutions.

Original: In addition, Table 3 report the Hypervolumeindicator (HV) values(a common metric for evaluating Pareto fronts Zitzler & Thiele (1999); Tian et al.
Revised: In addition, Table 3 reports the Hypervolume (HV) values Tian et al.

Original: (2017)) obtained by Handcrafted-COSMOS and DMTGAS-COSMOS  over 5 independent runs respectively.
Revised: (2017) obtained by Handcrafted-COSMOS and DMTGAS-COSMOS to solve GC, NC, and LP over 5 independent runs.

Original: The HV is calculated with reference point (2.5, 2.5, 2.5) and 25 equally distributed r .
Revised: The HV is calculated with reference point (2.5, 2.5, 2.5) and 25 equally distributed r .

Original: Experimental results show that DMTGASCOSMOS outperforms Handcrafted-COSMOS in terms of HV, i.e. our proposal can generate a higher quality Pareto front.
Revised: Experimental results show that DMTGAS-COSMOS outperforms Handcrafted-COSMOS in terms of HV, i.e. our proposal can generate a higher quality Pareto front.

Original: The convergence curves of HV on all datasets are further reported and analysed in the Appendix 4.
Revised: The convergence curves of HV on all datasets are further reported and analysed in the Appendix 4.

Original: Searched Architectures We visualize the architectures searched by DMTGAS-EPO and
Revised: Searched Architectures We visualize the architectures searched by DMTGAS-EPO and

Original: DMTGAS-COSMOS on different datasets as shown in Figure 3 and the Appendix 4.
Revised: DMTGAS-COSMOS on different datasets as shown in Figure 3 and the Appendix 4.

Original: It can be seen that these architectures are data-dependent, whose node aggregation, skip connections, and layer aggregation are significantly different.
Revised: It can be seen that these architectures are data-dependent, whose node aggregation, skip connections, and layer aggregation are significantly different.

Original: This phenomenon further shows the importance of automatically searching for data-dependent model architectures.h i [Figure]
Revised: This phenomenon further shows the importance of automatically searching for data-dependent model architectures.

Original: h
Revised: h

Revised: ih i

Original: [Figure]
Revised: [Figure]

Original: h
Revised: h

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: h
Revised: h

Revised: i

Original: [Figure]
Revised: [Figure]

Original: SK l ( h 
Revised: SK l ( h i 2 )

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: GIN
Revised: GIN

Original: MAX
Revised: MAX

Original: [Figure]
Revised: [Figure]

Original: z
Revised: z

Original: [Figure]
Revised: [Figure]

Revised: Task

Revised: [Figure]

Revised: Task

Original:  MAX
Revised: [Figure] Task [Figure] MAX

Original: IDENTIFY
Revised: IDENTIFY

Original: [Figure]
Revised: [Figure]

Original: h 
Revised: h ih ih

Original: i
Revised: i

Original: [Figure] h
Revised: 3 h

Original: ih i
Revised: i

Original: [Figure] h
Revised: SK l ( h i 1 , h i 2 , h

Original: i
Revised: i

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: SK l ( h i
Revised: SUM

Original: [Figure]
Revised: [Figure]

Original: h
Revised: SAGE z i Task

Original: i
Revised: [Figure] Task [Figure] Task

Original: [Figure]
Revised: [Figure]

Original: h
Revised: CONCAT IDENTIFY IDENTIFY IDENTIFY

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: z i
Revised: h i

Original: [Figure]
Revised: [Figure]

Original: IDENTIFY

Original:  [Figure] 
Revised: h i [Figure] h

Revised: i

Revised: [Figure]

Revised: h

Original: IDENTIFY
Revised: i [Figure] SK l ( h i 1 , h i

Original: [Figure]
Revised: [Figure]

Original: h i [Figure] h
Revised: h i 3 )

Original: [Figure]
Revised: [Figure]

Original: h

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: h
Revised: IDENTIFY

Original: [Figure]
Revised: [Figure]

Original: l ( h

Original: [Figure]
Revised: [Figure]

Original: h [Figure] [Figure] b are the tasks for training the multi-task model, and c is the unseen task.
Revised: model, and c is the unseen task.

Original: The preference r is set to (0 . 5 , 0 . 5) .
Revised: The preference r is set to (0 . 5 , 0 . 5) .

Original: Transferability We employ the node embeddings obtained by DMTGAS-EPO and HandcraftedEPO on any two tasks to train a task-specific network on unseen tasks to demonstrate the transferability of our method.
Revised: Transferability We employ the node embeddings obtained by DMTGAS-EPO and HandcraftedEPO on any two tasks to train a task-specific network on unseen tasks to demonstrate the transferability of our method.

Original: Figure 4 shows the obtained accuracy on all datasets.
Revised: Figure 4 shows the obtained accuracy on all datasets.

Original: DMTGAS-EPO achieves the highest accuracy on all unseen tasks, which shows that our proposal can get higher quality node embedding information than handcrafted methods.
Revised: DMTGAS-EPO achieves the highest accuracy on all unseen tasks, which shows that our proposal can get higher quality node embedding information than handcrafted methods.

Original: [Table]
Revised: [Table]

Original: Ablation Study DMTGAS need to maintain preference-consistency of architecture and weights.
Revised: Ablation Study DMTGAS need to maintain preference-consistency of architecture and weights.

Original: In this section, we construct a baseline with inconsistent trade-offs during optimization process, called
Revised: In this section, we construct a baseline with inconsistent trade-offs during optimization process, called

Original: DMTGASIC.
Revised: DMTGASIC.

Original: In each iteration, for
Revised: In each iteration, for

Original: EPO, a fixed preference r = (0 . 333 , 0 . 333 , 0 . 333) is provided to the outer optimization, and a randomly sampled preference vectors r ∼ P r is provided to the inner optimization.
Revised: EPO, a fixed preference r = (0 . 333 , 0 . 333 , 0 . 333) is provided to the outer optimization, and a randomly sampled preference vectors r ∼ P r is provided to the inner optimization.

Original: For COSMOS, two randomly sampled preference vectors r 1 and r 2 from the distribution
Revised: For COSMOS, two randomly sampled preference vectors r 1 and r 2 from the distribution

Original: P r are provided to inner and outer optimizations, respectively.
Revised: P r are provided to inner and outer optimizations, respectively.

Original: The experimental results on the ENZYMES dataset are shown in Table 4.
Revised: The experimental results on the ENZYMES dataset are shown in Table 4.

Original: It can be seen that DMTGAS outperforms DMTGASIC on all tasks, which shows the importance of maintaining preference-consistency of ULP and LLP.
Revised: It can be seen that DMTGAS outperforms DMTGASIC on all tasks, which shows the importance of maintaining preference-consistency of UL-MOP and LL-MOP.

Original: We also observed this phenomenon on other datasets (see the Appendix 4 in detail).
Revised: We also observed this phenomenon on other datasets (see the Appendix 4 in detail).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we firstly introduce a differentiable multi-task GNN architecture search framework based on bi-level multi-objective optimization for automatic multi-task graph representation learning, called DMTGAS.
Revised: We firstly introduce a differentiable multi-task GNN architecture search framework based on BLMOP for automatic multi-task graph representation learning, called DMTGAS.

Original: DMTGAS can provide a multi-taskmodel’s architecture and its weights, which match user-defined task trade-offs.
Revised: DMTGAS can provide a multi-task architecture and its weights, which match user-defined task trade-offs.

Original: Experimental results onthree real-world datasets confirm that our proposal can generate novel multi-task graph models that outperform current hand-designed models.
Revised: Experimental results on real-world datasets confirm that our proposal can generate novel multi-task GNN models that outperform current hand-designed models.

Original: In the future, we will  apply DMTGAS to more graph learning problems and further analyze a non-asymptotic convergence rate of DMTGAS, which is related to the convergence rate of the used preference-based MOO methods and Pareto-front learning methods.
Revised: In the future, we will further analyze relationships across downstream tasks and apply DMTGAS to more graph learning problems, such as self-supervised graph learningJin et al. ; Ju et al. A non-asymptotic convergence rate of DMTGAS is also worth exploring, which is related to the convergence rate of the used preferencebased MOO methods and Pareto-front learning methods.

Original: A A PPENDIX A. D ETAILED R ELATED W ORK
Revised: A A PPENDIX A. D ETAILED R ELATED W ORK

Original: Multi-Task Graph Representation Learning Most works on multi-task graph representation learning have focused on learning an embedding model for several applications on graphs.
Revised: Multi-Task Graph Representation Learning Most works on multi-task graph representation learning have focused on learning an embedding model for several applications on graphs.

Original: Avelar et al.
Revised: Avelar et al.

Original: Avelar et al.
Revised: Avelar et al.

Original: (2019) proposed a unified multi-task graph neural network that learns multiple graph centrality measures.
Revised: (2019) proposed a unified multi-task graph neural network that learns multiple graph centrality measures.

Original: Holtz et al. Holtz et al.
Revised: Holtz et al. Holtz et al.

Original: (2019) developed a supervised learning technique for two-level labels using a multi-task architecture and validated the performance of their framework on PROTEIN and ENZYME datasets.
Revised: (2019) developed a supervised learning technique for two-level labels using a multi-task architecture and validated the performance of their framework on PROTEIN and ENZYME datasets.

Original: Li et al.
Revised: Li et al.

Original: Li & Ji (2019) proposed a syntax-aware multi-task graph convolutional network for biomedical relation extraction, which simultaneously performs relation recognition and classification tasks, achieving state-of-the-art performance on the drug-drug interaction extraction task.
Revised: Li & Ji (2019) proposed a syntax-aware multi-task graph convolutional network for biomedical relation extraction, which simultaneously performs relation recognition and classification tasks, achieving state-of-the-art performance on the drug-drug interaction extraction task.

Original: Montanari et al. Montanari et al.
Revised: Montanari et al. Montanari et al.

Original: (2019) employed a multi-task graph convolutional network to model physical-chemical ADMET endpoints.
Revised: (2019) employed a multi-task graph convolutional network to model physical-chemical ADMET endpoints.

Original: Wang et al. Wang et al.
Revised: Wang et al. Wang et al.

Original: (2020) presented a multi-task network embedding method, where one of the tasks aims to maintain high-order proximity between paired nodes throughout the network.
Revised: (2020) presented a multi-task network embedding method, where one of the tasks aims to maintain high-order proximity between paired nodes throughout the network.

Original: Another task is to maintain low-order proximity in the one-hop region of each node.
Revised: Another task is to maintain low-order proximity in the one-hop region of each node.

Original: Xie et al. Xie et al.
Revised: Xie et al. Xie et al.

Original: (2020a) developed a multi-task representation learning architecture for enhanced graph classification, which leverages the knowledge of the node classification task to better enhance the performance of the graph classification task.
Revised: (2020a) developed a multi-task representation learning architecture for enhanced graph classification, which leverages the knowledge of the node classification task to better enhance the performance of the graph classification task.

Original: They further proposed multi-task network representation learning for handling node classification and link prediction tasks Xie et al.
Revised: They further proposed multi-task network representation learning for handling node classification and link prediction tasks Xie et al.

Original: (2020b).
Revised: (2020b).

Original: Buffell et al.
Revised: Buffell et al.

Original: Buffelli & Vandin (2020) designed a meta-learning method for multi-task graph representation learning.
Revised: Buffelli & Vandin (2022) designed a meta-learning method for multi-task graph representation learning.

Original: However, these application-specific handcrafted methods rely heavily on expert knowledge and ignore task conflicts due to excessive parameter sharing between tasks.
Revised: However, these application-specific handcrafted methods rely heavily on expert knowledge and ignore task conflicts due to excessive parameter sharing between tasks.

Original: Automatic Graph Representation Learning This section mainly focuses on the neural architecture search (NAS) for automatic graph representation learning.
Revised: Automatic Graph Representation Learning This section mainly focuses on the neural architecture search (NAS) for automatic graph representation learning.

Original: The challenges of NAS generally include the design of search space and search strategy.
Revised: The challenges of NAS generally include the design of search space and search strategy.

Original: According to the construction principles of graph neural modules, the search space can be divided into four categories: micro search space, macro search space, pooling methods, and hyperparameters.
Revised: According to the construction principles of graph neural modules, the search space can be divided into four categories: micro search space, macro search space, pooling methods, and hyperparameters.

Original: Search strategies include reinforcement learning, differentiable methods, evolutionary algorithms, and hybrid methods Wang & Zhu (2021).
Revised: Search strategies include reinforcement learning, differentiable methods, evolutionary algorithms, and hybrid methods Wang & Zhu (2021).

Original: Differentiable methods for GNN have gained popularity in recent years, which builds a single supernet that contains all operations Huan et al.
Revised: Differentiable methods for GNN have gained popularity in recent years, which builds a single supernet that contains all operations Huan et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: ; Qin et al. (2021).
Revised: ; Qin et al. (2021).

Original: Each candidate operation is treated as a probability distribution over all possible operations.
Revised: Each candidate operation is treated as a probability distribution over all possible operations.

Original: The objective function of architecture search is differentiable through continuous relaxation of the search space.
Revised: The objective function of architecture search is differentiable through continuous relaxation of the search space.

Original: Thus, gradient-based methods can jointly optimize the architecture and weights.
Revised: Thus, gradient-based methods can jointly optimize the architecture and weights.

Original: Existing works mainly focus on the search space and search strategy to learn a graph representation model customized for a given task Wang & Zhu (2021).
Revised: Existing works mainly focus on the search space and search strategy to learn a graph representation model customized for a given task Wang & Zhu (2021).

Original: However, automatic learning methods for multiple tasks on graphs have not been noticed.
Revised: However, automatic learning methods for multiple tasks on graphs have not been noticed.

Revised: Bi-level Multi-objective Optimization (BL-MOP)

Original: A. P ROOFS OF T HEOREMS IN S ECTION 3.
Revised: Due to the complex interaction mechanism between the upper level (UL) and lower level (LL) and the high computational cost, there are limited works on the BL-MOPsSinha et al. Existing efforts are divided into three categories: classicalShi & Xia (2001); Eichfelder (2010), evolution-basedDeb & Sinha (2010); Sinha (2011); Ruuska & Miettinen (2012); Said et al. ; Cai et al.

Revised: ; Wang et al.

Revised: Deb et al.

Revised: (2022) and gradient-based methodsYe et al.

Revised: Classical methods require some strong mathematical assumptions, which make them incapable of being directly applicable to many practical scenarios.

Revised: Despite good performance on traditional NP-hard problems, evolution-based methods are not directly applicable to high-dimensional BL-MOPs in deep learning due to the expensive training cost.

Revised: In addition, the high computational cost of maintaining two solution sets is unacceptable in practical resource-constrained environments.

Revised: The gradient-based method directly approximates the

Revised: LL-MOP as a single objective problem to guarantee that each architecture corresponds to an optimal weight.

Revised: But this method may lead to inconsistencies in task trade-offs at the upper and lower levels.

Revised: Furthermore, the method’s performance is severely affected by the manually tuned task loss weights.

Revised: However, our proposed schemes guarantee the consistency of task preference at the upper and lower levels and do not need to maintain multiple weights for each candidate architecture in the optimization process, which cleverly avoids the above optimization difficulties.

Revised: A. P ROOFS OF T HEOREMS IN S ECTION 3.

Original: Proof of Theorem 3.1 Let α ∈ A and { α t } Tt =1 is a sequence in A satisfying α t → α , we should prove that L ( w ∗ ( α ) , α, D valid , r )
Revised: Proof of Theorem 3.1 Let α ∈ A and { α t } Tt =1 is a sequence in A satisfying α t → α , we should prove that L ( w ∗ ( α ) , α, D valid , r )

Original: → L ( w ∗ ( α ) , α, D valid , r ) .
Revised: → L ( w ∗ ( α ) , α, D valid , r ) .

Original: Because { w ∗ ( α ) } is uniformly bounded on A ( Assumption (3.1) 3) ), for some w , there is a sub-sequence { w ∗ ( α tk ) } → w .
Revised: Because { w ∗ ( α ) } is uniformly bounded on A ( Assumption (3.1) 3) ), for some w , there is a sub-sequence { w ∗ ( α tk ) } → w .

Original: As α tk → α , w ∗ ( α ) = argmin w ∈W L ( w, α, D train , r ) ( Assumption (3.1) 2) ).
Revised: As α tk → α , w ∗ ( α ) = argmin w ∈W L ( w, α, D train , r ) ( Assumption (3.1) 2) ).

Original: Thus, w ∗ ( α ) = w .
Revised: Thus, w ∗ ( α ) = w .

Original: That is, { w ∗ ( α tk ) } is convergence to a unique cluster point w ∗ ( α ) .
Revised: That is, { w ∗ ( α tk ) } is convergence to a unique cluster point w ∗ ( α ) .

Original: Therefore, as α t → α , w ∗ ( α t ) → w ∗ ( α ) .
Revised: Therefore, as α t → α , w ∗ ( α t ) → w ∗ ( α ) .

Original: Since L ( w ( α ) , α, D valid , r ) is jointly continuous ( Assumption (3.1) 4) ),
Revised: Since L ( w ( α ) , α, D valid , r ) is jointly continuous ( Assumption (3.1) 4) ),

Original: L ( w ∗ ( α t ) , α t , D valid , r ) →
Revised: L ( w ∗ ( α t ) , α t , D valid , r ) →

Original: L ( w ∗ ( α ) , α, D valid , r ) as α t → α .
Revised: L ( w ∗ ( α ) , α, D valid , r ) as α t → α .

Original: In addition, because A is a compact set ( Assumption (3.1) 1) ), the existence of the solution is proved according to the Weierstrass theorem.
Revised: In addition, because A is a compact set ( Assumption (3.1) 1) ), the existence of the solution is proved according to the Weierstrass theorem.

Original: Proof of Theorem 3.2 Let { α t } Tt
Revised: Proof of Theorem 3.2 Let { α t } Tt

Original: =1 is a sequence in A satisfying α t → α .
Revised: =1 is a sequence in A satisfying α t → α .

Original: For every sequence { α t } Tt =1 ,
Revised: For every sequence { α t } Tt =1 ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: According to Theorem 3.1, as α t → α ,
Revised: According to Theorem 3.1, as α t → α ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: In addition, since L ( ∗ , α, D valid , r ) is uniformly Lipschitz continuous ( Assumption (3.2) 1) ), we have
Revised: In addition, since L ( ∗ , α, D valid , r ) is uniformly Lipschitz continuous ( Assumption (3.2) 1) ), we have

Original: [Equation]
Revised: [Equation]

Original: Because { w k ( α ) } Kk =1 converges uniformly to w ∗ ( α ) on A as K → + ∞ ( Assumption (3.2) 2) ),
Revised: Because { w k ( α ) } Kk =1 converges uniformly to w ∗ ( α ) on A as K → + ∞ ( Assumption (3.2) 2) ),

Original: L ( w K ( α ) , α, D valid , r ) continuously converges to L ( w ∗ ( α ) , α, D valid , r ) .
Revised: L ( w K ( α ) , α, D valid , r ) continuously converges to L ( w ∗ ( α ) , α, D valid , r ) .

Original: A. M ORE E XPERIMENTAL D ETAILS
Revised: A. M ORE E XPERIMENTAL D ETAILS

Original: In this section, we provide more experimental details and results in our experimental section.
Revised: In this section, we provide more experimental details and results in our experimental section.

Original: Our experiments are running with PyTorch (1.10.1) and PyTorch Geometric (2.0.3) on a GPU 2080Ti (Memory: 12GB, Cuda version:11.3).
Revised: Our experiments are running with PyTorch (1.10.1) and PyTorch Geometric (2.0.3) on a GPU 2080Ti (Memory: 12GB, Cuda version:11.3).

Original: We implement our proposed DMTGAS 2 based on the source code provided by SANE 3 .
Revised: We implement our proposed DMTGAS 2 based on the source code provided by SANE 3 .

Original: We independently run all automated search methods for 5 times with different random seeds to obtain the best architecture on the validation dataset.
Revised: We independently run all automated search methods for 5 times with different random seeds to obtain the best architecture on the validation dataset.

Original: Then the best architecture is retrained over 5 independent runs.
Revised: Then the best architecture is retrained over 5 independent runs.

Original: The final performance on the test set is reported.
Revised: The final performance on the test set is reported.

Original: In each run, we split the nodes in all graphs into 70%, 10%, 20% for training, validation, and test.
Revised: In each run, we split the nodes in all graphs into 70%, 10%, 20% for training, validation, and test.

Original: Datasets We employ three popular read-world datasets (ENZYMES, PROTEINS, and DHFR) with graph labels, node attributes, and node labels from the TUDataset library Morris et al.
Revised: Datasets We employ three popular read-world datasets (ENZYMES, PROTEINS, and DHFR) with graph labels, node attributes, and node labels from the TUDataset library Morris et al.

Original: These datasets satisfy multi-task conditions (node classification, graph classification, and link prediction) execution.
Revised: These datasets satisfy multi-task conditions (node classification, graph classification, and link prediction) execution.

Revised: For the node classification, only a small fraction of nodes are labeled.

Revised: For the link prediction, in each graph, some randomly removed edges and unremoved edges are treated as positive examples.

Revised: Couples of non-adjacent nodes are randomly sampled as negative examples.

Original:  Task-Specific Layer We employ the same task-specific layers as state-of-the-art hand-designed multi-task graph models Buffelli & Vandin (2020).
Revised: More implementation details can be found in Buffelli & Vandin (2022). Task-Specific Layer We employ the same task-specific layers as state-of-the-art hand-designed multi-task graph models Buffelli & Vandin (2022).

Original: The task-specific layer is composed of three heads.
Revised: The task-specific layer is composed of three heads.

Original: For node classification head, a single-layer neural network with a Softmax activation is employed, which maps embeddings to class predictions.
Revised: For node classification head, a single-layer neural network with a Softmax activation is employed, which maps embeddings to class predictions.

Original: For graph classification head, two-layer neural network is used to map embeddings to class predictions.
Revised: For graph classification head, two-layer neural network is used to map embeddings to class predictions.

Original: The first one is a linear transformation with a ReLU activation.
Revised: The first one is a linear transformation with a ReLU activation.

Original: And then its output is averaged.
Revised: And then its output is averaged.

Original: The second one is a simple neural network layer with a Softmax activation.
Revised: The second one is a simple neural network layer with a Softmax activation.

Original: For link prediction head, two-layer neural network is used to map embeddings to the probability of a link between nodes.
Revised: For link prediction head, two-layer neural network is used to map embeddings to the probability of a link between nodes.

Original: The first one is a linear transformation with a
Revised: The first one is a linear transformation with a

Original: ReLU activation.
Revised: ReLU activation.

Original: The second one is a single-layer neural network whose input is the concatenation of two node embeddings and whose output is the probability of the link between them.
Revised: The second one is a single-layer neural network whose input is the concatenation of two node embeddings and whose output is the probability of the link between them.

Original: Baseline We compare DMTGAS with three groups of state-of-the-art methods: single-task baseline, human-designed MTGNN, and auto-designed NAS methods.
Revised: Baseline To verify the efficacy of DMTGAS, we provide four types of baselines for experiments with respect to two levels - MTL and Graph NAS: single-task handcrafted baselines (single-level single-objective optimization), single-task Graph NAS baselines (bi-level single-objective optimization), multi-task handcrafted baselines (single-level multi-objective optimization), and multi-task Graph NAS baselines (bi-level multi-objective optimization).

Original: 1) Single Task .
Revised: 1) Single-task Handcrafted Baselines .

Original: A single task baseline trains each task independently by the advanced backbone (GCN, SAGE, GAT, and GIN) and a task-specific layer.
Revised: Single task handcrafted baselines (ST-GCN, STSAGE, ST-GAT, and ST-GIN) train each task independently by the advanced backbone (GCN, SAGE, GAT, and GIN) and a task-specific layer.

Original: In addition, an advanced Graph NAS method (ST-SANE) with a task-specific layer is employed as a single task  NAS baseline. 
Revised: Single-task Graph NAS Baselines . An advanced Graph NAS method (ST-SANE) with a task-specific layer is employed as a single task Graph NAS baseline. 3) Multi-task Handcrafted Baselines . We adopt

Original: 2) Human-Designed MTGNN.
Revised: the state-of-the-art handcrafted MTGNN

Original: We adopt the state-of-the-art handcrafted MTGNN Buffelli & Vandin (2020) with three MOO strategies, including linear scalarization (LS),  exact Pareto optimal search (EPO),  conditioned one-shot multi-objective search (COSMOS).
Revised: architecture Buffelli & Vandin (2022), and optimize it with four MOO strategies, including linear scalarization (LS), multiple-gradient descent algorithm (MGDA), exact Pareto optimal search (EPO), and conditioned one-shot multi-objective search (COSMOS).

Revised: These multi-task handcrafted baselines are called Handcrafted LS, Handcrafted MGDA, Handcrafted EPO, and Handcrafted COSMOS.

Original: This handcrafted architecture is an encoder-decoder model.
Revised: The handcrafted architecture is an encoderdecoder model.

Original: The encoder is the same as the single task GCN baseline, and the decoder is as described in the previous section.
Revised: The encoder is the same as the single task GCN baseline, and the decoder is as described in the previous section.

Original: 3) Auto-Designed NAS Methods .
Revised: 4) Multi-task Graph NAS Baselines .

Original: Random search is a simple baseline in  NAS, which randomly samples architectures from the discrete search space.
Revised: Random search is a simple baseline in Graph NAS, which randomly samples architectures from the discrete search space.

Original: And LS is used as the optimization strategy for each sampled architecture.
Revised: And LS is used as the optimization strategy for each sampled architecture.

Original: 2 https://github. .https: SANE .
Revised: In addition, LS is also embedded in DMTGAS to obtain DMTGAS-LS as a multi-task graph NAS baseline.

Original: Implementation Details
Revised: Implementation Details

Original: In this part, we give more implementation details of all compared methods.
Revised: In this part, we give more implementation details of all compared methods.

Original: • For all baselines, we set the batch size to 256 and train all models for 1000 epochs using Adam with a learning rate of 0 . 001 .
Revised: • For all baselines, we set the batch size to 256 and train all models for 1000 epochs using Adam with a learning rate of 0 . 001 .

Original: • For the single task baseline and human-designed MTGNN, node embeddings are normalized to the unit norm between GCN layers for better performance.
Revised: • For the single task baseline and human-designed MTGNN, node embeddings are normalized to the unit norm between GCN layers for better performance.

Original: • For the random search, the number of samples is set to 200.
Revised: • For the random search, the number of samples is set to 200.

Original: And we employ grid search to tune the hyper-parameters.
Revised: And we employ grid search to tune the hyper-parameters.

Original: • For methods with EPO and LS, we set a uniform preference vector r = (0 . 33 , 0 .
Revised: • For methods with EPO and LS, we set a uniform preference vector r = (0 . 33 , 0 .

Original: • For methods with COSMOS, the preference vector r is sampled from a Dirichlet distribution controlled with α .
Revised: • For methods with COSMOS, the preference vector r is sampled from a Dirichlet distribution controlled with α .

Original: We use grid search to tune the penalty parameter τ ( σ ) and the Dirichlet sampling parameter α with the search space [0 . 001 , 0 . 01 , 2 , 3 , 8] and [0 . 1 , 0 . 5 , 1 , 5 , 10] , respectively.
Revised: We use grid search to tune the penalty parameter τ ( σ ) and the Dirichlet sampling parameter α with the search space [0 . 001 , 0 . 01 , 2 , 3 , 8] and [0 . 1 , 0 . 5 , 1 , 5 , 10] , respectively.

Original: We run the model over 25 uniformly distributed preference vectors to obtain a Pareto front in the inference phase.
Revised: We run the model over 25 uniformly distributed preference vectors to obtain a Pareto front in the inference phase.

Original: • For the DMTGAS, both the architecture optimizer and the weight optimizer are Adam with a learning rate of 0 .
Revised: • For the DMTGAS, both the architecture optimizer and the weight optimizer are Adam with a learning rate of 0 .

Original: In this search phase, we set the hidden layer to 64 for the sake of computational resources.
Revised: In this search phase, we set the hidden layer to 64 for the sake of computational resources.

Original: Metric Hypervolume (HV) is a metric commonly used as a performance indicator of multiobjective algorithms and is calculated as the size of the dominated space.
Revised: Metric Hypervolume (HV) is a metric commonly used as a performance indicator of multiobjective algorithms and is calculated as the size of the dominated spaceZitzler & Thiele (1999);

Revised: Tian et al.

Original: It represents the volume of the hypercube enclosed by the individuals in the solution set and the reference points in the objective space.
Revised: It represents the volume of the hypercube enclosed by the individuals in the solution set and the reference points in the objective space.

Original: Let S = { q 1 , q 2 , ..., q n } be the non-dominated set, and
Revised: Let S = { q 1 , q 2 , ..., q n } be the nondominated set, and

Original: Z = { z 1 , z 2 , ..., z n } as the reference point, the HV is expressed as:
Revised: Z = { z 1 , z 2 , ..., z n } as the reference point, the HV is expressed as:

Original: [Equation]
Revised: [Figure]

Original: where (cid:81) ni =1 [ q i , z i ] denotes the region of the non-dominated points bounded by Z , and V OL is the

Original: Euclidean volume.

Original: The accuracy of the HV metric depends on the choice of reference points, and different choices of reference points will yield different results when evaluating the same solution set.

Original: In general, a solution with a larger HV is considered to be of better quality.

Original: A. M ORE E XPERIMENTAL R ESULTS

Original: The Hypervolume indicator (HV) is a common metric for evaluating Pareto fronts Zitzler &

Original: Thiele (1999); Tian et al.

Original: Figure 5 reports convergence curves of HV values obtained by

Original: Handcrafted-COSMOS and DMTGAS-COSMOS on all datasets over five independent runs, respectively.

Original: As can be seen from Figure 5, our searched architectures converge more easily than manually designed ones, which further illustrates the advantages of automatically designing architectures.

Original: Figure 6 shows the visualization of the architectures searched by DMTGAS-EPO and DMTGASCOSMOS on all datasets.

Original: Table 5 lists the experimental results obtained by DMTGASIC and DMTGAS on all datasets in terms of accuracy (%) for NC and GC and AUC (%) for LP.

Original: [Table] [Figure] [Figure]
Revised: [Figure]

Original: MAX
Revised: MAX

Original: IDENTIFY IDENTIFY (e) PROTEINS h
Revised: IDENTIFY IDENTIFY (e) PROTEINS h

Original: ih ih
Revised: ih ih

Original: i
Revised: i

Original: 3 h
Revised: 3 h

Original: i
Revised: i

Original: SK l ( h i 1 , h
Revised: SK l ( h i 1 , h

Original: i 2 , h i 3 )
Revised: i 2 , h i 3 )

Original: SUM-SAGE
Revised: SUM-SAGE

Original: GIN
Revised: GIN

Original: SUM-SAGE z i
Revised: SUM-SAGE z i

Original: Task 1 Task 2 Task
Revised: Task 1 Task 2 Task

Original: MAX
Revised: MAX

Original: IDENTIFY IDENTIFY
Revised: IDENTIFY IDENTIFY

Original: IDENTIFY (f) DHFR
Revised: IDENTIFY (f) DHFR

Revised: [Equation]

Revised: where (cid:81) ni =1 [ q i , z i ] denotes the region of the non-dominated points bounded by Z , and V OL is the

Revised: Euclidean volume.

Revised: The accuracy of the HV metric depends on the choice of reference points, and different choices of reference points will yield different results when evaluating the same solution set.

Revised: In general, a solution with a larger HV is considered to be of better quality.

Original: Table 6 lists the search times (clock time in seconds) of  all NAS baselines.
Revised: Figure 5 reports convergence curves of HV values obtained by Handcrafted-COSMOS and DMTGAS-COSMOS on all datasets over five independent runs, respectively. As can be seen from

Revised: Figure 5, our searched architectures converge more easily than manually designed ones, which further illustrates the advantages of automatically designing architectures.

Revised: Figure 6 shows the visualization of the architectures searched by DMTGAS-EPO and DMTGASCOSMOS on all datasets.

Revised: Table 5 lists the experimental results obtained by DMTGASIC and DMTGAS on all datasets in terms of accuracy (%) for NC and GC and AUC (%) for LP.

Revised: Table 6 lists the search times (clock time in seconds) of all NAS baselines.

Original: The search times of our proposed method are shorter than those of NAS baselines, which further demonstrates the superiority of DMTGAS in terms of search efficiency.
Revised: The search times of our proposed method are shorter than those of NAS baselines, which further demonstrates the superiority of DMTGAS in terms of search efficiency.

Original: Compared with linear scalarization, the MOO method searches for better-performing architectures (see Table 2) in a similar search time, which shows the efficiency of applying MOO.
Revised: Compared with linear scalarization, the MOO method searches for better-performing architectures (see Table 2) in a similar search time, which shows the efficiency of applying MOO.

Original: This is due to the fact that both the upper and lower levels only need to maintain one solution  instead of Pareto sets in our proposed optimization strategy. 
Revised: Both the upper and lower levels only need to maintain one solution [Table] instead of Pareto sets in our proposed optimization strategy. Furthermore

Original: Furthermore, our proposed method obtains one model that handles multiple tasks in a single run, whereas ST-SANE requires a model that is individually tailored for each task.
Revised: [Table] our proposed method obtains one model that handles multiple tasks in a single run [Table] whereas ST-SANE requires a model that is individually tailored for each task.

Original: Therefore, the search times of our method are much shorter than those of ST-SANE, indicating the necessity of multi-task graph representation learning.ENZYMES
Revised: Therefore, the search times of our method are much shorter than those of ST-SANE, indicating the necessity of multi-task graph representation learning.

Original: [Table]
Revised: [Table]

Original: DMTGAS-COSMOS 1, Table 7 lists experimental results of DMTGAS-EPO with different non-uniform preferences r including (0.80,0.10,0.10), (0.10,0.80,0.10), (0.10,0.10,0.80), and (0.33,0.33,0.33) in terms of accuracy (%) for NC and GC and AUC (%) for LP.
Revised: Table 7 lists experimental results of DMTGAS-EPO with different non-uniform preferences r including (0.80,0.10,0.10), (0.10,0.80,0.10), (0.10,0.10,0.80), and (0.33,0.33,0.33) in terms of accuracy (%) for NC and GC and AUC (%) for LP.

Original: By introducing different preferences r , DMTGASEPO can achieve excellent performance on a certain task that cannot be achieved by single-task baselines (see Table 2 and Table 7).
Revised: By introducing different preferences r , DMTGASEPO can achieve excellent performance on a certain task that cannot be achieved by single-task baselines (see Table 2 and Table 7).

Original: This further suggests that the MTGNN may have stronger representational capabilities than the single-task GNN through knowledge transfer.
Revised: This further suggests that the MTGNN may have stronger representational capabilities than the single-task GNN through knowledge transfer.

Original: [Table]
Revised: [Table]