Original: G RAPH R EPRESENTATION L EARNING FOR M ULTIT ASK S ETTINGS : A M ETA -L EARNING A PPROACH
Revised: G RAPH R EPRESENTATION L EARNING FOR M ULTIT ASK S ETTINGS : A M ETA -L EARNING A PPROACH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph Neural Networks (GNNs) have become the state-of-the-art method for many applications on graph structured data.
Revised: Graph Neural Networks (GNNs) have become the state-of-the-art method for many applications on graph structured data.

Original: GNNs are a framework for graph representation learning , where a model learns to generate low dimensional node embeddings that encapsulate structural and feature-related information.
Revised: GNNs are a framework for graph representation learning , where a model learns to generate low dimensional node embeddings that encapsulate structural and feature-related information.

Original: GNNs are usually trained in an end-to-end fashion, leading to highly specialized node embeddings.
Revised: GNNs are usually trained in an end-to-end fashion, leading to highly specialized node embeddings.

Original: While this approach achieves great results in the single-task setting, generating node embeddings that can be used to perform multiple tasks (with performance comparable to single-task models) is an open problem.
Revised: While this approach achieves great results in the single-task setting, generating node embeddings that can be used to perform multiple tasks (with performance comparable to single-task models) is an open problem.

Original: We propose a novel representation learning strategy, based on meta-learning, capable of producing multi-task node embeddings.
Revised: We propose a novel representation learning strategy, based on meta-learning, capable of producing multi-task node embeddings.

Original: Our method avoids the difﬁculties arising when learning to perform multiple tasks concurrently by, instead, learning to quickly (i.e. with a few steps of gradient descent) adapt to multiple tasks singularly .
Revised: Our method avoids the difﬁculties arising when learning to perform multiple tasks concurrently by, instead, learning to quickly (i.e. with a few steps of gradient descent) adapt to multiple tasks singularly .

Original: We show that the embeddings produced by our method can be used to perform multiple tasks with comparable or higher performance than both single-task and multitask end-to-end models.
Revised: We show that the embeddings produced by our method can be used to perform multiple tasks with comparable or higher performance than both single-task and multitask end-to-end models.

Original: Our method is model-agnostic and task-agnostic and can hence be applied to a wide variety of multi-task domains.
Revised: Our method is model-agnostic and task-agnostic and can hence be applied to a wide variety of multi-task domains.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: Graph Neural Networks (GNNs) are deep learning models that operate on graph structured data, and have become one of the main topics of the deep learning research community.
Revised: Graph Neural Networks (GNNs) are deep learning models that operate on graph structured data, and have become one of the main topics of the deep learning research community.

Original: Part of their success is given by great empirical performance on many graph-related tasks.
Revised: Part of their success is given by great empirical performance on many graph-related tasks.

Original: Three tasks in particular, with many practical applications, have received the most attention: graph classiﬁcation, node classiﬁcation, and link prediction.
Revised: Three tasks in particular, with many practical applications, have received the most attention: graph classiﬁcation, node classiﬁcation, and link prediction.

Original: GNNs are centered around the concept of node representation learning , and typically follow the same architectural pattern with an encoderdecoder structure (Hamilton et al., 2017; Chami et al., 2020; Wu et al., 2020).
Revised: GNNs are centered around the concept of node representation learning , and typically follow the same architectural pattern with an encoderdecoder structure (Hamilton et al., 2017; Chami et al., 2020; Wu et al., 2020).

Original: The encoder produces node embeddings (low-dimensional vectors capturing relevant structural and feature-related information about each node), while the decoder uses the embeddings to carry out the desired downstream task.
Revised: The encoder produces node embeddings (low-dimensional vectors capturing relevant structural and feature-related information about each node), while the decoder uses the embeddings to carry out the desired downstream task.

Original: The model is then trained in an end-to-end manner, giving rise to highly specialized node embeddings.
Revised: The model is then trained in an end-to-end manner, giving rise to highly specialized node embeddings.

Original: While this can lead to state-of-the-art performance, it also affects the generalization and reusability of the embeddings.
Revised: While this can lead to state-of-the-art performance, it also affects the generalization and reusability of the embeddings.

Original: In fact, taking the encoder from a GNN trained on a given task and using its node embeddings to train a decoder for a different task leads to substantial performance loss, as shown in Figure 1.
Revised: In fact, taking the encoder from a GNN trained on a given task and using its node embeddings to train a decoder for a different task leads to substantial performance loss, as shown in Figure 1.

Original: The low transferability of node embeddings requires the use of one specialized encoder and one specialized decoder for each considered task.
Revised: The low transferability of node embeddings requires the use of one specialized encoder and one specialized decoder for each considered task.

Original: However, many practical machine learning applications operate in resource-constrained environments where being able to share part of the model architecture between tasks is of great importance.
Revised: However, many practical machine learning applications operate in resource-constrained environments where being able to share part of the model architecture between tasks is of great importance.

Original: Furthermore, the training signal from multiple related tasks can lead to higher generalization.
Revised: Furthermore, the training signal from multiple related tasks can lead to higher generalization.

Original: Nevertheless, making sure tasks do not negatively interfere with each other is not trivial (Standley et al., 2020).
Revised: Nevertheless, making sure tasks do not negatively interfere with each other is not trivial (Standley et al., 2020).

Original: The problem of learning models that can perform multiple tasks is known as Multi-Task Learning (MTL), and is an open area of research, attracting many researchers in the deep learning community (Vandenhende et al., 2020).
Revised: The problem of learning models that can perform multiple tasks is known as Multi-Task Learning (MTL), and is an open area of research, attracting many researchers in the deep learning community (Vandenhende et al., 2020).

Original: MTL on graphs has not received much attention, and no single model capable of performing the three most common graph-related tasks has yet been proposed.
Revised: MTL on graphs has not received much attention, and no single model capable of performing the three most common graph-related tasks has yet been proposed.

Original: In fact, we notice that training a multi-head model with the classical procedure, i.e. by performing multiple tasks concurrently on each graph, and updating the parameters with some form of gradient descent to minimize the sum of the single-task losses, can lead to a performance loss with respect to single-task models.
Revised: In fact, we notice that training a multi-head model with the classical procedure, i.e. by performing multiple tasks concurrently on each graph, and updating the parameters with some form of gradient descent to minimize the sum of the single-task losses, can lead to a performance loss with respect to single-task models.

Original: Thus, we propose a novel optimization-based meta-learning (Finn et al., 2017) procedure with a focus on representation learning that can generate node embeddings that generalize across tasks.
Revised: Thus, we propose a novel optimization-based meta-learning (Finn et al., 2017) procedure with a focus on representation learning that can generate node embeddings that generalize across tasks.

Original: Our proposed meta-learning procedure produces task-generalizing node embeddings by not aiming at a setting of the parameters that can perform multiple tasks concurrently (like a classical method would do), or to a setting that allows fast multi-task adaptation (like traditional meta-learning), but to a setting that can easily be adapted to perform each of the tasks singularly .
Revised: Our proposed meta-learning procedure produces task-generalizing node embeddings by not aiming at a setting of the parameters that can perform multiple tasks concurrently (like a classical method would do), or to a setting that allows fast multi-task adaptation (like traditional meta-learning), but to a setting that can easily be adapted to perform each of the tasks singularly .

Original: In fact, our metalearning procedure aims at a setting of the parameters where a few steps of gradient descent on a given task, can lead to good performance on that task, hence removing the burden of directly learning to solve multiple tasks concurrently .
Revised: In fact, our metalearning procedure aims at a setting of the parameters where a few steps of gradient descent on a given task, can lead to good performance on that task, hence removing the burden of directly learning to solve multiple tasks concurrently .

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows:

Original: • We propose a novel method for learning representations that can generalize to multiple tasks.
Revised: • We propose a novel method for learning representations that can generalize to multiple tasks.

Original: We apply it to the challenging setting of graph MTL, and show that a GNN trained with our method produces higher quality node embeddings with respect to classical end-toend training procedures.
Revised: We apply it to the challenging setting of graph MTL, and show that a GNN trained with our method produces higher quality node embeddings with respect to classical end-toend training procedures.

Original: Our method is based on meta-learning and is model-agnostic and task-agnostic , which makes it easily applicable to a wide range of multi-task domains.
Revised: Our method is based on meta-learning and is model-agnostic and task-agnostic , which makes it easily applicable to a wide range of multi-task domains.

Original: • To the best of our knowledge, we are the ﬁrst to propose a GNN model generating a single set of node embeddings that can be used to perform the three most common graph-related tasks (graph classiﬁcation, node classiﬁcation, and link prediction).
Revised: • To the best of our knowledge, we are the ﬁrst to propose a GNN model generating a single set of node embeddings that can be used to perform the three most common graph-related tasks (graph classiﬁcation, node classiﬁcation, and link prediction).

Original: In particular, our embeddings lead to comparable or higher performance with respect to single-task models even when used as input to a simple linear classiﬁer.
Revised: In particular, our embeddings lead to comparable or higher performance with respect to single-task models even when used as input to a simple linear classiﬁer.

Original: • We show that the episodic training strategy at the base of our proposed meta-learning procedure leads to better node embeddings even for models trained on a single task.
Revised: • We show that the episodic training strategy at the base of our proposed meta-learning procedure leads to better node embeddings even for models trained on a single task.

Original: This unexpected ﬁnding provides interesting directions that we believe can be useful to the whole deep representation learning community.
Revised: This unexpected ﬁnding provides interesting directions that we believe can be useful to the whole deep representation learning community.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: GNNs, MTL, and meta-learning are very active areas of research.
Revised: GNNs, MTL, and meta-learning are very active areas of research.

Original: We highlight works that are at the intersections of these subjects, and point the interested reader to comprehensive reviews of each ﬁeld.
Revised: We highlight works that are at the intersections of these subjects, and point the interested reader to comprehensive reviews of each ﬁeld.

Original: To the best of our knowledge there is no work using meta-learning for graph MTL, or proposing a model performing graph classiﬁcation, node classiﬁcation, and link prediction concurrently.
Revised: To the best of our knowledge there is no work using meta-learning for graph MTL, or proposing a GNN performing graph classiﬁcation, node classiﬁcation, and link prediction concurrently .

Original: Graph Neural Networks.
Revised: Graph Neural Networks.

Original: GNNs have a long history (Scarselli et al., 2009), but in the past few years the ﬁeld has grown exponentially.
Revised: GNNs have a long history (Scarselli et al., 2009), but in the past few years the ﬁeld has grown exponentially; we refer the reader to Chami et al. ; Wu et al.

Original: Seminal works include ChebNet Defferrard et al. 
Revised: for a thorough review of the ﬁeld. The ﬁrst popular GNN approaches were based on ﬁlters in the graph spectral domain (Bronstein et al., 2017), and presented many challenges including high computational complexity. Defferrard et al. (2016) introduced ChebNet, which uses Chebyshev polynomials to produce localized and efﬁcient ﬁlters in the graph spectral domain.

Original: Graph Convolutional Networks (Kipf & Welling, 2017), Graph Attention Networks (Veliˇckovi´c et al., 2018), and Graph Isomorphism Networks (Xu et al., 2019).
Revised: Graph Convolutional Networks (Kipf & Welling, 2017) then introduced a localized ﬁrst-order approximation of spectral graph convolutions which was then extended to include attention mechanisms (Veliˇckovi´c et al.

Original: For a thorough review of the ﬁeld we refer the reader to Chami et al.
Revised: Recently

Original: ; Wu et al.
Revised: Xu et al.

Revised: (2019) provides theoretical proofs for the expressivity of GNNs

Original: Multi-Task Learning.
Revised: Multi-Task Learning.

Original: Works at the intersection of MTL and GNNs have mostly focused on multihead architectures forseveral applications including: the prediction of physico-chemical properties (Montanari et al., 2019), node and graph classiﬁcation (Holtz et al., 2019; Xu et al., 2019), learning multiple graph centrality measures (Avelar et al., 2019), and biomedical relation extraction (Li & Ji, 2019).
Revised: Works at the intersection of MTL and GNNs have mostly focused on multihead architectures. These models are all composed of a series of GNN layers followed by multiple heads that perform the desired downstream tasks. In this category, Montanari et al. (2019) propose a model for the prediction of physico-chemical properties.

Revised: Holtz et al.

Revised: and Xie et al.

Revised: (2020) propose multi-task models for concurrently performing node and graph classiﬁcation.

Revised: Finally, Avelar et al.

Revised: (2019) introduce a multi-head GNN for learning multiple graph centrality measures, and Li & Ji (2019) propose a MTL method for the extraction of multiple biomedical relations.

Revised: The work by (Haonan et al., 2019) introduces a model that can be trained for several tasks singularly, hence, unlike the previously mentioned approaches and our proposed method, it can not perform multiple tasks concurrently.

Original: Other works  use GNNs as a tool for MTL: Liu et al.
Revised: There are also some works that use GNNs as a tool for MTL: Liu et al.

Original: (2019b) use GNNs to allow communica- tion between tasks, while Zhang et al.
Revised: (2019b) use GNNs to allow communication between tasks, while Zhang et al.

Original: (2018) use GNNs to estimate the test error of a MTL model.
Revised: (2018) use GNNs to estimate the test error of a MTL model.

Revised: We further mention the work by Wang et al.

Revised: (2020) which considers the task of generating “general” node embeddings, however their method is not based on GNNs, does not consider node attributes (unlike our method), and is not focused on the three most common graph related tasks, which we consider.

Original: For an exhaustive review of deep MTL techniques we refer the reader to Vandenhende et al.
Revised: For an exhaustive review of deep MTL techniques we refer the reader to Vandenhende et al.

Original: Meta-Learning.
Revised: Meta-Learning.

Original: Meta-Learning consists in learning to learn .
Revised: Meta-Learning consists in learning to learn .

Original: Many methods have been proposed (see the review by Hospedales et al.
Revised: Many methods have been proposed (see the review by Hospedales et al.

Original: (2020)), specially in the area of few-shot learning .
Revised: (2020)), specially in the area of few-shot learning .

Original: Combinations of meta-learning and GNNs have been used prevalently in the few-shot learning setting.
Revised: Garcia & Bruna (2018) frame the few-shot learning problem with a partially observed graphical model and use GNNs as an inference algorithm.

Original: Some works use GNNs directly for few-shot learning (Garcia & Bruna, 2018), others as a tool for enhancing meta-learning (Liu et al., 2019a; Suo et al., 2020), and others use meta-learning to train GNNs in few-shot learning scenarios with applications to node classiﬁcation (Zhou et al., 2019; Yao et al., 2020), edge labelling (Kim et al., 2019), link prediction (Alet et al., 2019;
Revised: Liu et al. (2019a) use GNNs to propagate messages between class prototypes and improve existing few-shot learning methods, while Suo et al. (2020) use GNNs to introduce domain-knowledge in the form of graphs. There are also several works that use metalearning to train GNNs in few-shot learning scenarios with applications to node classiﬁcation (Zhou et al., 2019; Yao et al., 2020), edge labelling (Kim et al., 2019), link prediction (Alet et al., 2019;

Original: Bose et al., 2019), and graph regression (Nguyen et al., 2020).
Revised: Bose et al., 2019), and graph regression (Nguyen et al., 2020).

Original: Other works combining meta-learning and GNNs involve adversarial attacks (Z¨ugner & G¨unnemann, 2019) and active learning (Madhawa & Murata, 2020).
Revised: Finally, other combinations of metalearning and GNNs involve adversarial attacks on GNN models (Z ¨ ugner & G ¨ unnemann, 2019) and active learning (Madhawa & Murata, 2020).

Original:  P RELIMINARIES 3. G RAPH N EURAL N ETWORKS
Revised:  P RELIMINARIES 3. G RAPH N EURAL N ETWORKS

Original: Many popular state-of-the-art GNN models follow the message-passing paradigm (Gilmer et al., 2017).
Revised: Many popular state-of-the-art GNN models follow the message-passing paradigm (Gilmer et al., 2017).

Original: Let us represent a graph G =
Revised: Let us represent a graph G =

Original: ( A , X ) with an adjacency matrix A ∈ { 0 , 1 } n × n , and a node feature matrix X ∈ R n × d , where the v -th row X v represents the d dimensional feature vector of node v .
Revised: ( A , X ) with an adjacency matrix A ∈ { 0 , 1 } n × n , and a node feature matrix X ∈ R n × d , where the v -th row X v represents the d dimensional feature vector of node v .

Original: Let
Revised: Let

Original: H ( (cid:96) ) ∈
Revised: H ( (cid:96) ) ∈

Original: R n × d (cid:48) be the matrix containing the node representations at layer (cid:96) .
Revised: R n × d (cid:48) be the matrix containing the node representations at layer (cid:96) .

Original: A message passing layer updates the representation of every node v as follows:
Revised: A message passing layer updates the representation of every node v as follows:

Original: [Equation]
Revised: [Equation]

Original: where H (0) =
Revised: where H (0) =

Original: X , N v is the set of neighbours of node v , AGGREGATE is a permutation invariant function, and UPDATE is usually a neural network.
Revised: X , N v is the set of neighbours of node v , AGGREGATE is a permutation invariant function, and UPDATE is usually a neural network.

Original: After L message-passing layers, the ﬁnal node embeddings H ( L ) are used to perform a given task, and the network is trained end-to-end.
Revised: After L message-passing layers, the ﬁnal node embeddings H ( L ) are used to perform a given task, and the network is trained end-to-end.

Original:  M ODEL -A GNOSTIC M ETA -L EARNING AND ANIL
Revised: 3. M ODEL -A GNOSTIC M ETA -L EARNING AND ANIL

Original: MAML (Model-Agnostic Meta-Learning) is an optimization-based meta-learning strategy proposed by Finn et al.
Revised: MAML (Model-Agnostic Meta-Learning) is an optimization-based meta-learning strategy proposed by Finn et al.

Original: Let f θ be a deep learning model, where θ represents its parameters.
Revised: Let f θ be a deep learning model, where θ represents its parameters.

Original: Let p ( E ) be a distribution over episodes 1 , where an episode E i ∼ p ( E ) is deﬁned as a tuple containing a loss function , a support set , and a target set : E i
Revised: Let p ( E ) be a distribution over episodes 1 , where an episode E i ∼ p ( E ) is deﬁned as a tuple containing a loss function , a support set , and a target set : E i

Original: =
Revised: =

Original: ( L E i ( · ) , S E i , T E i ) , where support and target sets are simply sets of labelled examples.
Revised: ( L E i ( · ) , S E i , T E i ) , where support and target sets are simply sets of labelled examples.

Original: MAML’s goal is to ﬁnd a value of θ that can quickly, i.e. in a few steps of gradient descent, be adapted to new episodes.
Revised: MAML’s goal is to ﬁnd a value of θ that can quickly, i.e. in a few steps of gradient descent, be adapted to new episodes.

Original: This is done with a nested loop optimization procedure: an inner loop adapts the parameters to the support set of an episode by performing some steps of gradient descent, and an outer loop updates the initial parameters aiming at a setting that  allows fast adaptation.
Revised: This is done with a nested loop optimization procedure: an inner loop adapts the parameters to the support set of an episode by performing some steps of gradient descent, and an outer loop updates the initial parameters aiming at a setting that [Figure] allows fast adaptation.

Original: Formally, by deﬁning θ (cid:48)
Revised: Formally, by deﬁning θ (cid:48)

Original: i ( t ) as the parameters after t adaptation steps on the support set of episode E i , we can express the computations in the inner loop as 
Revised: i ( t ) as the parameters after t adaptation steps on the support set of episode E i , we can express the computations in the inner loop as [Equation] [Figure]

Original: [Equation]
Revised: [Equation]

Revised: , S E i ) , with θ (cid:48)

Revised: i (0)

Original:  where L ( f θ (cid:48)
Revised: = θ where L ( f θ (cid:48)

Original: i ( t − 1) , S E i ) indicates the loss over the support set S E i of the model with parameters θ (cid:48)
Revised: i ( t − 1) , S E i ) indicates the loss over the support set S E i of the model with parameters θ (cid:48)

Original: i ( t − 1) , and α is the learning rate.
Revised: i ( t − 1) , and α is the learning rate.

Original: The meta-objective that the outer loop tries to minimize is deﬁned as L meta =
Revised: The meta-objective that the outer loop tries to minimize is deﬁned as L meta =

Original: (cid:80) E i ∼ p ( E ) L E
Revised: (cid:80) E i ∼ p ( E ) L E

Original: i ( f θ [Equation] i ( t ) , T E i ) , which leads to the following parameter update
Revised: i ( f θ (cid:48) i ( t ) , T E i ) , which leads to the following parameter update

Original: [Equation]
Revised: [Equation]

Original: Raghu et al.
Revised: Raghu et al.

Original: (2020) showed that feature reuse is the dominant factor in MAML: in the adaptation loop, only the last layer(s) in the network are updated, while the ﬁrst layer(s) remain almost unchanged.
Revised: (2020) showed that feature reuse is the dominant factor in MAML: in the adaptation loop, only the last layer(s) in the network are updated, while the ﬁrst layer(s) remain almost unchanged.

Original: The authors then propose ANIL (Almost No Inner Loop) where they split the parameters in two sets: one that is used for adaptation in the inner loop, and one that is only updated in the outer loop.
Revised: The authors then propose ANIL (Almost No Inner Loop) where they split the parameters in two sets: one that is used for adaptation in the inner loop, and one that is only updated in the outer loop.

Original: This simpliﬁcation leads to computational improvements while maintaining performance.
Revised: This simpliﬁcation leads to computational improvements while maintaining performance.

Original: [Figure]

Original:  O UR M ETHOD
Revised:  O UR M ETHOD

Original: Our novel representation learning technique, based on meta-learning, is built on three insights: (i) optimization-based meta-learning is implicitly learning robust representations.
Revised: Our novel representation learning technique, based on meta-learning, is built on three insights: (i) optimization-based meta-learning is implicitly learning robust representations.

Original: The ﬁndings by Raghu et al.
Revised: The ﬁndings by Raghu et al.

Original: (2020) suggest that in a model trained with MAML, the ﬁrst layer(s) learn features that are reusable across episodes, while the last layer(s) are set up for fast adaptation.
Revised: (2020) suggest that in a model trained with MAML, the ﬁrst layer(s) learn features that are reusable across episodes, while the last layer(s) are set up for fast adaptation.

Original: MAML is then implicitly focusing on learning reusable representations that generalize across episodes.
Revised: MAML is then implicitly focusing on learning reusable representations that generalize across episodes.

Original: (ii) meta-learning episodes can be designed to encourage generalization. 
Revised: ii) meta-learning episodes can be designed to encourage generalization. If we design support and target set

Revised: to

Revised: mimic

Original: If we design support and target set to mimic the training and validation sets of a classical trainingprocedure, then the meta-learning procedureis effectively optimizing for generalization.
Revised: the training and validation sets of a classical training procedure

Original:  (iii) meta-learning can learn to quickly adapt to multiple tasks singularly , without having to learn to solve multiple tasks concurrently .
Revised: , then the meta -learning procedure is effectively optimizing for generalization. (iii) meta-learning can learn to quickly adapt to multiple tasks singularly , without having to learn to solve multiple tasks concurrently .

Original: We can design the meta-learning procedure so that, for each considered task, the inner loop adapts the parameters to a task-speciﬁc support set, and tests the adaptation on a task-speciﬁc target set.
Revised: We can design the meta-learning procedure so that, for each considered task, the inner loop adapts the parameters to a task-speciﬁc support set, and tests the adaptation on a task-speciﬁc target set.

Original: The outer loop then updates the parameters to allow this fast multiple single-task adaptation .
Revised: The outer loop then updates the parameters to allow this fast multiple single-task adaptation .

Original: This procedure is effectively searching for a parameter setting that can be easily adapted to obtain good single-task performance, without learning to solve multiple tasks concurrently.
Revised: This procedure is effectively searching for a parameter setting that can be easily adapted to obtain good single-task performance, without learning to solve multiple tasks concurrently.

Original: This procedure differs from classical training methods (which aim at solving multiple tasks concurrently), and from traditional meta-learning approaches (which aim at parameters that allow fast multi-task adaptation, inheriting the problems of classical methods) 3 .Based on ii ) and (iii ) , we develop
Revised: This procedure differs from classical training methods (which aim at solving multiple tasks concurrently), and from traditional meta-learning approaches (which aim at parameters that allow fast multi-task adaptation, inheriting the problems of classical methods) 3 .

Original:  a novel meta-learning procedure where the inner loop adapts to multiple tasks singularly , each time 
Revised: Based on (ii ) and (iii ) , we develop a novel meta-learning procedure where the inner loop adapts to multiple tasks singularly , each time with the goal of single-task generalization.

Original: with

Original: the

Original: goal of single-task generalization

Original: Using an encoder

Original: - decoder architecture

Original: (i )

Original:  suggests that this procedure leads to an encoder that learns features that are reusable across episodes.
Revised: Using an encoderdecoder architecture, (i ) suggests that this procedure leads to an encoder that learns features that are reusable across episodes.

Original: Furthermore

Original:  in each episode, the learner is adapting to multiple tasks, hence the encoder is learning features that are general across multiple tasks.
Revised: Furthermore, in each episode, the learner is adapting to multiple tasks, hence the encoder is learning features that are general across multiple tasks.

Original: Intuition.
Revised: Intuition.

Original: Training multi-task models is very challenging (Standley et al., 2020), as some losses may dominate over others, or gradients for different tasks may point in opposite directions.
Revised: Training multi-task models is very challenging (Standley et al., 2020), as some losses may dominate over others, or gradients for different tasks may point in opposite directions.

Original: Some methods have been proposed to counteract this issues (Kendall et al., 2018; Chen et al., 2018), but they are not always effective and it is not clear how one should choose which method to apply (Vandenhende et al., 2020).
Revised: Some methods have been proposed to counteract this issues (Kendall et al., 2018; Chen et al., 2018), but they are not always effective and it is not clear how one should choose which method to apply (Vandenhende et al., 2020).

Original: We design a meta-learning procedure where the learner does not have to ﬁnd a conﬁguration of the parameters that concurrently performs all tasks, but it has to ﬁnd a conﬁguration that can easily be adapted to perform each of the tasks singularly .
Revised: We design a meta-learning procedure where the learner does not have to ﬁnd a conﬁguration of the parameters that concurrently performs all tasks, but it has to ﬁnd a conﬁguration that can easily be adapted to perform each of the tasks singularly .

Original: By then leveraging the implicit/explicit robust representation learning that happens with MAML and ANIL, we can extract an encoder capable of generating node representations that generalize across tasks.
Revised: By then leveraging the implicit/explicit robust representation learning that happens with MAML and ANIL, we can extract an encoder capable of generating node representations that generalize across tasks.

Original: In the rest of this section, we formally present our novel meta-learning procedure for multi-task graph representation learning.
Revised: In the rest of this section, we formally present our novel meta-learning procedure for multi-task graph representation learning.

Original: There are three aspects that we need to deﬁne: (1) Episode Design: how is a an episode composed, (2) Model Architecture Design: what is the architecture of our model, (3) Meta-Training Design: how, and which, parameters are adapted/updated.
Revised: There are three aspects that we need to deﬁne: (1) Episode Design: how is a an episode composed, (2) Model Architecture Design: what is the architecture of our model, (3) Meta-Training Design: how, and which, parameters are adapted/updated.

Original: 4. E PISODE D ESIGN
Revised: 4. E PISODE D ESIGN

Original: In our case, an episode becomes a multi-task episode (Figure 2 (a)).
Revised: In our case, an episode becomes a multi-task episode (Figure 2 (a)).

Original: To formally introduce the concept, let us consider the case where the tasks are graph classiﬁcation (GC), node classiﬁcation (NC), and link prediction (LP).
Revised: To formally introduce the concept, let us consider the case where the tasks are graph classiﬁcation (GC), node classiﬁcation (NC), and link prediction (LP).

Original: We deﬁne a multi-task episode E ( m ) i [Equation] p [Equation]
Revised: We deﬁne a multi-task episode E ( m ) i ∼ p ( E ( m ) ) as a tuple

Original: E

Original: [Equation]

Original: m

Original: [Equation]

Original: ) as a tuple

Original: [Equation]
Revised: [Equation]

Original: where λ ( · ) are balancing coefﬁcients.
Revised: where λ ( · ) are balancing coefﬁcients.

Original: The meta-objective of our method then becomes:
Revised: The meta-objective of our method then becomes:

Original: [Equation]
Revised: [Equation]

Original: Support and target sets are set up to resemble a training and a validation set.
Revised: Support and target sets are set up to resemble a training and a validation set.

Original: This way the outer loop’s objective becomes to maximize the performance on a validation set, given a training set , hence pushing towards generalization.
Revised: This way the outer loop’s objective becomes to maximize the performance on a validation set, given a training set , hence pushing towards generalization.

Original: In more detail, given a batch of graphs, we divide it in equally sized splits (one per task), and we create support and target sets as follows:
Revised: In more detail, given a batch of graphs, we divide it in equally sized splits (one per task), and we create support and target sets as follows:

Original: Graph Classiﬁcation: S ( GC ) E i and T ( GC ) E i contain labeled graphs, obtained with a random split.
Revised: Graph Classiﬁcation: S ( GC ) E i and T ( GC ) E i contain labeled graphs, obtained with a random split.

Original: Node Classiﬁcation : S  NC ) E i and T ( NC ) E i are composed of the same graphs, with different labelled nodes.
Revised: Node Classiﬁcation: S ( NC ) E i and T ( NC ) E i are composed of the same graphs, with different labelled nodes.

Original: We mimic the common semi-supervised setting (Kipf & Welling, 2017) where feature vectors are available for all nodes, and only a small subset of nodes is labelled.
Revised: We mimic the common semi-supervised setting (Kipf & Welling, 2017) where feature vectors are available for all nodes, and only a small subset of nodes is labelled.

Original: Link Prediction: S ( LP ) E 
Revised: Link Prediction: S ( LP ) E i and T ( LP ) E i are composed of the same graphs, with different query edges In every

Revised: graph

Original: i
Revised: we

Original: and T ( LP ) E i are composed of the same graphs, with different query edges.
Revised: randomly

Original: In every graph we randomly remove some edges, used as positive examplestogether with nonremoved edges, and randomly sample couples of non-adjacent nodes as negative examples.
Revised: remove some edges , used as positive examples

Revised: together with non

Revised: removed edges, and

Revised: randomly sample couples of non adjacent nodes

Original:  The full algorithm for the creation of multi-task episodes is provided in Appendix A.
Revised: as negative examples The full algorithm for the creation of multi-task episodes is provided in Appendix A.

Original: 4. M ODEL A RCHITECTURE D ESIGN
Revised: M ODEL A RCHITECTURE D ESIGN

Original: We use an encoder-decoder model with a multi-head architecture.
Revised: We use an encoder-decoder model with a multi-head architecture.

Original: The backbone (which represents the encoder) is composed of 3 GCN (Kipf & Welling, 2017) layers with ReLU non-linearities and residual connections (He et al., 2016).
Revised: The backbone (which represents the encoder) is composed of 3 GCN (Kipf & Welling, 2017) layers with ReLU non-linearities and residual connections (He et al., 2016).

Original: The decoder is composed of three heads .
Revised: The decoder is composed of three heads .

Original: The node classiﬁcation head is a single layer neural network with a Softmax activation that is shared across nodes and maps node embeddings to class predictions.
Revised: The node classiﬁcation head is a single layer neural network with a Softmax activation that is shared across nodes and maps node embeddings to class predictions.

Original: In the graph classiﬁcation head, ﬁrst a single layer neural network (shared across nodes) performs a linear transformation (followed by a ReLU activation) of the node embeddings.
Revised: In the graph classiﬁcation head, ﬁrst a single layer neural network (shared across nodes) performs a linear transformation (followed by a ReLU activation) of the node embeddings.

Original: The transformed node embeddings are then averaged and a ﬁnal single layer neural network with Softmax activation outputs the class predictions.
Revised: The transformed node embeddings are then averaged and a ﬁnal single layer neural network with Softmax activation outputs the class predictions.

Original: The link prediction head is composed of a single layer neural network with a ReLU non-linearity that transforms node embeddings, and another single layer neural network that takes as input the concatenation of two node embeddings and outputs the probability of a link between them.
Revised: The link prediction head is composed of a single layer neural network with a ReLU non-linearity that transforms node embeddings, and another single layer neural network that takes as input the concatenation of two node embeddings and outputs the probability of a link between them.

Original: 4. M ETA -T RAINING D ESIGN
Revised: 4. M ETA -T RAINING D ESIGN

Original: We ﬁrst present our meta-learning training procedure, and successively describe which parameters are adapted/updated in the inner and outer loops.
Revised: We ﬁrst present our meta-learning training procedure, and successively describe which parameters are adapted/updated in the inner and outer loops.

Original: Meta-Learning Training Procedure.
Revised: Meta-Learning Training Procedure.

Original: To avoid the problems arising from training a model that performs multiple tasks concurrently, we design a meta-learning procedure where the inner loop adaptation and the meta-objective computation involves a single task at a time.
Revised: To avoid the problems arising from training a model that performs multiple tasks concurrently, we design a meta-learning procedure where the inner loop adaptation and the meta-objective computation involves a single task at a time.

Original: Only the parameter update performed to minimize the meta-objective involves multiple tasks, but, crucially, it does not aim at a setting of parameters that can solve, or quickly adapt to, multiple tasks concurrently , but to a setting that allows multiple fast single-task adaptation .
Revised: Only the parameter update performed to minimize the meta-objective involves multiple tasks, but, crucially, it does not aim at a setting of parameters that can solve, or quickly adapt to, multiple tasks concurrently , but to a setting that allows multiple fast single-task adaptation .

Original: Input: Model f θ ; Episodes E = {E 1 , .., E n } init ( θ ) for E i in E do o loss ←for t in (GC, NC, LP) do θ (cid:48)
Revised: Input: Model f θ ; Episodes E = {E 1 , .., E n } init ( θ ) for E i in E do o loss ←for t in (GC, NC, LP) do θ (cid:48)

Original: ( t ) ← θ θ (cid:48)
Revised: ( t ) ← θ θ (cid:48)

Original: ( t ) ← ADAPT ( f θ , S ( t ) E
Revised: ( t ) ← ADAPT ( f θ , S ( t ) E

Original: i , L ( t ) E i )
Revised: i , L ( t ) E i )

Original: o loss ← o loss + TEST ( f θ (cid:48) ( t ) , T ( t ) E i , L ( t ) E i ) end θ ← UPDATE ( θ, o loss , θ (cid:48)
Revised: o loss ← o loss + TEST ( f θ (cid:48) ( t ) , T ( t ) E i , L ( t ) E i ) end θ ← UPDATE ( θ, o loss , θ (cid:48)

Original: ( GC ) , θ (cid:48)
Revised: ( GC ) , θ (cid:48)

Original: ( NC ) , θ (cid:48) ( LP ) ) end
Revised: ( NC ) , θ (cid:48) ( LP ) ) end

Original: The pseudocode of our procedure is in Algorithm 1.
Revised: The pseudocode of our procedure is in Algorithm 1.

Original: ADAPT performs a few steps of gradient descent on a task speciﬁc loss function and support set,
Revised: ADAPT performs a few steps of gradient descent on a task speciﬁc loss function and support set,

Original: TEST computes the value of the metaobjective component on a task speciﬁc loss function and target set, and
Revised: TEST computes the value of the metaobjective component on a task speciﬁc loss function and target set, and

Original: UPDATE optimizes the parameters by minimizing the meta-objective.
Revised: UPDATE optimizes the parameters by minimizing the meta-objective.

Original: Notice how the multiple heads of the decoder in our model are never used concurrently.
Revised: Notice how the multiple heads of the decoder in our model are never used concurrently.

Original: Parameter Update in Inner/Outer
Revised: Parameter Update in Inner/Outer

Original: Loop.
Revised: Loop.

Original: Let us partition the parameters of our model in four sets: θ =
Revised: Let us partition the parameters of our model in four sets: θ =

Original: [ θ GCN , θ NC , θ GC , θ LP ] representing the parameters of the backbone ( θ GCN ), node classiﬁcation head ( θ NC ), graph classiﬁcation head ( θ GC ), and link prediction head ( θ LP ).
Revised: [ θ GCN , θ NC , θ GC , θ LP ] representing the parameters of the backbone ( θ GCN ), node classiﬁcation head ( θ NC ), graph classiﬁcation head ( θ GC ), and link prediction head ( θ LP ).

Original: We name our proposed meta-learning strategy SAME(
Revised: We name our proposed meta-learning strategy SAME

Original: Single-Task Adaptation for Multi-Task Embeddings), and present two variants (Figure 2 (b)-(c)):
Revised: (Single-Task Adaptation for Multi-Task Embeddings), and present two variants (Figure 2 (b)-(c)):

Original: Implicit SAME (iSAME): all the parameters θ are used for adaptation.
Revised: Implicit SAME (iSAME): all the parameters θ are used for adaptation.

Original: This strategy makes use of the implicit feature-reuse factor of MAML, leading to parameters θ GCN that are general across multi-task episodes .
Revised: This strategy makes use of the implicit feature-reuse factor of MAML, leading to parameters θ GCN that are general across multi-task episodes .

Original: Explicit SAME (eSAME): only the head parameters θ NC , θ GC , θ LP are used for adaptation.
Revised: Explicit SAME (eSAME): only the head parameters θ NC , θ GC , θ LP are used for adaptation.

Original: Contrary to the previous, this strategy explicitly aims at learning the parameters θ GCN to be general across multi-task episodes by only updating them in the outer loop.
Revised: Contrary to the previous, this strategy explicitly aims at learning the parameters θ GCN to be general across multi-task episodes by only updating them in the outer loop.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our goal is to assess the quality of the representations learned by our proposed methods.
Revised: Our goal is to assess the quality of the representations learned by our proposed methods.

Original: In more detail, we aim to answer the following questions:
Revised: In more detail, we aim to answer the following questions:

Original: Q1:
Revised: Q1:

Original: Do our proposed meta-learning procedures lead to high quality node embeddings in the singletask setting?
Revised: Do our proposed meta-learning procedures lead to high quality node embeddings in the singletask setting?

Original: Q2:
Revised: Q2:

Original: Do our proposed meta-learning procedures lead to high quality node embeddings in the multitask setting?
Revised: Do our proposed meta-learning procedures lead to high quality node embeddings in the multitask setting?

Original: Q3:
Revised: [Table]

Original:  Do our proposed meta-learning procedures for multiple tasks extract information that is not captured by classically trained multi-task models?
Revised: : Do our proposed meta-learning procedures for multiple tasks extract information that is not captured by classically trained multi-task models?

Original: Q4:
Revised: Q4:

Original: Can the node embeddings learned using our proposed meta-learning procedures be used for multiple tasks with comparable or better performance than classical multi-task models?
Revised: Can the node embeddings learned using our proposed meta-learning procedures be used for multiple tasks with comparable or better performance than classical multi-task models?

Original: Throughout this section we use GC to refer to graph classiﬁcation, NC for node classiﬁcation, and
Revised: Throughout this section we use GC to refer to graph classiﬁcation, NC for node classiﬁcation [Table] and

Original: LP for link prediction.
Revised: LP for link prediction.

Original: Unless otherwise stated, accuracy (%) is used for NC and GC, while ROC
Revised: Unless otherwise stated, accuracy (%) is used for NC and GC, while ROC

Original: AUC (%) is used for LP.
Revised: AUC (%) is used for LP.

Original: Experimental Setup.
Revised: Datasets.

Original: To perform multiple tasks, we consider datasets with graph labels, node attributes, and node labels from the TUDataset library (Morris et al., 2020).
Revised: To perform multiple tasks, we consider datasets with graph labels, node attributes, and node labels from the TUDataset library (Morris et al., 2020): ENZYMES (Schomburg et al., 2004), PROTEINS (Dobson & Doig, 2003), DHFR(Sutherland et al., 2003), and COX2 (Sutherland et al., 2003).

Revised: ENZYMES is a dataset of protein structures belonging to six classes.

Revised: PROTEINS is a dataset of chemical compounds with two classes (enzyme and non-enzyme).

Revised: DHFR, and COX2 are datasets of chemical inhibitors which can be active or inactive.

Revised: In all datasets, each node has a series of attributes containing physical and chemical measurements.

Revised: Experimental Setup.

Original: We perform a 10-fold cross validation, and average results across folds.
Revised: We perform a 10-fold cross validation, and average results across folds.

Original: To ensure a fair comparison, we use the same architecture for all training strategies.
Revised: To ensure a fair comparison, we use the same architecture for all training strategies.

Original: We tested loss balancing techniques (e.g. uncertainty weights (Kendall et al., 2018), and gradnorm (Chen et al., 2018)) but found that they were not effective. 
Revised: We tested loss balancing techniques (e.g. uncertainty weights (Kendall et al., 2018), and gradnorm (Chen et al., 2018)) but found that they were not effective. In our experiments we notice that the losses do not need to be balanced, and we set λ ( GC ) = λ ( NC ) = λ ( LP )

Revised: = 1 without performing any tuning.

Original: For more information we refer to Appendix B, and we provide source code as supplementary material.
Revised: For more information we refer to Appendix B, and we provide source code as supplementary material.

Original: Q
Revised: Q1:

Original: For every task we train a linear classiﬁer on top of the embeddings produced by a model trained using our proposed methods, and compare against a network with the same architecture trained in a classical manner.
Revised: For every task, we train a linear classiﬁer on top of the embeddings produced by a model trained using our proposed methods, and compare against a network with the same architecture trained in a classical manner.

Original: Results are shown in Table 1.
Revised: Results are shown in Table 1.

Original: For all three tasks, a linear classiﬁer on the embeddings produced by our methods achieves comparable, if not superior, performance to an end-to-end model.
Revised: For all three tasks, a linear classiﬁer on the embeddings produced by our methods achieves comparable, if not superior, performance to an end-to-end model.

Original: In fact, the linear classiﬁer is never outperformed by more than 2%, and it can [Table] [Table] [Table]
Revised: In fact, the linear classiﬁer is never outperformed by more than 2%, and it can outperform the classical end-to-end model by up to 12%.

Original: [Table]

Original: outperform the classical end-to-end model by up to 12%.

Original: These results show that our meta-learning procedures are learning high quality node embeddings
Revised: These results show that our meta-learning procedures are learning high quality node embeddings.

Original: A cc u r a c y

Original: GC,NC -> LP (a) 0.

Original: [Figure]

Original: Q2:
Revised: Q2:

Original: We train a model with our proposed methods, on all multi-task combinations, and use the embeddings as the input for a linear classiﬁer .
Revised: We train a model with our proposed methods, on all multi-task combinations, and use the embeddings as the input for a linear classiﬁer .

Original: We compare against models with the same taskspeciﬁc architecture trained in the classical manner on a single task.
Revised: We compare against models with the same task-speciﬁc architecture trained in the classical manner on a single task, and with a ﬁne-tuning baseline.

Revised: The latter is a model that has been trained on all three tasks, and then ﬁne-tuned on two speciﬁc tasks.

Revised: The idea is that the initial training on all tasks should lead the model towards the extraction of features that it would otherwise not consider (by only seeing 2 tasks), and the ﬁne-tuning process should then allow the model to use these features to target the speciﬁc tasks of interest.

Original: Results are shown in Table 2 (we omit standard deviation for space limitations).
Revised: Results are shown in Table 2 (we omit standard deviation for space limitations).

Original: We notice that the embeddings produced by our procedures in a multi-task setting, followed by a linear classiﬁer, achieve comparable performance to end-to-end single-task models.
Revised: We notice that the embeddings produced by our procedures in a multi-task setting, followed by a linear classiﬁer, achieve comparable performance to end-to-end single-task models.

Original: In fact, the linear classiﬁer is never outperformed by more than 3%, and in 50% of the cases it actually achieves higher performance.
Revised: In fact, the linear classiﬁer is never outperformed by more than 3%, and in 50% of the cases it actually achieves higher performance.

Revised: We further notice that the ﬁne-tuning baseline severely struggles, and is almost always outperformed by both single-task

Original: [Table]
Revised: [Table]

Original: Q3:
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: 8 (cid:88)

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: (cid:88) (cid:88)

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: models, and our proposed methods.

Revised: These results indicate that the episodic meta-learning procedure adopted by SAME is extracting features that are otherwise not accessible with standard training techniques.

Original:  We train a multi-task model, and we then train a new simple network (with the same architecture as the heads described in Section 4.2), which we refer to as classiﬁer , on the embeddings to perform a task that was not seen during training.
Revised: [Figure] Q3: We train a multi-task model, and we then train a new simple network (with the same architecture as the heads described in Section 4.2), which we refer to as classiﬁer , on the embeddings to perform a task that was not seen during training.

Original: We compare the performance of the classiﬁer on the embeddings learned by a model trained in a classical manner, and with our proposed procedure.
Revised: We compare the performance of the classiﬁer on the embeddings learned by a model trained in a classical manner, and with our proposed procedure.

Original: Intuitively, this tests gives us a way to analyse if the embeddings learned by our proposed approaches contain “more information” than embeddings learned in a classical manner.
Revised: Intuitively, this tests gives us a way to analyse if the embeddings learned by our proposed approaches contain “more information” than embeddings learned in a classical manner.

Original: Results on the ENZYMES dataset are shown in Figure 3, where we notice that embeddings learned by our proposed approaches lead to at least 10% higher performance.
Revised: Results on the ENZYMES dataset are shown in Figure 3, where we notice that embeddings learned by our proposed approaches lead to at least 10% higher performance.

Original: We observe an analogous trend on the other datasets, and report all results in Appendix D.
Revised: We observe an analogous trend on the other datasets, and report all results in Appendix D.

Original: Q4: We train the same multi-task model, both in the classical supervised manner, and with our proposed approaches, on all multi-task combinations.
Revised: Q4: We train the same multi-task model, both in the classical supervised manner, and with our proposed approaches, on all multi-task combinations.

Original: For our approaches, we then train a linear classiﬁer on top of the node embeddings.
Revised: For our approaches, we then train a linear classiﬁer on top of the node embeddings.

Revised: We further consider the ﬁne-tuning baseline introduced in

Original:  We use the multi-task performance ( ∆ m ) metric (Maninis et al., 2019), deﬁned as the average per-task drop with respect to the single-task baseline: ∆ m = 
Revised: Q2 . We use the multi-task performance ( ∆ m ) metric (Maninis et al., 2019), deﬁned as the average per-task drop with respect to the single-task baseline: ∆ m = 1 T (cid:80)

Original: T (cid:80) Ti =1 ( M m,i − M b,i ) /M
Revised: Ti =1 ( M m,i − M b,i ) /M

Original: b,i , where M m,i is the value of a task’s metric for the multi-task model, and M b,i is the value for the baseline.
Revised: b,i , where M m,i is the value of a task’s metric for the multi-task model, and M b,i is the value for the baseline.

Original:  Results are shown in Table 4.
Revised: [Table] Results are shown in Table 4.

Original: We ﬁrst notice that usually multitask models achieve lower performance than specialized single-task ones.
Revised: We ﬁrst notice that usually multi-task models achieve lower performance than specialized single-task ones.

Original: We then highlight that linear classiﬁers trained on the embeddings produced by our procedures are not only comparable, but in many cases signiﬁcantly superior to classically trained multi-task models.
Revised: We then highlight that linear classiﬁers trained on the embeddings produced by our procedures are not only comparable, but in many cases signiﬁcantly superior to classically trained multi-task models.

Original: In fact, a multi-task model trained in a classical manner is highly sensible to the tasks that are being learned (e.g. GC and LP negatively interfere with each other in every dataset), while our methods seem much less sensible: the former has a worst-case average drop in performance of 29%, while our method has a worst-case average drop of 4%.
Revised: In fact, a multi-task model trained in a classical manner is highly sensible to the tasks that are being learned (e.g. GC and LP negatively interfere with each other in every dataset), while our methods seem much less sensible: the former has a worst-case average drop in performance of 29%, while our method has a worst-case average drop of less than 3%.

Revised: Finally, we also notice that the ﬁne-tuning baseline generally performs worst than classically trained models, conﬁrming that transferring knowledge in multi-task settings is not easy, and more advanced techniques, like our proposed method SAME, are needed.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work we propose a novel representation learning strategy for multi-task settings.
Revised: In this work we propose a novel representation learning strategy for multi-task settings.

Original: Our method overcomes the problems that arise when learning to solve multiple tasks concurrently by optimizing for a parameter setting that can quickly, i.e. with few steps of gradient descent, be adapted for high single-task performance on multiple tasks.
Revised: Our method overcomes the problems that arise when learning to solve multiple tasks concurrently by optimizing for a parameter setting that can quickly, i.e. with few steps of gradient descent, be adapted for high single-task performance on multiple tasks.

Original: We apply our method to graph representation learning, and ﬁnd that our training procedure leads to higher quality node embeddings, both in the multi-task setting, and in the single-task setting.
Revised: We apply our method to graph representation learning, and ﬁnd that our training procedure leads to higher quality node embeddings, both in the multi-task setting, and in the single-task setting.

Original: In fact, we show that a linear classiﬁer trained on the embeddings produced by our method has comparable or better performance than classical end-to-end supervised models.
Revised: In fact, we show that a linear classiﬁer trained on the embeddings produced by our method has comparable or better performance than classical end-to-end supervised models.

Original: Furthermore, we ﬁnd that the embeddings learned with our proposed procedures lead to higher performance on downstream tasks that were not seen during training.
Revised: Furthermore, we ﬁnd that the embeddings learned with our proposed procedures lead to higher performance on downstream tasks that were not seen during training.

Original: We believe this work can be highly useful to the whole deep representation learning community, as our method is model agnostic and task agnostic, and can therefore be applied on a wide variety of multi-task domains.
Revised: We believe this work can be highly useful to the whole deep representation learning community, as our method is model agnostic and task agnostic, and can therefore be applied on a wide variety of multi-task domains.

Original: A E PISODE D ESIGN A LGORITHM
Revised: A E PISODE D ESIGN A LGORITHM

Original: Algorithm 2 contains the procedure for the creation of the episodes for our meta-learning procedures.
Revised: Algorithm 2 contains the procedure for the creation of the episodes for our meta-learning procedures.

Original: The algorithm takes as input a batch of graphs (with graph labels, node labels, and node features) and the loss function balancing weights, and outputs a multi-task episode .
Revised: The algorithm takes as input a batch of graphs (with graph labels, node labels, and node features) and the loss function balancing weights and outputs a multi-task episode .

Original: We assume that each graph has a set of attributes that can be accessed with a dot-notation (like in most object-oriented programming languages).
Revised: We assume that each graph has a set of attributes that can be accessed with a dot-notation (like in most object-oriented programming languages).

Original: Notice how the episodes are created so that only one task is performed on each graph.
Revised: Notice how the episodes are created so that only one task is performed on each graph.

Original: This is important as in the inner loop of our meta-learning procedure, the learner adapts and tests the adaptated parameters on one task at a time.
Revised: This is important as in the inner loop of our meta-learning procedure, the learner adapts and tests the adaptated parameters on one task at a time.

Original: The outer loop then updates the parameters, optimizing for a representation that leads to fast single-task adaptation .
Revised: The outer loop then updates the parameters, optimizing for a representation that leads to fast single-task adaptation .

Original: This procedure bypasses the problem of learning parameters that directly solve multiple tasks, which can be very challenging.
Revised: This procedure bypasses the problem of learning parameters that directly solve multiple tasks, which can be very challenging.

Original: Another important aspect to notice is that the support and target sets are designed as if they were the training and validation splits for training a single-task model with the classical procedure.
Revised: Another important aspect to notice is that the support and target sets are designed as if they were the training and validation splits for training a single-task model with the classical procedure.

Original: This way the meta-objective becomes to train a model that can generalize well.
Revised: This way the meta-objective becomes to train a model that can generalize well.

Revised: B A DDITIONAL E XPERIMENTAL D ETAILS

Revised: In this section we provide additional information on the implementation of the models used in our experimental section.

Revised: We implement our models using PyTorch (Paszke et al., 2019), PyTorch

Revised: Geometric (Fey & Lenssen, 2019) and Torchmeta (Deleu et al., 2019).

Original: Algorithm
Revised: For all models the number and structure of the layers is as described in Section 4.2 of the paper, where we use 256-dimensional node embeddings at every layer. At every cross-validation fold the datasets are split into 70% for training, 10% for validation, and 20% for testing. For each model we perform 100 iterations of hyperparameter optimization over the same search space (for shared parameters) using Ax (Bakshy et al., 2018). We tried some sophisticated methods to balance the contribution of loss functions during multi-task training like GradNorm (Chen et al., 2018) and Uncertainty Weights (Kendall et al., 2018), but we saw that usually they do not positively impact performance.

Revised: Furthermore, in the few cases where they increase performance, they work for both classically trained models, and for models trained with our proposed procedures.

Revised: We then set the balancing weights to λ ( GC )

Revised: =

Revised: λ ( NC ) = λ ( LP )

Revised: =to provide better comparisons between the training strategies.

Revised: Algorithm 2:

Original: Input : Batch of n randomly sampled graphs B = {G 1 , .., G n }
Revised: Input : Batch of n randomly sampled graphs B = {G 1 , .., G n }

Original: Loss weights λ ( GC ) , λ ( NC ) , λ ( LP ) ∈
Revised: Loss weights λ ( GC ) , λ ( NC ) , λ ( LP ) ∈

Original: Output: Episode E i =
Revised: Output: Episode E i =

Original: ( L ( m ) E i , S ( m ) E i , T ( m ) E i )
Revised: ( L ( m ) E i , S ( m ) E i , T ( m ) E i )

Original: B ( GC ) , B ( NC ) , B ( LP ) ← equally divide the graphs in B in three sets /* Graph Classification */
Revised: B ( GC ) , B ( NC ) , B ( LP ) ← equally divide the graphs in B in three sets /* Graph Classification */

Original: S ( GC ) E i , T ( GC ) E i ← randomly divide B ( GC ) with a 60/40 split /* Node Classification */ for G i in B ( NC ) do num labelled nodes ← G i .num
Revised: S ( GC ) E i , T ( GC ) E i ← randomly divide B ( GC ) with a 60/40 split / * Node Classification * / for G i in B ( NC ) do num labelled nodes ← G i .num

Original: nodes × 0 .
Revised: nodes × 0 .

Original: N ← divide nodes per class, then iteratively randomly sample one node per class without replacement and add it to N until |N|
Revised: N ← divide nodes per class, then iteratively randomly sample one node per class without replacement and add it to N until |N|

Original: =
Revised: =

Original: num labelled nodes
Revised: num labelled nodes

Original: G (cid:48)
Revised: G (cid:48)

Original: i ← copy ( G i )
Revised: i ← copy ( G i )

Original: G i .labelled nodes ← N ;
Revised: G i .labelled nodes ← N ;

Original: G (cid:48) i.labelled nodes ← G
Revised: i

Original: i .nodes \ N
Revised: N

Original: S ( NC ) E i .add
Revised: S ( NC ) E i .add

Original: ( G i ) ;
Revised: ( G i ) ;

Original: T ( NC ) E i.add
Revised: T ( NC ) E i

Original: ( G (cid:48) i ) end /
Revised: ( G (cid:48) i ) end /

Original: * Link Prediction */ for G i in B ( LP ) do
Revised: * Link Prediction */ for G i in B ( LP ) do

Original: E ( N ) i ← randomly pick negative samples (edges that are not in the graph; possibly in the same number as the number of edges in the graph)
Revised: E ( N ) i ← randomly pick negative samples (edges that are not in the graph; possibly in the same number as the number of edges in the graph)

Original: E 1 , ( N ) i , E 2 , ( N ) i ← divide E ( N ) i with an 80 / 20 split
Revised: E 1 , ( N ) i , E 2 , ( N ) i ← divide E ( N ) i with an 80 / 20 split

Original: E ( P ) i ← randomly remove 20% of the edges in G
Revised: E ( P ) i ← randomly remove 20% of the edges in G

Original: i
Revised: i

Original: G (cid:48) (1) i ← G i removed of E ( P )
Revised: G (cid:48) (1) i ← G i removed of E ( P )

Original: i
Revised: i

Original: G (cid:48) (2) i ←  ( G (cid:48) (1) i )
Revised: G (cid:48) (2) i ← copy ( G (cid:48) (1) i )

Original: G (cid:48) (
Revised: G (cid:48) (1)

Original: i
Revised: i

Revised: .positive edges ← G

Revised: i .edges ;

Original: S ( m ) E i  ← {S ( GC )E
Revised: i .positive edges ← S ← {S ( GC )

Original: i , S (NC ) E i , S ( LP ) E i }
Revised: , S ( , S ( LP ) E i }

Original: T ( m ) E i ← {T ( GC ) E i , T ( NC ) E i , T ( LP ) E i }
Revised: T ( m ) E i ← {T ( GC ) E i , T ( NC ) E i , T ( LP ) E i }

Original: L ( GC ) T i ← Cross-Entropy ( · ) ;
Revised: L ( GC ) T i ← Cross-Entropy ( · ) ;

Original: L ( NC ) T
Revised: L ( NC ) T

Original: i ← Cross-Entropy ( · )
Revised: i ← Cross-Entropy ( · )

Original: L ( LP ) T i ← Binary Cross-Entropy ( · )
Revised: L ( LP ) T i ← Binary Cross-Entropy ( · )

Original: L ( m ) E i
Revised: L ( m ) E i

Original: = λ ( GC ) L ( GC ) T
Revised: = λ ( GC ) L ( GC ) T

Original: i + λ ( NC ) L ( NC ) T
Revised: i + λ ( NC ) L ( NC ) T

Original: i + λ ( LP ) L ( LP ) T
Revised: i + λ ( LP ) L ( LP ) T

Original: i
Revised: i

Original: Return E =
Revised: Return E =

Original: ( L ( m ) E i , S ( m ) E i , T ( m ) E i )
Revised: ( L ( m ) E i , S ( m ) E i , T ( m ) E i )

Original: B A DDITIONAL E XPERIMENTAL D ETAILS

Original: In this section we provide additional information on the implementation of the models used in our experimental section.

Original: We implement our models using PyTorch (Paszke et al., 2019), PyTorch

Original: Geometric (Fey & Lenssen, 2019) and Torchmeta (Deleu et al., 2019).

Original: For all models the number and structure of the layers is as described in Section 4.2 of the paper, where we use 256-dimensional node embeddings at every layer.

Original: At every cross-validation fold the datasets are split into 70% for training, 10% for validation, and 20% for testing.

Original: For each model we perform 100 iterations of hyperparameter optimization over the same search space (for shared parameters) using Ax (Bakshy et al., 2018).

Original: We tried some sophisticated methods to balance the contribution of loss functions during multi-task training like GradNorm (Chen et al., 2018) and Uncertainty Weights (Kendall et al., 2018), but we saw that usually they do not positively impact performance.

Original: Furthermore, in the few cases where they increase performance, they work for both classically trained models, and for models trained with our proposed procedures.

Original: We then set the balancing weights to λ ( GC )

Original: =

Original: λ ( NC ) = λ ( LP )

Original: =to provide better comparisons between the training strategies.

Original: Linear Model.
Revised: Linear Model.

Original: The linear model trained on the embeddings produced by our proposed method is a standard linear SVM.
Revised: The linear model trained on the embeddings produced by our proposed method is a standard linear SVM.

Original: In particular we use the implementation available in Scikit-learn (Pedregosa et al., 2011) with default hyperparameters.
Revised: In particular we use the implementation available in Scikit-learn (Pedregosa et al., 2011) with default hyperparameters.

Original: For graph classiﬁcation, we take the mean of the node embeddings as input.
Revised: For graph classiﬁcation, we take the mean of the node embeddings as input.

Original: For link prediction we take the concatenation of the embeddings of two nodes.
Revised: For link prediction we take the concatenation of the embeddings of two nodes.

Original: For node classiﬁcation we keep the embeddings unaltered.
Revised: For node classiﬁcation we keep the embeddings unaltered.

Original: Deep Learning Baselines.
Revised: Deep Learning Baselines.

Original: We train the single task models for 1000 epochs, and the multi-task models for 5000 epochs, with early stopping on the validation set (for multi-task models we use the sum of the task validation losses or accuracies as metrics for early-stopping).
Revised: We train the single task models for 1000 epochs, and the multi-task models for 5000 epochs, with early stopping on the validation set (for multi-task models we use the sum of the task validation losses or accuracies as metrics for early-stopping).

Original: Optimization is done using Adam (Kingma & Ba, 2015).
Revised: Optimization is done using Adam (Kingma & Ba, 2015).

Original: For node classiﬁcation and link prediction we found that normalizing the node embeddings to unit norm in between GCN layers helps performance.
Revised: For node classiﬁcation and link prediction we found that normalizing the node embeddings to unit norm in between GCN layers helps performance.

Original: Our Meta-Learning Procedure.
Revised: Our Meta-Learning Procedure.

Original: We train the single task models for 5000 epochs, and the multitask models for 15000 epochs, with early stopping on the validation set (for multi-task models we use the sum of the task validation losses or accuracies as metrics for early-stopping).
Revised: We train the single task models for 5000 epochs, and the multitask models for 15000 epochs, with early stopping on the validation set (for multi-task models we use the sum of the task validation losses or accuracies as metrics for early-stopping).

Original: Early stopping is very important in this case as it is the only way to check if the meta-learned model is overﬁtting the training data.
Revised: Early stopping is very important in this case as it is the only way to check if the meta-learned model is overﬁtting the training data.

Original: The inner loop adaptation consists of 1 step of gradient descent.
Revised: The inner loop adaptation consists of 1 step of gradient descent.

Original: Optimization in the outer loop is done using Adam (Kingma & Ba, 2015).
Revised: Optimization in the outer loop is done using Adam (Kingma & Ba, 2015).

Original: We found that normalizing the node embeddings to unit norm in between GCN layers helps performance.
Revised: We found that normalizing the node embeddings to unit norm in between GCN layers helps performance.

Original: C C OMPARISON WITH T RADITIONAL T RAINING A PPORACHES
Revised: C C OMPARISON WITH T RADITIONAL T RAINING A PPORACHES

Original: Our proposed meta-learning approach is signiﬁcantly different from the classical training strategy (Algorithm 3), and the traditional meta-learning approaches (Algorithm 4).
Revised: Our proposed meta-learning approach is signiﬁcantly different from the classical training strategy (Algorithm 3), and the traditional meta-learning approaches (Algorithm 4).

Original: The classical training approach for multi-task models takes as input a batch of graphs, which is simply a set of graphs, where on each graph the model has to execute all the tasks.
Revised: The classical training approach for multi-task models takes as input a batch of graphs, which is simply a set of graphs, where on each graph the model has to execute all the tasks.

Original: Based on the cumulative loss on all tasks
Revised: Based on the cumulative loss on all tasks

Original: [Equation]
Revised: [Equation]

Original: for all the graphs in the batch, the parameters are updated with some form of gradient descent, and the procedure is repeated for each batch.
Revised: for all the graphs in the batch, the parameters are updated with some form of gradient descent, and the procedure is repeated for each batch.

Original: The traditional meta-learning approach takes as input an episode, like our approach, but for every graph in the episode all the tasks are performed.
Revised: The traditional meta-learning approach takes as input an episode, like our approach, but for every graph in the episode all the tasks are performed.

Original: The support set and target set are single sets of graphs, where every task can be performed on all graphs.
Revised: The support set and target set are single sets of graphs, where every task can be performed on all graphs.

Original: The support set is used to obtain the adapted parameters θ (cid:48) , which have the goal of concurrently solving all tasks on all graphs in the target set.
Revised: The support set is used to obtain the adapted parameters θ (cid:48) , which have the goal of concurrently solving all tasks on all graphs in the target set.

Original: The loss functions, both for the inner loop and for the outer loop, are the same as the one used by the classical training approach.
Revised: The loss functions, both for the inner loop and for the outer loop, are the same as the one used by the classical training approach.

Original: The outer loop then updates the parameters aiming at a setting that can easily, i.e. with a few steps of gradient descent, be adapted to perform multiple tasks concurrently given a support set.
Revised: The outer loop then updates the parameters aiming at a setting that can easily, i.e. with a few steps of gradient descent, be adapted to perform multiple tasks concurrently given a support set.

Original: [Table]
Revised: Algorithm 3:

Original: Input: Model f θ ; Batches B = {B 1 , .., B n } init ( θ ) for B i in B do loss ← concurrently perform all tasks on all graphs in B i θ ← UPDATE ( θ, loss ) end 
Revised: Input: Model f θ ; Batches B = {B 1 , .., B n } init ( θ ) for B i in B do loss ← concurrently perform all tasks on all graphs in B i θ ← UPDATE ( θ, loss ) end Algorithm 4:

Original:  Input: Model f θ ; Episodes E = {E 1 , .., E n } init ( θ ) for E i in E do i loss ← concurrently perform all tasks on all support set graphs θ (cid:48) ← ADAPT ( θ, i loss ) o loss ← concurrently perform all tasks on all target set graphs using parameters θ (cid:48) θ ← UPDATE ( θ, θ (cid:48) , o loss ) end 
Revised: Meta- Input: Model f θ ; Episodes E = {E 1 , .., E n } init ( θ ) for E i in E do i loss ← concurrently perform all tasks on all support set graphs θ (cid:48) ← ADAPT ( θ, i loss ) o loss ← concurrently perform all tasks on all target set graphs using parameters θ (cid:48) θ ← UPDATE ( θ, θ (cid:48) , o loss ) end [Table]

Original: D F ULL R ESULTS FOR THE G ENERALIZATION OF N ODE E MBEDDINGS
Revised: D F ULL R ESULTS FOR THE G ENERALIZATION OF N ODE E MBEDDINGS

Original: Table 4 contains results for a neural network, trained on the embeddings generated by a multi-task model, to perform a task that was not seen during the training of the multi-task model.
Revised: Table 4 contains results for a neural network, trained on the embeddings generated by a multi-task model, to perform a task that was not seen during the training of the multi-task model.

Original: Accuracy (%) is used for node classiﬁcation (NC) and graph classiﬁcation (GC); ROC AUC (%) is used for link prediction (LP).
Revised: Accuracy (%) is used for node classiﬁcation (NC) and graph classiﬁcation (GC); ROC AUC (%) is used for link prediction (LP).

Original: The embeddings produced by our meta-learning methods lead to higher performance (up to 35% ), showing that our procedures lead to the extraction of more informative node embeddings with respect to the classical end-to-end training procedure.
Revised: The embeddings produced by our meta-learning methods lead to higher performance (up to 35% ), showing that our procedures lead to the extraction of more informative node embeddings with respect to the classical end-to-end training procedure.