Original: O N S IZE G ENERALIZATION IN G RAPH N EURAL N ETWORKS
Revised: O N S IZE G ENERALIZATION IN G RAPH N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph neural networks (GNNs) can process graphs of different sizes but their capacity to generalize across sizes is still not well understood.
Revised: Graph neural networks (GNNs) can process graphs of different sizes but their capacity to generalize across sizes is still not well understood.

Original: Size generalization is key to numerous GNN applications, from solving combinatorial optimization problems to learning in molecular biology.
Revised: Size generalization is key to numerous GNN applications, from solving combinatorial optimization problems to learning in molecular biology.

Original: In such problems, obtaining labels and training on large graphs can be prohibitively expensive, but training on smaller graphs is possible.
Revised: In such problems, obtaining labels and training on large graphs can be prohibitively expensive, but training on smaller graphs is possible.

Original: This paper puts forward the size-generalization question and characterizes important aspects of that problem theoretically and empirically.
Revised: This paper puts forward the size-generalization question and characterizes important aspects of that problem theoretically and empirically.

Original: We show that even for very simple tasks,  GNNs do not naturally generalize to graphs of larger size.
Revised: We prove that even for very simple tasks, such as counting the number of nodes or edges in a graph, GNNs do not naturally generalize to graphs of larger size.

Original: Instead, their generalization performance is closely related to the distribution of local patterns of connectivity and features and how that distribution changes from small to large graphs.
Revised: Instead, their generalization performance is closely related to the distribution of local patterns of connectivity and features and how that distribution changes from small to large graphs.

Original: Speciﬁcally, we show that in many cases, there are  GNNs that can perfectly solve a task on small graphs but generalize poorly to large graphs and that these GNNs are encountered in practice.
Revised: Speciﬁcally, we prove that for many tasks, there are weight assignments for GNNs that can perfectly solve the task on small graphs but fail on large graphs, if there is a discrepancy between their local patterns. We further demonstrate on several tasks, that training GNNs on small graphs results in solutions which do not generalize to larger graphs.

Original: We then formalize size generalization as a domain-adaption problem and describe two learning setups where size generalization can be improved.
Revised: We then formalize size generalization as a domainadaption problem and describe two learning setups where size generalization can be improved.

Original: First, as a self-supervised learning problem (SSL) over the target domain of large graphs.
Revised: First, as a self-supervised learning problem (SSL) over the target domain of large graphs.

Original: Second as a semi-supervised learning problem when few samples are available in the target domain.
Revised: Second as a semi-supervised learning problem when few samples are available in the target domain.

Original: We demonstrate the efﬁcacy of these solutions on a diverse set of benchmark graph datasets.
Revised: We demonstrate the efﬁcacy of these solutions on a diverse set of benchmark graph datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graphs are a ﬂexible representation, widely used for representing diverse data and phenomena.
Revised: Graphs are a ﬂexible representation, widely used for representing diverse data and phenomena.

Original: Graph neural networks (GNNs) – Deep models that operate over graphs – have emerged as a prominent learning model (Bruna et al., 2013; Kipf and Welling, 2016;
Revised: Graph neural networks (GNNs) – Deep models that operate over graphs – have emerged as a prominent learning model (Bruna et al., 2013; Kipf and Welling, 2016;

Original: Veli ˇ ckovi ´ c et al., 2017).
Revised: Veli ˇ ckovi ´ c et al., 2017).

Original: They are used in natural sciences (Gilmer et al., 2017), social network analysis (Fan et al., 2019), for solving difﬁcult mathematical problems (Luz et al., 2020) and for approximating solutions to combinatorial optimization problems (Li et al., 2018).
Revised: They are used in natural sciences (Gilmer et al., 2017), social network analysis (Fan et al., 2019), for solving difﬁcult mathematical problems (Luz et al., 2020) and for approximating solutions to combinatorial optimization problems (Li et al., 2018).

Original: In many domains, graphs data vary signiﬁcantly in size.
Revised: In many domains, graphs data vary signiﬁcantly in size.

Original: This is the case for molecular biology, where molecules – represented as graphs over atoms as nodes – span from small compounds to proteins with many thousands of nodes.
Revised: This is the case for molecular biology, where molecules – represented as graphs over atoms as nodes – span from small compounds to proteins with many thousands of nodes.

Original: It is even more severe in social networks, which can reach billions of nodes.
Revised: It is even more severe in social networks, which can reach billions of nodes.

Original: The success of GNNs for such data stems from the fact that the same GNN model can process input graphs regardless of their size.
Revised: The success of GNNs for such data stems from the fact that the same GNN model can process input graphs regardless of their size.

Original: Indeed, it has been proposed that GNNs can generalize to graphs whose size is different from what they were trained on , but it is largely unknown in what problems such generalization occurs.
Revised: Indeed, it has been proposed that GNNs can generalize to graphs whose size is different from what they were trained on , but it is largely unknown in what problems such generalization occurs.

Original: Empirically, several papers report good generalization performance on speciﬁc tasks (Li et al., 2018; Luz et al., 2020).
Revised: Empirically, several papers report good generalization performance on speciﬁc tasks (Li et al., 2018; Luz et al., 2020).

Original: Other papers, like
Revised: Other papers, like

Original: Veliˇckovi´c et al.
Revised: Veliˇckovi´c et al.

Original: (2019), show that size generalization can fail on several simple graph algorithms, and can be improved by using task-speciﬁc training procedures and speciﬁc architectures.
Revised: (2019), show that size generalization can fail on several simple graph algorithms, and can be improved by using task-speciﬁc training procedures and speciﬁc architectures.

Original: Given their ﬂexibility to operate on variable-sized graphs, A fundamental question arises about generalization in GNNs: ”When do GNNs trained on small graphs generalize to large graphs?”
Revised: Given their ﬂexibility to operate on variable-sized graphs, A fundamental question arises about generalization in GNNs: ”When do GNNs trained on small graphs generalize to large graphs?”

Original: Aside from being an intriguing theoretical question, this problem has important practical implications.
Revised: Aside from being an intriguing theoretical question, this problem has important practical implications.

Original: In many domains, it is hard to label large graphs.
Revised: In many domains, it is hard to label large graphs.

Original: For instance, in combinatorial optimization problems, labeling a large graph boils down to solving large and hard optimization problems.
Revised: For instance, in combinatorial optimization problems, labeling a large graph boils down to solving large and hard optimization problems.

Original: In other domains, it is often very hard for human raters to correctly label complex networks.
Revised: In other domains, it is often very hard for human raters to correctly label complex networks.

Original: One approach to this problem could have been to resize graphs into a homogeneous size.
Revised: One approach to this problem could have been to resize graphs into a homogeneous size.

Original: This is the strategy taken in computer vision, where it is well understood how to resize an image while keeping its content.
Revised: This is the strategy taken in computer vision, where it is well understood how to resize an image while keeping its content.

Original: Unfortunately, there are no effective resizing procedures for graphs.
Revised: Unfortunately, there are no effective resizing procedures for graphs.

Original: It would therefore be extremely valuable to develop techniques that can generalize from training on small graphs.
Revised: It would therefore be extremely valuable to develop techniques that can generalize from training on small graphs.

Original: As we discuss below, a theoretical analysis of size generalization is very challenging because it depends on several different factors, including the task, the architecture, and the data.
Revised: As we discuss below, a theoretical analysis of size generalization is very challenging because it depends on several different factors, including the task, the architecture, and the data.

Original: For tasks, we argue that it is important to distinguish two types of tasks, local and global .
Revised: For tasks, we argue that it is important to distinguish two types of tasks, local and global .

Original: Local tasks can be solved by GNNs whose depth does not depend on the size of the input graph.
Revised: Local tasks can be solved by GNNs whose depth does not depend on the size of the input graph.

Original: For example, the task of ﬁnding a constant-size pattern.
Revised: For example, the task of ﬁnding a constant-size pattern.

Original: Global tasks require that the depth of the GNN grows with the size of the input graph.
Revised: Global tasks require that the depth of the GNN grows with the size of the input graph.

Original: For example, calculating the diameter of a graph.
Revised: For example, calculating the diameter of a graph.

Original: While there are a few previous works that explore depth-dependant GNNs (e.g., Tang et al.
Revised: While there are a few previous works that explore depth-dependant GNNs (e.g., Tang et al.

Original: (2020), constant depth GNNs are by far the most widely used GNN models today and are therefore the focus of this paper.
Revised: (2020), constant depth GNNs are by far the most widely used GNN models today and are therefore the focus of this paper.

Original: In this paper, we focus on GNNs with constant depth and study the ability of the most expressive message passing neural networks (Xu et al., 2018; Morris et al., 2019) to generalize to unseen sizes.
Revised: In this paper, we focus on GNNs with constant depth and study the ability of the most expressive message passing neural networks (Xu et al., 2018; Morris et al., 2019) to generalize to unseen sizes.

Original: Our key observation is that generalization to graphs of different sizes is strongly related to the distribution of patterns around nodes in the graphs of interest.
Revised: Our key observation is that generalization to graphs of different sizes is strongly related to the distribution of patterns around nodes in the graphs of interest.

Original: These patterns, dubbed d -patterns (where d is the radius of the local neighborhood), describe the local feature-connectivity structure around each node, as seen by message-passing neural networks and are deﬁned in Section 3.
Revised: These patterns, dubbed d -patterns (where d is the radius of the local neighborhood), describe the local feature-connectivity structure around each node, as seen by message-passing neural networks and are deﬁned in Section 3.

Original: We study the role of d -patterns both empirically and theoretically.
Revised: We study the role of d -patterns both empirically and theoretically.

Original: First, we theoretically show that when there is a signiﬁcant discrepancy between the d -pattern distributions, GNNs have multiple global minima for graphs of a speciﬁc size range, out of which only a subset of models can generalize well to larger graphs.
Revised: First, we theoretically show that when there is a signiﬁcant discrepancy between the d -pattern distributions, GNNs have multiple global minima for graphs of a speciﬁc size range, out of which only a subset of models can generalize well to larger graphs.

Original: We complement our theoretical analysis with an experimental study and show that GNNs tend to converge to non-generalizing global minima, when d -patterns from the large graph distribution are not well-represented in the small graph distribution.
Revised: We complement our theoretical analysis with an experimental study and show that GNNs tend to converge to non-generalizing global minima, when d -patterns from the large graph distribution are not well-represented in the small graph distribution.

Original: Furthermore we demonstrate that the size generalization problem is accentuated in deeper GNNs.
Revised: Furthermore we demonstrate that the size generalization problem is accentuated in deeper GNNs.

Original: Following these observations, in the ﬁnal part of this paper, we discuss two learning setups that help improve size-generalization by formulating the learning problem as a domain adaptation problem: (1) Training the GNNs on self-supervised tasks aimed at learning the d -pattern distribution of both the target (large graphs) and source (small graphs) domains.
Revised: Following these observations, in the ﬁnal part of this paper, we discuss two learning setups that help improve size-generalization by formulating the learning problem as a domain adaptation problem: (1) Training the GNNs on self-supervised tasks aimed at learning the d -pattern distribution of both the target (large graphs) and source (small graphs) domains.

Original: We also propose a novel SSL task that addresses over-ﬁtting of d -patterns.
Revised: We also propose a novel SSL task that addresses over-ﬁtting of d -patterns.

Original: (2) A semi-supervised learning setup with a limited number of labeled examples from the target domain.
Revised: (2) A semi-supervised learning setup with a limited number of labeled examples from the target domain.

Original: The idea behind both setups is to promote convergence of GNNs to local/global minima with good size generalization properties.
Revised: The idea behind both setups is to promote convergence of GNNs to local/global minima with good size generalization properties.

Original: We show that both setups are useful in a series of experiments on synthetic and real data.
Revised: We show that both setups are useful in a series of experiments on synthetic and real data.

Original: To summarize, this paper makes the following contributions.
Revised: To summarize, this paper makes the following contributions.

Original: (1) We identify a size generalization problem when learning local tasks with GNNs and analyze it empirically and theoretically.
Revised: (1) We identify a size generalization problem when learning local tasks with GNNs and analyze it empirically and theoretically.

Original: (2) We link the size-generalization problem with the distribution of d -patterns and suggest to approach it as a domain adaptation problem (3) We empirically show how several learning setups help improve size generalization.
Revised: (2) We link the size-generalization problem with the distribution of d -patterns and suggest to approach it as a domain adaptation problem (3) We empirically show how several learning setups help improve size generalization.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Size generalization in set and graph learning.
Revised: Size generalization in set and graph learning.

Original: Several papers observed successful generalization across graph sizes, but the underlying reasons were not investigated (Li et al., 2018;
Revised: Several papers observed successful generalization across graph sizes, but the underlying reasons were not investigated (Li et al., 2018;

Original: Maron et al., 2018; Luz et al., 2020).
Revised: Maron et al., 2018; Luz et al., 2020).

Original: More recently, (Veliˇckovi´c et al., 2019) showed that when training GNNs to perform simple graph algorithms step by step they generalize better to graphs of different sizes.
Revised: More recently, (Veliˇckovi´c et al., 2019) showed that when training GNNs to perform simple graph algorithms step by step they generalize better to graphs of different sizes.

Original: Unfortunately, such training procedures cannot be easily applied to general tasks.
Revised: Unfortunately, such training procedures cannot be easily applied to general tasks.

Original: Knyazev et al.
Revised: Knyazev et al.

Original: (2019) studied the relationship between generalization and attention mechanisms.
Revised: (2019) studied the relationship between generalization and attention mechanisms.

Original: Tang et al.
Revised: Tang et al.

Original: (2020) observed two issues that can harm generalization: (1) There are tasks for which a constant number of layers is not sufﬁcient.
Revised: (2020) observed two issues that can harm generalization: (1) There are tasks for which a constant number of layers is not sufﬁcient.

Original: (2) Some graph learning tasks are homogeneous functions.
Revised: (2) Some graph learning tasks are homogeneous functions.

Original: They then suggest a new GNN architecture to deal with these issues.
Revised: They then suggest a new GNN architecture to deal with these issues.

Original: Our work is complementary to these works as it explores another fundamental size generalization problem, focusing on constant depth GNNs.
Revised: Our work is complementary to these works as it explores another fundamental size generalization problem, focusing on constant depth GNNs.

Original: For more details on the distinction between constant depth and variable depth tasks see Appendix A.
Revised: For more details on the distinction between constant depth and variable depth tasks see Appendix A.

Original: Several works also studied size generalization and expressivity when learning set-structured inputs (Zweig and Bruna, 2020; Bueno and Hylton, 2020).
Revised: Several works also studied size generalization and expressivity when learning set-structured inputs (Zweig and Bruna, 2020; Bueno and Hylton, 2020).

Original: On the more practical side, Joshi et al.
Revised: On the more practical side, Joshi et al.

Original: Joshi et al.
Revised: Joshi et al.

Original: (2020) study the combinatorial problem of traveling salesman and whether it is possible to generalize to larger sizes.
Revised: (2020) study the combinatorial problem of traveling salesman and whether it is possible to generalize to larger sizes.

Original: Corso et al.
Revised: Corso et al.

Original: (2020) study several multitask learning problems on graphs and evaluate how the performance changes as the size of the graphs change.
Revised: (2020) study several multitask learning problems on graphs and evaluate how the performance changes as the size of the graphs change.

Original: Expressivity and generalization in graph neural networks.
Revised: Expressivity and generalization in graph neural networks.

Original: (Xu et al., 2018; Morris et al., 2019) established a fundamental connection between message-passing neural networks and the WeisfeilerLeman (WL) graph-isomorphism test.
Revised: (Xu et al., 2018; Morris et al., 2019) established a fundamental connection between message-passing neural networks and the WeisfeilerLeman (WL) graph-isomorphism test.

Original: We use similar arguments to show that GNNs have enough expressive power to solve a task on a set of small graphs and to fail on it on a set of large graphs.
Revised: We use similar arguments to show that GNNs have enough expressive power to solve a task on a set of small graphs and to fail on it on a set of large graphs.

Original: Several works studied generalization bounds for certain classes of GNNs (Garg et al., 2020; Puny et al., 2020; Verma and Zhang, 2019), but did not discuss size-generalization.
Revised: Several works studied generalization bounds for certain classes of GNNs (Garg et al., 2020; Puny et al., 2020; Verma and Zhang, 2019), but did not discuss size-generalization.

Original: Sinha et al.
Revised: Sinha et al.

Original: (2020) proposed a benchmark for assessing the logical generalization abilities of GNNs.
Revised: (2020) proposed a benchmark for assessing the logical generalization abilities of GNNs.

Original:  T HE SIZE GENERALIZATION PROBLEM
Revised:  T HE SIZE GENERALIZATION PROBLEM

Original: We now present the main problem discussed in the paper, that is, what determines if a GNN generalizes well to graphs of sizes not seen during training.
Revised: We now present the main problem discussed in the paper, that is, what determines if a GNN generalizes well to graphs of sizes not seen during training.

Original: We start with a simple motivating example showing the problem on single layer GNNs.
Revised: We start with a simple motivating example showing the problem on single layer GNNs.

Original: We then show that the question of size generalization actually depends on d-patterns , the local patterns of connectivity and features of the graphs, and not only on their actual size.
Revised: We then show that the question of size generalization actually depends on d-patterns , the local patterns of connectivity and features of the graphs, and not only on their actual size.

Original: Setup.
Revised: Setup.

Original: We are given two distributions over graphs P 1 , P 2 that contain small and large graphs accordingly, and a task that can be solved with 0 error for all graph sizes using a constant depth GNN.
Revised: We are given two distributions over graphs P 1 , P 2 that contain small and large graphs accordingly, and a task that can be solved with 0 error for all graph sizes using a constant depth GNN.

Original: We train a GNN on a training set S sampled i.i.d from P 1 and study its performance on P 2 .
Revised: We train a GNN on a training set S sampled i.i.d from P 1 and study its performance on P 2 .

Original: GNN model.
Revised: GNN model.

Original: We focus on the ﬁrst order GNN (1-GNN) architecture from Morris et al.
Revised: We focus on the ﬁrst order GNN (1-GNN) architecture from Morris et al.

Original: (2019) deﬁned in the following way:
Revised: (2019) deﬁned in the following way:

Original: [Equation]
Revised: [Equation]

Original: Here, h ( t ) v is the feature vector of node v after t layers, W ( t ) 1 , W ( t ) 2 ∈ R d t − 1 × d t , b
Revised: Here, h ( t ) v is the feature vector of node v after t layers, W ( t ) 1 , W ( t ) 2 ∈ R d t − 1 × d t , b

Original: ( t ) ∈
Revised: ( t ) ∈

Original: R d t denotes the parameters of the t -th layer of the GNN, and σ is some non-linear activation (e.g ReLU).
Revised: R d t denotes the parameters of the t -th layer of the GNN, and σ is some non-linear activation (e.g ReLU).

Original: It was shown in Morris et al.
Revised: It was shown in Morris et al.

Original: that GNNs composed from these layers have maximal expressive power with respect to all message-passing neural networks.
Revised: that GNNs composed from these layers have maximal expressive power with respect to all message-passing neural networks.

Original: In the experimental section we also experiment with Graph Isomorphism Network (GIN) (Xu et al., 2018).
Revised: In the experimental section we also experiment with Graph Isomorphism Network (GIN) (Xu et al., 2018).

Original: For further details on GNNs see Appendix A.
Revised: For further details on GNNs see Appendix A.

Original: In this work we use the most expressive GNN variants that use the ”sum” aggregation function.
Revised: In this work we use the most expressive GNN variants that use the ”sum” aggregation function.

Original: Using a ”max” or ”mean” reduces the expressive power of the network, making it not powerful enough to solve simple counting problems (e.g. counting edges or computing node degrees).
Revised: Using a ”max” or ”mean” reduces the expressive power of the network, making it not powerful enough to solve simple counting problems (e.g. counting edges or computing node degrees).

Original: On the other hand, these networks give rise to slightly different deﬁnitions of patterns and can generalize better in some cases as shown in (Veli ˇ ckovi ´ c et al., 2019), yet still suffer from size overﬁt.
Revised: On the other hand, these networks give rise to slightly different deﬁnitions of patterns and can generalize better in some cases as shown in (Veli ˇ ckovi ´ c et al., 2019), yet still suffer from size overﬁt.

Original: Exploring these networks is beyond the scope of this work.
Revised: Exploring these networks is beyond the scope of this work.

Original: 3. S IZE GENERALIZATION IN S INGLE - LAYER GNN S
Revised: 3. S IZE GENERALIZATION IN S INGLE - LAYER GNN S

Original: We start our discussion on size generalization with a theoretical analysis of a simple setup.
Revised: We start our discussion on size generalization with a theoretical analysis of a simple setup.

Original: We consider a single-layer GNN and an easy task and show that: (1) The training objective has many different solutions, but only a small subset of these solutions generalizes to larger graphs (2)
Revised: We consider a single-layer GNN and an easy task and show that: (1) The training objective has many different solutions, but only a small subset of these solutions generalizes to larger graphs (2)

Original: Simple regularization techniques cannot mitigate the problem.
Revised: Simple regularization techniques cannot mitigate the problem.

Original: This subsection serves as a warm-up for the next subsections that contain our main results.
Revised: This subsection serves as a warm-up for the next subsections that contain our main results.

Original: Assume we train on a distribution of graphs with a ﬁxed number of nodes n and a ﬁxed number of edges m .
Revised: Assume we train on a distribution of graphs with a ﬁxed number of nodes n and a ﬁxed number of edges m .

Original: Our goal is to predict the number of edges in the graph using a 1-GNN with a single linear layer and additive readout function, for simplicity also consider the squared loss.
Revised: Our goal is to predict the number of edges in the graph using a 1-GNN with a single linear layer and additive readout function, for simplicity also consider the squared loss.

Original: The objective boils down to the following function for any graph G in the training set::
Revised: The objective boils down to the following function for any graph G in the training set::

Original: [Equation]
Revised: L ( w 1 , w 2 , b ; G ) =

Original:  w 1 · x u + (cid:80) v ∈N ( u ) w 2 · x v + b (cid:17) − y (cid:17) 2 .
Revised: (cid:16)(cid:80) u ∈ V ( G ) (cid:16) w 1 · x u + (cid:80) v ∈N ( u ) w 2 · x v + b (cid:17) − y (cid:17) 2 .

Original: Here, G is an input graph, V ( G ) are the nodes of G , N ( v ) are all the neighbors of node v , w 1 , w 

Revised: Here, G is an input graph, V ( G ) are the nodes of G , N ( v ) are all the neighbors of node v , w 1 , w

Original: and b are the trainable parameters, y is the target ( m in this case) and x v is the node feature for node v .
Revised: and b are the trainable parameters, y is the target ( m in this case) and x v is the node feature for node v .

Original: Further, assume that we have no additional information on the nodes, so we can just embed each node as a one-dimensional feature vector with a ﬁxed value of 1 .
Revised: Further, assume that we have no additional information on the nodes, so we can just embed each node as a one-dimensional feature vector with a ﬁxed value of 1 .

Original: In this simple case, the trainable parameters are also one-dimensional.
Revised: In this simple case, the trainable parameters are also one-dimensional.

Original: We note that the training objective can also be written in the following form L ( w 1 , w 2 , b ; G )
Revised: We note that the training objective can also be written in the following form L ( w 1 , w 2 , b ; G )

Original: = ( nw 1 + 2 mw 2 + nb − m ) 2 , and that one can easily ﬁnd its solutions space, which is an afﬁne subspace deﬁned by w 2 = m − n ( w 1 + b ) 2 m .
Revised: = ( nw 1 + 2 mw 2 + nb − m ) 2 , and that one can easily ﬁnd its solutions space, which is an afﬁne subspace deﬁned by w 2 = m − n ( w 1 + b ) 2 m .

Original: In particular, the solutions with b + w 1 = 0 , w 2 = 1 / 2 are the only ones which do not depend on the speciﬁc training set graph size n , and generalize to graphs of any size.
Revised: In particular, the solutions with b + w 1 = 0 , w 2 = 1 / 2 are the only ones which do not depend on the speciﬁc training set graph size n , and generalize to graphs of any size.

Original: It can be readily seen that when training the model on graphs of ﬁxed size (ﬁxed m, n ), gradient descent will have no reason to favor one solution over another and we will not be able to generalize.
Revised: It can be readily seen that when training the model on graphs of ﬁxed size (ﬁxed m, n ), gradient descent will have no reason to favor one solution over another and we will not be able to generalize.

Original: We also note that the generalizing solution is not always the least norm solution (with respect to both L 1 and L 2 norms) so simple regularization will not help here.
Revised: We also note that the generalizing solution is not always the least norm solution (with respect to both L 1 and L 2 norms) so simple regularization will not help here.

Original: On the other hand, it is easy to show that training on graphs with different number of edges will favor the generalizing solution.
Revised: On the other hand, it is easy to show that training on graphs with different number of edges will favor the generalizing solution.

Original: As we will see next, the problem gets worse when considering
Revised: As we will see next, the problem gets worse when considering

Original: GNNs with multiple non-linear layers, and this simple solution will not help in this case: we can train deeper GNNs on a wide variety of sizes and the solution will not generalize to other sizes.
Revised: GNNs with multiple non-linear layers, and this simple solution will not help in this case: we can train deeper GNNs on a wide variety of sizes and the solution will not generalize to other sizes.

Original: 3. d - PATTERNS
Revised: 3. d - PATTERNS

Original: We wish to understand theoretically when does a GNN which was trained on on graphs with a small number of nodes can generalize to graphs with a large number of nodes.
Revised: We wish to understand theoretically when does a GNN which was trained on on graphs with a small number of nodes can generalize to graphs with a large number of nodes.

Original: To answer that question, we ﬁrst analyze what information is received by each node in the graph from its neighboring nodes after a graph is processed by a GNN with T layers.
Revised: To answer that question, we ﬁrst analyze what information is received by each node in the graph from its neighboring nodes after a graph is processed by a GNN with T layers.

Original: It is easy to see that every node can receive information about its neighbors which are at most T hops away.
Revised: It is easy to see that every node can receive information about its neighbors which are at most T hops away.

Original: We also know that nodes do not have full information about their order T environment.
Revised: We also know that nodes do not have full information about their order T environment.

Original: For example, GNNs cannot determine if a triangle is present in a neighborhood of a given node Chen et al.
Revised: For example, GNNs cannot determine if a triangle is present in a neighborhood of a given node Chen et al.

Original: In order to characterize the exact information that can be found in each node after a T layer GNN, we use the deﬁnition of the WL test, speciﬁcally its iteration structure, which has the same representational power as GNNs (see Xu et al.
Revised: In order to characterize the exact information that can be found in each node after a T layer GNN, we use the deﬁnition of the WL test, speciﬁcally its iteration structure, which has the same representational power as GNNs (see Xu et al.

Original: ; Morris et al.
Revised: ; Morris et al.

Original: , For more details on the WL test see Appendix A.
Revised: , For more details on the WL test see Appendix A.

Original: Deﬁnition 3.1 ( d -patterns) .
Revised: Deﬁnition 3.1 ( d -patterns) .

Original: Let C be a ﬁnite set of node features and N ∈ N .
Revised: Let C be a ﬁnite set of node features and N ∈ N .

Original: For d ≥ 

Revised: For d ≥

Original: we deﬁne the set of d -patterns P d on graphs with maximal degree N and node features from C .
Revised: we deﬁne the set of d -patterns P d on graphs with maximal degree N and node features from C .

Original: The deﬁnition is recursive in the following way: For d = 0 , P 0 = C .
Revised: The deﬁnition is recursive in the following way: For d = 0 , P 0 = C .

Original: We deﬁne P d to be the set of all tuples ( a, b ) where a ∈ P d − 1 and b is in multisets of size at most N consisting of elements from P d − 1 .
Revised: We deﬁne P d to be the set of all tuples ( a, b ) where a ∈ P d − 1 and b is in multisets of size at most N consisting of elements from P d − 1 .

Original: Let G =
Revised: Let G =

Original: ( V, E ) be a graph with maximal degree N and a node feature c v ∈ C for every node v ∈ V .
Revised: ( V, E ) be a graph with maximal degree N and a node feature c v ∈ C for every node v ∈ V .

Original: We deﬁne the d -pattern of a node v ∈ V for d ≥ 0 recursively:
Revised: We deﬁne the d -pattern of a node v ∈ V for d ≥ 0 recursively:

Original: For d = 0 , its 0pattern is c v .
Revised: For d = 0 , its 0pattern is c v .

Original: For d > 0 we say that v with (cid:96) neighboring d − 1 patterns has a d -pattern p
Revised: For d > 0 we say that v with (cid:96) neighboring d − 1 patterns has a d -pattern p

Original: =
Revised: =

Original: ( p v , { ( p i 1 , m p i 1 ) , . . .
Revised: ( p v , { ( p i 1 , m p i 1 ) , . . .

Original: , ( p i (cid:96) , m p i(cid:96) ) } )
Revised: , ( p i (cid:96) , m p i(cid:96) ) } )

Original: iff node v has ( d − 1) -pattern
Revised: iff node v has ( d − 1) -pattern

Original: p v and for every j ∈ { 1 , . . .
Revised: p v and for every j ∈ { 1 , . . .

Original: , (cid:96) } the number of neighbors of v with ( d − 1) -pattern p
Revised: , (cid:96) } the number of neighbors of v with ( d − 1) -pattern p

Original: i j is exactly m p ij .
Revised: i j is exactly m p ij .

Original: The d -pattern of a node is an encoding of the ( d − 1) -patterns of itself and of its neighbors.
Revised: The d -pattern of a node is an encoding of the ( d − 1) -patterns of itself and of its neighbors.

Original: For example, assume a graph has a maximal degree of N and all the nodes start with the same node feature.
Revised: For example, assume a graph has a maximal degree of N and all the nodes start with the same node feature.

Original: The 1 -pattern of each node is its degree.
Revised: The 1 -pattern of each node is its degree.

Original: The 2 -pattern of each node is for each possible degree i ∈ { 1 , . .
Revised: The 2 -pattern of each node is for each possible degree i ∈ { 1 , . .

Original: , N } the number of neighbors with degree i .
Revised: , N } the number of neighbors with degree i .

Original: In the same manner, the 3 -pattern of a node is for each possible 2 -pattern, the number of its neighbors with this exact 2 -pattern.
Revised: In the same manner, the 3 -pattern of a node is for each possible 2 -pattern, the number of its neighbors with this exact 2 -pattern.

Original: The deﬁnition of d -patterns can naturally be extended to the case of unbounded degrees.
Revised: The deﬁnition of d -patterns can naturally be extended to the case of unbounded degrees.

Original: We have the following theorem which connects the d -patterns with the expressive power of GNNs:
Revised: We have the following theorem which connects the d -patterns with the expressive power of GNNs:

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Any function that can be represented by a d -layer GNN is constant on d -patterns.
Revised: Any function that can be represented by a d -layer GNN is constant on d -patterns.

Original: In particular, the theorem shows that for any two graphs (of any size) and two nodes, one in each graph, if the nodes have the exact same d -pattern, then any d -layer GNN will output the same result for the two nodes.
Revised: In particular, the theorem shows that for any two graphs (of any size) and two nodes, one in each graph, if the nodes have the exact same d -pattern, then any d -layer GNN will output the same result for the two nodes.

Original: The full proof can be found in Appendix B, and follows directly from the analogy between the WL algorithm (see Appendix A) and d -patterns.
Revised: The full proof can be found in Appendix B, and follows directly from the analogy between the WL algorithm (see Appendix A) and d -patterns.

Original: Thm. 3.2 implies that d -patterns don’t represent more expressive power than GNN.
Revised: Thm. 3.2 implies that d -patterns don’t represent more expressive power than GNN.

Original: In the next subsection, we prove that GNNs can exactly compute d -patterns, and show that this capacity is tightly related to size generalization.
Revised: In the next subsection, we prove that GNNs can exactly compute d -patterns, and show that this capacity is tightly related to size generalization.

Original: It is also easy to see from the deﬁnition of d -patterns and the proof of Theorem 2 from Morris et al.
Revised: It is also easy to see from the deﬁnition of d -patterns and the proof of Theorem 2 from Morris et al.

Original: (2019) that d -patterns exactly represent the expressive power of GNNs (with additive aggregation), thus this deﬁnition is a natural tool to study the properties of GNNs, such as size generalization.
Revised: (2019) that d -patterns exactly represent the expressive power of GNNs (with additive aggregation), thus this deﬁnition is a natural tool to study the properties of GNNs, such as size generalization.

Original: 3. GNN S MAY OVERFIT d - PATTERNS
Revised: 3. GNN S MAY OVERFIT d - PATTERNS

Original: We can now connect the size generalization problem to the concept of d -patterns.
Revised: We can now connect the size generalization problem to the concept of d -patterns.

Original: We start with an example: consider a node prediction task in which an output is speciﬁed for each node in an input graph, and is solvable by a d -layer GNN.
Revised: We start with an example: consider a node prediction task in which an output is speciﬁed for each node in an input graph, and is solvable by a d -layer GNN.

Original: To perfectly solve this task, the model should produce the correct output for the d -pattern of all the nodes in the training set.
Revised: To perfectly solve this task, the model should produce the correct output for the d -pattern of all the nodes in the training set.

Original: Testing this GNN on a different set of graphs will succeed if the test set has graphs with similar d -patterns to those in the training set.
Revised: Testing this GNN on a different set of graphs will succeed if the test set has graphs with similar d -patterns to those in the training set.

Original: Note that this requirement is not related to the size of the graphs but to the distribution of the d -patterns of the nodes in the test set.
Revised: Note that this requirement is not related to the size of the graphs but to the distribution of the d -patterns of the nodes in the test set.

Original: In the following theorem we show rigorously, that given a set of d -patterns and output for each such pattern, there is an assignment of weights to a GNN with O ( d ) layers that perfectly ﬁts the output for each pattern.
Revised: In the following theorem we show rigorously, that given a set of d -patterns and output for each such pattern, there is an assignment of weights to a GNN with O ( d ) layers that perfectly ﬁts the output for each pattern.

Original: We will then use this theorem in order to show that, under certain assumptions on the distribution of d -patterns of the large graphs, GNNs can perfectly solve a task on a set of small graphs, and completely fail on a set on large graphs.
Revised: We will then use this theorem in order to show that, under certain assumptions on the distribution of d -patterns of the large graphs, GNNs can perfectly solve a task on a set of small graphs, and completely fail on a set on large graphs.

Original: In other words, we show that there are multiple global minima for the training objective that do not generalize to larger graphs.
Revised: In other words, we show that there are multiple global minima for the training objective that do not generalize to larger graphs.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Let C be a ﬁnite set of node features, P be a ﬁnite set of d -patterns on graphs with maximal degree N ∈ N , and for each pattern p ∈ P let y p ∈
Revised: Let C be a ﬁnite set of node features, P be a ﬁnite set of d -patterns on graphs with maximal degree N ∈ N , and for each pattern p ∈ P let y p ∈

Original: [ − 1 , 1] be some target label.
Revised: [ − 1 , 1] be some target label.

Original: Then there exists a 1-GNN F with d + 2 layers, width bounded by max (cid:110)
Revised: Then there exists a 1-GNN F with d + 2 layers, width bounded by max (cid:110)

Original: ( N + 1) d · | C | , 2 (cid:112) | P | (cid:111) and ReLU activation such that for every graph G with nodes v 1 , . .
Revised: ( N + 1) d · | C | , 2 (cid:112) | P | (cid:111) and ReLU activation such that for every graph G with nodes v 1 , . .

Original: . , v n , corresponding d -patterns p 1 , . . .
Revised: . , v n , corresponding d -patterns p 1 , . . .

Original: , p n ⊆ P and node features from C , the output of F on node v
Revised: , p n ⊆ P and node features from C , the output of F on node v

Original: i is exactly y p i .
Revised: i is exactly y p i .

Original: The full proof is in Appendix B. Note that the width of the required GNN from the theorem is not very large if d is small, where d represents the depth of the 1-GNN.
Revised: The full proof is in Appendix B. Note that the width of the required GNN from the theorem is not very large if d is small, where d represents the depth of the 1-GNN.

Original: In practice, shallow GNNs are very commonly used and are proven empirically successful, while training deep GNNs was shown to be hard due to many problems like over-smoothing (Zhao and Akoglu, 2019).
Revised: In practice, shallow GNNs are very commonly used and are proven empirically successful, while training deep GNNs was shown to be hard due to many problems like over-smoothing (Zhao and Akoglu, 2019).

Original: Using the above theorem we can claim that there are assignments of weights to GNN that cannot ”size-generalize”, that is, given a speciﬁc task, the GNN succeeds on the task for small graphs (up to some bound) and fails on larger graphs, as long as there is a notable discrepancy between their d -patterns distributions:
Revised: Using the above theorem we can claim that there are assignments of weights to GNN that cannot ”size-generalize”, that is, given a speciﬁc task, the GNN succeeds on the task for small graphs (up to some bound) and fails on larger graphs, as long as there is a notable discrepancy between their d -patterns distributions:

Original: Corollary 3.4.
Revised: Corollary 3.4.

Original: Let P 1 and P 2 be distributions of small and large graphs respectively with ﬁnite support, and let P d − patbe the distribution of d -patterns over small graphs and similarly P
Revised: Let P 1 and P 2 be distributions of small and large graphs respectively with ﬁnite support, and let P d − patbe the distribution of d -patterns over small graphs and similarly P

Original: d − patfor large graphs.
Revised: d − patfor large graphs.

Original: For any node prediction task which is solvable by a 1-GNN with depth d and (cid:15) >there exists a 1-GNN with depth at most d + 2 that has 0-1 loss smaller then (cid:15) on P 1 and 0-1 loss ∆ on P 2 , where
Revised: For any node prediction task which is solvable by a 1-GNN with depth d and (cid:15) >there exists a 1-GNN with depth at most d + 2 that has 0-1 loss smaller then (cid:15) on P 1 and 0-1 loss ∆ on P 2 , where

Original: [Equation]
Revised: [Equation]

Original: Here, A is a set of d -patterns and P ( A ) is the total probability mass for that set under P .
Revised: Here, A is a set of d -patterns and P ( A ) is the total probability mass for that set under P .

Original: Intuitively, large ∆ means that there exists a set of d-patterns that have a low probability for small graphs and high probability for large graphs.
Revised: Intuitively, large ∆ means that there exists a set of d-patterns that have a low probability for small graphs and high probability for large graphs.

Original: Corollary 3.4 implies that the major factor in the success of GNN to generalize to larger graphs is not the graph size, but the distribution of the dpatterns.
Revised: Corollary 3.4 implies that the major factor in the success of GNN to generalize to larger graphs is not the graph size, but the distribution of the dpatterns.

Original: Different distributions of d -patterns lead to large ∆ and thus to bad generalization to larger graphs.
Revised: Different distributions of d -patterns lead to large ∆ and thus to bad generalization to larger graphs.

Original: On the other hand, from Thm.
Revised: On the other hand, from Thm.

Original: 3.2 we immediately get that similar distributions of d -patterns imply that every GNN model that succeeds on small graphs will also succeed on large graphs, since
Revised: 3.2 we immediately get that similar distributions of d -patterns imply that every GNN model that succeeds on small graphs will also succeed on large graphs, since

Original: GNNs are constant on d -patterns:
Revised: GNNs are constant on d -patterns:

Original: Corollary 3.5.
Revised: Corollary 3.5.

Original: In the setting of Corollary 3.4, also assume that all the patterns that have a positive probability in P d − patalso have a positive probability in P d − pat.
Revised: In the setting of Corollary 3.4, also assume that all the patterns that have a positive probability in P d − patalso have a positive probability in P d − pat.

Original: Then, for any node prediction task solvable by a depth d GNN, any 1 − GNN that have 0 loss (w.r.t the 0 − 1 loss) on P 1 will also have 0 loss on P 2 .
Revised: Then, for any node prediction task solvable by a depth d GNN, any 1 − GNN that have 0 loss (w.r.t the 0 − 1 loss) on P 1 will also have 0 loss on P 2 .

Original: Examples.
Revised: Examples.

Original: Corollary 3.4 shows that even for simple tasks, GNN may fail, here are two simple examples.
Revised: Corollary 3.4 shows that even for simple tasks, GNN may fail, here are two simple examples.

Original: (i) Consider the task of calculating the node degree.
Revised: (i) Consider the task of calculating the node degree.

Original: From Corollary 3.4 there is a GNN that successfully output the degree of nodes with max degree up to N and fails on nodes with larger degrees.
Revised: From Corollary 3.4 there is a GNN that successfully output the degree of nodes with max degree up to N and fails on nodes with larger degrees.

Original: Note that this problem can easily be solved for any node degree with a 1 -layer GNN.
Revised: Note that this problem can easily be solved for any node degree with a 1 -layer GNN.

Original: (ii)
Revised: (ii)

Original: Consider some node regression task, when the training set consists of graphs sampled i.i.d from an
Revised: Consider some node regression task, when the training set consists of graphs sampled i.i.d from an

Original: Erdos-Renyi graph G ( n, p ) 1 , and the test set contains graphs sampled i.i.d from G (2 n, p ) .
Revised: Erdos-Renyi graph G ( n, p ) 1 , and the test set contains graphs sampled i.i.d from G (2 n, p ) .

Original: In this case, a GNN trained on the training set will be trained on graphs with an average degree np , while the test set contains graphs with an average degree 2 np .
Revised: In this case, a GNN trained on the training set will be trained on graphs with an average degree np , while the test set contains graphs with an average degree 2 np .

Original: This means that the d -patterns in the train and test set are very different, and by Corollary 3.4 the GNN may overﬁt.
Revised: This means that the d -patterns in the train and test set are very different, and by Corollary 3.4 the GNN may overﬁt.

Original: 1 Graphs with n nodes such that each edge exists with probability p .

Original: Graph prediction tasks.
Revised: Graph prediction tasks.

Original: Our theoretical results discuss node prediction tasks.
Revised: Our theoretical results discuss node prediction tasks.

Original: We note that they are also relevant for graph prediction tasks where there is a single output to each input graph.
Revised: We note that they are also relevant for graph prediction tasks where there is a single output to each input graph.

Original: The reason for that is that in order to solve graph prediction tasks, a GNN ﬁrst calculates node features and then pools them into a single global graph feature.
Revised: The reason for that is that in order to solve graph prediction tasks, a GNN ﬁrst calculates node features and then pools them into a single global graph feature.

Original: Our analysis shows that the ﬁrst part of the
Revised: Our analysis shows that the ﬁrst part of the

Original: GNN, which is responsible for calculating the node features, might not generalize to large graphs.
Revised: GNN, which is responsible for calculating the node features, might not generalize to large graphs.

Original: As a result, the GNN will generate an uninformative global graph feature and the GNN will fail on the original graph prediction task.
Revised: As a result, the GNN will generate an uninformative global graph feature and the GNN will fail on the original graph prediction task.

Original: In the experimental sections, we show that the size generalization problem is indeed relevant for both node and graph prediction tasks.
Revised: In the experimental sections, we show that the size generalization problem is indeed relevant for both node and graph prediction tasks.

Original: Here is a formal statement regarding graph prediction tasks, the full proof can be found in Appendix B.
Revised: Here is a formal statement regarding graph prediction tasks, the full proof can be found in Appendix B.

Original: Corollary 3.6.
Revised: Corollary 3.6.

Original: Let P 1 and P 2 be distributions of small and large graphs respectively with ﬁnite support.
Revised: Let P 1 and P 2 be distributions of small and large graphs respectively with ﬁnite support.

Original: Let P d − patbe the distribution of d -patterns over small graphs and similarly P
Revised: Let P d − patbe the distribution of d -patterns over small graphs and similarly P

Original: d − patfor large graphs, and assume that the supports of P d − patand P d − patare disjoint.
Revised: d − patfor large graphs, and assume that the supports of P d − patand P d − patare disjoint.

Original: For any graph prediction task solvable by a 1-GNN with depth d and summation readout function, there exists a 1-GNN with depth at most d + 3 that perfectly solves the task on P 1 and fails on all graphs from P 2 .
Revised: For any graph prediction task solvable by a 1-GNN with depth d and summation readout function, there exists a 1-GNN with depth at most d + 3 that perfectly solves the task on P 1 and fails on all graphs from P 2 .

Original: Relation to Morris et al.
Revised: Relation to Morris et al.

Original: ; Xu et al.
Revised: ; Xu et al.

Original: We note that Theorem 3.3 and Corollary 3.4 are somewhat related to the expressivity results in (Xu et al., 2018; Morris et al., 2019) that show that
Revised: We note that Theorem 3.3 and Corollary 3.4 are somewhat related to the expressivity results in (Xu et al., 2018; Morris et al., 2019) that show that

Original: GNNs can be as powerful as the WL test.
Revised: GNNs can be as powerful as the WL test.

Original: Here, we show that the expressive power of GNNs can cause negative effects when there is a discrepancy between the training and test sets.
Revised: Here, we show that the expressive power of GNNs can cause negative effects when there is a discrepancy between the training and test sets.

Original: 3. E MPIRICAL V ALIDATION
Revised: 3. E MPIRICAL V ALIDATION

Original: In the previous subsection we have shown that for any node task, and any two datasets of graphs with different sizes that signiﬁcantly differ in their d -patterns distributions, there is a 1-GNN that successfully solves the task on one dataset but fails on the second.
Revised: In the previous subsection we have shown that for any node task, and any two datasets of graphs with different sizes that signiﬁcantly differ in their d -patterns distributions, there is a 1-GNN that successfully solves the task on one dataset but fails on the second.

Original: In this subsection, we show empirically that reaching these ”overﬁtting” GNNs is actually very common.
Revised: In this subsection, we show empirically that reaching these ”overﬁtting” GNNs is actually very common.

Original: Speciﬁcally, the sizeoverﬁt phenomenon is prevalent when the d -patterns of in the large graph distribution are not found in the small graph distribution.
Revised: Speciﬁcally, the sizeoverﬁt phenomenon is prevalent when the d -patterns of in the large graph distribution are not found in the small graph distribution.

Original: We also show that GNNs can generalize to larger graphs if the distribution of d -patterns remains similar to the distribution of patterns in the small graphs.
Revised: We also show that GNNs can generalize to larger graphs if the distribution of d -patterns remains similar to the distribution of patterns in the small graphs.

Original: To show this, we use a controlled regression task in a student-teacher setting.
Revised: To show this, we use a controlled regression task in a student-teacher setting.

Original: In this setting, we sample a ”teacher” GNN with random weights, freeze the network, and label each graph in the dataset using the output of the ”teacher” network.
Revised: In this setting, we sample a ”teacher” GNN with random weights, freeze the network, and label each graph in the dataset using the output of the ”teacher” network.

Original: Our goal is to train a ”student” network, which has the same architecture as the ”teacher” network, to ﬁt the labels of the teacher network.
Revised: Our goal is to train a ”student” network, which has the same architecture as the ”teacher” network, to ﬁt the labels of the teacher network.

Original: The advantages of this setting are two-fold: (1) A solution is guaranteed to exist : We know that there is a weight assignment of the student network which perfectly solve the task for graphs of any size.
Revised: The advantages of this setting are two-fold: (1) A solution is guaranteed to exist : We know that there is a weight assignment of the student network which perfectly solve the task for graphs of any size.

Original: Generality : It includes all tasks solvable by constant depth GNNs.
Revised: Generality : It includes all tasks solvable by constant depth GNNs.

Original: We discuss more settings below.
Revised: We discuss more settings below.

Original: Architecture and training protocol.
Revised: Architecture and training protocol.

Original: We use 1-GNN as deﬁned in (Morris et al., 2019).
Revised: We use 1-GNN as deﬁned in (Morris et al., 2019).

Original: The number of GNN layers in the network we use is either 1 , 2 or 3 ; the width of the teacher network is 32 , and of the student network 64 , providing more expressive power to the student network.
Revised: The number of GNN layers in the network we use is either 1 , 2 or 3 ; the width of the teacher network is 32 , and of the student network 64 , providing more expressive power to the student network.

Original: We obtained similar results when testing with a width of 32, same as the teacher network.
Revised: We obtained similar results when testing with a width of 32, same as the teacher network.

Original: We use a summation readout function followed by a two-layer fully connected sufﬁx.
Revised: We use a summation readout function followed by a two-layer fully connected sufﬁx.

Original: We use ADAM with learning rate 10 − 3 .
Revised: We use ADAM with learning rate 10 − 3 .

Original: We performed a hyper-parameters search on the learning rate and weight decay and use validation-based early stopping on the source domain (small graphs).
Revised: We performed a hyper-parameters search on the learning rate and weight decay and use validation-based early stopping on the source domain (small graphs).

Original: The results are averaged over 10 random seeds.
Revised: The results are averaged over 10 random seeds.

Original: All runs used Pytorch Geometric (Fey and Lenssen, 2019) on NVIDIA DGX-1.
Revised: All runs used Pytorch Geometric (Fey and Lenssen, 2019) on NVIDIA DGX-1.

Original: Results.
Revised: Results.

Original: Fig.
Revised: Fig.

Original: 1 compares the loss of GNNs as the distribution of d -patterns changes, for the task of teacher-student graph level regression.
Revised: 1 compares the loss of GNNs as the distribution of d -patterns changes, for the task of teacher-student graph level regression.

Original: The model was trained on graphs generated using the
Revised: The model was trained on graphs generated using the

Original: G ( n, p ) model.
Revised: G ( n, p ) model.

Original: We show the normalized L 2 loss computed on test, where output is normalized by the average test-set (target) output.
Revised: We show the normalized L 2 loss computed on test, where output is normalized by the average test-set (target) output.

Original: The left panel shows the test loss when training on n ∈ [40 , 50] and p = 0 .
Revised: The left panel shows the test loss when training on n ∈ [40 , 50] and p = 0 .

Original: 3 and testing on G ( n, p ) graphs with n = 100 and p varying from 0 .
Revised: 3 and testing on G ( n, p ) graphs with n = 100 and p varying from 0 .

Original: 05 to 0 .
Revised: 05 to 0 .

Original: In this experiment, the expected node degree is np , hence the distribution of d -patterns is most similar to the one observed in the training set when p = 0 .
Revised: In this experiment, the expected node degree is np , hence the distribution of d -patterns is most similar to the one observed in the training set when p = 0 .

Original: Indeed, this is the value of p where the test loss is minimized.
Revised: Indeed, this is the value of p where the test loss is minimized.

Original: The right panel is discussed in the caption.
Revised: The right panel is discussed in the caption.

Original: These results are consistent with 
Revised: These results are consistent with 1 Graphs with n nodes such that each edge exists with probability p . [Figure] [Figure]

Original: Corollary 3.4, since when the distributions of d -patterns are far the model is not able to generalize well, and it does generalize well when these distributions are similar.
Revised: Corollary 3.4, since when the distributions of d -patterns are far the model is not able to generalize well, and it does generalize well when these distributions are similar.

Original: To give further conﬁrmation to the effect of the local distributions on the generalization capabilities of GNN we conducted the following two experiments: (1) We tested on the teacher-student setup with a 3-layer GNN on graphs of sizes uniformly from n = 40 , . .
Revised: To give further conﬁrmation to the effect of the local distributions on the generalization capabilities of GNN we conducted the following two experiments: (1) We tested on the teacher-student setup with a 3-layer GNN on graphs of sizes uniformly from n = 40 , . .

Original: , 50 and sampled from G ( n, 0 . 3) .
Revised: , 50 and sampled from G ( n, 0 . 3) .

Original: [Figure]

Original: [Figure] We tested on graphs sampled from G ( N, 0 . 3) where N varies from 50 up to 150 .
Revised: We tested on graphs sampled from G ( N, 0 . 3) where N varies from 50 up to 150 .

Original: It is evident that as the graph size in the test set increases, the model performs worse.
Revised: It is evident that as the graph size in the test set increases, the model performs worse.

Original: (2) We did the same test as in (1), but this time we normalize p on the test set so that p · N = 15 , which is the approximate ratio for the training set.
Revised: (2) We did the same test as in (1), but this time we normalize p on the test set so that p · N = 15 , which is the approximate ratio for the training set.

Original: Here we even went further to train up to sizes N = 250 .
Revised: Here we even went further to train up to sizes N = 250 .

Original: In this experiment, the
Revised: In this experiment, the

Original: GNN successfully generalized to larger graphs, since the local distributions of the train and test set are indeed very similar.
Revised: GNN successfully generalized to larger graphs, since the local distributions of the train and test set are indeed very similar.

Original: For the results see Fig.
Revised: For the results see Fig.

Original: We also tested on the tasks of ﬁnding the max clique in a graph, calculating the number of edges, and the node prediction tasks of calculating the node degree, and the student-teacher task at the node level.
Revised: We also tested on the tasks of ﬁnding the max clique in a graph, calculating the number of edges, and the node prediction tasks of calculating the node degree, and the student-teacher task at the node level.

Original: In addition, we tested on the popular GIN architecture (Xu et al., 2018), and show that the size generalization problem also occurs there.
Revised: In addition, we tested on the popular GIN architecture (Xu et al., 2018), and show that the size generalization problem also occurs there.

Original: We also tested on ReLU, Tanh, and sigmoid activations.
Revised: We also tested on ReLU, Tanh, and sigmoid activations.

Original: See additional experiments in Appendix C.
Revised: See additional experiments in Appendix C.

Original:  T OWARDS IMPROVING SIZE - GENERALIZATION
Revised:  T OWARDS IMPROVING SIZE - GENERALIZATION

Original: The results from the previous section show that the problem of size generalization is not only related to the size of the graph in terms of the number of nodes or edges but to the distribution of d -patterns induced by the distributions from which the graphs are sampled.
Revised: The results from the previous section show that the problem of size generalization is not only related to the size of the graph in terms of the number of nodes or edges but to the distribution of d -patterns induced by the distributions from which the graphs are sampled.

Original: Based on this observation, we now formulate the size generalization problem as a domain adaptation (DA) problem.
Revised: Based on this observation, we now formulate the size generalization problem as a domain adaptation (DA) problem.

Original: We then build on techniques from domain adaptation and suggest two approaches to improve size generalization.
Revised: We then build on techniques from domain adaptation and suggest two approaches to improve size generalization.

Original: Self-supervised learning on the target domain (large graphs) and (2) Semi-supervised learning with a few labeled target samples.
Revised: Self-supervised learning on the target domain (large graphs) and (2) Semi-supervised learning with a few labeled target samples.

Original: We consider the DA setting where we are given two distributions over graphs: a source distribution D S (say, for small graphs) and a target distribution D T (say, for large graphs).
Revised: We consider the DA setting where we are given two distributions over graphs: a source distribution D S (say, for small graphs) and a target distribution D T (say, for large graphs).

Original: We consider two settings.
Revised: We consider two settings.

Original: First, the unlabeled DA setting, where we have access to labeled samples from the source D S but the target data from D T is unlabeled.
Revised: First, the unlabeled DA setting, where we have access to labeled samples from the source D S but the target data from D T is unlabeled.

Original: Our goal is to infer labels on a test dataset sampled from the target D T .
Revised: Our goal is to infer labels on a test dataset sampled from the target D T .

Original: Second, we consider a semi-supervised setup, where we also have access to a small number of labeled examples from the target D T .
Revised: Second, we consider a semi-supervised setup, where we also have access to a small number of labeled examples from the target D T .

Original: Size generalization with Self-supervised learning.
Revised: Size generalization with Self-supervised learning.

Original: In Self-supervised learning (SSL) for DA, a model is trained on unlabeled data to learn a pretext task, which is different from the main task at hand.
Revised: In Self-supervised learning (SSL) for DA, a model is trained on unlabeled data to learn a pretext task, which is different from the main task at hand.

Original: If the pretext task is chosen wisely, the model learns useful representations (Doersch et al., 2015; Gidaris et al., 2018) that can help with the main task.
Revised: If the pretext task is chosen wisely, the model learns useful representations (Doersch et al., 2015; Gidaris et al., 2018) that can help with the main task.

Original: Here, we train the pretext task on both the source and target domains, as was done for images and point clouds (Sun et al., 2019; Achituve et al., 2020).
Revised: Here, we train the pretext task on both the source and target domains, as was done for images and point clouds (Sun et al., 2019; Achituve et al., 2020).

Original: The idea is that the pretext task aligns the representations of the source and target domains leading to better predictions of the main task fortarget graphs.
Revised: The idea is that the pretext task aligns the representations of the source and target domains leading to better predictions of the main task for graphs.

Original: { For a detailed review of the training procedures and the losses see
Revised: { For a detailed review of the training procedures and the losses see

Original: Appendix E. Given a pretext task we consider two different training procedures: (1) Multi-task learning (MTL) : parallel training of the main task on a source domain and a pretext task on both the source of target domain (You et al., 2020).
Revised: Appendix E. Given a pretext task we consider two different training procedures: (1) Multi-task learning (MTL) : parallel training of the main task on a source domain and a pretext task on both the source of target domain (You et al., 2020).

Original: In this case, the architecture consists of a main GNN that acts as a feature extractor and two secondary networks (heads) that operate on the extracted features and try to predict the main task and the pretext task.
Revised: In this case, the architecture consists of a main GNN that acts as a feature extractor and two secondary networks (heads) that operate on the extracted features and try to predict the main task and the pretext task.

Original: (2) Pretraining (PT) : in this procedure (Hu et al., 2019), the GNN feature extractor is trained until convergence on the pretext task on both the source and target examples.
Revised: (2) Pretraining (PT) : in this procedure (Hu et al., 2019), the GNN feature extractor is trained until convergence on the pretext task on both the source and target examples.

Original: Then, the GNN part is frozen, and only the head of the model is trained on labeled examples from the source.
Revised: Then, the GNN part is frozen, and only the head of the model is trained on labeled examples from the source.

Original: Pattern-tree pretext task.
Revised: Pattern-tree pretext task.

Original: We propose a novel pretext task which is motivated by the deﬁnition of d -patterns.
Revised: We propose a novel pretext task which is motivated by the deﬁnition of d -patterns.

Original: We do that by constructing a tree that fully represents the d -patterns of each node (see e.g., Xu et al.
Revised: We do that by constructing a tree that fully represents the d -patterns of each node (see e.g., Xu et al.

Original: We then calculate a descriptor of the tree, which is a vector containing counts of the number of nodes from each class in each layer of the tree.
Revised: We then calculate a descriptor of the tree, which is a vector containing counts of the number of nodes from each class in each layer of the tree.

Original: We treat this descriptor as the target label to be reconstructed by the SSL task.
Revised: We treat this descriptor as the target label to be reconstructed by the SSL task.

Original: For more details see Figure 3.
Revised: For more details see Figure 3.

Original: Intuitively, in order to be successful on a task on graphs, the GNN needs to correctly represent the pattern trees of the nodes of the graphs.
Revised: Intuitively, in order to be successful on a task on graphs, the GNN needs to correctly represent the pattern trees of the nodes of the graphs.

Original: This means, that to generalize to the target domain D t , the GNN needs to be forced to represent pattern trees from both the source and the target distributions.
Revised: This means, that to generalize to the target domain D t , the GNN needs to be forced to represent pattern trees from both the source and the target distributions.

Original: For more details about the construction of the pattern tree see Appendix D.
Revised: For more details about the construction of the pattern tree see Appendix D.

Original: In short, each tree corresponds to a d -pattern in the following way: the d -pattern tree of a node can be seen as a multiset of the children of the root, and each child is a multiset of its children, etc.
Revised: In short, each tree corresponds to a d -pattern in the following way: the d -pattern tree of a node can be seen as a multiset of the children of the root, and each child is a multiset of its children, etc.

Original: The pattern tree is a different description of the d -pattern of a node.
Revised: The pattern tree is a different description of the d -pattern of a node.

Original: This means that a GNN that successfully represent a pattern tree also represents its corresponding d -pattern, thus connecting this SSL task to the theory from Sec. 3.
Revised: This means that a GNN that successfully represent a pattern tree also represents its corresponding d -pattern, thus connecting this SSL task to the theory from Sec. 3.

Original: Semi-supervised setup.
Revised: Semi-supervised setup.

Original: We also consider a case where a small number of labeled samples are available for the target domain.
Revised: We also consider a case where a small number of labeled samples are available for the target domain.

Original: A natural approach is to train an SSL pretext task on samples from both the source and target domain, and train the main task on all the labeled samples available.
Revised: A natural approach is to train an SSL pretext task on samples from both the source and target domain, and train the main task on all the labeled samples available.

Original: We tested this setup with 1, 5, or 10 labeled examples from the target domain.
Revised: We tested this setup with 1, 5, or 10 labeled examples from the target domain.

Original: 4. E XPERIMENTS
Revised: 4. E XPERIMENTS

Original: Architecture and training protocol.
Revised: Architecture and training protocol.

Original: The setup is the same as in Subsection 3.4 with the following changes.
Revised: The setup is the same as in Subsection 3.4 with the following changes.

Original: We use a three-layer GNN in all experiments.
Revised: We use a three-layer GNN in all experiments.

Original: Multi-task learning is used with equal weight to the main and SSL tasks.
Revised: Multi-task learning is used with equal weight to the main and SSL tasks.

Original: In the semi-supervised setup, we used an equal weight for the main task and the labeled examples from the target domain.
Revised: In the semi-supervised setup, we used an equal weight for the main task and the labeled examples from the target domain.

Original: Baselines.
Revised: Baselines.

Original: We compare our new pretext task to the following baselines: (1) Vanilla : standard training on the source domain; (2) HomoGNN (Tang et al., 2020) a homogeneous GNN without the bias term trained on the source domain; (3) Graph autoencoder (GAE) pretext task (Kipf and
Revised: We compare our new pretext task to the following baselines: (1) Vanilla : standard training on the source domain; (2) HomoGNN (Tang et al., 2020) a homogeneous GNN without the bias term trained on the source domain; (3) Graph autoencoder (GAE) pretext task (Kipf and

Original: Welling, 2016); (4) Node masking (NM) pretext task from Hu et al.
Revised: Welling, 2016); (4) Node masking (NM) pretext task from Hu et al.

Original: (2019) where at each training iteration we mask 10% of the node features and the goal is to reconstruct them.
Revised: (2019) where at each training iteration we mask 10% of the node features and the goal is to reconstruct them.

Original: In case the graph does not have node features then the task was to predict the degree of the masked nodes.
Revised: In case the graph does not have node features then the task was to predict the degree of the masked nodes.

Original: (5) Node metric learning (NML) : we use metric learning to learn useful node representations.
Revised: (5) Node metric learning (NML) : we use metric learning to learn useful node representations.

Original: We use a corruption function that given a graph and corruption parameter p ∈ [0 , 1] , replaces p | E | of theedges with random edges, and thus can generate positive ( p = 0 . 1 ) and negative ( p = 0 . 3 ) examples for all nodes of the graph. we train with the triplet loss (Weinberger and Saul, 2009).
Revised: We use a corruption function that given a graph and corruption parameter p ∈ [0 , 1] , replaces p | E | of the

Original: [Table]
Revised: [Table]

Original: Datasets.
Revised: Datasets.

Original: We use datasets from Morris et al.
Revised: We use datasets from Morris et al.

Original: and Rozemberczki et al.
Revised: and Rozemberczki et al.

Original: (2020) (Twitch egos and Deezer egos).
Revised: (2020) (Twitch egos and Deezer egos).

Original: We selected datasets that have a sufﬁcient number of graphs (more than 1,000) and with a non-trivial split to small and large graphs as detailed in Appendix F.1.
Revised: We selected datasets that have a sufﬁcient number of graphs (more than 1,000) and with a non-trivial split to small and large graphs as detailed in Appendix F.1.

Original: In total we used 7 datasets, 4 in molecular biology (NCI1, NCI109, D&D, Proteins), andof social networks (Twitch ego nets, Deezer ego nets, IMDB-Binary).
Revised: In total we used 7 datasets, 4 in molecular biology (NCI1, NCI109, D&D, Proteins), andof social networks (Twitch ego nets, Deezer ego nets, IMDB-Binary).

Original: In all datasets, 50% smallest graphs were assigned to the training set, and the largest 10% of graphs assigned to the test set.
Revised: In all datasets, 50% smallest graphs were assigned to the training set, and the largest 10% of graphs assigned to the test set.

Original: We further split a random 10% of the small graphs as a validation set.
Revised: We further split a random 10% of the small graphs as a validation set.

Original: [Figure]
Revised: [Figure]

Original: Results.
Revised: Results.

Original: Table 1 compares the effect of using the Patterntree pretext task to the baselines described above.
Revised: Table 1 compares the effect of using the Patterntree pretext task to the baselines described above.

Original: The small graphs row presents vanilla results on a validation set with small graphs.
Revised: The small graphs row presents vanilla results on a validation set with small graphs.

Original: The small graph accuracy on 5 out ofdatasets is larger by 7.3%-15.5% than on large graphs, indicating that the size-generalization problem is indeed prevalent in real datasets.
Revised: The small graph accuracy on 5 out ofdatasets is larger by 7.3%-15.5% than on large graphs, indicating that the size-generalization problem is indeed prevalent in real datasets.

Original: Pretraining with the d -patterns pretext task outperforms other baselines in 5 out 7 datasets, with an average 4% improved accuracy on all datasets.
Revised: Pretraining with the d -patterns pretext task outperforms other baselines in 5 out 7 datasets, with an average 4% improved accuracy on all datasets.

Original: HOMOGNN slightly improves over the vanilla while other pretext tasks do not improve average accuracy.
Revised: HOMOGNN slightly improves over the vanilla while other pretext tasks do not improve average accuracy.

Original: Naturally, the accuracy here is much lower than SOTA on these datasets because the domain shift makes the problem much harder.
Revised: Naturally, the accuracy here is much lower than SOTA on these datasets because the domain shift makes the problem much harder.

Original: In
Revised: In

Original: Appendix F.2 we show the 1-pattern distribution discrepancy between large and small graphs in two real datasets:
Revised: Appendix F.2 we show the 1-pattern distribution discrepancy between large and small graphs in two real datasets:

Original: IMDB (large discrepancy) and D&D (small discrepancy).
Revised: IMDB (large discrepancy) and D&D (small discrepancy).

Original: Correspondingly, the pattern tree SSL task improved performance on the IMDB dataset, while not improving performance on the D& D dataset.
Revised: Correspondingly, the pattern tree SSL task improved performance on the IMDB dataset, while not improving performance on the D& D dataset.

Original: This gives further evidence that a discrepancy between the d -patterns leads to bad generalization, and that correctly representing the patterns of the test set can improve performance.
Revised: This gives further evidence that a discrepancy between the d -patterns leads to bad generalization, and that correctly representing the patterns of the test set can improve performance.

Original: Figure 4 compares the performance of vanilla training versus pretraining with the pattern-tree pretext task in the semi-supervised setup.
Revised: Figure 4 compares the performance of vanilla training versus pretraining with the pattern-tree pretext task in the semi-supervised setup.

Original: The accuracy monotonically increases with respect to the number of labeled examples in both cases.
Revised: The accuracy monotonically increases with respect to the number of labeled examples in both cases.

Original: Moreover, pretraining with the pretext task yields better results in the case of 0,1,5 labeled examples and comparable results in the case we use 10 labeled examples.
Revised: Moreover, pretraining with the pretext task yields better results in the case of 0,1,5 labeled examples and comparable results in the case we use 10 labeled examples.

Original: We additionally tested on the synthetic tasks discussed in Sec. 3, and show that the pattern-tree pretext task improves in the student-teacher setting, while it does not solve the edge count or degree prediction tasks.
Revised: We additionally tested on the synthetic tasks discussed in Sec. 3, and show that the pattern-tree pretext task improves in the student-teacher setting, while it does not solve the edge count or degree prediction tasks.

Original: On the other hand, adding even a single labeled sample from the target distribution signiﬁcantly improves performance on the synthetic tasks we tested on.
Revised: On the other hand, adding even a single labeled sample from the target distribution signiﬁcantly improves performance on the synthetic tasks we tested on.

Original: For more details see Sec.
Revised: For more details see Sec.

Original: F.
Revised: F.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: This paper is a ﬁrst step towards understanding the size generalization problem in graph neural networks.
Revised: This paper is a ﬁrst step towards understanding the size generalization problem in graph neural networks.

Original: We showed that GNNs do not naturally generalize to larger graphs even on simple tasks, characterized how this failure depends on d -patterns, and suggested two approaches that can improve generalization.
Revised: We showed that GNNs do not naturally generalize to larger graphs even on simple tasks, characterized how this failure depends on d -patterns, and suggested two approaches that can improve generalization.

Original: While these approaches are found useful for multiple tasks, there are tasks on which we could not improve generalization.

Original: Our characterization of d -patterns is likely to have implications to other problems where generalization is harmed by distribution shifts, and offer a way to mitigate those problems.
Revised: Our characterization of d -patterns is likely to have implications to other problems where generalization is harmed by distribution shifts, and offer a way to mitigate those problems.

Original: A P RELIMINARIES
Revised: A P RELIMINARIES

Original: This section discusses two key concepts that we use throughout the paper: (1) Graph Neural Networks; (2) The Weisfeiler-Lehman graph isomorphism test.
Revised: This section discusses two key concepts that we use throughout the paper: (1) Graph Neural Networks; (2) The Weisfeiler-Lehman graph isomorphism test.

Original: At the end of the section, we also discuss the setup of constant GNN depth that we use in this paper.
Revised: At the end of the section, we also discuss the setup of constant GNN depth that we use in this paper.

Original: Notation.
Revised: Notation.

Original: We denote by { ( a 1 , m a 1 ) , . . .
Revised: We denote by { ( a 1 , m a 1 ) , . . .

Original: , ( a n , m a n ) } a multiset , that is a set where we allow multiple instances of the same element.
Revised: , ( a n , m a n ) } a multiset , that is a set where we allow multiple instances of the same element.

Original: Here a 1 , . .
Revised: Here a 1 , . .

Original: . , a n are distinct elements, and m a
Revised: . , a n are distinct elements, and m a

Original: i is the number of times a i appears in the multiset.
Revised: i is the number of times a i appears in the multiset.

Original: bold-face letters represent vectors.
Revised: bold-face letters represent vectors.

Original: A. G RAPH NEURAL NETWORKS
Revised: A. G RAPH NEURAL NETWORKS

Original: We consider the common message-passing GNN architecture (38) deﬁned as follows: Let G
Revised: We consider the common message-passing GNN architecture (38) deﬁned as follows: Let G

Original: =
Revised: =

Original: ( V, E ) be a graph, and for each node v ∈ V let h (0) v ∈ R d 0 be a node feature vector.
Revised: ( V, E ) be a graph, and for each node v ∈ V let h (0) v ∈ R d 0 be a node feature vector.

Original: Then for every t > 0 we deﬁne:
Revised: Then for every t > 0 we deﬁne:

Original: [Equation]
Revised: [Equation]

Original: Here, A GG is an permutation invariant aggregation function such as summation, averaging or taking max, U PDATE is an update function such as an MLP, and N ( v ) denotes the set of neighbors of node v .
Revised: Here, A GG is an permutation invariant aggregation function such as summation, averaging or taking max, U PDATE is an update function such as an MLP, and N ( v ) denotes the set of neighbors of node v .

Original: In case the graph is directed then N ( v ) denotes the set of nodes with edges incoming to v .
Revised: In case the graph is directed then N ( v ) denotes the set of nodes with edges incoming to v .

Original: For node prediction tasks, the output of a T -layer GNN for node v is h ( T ) v , while for graph prediction tasks an additional readout layer is used:
Revised: For node prediction tasks, the output of a T -layer GNN for node v is h ( T ) v , while for graph prediction tasks an additional readout layer is used:

Original: [Equation]
Revised: [Equation]

Original: where R EAD O UT is some invariant function such as summation, averaging or taking the max, possibly followed by a fully connected network.
Revised: where R EAD O UT is some invariant function such as summation, averaging or taking the max, possibly followed by a fully connected network.

Original: In our theoretical results, we focus on the ﬁrst order GNN architecture from (25) deﬁned in the following way:
Revised: In our theoretical results, we focus on the ﬁrst order GNN architecture from (25) deﬁned in the following way:

Original: [Equation]
Revised: [Equation]

Original: Here, W ( t ) 1 , W ( t )∈ R d t
Revised: Here, W ( t ) 1 , W ( t )∈ R d t

Original: [Equation]
Revised: [Equation]

Original: 1 × d t , b ( t ) ∈ R d t denotes the parameters of the t -th layer of the GNN, and σ is some non-linear activation (e.g ReLU).
Revised: 1 × d t , b ( t ) ∈ R d t denotes the parameters of the t -th layer of the GNN, and σ is some non-linear activation (e.g ReLU).

Original: It was shown in (25) that GNNs composed from these layers have maximal expressive power with respect to all message passing neural networks.
Revised: It was shown in (25) that GNNs composed from these layers have maximal expressive power with respect to all message passing neural networks.

Original: In general, we focus on GNNs that use the popular sum aggregation as it has maximal expressive power with respect to other aggregations used in message passing GNNs (38).
Revised: In general, we focus on GNNs that use the popular sum aggregation as it has maximal expressive power with respect to other aggregations used in message passing GNNs (38).

Original: We discuss other aggregation functions and the trade-of they present in Section 5.
Revised: We discuss other aggregation functions and the trade-of they present in Section 5.

Original: A. T HE W EISFEILER -L EHMAN TEST
Revised: A. T HE W EISFEILER -L EHMAN TEST

Original: The deﬁnition of d -patterns, which are used frequently in this paper, is closely related to the
Revised: The deﬁnition of d -patterns, which are used frequently in this paper, is closely related to the

Original: Weisfeiler-Lehman (WL) test (37; 13).
Revised: Weisfeiler-Lehman (WL) test (37; 13).

Original: The WL test is an algorithm to test whether two graphs are isomorphic and was recently coupled with the expressive power of GNNs.
Revised: The WL test is an algorithm to test whether two graphs are isomorphic and was recently coupled with the expressive power of GNNs.

Original: It is easiest to describe WL by using an equivalent algorithm called the color-reﬁnement algorithm (13).
Revised: It is easiest to describe WL by using an equivalent algorithm called the color-reﬁnement algorithm (13).

Original: The color reﬁnement algorithm is executed sequentially, wherein each stage the algorithm generates a descriptor for each node according to the descriptors of its direct neighbors.
Revised: The color reﬁnement algorithm is executed sequentially, wherein each stage the algorithm generates a descriptor for each node according to the descriptors of its direct neighbors.

Original: The features in each stage can be used to deﬁne equivalence classes of nodes and the process continues until these equivalence classes cannot be reﬁned anymore.
Revised: The features in each stage can be used to deﬁne equivalence classes of nodes and the process continues until these equivalence classes cannot be reﬁned anymore.

Original: The ﬁnal graph descriptor is a histogram of the node features, and is closely related to the deﬁnition of d -patterns.
Revised: The ﬁnal graph descriptor is a histogram of the node features, and is closely related to the deﬁnition of d -patterns.

Original: A. C ONSTANT DEPTH VS . ADAPTIVE DEPTH TASKS
Revised: A. C ONSTANT DEPTH VS . ADAPTIVE DEPTH TASKS

Original: As stated earlier, multiple factors control size generalization.
Revised: As stated earlier, multiple factors control size generalization.

Original: One important factor is the type of task: We make a distinction between two types of graph tasks.
Revised: One important factor is the type of task: We make a distinction between two types of graph tasks.

Original: (1) Tasks that can be solved by constant depth GNN; A good example for such task is determining whether a certain, constant size connectivity pattern is found in the graph (2) Tasks that require the depth of the GNN to be related to different parameters of the problem, such as the diameter of the graph.
Revised: (1) Tasks that can be solved by constant depth GNN; A good example for such task is determining whether a certain, constant size connectivity pattern is found in the graph (2) Tasks that require the depth of the GNN to be related to different parameters of the problem, such as the diameter of the graph.

Original: To exemplify, the task of calculating the diameter of a graph can be solved with a GNN only if its depth depends on the size of the graph (For more details see (32; 21)).
Revised: To exemplify, the task of calculating the diameter of a graph can be solved with a GNN only if its depth depends on the size of the graph (For more details see (32; 21)).

Original: On the other hand, tasks that require local information about each node can be solved with a constant depth GNN.
Revised: On the other hand, tasks that require local information about each node can be solved with a constant depth GNN.

Original: We note that although there are several recent works about adaptive depth GNN, e.g. (32; 27), most currently used GNNs have constant and relatively small depth (7)).
Revised: We note that although there are several recent works about adaptive depth GNN, e.g. (32; 27), most currently used GNNs have constant and relatively small depth (7)).

Original: In this paper, we focus on the ﬁrst kind of tasks that require a constant number of GNN layers.
Revised: In this paper, we focus on the ﬁrst kind of tasks that require a constant number of GNN layers.

Original: We note that the size-generalization problem we discuss in this paper is also relevant for the second type of tasks, but we chose to discuss the problem in the simpler setup for clarity.
Revised: We note that the size-generalization problem we discuss in this paper is also relevant for the second type of tasks, but we chose to discuss the problem in the simpler setup for clarity.

Original: B P ROOFS FROM S EC .
Revised: B P ROOFS FROM S EC .

Original: Proof of Thm. 3.2.
Revised: Proof of Thm. 3.2.

Original: We show that from the deﬁnition of d -patterns, and the 1-WL algorithm (see (37)), the color given by the WL algorithm to two nodes is equal iff their d -pattern is equal.
Revised: We show that from the deﬁnition of d -patterns, and the 1-WL algorithm (see (37)), the color given by the WL algorithm to two nodes is equal iff their d -pattern is equal.

Original: For the case of d = 0 it is clear.
Revised: For the case of d = 0 it is clear.

Original: Suppose it is true for d − 1 , the WL algorithm at iteration d give node v a new color based on the colors given in iteration d − 1 to the neighbors of v .
Revised: Suppose it is true for d − 1 , the WL algorithm at iteration d give node v a new color based on the colors given in iteration d − 1 to the neighbors of v .

Original: This means, that the color of v at iteration d depends on the multiset of colors at iteration d − 1 of its neighbors, which by induction is the d − 1 -pattern of the neighbors of v .
Revised: This means, that the color of v at iteration d depends on the multiset of colors at iteration d − 1 of its neighbors, which by induction is the d − 1 -pattern of the neighbors of v .

Original: To conclude, we use Theorem 1 from (25) which shows that GNNs are constant on the colors of WL, hence also constant on the d -patterns.
Revised: To conclude, we use Theorem 1 from (25) which shows that GNNs are constant on the colors of WL, hence also constant on the d -patterns.

Original: To prove Thm.
Revised: To prove Thm.

Original: 3.3 we will ﬁrst need to following claim from (40) about the memorization power of
Revised: 3.3 we will ﬁrst need to following claim from (40) about the memorization power of

Original: ReLU networks:
Revised: ReLU networks:

Original: Theorem B.1.
Revised: Theorem B.1.

Original: Let { x i , y i } Ni =1 ∈ R d × R such that all the x i are distinct and y i ∈
Revised: Let { x i , y i } Ni =1 ∈ R d × R such that all the x i are distinct and y i ∈

Original: [ − 1 , 1] for all i .
Revised: [ − 1 , 1] for all i .

Original: Then there exists a 3 -layer fully connected ReLU neural network f : R d → R with width 2 √ N such that f ( x i ) =
Revised: Then there exists a 3 -layer fully connected ReLU neural network f : R d → R with width 2 √ N such that f ( x i ) =

Original: y i for every i .
Revised: y i for every i .

Original: We will also need the following lemma which will be used in the construction of each layer of the 1-GNN:
Revised: We will also need the following lemma which will be used in the construction of each layer of the 1-GNN:

Original: Lemma B.2.
Revised: Lemma B.2.

Original: Let N ∈ N and f : N → R be a function deﬁned by f ( n )
Revised: Let N ∈ N and f : N → R be a function deﬁned by f ( n )

Original: = w (cid:62) 2 σ ( w 1 n − b ) where w 1 , w 2 , b ∈ R N , and σ is the ReLU function.
Revised: = w (cid:62) 2 σ ( w 1 n − b ) where w 1 , w 2 , b ∈ R N , and σ is the ReLU function.

Original: Then for every y 1 , . . .
Revised: Then for every y 1 , . . .

Original: , y N ∈ R there exists w 1 , w 2 , b such that f ( n ) =
Revised: , y N ∈ R there exists w 1 , w 2 , b such that f ( n ) =

Original: y n for n ≤ N and f ( n ) =
Revised: y n for n ≤ N and f ( n ) =

Original: ( n − N + 1) y N − ( n − N )
Revised: ( n − N + 1) y N − ( n − N )

Original: y N − 1 for n > N .
Revised: y N − 1 for n > N .

Original: Proof.
Revised: Proof.

Original: Deﬁne w 1 =    . . .
Revised: Deﬁne w 1 =    . . .

Original:    , b =  
Revised:    , b =  

Original:  . ..
Revised:  . ..

Original: N −   .
Revised: N −   .

Original: Let a i be the i -th coordinate of w 2 , we will deﬁne a i recursively in the following way: Let a 1 = y 1 , suppose we deﬁned a 1 , . . .
Revised: Let a i be the i -th coordinate of w 2 , we will deﬁne a i recursively in the following way: Let a 1 = y 1 , suppose we deﬁned a 1 , . . .

Original: , a i − 1 , then deﬁne a i =
Revised: , a i − 1 , then deﬁne a i =

Original: y
Revised: y

Original: i − 2 a
Revised: i − 2 a

Original: i − 1 − · · · − ia 1 .
Revised: i − 1 − · · · − ia 1 .

Original: Now we have for every n ≤ N :
Revised: Now we have for every n ≤ N :

Original: [Equation]
Revised: [Equation]

Original: 2 σ
Revised: 2 σ

Original: [Equation]
Revised: [Equation]

Original: w n − b ) =
Revised: w n − b ) =

Original: na
Revised: na

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: n −
Revised: n −

Original: ) a
Revised: ) a

Original: + · · · y n
Revised: + · · · y n

Original: [Equation]
Revised: [Equation]

Original: For n > N we can write n = N + k for k ≥ 1 , then we have:
Revised: For n > N we can write n = N + k for k ≥ 1 , then we have:

Original: [Equation]
Revised: [Equation]

Original: Now we are ready to prove the main theorem:
Revised: Now we are ready to prove the main theorem:

Original: Proof of Thm.
Revised: Proof of Thm.

Original: We assume w.l.o.g that at the ﬁrst iteration each node i is represented as a onehot vector h (0)
Revised: We assume w.l.o.g that at the ﬁrst iteration each node i is represented as a onehot vector h (0)

Original: i of dimension | C | , with its corresponding node feature.
Revised: i of dimension | C | , with its corresponding node feature.

Original: Otherwise, since there are | C | node features we can use one GNN layer that ignores all neighbors and only represent each node as a one-hot vector.
Revised: Otherwise, since there are | C | node features we can use one GNN layer that ignores all neighbors and only represent each node as a one-hot vector.

Original: We construct the ﬁrst d layers of the 1 -GNN F by induction on d .
Revised: We construct the ﬁrst d layers of the 1 -GNN F by induction on d .

Original: Denote by a i = | C | · (cid:0) N i + N i − 1 (cid:1) the dimension of the i -th layer of the GNN for 1 ≤ i ≤ d , and a 0 = | C | .
Revised: Denote by a i = | C | · (cid:0) N i + N i − 1 (cid:1) the dimension of the i -th layer of the GNN for 1 ≤ i ≤ d , and a 0 = | C | .

Original: The mapping has two parts, one takes the neighbors information and maps it into a feature representing the multiset of d -patterns, the other part is simply the identity saving information regarding the d-pattern of the node itself.
Revised: The mapping has two parts, one takes the neighbors information and maps it into a feature representing the multiset of d -patterns, the other part is simply the identity saving information regarding the d-pattern of the node itself.

Original: The d layer structure is
Revised: The d layer structure is

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: We set W ( d )=
Revised: We set W ( d )=

Original: [0 , I ] T , W ( d )=
Revised: [0 , I ] T , W ( d )=

Original: [ ˜ W ( d ) T, 0] T and U ( d +1) =
Revised: [ ˜ W ( d ) T, 0] T and U ( d +1) =

Original: [ ˜ U ( d +1) T , 0] T with
Revised: [ ˜ U ( d +1) T , 0] T with

Original: ˜ W ( d )∈ R
Revised: ˜ W ( d )∈ R

Original: Na d − 1 × a d − 1 and ˜ U
Revised: Na d − 1 × a d − 1 and ˜ U

Original: ( d +1) ∈
Revised: ( d +1) ∈

Original: R Na d − 1 ×
Revised: R Na d − 1 ×

Original: Na d − 1 .
Revised: Na d − 1 .

Original: For ˜ W ( d )we set w (1) i , its i -th row, to be equal to e n where n = (cid:6) iN (cid:7) .
Revised: For ˜ W ( d )we set w (1) i , its i -th row, to be equal to e n where n = (cid:6) iN (cid:7) .

Original: Let b (1) i be the i -th coordinate of b ( d ) , be equal to i − 1 ( mod N ) for i ≤ N · a d − 1 and zero otherwise.
Revised: Let b (1) i be the i -th coordinate of b ( d ) , be equal to i − 1 ( mod N ) for i ≤ N · a d − 1 and zero otherwise.

Original: What this does for the ﬁrst a d − 1 elements of W ( d ) 2 h ( d ) v + (cid:80) u ∈N ( v ) W ( d ) 1 h ( d ) u is that each dimension i hold the number of neighbors with a speciﬁc (d-1)-pattern.
Revised: What this does for the ﬁrst a d − 1 elements of W ( d ) 2 h ( d ) v + (cid:80) u ∈N ( v ) W ( d ) 1 h ( d ) u is that each dimension i hold the number of neighbors with a speciﬁc (d-1)-pattern.

Original: We then replicate this vector N times, and for each replica we subtract a different bias integer ranging from 0 to N − 1 .
Revised: We then replicate this vector N times, and for each replica we subtract a different bias integer ranging from 0 to N − 1 .

Original: To that output we concatenate the original h ( d − 1) v
Revised: To that output we concatenate the original h ( d − 1) v

Original: Next we construct ˜ U
Revised: Next we construct ˜ U

Original: ( d +1) ∈
Revised: ( d +1) ∈

Original: R Na d − 1 × Na d − 1 in the following way: Let u ( d +1)
Revised: R Na d − 1 × Na d − 1 in the following way: Let u ( d +1)

Original: i be its i -th row, and u ( d +1)
Revised: i be its i -th row, and u ( d +1)

Original: i,j its j -th coordinate.
Revised: i,j its j -th coordinate.

Original: We set u ( d +1) i,j = 0 for every j with j (cid:54) = (cid:6) iN (cid:7) and the rest N coordinates to be equal to the vector w 2 from Lemma B.2 with labels y (cid:96) = 0 for (cid:96) ∈ { 1 , . . .
Revised: We set u ( d +1) i,j = 0 for every j with j (cid:54) = (cid:6) iN (cid:7) and the rest N coordinates to be equal to the vector w 2 from Lemma B.2 with labels y (cid:96) = 0 for (cid:96) ∈ { 1 , . . .

Original: , N }\{ ( i mod N )+1 } and y (cid:96) =
Revised: , N }\{ ( i mod N )+1 } and y (cid:96) =

Original: 1 for (cid:96) =
Revised: 1 for (cid:96) =

Original: ( i mod N ) + 1 .
Revised: ( i mod N ) + 1 .

Original: Using the above construction we encoded the output on node v of the ﬁrst layer of F as a vector:
Revised: Using the above construction we encoded the output on node v of the ﬁrst layer of F as a vector:

Original: This encoding is such that the i -th coordinate of h ( d +1) v for 1 ≤ i ≤ N · a d − 1 is equal to 1 iff node v have ( i mod N ) + 1 neighbors with node feature (cid:6) iN (cid:7) ∈ { 1 , . .
Revised: This encoding is such that the i -th coordinate of h ( d +1) v for 1 ≤ i ≤ N · a d − 1 is equal to 1 iff node v have ( i mod N ) + 1 neighbors with node feature (cid:6) iN (cid:7) ∈ { 1 , . .

Original: | C |} .
Revised: | C |} .

Original: The last a d − 1 rows are a copy of h ( d − 1) v .
Revised: The last a d − 1 rows are a copy of h ( d − 1) v .

Original: Construction of the sufﬁx.
Revised: Construction of the sufﬁx.

Original: Next, we construct the last two layers.
Revised: Next, we construct the last two layers.

Original: First we note that for a node v with d -pattern p there is a unique vector z p such that the output of the 1-GNN on node v , h ( d ) v , is equal to z p .
Revised: First we note that for a node v with d -pattern p there is a unique vector z p such that the output of the 1-GNN on node v , h ( d ) v , is equal to z p .

Original: From our previous construction one can reconstruct exactly the (d-1)-pattern of each node, and the exact number of neighbors with each (d-1)-pattern and therefore can recover the d-pattern correctly from the h ( d ) v embedding.
Revised: From our previous construction one can reconstruct exactly the (d-1)-pattern of each node, and the exact number of neighbors with each (d-1)-pattern and therefore can recover the d-pattern correctly from the h ( d ) v embedding.

Original: Finally, we use Thm.
Revised: Finally, we use Thm.

Original: B.1 to construct the 3 -layer fully connected neural network with width at most 2 (cid:112) | P | such that for every pattern p i ∈ P with corresponding unique representation z p i and label y
Revised: B.1 to construct the 3 -layer fully connected neural network with width at most 2 (cid:112) | P | such that for every pattern p i ∈ P with corresponding unique representation z p i and label y

Original: i , the output of the network on z p i is equal to y i .
Revised: i , the output of the network on z p i is equal to y i .

Original: We construct the last two layers of the 1-GNN such that W ( d +1), W ( d )= 0 , and W ( d +1), b ( d +1) , W ( d +2), b ( d +2) , W ( d +3) are the matrices produced from Thm.
Revised: We construct the last two layers of the 1-GNN such that W ( d +1), W ( d )= 0 , and W ( d +1), b ( d +1) , W ( d +2), b ( d +2) , W ( d +3) are the matrices produced from Thm.

Original: B.1.
Revised: B.1.

Original: Note that W ( d +3) is the linear output layer constructed from the theorem, thus the ﬁnal output of the 1-GNN for node v is W ( d +3) · h ( d +2) v , where h ( d +
Revised: Note that W ( d +3) is the linear output layer constructed from the theorem, thus the ﬁnal output of the 1-GNN for node v is W ( d +3) · h ( d +2) v , where h ( d +

Original: ) v is the output after d + layers.
Revised: ) v is the output after d + layers.

Original: Proof of Corollary 3.4.
Revised: Proof of Corollary 3.4.

Original: By the assumption, the output of the task is determined by the d -patterns of the nodes.
Revised: By the assumption, the output of the task is determined by the d -patterns of the nodes.

Original: For each node with pattern p i let y i be the output of the node prediction task.
Revised: For each node with pattern p i let y i be the output of the node prediction task.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: By Thm.
Revised: By Thm.

Original: 3.3 there exists a 1-GNN such that for any d -pattern p
Revised: 3.3 there exists a 1-GNN such that for any d -pattern p

Original: i ∈
Revised: i ∈

Original: A gives a wrong label and for every pattern outside A ,
Revised: A gives a wrong label and for every pattern outside A ,

Original: p j ∈
Revised: p j ∈

Original: A c gives the correct label.
Revised: A c gives the correct label.

Original: Note that we can use Thm. 3.3 since both
Revised: Note that we can use Thm. 3.3 since both

Original: A and A c are ﬁnite, because we assumed that distributions on the graphs have ﬁnite supports.
Revised: A and A c are ﬁnite, because we assumed that distributions on the graphs have ﬁnite supports.

Original: The 0-1 loss for small and large graphs is exactly P d − pat
Revised: The 0-1 loss for small and large graphs is exactly P d − pat

Original: ( A
Revised: ( A

Original: [Equation]
Revised: [Equation]

Original: and P
Revised: and P

Original: [Equation]
Revised: [Equation]

Original: pat( A ) respectively.
Revised: pat( A ) respectively.

Original: Proof of Corollary 3.6.
Revised: Proof of Corollary 3.6.

Original: Let D 1 and D 2 be all the d -patterns that appear in graphs from P 1 and Prespectively.
Revised: Let D 1 and D 2 be all the d -patterns that appear in graphs from P 1 and Prespectively.

Original: By the assumption that the task is solvable by a d -layer GNN, and the same reasoning as in the proof of Corollary 3.4, the output of the GNN which solves the task depends only on the d -patterns.
Revised: By the assumption that the task is solvable by a d -layer GNN, and the same reasoning as in the proof of Corollary 3.4, the output of the GNN which solves the task depends only on the d -patterns.

Original: Thus, by Thm.
Revised: Thus, by Thm.

Original: 3.3 there exists a d +2 -layer GNN that perfectly solves the task on graphs with patterns from D 1 and in addition sends every pattern from D 2 to zero.
Revised: 3.3 there exists a d +2 -layer GNN that perfectly solves the task on graphs with patterns from D 1 and in addition sends every pattern from D 2 to zero.

Original: If zero is not a solution of the task for any graph from P 2 we are ﬁnished.
Revised: If zero is not a solution of the task for any graph from P 2 we are ﬁnished.

Original: Otherwise, if zero is a solution of a graph from Pbut not for a graph from P 1 we can add an MLP after the summation readout function that outputs the identity on every input except zero, for which it outputs a value which is not a solution for the task for any graph in P 2 .
Revised: Otherwise, if zero is a solution of a graph from Pbut not for a graph from P 1 we can add an MLP after the summation readout function that outputs the identity on every input except zero, for which it outputs a value which is not a solution for the task for any graph in P 2 .

Original: Note that this is possible since P 2 has ﬁnite support.
Revised: Note that this is possible since P 2 has ﬁnite support.

Original: In the case that zero is also a solution of the task on a graph from P 1 , let n 1 , . . . , n (cid:96) be the sizes of graphs from P 2 .
Revised: In the case that zero is also a solution of the task on a graph from P 1 , let n 1 , . . . , n (cid:96) be the sizes of graphs from P 2 .

Original: We pick some y ∈ R such that y · n i is not an output of any graph from P 1 and P 2 , there is such y since the distributions have a ﬁnite support.
Revised: We pick some y ∈ R such that y · n i is not an output of any graph from P 1 and P 2 , there is such y since the distributions have a ﬁnite support.

Original: Now, we can change the proof so that the output of the
Revised: Now, we can change the proof so that the output of the

Original: GNN on all patterns from D 2 is equal to y .
Revised: GNN on all patterns from D 2 is equal to y .

Original: Note that now the output of any graph from P 2 after the summation readout function is not the correct output.
Revised: Note that now the output of any graph from P 2 after the summation readout function is not the correct output.

Original: In case there is an MLP after the readout function, the proof can be readily changed to account for that.
Revised: In case there is an MLP after the readout function, the proof can be readily changed to account for that.

Original: Proof or Corollary 3.5.
Revised: Proof or Corollary 3.5.

Original: Let h be a 1 -GNN that have 0 loss on P 1 .
Revised: Let h be a 1 -GNN that have 0 loss on P 1 .

Original: In particular, for any pattern in P d − pat, h successfully predict the correct label.
Revised: In particular, for any pattern in P d − pat, h successfully predict the correct label.

Original: By the assumption, the patterns that appear in
Revised: By the assumption, the patterns that appear in

Original: P d − patare contained in the pattern that appear in P d − pat.
Revised: P d − patare contained in the pattern that appear in P d − pat.

Original: Using Thm. 3.2, that any 1 -GNN is constant on d -patterns, we get that h will also succeed on the pattern from P d − pat, hence will have 0 loss on P 2 .
Revised: Using Thm. 3.2, that any 1 -GNN is constant on d -patterns, we get that h will also succeed on the pattern from P d − pat, hence will have 0 loss on P 2 .

Original: C A DDITIONAL EXPERIMENTS FROM S EC . 3. 
Revised: C A DDITIONAL EXPERIMENTS FROM S EC . 3.

Original: First, we consider the max-clique problem.
Revised: First, we consider the max-clique problem.

Original: The goal of this problem is given a graph, to output the size of the maximal clique.
Revised: The goal of this problem is given a graph, to output the size of the maximal clique.

Original: This is in general an NP-hard problem, hence a constant depth GNN will not be able to solve it for graphs of all sizes.
Revised: This is in general an NP-hard problem, hence a constant depth GNN will not be able to solve it for graphs of all sizes.

Original: For this task we sampled both the train and test graphs from a geometrical distribution deﬁned as follows: given a number of nodes n and radius ρ we draw n points uniformly in [0 , 1] 2 , each point correspond to a node in the graph, and two nodes are connected if their corresponding points have a distance less than ρ .
Revised: For this task we sampled both the train and test graphs from a geometrical distribution deﬁned as follows: given a number of nodes n and radius ρ we draw n points uniformly in [0 , 1] 2 , each point correspond to a node in the graph, and two nodes are connected if their corresponding points have a distance less than ρ .

Original: We further analyzed the effects of how the network depth and architecture affect size generalization.
Revised: We further analyzed the effects of how the network depth and architecture affect size generalization.

Original: Table 2 presents the test loss on the max-clique problem.
Revised: Table 2 presents the test loss on the max-clique problem.

Original: Deeper networks are substantially more stricken due to the domain shift.
Revised: Deeper networks are substantially more stricken due to the domain shift.

Original: If the test domain has a similar pattern distribution, increasing the neural network depth from one layer to three layers results in a small decrement of at most 20% to the loss.
Revised: If the test domain has a similar pattern distribution, increasing the neural network depth from one layer to three layers results in a small decrement of at most 20% to the loss.

Original: However, if the pattern distribution is different than the train pattern distribution, such change may increase the loss by more than 2 .
Revised: However, if the pattern distribution is different than the train pattern distribution, such change may increase the loss by more than 2 .

Original: 5 × .
Revised: 5 × .

Original: We also show that the problem is consistent in both 1-GNN and GIN architectures.
Revised: We also show that the problem is consistent in both 1-GNN and GIN architectures.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: task; (b) A graph regression task, where the graph label is the number of edges; (c) A student-teacher node regression task; (d) A node regression task, where the node label is its degree.
Revised: task; (b) A graph regression task, where the graph label is the number of edges; (c) A student-teacher node regression task; (d) A node regression task, where the node label is its degree.

Original: In the edge count/degree tasks the loss is the mean difference from the ground-truths, divided by the average degree/number of edges.
Revised: In the edge count/degree tasks the loss is the mean difference from the ground-truths, divided by the average degree/number of edges.

Original: In the student-teacher tasks the loss is the mean L 2 loss between the teacher’s value and the student’s prediction, divided by the averaged student’s prediction.
Revised: In the student-teacher tasks the loss is the mean L 2 loss between the teacher’s value and the student’s prediction, divided by the averaged student’s prediction.

Original: Both the student and teacher share the same 3-layer architecture
Revised: Both the student and teacher share the same 3-layer architecture

Original: Next, we tested on the student-teacher task, on both graph and node levels, on the degree prediction task of each node, and the task of predicting the number of edges in the graph.
Revised: Next, we tested on the student-teacher task, on both graph and node levels, on the degree prediction task of each node, and the task of predicting the number of edges in the graph.

Original: The goal of these experiments is to show that the size generalization problem persists on different tasks, different architectures, and its intensity is increased for deeper GNN.
Revised: The goal of these experiments is to show that the size generalization problem persists on different tasks, different architectures, and its intensity is increased for deeper GNN.

Original: In all the experiments we draw the graphs from G ( n, p ) distribution, wherein the test set n is drawn uniformly between 40 and 50 , and p = 0 .
Revised: In all the experiments we draw the graphs from G ( n, p ) distribution, wherein the test set n is drawn uniformly between 40 and 50 , and p = 0 .

Original: 3 , and in the test set n = 100 and p is either 0 .
Revised: 3 , and in the test set n = 100 and p is either 0 .

Original: 3 or 0 .
Revised: 3 or 0 .

Original: We note that when p = 0 .
Revised: We note that when p = 0 .

Original: 15 , the average degree of the test graph is equal to (approximately) the average degree of the train graph, while when p = 0 .
Revised: 15 , the average degree of the test graph is equal to (approximately) the average degree of the train graph, while when p = 0 .

Original: 3 it is twice as large.
Revised: 3 it is twice as large.

Original: We would expect that the model will generalize better when the average degree in the train and test set is similar, because then their d -patterns will also be more similar.
Revised: We would expect that the model will generalize better when the average degree in the train and test set is similar, because then their d -patterns will also be more similar.

Original: Table 3 compares the performance of these tasks when changing the graph size of the test data.
Revised: Table 3 compares the performance of these tasks when changing the graph size of the test data.

Original: We tested the performance with normalized test loss, where we normalized the output by the average output of the test set.
Revised: We tested the performance with normalized test loss, where we normalized the output by the average output of the test set.

Original: This metric allows us to estimate the percentage of mistakes from the average output.
Revised: This metric allows us to estimate the percentage of mistakes from the average output.

Original: D SSL TASK ON WL TREE
Revised: D SSL TASK ON WL TREE

Original: First, we will need the following deﬁnition which constructs a tree out of the d -pattern introduced in the previous section.
Revised: First, we will need the following deﬁnition which constructs a tree out of the d -pattern introduced in the previous section.

Original: This tree enables us to extract certain properties for each node which can, later on, be learned using a GNN.
Revised: This tree enables us to extract certain properties for each node which can, later on, be learned using a GNN.

Original: This deﬁnition is similar to the deﬁnition of ”unrolled tree” from (24).
Revised: This deﬁnition is similar to the deﬁnition of ”unrolled tree” from (24).

Original: Deﬁnition D.1 ( d -pattern tree) .
Revised: Deﬁnition D.1 ( d -pattern tree) .

Original: Let G =
Revised: Let G =

Original: ( V, E ) a graph, C a ﬁnite set of node features, where each v ∈ V have a corresponding feature c v , and d ≥ 0 .
Revised: ( V, E ) a graph, C a ﬁnite set of node features, where each v ∈ V have a corresponding feature c v , and d ≥ 0 .

Original: For a node v ∈ V , its d -pattern tree
Revised: For a node v ∈ V , its d -pattern tree

Original: T ( d ) v = ( V ( d ) v , E ( d ) v ) is directed tree where each node corresponds to some node in G. It is deﬁned recursively in the following way: For d = 0 , V (0) v = u (0 ,v ) , and E (0) v = ∅ .
Revised: T ( d ) v = ( V ( d ) v , E ( d ) v ) is directed tree where each node corresponds to some node in G. It is deﬁned recursively in the following way: For d = 0 , V (0) v = u (0 ,v ) , and E (0) v = ∅ .

Original: Suppose we deﬁned
Revised: Suppose we deﬁned

Original: T ( d − 1) v , and let ˜ V ( d − 1) v be all the leaf nodes in V ( d − 1) v (i.e. nodes without incoming edges).
Revised: T ( d − 1) v , and let ˜ V ( d − 1) v be all the leaf nodes in V ( d − 1) v (i.e. nodes without incoming edges).

Original: We deﬁne:
Revised: We deﬁne:

Original: [Equation]
Revised: [Equation]

Original: The main advantage of pattern trees is that they encode all the information that a GNN can produce for a given node by running the same GNN on the pattern tree.
Revised: The main advantage of pattern trees is that they encode all the information that a GNN can produce for a given node by running the same GNN on the pattern tree.

Original: This tree corresponds to the local patterns in the following way: the d -pattern tree of a node can be seen as a multiset of the children of the root, and each child is a multiset of its children, etc.
Revised: This tree corresponds to the local patterns in the following way: the d -pattern tree of a node can be seen as a multiset of the children of the root, and each child is a multiset of its children, etc.

Original: This completely describes the d -pattern of a node.
Revised: This completely describes the d -pattern of a node.

Original: In other words, there is a one-to-one correspondence between d -patterns and pattern trees of depth d .
Revised: In other words, there is a one-to-one correspondence between d -patterns and pattern trees of depth d .

Original: Thus, a GNN that successfully represents the pattern trees of the target distribution will also successfully represent the d -patterns of the target distribution.
Revised: Thus, a GNN that successfully represents the pattern trees of the target distribution will also successfully represent the d -patterns of the target distribution.

Original: Using the d -pattern tree we construct a simple regression SSL task where its goal is for each node to count the number of nodes in each layer of its d -pattern tree.
Revised: Using the d -pattern tree we construct a simple regression SSL task where its goal is for each node to count the number of nodes in each layer of its d -pattern tree.

Original: This is a simple descriptor of the tree, which although loses some information about connectivity, does hold information about the structure of the layers.
Revised: This is a simple descriptor of the tree, which although loses some information about connectivity, does hold information about the structure of the layers.

Original: For example, in Fig.
Revised: For example, in Fig.

Original: 3 the descriptor for the tree would be that the root (zero) layer has a single black node, the ﬁrst layer has two yellow nodes, the second layer has two yellow, two gray, and two black nodes, and the third layer has ten yellow, two black and two gray nodes.
Revised: 3 the descriptor for the tree would be that the root (zero) layer has a single black node, the ﬁrst layer has two yellow nodes, the second layer has two yellow, two gray, and two black nodes, and the third layer has ten yellow, two black and two gray nodes.

Original: E T RAINING P ROCEDURE
Revised: E T RAINING P ROCEDURE

Original: In this section we explain in details the training procedure used in the experiments of Sec. 4.
Revised: In this section we explain in details the training procedure used in the experiments of Sec. 4.

Original: Let
Revised: Let

Original: X Main and X SSL be two labeled datasets, the ﬁrst contains the labeled examples for the main task from the source distribution, and the second contains examples labeled by the SSL task from both the source and target distributions.
Revised: X Main and X SSL be two labeled datasets, the ﬁrst contains the labeled examples for the main task from the source distribution, and the second contains examples labeled by the SSL task from both the source and target distributions.

Original: Let (cid:96) : R × R
Revised: Let (cid:96) : R × R

Original: → R be a loss function, in all our experiments we use cross entropy loss for classiﬁcation tasks and squared loss for regression tasks.
Revised: → R be a loss function, in all our experiments we use cross entropy loss for classiﬁcation tasks and squared loss for regression tasks.

Original: We construct the following models:
Revised: We construct the following models:

Original: (1) f GNN is a GNN feature extractor.
Revised: (1) f GNN is a GNN feature extractor.

Original: Its input is a graph and its output is a feature vector for each node in the
Revised: Its input is a graph and its output is a feature vector for each node in the

Original: graph
Revised: graph

Original: (2) h
Revised: (2) h

Original: Main is a head (a small
Revised: Main is a head (a small

Original: neural
Revised: neural

Original: network) for the
Revised: network) for the

Original: main
Revised: main

Original: task.
Revised: task.

Original: Its inputs are the node feature and it outputs a prediction (for
Revised: Its inputs are the node feature and it outputs a prediction (for

Original: graph prediction
Revised: graph prediction

Original: tasks this
Revised: tasks this

Original: head
Revised: head

Original: contains
Revised: contains

Original: a global pooling layer)
Revised: a global pooling layer)

Original: (3) h
Revised: (3) h

Original: SSL
Revised: SSL

Original: is the head for the
Revised: is the head for the

Original: SSL
Revised: SSL

Original: task.
Revised: task.

Original: Its inputs
Revised: Its inputs

Original: are
Revised: are

Original: node
Revised: node

Original: features, and
Revised: features, and

Original: it
Revised: it

Original: outputs
Revised: outputs

Original: a
Revised: a

Original: prediction for each
Revised: prediction for each

Original: node
Revised: node

Original: of
Revised: of

Original: the graph
Revised: the graph

Original: , depending
Revised: , depending

Original: on the speciﬁc
Revised: on the speciﬁc

Original: SSL used
Revised: SSL used

Original: Pretraining.
Revised: Pretraining.

Original: Here, there are two phases for the learning procedure.
Revised: Here, there are two phases for the learning procedure.

Original: In the ﬁrst phase, at each iteration we sample a batch ( x 1 , y 1 ) from X SSL , and train ny minimizing the objective: (cid:96) ( h SSL ◦ f GNN ( x 1 ) , y 1 ) .
Revised: In the ﬁrst phase, at each iteration we sample a batch ( x 1 , y 1 ) from X SSL , and train ny minimizing the objective: (cid:96) ( h SSL ◦ f GNN ( x 1 ) , y 1 ) .

Original: In this phase both h SSL and f GNN are trained.
Revised: In this phase both h SSL and f GNN are trained.

Original: In the second phase, at each iteration we sample ( x 2 , y 2 ) from X main and train on the loss (cid:96) ( h Main ◦ f GNN ( x 2 ) , y 2 ) , where we only train the head h Main , while the weights of f GNN are ﬁxed.
Revised: In the second phase, at each iteration we sample ( x 2 , y 2 ) from X main and train on the loss (cid:96) ( h Main ◦ f GNN ( x 2 ) , y 2 ) , where we only train the head h Main , while the weights of f GNN are ﬁxed.

Original: Multitask training.
Revised: Multitask training.

Original: Here we train all the functions at the same time.
Revised: Here we train all the functions at the same time.

Original: At each iteration we sample a batch ( x 1 , y 1 ) from X SSL and a batch ( x 2 , y 2 ) from X Main and train by minimizing the objective:
Revised: At each iteration we sample a batch ( x 1 , y 1 ) from X SSL and a batch ( x 2 , y 2 ) from X Main and train by minimizing the objective:

Original: [Equation]
Revised: [Equation]

Original: Here α ∈ [0 , 1] is the weight for the SSL task, in all our experiments we used α = 1 / 2 .
Revised: Here α ∈ [0 , 1] is the weight for the SSL task, in all our experiments we used α = 1 / 2 .

Original: For an illustration of the training procedures see Fig.
Revised: For an illustration of the training procedures see Fig.

Original: These procedures are common practices for training with SSL tasks (see e.g. (39)).
Revised: These procedures are common practices for training with SSL tasks (see e.g. (39)).

Original: We additionally use a semi-supervised setup in which we are given a dataset X FS of few-shot examples from the target distribution with their correct label.
Revised: We additionally use a semi-supervised setup in which we are given a dataset X FS of few-shot examples from the target distribution with their correct label.

Original: In both training procedures, at each iteration we sample a batch ( x 3 , y 3 ) from X FS and add a loss term β(cid:96)
Revised: In both training procedures, at each iteration we sample a batch ( x 3 , y 3 ) from X FS and add a loss term β(cid:96)

Original: ( h Main ◦ f GNN ( x 3 ) , y ) where β ∈ [0 , 1] is the weight of the few-shot loss.
Revised: ( h Main ◦ f GNN ( x 3 ) , y ) where β ∈ [0 , 1] is the weight of the few-shot loss.

Original: In pretraining this term is only added to the
Revised: In pretraining this term is only added to the

Original: [Figure]
Revised: [Figure]

Original: second phase, with weight 1 / 2 and adjust the weight of the main task to 1 / 2 as well (equal weight to the main task).
Revised: second phase, with weight 1 / 2 and adjust the weight of the main task to 1 / 2 as well (equal weight to the main task).

Original: In the multitask setup, we add this term with a weight 1 / 3 and adjust the weights of the two other losses to 1 / 3 as well, so all the losses have the same weight.
Revised: In the multitask setup, we add this term with a weight 1 / 3 and adjust the weights of the two other losses to 1 / 3 as well, so all the losses have the same weight.

Original: F M ORE EXPERIMENTS FROM S EC .
Revised: F M ORE EXPERIMENTS FROM S EC .

Original: For synthetic datasets, we used the setting of Section 3.4.
Revised: For synthetic datasets, we used the setting of Section 3.4.

Original: Source graphs were generated with G ( n, p ) with n sampled uniformly in [40 , 50] and p = 0 .
Revised: Source graphs were generated with G ( n, p ) with n sampled uniformly in [40 , 50] and p = 0 .

Original: Target graphs were sampled from G ( n, p ) with n = 100 and p = 0 . 3 .
Revised: Target graphs were sampled from G ( n, p ) with n = 100 and p = 0 . 3 .

Original: Table 4 depicts the results of using the d -patterns SSL tasks, in addition to using the semi-supervised setting.
Revised: Table 4 depicts the results of using the d -patterns SSL tasks, in addition to using the semi-supervised setting.

Original: It can be seen that adding the d -patterns SSL task signiﬁcantly improves the performance on the teacher-student task, although it does not completely solve it.
Revised: It can be seen that adding the d -patterns SSL task signiﬁcantly improves the performance on the teacher-student task, although it does not completely solve it.

Original: We also observe that adding labeled examples from the target domain signiﬁcantly improves the performance of all tasks.
Revised: We also observe that adding labeled examples from the target domain signiﬁcantly improves the performance of all tasks.

Original: Note that adding even a single example signiﬁcantly improves performance.
Revised: Note that adding even a single example signiﬁcantly improves performance.

Original: In all the experiments, the network was successful at learning the task on the source domain with less than 0 .
Revised: In all the experiments, the network was successful at learning the task on the source domain with less than 0 .

Original: 15 averaged error, and in most cases much less.
Revised: 15 averaged error, and in most cases much less.

Original: Fig.
Revised: Fig.

Original: 6 depicts a side-by-side plot of the 3-layer case of Fig.
Revised: 6 depicts a side-by-side plot of the 3-layer case of Fig.

Original: 2 (left), where training is done on graphs sampled from G ( n, p ) with 40 to 50 nodes and p = 0 .
Revised: 2 (left), where training is done on graphs sampled from G ( n, p ) with 40 to 50 nodes and p = 0 .

Original: 3 , and testing on graphs with 100 nodes and p varies.
Revised: 3 , and testing on graphs with 100 nodes and p varies.

Original: We compare Vanilla training, and our pattern tree SSL task with pretraining.
Revised: We compare Vanilla training, and our pattern tree SSL task with pretraining.

Original: It is clear that for all values of p our SSL task improves over vanilla training.
Revised: It is clear that for all values of p our SSL task improves over vanilla training.

Original: F. D ATASETS STATISTICS
Revised: F. D ATASETS STATISTICS

Original: Table F.1 shows the statistics of the datasets that were used in the paper.
Revised: Table F.1 shows the statistics of the datasets that were used in the paper.

Original: In particular the table presents the split that was used in the experiments, we trained on graphs with sizes smaller or equal to the 50-th percentile and tested on graphs with sizes larger or equal to the 90-th percentile.
Revised: In particular the table presents the split that was used in the experiments, we trained on graphs with sizes smaller or equal to the 50-th percentile and tested on graphs with sizes larger or equal to the 90-th percentile.

Original: In all
Revised: In all

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the datasets there is a signiﬁcant difference between the graph sizes in the train and test sets, and in some datasets there is also a difference between the distribution of the output class in the small and large graphs.
Revised: the datasets there is a signiﬁcant difference between the graph sizes in the train and test sets, and in some datasets there is also a difference between the distribution of the output class in the small and large graphs.

Original: F. D EGREE D ISCREPANCY C ORRESPONDS TO S IZE D ISCREPANCY
Revised: F. D EGREE D ISCREPANCY C ORRESPONDS TO S IZE D ISCREPANCY

Original: In this subsection we calculated the degree histogram of two of the real datasets that we tested on:
Revised: In this subsection we calculated the degree histogram of two of the real datasets that we tested on:

Original: IMDB-Binary and D & D. For the degree histogram see Fig.
Revised: IMDB-Binary and D & D. For the degree histogram see Fig.

Original: Recall the the degree is the 1pattern of a node.
Revised: Recall the the degree is the 1-pattern of a node.

Original: It is clear from the plots that for the IMDB dataset there is a difference in the distribution of degrees between small and large graphs, while in D & D the distributions look almost the same.
Revised: It is clear from the plots that for the IMDB dataset there is a difference in the distribution of degrees between small and large graphs, while in D & D the distributions look almost the same.

Original: Correspondingly, In our experiments the pattern tree SSL task signiﬁcantly improved performance on the IMDB dataset, while not improving performance on the D& D dataset.
Revised: Correspondingly, In our experiments the pattern tree SSL task signiﬁcantly improved performance on the IMDB dataset, while not improving performance on the D& D dataset.

Original: This gives further evidence that a discrepancy between the d -patterns leads to bad generalization, and that correctly representing the patterns of the test set can improve performance.
Revised: This gives further evidence that a discrepancy between the d -patterns leads to bad generalization, and that correctly representing the patterns of the test set can improve performance.

Original: While in this simple experiment we only discussed 1-paterns it is worth mentioning that the discrepancy is  accentuated when considering deeper patterns.
Revised: While in this simple experiment we only considered 1-patterns it is worth mentioning that the discrepancy is can only be accentuated when considering deeper patterns.

Original: [Figure]
Revised: [Figure]