Original: A H IERARCHY OF G RAPH N EURAL N ETWORKS B ASED ON L EARNABLE L OCAL F EATURES
Revised: A H IERARCHY OF G RAPH N EURAL N ETWORKS B ASED ON L EARNABLE L OCAL F EATURES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph neural networks (GNNs) are a powerful tool to learn representations on graphs by iteratively aggregating features from node neighbourhoods.
Revised: Graph neural networks (GNNs) are a powerful tool to learn representations on graphs by iteratively aggregating features from node neighbourhoods.

Original: Many variant models have been proposed, but there is limited understanding on both how to compare different architectures and how to construct GNNs systematically.
Revised: Many variant models have been proposed, but there is limited understanding on both how to compare different architectures and how to construct GNNs systematically.

Original: Here, we propose a hierarchy of GNNs based on their aggregation regions.
Revised: Here, we propose a hierarchy of GNNs based on their aggregation regions.

Original: We derive theoretical results about the discriminative power and feature representation capabilities of each class.
Revised: We derive theoretical results about the discriminative power and feature representation capabilities of each class.

Original: Then, we show how this framework can be utilized to systematically construct arbitrarily powerful GNNs.
Revised: Then, we show how this framework can be utilized to systematically construct arbitrarily powerful GNNs.

Original: As an example, we construct a simple architecture that exceeds the expressiveness of the Weisfeiler-Lehman graph isomorphism test.
Revised: As an example, we construct a simple architecture that exceeds the expressiveness of the Weisfeiler-Lehman graph isomorphism test.

Original: We empirically validate our theory on both synthetic and real-world benchmarks, and demonstrate our example’s theoretical power translates to state-of-the-art results on node classiﬁcation, graph classiﬁcation, and graph regression tasks.
Revised: We empirically validate our theory on both synthetic and real-world benchmarks, and demonstrate our example’s theoretical power translates to strong results on node classiﬁcation, graph classiﬁcation, and graph regression tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graphs arise naturally in the world and are key to applications in chemistry, social media, ﬁnance, and many other areas.
Revised: Graphs arise naturally in the world and are key to applications in chemistry, social media, ﬁnance, and many other areas.

Original: Understanding graphs is important and learning graph representations is a key step.
Revised: Understanding graphs is important and learning graph representations is a key step.

Original: Recently, there has been an explosion of interest in utilizing graph neural networks (GNNs), which have shown outstanding performance across tasks (e.g. Kipf & Welling (2016), Veliˇckovi´c et al.
Revised: Recently, there has been an explosion of interest in utilizing graph neural networks (GNNs), which have shown outstanding performance across tasks (e.g. Kipf & Welling (2016), Veliˇckovi´c et al.

Original: Generally, we consider node-feature GNNs which operate recursively to aggregate representations from a neighbouring region (Gilmer et al., 2017).
Revised: Generally, we consider node-feature GNNs which operate recursively to aggregate representations from a neighbouring region (Gilmer et al., 2017).

Original: In this work, we propose a representational hierarchy of GNNs, and derive the discriminative power and feature representation capabilities in each class.
Revised: In this work, we propose a representational hierarchy of GNNs, and derive the discriminative power and feature representation capabilities in each class.

Original: Importantly, while most previous work has focused on GNNs aggregating over vertices in the immediate neighbourhood, we consider GNNs aggregating over arbitrary subgraphs containing the node.
Revised: Importantly, while most previous work has focused on GNNs aggregating over vertices in the immediate neighbourhood, we consider GNNs aggregating over arbitrary subgraphs containing the node.

Original: We show that, under mild conditions, there is only in fact a small class of subgraphs that are valid aggregation regions.
Revised: We show that, under mild conditions, there is only in fact a small class of subgraphs that are valid aggregation regions.

Original: These subgraphs provide a systematic way of deﬁning a hierarchy for GNNs.
Revised: These subgraphs provide a systematic way of deﬁning a hierarchy for GNNs.

Original: Using this hierarchy, we can derive theoretical results which provide insight into GNNs.
Revised: Using this hierarchy, we can derive theoretical results which provide insight into GNNs.

Original: For example, we show that no matter how many layers are added, networks which only aggregate over immediate neighbors cannot learn the number of triangles in a node’s neighbourhood.
Revised: For example, we show that no matter how many layers are added, networks which only aggregate over immediate neighbors cannot learn the number of triangles in a node’s neighbourhood.

Original: We demonstrate that many popular frameworks, including GCN 1 (Kipf & Welling, 2016), GAT (Veliˇckovi´c et al., 2017), and
Revised: We demonstrate that many popular frameworks, including GCN 1 (Kipf & Welling, 2016), GAT (Veliˇckovi´c et al., 2017), and

Original: N-GCN (Abu-El-Haija et al., 2018) are uniﬁed under our framework.
Revised: N-GCN (Abu-El-Haija et al., 2018) are uniﬁed under our framework.

Original: We also compare each class using the Weisfeiler-Lehman (WL) isomorphism test (Weisfeiler & Lehman, 1968), and conclude our hierarchy is able to generate arbitrarily powerful GNNs.
Revised: We also compare each class using the Weisfeiler-Lehman (WL) isomorphism test (Weisfeiler & Lehman, 1968), and conclude our hierarchy is able to generate arbitrarily powerful GNNs.

Original: Then we utilize it to systematically generate
Revised: Then we utilize it to systematically generate

Original: GNNs exceeding the discriminating power of the 1-WL test.
Revised: GNNs exceeding the discriminating power of the 1-WL test.

Original: Experiments utilize both synthetic datasets and standard GNN benchmarks.
Revised: Experiments utilize both synthetic datasets and standard GNN benchmarks.

Original: We show that the method is able to learn difﬁcult graph properties where standard GCNs fail, even with multiple layers.
Revised: We show that the method is able to learn difﬁcult graph properties where standard GCNs fail, even with multiple layers.

Original: On benchmark datasets, our proposed GNNs are able to match or exceed state-of-the-art results on multiple datasets covering node classiﬁcation, graph classiﬁcation, and graph regression.
Revised: On benchmark datasets, our proposed GNNs are able to achieve strong results on multiple datasets covering node classiﬁcation, graph classiﬁcation, and graph regression.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Numerous works (see Li et al. (2015), Atwood & Towsley (2016), Defferrard et al.
Revised: Numerous works (see Li et al. (2015), Atwood & Towsley (2016), Defferrard et al.

Original: (2016), Kipf & Welling (2016), Niepert et al.
Revised: (2016), Kipf & Welling (2016), Niepert et al.

Original: (2016), Santoro et al.
Revised: (2016), Santoro et al.

Original: (2017), Veliˇckovi´c et al.
Revised: (2017), Veliˇckovi´c et al.

Original: (2017), Verma &
Revised: (2017), Verma &

Original: Zhang (2018), Zhang et al.
Revised: Zhang (2018), Zhang et al.

Original: (2018), Ivanov & Burnaev (2018), Wu et al. (2019a) for examples) have constructed different architectures to learn graph representations.
Revised: (2018a), Ivanov & Burnaev (2018), Wu et al. (2019a) for examples) have constructed different architectures to learn graph representations.

Original: Collectively, GNNs have pushed the state-of-the-art on many different tasks on graphs, including node classiﬁcation, and graph classiﬁcation/regression.
Revised: Collectively, GNNs have pushed the state-of-the-art on many different tasks on graphs, including node classiﬁcation, and graph classiﬁcation/regression.

Original: However, there are relatively few works that attempt to understand or categorize GNNs theoretically.
Revised: However, there are relatively few works that attempt to understand or categorize GNNs theoretically.

Original: Scarselli et al.
Revised: Scarselli et al.

Original: (2009) presented one of the ﬁrst works that investigated the capabilities of GNNs.
Revised: (2009) presented one of the ﬁrst works that investigated the capabilities of GNNs.

Original: They showed that the GNNs are able to approximate a large class of functions (those satisfying preservation of the unfolding equivalence) on graphs arbitrarily well.
Revised: They showed that the GNNs are able to approximate a large class of functions (those satisfying preservation of the unfolding equivalence) on graphs arbitrarily well.

Original: A recent work by Xu et al.
Revised: A recent work by Xu et al.

Original: (2018) also explored the theoretical properties of GNNs.
Revised: (2018) also explored the theoretical properties of GNNs.

Original: Its deﬁnition of GNNs is limited to those that aggregate features in the immediate neighbourhood, and thus is a special case of our general framework.
Revised: Its deﬁnition of GNNs is limited to those that aggregate features in the immediate neighbourhood, and thus is a special case of our general framework.

Original: We also show that the paper’s conclusion that GNNs are at most as powerful as the Weisfeiler-Lehman test fails to hold in a simple extension.
Revised: We also show that the paper’s conclusion that GNNs are at most as powerful as the Weisfeiler-Lehman test fails to hold in a simple extension.

Original: Survey works including Zhou et al.
Revised: Survey works including Zhou et al.

Original: (2018) and Wu et al.
Revised: (2018) and Wu et al.

Original: (2019b) give an overview of the current ﬁeld of research in GNNs, and provide structural classiﬁcations of GNNs.
Revised: (2019b) give an overview of the current ﬁeld of research in GNNs, and provide structural classiﬁcations of GNNs.

Original: We differ in our motivation to categorize GNNs from a computational perspective.
Revised: We differ in our motivation to categorize GNNs from a computational perspective.

Original: We also note that our classiﬁcation only covers static node feature graphs, though extensions to more general settings are possible.
Revised: We also note that our classiﬁcation only covers static node feature graphs, though extensions to more general settings are possible.

Original: The disadvantages of GNNs using localized ﬁlter to propagate information are analyzed in Li et al.
Revised: The disadvantages of GNNs using localized ﬁlter to propagate information are analyzed in Li et al.

Original: One major problem is their incapability of exploring global graph structures.
Revised: One major problem is their incapability of exploring global graph structures.

Original: To alleviate this,  N-GCN (Abu-El-Haija et al., 2018) feeds higher-degree polynomials of adjacency matrix to multiple instantiations of GCNs.
Revised: To alleviate this, there has been two important works in expanding neighbourhoods: N-GCN (Abu-El-Haija et al., 2018) feeds higher-degree polynomials of adjacency matrix to multiple instantiations of GCNs, and

Original: In Morris et al.
Revised: Morris et al.

Original: (2018), GNNsare generalized to k -GNNs by constructing a set-based k -WL to consider higher-order neighbourhoods and capture information beyond node-level.
Revised: (2018) generalizes GNNs to k -GNNs by constructing a set-based k -WL to consider higher-order neighbourhoods and capture information beyond node-level.

Original: We compare architectures constructed using our hierarchy to these state-of-the-art baselines in the experiments, and show thatthe systematic construction  brings an advantage across different tasks.
Revised: We compare architectures constructed using our hierarchy to these previous works in the experiments, and show that systematic construction of higher-order neighbourhoods brings an advantage across different tasks.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Let G =
Revised: Let G =

Original: ( V, E ) denote an undirected and unweighted graph, where | V | = N , and | E | =
Revised: ( V, E ) denote an undirected and unweighted graph, where | V | = N , and | E | =

Original: Ω .
Revised: Ω .

Original: Unless otherwise speciﬁed, we include self-loops for every node v ∈ V .
Revised: Unless otherwise speciﬁed, we include self-loops for every node v ∈ V .

Original: Let A be the graph’s adjacency matrix.
Revised: Let A be the graph’s adjacency matrix.

Original: Denote d ( u, v ) as the distance between two nodes u and v on a graph, deﬁned as the minimum length of walk between u and v .
Revised: Denote d ( u, v ) as the distance between two nodes u and v on a graph, deﬁned as the minimum length of walk between u and v .

Original: We further write d v as the degree of node v , and N ( v ) as the set of nodes in the direct neighborhood of v (including v itself).
Revised: We further write d v as the degree of node v , and N ( v ) as the set of nodes in the direct neighborhood of v (including v itself).

Original: Graph Neural Networks (GNNs) utilize the structure of a graph G and node features X ∈ R N × p to learn a reﬁned representation of each node, where p is input feature size, i.e. for each node v ∈ V , we have features X v ∈ R p .
Revised: Graph Neural Networks (GNNs) utilize the structure of a graph G and node features X ∈ R N × p to learn a reﬁned representation of each node, where p is input feature size, i.e. for each node v ∈ V , we have features X v ∈ R p .

Original: A GNN is a function that for every layer l at every node v aggregates features over a connected subgraph G v ⊆
Revised: A GNN is a function that for every layer l at every node v aggregates features over a connected subgraph G v ⊆

Original: G containing the node v , and updates a hidden representation H ( l ) =
Revised: G containing the node v , and updates a hidden representation H ( l ) =

Original: [ h ( l ) 1 , · · · , h ( l ) N ] .
Revised: [ h ( l ) 1 , · · · , h ( l ) N ] .

Original: Formally, we can deﬁne the l th layer of a GNN (with h (0) v = X v ):
Revised: Formally, we can deﬁne the l th layer of a GNN (with h (0) v = X v ):

Original: [Equation]
Revised: [Equation]

Original: where | is the restriction symbol over the domain G v , the aggregation subgraph.
Revised: where | is the restriction symbol over the domain G v , the aggregation subgraph.

Original: The aggregation function Agg( · ) is invariant with respect to the labeling of the nodes.
Revised: The aggregation function Agg( · ) is invariant with respect to the labeling of the nodes.

Original: The aggregation function,
Revised: The aggregation function,

Original: Agg( · ) , summarizes information from a neighbouring region G v , while the combination function
Revised: Agg( · ) , summarizes information from a neighbouring region G v , while the combination function

Original: Com( · ) joins such information with the previous hidden features to produce a new representation.
Revised: Com( · ) joins such information with the previous hidden features to produce a new representation.

Original: For different tasks, these GNNs are combined with an output layer to coerce the ﬁnal output into an appropriate shape.
Revised: For different tasks, these GNNs are combined with an output layer to coerce the ﬁnal output into an appropriate shape.

Original: Examples include fully-connected layers (Xu et al., 2018), convolutional layers (Zhang et al., 2018), and simple summation (Verma & Zhang, 2018).
Revised: Examples include fully-connected layers (Xu et al., 2018), convolutional layers (Zhang et al., 2018a), and simple summation (Verma & Zhang, 2018).

Original: These output layers are task-dependent and not graph-dependent, so we would omit these in our framework, and consider the node level output H ( L ) of the ﬁnal L th layer as the output of the GNN.
Revised: These output layers are task-dependent and not graph-dependent, so we would omit these in our framework, and consider the node level output H ( L ) of the ﬁnal L th layer as the output of the GNN.

Original: We consider three representative GNN variants in terms of this notation, where W ( l ) is a learnable weight matrix at layer l :
Revised: We consider three representative GNN variants in terms of this notation, where W ( l ) is a learnable weight matrix at layer l :

Original: • Graph Convolutional Networks (GCNs) (Kipf & Welling, 2016):
Revised: • Graph Convolutional Networks (GCNs) (Kipf & Welling, 2016):

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation]
Revised: [Equation]

Original: • Graph Attention Networks (GAT) (Veliˇckovi´c et al., 2017):
Revised: • Graph Attention Networks (GAT) (Veliˇckovi´c et al., 2017):

Original: Agg
Revised: Agg

Original: [Equation]
Revised: [Equation]

Original: ) =(cid:88)
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: • N-GCN (Abu-El-Haija et al., 2018) (2-layer case):
Revised: • N-GCN (Abu-El-Haija et al., 2018) (2-layer case):

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  H IERARCHICAL F RAMEWORK FOR C ONSTRUCTING GNN S
Revised:  H IERARCHICAL F RAMEWORK FOR C ONSTRUCTING GNN S

Original: Our proposed framework usesrandom walks to specify a hierarchy of aggregation ranges.
Revised: Our proposed framework uses walks to specify a hierarchy of aggregation ranges.

Original: The aggregation function over a node v ∈ G is a permutation-invariant function over a connected subgraph G v .
Revised: The aggregation function over a node v ∈ G is a permutation-invariant function over a connected subgraph G v .

Original: Consider the simplest case, using the neighbouring vertices u ∈ N ( v ) , utilized by many popular architectures (e.g. GCN, GAT).
Revised: Consider the simplest case, using the neighbouring vertices u ∈ N ( v ) , utilized by many popular architectures (e.g. GCN, GAT).

Original: Then G v in this case is a star-shaped subgraph, as illustrated below in Figure 1.
Revised: Then G v in this case is a star-shaped subgraph, as illustrated below in Figure 1.

Original: We refer to that as D 1 ( v ) , which in terms ofrandom walks, is the union of all edges and nodes in length-2 walks that start and end at v .
Revised: We refer to that as D 1 ( v ) , which in terms of walks, is the union of all edges and nodes in length-2 walks that start and end at v .

Original: To build a hierarchy, we consider beneﬁts of longerrandom walks.
Revised: To build a hierarchy, we consider beneﬁts of longer walks.

Original: The next simplest graph feature is the triangles in the neighbourhood of v .
Revised: The next simplest graph feature is the triangles in the neighbourhood of v .

Original: Knowledge on connections between the neighbouring nodes of v are necessary for considering triangles.
Revised: Knowledge on connections between the neighbouring nodes of v are necessary for considering triangles.

Original: A natural formulation usingrandom walks would be length-3 walks that start and end at v .
Revised: A natural formulation using walks would be length-3 walks that start and end at v .

Original: A length-3 returning walk outlines a triangle, and the union of all length-3 returning walks induces a subgraph, formed by all nodes and edges included in those walks.
Revised: A length-3 returning walk outlines a triangle, and the union of all lengthreturning walks induces a subgraph, formed by all nodes and edges included in those walks.

Original: This is illustrated in Figure 1 as L 1 ( v ) .
Revised: This is illustrated in Figure 1 as L 1 ( v ) .

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Deﬁne the set of all walks of length ≤ m returning to v as W m ( v ) .
Revised: Deﬁne the set of all walks of length ≤ m returning to v as W m ( v ) .

Original: For k ∈ Z + , we deﬁne D k ( v ) as the subgraph formed by all the edges and nodes in W 2 k ( v ) , while L k ( v ) is deﬁned as the subgraph formed by all the nodes and edges in W 2 k +1 ( v ) .
Revised: For k ∈ Z + , we deﬁne D k ( v ) as the subgraph formed by all the edges and nodes in W 2 k ( v ) , while L k ( v ) is deﬁned as the subgraph formed by all the nodes and edges in W 2 k +1 ( v ) .

Original: Intuitively, L k ( v ) is a subgraph of G consisting of all nodes and edges in the k -hop neighbourhood of node v , and D k ( v ) only differs from L k ( v ) by excluding the edges between the distance- k neighbors of v .
Revised: Intuitively, L k ( v ) is a subgraph of G consisting of all nodes and edges in the k -hop neighbourhood of node v , and D k ( v ) only differs from L k ( v ) by excluding the edges between the distance- k neighbors of v .

Original: We explore this further in Section 5.
Revised: We explore this further in Section 5.

Original: An example illustration of the neighbourhoods deﬁned above is shown in Figure 1.
Revised: An example illustration of the neighbourhoods deﬁned above is shown in Figure 1.

Original: This set of subgraphs naturally induces a hierarchy with increasing aggregation region:
Revised: This set of subgraphs naturally induces a hierarchy with increasing aggregation region:

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: The D-L hierarchy of aggregation regions for a node v , A D − L ( v ) in a graph G is, in increasing order:
Revised: The D-L hierarchy of aggregation regions for a node v , A D − L ( v ) in a graph G is, in increasing order:

Original: [Equation]
Revised: [Equation]

Original: Where D 1 ( v ) ⊆
Revised: Where D 1 ( v ) ⊆

Original: L 1 ( v ) ⊆
Revised: L 1 ( v ) ⊆

Original: D 2 ( v ) ⊆
Revised: D 2 ( v ) ⊆

Original: L 2 ( v ) · · · .
Revised: L 2 ( v ) · · · .

Original: Next, we consider the properties of this hierarchy.
Revised: Next, we consider the properties of this hierarchy.

Original: One important property is completeness - that the hierarchy can classify every possible GNN.
Revised: One important property is completeness - that the hierarchy can classify every possible GNN.

Original: Note that there is no meaningful complete hierarchy if G v is arbitrary.
Revised: Note that there is no meaningful complete hierarchy if G v is arbitrary.

Original: Therefore, we propose to limit our focus to those G v that can be deﬁned as a function of the distance from v .
Revised: Therefore, we propose to limit our focus to those G v that can be deﬁned as a function of the distance from v .

Original: Absent speciﬁc graph structures, distance is a canonical metric between vertices and this deﬁnition includes all examples listed in Section 3.
Revised: Absent speciﬁc graph structures, distance is a canonical metric between vertices and this deﬁnition includes all examples listed in Section 3.

Original: With such assumption, we can show that the D-L hierarchy is complete:
Revised: With such assumption, we can show that the D-L hierarchy is complete:

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider a GNN deﬁned by its action at each layer:
Revised: Consider a GNN deﬁned by its action at each layer:

Original: [Equation]
Revised: [Equation]

Original: Assume G v can be deﬁned as a univariate function of the distance from v .
Revised: Assume G v can be deﬁned as a univariate function of the distance from v .

Original: Then both of the following statements are true for all k ∈ Z + :
Revised: Then both of the following statements are true for all k ∈ Z + :

Original: • If D k ( v ) ⊆
Revised: • If D k ( v ) ⊆

Original: G v ⊆
Revised: G v ⊆

Original: L k ( v ) , then G v ∈ { D k ( v ) , L k ( v ) } .
Revised: L k ( v ) , then G v ∈ { D k ( v ) , L k ( v ) } .

Original: • If L k ( v ) ⊆
Revised: • If L k ( v ) ⊆

Original: G v ⊆
Revised: G v ⊆

Original: D k +1 ( v ) , then G v ∈ { L k ( v ) ,
Revised: D k +1 ( v ) , then G v ∈ { L k ( v ) ,

Original: D k +1 ( v ) } .
Revised: D k +1 ( v ) } .

Original: This theorem shows that one cannot create an aggregation region based on node distance that is "in between" the hierarchy deﬁned.
Revised: This theorem shows that one cannot create an aggregation region based on node distance that is "in between" the hierarchy deﬁned.

Original: With Theorem 1, we can use the D-L aggregation hierarchy to create a hierarchy of GNNs based on their aggregation regions.
Revised: With Theorem 1, we can use the D-L aggregation hierarchy to create a hierarchy of GNNs based on their aggregation regions.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: For k ∈ Z + , G ( D k ) is the set of all graph neural networks with aggregation region
Revised: For k ∈ Z + , G ( D k ) is the set of all graph neural networks with aggregation region

Original: G v = D k ( v ) that is not a member of G ( L k ) .
Revised: G v = D k ( v ) that is not a member of G ( L k ) .

Original: G ( L k ) is the set of all graph neural networks with aggregation region G v =
Revised: G ( L k ) is the set of all graph neural networks with aggregation region G v =

Original: L k ( v ) that is not a member of G ( D k − 1 ) .
Revised: L k ( v ) that is not a member of G ( D k − 1 ) .

Original: We explicitly exclude those belonging to a lower aggregation region in order to make the hierarchy well-deﬁned (otherwise a GNN of order G ( D 1 ) is trivially one of order G ( L 1 ) ).
Revised: We explicitly exclude those belonging to a lower aggregation region in order to make the hierarchy well-deﬁned (otherwise a GNN of order G ( D 1 ) is trivially one of order G ( L 1 ) ).

Original: We also implicitly deﬁne D 

Revised: We also implicitly deﬁne D

Original: = L 0 = ∅ .
Revised: = L 0 = ∅ .

Original: 4. C ONSTRUCTING D-L GNN S
Revised: 4. C ONSTRUCTING D-L GNN S

Original: The D-L Hierarchy can be used both to classify existing GNNs and also to construct new models.
Revised: The D-L Hierarchy can be used both to classify existing GNNs and also to construct new models.

Original: We ﬁrst note that all GNNs which aggregate over immediate neighbouring nodes fall in the class of
Revised: We ﬁrst note that all GNNs which aggregate over immediate neighbouring nodes fall in the class of

Original: G ( D 1 ) .
Revised: G ( D 1 ) .

Original: For example, Graph Convolutional Networks (GCNs) deﬁned in Section 3 is in G ( D 1 ) since its aggregation is Agg( · ) = (cid:80) u ∈ D 1 ( v ) h
Revised: For example, Graph Convolutional Networks (GCNs) deﬁned in Section 3 is in G ( D 1 ) since its aggregation is Agg( · ) = (cid:80) u ∈ D 1 ( v ) h

Original: ( l − 1) u , and similarly the N-GCN example is in G ( D 2 ) .
Revised: ( l − 1) u , and similarly the N-GCN example is in G ( D 2 ) .

Original: Note that these classes are deﬁned by the subgraph used by Agg , but does not imply that these networks reach the maximum discriminatory power of their class (deﬁned in the next section).
Revised: Note that these classes are deﬁned by the subgraph used by Agg , but does not imply that these networks reach the maximum discriminatory power of their class (deﬁned in the next section).

Original: We can use basic building blocks to implement different levels of GNNs.
Revised: We can use basic building blocks to implement different levels of GNNs.

Original: These examples are not meant to be exhaustive and only serve as a glimpse of what could be achieved with this framework.
Revised: These examples are not meant to be exhaustive and only serve as a glimpse of what could be achieved with this framework.

Original: Examples.
Revised: Examples.

Original: For every k ∈ Z + :
Revised: For every k ∈ Z + :

Original: • Any GNN with Agg( · ) =
Revised: • Any GNN with Agg( · ) =

Original: (cid:88) u ∈ D k ( v ) ( A k ) vu h ( l − 1) u is a GNN of class G ( D k ) .
Revised: (cid:88) u ∈ D k ( v ) ( A k ) vu h ( l − 1) u is a GNN of class G ( D k ) .

Original: • Any GNN with Agg( · ) =
Revised: • Any GNN with Agg( · ) =

Original: (cid:88) u ∈ D k ( v ) ( A k +1 )
Revised: (cid:88) u ∈ D k ( v ) ( A k +1 )

Original: vu h ( l − 1) u is a GNN of class G ( L k ) .
Revised: vu h ( l − 1) u is a GNN of class G ( L k ) .

Original: • Any GNN with Agg( · ) =
Revised: • Any GNN with Agg( · ) =

Original: ( A 2 k +1 )
Revised: ( A 2 k +1 )

Original: vv · h ( l − 1) v is a GNN of class G ( L k ) .
Revised: vv · h ( l − 1) v is a GNN of class G ( L k ) .

Original: Intuitively, ( A k ) vu counts all k -length walks from v to u , which includes all nodes in the k -hop neighbourhood.
Revised: Intuitively, ( A k ) vu counts all k -length walks from v to u , which includes all nodes in the k -hop neighbourhood.

Original: The difference between the ﬁrst and the second example above is that in the second one, we allow ( k + 1) -length walks from the nodes in the k -hop neighbourhood, which promotes it to be class of G ( L k ) .
Revised: The difference between the ﬁrst and the second example above is that in the second one, we allow ( k + 1) -length walks from the nodes in the k -hop neighbourhood, which promotes it to be class of G ( L k ) .

Original: Note the simplicity of the ﬁrst and the last examples: in matrix form the ﬁrst is
Revised: Note the simplicity of the ﬁrst and the last examples: in matrix form the ﬁrst is

Original: A k H ( l − 1) while the last form is Diag( A 2 k +1 ) · H ( l − 1) .
Revised: A k H ( l − 1) while the last form is Diag( A 2 k +1 ) · H ( l − 1) .

Original: The building blocks can be gradually added to the original aggregation function.
Revised: The building blocks can be gradually added to the original aggregation function.

Original: This is particularly useful if an experimenter knows there are higher-level properties that are necessary to compute, for instance to incorporate knowledge of triangles, one can design the following network (see Sectionfor more details):
Revised: This is particularly useful if an experimenter knows there are higher-level properties that are necessary to compute, for instance to incorporate knowledge of triangles, one can design the following network (see Sectionfor more details):

Original: [Equation]
Revised: [Equation]

Original: where w 1 , w 2 ∈ R are learnable weights.
Revised: where w 1 , w 2 ∈ R are learnable weights.

Original:  T HEORETICAL P ROPERTIES
Revised:  T HEORETICAL P ROPERTIES

Original: We can prove interesting theoretical properties for each class of graph neural networks on this hierarchy.
Revised: We can prove interesting theoretical properties for each class of graph neural networks on this hierarchy.

Original: To do this, we utilize the Weisfeiler-Lehman test, a powerful classical algorithm used to discriminate between potentially isomorphic graphs.
Revised: To do this, we utilize the Weisfeiler-Lehman test, a powerful classical algorithm used to discriminate between potentially isomorphic graphs.

Original: In interest of brevity, its introduction is included in the Appendix in Section 8.1.
Revised: In interest of brevity, its introduction is included in the Appendix in Section 8.1.

Original: We deﬁne the terminology of "discriminating graphs" formally below:
Revised: We deﬁne the terminology of "discriminating graphs" formally below:

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: The discriminative power of a function f over graphs G is the set of graphs S fG such that for every pair of graphs G 1 , G 2 ∈ S fG , the function has f ( G 1 ) =
Revised: The discriminative power of a function f over graphs G is the set of graphs S fG such that for every pair of graphs G 1 , G 2 ∈ S fG , the function has f ( G 1 ) =

Original: f ( G 2 )
Revised: f ( G 2 )

Original: iff G 1 ∼ =
Revised: iff G 1 ∼ =

Original: G 2 and f ( G 1 ) (cid:54) =
Revised: G 2 and f ( G 1 ) (cid:54) =

Original: f ( G 2 )
Revised: f ( G 2 )

Original: iff G 1 (cid:54)∼ =
Revised: iff G 1 (cid:54)∼ =

Original: G 2 .
Revised: G 2 .

Original: We say f decides G 1 , G 2 as isomorphic if f ( G 1 ) =
Revised: We say f decides G 1 , G 2 as isomorphic if f ( G 1 ) =

Original: f ( G 2 ) and vice versa.
Revised: f ( G 2 ) and vice versa.

Original: Essentially, S fG is the set of graphs that f can decide correctly whether any two of them are isomorphic or not.
Revised: Essentially, S fG is the set of graphs that f can decide correctly whether any two of them are isomorphic or not.

Original: We say f has a greater discriminative power than g if S fG (cid:41) S gG .
Revised: We say f has a greater discriminative power than g if S fG (cid:41) S gG .

Original: Now we ﬁrst introduce a theorem proven by Xu et al.
Revised: Now we ﬁrst introduce a theorem proven by Xu et al.

Original: Theorem 2.
Revised: Theorem 2.

Original: The maximum discriminative power of the set of GNNs in G ( D 1 ) is strictly less than or equal to the 1-dimensional WL test.
Revised: The maximum discriminative power of the set of GNNs in G ( D 1 ) is strictly less than or equal to the 1-dimensional WL test.

Original: Their framework only included G ( D 1 ) GNNs, and they upper bounded the discriminative power of such GNNs.
Revised: Their framework only included G ( D 1 ) GNNs, and they upper bounded the discriminative power of such GNNs.

Original: With our generalized framework, we are able to prove a slightly surprising result:
Revised: With our generalized framework, we are able to prove a slightly surprising result:

Original: Theorem 3.
Revised: Theorem 3.

Original: The maximum discriminative power of the set of GNNs in G ( L 1 ) is strictly greater than the 1-dimensional WL test.
Revised: The maximum discriminative power of the set of GNNs in G ( L 1 ) is strictly greater than the 1-dimensional WL test.

Original: This result is central to understanding GNNs.
Revised: This result is central to understanding GNNs.

Original: Even though the discriminative power of G ( D 1 ) is strictly less than or equal to the 1-WL test, Theorem 3 shows that just by adding the connections between the immediate neighbors of each node ( L 1 \ D 1 ), we can achieve theoretically greater discriminative power.
Revised: Even though the discriminative power of G ( D 1 ) is strictly less than or equal to the 1-WL test, Theorem 3 shows that just by adding the connections between the immediate neighbors of each node ( L 1 \ D 1 ), we can achieve theoretically greater discriminative power.

Original: One particular implication is that GNNs with maximal discriminative power in G ( L 1 ) can count the number of triangles in a graph, while those in G ( D 1 ) cannot, no matter how many layers are added.
Revised: One particular implication is that GNNs with maximal discriminative power in G ( L 1 ) can count the number of triangles in a graph, while those in G ( D 1 ) cannot, no matter how many layers are added.

Original: This goes against the intuition that more layers allow GNNs to aggregate information from further nodes, as G ( D 1 ) is unable to aggregate the information of triangles from the L 1 region, which is important in many applications (see Frank & Strauss (1986), Tsourakakis et al.
Revised: This goes against the intuition that more layers allow GNNs to aggregate information from further nodes, as G ( D 1 ) is unable to aggregate the information of triangles from the L 1 region, which is important in many applications (see Frank & Strauss (1986), Tsourakakis et al.

Original: (2011), Becchetti et al.
Revised: (2011), Becchetti et al.

Original: (2008), Eckmann & Moses (2002)).
Revised: (2008), Eckmann & Moses (2002)).

Original: Unfortunately, this is the only positive result we are able to establish regarding the WL test as the k -dim WL-test is not a local method for k > 1 .
Revised: Unfortunately, this is the only positive result we are able to establish regarding the WL test as the k -dim WL-test is not a local method for k > 1 .

Original: Nevertheless, we are able to prove that our hierarchy admits arbitrarily powerful GNNs through the following theorem:
Revised: Nevertheless, we are able to prove that our hierarchy admits arbitrarily powerful GNNs through the following theorem:

Original: [Table]
Revised: [Table]

Original: Theorem 4.
Revised: Theorem 4.

Original: For all k ∈ Z + , there exists a GNN within the class of G ( L k ) that is able to discriminate all graphs with ≤ k + 1 nodes.
Revised: For all k ∈ Z + , there exists a GNN within the class of G ( L k ) that is able to discriminate all graphs with ≤ k + 1 nodes.

Original: This shows that as k → ∞ , we are able to discriminate all graphs.
Revised: This shows that as k → ∞ , we are able to discriminate all graphs.

Original: We record the full set of results proven in Table 1.
Revised: We record the full set of results proven in Table 1.

Original: The key ingredients for proving these results are contained in Appendix 8.3 and 8.4.
Revised: The key ingredients for proving these results are contained in Appendix 8.3 and 8.4.

Original: Here we see that at the G ( L 1 ) class, theoretically we are able to learn all cliques (as cliques by deﬁnition are fully connected).
Revised: Here we see that at the G ( L 1 ) class, theoretically we are able to learn all cliques (as cliques by deﬁnition are fully connected).

Original: As we gradually move upward in the hierarchy, we are able to learn more far-reaching features such as higher length walks and cycles, while the discriminatory power improves.
Revised: As we gradually move upward in the hierarchy, we are able to learn more far-reaching features such as higher length walks and cycles, while the discriminatory power improves.

Original: We also note that the theoretical complexity increases as k increases.
Revised: We also note that the theoretical complexity increases as k increases.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We consider the capability of two speciﬁc GNNs instantiations that are motivated by this framework: w 1 AH ( l − 1) + w 2 Diag( A 3 ) · H ( l − 1) (GCN-L1) and w 1 AH ( l − 1) + w 2 Diag( A 3 ) · H ( l − 1) + w 3 A 2 H
Revised: We consider the capability of two speciﬁc GNNs instantiations that are motivated by this framework: w 1 AH ( l − 1) + w 2 Diag( A 3 ) · H ( l − 1) (GCN-L1) and w 1 AH ( l − 1) + w 2 Diag( A 3 ) · H ( l − 1) + w 3 A 2 H

Original: ( l − 1) (GCN-D2).
Revised: ( l − 1) (GCN-D2).

Original: These can be seen as extensions of the GCN introduced in Kipf & Welling (2016).
Revised: These can be seen as extensions of the GCN introduced in Kipf & Welling (2016).

Original: The ﬁrst, GCN-L1, equips the GNN with the ability to count triangles.
Revised: The ﬁrst, GCN-L1, equips the GNN with the ability to count triangles.

Original: The second, GCN-D2, can further count the number of 4-cycles.
Revised: The second, GCN-D2, can further count the number of 4-cycles.

Original: We note their theoretical power below (proof follows from
Revised: We note their theoretical power below (proof follows from

Original: Theorem 3):
Revised: Theorem 3):

Original: Corollary 1.
Revised: Corollary 1.

Original: The maximum discriminative power of GCN-L1 and GCN-D2 is strictly greater than the 1-dimensional WL test.
Revised: The maximum discriminative power of GCN-L1 and GCN-D2 is strictly greater than the 1-dimensional WL test.

Original: We compare the performance of GCN-L1, GCN-D2 and other state-of-art GNN variants on both synthetic and real-world tasks.
Revised: We compare the performance of GCN-L1, GCN-D2 and other state-of-art GNN variants on both synthetic and real-world tasks.

Original: For the combine function of GCN, GCN-L1, and GCN-D2, we use
Revised: For the combine function of GCN, GCN-L1, and GCN-D2, we use

Original: Com( · ) =
Revised: Com( · ) =

Original: MLP( a ( k ) v ) , where MLP is a multi-layer perceptron (MLP) with LeakyReLu activation similar to Xu et al.
Revised: MLP( a ( k ) v ) , where MLP is a multi-layer perceptron (MLP) with LeakyReLu activation similar to Xu et al.

Original: All of our experiments are run with PyTorch 1.2.0, PyTorch-Geometric 1.2.1, and we use NVIDIA
Revised: All of our experiments are run with PyTorch 1.2.0, PyTorch-Geometric 1.2.1, and we use NVIDIA

Original: Tesla P100 GPUs with 16GB memory.
Revised: Tesla P100 GPUs with 16GB memory.

Original: 6. S YNTHETIC E XPERIMENTS
Revised: 6. S YNTHETIC E XPERIMENTS

Original: To verify our previous claim that in our proposed hierarchy, GNNs from certain classes are able to learn speciﬁc features more effectively, we created two tasks: predict the number of triangles and number of 4-cycles in the graphs.
Revised: To verify our previous claim that in our proposed hierarchy, GNNs from certain classes are able to learn speciﬁc features more effectively, we created two tasks: predict the number of triangles and number of 4-cycles in the graphs.

Original: For each task, the dataset contains 1000 graphs and is generated in a procedure as follows:
Revised: For each task, the dataset contains 1000 graphs and is generated in a procedure as follows:

Original: We ﬁx the number of nodes in each graph to be 100 and use the Erd˝os–Rényi random graph model to generate random graphs with edge probability 0.07.
Revised: We ﬁx the number of nodes in each graph to be 100 and use the Erd˝os–Rényi random graph model to generate random graphs with edge probability 0.07.

Original: Then we count the number of patterns of interest.
Revised: Then we count the number of patterns of interest.

Original: In the 4-cycle dataset, the average number of 4-cycles in each graph is 1350 and in the triangle dataset, there are 54 triangles on average in each graph.
Revised: In the 4-cycle dataset, the average number of 4-cycles in each graph is 1350 and in the triangle dataset, there are 54 triangles on average in each graph.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: classiﬁcation, GR for graph regression.

Original: We perform 10-fold cross-validation and record the average and standard deviation of evaluation metrics across the 10 folds within the cross-validation.
Revised: We perform 10-fold cross-validation and record the average and standard deviation of evaluation metrics across the 10 folds within the cross-validation.

Original: We used 16 hidden features, and trained the networks using Adam optimizer with 0.001 initial learning rate, L 2 regularization λ = 0 . 0005 .
Revised: We used 16 hidden features, and trained the networks using Adam optimizer with 0.001 initial learning rate, L 2 regularization λ = 0 . 0005 .

Original: We further apply early stopping on validation loss with a delay window size of 10.
Revised: We further apply early stopping on validation loss with a delay window size of 10.

Original: The dropout rate is 0.1.
Revised: The dropout rate is 0.1.

Original: The learning rate is scheduled to reduce 50% if the validation accuracy stops increasing for 10 epochs.
Revised: The learning rate is scheduled to reduce 50% if the validation accuracy stops increasing for 10 epochs.

Original: We utilized a two-layer MLP in our combine function for GCN, GCN-L1 and GCN-L2, similar to the implementation in Xu et al.
Revised: We utilized a two-layer MLP in our combine function for GCN, GCN-L1 and GCN-L2, similar to the implementation in Xu et al.

Original: For training stability, we limited w 1 , w 2 ∈ (0 , 1) in our models using the sigmoid function.
Revised: For training stability, we limited w 1 , w 2 ∈ (0 , 1) in our models using the sigmoid function.

Original: To test our theoretical results under the most stringent conditions, we severely handicapped the GCN-L1 and GCN-D2models by limiting them to a 1-layer network. 
Revised: Results In our testing, we limited GCN-L1 and GCN-D2 to a 1-layer network. This prevents GCNL1 and GCN-D2 from utilizing higher order features to reverse predict the triangles and 4-cycles.

Original: Furthermore, we ensured GCN had the same receptive ﬁeld as such networks by using 2-layer and 3-layer GCNs, which provided GCN with additional feature representational capability.
Revised: Simultaneously, we ensured GCN had the same receptive ﬁeld as such networks by using 2-layer and 3-layer GCNs, which provided GCN with additional feature representational capability.

Revised: The baseline is a model that predicts the training mean on the testing set.

Original: The results are in Table 2.
Revised: The results are in Table 2.

Original: We see that even the restricted GCN-L1 and GCN-D2 can learn how to count the patterns signiﬁcantly better.
Revised: GCN completely fails to learn the features (worse than a simple mean prediction). However, we see that

Original: GCN-L1layer effectively learns the triangle counts and  outperforms a 3-layer GCN network, while
Revised: GCN-L1 effectively learns the triangle counts and greatly outperforms the mean baseline, while

Original: GCN-D2 is furthermore able to provide a considerably more exact approximation on the count of 4-cycles, without losing the ability to count triangles.
Revised: GCN-D2 is similarly able in providing a good approximation on the count of 4-cycles, without losing the ability to count triangles.

Original: This validates the "possible features learned" in Table 1.
Revised: This validates the "possible features learned" in Table 1.

Original: 6. R EAL -W ORLD B ENCHMARKS
Revised: 6. R EAL -W ORLD B ENCHMARKS

Original: We next consider standard benchmark datasets for (i) node classiﬁcation, (ii) graph classiﬁcation, (iii) graph regression tasks.
Revised: We next consider standard benchmark datasets for (i) node classiﬁcation, (ii) graph classiﬁcation, (iii) graph regression tasks.

Original: The details of these datasets are presented in Table 3.
Revised: The details of these datasets are presented in Table 3.

Original: The setup of the learning rate scheduling and L 2 regularization rate are the same as in synthetic tasks.
Revised: The setup of the learning rate scheduling and L 2 regularization rate are the same as in synthetic tasks.

Original: For the citation tasks, we used 16 hidden features, while we used 64 for the biological datasets.
Revised: For the citation tasks, we used 16 hidden features, while we used 64 for the biological datasets.

Original: We compare our model to results in the papers of  k -GNN (Morris et al., 2018), N-GCN (Abu-El-Haija et al., 2018), GAT (Verma & Zhang, 2018), WL-subtree (Shervashidze et al., 2011), the shortest path kernel SHT-PATH, (Borgwardt & Kriegel, 2005) and PATCHYSAN (Niepert et al., 2016). k -GNN and N-GCN are chosen as they are state-of-the-art baselines for GNNs that have an aggregation region beyond the immediate neighbors. Others were selected as they achieve the absolute state-of-the-art on at least one of the datasets tested below.
Revised: Since our main modiﬁcation is the expansion of aggregation region, our main comparison benchmarks are k -GNN (Morris et al., 2018) and N-GCN (Abu-El-Haija et al., 2018), two previous best attempts in incorporating aggregation regions beyond the immediate nodes.

Original: Note that we can view a m th order N-GCN as aggregating over D 1 , D 2 , · · · , D m .
Revised: Note that we can view a m th order N-GCN as aggregating over D 1 , D 2 , · · · , D m .

Revised: We further include results on GAT (Verma & Zhang, 2018), GIN (Xu et al., 2018), RetGK (Zhang et al., 2018b), GNTK (Du et al., 2019), WL-subtree (Shervashidze et al., 2011), SHT-PATH, (Borgwardt &

Revised: Kriegel, 2005) and PATCHYSAN (Niepert et al., 2016) to show some best performing architectures.

Original: [Table]
Revised: [Table]

Original: Baseline neural network models use a 1-layer perceptron combine function, with the exception of k -GNN, which uses a 2-layer perceptron combine function.
Revised: Baseline neural network models use a 1-layer perceptron combine function, with the exception of k -GNN, which uses a 2-layer perceptron combine function.

Original: Thus, to illustrate the effectiveness of the framework, we only utilize a 1-layer perceptron combine function for all tasks for our GCN models, with the exception of NCI1.
Revised: Thus, to illustrate the effectiveness of the framework, we only utilize a 1-layer perceptron combine function for all tasks for our GCN models, with the exception of NCI1.

Original: 2-layer perceptrons seemed necessary for good performance in NCI1, and thus we implemented all neural networks with 2-layer perceptrons for this task to ensure a fair comparison.
Revised: 2-layer perceptrons seemed necessary for good performance in NCI1, and thus we implemented all neural networks with 2-layer perceptrons for this task to ensure a fair comparison.

Original: We tuned the learning rates ∈ { 0 . 001 , 0 .
Revised: We tuned the learning rates ∈ { 0 . 001 , 0 .

Original: 005 , 0 . 01 , 0 . 05 } and dropout rates ∈ { 0 . 1 , 0 . 5 } .
Revised: 005 , 0 . 01 , 0 . 05 } and dropout rates ∈ { 0 . 1 , 0 . 5 } .

Original: For numerical stability, we normalize the aggregation function using the degree of v only.
Revised: For numerical stability, we normalize the aggregation function using the degree of v only.

Original: For the node classiﬁcation tasks, we directly utilized the ﬁnal layer output, while we summed over the node representations for the graph-level tasks.
Revised: For the node classiﬁcation tasks, we directly utilized the ﬁnal layer output, while we summed over the node representations for the graph-level tasks.

Original: Results Experimental results on real-world data are summarized in Table 4.
Revised: Results Experimental results on real-world data are summarized in Table 4.

Original: According to our experiments, GCN-L1 and GCN-D2 noticeably improve upon GCN across all datasets.
Revised: According to our experiments, GCN-L1 and GCN-D2 noticeably improve upon GCN across all datasets, due to its ability to combine node features in more complex and nonlinear ways.

Original: The improvement is statistically signiﬁcant on the 5% level for all datasets except Proteins.
Revised: The improvement is statistically signiﬁcant on the 5% level for all datasets except Proteins.

Original: The results of 
Revised: The results of GCN-L1 and

Original: GCN-L1 and GCN-D2 match the state-of-the-art in most datasets, and lead in numerical averages for Cora, Proteins, QM7b, and QM9 ( though not statistically signiﬁcant for all).
Revised: GCN-D2 match the best performing architectures in most datasets, and lead in numerical averages for Cora, Proteins, QM7b, and QM9 (though not statistically signiﬁcant for all).

Original: The results also show a signiﬁcant improvement from the state-of-the-
Revised: Importantly, the results also show a signiﬁcant improvement from the two main comparison architectures, k -GNN and N-GCN.

Original: Between the methods, we see that further expanding aggregation regions generates diminishing returns on these datasets, and the majority of the beneﬁt is gained in the ﬁrst-order extension G ( L 1 ) .
Revised: We see that further expanding aggregation regions generates diminishing returns on these datasets, and the majority of the beneﬁt is gained in the ﬁrst-order extension G ( L 1 ) .

Original: This is in contrast to N-GCN which skipped L 1 to only used D -type aggregation regions ( D 2 , D 3 , · · · ), which is an incomplete hierarchy of aggregation regions.
Revised: This is in contrast to N-GCN which skipped L 1 to only used D -type aggregation regions ( D 2 , D 3 , · · · ), which is an incomplete hierarchy of aggregation regions.

Original: The differential in results illustrates the power of the complete hierarchy as proven in Theorem 1.
Revised: The differential in results illustrates the power of the complete hierarchy as proven in Theorem 1.

Original: We especially would like to stress the outsized improvement of GCN-L1 on the biological datasets.
Revised: We especially would like to stress the outsized improvement of GCN-L1 on the biological datasets.

Original: As described in 1, GCN-L1 is able to capture information about triangles, which are highly relevant for the properties of biological molecules.
Revised: As described in 1, GCN-L1 is able to capture information about triangles, which are highly relevant for the properties of biological molecules.

Original: The experimental results verify such intuition, and show how knowledge about the task can lead to targeted GNN design using our framework.
Revised: The experimental results verify such intuition, and show how knowledge about the task can lead to targeted GNN design using our framework.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a theoretical framework to classify GNNs by their aggregation region and discriminative power, proving that the presented framework deﬁnes a complete hierarchy for GNNs.
Revised: We propose a theoretical framework to classify GNNs by their aggregation region and discriminative power, proving that the presented framework deﬁnes a complete hierarchy for GNNs.

Original: We also provide methods to construct powerful GNN models of any class with various building blocks.
Revised: We also provide methods to construct powerful GNN models of any class with various building blocks.

Original: Our experimental results show that example models constructed in the proposed way can effectively learn the corresponding features exceeding the capability of 1-WL algorithm in graphs.
Revised: Our experimental results show that example models constructed in the proposed way can effectively learn the corresponding features exceeding the capability of 1-WL algorithm in graphs.

Original: Aligning with our theoretical analysis, experimental results show that these stronger GNNs can better represent the complex properties of a number of real-world graphs.
Revised: Aligning with our theoretical analysis, experimental results show that these stronger GNNs can better represent the complex properties of a number of real-world graphs.

Original:  A PPENDIXES 8. I NTRODUCTION TO W EISFEILER -L EHMAN T EST
Revised:  A PPENDIXES 8. I NTRODUCTION TO W EISFEILER -L EHMAN T EST

Original: The 1-dimensional Weisfeiler-Lehman (WL) test is an iterative vertex classiﬁcation method widely used in checking graph isomorphism.
Revised: The 1-dimensional Weisfeiler-Lehman (WL) test is an iterative vertex classiﬁcation method widely used in checking graph isomorphism.

Original: In the ﬁrst iteration, the vertices are labeled by their valences.
Revised: In the ﬁrst iteration, the vertices are labeled by their valences.

Original: Then at each following step, the labels of vertices are updated by the multiset of the labels of themselves and their neighbors.
Revised: Then at each following step, the labels of vertices are updated by the multiset of the labels of themselves and their neighbors.

Original: The algorithm terminates when a stable set of labels is reached.
Revised: The algorithm terminates when a stable set of labels is reached.

Original: The details of 1-dimensional Weisfeiler-Lehman (WL) is shown in Algorithm 1.
Revised: The details of 1-dimensional Weisfeiler-Lehman (WL) is shown in Algorithm 1.

Original: Regarding the limitation of 1-dimensional Weisfeiler-Lehman (WL), Cai et al.
Revised: Regarding the limitation of 1-dimensional Weisfeiler-Lehman (WL), Cai et al.

Original: (1992) described families of non-isomorphic graphs which 1-dimensional WL test cannot distinguish.
Revised: (1992) described families of non-isomorphic graphs which 1-dimensional WL test cannot distinguish.

Original: end
Revised: end

Original: The k-dimensional Weisfeiler-Lehman (WL) algorithm extends the above procedure from operations on nodes to operations on tuples V ( G ) k .
Revised: The k-dimensional Weisfeiler-Lehman (WL) algorithm extends the above procedure from operations on nodes to operations on tuples V ( G ) k .

Original: 8. P ROOF OF T HEOREM
Revised: 8. P ROOF OF T HEOREM

Original: Theorem.
Revised: Theorem.

Original: Consider a GNN deﬁned by its action at each layer:
Revised: Consider a GNN deﬁned by its action at each layer:

Original: [Equation]
Revised: [Equation]

Original: Assume G v can be deﬁned as a univariate function of the distance from v .
Revised: Assume G v can be deﬁned as a univariate function of the distance from v .

Original: Then both of the following statements are true for all k ∈ Z + :
Revised: Then both of the following statements are true for all k ∈ Z + :

Original: • If D k ( v ) ⊆
Revised: • If D k ( v ) ⊆

Original: G v ⊆
Revised: G v ⊆

Original: L k ( v ) , then G v ∈ { D k ( v ) , L k ( v ) } .
Revised: L k ( v ) , then G v ∈ { D k ( v ) , L k ( v ) } .

Original: • If L k ( v ) ⊆
Revised: • If L k ( v ) ⊆

Original: G v ⊆
Revised: G v ⊆

Original: D k +1 ( v ) , then G v ∈ { D k +1 ( v ) ,
Revised: D k +1 ( v ) , then G v ∈ { D k +1 ( v ) ,

Original: L k ( v ) } .
Revised: L k ( v ) } .

Original: Proof.
Revised: Proof.

Original: We would prove by contradiction.
Revised: We would prove by contradiction.

Original: Assume, in contrary, that one of the statements in Theorem 1 is false.
Revised: Assume, in contrary, that one of the statements in Theorem 1 is false.

Original: Let k ∈ Z
Revised: Let k ∈ Z

Original: + .
Revised: + .

Original: Then we would separate these two cases as below:
Revised: Then we would separate these two cases as below:

Original: D k
Revised: D k

Original: [Equation]
Revised: [Equation]

Original: v ) (cid:40)(cid:40)(cid:40) G v (cid:40)(cid:40)(cid:40)
Revised: v ) (cid:40)(cid:40)(cid:40) G v (cid:40)(cid:40)(cid:40)

Original: L k ( v ) Assume that G v satisﬁes such relationship.
Revised: L k ( v ) Assume that G v satisﬁes such relationship.

Original: Since L k ( v ) and D k ( v ) only differ by the set C k ( v ) =
Revised: Since L k ( v ) and D k ( v ) only differ by the set C k ( v ) =

Original: { e ij ∈ E | d ( i, v ) =
Revised: { e ij ∈ E | d ( i, v ) =

Original: d ( j, v ) =
Revised: d ( j, v ) =

Original: k } , G v can only contain this set partially.
Revised: k } , G v can only contain this set partially.

Original: Let M k ( v ) (cid:40)
Revised: Let M k ( v ) (cid:40)

Original: C k ( v ) be the non-empty maximal subset of C k ( v ) that is contained in
Revised: C k ( v ) be the non-empty maximal subset of C k ( v ) that is contained in

Original: G v .
Revised: G v .

Original: Since M k ( v ) (cid:54) =
Revised: Since M k ( v ) (cid:54) =

Original: C k ( v ) , there exists m, n with d ( v, m )
Revised: C k ( v ) , there exists m, n with d ( v, m )

Original: =
Revised: =

Original: d ( v, n ) =
Revised: d ( v, n ) =

Original: k such that e mn ∈ C k ( v ) but e mn (cid:54)∈ M k ( v ) .
Revised: k such that e mn ∈ C k ( v ) but e mn (cid:54)∈ M k ( v ) .

Original: Consider a non-identity permutation of vertices ﬁxing v .
Revised: Consider a non-identity permutation of vertices ﬁxing v .

Original: Then since G v is deﬁned only using the distance function, and it needs to be permutation invariant, all e ij with d ( i, v ) =
Revised: Then since G v is deﬁned only using the distance function, and it needs to be permutation invariant, all e ij with d ( i, v ) =

Original: d ( j, v )
Revised: d ( j, v )

Original: =
Revised: =

Original: k must be in C k
Revised: k must be in C k

Original: ( v ) and not in M k
Revised: ( v ) and not in M k

Original: ( v ) .
Revised: ( v ) .

Original: But then M k ( v ) is empty, a contradiction.
Revised: But then M k ( v ) is empty, a contradiction.

Original: [Equation]
Revised: [Equation]

Original: v ) Assume that G v satisﬁes such relationship.
Revised: v ) Assume that G v satisﬁes such relationship.

Original: Consider the set difference between D k +1 ( v ) and L k ( v ) , denoted as a subgraph C k ( v ) :
Revised: Consider the set difference between D k +1 ( v ) and L k ( v ) , denoted as a subgraph C k ( v ) :

Original: [Equation]
Revised: [Equation]

Original: Then G v can only contain this set partially.
Revised: Then G v can only contain this set partially.

Original: Let M k ( v ) (cid:40)
Revised: Let M k ( v ) (cid:40)

Original: C k ( v ) be the maximal subset of C k ( v ) that is contained in G v .
Revised: C k ( v ) be the maximal subset of C k ( v ) that is contained in G v .

Original: Since M k ( v ) (cid:54) =
Revised: Since M k ( v ) (cid:54) =

Original: C k ( v ) , at least one of the followings must be true:
Revised: C k ( v ) , at least one of the followings must be true:

Original: • There exists u with d ( u, v ) =
Revised: • There exists u with d ( u, v ) =

Original: k such that u ∈ C k ( v ) but u (cid:54)∈ M k ( v ) .
Revised: k such that u ∈ C k ( v ) but u (cid:54)∈ M k ( v ) .

Original: • There exists m, n with d ( v, m )
Revised: • There exists m, n with d ( v, m )

Original: =
Revised: =

Original: k , d ( v, n ) =
Revised: k , d ( v, n ) =

Original: k + 1 such that e mn ∈ C k ( v ) but e mn (cid:54)∈ M k ( v )
Revised: k + 1 such that e mn ∈ C k ( v ) but e mn (cid:54)∈ M k ( v )

Original: For the ﬁrst case, consider a non-identity permutation of the vertices ﬁxing v .
Revised: For the ﬁrst case, consider a non-identity permutation of the vertices ﬁxing v .

Original: Then since G v is permutation invariant and deﬁned only using the distance, then thus all vertices w with d ( w, v ) =
Revised: Then since G v is permutation invariant and deﬁned only using the distance, then thus all vertices w with d ( w, v ) =

Original: k are in C k
Revised: k are in C k

Original: ( v ) but not in M k
Revised: ( v ) but not in M k

Original: ( v ) .
Revised: ( v ) .

Original: This implies V Mk ( v ) is empty.
Revised: This implies V Mk ( v ) is empty.

Original: Using the same logic for the second case, one can conclude that all e ij with d ( v, i ) =
Revised: Using the same logic for the second case, one can conclude that all e ij with d ( v, i ) =

Original: k and d ( v, m )
Revised: k and d ( v, m )

Original: =
Revised: =

Original: k + 1 must be in C k
Revised: k + 1 must be in C k

Original: ( v ) but not in M k
Revised: ( v ) but not in M k

Original: ( v ) .
Revised: ( v ) .

Original: That means E Mk ( v ) is empty.
Revised: That means E Mk ( v ) is empty.

Original: Therefore, we can conclude that at least one of V Mk ( v ) and E Mk ( v ) is empty.
Revised: Therefore, we can conclude that at least one of V Mk ( v ) and E Mk ( v ) is empty.

Original: Since both cannot be empty (as that means G v =
Revised: Since both cannot be empty (as that means G v =

Original: L k ( v ) ), we must have either V Mk ( v ) = ∅ or E Mk ( v ) = ∅ .
Revised: L k ( v ) ), we must have either V Mk ( v ) = ∅ or E Mk ( v ) = ∅ .

Original: With the former case M k ( v ) is not a valid subgraph (as some edges to nodes with distance k + 1 from v are in the set, but the nodes are not), and with the latter case it is not connected (as the nodes with distance k + 1 from v are in the set but none of the edges are), so neither of them are valid aggregation regions in our framework (our deﬁnition of GNN requires the region to be a connected graph).
Revised: With the former case M k ( v ) is not a valid subgraph (as some edges to nodes with distance k + 1 from v are in the set, but the nodes are not), and with the latter case it is not connected (as the nodes with distance k + 1 from v are in the set but none of the edges are), so neither of them are valid aggregation regions in our framework (our deﬁnition of GNN requires the region to be a connected graph).

Original: Thus, we reach a contradiction.
Revised: Thus, we reach a contradiction.

Original: 8. P ROOF OF T HEOREM
Revised: 8. P ROOF OF T HEOREM

Original: Theorem.
Revised: Theorem.

Original: The maximum discriminative power of the set of GNNs in G ( L 1 ) is strictly greater than the 1-dimensional WL test.
Revised: The maximum discriminative power of the set of GNNs in G ( L 1 ) is strictly greater than the 1-dimensional WL test.

Original: Proof.
Revised: Proof.

Original: We ﬁrst note that a GNN with the aggregation function (in matrix notation):
Revised: We ﬁrst note that a GNN with the aggregation function (in matrix notation):

Original: [Equation]
Revised: [Equation]

Original: is a G ( L 1 ) -class GNN.
Revised: is a G ( L 1 ) -class GNN.

Original: Then note that ( A 3 ) ii is the number of length 3 walks that start and end at i .
Revised: Then note that ( A 3 ) ii is the number of length 3 walks that start and end at i .

Original: These walks must be simple 3-cycles, and thus ( A 3 ) ii is twice the number of triangles that contains i (since for a triangle { i, j, k } , there would be two walks i → j → k → i and i
Revised: These walks must be simple 3-cycles, and thus ( A 3 ) ii is twice the number of triangles that contains i (since for a triangle { i, j, k } , there would be two walks i → j → k → i and i

Original: → k → j → i ).
Revised: → k → j → i ).

Original: Fürer (2017) showed that 1-WL test cannot measure the number of triangles in a graph (while 2-WL test can), so there exist graphs G 1 and G 2 such that 1-WL test cannot differentiate but the GNN above can due to different number of triangles in these two graphs (an example are two regular graphs with the same degree and same number of nodes).
Revised: Fürer (2017) showed that 1-WL test cannot measure the number of triangles in a graph (while 2-WL test can), so there exist graphs G 1 and G 2 such that 1-WL test cannot differentiate but the GNN above can due to different number of triangles in these two graphs (an example are two regular graphs with the same degree and same number of nodes).

Original: Now Xu et al.
Revised: Now Xu et al.

Original: (2018) proved that G ( D 1 ) has a maximum discriminatory power of 1-WL, and since
Revised: (2018) proved that G ( D 1 ) has a maximum discriminatory power of 1-WL, and since

Original: L 1 (cid:41) D 1 , the maximum discriminatory power of G ( L 1 ) is at least as great than that of G ( D 1 ) , which is 1-WL.
Revised: L 1 (cid:41) D 1 , the maximum discriminatory power of G ( L 1 ) is at least as great than that of G ( D 1 ) , which is 1-WL.

Original: Thus combining these two results give the required theorem.
Revised: Thus combining these two results give the required theorem.

Original: We here note the computational complexity of A 3 using naive matrix multiplication requires O ( N 3 ) multiplications.
Revised: We here note the computational complexity of A 3 using naive matrix multiplication requires O ( N 3 ) multiplications.

Original: However, by exploiting the sparsity and binary nature of A , there exist algorithms that can calculate A k with O (Ω N ) additions (Razzaque et al.
Revised: However, by exploiting the sparsity and binary nature of A , there exist algorithms that can calculate A k with O (Ω N ) additions (Razzaque et al.

Original: , and we thus derive a more favorable bound.
Revised: , and we thus derive a more favorable bound.

Original: 8. P ROOF OF T HEOREM
Revised: 8. P ROOF OF T HEOREM

Original: Theorem.
Revised: Theorem.

Original: For all k ∈ Z + , there exists a GNN within the class of G ( L k ) that is able to discriminate all graphs with k + 1 nodes.
Revised: For all k ∈ Z + , there exists a GNN within the class of G ( L k ) that is able to discriminate all graphs with k + 1 nodes.

Original: Proof.
Revised: Proof.

Original: We would prove by induction on k .
Revised: We would prove by induction on k .

Original: The base case for k = 1 is simple.
Revised: The base case for k = 1 is simple.

Original: Assume the statement is true for all k ≤ k (cid:48) − 1 .
Revised: Assume the statement is true for all k ≤ k (cid:48) − 1 .

Original: Let us prove the case for k = k (cid:48) ≥ 2 .
Revised: Let us prove the case for k = k (cid:48) ≥ 2 .

Original: We would separate the proof into three cases:
Revised: We would separate the proof into three cases:

Original: • G 1 and G 2 are both disconnected.
Revised: • G 1 and G 2 are both disconnected.

Original: • One of G 1 , G 2 is disconnected.
Revised: • One of G 1 , G 2 is disconnected.

Original: • G 1 and G 2 are both connected.
Revised: • G 1 and G 2 are both connected.

Original: If we can create appropriate GNNs to discriminate graphs in each of the three cases (say f 1 , f 2 , f 3 ), then the concatenated function f :=
Revised: If we can create appropriate GNNs to discriminate graphs in each of the three cases (say f 1 , f 2 , f 3 ), then the concatenated function f :=

Original: [ f 1 , f 2 , f 3 ] can discriminate all graphs.
Revised: [ f 1 , f 2 , f 3 ] can discriminate all graphs.

Original: Therefore, we would prove the induction step separately for the three cases below.
Revised: Therefore, we would prove the induction step separately for the three cases below.

Original: 8.4. G 1 AND G 2 DISCONNECTED
Revised: 8.4. G 1 AND G 2 DISCONNECTED

Original: We would use the Graph Reconstruction Conjecture as proved in the disconnected case (Harary, 1974):
Revised: We would use the Graph Reconstruction Conjecture as proved in the disconnected case (Harary, 1974):

Original: Lemma 1.
Revised: Lemma 1.

Original: For all k ≥ 2 , two disconnected graphs G 1 , G 2 with | V 1 | = | V 2 | =
Revised: For all k ≥ 2 , two disconnected graphs G 1 , G 2 with | V 1 | = | V 2 | =

Original: k are isomorphic if and only if the set of k − 1 sized subgraphs for these two graphs are isomorphic.
Revised: k are isomorphic if and only if the set of k − 1 sized subgraphs for these two graphs are isomorphic.

Original: Let G 1 and G 2 be any two disconnected graphs with k (cid:48)
Revised: Let G 1 and G 2 be any two disconnected graphs with k (cid:48)

Original: + 1 nodes.
Revised: + 1 nodes.

Original: By the induction assumption, there exist a GNN in G ( L k (cid:48) )
Revised: By the induction assumption, there exist a GNN in G ( L k (cid:48) )

Original: such that it discriminates all graphs with k (cid:48) nodes.
Revised: such that it discriminates all graphs with k (cid:48) nodes.

Original: Denote that f k (cid:48) :
Revised: Denote that f k (cid:48) :

Original: G → R k (cid:48) × p .
Revised: G → R k (cid:48) × p .

Original: Then by Lemma 1 above, we know that G 1 and G 2 are isomorphic iff:
Revised: Then by Lemma 1 above, we know that G 1 and G 2 are isomorphic iff:

Original: [Equation]
Revised: [Equation]

Original: Where G 11 , · · · G k (cid:48)
Revised: Where G 11 , · · · G k (cid:48)

Original: +1are the k (cid:48)
Revised: +1are the k (cid:48)

Original: + 1 subgraphs of G 1 with size k (cid:48) , and similarly for G 2 .
Revised: + 1 subgraphs of G 1 with size k (cid:48) , and similarly for G 2 .

Original: Then we deﬁne:
Revised: Then we deﬁne:

Original: [Equation]
Revised: [Equation]

Original: Then, by Theorem 4.3 in Wagstaff et al.
Revised: Then, by Theorem 4.3 in Wagstaff et al.

Original: (2019), f k (cid:48)
Revised: (2019), f k (cid:48)

Original: +1 ( G ) is an injective, permutation-invariant function on { f k (cid:48)
Revised: +1 ( G ) is an injective, permutation-invariant function on { f k (cid:48)

Original: ( G 1 ) , · · · f k (cid:48)
Revised: ( G 1 ) , · · · f k (cid:48)

Original: ( G k (cid:48) +1 ) } .
Revised: ( G k (cid:48) +1 ) } .

Original: Therefore, f k (cid:48) +1 ( G ) is a GNN with an aggregation region of L k (cid:48)
Revised: Therefore, f k (cid:48) +1 ( G ) is a GNN with an aggregation region of L k (cid:48)

Original: +1 that can discriminate G 1 and G 2 .
Revised: +1 that can discriminate G 1 and G 2 .

Original: Thus, the induction step is proven.
Revised: Thus, the induction step is proven.

Original: 8.4. G 1 AND G 2 CONNECTED
Revised: 8.4. G 1 AND G 2 CONNECTED

Original: In the case where both G 1 and G 2 are connected, let v 1 , u 1 be two vertices from each of the two graphs.
Revised: In the case where both G 1 and G 2 are connected, let v 1 , u 1 be two vertices from each of the two graphs.

Original: Note that since G 1 and G 2 have k (cid:48)
Revised: Note that since G 1 and G 2 have k (cid:48)

Original: + 1 nodes, by deﬁnition of L k (cid:48) , we have L k (cid:48)
Revised: + 1 nodes, by deﬁnition of L k (cid:48) , we have L k (cid:48)

Original: ( v 1 ) =
Revised: ( v 1 ) =

Original: Gand L k (cid:48)
Revised: Gand L k (cid:48)

Original: ( u 1 ) =
Revised: ( u 1 ) =

Original: G 2 .
Revised: G 2 .

Original: Since v 1 , u 1 are arbitrary, every node in the two graphs has an aggregation region of its entire graph.
Revised: Since v 1 , u 1 are arbitrary, every node in the two graphs has an aggregation region of its entire graph.

Original: Then we deﬁne the Aggregation function as followed:
Revised: Then we deﬁne the Aggregation function as followed:

Original: [Equation]
Revised: [Equation]

Original: Where A L k (cid:48) is the adjacency matrix restricted to the L k (cid:48) ( v ) subgraph, and LO( A ) returns the lexicographical smallest ordering of the adjacency matrix as a row-major vector among all isomorphic permutations (where nodes are relabeled).
Revised: Where A L k (cid:48) is the adjacency matrix restricted to the L k (cid:48) ( v ) subgraph, and LO( A ) returns the lexicographical smallest ordering of the adjacency matrix as a row-major vector among all isomorphic permutations (where nodes are relabeled).

Original: For example, take:
Revised: For example, take:

Original: [Equation]
Revised: [Equation]

Original: There are two isomorphic permutations of this adjacency matrix, which are:
Revised: There are two isomorphic permutations of this adjacency matrix, which are:

Original: [Equation]
Revised: [Equation]

Original: The row-major vectors of these two adjacency matrices are [1 , 1 , 1 , 0] and [0 , 1 , 1 , 1] , in which [0 , 1 , 1 , 1] is lexicographical smaller, so LO( A ) =
Revised: The row-major vectors of these two adjacency matrices are [1 , 1 , 1 , 0] and [0 , 1 , 1 , 1] , in which [0 , 1 , 1 , 1] is lexicographical smaller, so LO( A ) =

Original: Note that this function is permutation invariant.
Revised: Note that this function is permutation invariant.

Original: Then for G 1 and G 2 connected, A L k (cid:48) ( v ) is always the adjacency matrix of the full graph.
Revised: Then for G 1 and G 2 connected, A L k (cid:48) ( v ) is always the adjacency matrix of the full graph.

Original: Therefore if G 1 and G 2 are connected and isomorphic, then their adjacency matrices are permutations of each other, and thus their lexicographical smallest ordering of the row-major vector form of the adjacency matrix are identical.
Revised: Therefore if G 1 and G 2 are connected and isomorphic, then their adjacency matrices are permutations of each other, and thus their lexicographical smallest ordering of the row-major vector form of the adjacency matrix are identical.

Original: The converse is also clearly true as the adjacency matrix determines the graph.
Revised: The converse is also clearly true as the adjacency matrix determines the graph.

Original: Therefore, this function discriminates all connected graphs of k (cid:48)
Revised: Therefore, this function discriminates all connected graphs of k (cid:48)

Original: + 1 nodes, and the induction step is proven.
Revised: + 1 nodes, and the induction step is proven.

Original: 8.4. G 1 DISCONNECTED AND G 2 CONNECTED
Revised: 8.4. G 1 DISCONNECTED AND G 2 CONNECTED

Original: In the case where G 1 is disconnected and G 2 is connected, we deﬁne the aggregation function as the number of vertices in L k (cid:48) ( v ) , denoted | V L k (cid:48)
Revised: In the case where G 1 is disconnected and G 2 is connected, we deﬁne the aggregation function as the number of vertices in L k (cid:48) ( v ) , denoted | V L k (cid:48)

Original: ( v ) | :
Revised: ( v ) | :

Original: [Equation]
Revised: [Equation]

Original: v )
Revised: v )

Original: [Equation]
Revised: [Equation]

Original: This is a permutation-invariant function.
Revised: This is a permutation-invariant function.

Original: Note that for the connected graph G 2 and any vertex v in
Revised: Note that for the connected graph G 2 and any vertex v in

Original: G 2 , this function returns Agg( v ) =
Revised: G 2 , this function returns Agg( v ) =

Original: k (cid:48)
Revised: k (cid:48)

Original: + 1 as L k (cid:48) ( v ) =
Revised: + 1 as L k (cid:48) ( v ) =

Original: G 2 .
Revised: G 2 .

Original: Therefore, every node has the same embedding k (cid:48)
Revised: Therefore, every node has the same embedding k (cid:48)

Original: + 1 .
Revised: + 1 .

Original: On the other hand, for the disconnected graph G 1 , let [ U 1 , · · · , U m ] be the connected components of
Revised: On the other hand, for the disconnected graph G 1 , let [ U 1 , · · · , U m ] be the connected components of

Original: G 1 .
Revised: G 1 .

Original: Then for a vertex u ∈ U i , it is clear that L k (cid:48) ( u )
Revised: Then for a vertex u ∈ U i , it is clear that L k (cid:48) ( u )

Original: =
Revised: =

Original: U i , and thus Agg( u ) =
Revised: U i , and thus Agg( u ) =

Original: | V U i | for all u ∈ U i .
Revised: | V U i | for all u ∈ U i .

Original: And since | V U i | < k (cid:48) + 1 by construction, Agg( u ) < k (cid:48) + 1 for all u ∈ U
Revised: And since | V U i | < k (cid:48) + 1 by construction, Agg( u ) < k (cid:48) + 1 for all u ∈ U

Original: i , so the embedding of Gand G 2 are never equal when G 1 is connected and G 2 is disconnected.
Revised: i , so the embedding of Gand G 2 are never equal when G 1 is connected and G 2 is disconnected.

Original: Therefore, this function discriminate all graphs of k (cid:48)
Revised: Therefore, this function discriminate all graphs of k (cid:48)

Original: + 1 nodes in which one is connected and one is disconnected, so the induction step is proven.
Revised: + 1 nodes in which one is connected and one is disconnected, so the induction step is proven.