Original: I MPROVING G RAPH N EURAL N ETWORK E XPRESSIVITY VIA S UBGRAPH I SOMORPHISM C OUNTING
Revised: I MPROVING G RAPH N EURAL N ETWORK E XPRESSIVITY VIA S UBGRAPH I SOMORPHISM C OUNTING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While Graph Neural Networks (GNNs) have achieved remarkable results in a variety of applications, recent studies exposed important shortcomings in their ability to capture the structure of the underlying graph.
Revised: While Graph Neural Networks (GNNs) have achieved remarkable results in a variety of applications, recent studies exposed important shortcomings in their ability to capture the structure of the underlying graph.

Original: It has been shown that the expressive power of standard GNNs is bounded by the Weisfeiler-Lehman (WL) graph isomorphism test, from which they inherit proven limitations such as the inability to detect and count graph substructures.
Revised: It has been shown that the expressive power of standard GNNs is bounded by the Weisfeiler-Lehman (WL) graph isomorphism test, from which they inherit proven limitations such as the inability to detect and count graph substructures.

Original: On the other hand, there is signiﬁcant empirical evidence, e.g. in network science and bioinformatics, that substructures are often informative for downstream tasks, suggesting that it is desirable to design GNNs capable of leveraging this important source of information.
Revised: On the other hand, there is signiﬁcant empirical evidence, e.g. in network science and bioinformatics, that substructures are often informative for downstream tasks, suggesting that it is desirable to design GNNs capable of leveraging this important source of information.

Original: To this end, we propose a novel topologically-aware message passing scheme based on substructure encoding.
Revised: To this end, we propose a novel topologically-aware message passing scheme based on substructure encoding.

Original: We show that our architecture allows incorporating domain-speciﬁc inductive biases and that it is strictly more expressive than the WL test.
Revised: We show that our architecture allows incorporating domain-speciﬁc inductive biases and that it is strictly more expressive than the WL test.

Original: Importantly, in contrast to recent works on the expressivity of GNNs, we do not attempt to adhere to the WL hierarchy; this allows us to retain multiple attractive properties of standard GNNs such as locality and linear network complexity, while being able to disambiguate even hard instances of graph isomorphism.
Revised: Importantly, in contrast to recent works on the expressivity of GNNs, we do not attempt to adhere to the WL hierarchy; this allows us to retain multiple attractive properties of standard GNNs such as locality and linear network complexity, while being able to disambiguate even hard instances of graph isomorphism.

Original: We extensively evaluate our method on graph classiﬁcation and regression tasks and show state-of-the-art results on multiple datasets including molecular graphs and social networks.
Revised: We extensively evaluate our method on graph classiﬁcation and regression tasks and show state-of-the-art results on multiple datasets including molecular graphs and social networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ﬁeld of graph representation learning has undergone a rapid growth in the past few years.
Revised: The ﬁeld of graph representation learning has undergone a rapid growth in the past few years.

Original: In particular, Graph Neural Networks (GNNs), a family of neural architectures designed for irregularly structured data, have been successfully applied to problems ranging from social networks and recommender systems (Ying et al., 2018a) to bioinformatics (Fout et al., 2017; Gainza et al., 2020), chemistry (Duvenaud et al., 2015; Gilmer et al., 2017; Sanchez-Lengeling et al., 2019) and physics (Kipf et al., 2018; Battaglia et al., 2016), to name a few.
Revised: In particular, Graph Neural Networks (GNNs), a family of neural architectures designed for irregularly structured data, have been successfully applied to problems ranging from social networks and recommender systems (Ying et al., 2018a) to bioinformatics (Fout et al., 2017; Gainza et al., 2020), chemistry (Duvenaud et al., 2015; Gilmer et al., 2017; Sanchez-Lengeling et al., 2019) and physics (Kipf et al., 2018; Battaglia et al., 2016), to name a few.

Original: Most GNN architectures are based on message passing (Gilmer et al., 2017), where at each layer the nodes are updated by information aggregated from their neighbours.
Revised: Most GNN architectures are based on message passing (Gilmer et al., 2017), where at each layer the nodes are updated by information aggregated from their neighbours.

Original: A crucial difference from traditional neural networks operating on grid-structured data is the absence of canonical ordering of the nodes in a graph.
Revised: A crucial difference from traditional neural networks operating on grid-structured data is the absence of canonical ordering of the nodes in a graph.

Original: To address this, the aggregation function is constructed to be invariant to neighbourhood permutations and, as a consequence, to graph isomorphism.
Revised: To address this, the aggregation function is constructed to be invariant to neighbourhood permutations and, as a consequence, to graph isomorphism.

Original: This kind of symmetry is not always desirable and thus different inductive biases that disambiguate the neighbours have been proposed.
Revised: This kind of symmetry is not always desirable and thus different inductive biases that disambiguate the neighbours have been proposed.

Original: For instance, in geometric graphs, such as 3D molecular graphs and meshes, directional biases are usually employed in order to model the positional information of the nodes (Masci et al., 2015; Monti et al., 2017; Bouritsas et al., 2019; Klicpera et al., 2020; de Haan et al., 2020b); for proteins, ordering information is used to disambiguate amino-acids at different positions in the sequence (Ingraham et al., 2019); in multi-relational knowledge graphs, a different aggregation is performed for each relation type (Schlichtkrull et al., 2018).
Revised: For instance, in geometric graphs, such as 3D molecular graphs and meshes, directional biases are usually employed in order to model the positional information of the nodes (Masci et al., 2015; Monti et al., 2017; Bouritsas et al., 2019; Klicpera et al., 2020; de Haan et al., 2020b); for proteins, ordering information is used to disambiguate amino-acids at different positions in the sequence (Ingraham et al., 2019); in multi-relational knowledge graphs, a different aggregation is performed for each relation type (Schlichtkrull et al., 2018).

Original: The structure of the graph itself does not usually explicitly take part in the aggregation function.
Revised: The structure of the graph itself does not usually explicitly take part in the aggregation function.

Original: In fact, most models rely on multiple message passing steps as a means for each node to discover the global structure of the graph.
Revised: In fact, most models rely on multiple message passing steps as a means for each node to discover the global structure of the graph.

Original: However, since message-passing GNNs are at most as powerful as the Weisfeiler Lehman test (WL)
Revised: However, since message-passing GNNs are at most as powerful as the Weisfeiler Lehman test (WL)

Original: (Xu et al., 2019; Morris et al., 2019), they are limited in their abilities to adequately exploit the graph structure, e.g. by counting substructures (Arvind et al., 2019;
Revised: (Xu et al., 2019; Morris et al., 2019), they are limited in their abilities to adequately exploit the graph structure, e.g. by counting substructures (Arvind et al., 2019;

Original: Chen et al., 2020).
Revised: Chen et al., 2020).

Original: This uncovers a crucial limitation of GNNs, as substructures have been widely recognised as important in the study of complex networks.
Revised: This uncovers a crucial limitation of GNNs, as substructures have been widely recognised as important in the study of complex networks.

Original: For example, in molecular chemistry, functional groups and rings are related to a plethora of chemical properties, while cliques are related to protein complexes in Protein-Protein Interaction networks and community structure in social networks, respectively (Granovetter, 1982; Girvan & Newman, 2002).
Revised: For example, in molecular chemistry, functional groups and rings are related to a plethora of chemical properties, while cliques are related to protein complexes in Protein-Protein Interaction networks and community structure in social networks, respectively (Granovetter, 1982; Girvan & Newman, 2002).

Original: Motivated by this observation, in this work we propose Graph Substructure Network (GSN) , a new symmetry breaking mechanism for GNNs based on introducing structural biases in the aggregation function.
Revised: Motivated by this observation, in this work we propose Graph Substructure Network (GSN) , a new symmetry breaking mechanism for GNNs based on introducing structural biases in the aggregation function.

Original: In particular, each message is transformed differently depending on the topological relationship between the endpoint nodes.
Revised: In particular, each message is transformed differently depending on the topological relationship between the endpoint nodes.

Original: This relationship is expressed by counting the appearance of certain substructures, the choice of which allows us to provide the model with different inductive biases, based on the graph distribution at hand.
Revised: This relationship is expressed by counting the appearance of certain substructures, the choice of which allows us to provide the model with different inductive biases, based on the graph distribution at hand.

Original: The substructures are encoded as structural identiﬁers that are assigned to either the nodes or edges of the graph and can thus disambiguate the neighbouring nodes that take part in the aggregation.
Revised: The substructures are encoded as structural identiﬁers that are assigned to either the nodes or edges of the graph and can thus disambiguate the neighbouring nodes that take part in the aggregation.

Original: We characterise the expressivity of substructure encoding in GNNs, showing that GSNs are strictly more expressive than traditional GNNs for the vast majority of substructures while retaining the locality of message passing, as opposed to higher-order methods (Maron et al., 2019b;c;a; Morris et al., 2019) that follow the WL hierarchy (see Section 2).
Revised: We characterise the expressivity of substructure encoding in GNNs, showing that GSNs are strictly more expressive than traditional GNNs for the vast majority of substructures while retaining the locality of message passing, as opposed to higher-order methods (Maron et al., 2019b;c;a; Morris et al., 2019) that follow the WL hierarchy (see Section 2).

Original: In the limit, our model can yield a unique representation for every isomorphism class and is thus universal.
Revised: In the limit, our model can yield a unique representation for every isomorphism class and is thus universal.

Original: We provide an extensive experimental evaluation on hard instances of graph isomorphism testing (strongly regular graphs), as well as on real-world networks from the social and biological domains, including the recently introduced large-scale benchmarks (Dwivedi et al., 2020;
Revised: We provide an extensive experimental evaluation on hard instances of graph isomorphism testing (strongly regular graphs), as well as on real-world networks from the social and biological domains, including the recently introduced large-scale benchmarks (Dwivedi et al., 2020;

Original: Hu et al., 2020).
Revised: Hu et al., 2020).

Original: We observe that when choosing the structural inductive biases based on domain-speciﬁc knowledge, GSN achieves state-ofthe-art results.
Revised: We observe that when choosing the structural inductive biases based on domain-speciﬁc knowledge, GSN achieves state-ofthe-art results.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Let G =
Revised: Let G =

Original: ( V G , E G ) be a graph with vertex set V G and undirected edge set E G .
Revised: ( V G , E G ) be a graph with vertex set V G and undirected edge set E G .

Original: A subgraph
Revised: A subgraph

Original: G S =
Revised: G S =

Original: ( V G S , E G S ) of G is any graph with V G S ⊆
Revised: ( V G S , E G S ) of G is any graph with V G S ⊆

Original: V G , E G S ⊆
Revised: V G , E G S ⊆

Original: E G .
Revised: E G .

Original: When E G S includes all the edges of G with endpoints in V G S , i.e. E G S
Revised: When E G S includes all the edges of G with endpoints in V G S , i.e. E G S

Original: = { ( v, u ) ∈
Revised: = { ( v, u ) ∈

Original: E : v, u ∈ V G S } , the subgraph is said to be induced .
Revised: E : v, u ∈ V G S } , the subgraph is said to be induced .

Original: Isomorphisms Two graphs G, H are isomorphic (denoted H (cid:39) G ), if there exists an adjacencypreserving bijective mapping ( isomorphism )
Revised: Isomorphisms Two graphs G, H are isomorphic (denoted H (cid:39) G ), if there exists an adjacencypreserving bijective mapping ( isomorphism )

Original: f : V G → V H , i.e. ( v,
Revised: f : V G → V H , i.e. ( v,

Original: u ) ∈
Revised: u ) ∈

Original: E G iff ( f ( v ) , f ( u ))
Revised: E G iff ( f ( v ) , f ( u ))

Original: ∈
Revised: ∈

Original: E H .
Revised: E H .

Original: Given some small graph H , the subgraph isomorphism problem amounts to ﬁnding a subgraph G S of G such that G S (cid:39) H .
Revised: Given some small graph H , the subgraph isomorphism problem amounts to ﬁnding a subgraph G S of G such that G S (cid:39) H .

Original: An automorphism of H is an isomorphism that maps H onto itself.
Revised: An automorphism of H is an isomorphism that maps H onto itself.

Original: The set of all the unique automorphisms form the automorphism group of the graph, denoted as Aut( H ) , contains all the possible symmetries of the graph.
Revised: The set of all the unique automorphisms form the automorphism group of the graph, denoted as Aut( H ) , contains all the possible symmetries of the graph.

Original: The automorphism group yields a partition of the vertices into disjoint subsets of V H called orbits .
Revised: The automorphism group yields a partition of the vertices into disjoint subsets of V H called orbits .

Original: Intuitively, this concept allows us to group the vertices based on their structural roles , e.g. the end vertices of a path, or all the vertices of a cycle (see Figure 1).
Revised: Intuitively, this concept allows us to group the vertices based on their structural roles , e.g. the end vertices of a path, or all the vertices of a cycle (see Figure 1).

Original: Formally, the orbit of a vertex v ∈ V H is the set of vertices to which it can be mapped via an automorphism:
Revised: Formally, the orbit of a vertex v ∈ V H is the set of vertices to which it can be mapped via an automorphism:

Original: Orb( v ) = { u ∈ V H : ∃ g ∈ Aut( H ) s .
Revised: Orb( v ) = { u ∈ V H : ∃ g ∈ Aut( H ) s .

Original: t .
Revised: t .

Original: g ( u ) = v } , and the set of all orbits H \ Aut( H ) =
Revised: g ( u ) = v } , and the set of all orbits H \ Aut( H ) =

Original: { Orb( v ) : v ∈ V H } is usually called the quotient of the automorphism when it acts on the graph H .
Revised: { Orb( v ) : v ∈ V H } is usually called the quotient of the automorphism when it acts on the graph H .

Original: We are interested in the unique elements of this set that we will denote as { O VH, 1 , O VH, 2 , . . .
Revised: We are interested in the unique elements of this set that we will denote as { O VH, 1 , O VH, 2 , . . .

Original: , O VH,d H } , where d H is the cardinality of the quotient.
Revised: , O VH,d H } , where d H is the cardinality of the quotient.

Original: Analogously, we deﬁne edge structural roles via edge automorphisms , i.e. bijective mappings from the edge set onto itself, that preserve edge adjacency (two edges are adjacent if they share a common endpoint).
Revised: Analogously, we deﬁne edge structural roles via edge automorphisms , i.e. bijective mappings from the edge set onto itself, that preserve edge adjacency (two edges are adjacent if they share a common endpoint).

Original: In particular, every vertex automorphism g induces an edge automorphism by mapping each edge { u, v } to { g ( u ) , g ( v ) } .
Revised: In particular, every vertex automorphism g induces an edge automorphism by mapping each edge { u, v } to { g ( u ) , g ( v ) } .

Original: In the same way as before, we construct the edge automorphism group, from which we deduce the partition of the edge set in edge orbits { O EH, 1 , O EH, 2 , . . .
Revised: In the same way as before, we construct the edge automorphism group, from which we deduce the partition of the edge set in edge orbits { O EH, 1 , O EH, 2 , . . .

Original: , O EH,d H } .
Revised: , O EH,d H } .

Original: Weisfeiler-Lehman tests: The Weisfeiler-Lehman graph-isomorphism test (Weisfeiler & Leman, 1968), also known as naive vertex reﬁnement, 1-WL , or just WL ), is a fast heuristic to decide if two graphs are isomorphic or not.
Revised: Weisfeiler-Lehman tests: The Weisfeiler-Lehman graph-isomorphism test (Weisfeiler & Leman, 1968), also known as naive vertex reﬁnement, 1-WL , or just WL ), is a fast heuristic to decide if two graphs are isomorphic or not.

Original: The WL test proceeds as follows: every vertex v is initially assigned a colour c 0 ( v ) that is later iteratively reﬁned by aggregating neighbouring information:
Revised: The WL test proceeds as follows: every vertex v is initially assigned a colour c 0 ( v ) that is later iteratively reﬁned by aggregating neighbouring information:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) (cid:17) , where (cid:72) · (cid:73) denotes a multiset (a set that allows element repetitions) and N ( v ) is the neighbourhood of v .
Revised: ) (cid:17) , where (cid:72) · (cid:73) denotes a multiset (a set that allows element repetitions) and N ( v ) is the neighbourhood of v .

Original: The WL algorithm terminates when the colours stop changing, and outputs a histogram of colours.
Revised: The WL algorithm terminates when the colours stop changing, and outputs a histogram of colours.

Original: Two graphs with different histograms are nonisomorphic; if the histograms are identical, the graphs are possibly, but not necessarily, isomorphic.
Revised: Two graphs with different histograms are nonisomorphic; if the histograms are identical, the graphs are possibly, but not necessarily, isomorphic.

Original: Note that the neighbour aggregation in the WL test is a form of message passing, and GNNs are the learnable analogue.
Revised: Note that the neighbour aggregation in the WL test is a form of message passing, and GNNs are the learnable analogue.

Original: A series of works on improving GNN expressivity mimic the higher-order generalisations of WL, known as k -WL and k -Folklore WL (WL hierarchy) and operate on k -tuples of nodes (see Appendix
Revised: A series of works on improving GNN expressivity mimic the higher-order generalisations of WL, known as k -WL and k -Folklore WL (WL hierarchy) and operate on k -tuples of nodes (see Appendix

Original: B.1).
Revised: B.1).

Original: The ( k + 1) -FWL is strictly stronger than k -FWL, k -FWL is as strong as ( k + 1) -WL and 2-FWL is strictly stronger than the simple 1-WL test.
Revised: The ( k + 1) -FWL is strictly stronger than k -FWL, k -FWL is as strong as ( k + 1) -WL and 2-FWL is strictly stronger than the simple 1-WL test.

Original:  G RAPH S UBSTRUCTURE N ETWORKS
Revised:  G RAPH S UBSTRUCTURE N ETWORKS

Original: Graphs consist of nodes (or edges) with repeated structural roles.
Revised: Graphs consist of nodes (or edges) with repeated structural roles.

Original: Thus, it is natural for a neural network to treat them in a similar manner, akin to weight sharing between local patches in CNNs for images (LeCun et al., 1989) or positional encodings in language models for sequential data (Sukhbaatar et al., 2015; Gehring et al., 2017; Vaswani et al., 2017).
Revised: Thus, it is natural for a neural network to treat them in a similar manner, akin to weight sharing between local patches in CNNs for images (LeCun et al., 1989) or positional encodings in language models for sequential data (Sukhbaatar et al., 2015; Gehring et al., 2017; Vaswani et al., 2017).

Original: Nevertheless, GNNs are usually unaware of the nodes’ different structural roles, since all nodes are treated equally when performing local operations.
Revised: Nevertheless, GNNs are usually unaware of the nodes’ different structural roles, since all nodes are treated equally when performing local operations.

Original: Despite the initial intuition that the neural network would be able to discover these roles by constructing deeper architectures, it has been shown that GNNs are ill-suited for this purpose and are blind to the existence of structural properties, e.g. triangles or larger cycles (Chen et al., 2020;
Revised: Despite the initial intuition that the neural network would be able to discover these roles by constructing deeper architectures, it has been shown that GNNs are ill-suited for this purpose and are blind to the existence of structural properties, e.g. triangles or larger cycles (Chen et al., 2020;

Original: Arvind et al., 2019).
Revised: Arvind et al., 2019).

Original: To this end, we propose to explicitly encode structural roles as part of message passing, in order to capture richer topological properties.
Revised: To this end, we propose to explicitly encode structural roles as part of message passing, in order to capture richer topological properties.

Original: Our method draws inspiration from Loukas (2020), where it was shown that GNNs become universal when the nodes in the graph are uniquely identiﬁed, i.e when they are equipped with different features.
Revised: Our method draws inspiration from Loukas (2020), where it was shown that GNNs become universal when the nodes in the graph are uniquely identiﬁed, i.e when they are equipped with different features.

Original: However, it is not clear how to choose these identiﬁers in a way that can allow the neural network to generalise.
Revised: However, it is not clear how to choose these identiﬁers in a way that can allow the neural network to generalise.

Original: Structural roles, when treated as identiﬁers, although not necessarily unique, are more amenable to generalisation due to their repetition across different graphs.
Revised: Structural roles, when treated as identiﬁers, although not necessarily unique, are more amenable to generalisation due to their repetition across different graphs.

Original: Thus, they can constitute a trade-off between uniqueness and generalisation.
Revised: Thus, they can constitute a trade-off between uniqueness and generalisation.

Original: Structural features: Structural roles are encoded into features by counting the appearance of certain substructures.
Revised: Structural features: Structural roles are encoded into features by counting the appearance of certain substructures.

Original: The larger the substructure collection, the more ﬁne-grained the partition of nodes into roles will be.
Revised: The larger the substructure collection, the more ﬁne-grained the partition of nodes into roles will be.

Original: Let H
Revised: Let H

Original: = { H 1 , H 2 . . .
Revised: = { H 1 , H 2 . . .

Original: H K } be a set of small (connected) graphs, for example cycles of ﬁxed length or cliques.
Revised: H K } be a set of small (connected) graphs, for example cycles of ﬁxed length or cliques.

Original: For each graph H ∈ H , we ﬁrst ﬁnd its isomorphic subgraphs in G denoted as G S .
Revised: For each graph H ∈ H , we ﬁrst ﬁnd its isomorphic subgraphs in G denoted as G S .

Original: For each node v ∈ V G S we infer its role w.r.t.
Revised: For each node v ∈ V G S we infer its role w.r.t.

Original: H by obtaining the orbit of its mapping f ( v ) in H , Orb H ( f ( v )) .
Revised: H by obtaining the orbit of its mapping f ( v ) in H , Orb H ( f ( v )) .

Original: By counting all the possible appearances of different orbits in v , we obtain the vertex structural feature x VH ( v ) of v , deﬁned as follows:
Revised: By counting all the possible appearances of different orbits in v , we obtain the vertex structural feature x VH ( v ) of v , deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: We divide the counts by the number of the automorphisms of H , since for every matched subgraph
Revised: We divide the counts by the number of the automorphisms of H , since for every matched subgraph

Original: G S there will always be | Aut( H ) | different ways to map it to H , thus these repetitions will be uninformative.
Revised: G S there will always be | Aut( H ) | different ways to map it to H , thus these repetitions will be uninformative.

Original: By combining the counts from different substructures in H and different orbits, we obtain the feature vector x V ( v ) =
Revised: By combining the counts from different substructures in H and different orbits, we obtain the feature vector x V ( v ) =

Original: [ x VH 1 ( v ) , . . .
Revised: [ x VH 1 ( v ) , . . .

Original: , x VH K ( v )]
Revised: , x VH K ( v )]

Original: ∈ N D × 1 of dimension D = (cid:80) H i ∈H d H i .
Revised: ∈ N D × 1 of dimension D = (cid:80) H i ∈H d H i .

Original: Similarly, we can deﬁne edge structural features x E H,i ( { u, v } ) by counting occurrences of edge automorphism orbits:
Revised: Similarly, we can deﬁne edge structural features x E H,i ( { u, v } ) by counting occurrences of edge automorphism orbits:

Original: [Equation]
Revised: [Equation]

Original: and the combined edge features x E ( { u, v } ) =
Revised: and the combined edge features x E ( { u, v } ) =

Original: [ x EH 1 ( { u, v } ) , . . .
Revised: [ x EH 1 ( { u, v } ) , . . .

Original: , x EH K ( { u, v } )] .
Revised: , x EH K ( { u, v } )] .

Original: An example of vertex and edge structural features is illustrated in Figure 1.
Revised: An example of vertex and edge structural features is illustrated in Figure 1.

Original: Structure-aware message passing: The key building block of our architecture is the graph substructure layer, deﬁned in a general manner as a Message Passing Neural Network (MPNN) (Gilmer et al., 2017), where now the messages from the neighbouring nodes also contain the structural information.
Revised: Structure-aware message passing: The key building block of our architecture is the graph substructure layer, deﬁned in a general manner as a Message Passing Neural Network (MPNN) (Gilmer et al., 2017), where now the messages from the neighbouring nodes also contain the structural information.

Original: In particular, each node v updates its state h t ( v ) by combining its previous state with the aggregated messages: h t +1 ( v ) =
Revised: In particular, each node v updates its state h t ( v ) by combining its previous state with the aggregated messages: h t +1 ( v ) =

Original: UP t +1 (cid:0) h t ( v ) , MSG t +1 ( v ) (cid:1) , where the UP t +1 function is a multilayer perceptron (MLP) and the message aggregation is a summation of features transformed by an MLP M t +1 as follows:
Revised: UP t +1 (cid:0) h t ( v ) , MSG t +1 ( v ) (cid:1) , where the UP t +1 function is a multilayer perceptron (MLP) and the message aggregation is a summation of features transformed by an MLP M t +1 as follows:

Original: [Equation]
Revised: [Equation]

Original: where the two variants, named GSN-v and GSN-e , correspond to vertex- or edge-counts, respectively, and e ( { u, v } ) denotes edge features.
Revised: where the two variants, named GSN-v and GSN-e , correspond to vertex- or edge-counts, respectively, and e ( { u, v } ) denotes edge features.

Original: The two variants are analogous to absolute and relative positional encodings in language models (Shaw et al., 2018; Dai et al., 2019). 
Revised: The two variants are analogous to absolute and relative positional encodings in language models (Shaw et al., 2018; Dai et al., 2019). It is important to note here that contrary to identiﬁer-based GNNs (Loukas, 2020; Sato et al., 2020;

Revised: Dasoulas et al., 2020) that obtain universality at the expense of permutation equivariance (since the identiﬁers are arbitrarily chosen with the sole requirement of being unique), GSNs retain the attractive inductive bias of the permutation equivariant property.

Revised: This stems from the fact that the process generating our structural identiﬁers (i.e. subgraph isomorphism) is permutation equivariant itself (proof provided in the appendix).

Original: How powerful are GSNs?
Revised: How powerful are GSNs?

Original: We now turn to the expressive power of GSNs in comparison to MPNNs and the WL tests, a key tool for the theoretical analysis of the expressivity of graph neural networks so far.
Revised: We now turn to the expressive power of GSNs in comparison to MPNNs and the WL tests, a key tool for the theoretical analysis of the expressivity of graph neural networks so far.

Original: Since GSN is a generalisation of MPNNs, it is easy to see that it is at least as powerful.
Revised: Since GSN is a generalisation of MPNNs, it is easy to see that it is at least as powerful.

Original: Importantly, GSNs have the capacity to learn functions that traditional MPNNs cannot learn.
Revised: Importantly, GSNs have the capacity to learn functions that traditional MPNNs cannot learn.

Original: The following observation derives directly from the analysis of the counting abilities of the 1-WL test (Arvind et al., 2019) and its extension to MPNNs (Chen et al., 2020) (for proofs, see Appendix A).
Revised: The following observation derives directly from the analysis of the counting abilities of the 1-WL test (Arvind et al., 2019) and its extension to MPNNs (Chen et al., 2020) (for proofs, see Appendix A.2).

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: GSN is at least as powerful as MPNN and the 1-WL test.
Revised: GSN is strictly more powerful than MPNN and the 1-WL test when one of the following holds: a) H is any graph except for star graphs of any size and structural features are inferred by subgraph matching, i.e. we count all subgraphs G S ∼ = H for which it holds that

Original: Moreover, GSN is strictly more powerful than MPNN and the 1-WL test when H is any induced subgraph except for single edges and single nodes, or any not necessarily induced subgraph except from star graphs of any size.
Revised: E G S ⊆ E G , or b) H is any graph except for single edges and single nodes and structural features are inferred by induced subgraph matching, i.e. we count all subgraphs G S ∼ = H for which it holds that E G S = E G ∩ (cid:0)

Original:  An important question is which substructures are the most informative and whether they can completely characterise the graph.
Revised: V G S × V G S (cid:1) . An important question is which substructures are the most informative and whether they can completely characterise the graph.

Original: As of today, we are not aware of any results in graph theory that can guarantee the reconstruction of a graph from a smaller collection of its subgraphs.
Revised: As of today, we are not aware of any results in graph theory that can guarantee the reconstruction of a graph from a smaller collection of its subgraphs.

Original: In fact, the
Revised: In fact, the

Original: Reconstruction Conjecture (Kelly et al., 1957; Ulam, 1960), states that a graph with size n ≥ 3 can be reconstructed from its vertex-deleted subgraphs.
Revised: Reconstruction Conjecture (Kelly et al., 1957; Ulam, 1960), states that a graph with size n ≥ 3 can be reconstructed from its vertex-deleted subgraphs.

Revised: Moreover, it is known (Dasoulas et al., 2020; Chen et al., 2019) that solving graph isomorphism is equivalent to universal approximation of permutation invariant functions deﬁned on graphs.

Original: Therefore, we can state the following for GSN:
Revised: Thus, we can state the following for GSN:

Original: Corollary 3.1.
Revised: Corollary 3.1.

Original: If the Reconstruction Conjecture holds, GSN can distinguish all non-isomorphic graphs of size n  when using all substructures of size k = n − 1 .
Revised: If the Reconstruction Conjecture holds, GSN can distinguish all non-isomorphic graphs of size n and is therefore universal, when H contains all substructures of size k = n − 1 .

Original: How to choose the substructures?
Revised: How to choose the substructures?

Original: The Reconstruction Conjecture has only been proven for n ≤ 11 (McKay, 1997) and still remains open for larger graphs, while to the best of our knowledge, there is no similar hypothesis for smaller values of k .
Revised: The Reconstruction Conjecture has only been proven for n ≤ 11 (McKay, 1997) and still remains open for larger graphs, while to the best of our knowledge, there is no similar hypothesis for smaller values of k .

Original: However, in practice we observe that small substructures of size k
Revised: However, in practice we observe that small substructures of size k

Original: = O (1) are sufﬁcient both for resolving hard instances of graph isomorphism as well as for tasks related to real-world networks.
Revised: = O (1) are sufﬁcient both for resolving hard instances of graph isomorphism as well as for tasks related to real-world networks.

Original: In particular, although our method does not attempt to align with the WL hierarchy, an interesting observation that we make is that small substructures have the capacity to distinguish strongly regular graphs, where 2-FWL provably fails (proven in Appendix B.3).
Revised: In particular, although our method does not attempt to align with the WL hierarchy, an interesting observation that we make is that small substructures have the capacity to distinguish strongly regular graphs, where 2-FWL provably fails (proven in Appendix B.3).

Original: This is illustrated in Figure 2 (right), where counting 4-cliques is sufﬁcient to disambiguate this exemplary pair, and in Section 5, where a large scale experimental study is conducted.
Revised: This is illustrated in Figure 2 (right), where counting 4-cliques is sufﬁcient to disambiguate this exemplary pair, and in Section 5, where a large scale experimental study is conducted.

Original: In real-world scenarios, using too many and too large subgraphs will not only be expensive to compute, but they might also lead to overﬁtting.
Revised: In real-world scenarios, using too many and too large subgraphs will not only be expensive to compute, but they might also lead to overﬁtting.

Original: As an alternative, one can choose only the most discriminative subgraphs, i.e. the ones that can achieve the maximum possible node disambiguation, similarly to identiﬁer-based approaches.
Revised: As an alternative, one can choose only the most discriminative subgraphs, i.e. the ones that can achieve the maximum possible node disambiguation, similarly to identiﬁer-based approaches.

Original: Note that this approach solely takes into account expressivity and does not provide any guarantee about generalisation.
Revised: Note that this approach solely takes into account expressivity and does not provide any guarantee about generalisation.

Original: This is empirically validated in Figure 3, where we observe that choosing substructures with strong node disambiguation properties allows for better ﬁtting of the training data, but the test set performance does not necessarily improve.
Revised: This is empirically validated in Figure 4, where we observe that choosing substructures with strong node disambiguation properties allows for better ﬁtting of the training data, but the test set performance does not necessarily improve.

Original: Aiming at better generalisation, it is desirable to make use of substructures for which there is substantial prior knowledge of their importance in certain network distributions and have been observed to be intimately related to various properties.
Revised: Aiming at better generalisation, it is desirable to make use of substructures for which there is substantial prior knowledge of their importance in certain network distributions and have been observed to be intimately related to various properties.

Original: For example, small substructures (graphlets) have been extensively analysed in protein-protein interaction networks (Prˇzulj et al., 2004), triangles and cliques characterise the structure of ego-nets and social networks in general (Granovetter, 1982), simple cycles (rings) are central in molecular distributions, directed and temporal motifs have been shown to explain the working mechanisms of gene regulatory networks, biological neural networks, transportation networks and food webs (Milo et al., 2002; Paranjape et al., 2017; Benson et al., 2016).
Revised: For example, small substructures (graphlets) have been extensively analysed in protein-protein interaction networks (Pr ˇ zulj et al., 2004), triangles and cliques characterise the structure of ego-nets and social networks in general (Granovetter, 1982), simple cycles (rings) are central in molecular distributions, directed and temporal motifs have been shown to explain the working mechanisms of gene regulatory networks, biological neural networks, transportation networks and food webs (Milo et al., 2002; Paranjape et al., 2017; Benson et al., 2016).

Original: In Figure 5 in the Appendix, we showcase the importance of these inductive biases: a cycle-based GSN predicting molecular properties achieves smaller generalisation gap compared to a traditional MPNN, while at the same time generalising better with less training data.
Revised: In Figure 6 in the Appendix, we showcase the importance of these inductive biases: a cycle-based GSN predicting molecular properties achieves smaller generalisation gap compared to a traditional MPNN, while at the same time generalising better with less training data.

Original: Choosing the best substructure collection is still an open problem; various heuristics can be used (motif frequencies, feature selection strategies) or ideally by learning the substructure dictionary in an end-to-end manner.
Revised: Choosing the best substructure collection is still an open problem; various heuristics can be used (motif frequencies, feature selection strategies) or ideally by learning the substructure dictionary in an end-to-end manner.

Original: Answering this question is left for future work.
Revised: Answering this question is left for future work.

Revised: GSN-v vs GSN-e : Finally, we examine the expressive power of the two proposed variants.

Revised: A crucial observation that we make, is that for each graph H in the collection, the vertex structural identiﬁers can be reconstructed by the corresponding edge identiﬁers.

Revised: Thus, we can show that for every GSN-v there exists a GSN-e that can simulate the behaviour of the former (proof provided in Appendix A.3).

Revised: Proposition 3.2.

Revised: For a given subgraph collection H , let C V the set of functions that can be expressed by a GSN-v with arbitrary depth and with, and C E the set of functions that can be expressed by a

Original:  Complexity: The complexity of GSN comprises two parts: precomputation (substructure counting) and training/testing.
Revised: GSN-e with the same properties. Then, it holds that C E ⊇ C V , or in other words GSN-e is at least as expressive as GSN-v. Complexity: The complexity of GSN comprises two parts: precomputation (substructure counting) and training/testing.

Original: The key appealing property is that training and inference are linear w.r.t the number of edges, O ( |E| ) , as opposed to higher-order methods (Maron et al., 2019a; Morris et al., 2019) with O ( n k ) training complexity and relational pooling (Murphy et al., 2019) with O ( n !)
Revised: The key appealing property is that training and inference are linear w.r.t the number of edges, O ( |E| ) , as opposed to higher-order methods (Maron et al., 2019a; Morris et al., 2019) with O ( n k ) training complexity and relational pooling (Murphy et al., 2019) with O ( n !)

Original: training complexity in absence of approximations.
Revised: training complexity in absence of approximations.

Original: The worst-case complexity of subgraph isomorphism of ﬁxed size k is O ( n k ) , by examining all the possible k -tuples in the graph.
Revised: The worst-case complexity of subgraph isomorphism of ﬁxed size k is O ( n k ) , by examining all the possible k -tuples in the graph.

Original: However, for speciﬁc types of subgraphs, such as paths and cycles, the problem can be solved even faster (see e.g. Giscard et al.
Revised: However, for speciﬁc types of subgraphs, such as paths and cycles, the problem can be solved even faster (see e.g. Giscard et al.

Original: Approximate counting algorithms are also widely used, especially for counting frequent network motifs (Kashtan et al., 2004; Wernicke, 2005; 2006; Wernicke & Rasche, 2006), and can provide a considerable speed-up.
Revised: Approximate counting algorithms are also widely used, especially for counting frequent network motifs (Kashtan et al., 2004; Wernicke, 2005; 2006; Wernicke & Rasche, 2006), and can provide a considerable speed-up.

Original: Furthermore, recent neural approaches (Ying et al., 2020b;a) provide fast approximate counting.
Revised: Furthermore, recent neural approaches (Ying et al., 2020b;a) provide fast approximate counting.

Original: In our experiments, we performed exact counting using the common isomorphism algorithm VF (Cordella et al., 2004).
Revised: In our experiments, we performed exact counting using the common isomorphism algorithm VF (Cordella et al., 2004).

Original: Although its worst case complexity is O ( n k ) , it scales better in practice, for instance when the candidate subgraph is infrequently matched or when the graphs are sparse, and is also trivially parallelisable.
Revised: Although its worst case complexity is O ( n k ) , it scales better in practice, for instance when the candidate subgraph is infrequently matched or when the graphs are sparse, and is also trivially parallelisable.

Revised: In Figure 3, we show a quantitative analysis of the empirical runtime of the counting algorithm against the worst case, for three different graph distributions: molecules, protein contact maps, social networks.

Revised: It is easy to see that when the graphs are sparse (for the ﬁrst two cases) and the number of matches is small, the algorithm is signiﬁcantly faster than the worst case, while it scales better with the size of the graph n .

Revised: Even, in the case of social networks, where several examples are near-complete graphs, both the runtime and the growth w.r.t both n and k are better than the worst case.

Original: Overall, the preprocessing computational burden in most of the cases remains negligible for relatively small and sparse graphs, as it is the case of molecules.
Revised: Overall, the preprocessing computational burden in most of the cases remains negligible for relatively small and sparse graphs, as it is the case of molecules.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Expressive power of GNNs and the WL hierarchy : The seminal results in the theoretical analysis of the expressivity of GNNs (Xu et al., 2019) and k-GNNs (Morris et al., 2019) established that traditional message passing-based GNNs are at most as powerful as the 1-WL test.
Revised: Expressive power of GNNs and the WL hierarchy : The seminal results in the theoretical analysis of the expressivity of GNNs (Xu et al., 2019) and k-GNNs (Morris et al., 2019) established that traditional message passing-based GNNs are at most as powerful as the 1-WL test.

Original: Chen et al.
Revised: Chen et al.

Original: (2019) showed that graph isomorphism is equivalent to universal invariant function approximation.
Revised: (2019) showed that graph isomorphism is equivalent to universal invariant function approximation.

Original: Higher-order Invariant Graph Networks (IGNs) have been studied in a series of works (Maron et al., 2019b;c;a;
Revised: Higher-order Invariant Graph Networks (IGNs) have been studied in a series of works (Maron et al., 2019b;c;a;

Original: Chen et al., 2019; Keriven & Peyr´e, 2019; Puny et al., 2020), establishing connections with the WL hierarchy, similarly to Morris et al.
Revised: Chen et al., 2019; Keriven & Peyr ´ e, 2019; Puny et al., 2020), establishing connections with the WL hierarchy, similarly to Morris et al.

Original: The main drawback of these methods is the training and inference time complexity and memory requirements of O ( n k ) as well as the super-exponential number of parameters for linear IGNs making them impractical.
Revised: The main drawback of these methods is the training and inference time complexity and memory requirements of O ( n k ) as well as the super-exponential number of parameters for linear IGNs making them impractical.

Original: From a different perspective, Sato et al.
Revised: From a different perspective, Sato et al.

Original: (2019) and Loukas (2020)
Revised: (2019) and Loukas (2020)

Original: showed the connections between
Revised: showed the connections between

Original: GNNs and distributed local algorithms (Angluin, 1980; Linial, 1992; Naor & Stockmeyer, 1993) and suggested more powerful alternatives based on either local orderings or unique global identiﬁers (in the form of random features in Sato et al.
Revised: GNNs and distributed local algorithms (Angluin, 1980; Linial, 1992; Naor & Stockmeyer, 1993) and suggested more powerful alternatives based on either local orderings or unique global identiﬁers (in the form of random features in Sato et al.

Original: that make GNNs universal.
Revised: that make GNNs universal.

Original: However, these methods lack a principled way to choose orderings/identiﬁers that can be shared across graphs (this would require a graph canonisation procedure).
Revised: However, these methods lack a principled way to choose orderings/identiﬁers that can be shared across graphs (this would require a graph canonisation procedure).

Original: Garg et al.
Revised: Garg et al.

Original: (2020) also analysed the expressive power of MPNNs and other more powerful variants and provided generalisation bounds.
Revised: (2020) also analysed the expressive power of MPNNs and other more powerful variants and provided generalisation bounds.

Original: Murphy et al.
Revised: Murphy et al.

Original: (2019) take into account all possible node permutations and can therefore be intractable.
Revised: (2019) take into account all possible node permutations and can therefore be intractable.

Original: Concurrently with our work, more expressive GNNs have been proposed using equivariant message passing with different kernels for each edge based on their isomorphism classes (de Haan et al., 2020a), message passing with matrices of order equal to the size of the graph instead of vectors (Vignac et al., 2020), or by enhancing node features with distance encodings (Li et al., 2020).
Revised: Concurrently with our work, more expressive GNNs have been proposed using equivariant message passing with different kernels for each edge based on their isomorphism classes (de Haan et al., 2020a), message passing with matrices of order equal to the size of the graph instead of vectors (Vignac et al., 2020), or by enhancing node features with distance encodings (Li et al., 2020).

Original: Solely quantifying the expressive power of GNNs in terms of their ability to distinguish nonisomorphic graphs does not provide the necessary granularity: even the 1-WL test can distinguish almost all (in the probabilistic sense) non-isomorphic graphs (Babai et al., 1980).
Revised: Solely quantifying the expressive power of GNNs in terms of their ability to distinguish nonisomorphic graphs does not provide the necessary granularity: even the 1-WL test can distinguish almost all (in the probabilistic sense) non-isomorphic graphs (Babai et al., 1980).

Original: As a result, there have been several efforts to analyse the power of k -WL tests in comparison to other graph invariants (F¨urer, 2010; 2017; Arvind et al., 2019;
Revised: As a result, there have been several efforts to analyse the power of k -WL tests in comparison to other graph invariants (F¨urer, 2010; 2017; Arvind et al., 2019;

Original: Dell et al., 2018), while recently Chen et al.
Revised: Dell et al., 2018), while recently Chen et al.

Original: (2020) approached GNN expressivity by studying their ability to count substructures.
Revised: (2020) approached GNN expressivity by studying their ability to count substructures.

Original: Substructures in Complex Networks:
Revised: Substructures in Complex Networks:

Original: The idea of analysing complex networks based on smallscale topological characteristics dates back to the 1970’s and the notion of triad census for directed graphs (Holland & Leinhardt, 1976).
Revised: The idea of analysing complex networks based on smallscale topological characteristics dates back to the 1970’s and the notion of triad census for directed graphs (Holland & Leinhardt, 1976).

Original: The seminal paper of Milo et al.
Revised: The seminal paper of Milo et al.

Original: (2002) coined the term network motifs as over-represented subgraph patterns that were shown to characterise certain functional properties of complex networks in systems biology.
Revised: (2002) coined the term network motifs as over-represented subgraph patterns that were shown to characterise certain functional properties of complex networks in systems biology.

Original: The closely related concept of graphlets (Prˇzulj et al., 2004; Pr ˇ zulj, 2007; Milenkovi ´ c & Pr ˇ zulj, 2008; Sarajli ´ c et al., 2016), different from motifs in being induced subgraphs, has been used to analyse the distribution of real-world networks and as a topological signature for network similarity.
Revised: The closely related concept of graphlets (Prˇzulj et al., 2004; Pr ˇ zulj, 2007; Milenkovi ´ c & Pr ˇ zulj, 2008; Sarajli ´ c et al., 2016), different from motifs in [Figure] [Figure] [Figure]

Revised: M e a n A b s o l u t e

Revised: E rr o r traintestPathsCyclesTreesMPNN -EF being induced subgraphs, has been used to analyse the distribution of real-world networks and as a topological signature for network similarity.

Original: Our work is similar in spirit with the graphlet degree vector (GDV)
Revised: Our work is similar in spirit with the graphlet degree vector (GDV)

Original: (Pr ˇ zulj, 2007), a node-wise descriptor based on graphlet counting.
Revised: (Pr ˇ zulj, 2007), a node-wise descriptor based on graphlet counting.

Original: Substructures have been also used in the context of ML.
Revised: Substructures have been also used in the context of ML.

Original: In particular, subgraph patterns have been used to deﬁne Graph Kernels (GKs)
Revised: In particular, subgraph patterns have been used to deﬁne Graph Kernels (GKs)

Original: (Horv ´ ath et al., 2004; Shervashidze et al., 2009; Costa &
Revised: (Horv ´ ath et al., 2004; Shervashidze et al., 2009; Costa &

Original: De Grave, 2010; Kriege & Mutzel, 2012; NT & Maehara, 2020), with the most prominent being the graphlet kernel (Shervashidze et al., 2009).
Revised: De Grave, 2010; Kriege & Mutzel, 2012; NT & Maehara, 2020), with the most prominent being the graphlet kernel (Shervashidze et al., 2009).

Original: Motif-based node embeddings (Dareddy et al., 2019;
Revised: Motif-based node embeddings (Dareddy et al., 2019;

Original: Rossi et al., 2018) and diffusion operators (Monti et al., 2018; Sankar et al., 2019; Lee et al., 2019) that employ adjacency matrices weighted according to motif occurrences, have recently been proposed for graph representation learning.
Revised: Rossi et al., 2018) and diffusion operators (Monti et al., 2018; Sankar et al., 2019; Lee et al., 2019) that employ adjacency matrices weighted according to motif occurrences, have recently been proposed for graph representation learning.

Original: Our formulation provides a unifying framework for these methods and it is the ﬁrst to analyse their expressive power.
Revised: Our formulation provides a unifying framework for these methods and it is the ﬁrst to analyse their expressive power.

Original: Finally, GNNs that operate in larger induced neighbourhoods (Li et al., 2019; Kim et al., 2019) or higher-order paths (Flam-Shepherd et al., 2020) have prohibitive complexity since the size of these neighbourhoods typically grows exponentially.
Revised: Finally, GNNs that operate in larger induced neighbourhoods (Li et al., 2019; Kim et al., 2019) or higher-order paths (Flam-Shepherd et al., 2020) have prohibitive complexity since the size of these neighbourhoods typically grows exponentially.

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: In the following section we evaluate GSN in comparison to the state-of-the-art in a variety of datasets from different domains.
Revised: In the following section we evaluate GSN in comparison to the state-of-the-art in a variety of datasets from different domains.

Original: We are interested in practical scenarios where the collection of subgraphs, as well as their size, are kept small.
Revised: We are interested in practical scenarios where the collection of subgraphs, as well as their size, are kept small.

Original: Depending on the dataset domain we experimented with typical substructure families ( cycles, paths and cliques ) and maximum substructure size k (note that for each setting, our substructure collection consists of all the substructures of the family with size ≤ k ).
Revised: Depending on the dataset domain we experimented with typical substructure families ( cycles, paths and cliques ) and maximum substructure size k (note that for each setting, our substructure collection consists of all the substructures of the family with size ≤ k ).

Original: We also experimented with both graphlets and motifs and observed similar performance in most cases.
Revised: We also experimented with both graphlets and motifs and observed similar performance in most cases.

Original: Appendix C provides details and shows additional experiments.
Revised: Appendix C provides details and shows additional experiments.

Original: Synthetic Graph Isomorphism test: We tested the ability of GSNs to decide if two graphs are non-isomorphic on a collection of Strongly Regular graphs of size up to 35 nodes, attempting to disambiguate pairs with the same number of nodes (for different sizes the problem becomes trivial).
Revised: Synthetic Graph Isomorphism test: We tested the ability of GSNs to decide if two graphs are non-isomorphic on a collection of Strongly Regular graphs of size up to 35 nodes, attempting to disambiguate pairs with the same number of nodes (for different sizes the problem becomes trivial).

Original: As we are only interested in the bias of the architecture itself, we use GSN with random weights to compute graph representations.
Revised: As we are only interested in the bias of the architecture itself, we use GSN with random weights to compute graph representations.

Original: Two graphs are deemed isomorphic if the Euclidean distance of their representations is smaller than a predeﬁned threshold (cid:15) .
Revised: Two graphs are deemed isomorphic if the Euclidean distance of their representations is smaller than a predeﬁned threshold (cid:15) .

Original: Figure 4 shows the failure percentage of our isomorphism test when using different graphlet substructures ( cycles, paths , and cliques ) of varying size k .
Revised: Figure 5 shows the failure percentage of our isomorphism test when using different graphlet substructures ( cycles, paths , and cliques ) of varying size k .

Original: Interestingly, the number of failure cases of GSN decreases rapidly as we increase k ; cycles and paths of maximum length k = 6 are enough to tell apart all the graphs in the dataset.
Revised: Interestingly, the number of failure cases of GSN decreases rapidly as we increase k ; cycles and paths of maximum length k = 6 are enough to tell apart all the graphs in the dataset.

Original: Note that the performance of cliques saturates, possibly because the largest clique in our dataset has 5 nodes.
Revised: Note that the performance of cliques saturates, possibly because the largest clique in our dataset has 5 nodes.

Original: Observe also the discrepancy between GSN-v and GSN-e. In particular, vertex-wise counts do not manage to distinguish all graphs, although missing only a few instances.
Revised: Observe also the discrepancy between GSN-v and GSN-e. In particular, vertex-wise counts do not manage to distinguish all graphs, although missing only a few instances.

Original: We hypothesise that this is because edge counts offer a higher level of granularity, allowing GSN to create a ﬁner partition of the nodes in the graph (thus stronger symmetry breaking properties).
Revised: We hypothesise that this is because edge counts offer a higher level of granularity, allowing GSN to create a ﬁner partition of the nodes in the graph (thus stronger symmetry breaking properties).

Original: Finally, 1-WL (Xu et al., 2019) and 2-FWL (Maron et al., 2019a) equivalent models demonstrate 100% failure, as expected from theory.[Figure] [Figure] [Figure]
Revised: Finally, 1-WL (Xu et al., 2019) and 2-FWL (Maron et al., 2019a) equivalent models demonstrate 100% failure, as expected from theory.

Original: M e a n A b s o l u t e

Original: E rr o r traintestPathsCyclesTreesMPNN -EF

Original: TUD Graph Classiﬁcation Benchmarks: We evaluate GSN on datasets from the classical TUD benchmarks.
Revised: TUD Graph Classiﬁcation Benchmarks: We evaluate GSN on datasets from the classical TUD benchmarks.

Original: We use seven datasets from the domains of bioinformatics and computational social science and compare against various GNNs and Graph Kernels.
Revised: We use seven datasets from the domains of bioinformatics and computational social science and compare against various GNNs and Graph Kernels.

Original: Our main GNN baselines are GIN
Revised: Our main GNN baselines are GIN

Original: Xu et al.
Revised: Xu et al.

Original: and PPGNs Maron et al.
Revised: and PPGNs Maron et al.

Original: (2019a).
Revised: (2019a).

Original: We follow the same evaluation protocol of Xu et al.
Revised: We follow the same evaluation protocol of Xu et al.

Original: (2019), performing 

Revised: (2019), performing

Original: -fold cross-validation and then reporting the performance at the epoch with the best average accuracy across the 10 folds.
Revised: -fold cross-validation and then reporting the performance at the epoch with the best average accuracy across the 10 folds.

Original: Table 1 lists all the methods evaluated with the split of
Revised: Table 1 lists all the methods evaluated with the split of

Original: Zhang et al.
Revised: Zhang et al.

Original: We select our model by tuning architecture and optimisation hyperparameters and substructure related parameters, that is: (i) k , (ii) motifs against graphlets.
Revised: We select our model by tuning architecture and optimisation hyperparameters and substructure related parameters, that is: (i) k , (ii) motifs against graphlets.

Original: Following domain evidence we choose the following substructure families: cycles for molecules, cliques for social networks.
Revised: Following domain evidence we choose the following substructure families: cycles for molecules, cliques for social networks.

Original: Best performing substructures both for GSN-e and GSN-v are reported.
Revised: Best performing substructures both for GSN-e and GSN-v are reported.

Original: As can be seen, our model obtains state-of-the-art performance in most of the datasets, with a considerable margin against the main GNN baselines in some cases.
Revised: As can be seen, our model obtains state-of-the-art performance in most of the datasets, with a considerable margin against the main GNN baselines in some cases

Original: , Second , Third best methods are highlighted. For GSN, the best performing structure is shown. ∗ Graph Kernel methods.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: ZINC
Revised: ZINC

Original: Molecular graphs: We evaluate GSN on the task of regressing the constrained solubility property of molecules from the ZINC database (Irwin et al., 2012; Kusner et al., 2017; G´omezBombarelli et al., 2018; Jin et al., 2018; Dwivedi et al., 2020).
Revised: Molecular graphs: We evaluate GSN on the task of regressing the constrained solubility property of molecules from the ZINC database (Irwin et al., 2012; Kusner et al., 2017; G ´ omezBombarelli et al., 2018; Jin et al., 2018; Dwivedi et al., 2020).

Original: Our main baselines are GIN (which we re-implement), as well as a stronger baseline (as in Eq.
Revised: Our main baselines are GIN (which we re-implement), as well as a stronger baseline (as in Eq.

Original: 3 and 4 without structural features (see Appendix C.1 for details) that can also take into account edge features (MPNN- EF ).
Revised: A.3 and 4 without structural features (see Appendix C.1 for details) that can also take into account edge features (MPNN- EF ).

Original: We then extend both baselines with structural features obtained with k -cycle counting (models denoted asGSN and
Revised: We then extend both baselines with structural features obtained with k -cycle counting (models denoted as

Original: GSN  -EF ) and report the result of the best performing substructure w.r.t.
Revised: GSN and GSN -EF ) and report the result of the best performing substructure w.r.t.

Original: the validation set.
Revised: the validation set.

Original: The data split into training, validation and test set is obtained from Dwivedi et al.
Revised: The data split into training, validation and test set is obtained from Dwivedi et al.

Original: The evaluation metric and the training loss is the Mean Absolute Error (MAE).
Revised: The evaluation metric and the training loss is the Mean Absolute Error (MAE).

Original: Table 2 shows that our model signiﬁcantly outperforms all the baselines.
Revised: Table 2 shows that our model signiﬁcantly outperforms all the baselines.

Original: In Figure 3, we compare the training and test error for different substructure families (cycles, paths and non-isomorphic trees – for each experiment we use all the substructures of size ≤ k in the family).
Revised: In Figure 4, we compare the training and test error for different substructure families (cycles, paths and non-isomorphic trees – for each experiment we use all the substructures of size ≤ k in the family).

Original: Additionally, we measure the “uniqueness” of the identiﬁers each substructure yields as follows: for each graph G in the dataset, we measure the number of unique node features u G (initial node features[Table] [Table] concatenated with node structural identiﬁers).
Revised: Additionally, we measure the “uniqueness” of the identiﬁers each substructure yields as follows: for each graph G in the dataset, we measure the number of unique node features u G (initial node features concatenated with node structural identiﬁers).

Original: Then, we sum them up over the entire training set and divide by the total number of nodes, yielding the disambiguation score δ =
Revised: Then, we sum them up over the entire training set and divide by the total number of nodes, yielding the disambiguation score δ =

Original: (cid:80) G u G [Table] G | V G | , shown as horizontal bars in Figure 3.
Revised: (cid:80) G u G (cid:80) G | V G | , shown as horizontal bars in Figure 4.

Original: A ﬁrst thing to notice is that the training error is tightly related to the disambiguation score.
Revised: A ﬁrst thing to notice is that the training error is tightly related to the disambiguation score.

Original: As identiﬁers become more discriminative, the model gains expressive power.
Revised: As identiﬁers become more discriminative, the model gains expressive power.

Original: On the other hand, the test error is not guaranteed to decrease when the identiﬁers become more discriminative.
Revised: On the other hand, the test error is not guaranteed to decrease when the identiﬁers become more discriminative.

Original: For example, although cycles have smaller disambiguation scores, they manage to generalise much better than the other substructures, the performance of which is similar to the baseline architecture.
Revised: For example, although cycles have smaller disambiguation scores, they manage to generalise much better than the other substructures, the performance of which is similar to the baseline architecture.

Original: This is also observed when comparing against Sato et al.
Revised: This is also observed when comparing against Sato et al.

Original: (2020) ( -r methods in
Revised: (2020) ( -r methods in

Original: Table 2), where, akin to unique identiﬁers, random features are used to strengthen the expressivity of
Revised: Table 2), where, akin to unique identiﬁers, random features are used to strengthen the expressivity of

Original: GNN architectures.
Revised: GNN architectures.

Original: This approach also fails to improve the baseline architectures in terms of mean test error.
Revised: This approach also fails to improve the baseline architectures in terms of mean test error.

Original: This validates our intuition that unique identiﬁers can be hard to generalise when chosen arbitrarily and motivates once more the importance of choosing the identiﬁers not only based on their discriminative power, but also in a way that allows incorporating the appropriate inductive biases.
Revised: This validates our intuition that unique identiﬁers can be hard to generalise when chosen arbitrarily and motivates once more the importance of choosing the identiﬁers not only based on their discriminative power, but also in a way that allows incorporating the appropriate inductive biases.

Original: Finally, we observe a substantial jump in performance when using GSN with cycles of size k ≥ 6 .
Revised: Finally, we observe a substantial jump in performance when using GSN with cycles of size k ≥ 6 .

Original: This is not surprising, as cyclical patterns of such sizes (e.g. aromatic rings) are very common in organic molecules.
Revised: This is not surprising, as cyclical patterns of such sizes (e.g. aromatic rings) are very common in organic molecules.

Original:  OGB-MOLHIV: We use ogbg-molhiv from the Open Graph Benchmark (OGB)
Revised: [Table] [Table] OGB-MOLHIV: We use ogbg-molhiv from the Open Graph Benchmark (OGB)

Original: Hu et al.
Revised: Hu et al.

Original: (2020) as a graph-level binary classiﬁcation task, where the aim is to predict if a molecule inhibits HIV replication or not.
Revised: (2020) as a graph-level binary classiﬁcation task, where the aim is to predict if a molecule inhibits HIV replication or not.

Original: The baseline architecture provided by the authors is a variation of GIN that allows for edge features and is extended with a virtual node , GIN-VN , or with additional node/edge features, GIN-AF , or both, GIN-VN-AF (more information in the supplementary material).
Revised: The baseline architecture provided by the authors is a variation of GIN that allows for edge features and is extended with a virtual node , GIN-VN , or with additional node/edge features, GIN-AF , or both, GIN-VN-AF (more information in the supplementary material).

Original: Similarly to the experiment on ZINC, we extend the baseline settings with cyclical substructure features by treating them in a similar way as node and edge features ( GSN-VN, GSN-AF, GSN-VN-AF ).
Revised: Similarly to the experiment on ZINC, we extend the baseline settings with cyclical substructure features by treating them in a similar way as node and edge features ( GSN-VN, GSN-AF, GSN-VN-AF ).

Original: Using the evaluator provided by the authors, we report the ROC-AUC metric at the epoch with the best validation performance (substructures are also chosen based on the validation set).
Revised: Using the evaluator provided by the authors, we report the ROC-AUC metric at the epoch with the best validation performance (substructures are also chosen based on the validation set).

Original: As shown in
Revised: As shown in

Original: Table 3, considerable improvement in the performance of the model in all splits is obtained, thus demonstrating strong generalisation capabilities.
Revised: Table 3, considerable improvement in the performance of the model in all splits is obtained, thus demonstrating strong generalisation capabilities.

Original: Tests with additional chemical features such as formal charge, hybridisation, etc.
Revised: Tests with additional chemical features such as formal charge, hybridisation, etc.

Original: (‘ -AF ’ setting in Table 3) show that structure provides important complementary information.
Revised: (‘ -AF ’ setting in Table 3) show that structure provides important complementary information.

Revised: Structural Features & Message Passing: We perform an ablation study on the abilities of the structural features to predict the task at the hand, when given as input to a graph-agnostic network.

Revised: In particular, we compare our best performing GSN with a DeepSets model (Zaheer et al., 2017) that treats the input features and the structural identiﬁers as a set.

Revised: For fairness of evaluation the same hyperparameter search is performed for both models (see Appendix C.5).

Revised: Interestingly, as we show in Table 4, our baseline attains particularly strong performance across a variety of datasets and often outperforms other traditional message passing baselines.

Original:   C ONCLUSION
Revised: This demonstrates the beneﬁts of these additional features and motivates their introduction in GNNs, which are unable to compute them. As expected, we observe that applying message passing on top of these features, brings performance improvements in the vast majority of the cases, sometimes considerably, as in the ZINC dataset. [Table]  C ONCLUSION

Original: In this paper, we propose a novel way to design structure-aware graph neural networks.
Revised: In this paper [Table] we propose a novel way to design structure-aware graph neural networks.

Original: Motivated by the limitations of traditional GNNs to capture important topological properties of the graph, we formulate a message passing scheme enhanced with structural features that are extracted by counting the appearances of prominent substructures, as domain knowledge suggests.
Revised: Motivated by the limitations of traditional GNNs to capture important topological properties of the graph, we formulate a message passing scheme enhanced with structural features that are extracted by counting the appearances of prominent substructures [Table] as domain knowledge suggests.

Original: We show both theoretically and empirically that our construction leads to improved expressive power and attains state-of-the-art performance in real-world scenarios. 
Revised: We show both theoretically and empirically that our construction leads to improved expressive power and attains state-of-the-art performance in real-world scenarios. In future work

Original: In future work, we will further explore the expressivity of GSNs as an alternative to the k -WL tests, as well as their generalisation capabilities.
Revised: [Table] we will further explore the expressivity of GSNs as an alternative to the k -WL tests [Table] as well as their generalisation capabilities.

Original: Another important direction is to infer prominent substructures directly from the data and explore the ability of graph neural networks to compose substructures.
Revised: Another important direction is to infer prominent substructures directly from the data and explore the ability of graph neural networks to compose substructures [Table]

Original: A  P ROOF OF P ROPOSITION 3.
Revised: A D EFERRED P ROOFS A. GSN IS PERMUTATION EQUIVARIANT

Original: Proof.
Revised: Proof.

Original: GSN is at least as powerful as MPNNs and the 1-WL test: It is easy to see that GSN model class contains MPNNs and is thus at least as expressive.
Revised: Let A ∈ R n × n the adjacency matrix of the graph, H ∈ R n × d Vin the input vertex features, E ∈ R n × n × d Ein the input edge features and E i ∈ R n × n the edge features at dimension i .

Revised: Let

Revised: S V ( A ) ∈ N n × d V , S E ( A ) ∈ N n × n × d E the functions generating the vertex and edge structural identiﬁers respectively.

Revised: It is obvious that subgraph isomorphism is invariant to the ordering of the vertices, i.e. we will always obtain the same matching between subgraphs G S and graphs

Revised: H in the subgraph collection.

Revised: Thus, each vertex v (edge ( v, u ) ) in the graph will be assigned the same structural identiﬁers x V ( v ) ( x E ( v, u ) ) regarless of the vertex ordering.

Revised: Thus S V and

Revised: S E are permutation equivariant, i.e. for any permutation matrix P ∈ { 0 , 1 } n × n it holds that

Revised: S V ( PAP T ) =

Revised: P S V ( A ) , S E ( PAP T ) =

Revised: P [ S E 1 ( A ); . .

Revised: ; S Ed E ( A )) P T , where the permutation is applied at each slice S E 1 ( A ) ∈ N n × n of the tensor S E ( A ) .

Revised: Let f ( A , H , E ) ∈ R n × d out a GSN layer.

Revised: We will show that f is permutation equivariant.

Revised: We need to show that if Y =

Revised: f

Revised: ( A , X , E ) the output of the GSN-layer, then PY = f

Revised: ( PAP T , PX , P [ E 1 ; . . .

Revised: ; E d E ] P T ) for any permutation matrix P .

Revised: It is easy to see that GSNv can can be expressed as a traditional MPNN g ( A , H , E ) ∈ R n × d out (similar to equations 3 and 4 of the main paper) by replacing the vertex features H with the concatenation of the input vertex features and the vertex structural identiﬁers, i.e. Y =

Revised: f ( A , H , E ) =

Revised: g ( A , [ H ; S V ( A )] , E ) .

Revised: Thus, f ( PAP T , PH , PE 1 P T , . . .

Revised: , PE d E P T ) =

Revised: g ( PAP T , [ PH ; S V ( PAP T )] , P [ E 1 , . . .

Revised: ; E d E ] P T ) =

Revised: g ( PAP T , P [ H 0 ; S V ( A )] , P [ E 1 ; . . .

Revised: ; E d E ] P T ) =

Revised: P g ( A , [ H 0 ; S V ( A )] , E ) = PY where in the last step we used the permutation equivariant property of MPNNs.

Revised: Similarly, we can show that a GSN-e layer is permutation equivariant, since it can be expressed as a traditional MPNN layer by replacing the edge features with the concatenation of the original edge features and the edge structural identiﬁers f

Revised: ( A , H , E ) =

Revised: g ( A , H , [ E ; S E ( A )]) .

Revised: Overall, a GSN network is permutation equivariant as composition of permutation equivariant functions, or permutation invariant when composed with a permutation invariant layer at the end, i.e. the READOUT function.

Revised: A. P ROOF OF P ROPOSITION 3.

Revised: Proof.

Revised: GSN is at least as powerful as MPNNs and the 1-WL test: It is easy to see that GSN model class contains MPNNs and is thus at least as expressive.

Original: To show that it is at least as expressive as the 1-WL test, one can repurpose the proof of Theorem 3 in Xu et al.
Revised: To show that it is at least as expressive as the 1-WL test, one can repurpose the proof of Theorem 3 in Xu et al.

Original: and demand the injectivity of the update function (w.r.t.
Revised: and demand the injectivity of the update function (w.r.t.

Original: both the hidden state h t ( v ) and the message aggregation
Revised: both the hidden state h t ( v ) and the message aggregation

Original: MSG t +1 ( v ) ), and the injectivity of the message aggregation w.r.t.
Revised: MSG t +1 ( v ) ), and the injectivity of the message aggregation w.r.t.

Original: the multiset of the hidden states of the neighbours (cid:72) h t ( u ) (cid:73) u ∈N ( v ) .
Revised: the multiset of the hidden states of the neighbours (cid:72) h t ( u ) (cid:73) u ∈N ( v ) .

Original: It sufﬁces then to show that if injectivity is preserved then GSNs are at least as powerful as the 1-WL.
Revised: It sufﬁces then to show that if injectivity is preserved then GSNs are at least as powerful as the 1-WL.

Original: We will show the above statement for node-labelled graphs, since traditionally the 1-WL test does not take into account edge labels.
Revised: We will show the above statement for node-labelled graphs, since traditionally the 1-WL test does not take into account edge labels.

Original: 2 We can rephrase the statement as follows: If GSN deems two graphs G 1 , G 2 as isomorphic, then also 1-WL deems them isomorphic .
Revised: 2 We can rephrase the statement as follows: If GSN deems two graphs G 1 , G 2 as isomorphic, then also 1-WL deems them isomorphic .

Original: Given that the graph-level representation is extracted by a readout function that receives the multiset of the node colours in its input (i.e. the graph-level representation is the node colour histogram at some iteration t ), then it sufﬁces to show that if for the two graphs the multiset of the node colours that GSN infers is the same, then also 1-WL will infer the same multiset for the two graphs.
Revised: Given that the graph-level representation is extracted by a readout function that receives the multiset of the node colours in its input (i.e. the graph-level representation is the node colour histogram at some iteration t ), then it sufﬁces to show that if for the two graphs the multiset of the node colours that GSN infers is the same, then also 1-WL will infer the same multiset for the two graphs.

Original: Consider the case where the two multisets that GSN extracts are the same: i.e.
Revised: Consider the case where the two multisets that GSN extracts are the same: i.e.

Original: [Equation]

Original:  u ) [Equation] u [Equation]
Revised: (cid:72) h t ( v ) (cid:73) v ∈V G 1 = (cid:72) h t ( u ) (cid:73) u ∈V G 2 .

Original: Then both multisets contain the same distinct colour/hidden rep- resentations with the exact same multiplicity.
Revised: Then both multisets contain the same distinct colour/hidden rep- resentations with the exact same multiplicity.

Original: Thus, it further sufﬁces to show that if two nodes v, u (that may belong to the same or to different graphs) have the same GSN hidden representations h t ( v )
Revised: Thus, it further sufﬁces to show that if two nodes v, u (that may belong to the same or to different graphs) have the same GSN hidden representations h t ( v )

Original: =
Revised: =

Original: h t ( u ) at any iteration t , then they will also have the same colours c t ( v ) =
Revised: h t ( u ) at any iteration t , then they will also have the same colours c t ( v ) =

Original: c t ( u ) , extracted by 1-WL.
Revised: c t ( u ) , extracted by 1-WL.

Original: Intuitively, this means that GSN creates a partition of the nodes of each graph that is at least as ﬁne-grained as the one created by 1-WL.
Revised: Intuitively, this means that GSN creates a partition of the nodes of each graph that is at least as ﬁne-grained as the one created by 1-WL.

Original: We prove by induction (similarly to Xu et al.
Revised: We prove by induction (similarly to Xu et al.

Original: that
Revised: that

Original: GSN model class contains a model where this holds (w.l.o.g.
Revised: GSN model class contains a model where this holds (w.l.o.g.

Original: we show that for GSN-v; same proof applies to GSN-e).
Revised: we show that for GSN-v; same proof applies to GSN-e).

Original: For t = 

Revised: For t =

Original: the statement holds since the initial node features are the same for both GSN and 1-WL, i.e. h 0 ( v ) =
Revised: the statement holds since the initial node features are the same for both GSN and 1-WL, i.e. h 0 ( v ) =

Original: c 0 ( v ) , ∀ v ∈ V G 1 ∪ V G 2 .
Revised: c 0 ( v ) , ∀ v ∈ V G 1 ∪ V G 2 .

Original: Suppose the statement holds for t − 1 , i.e. h t − 1 ( v ) =
Revised: Suppose the statement holds for t − 1 , i.e. h t − 1 ( v ) =

Original: h t − 1 ( u ) ⇒ c
Revised: h t − 1 ( u ) ⇒ c

Original: t − 1 ( v ) =
Revised: t − 1 ( v ) =

Original: c t − 1 ( u ) .
Revised: c t − 1 ( u ) .

Original: Then we show that it also holds for t .
Revised: Then we show that it also holds for t .

Original: Every node hidden representation at step t is updated as follows:
Revised: Every node hidden representation at step t is updated as follows:

Original: h t ( v ) = UP t (cid:0) h t − 1 ( v ) , MSG t ( v ) (cid:1) .
Revised: h t ( v ) = UP t (cid:0) h t − 1 ( v ) , MSG t ( v ) (cid:1) .

Original: Assuming that the update function UP t is injective, we have the following: if h t ( v ) =
Revised: Assuming that the update function UP t is injective, we have the following: if h t ( v ) =

Original: h t ( u ) , then:
Revised: h t ( u ) , then:

Original: h t − 1 ( v ) =
Revised: h t − 1 ( v ) =

Original: h t − 1 ( u ) , which from the induction hypothesis implies that c t − 1 ( v ) =
Revised: h t − 1 ( u ) , which from the induction hypothesis implies that c t − 1 ( v ) =

Original: c t − 1 ( u ) .
Revised: c t − 1 ( u ) .

Original: MSG t ( v ) =
Revised: MSG t ( v ) =

Original: MSG t ( u ) , where the message function is deﬁned as in Eq.
Revised: MSG t ( u ) , where the message function is deﬁned as in Eq.

Original: 3 of the main paper: MSG t ( v ) = (cid:80) w ∈N ( v ) M t (cid:0) h t − 1 ( v ) , h t − 1 ( w ) , x V ( v ) , x V ( w ) (cid:1) .
Revised: A.3 of the main paper: MSG t ( v ) = (cid:80) w ∈N ( v ) M t (cid:0) h t − 1 ( v ) , h t − 1 ( w ) , x V ( v ) , x V ( w ) (cid:1) .

Original: Additionally here we require
Revised: Additionally here we require

Original: MSG t to be injective w.r.t.
Revised: MSG t to be injective w.r.t.

Original: the multiset of the hidden representations of the neighbours.
Revised: the multiset of the hidden representations of the neighbours.

Original: In fact, using Lemma 5 from Xu et al.
Revised: In fact, using Lemma 5 from Xu et al.

Original: we know that there always exists a function M t , such that
Revised: we know that there always exists a function M t , such that

Original: MSG t ( v ) is unique for each multiset (cid:72) (cid:0) h t − 1 ( v ) , h t − 1 ( w ) , x V ( v ) , x V ( w ) (cid:1) (cid:73) w ∈N v , assuming that the domain from where the elements of the multiset originate is countable.
Revised: MSG t ( v ) is unique for each multiset (cid:72) (cid:0) h t − 1 ( v ) , h t − 1 ( w ) , x V ( v ) , x V ( w ) (cid:1) (cid:73) w ∈N v , assuming that the domain from where the elements of the multiset originate is countable.

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: From the induction hypothesis we know that h t − 1 ( w ) =
Revised: From the induction hypothesis we know that h t − 1 ( w ) =

Original: h t − 1 ( z ) implies that c t − 1 ( w ) =
Revised: h t − 1 ( z ) implies that c t − 1 ( w ) =

Original: c t − 1 ( z ) for any w ∈ N v , z ∈ N u , thus (cid:72) c t − 1 ( w ) (cid:73) w ∈N v = (cid:72) c t − 1 ( z ) (cid:73) z ∈N u .
Revised: c t − 1 ( z ) for any w ∈ N v , z ∈ N u , thus (cid:72) c t − 1 ( w ) (cid:73) w ∈N v = (cid:72) c t − 1 ( z ) (cid:73) z ∈N u .

Original: Concluding, given the update rule of 1-WL: c t ( v ) =
Revised: Concluding, given the update rule of 1-WL: c t ( v ) =

Original: HASH (cid:16) c t − 1 ( v ) , (cid:72) c t − 1 ( w ) (cid:73) w ∈N v (cid:17) , it holds that c t ( v ) =
Revised: HASH (cid:16) c t − 1 ( v ) , (cid:72) c t − 1 ( w ) (cid:73) w ∈N v (cid:17) , it holds that c t ( v ) =

Original: c t ( u ) .
Revised: c t ( u ) .

Original: When are GSNs strictly more powerful than the 1-WL test?
Revised: When are GSNs strictly more powerful than the 1-WL test?

Original: Arvind et al.
Revised: Arvind et al.

Original: (2019) showed that 1-WL, and consequently MPNNs, can count only forests of stars .
Revised: (2019) showed that 1-WL, and consequently MPNNs, can count only forests of stars .

Original: Thus, if the subgraphs are required to be connected, then they can only be star graphs of any size (note that this contains single nodes and single edges).
Revised: Thus, if the subgraphs are required to be connected, then they can only be star graphs of any size (note that this contains single nodes and single edges).

Original: In addition, Chen et al.
Revised: In addition, Chen et al.

Original: (2020) showed that 1-WL, and consequently MPNNs, cannot count any connected induced subgraph with 3 or more nodes, i.e. any connected subgraph apart from single nodes and single edges.
Revised: (2020) showed that 1-WL, and consequently MPNNs, cannot count any connected induced subgraph with 3 or more nodes, i.e. any connected subgraph apart from single nodes and single edges.

Original: Given the ﬁrst part of the proposition, in order to show that GSNs are strictly more expressive than the 1-WL test, it sufﬁces to show that GSN can distinguish a pair of graphs that 1-WL test deem isomorphic.
Revised: Given the ﬁrst part of the proposition, in order to show that GSNs are strictly more expressive than the 1-WL test, it sufﬁces to show that GSN can distinguish a pair of graphs that 1-WL test deem isomorphic.

Original: If H is a substructure that 1-WL cannot learn to count, i.e. the ones mentioned above, then there is at least one pair of graphs with different number of counts of H , that 1-WL deems isomorphic.
Revised: If H is a substructure that 1-WL cannot learn to count, i.e. the ones mentioned above, then there is at least one pair of graphs with different number of counts of H , that 1-WL deems isomorphic.

Original: Thus, by assigning counting features to the nodes/edges of the two graphs based on appearances of H , a GSN can obtain different representations for G 1 and G 2 by summing up the features.
Revised: Thus, by assigning counting features to the nodes/edges of the two graphs based on appearances of H , a GSN can obtain different representations for G 1 and G 2 by summing up the features.

Original: Hence, G 1 , G 2 are deemed non-isomorphic.
Revised: Hence, G 1 , G 2 are deemed non-isomorphic.

Original: An example is depicted in Figure 2 (left), where the two non-isomorphic graphs are distinguishable by GSN via e.g. cycle counting, but not by 1-WL.  
Revised: An example is depicted in Figure 2 (left), where the two non-isomorphic graphs are distinguishable by GSN via e.g. cycle counting, but not by 1-WL. A. P ROOF OF P ROPOSITION 3.

Revised: Without loss of generality we will show Proposition 3.2 for the case of a single substructure H .

Revised: Moreover, in the context of this proof, for reasons that will be made clear below, we will use directed edge structural identiﬁers, i.e. edge orbits are deﬁned on directed edges, thus potentially different edge identiﬁers are assigned to different directions of the same edge.

Revised: Note that this is not the case for substructures that have a single vertex orbit, such as the ones that we use in our experiments - cliques and cycles.

Revised: This is because the single vertex orbit induces a single edge orbit as well which is the same both for the directed and the undirected case.

Original: B C OMPARISON WITH HIGHER - ORDER W EISFEILER -L EHMAN TESTS B. T HE WL HIERARCHY
Revised: Proof. In order to show that GSN-e can express GSN-v, we will ﬁrst prove that the vertex identiﬁer of each vertex v can be inferred by the edge identiﬁers of its incident edges { ( v, u ) : u ∈ N ( v ) } . We ﬁrst introduce the following deﬁnitions to simplify notation: Recall the deﬁnition of a vertex orbit Orb V ( v ) = { u ∈ V H : ∃ g ∈ Aut( H ) s .

Revised: t .

Revised: g ( w ) = v } .

Revised: For an orbit Orb V ( v ) , denote the orbit neighbourhood N (cid:0)

Revised: Orb V ( v ) (cid:1) as the multiset of the orbits of the neighbours of v :

Revised: N (cid:0) Orb V ( v ) (cid:1) = (cid:72)

Revised: Orb V ( u ) : u ∈ N ( v ) (cid:73) .

Revised: Note that the orbit neighbourhood is the same for any vertex w ∈ Orb V ( v ) and that the neighbourhood is a multiset, since different neighbours might belong to the same orbit.

Revised: Denote the orbit degree deg (Orb V ( v )) as the degree of any vertex in

Revised: Orb V ( v ) as deg ( v ) =

Revised: deg (Orb V ( v )) =

Revised: |N (cid:0) Orb V ( v ) (cid:1) | .

Revised: We will be indexing different vertex orbits as follows:

Revised: O Vi =

Revised: Orb V ( v ) .

Revised: Regarding the edge orbits, recall that each edge orbit is induced by the orbits of the vertex automorphism group, i.e. Orb E ( v, u )

Revised: = { ( w, z ) ∈

Revised: E H : ∃ g ∈ Aut( H ) s .

Revised: t .

Revised: g ( w ) = v and g ( z )

Revised: = u } , or al- ternatively Orb E ( v, u ) = {

Revised: ( w, z ) ∈

Revised: E H :

Revised: Orb V ( w ) =

Revised: Orb V ( v ) and Orb V ( z ) =

Revised: Orb V ( u ) } .

Revised: Observe here, that we deﬁne edge orbits for directed edges.

Revised: Let’s assume that Orb V ( v ) =

Revised: O Vi ,

Revised: Orb V ( u ) =

Revised: O Vj and Orb E ( v, u ) =

Revised: O Eij .

Revised: For ease of notation we will denote O Eij = (cid:0)

Revised: O Vi , O Vj (cid:1) .

Revised: Recall that vertex structural identiﬁers are deﬁned as follows (we drop the subscript H for ease of notation):

Revised: [Equation]

Revised: and edge structural identiﬁers for directed edges:

Revised: [Equation]

Revised: Let as assume that there exists only one matched subgraph G S ∼ = H and the bijection between V G S and H is denoted as f .

Revised: Then, concerning the vertex orbit O Vi , one of the following holds:

Revised: • v (cid:54)∈ V G S .

Revised: Then x Vi ( v ) =

Revised: and x Eij ( v, u ) = 0 , ∀ u ∈ N ( v ) , • v ∈ V G S and Orb( f ( v ))

Revised: (cid:54) =

Revised: O Vi .

Revised: Then, x Vi ( v ) =

Revised: and x Eij ( v, u ) = 0 , ∀ u ∈ N ( v ) , otherwise the orbit of f ( v ) would be O Vi .

Revised: Note that here the directionality of the edge is important, otherwise an ambiguity would be created and x Eij ( v, u ) could have been equal to 1 if the orbit of f ( u ) is O Vi .

Revised: • v ∈ V G S and Orb( f ( v ))

Revised: =

Revised: O Vi .

Revised: Then, x Vi ( v ) = 1 and since f ( v ) has exactly deg ( O Vi ) neighbours in H , then there exist exactly deg ( O Vi ) edges incident to v that belong to G S as well and the set of their edge orbits is exactly N (cid:0)

Revised: O Vi (cid:1) .

Revised: In other words, (cid:80)

Revised: [Equation]

Revised: Thus, by induction, we can easily see that for m matched subgraphs G S ∼ = H where v ∈ V G S and

Revised: Orb( f ( v )) =

Revised: O Vi , it holds that x Vi ( v ) = m

Revised: and (cid:80) u ∈N ( v ) (cid:80) j

Revised: : O Vj ∈N ( O

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: deg ( O

Revised: [Equation]

Revised: Then it follows that:

Revised: [Equation]

Revised: This expression shows that vertex identiﬁers can be inferred by edge identiﬁers.

Revised: The opposite does not necessary hold, since if we know the vertex identiﬁers x Vi ( v ) , x Vj ( u ) of an edge ( v, u ) we might not be able to infer x Eij ( u, v ) .

Revised: That is because we do not know how many of the subgraphs in x Vi ( v ) and x V j ( u ) are common, i.e. they contain both nodes.

Revised: The rest of the proof is straightforward: we will assume a GSN-v using substructure counts of the graph H , with L layers and width w deﬁned as in the main paper, i.e. at each layer vertex hidden states are updated by h t +1 ( v ) =

Revised: UP t +1 (cid:0) h t ( v ) , MSG t +1 ( v ) (cid:1) , where MSG t +1 ( v ) =

Revised: (cid:80) u ∈N ( v ) M t +1 (cid:0) h t ( v ) , h t ( u ) , x V ( v ) , x V ( u ) , e ( { u, v } ) (cid:1) .

Revised: Then, there exists a GSN-e with L + 1 layers, where the ﬁrst layer has width d in V + d V and implements the following function:

Revised: UP t +1 (cid:0) h t ( v ) , MSG t +1 ( v ) (cid:1) =

Revised: [ h t ( v ); MSG t +1 ( v )] , where:

Revised: [Equation]

Revised: Note that since M is a universal approximator, then there exist parameters of M with which the above function can be simulated.

Revised: The next L layers of GSN-e can implement a traditional MPNN where now the input vertex features are [ h t ( v ); x V ( v )] (which is exactly the formulation of GSN-v) and this concludes the proof.

Revised: B C OMPARISON WITH HIGHER - ORDER W EISFEILER -L EHMAN TESTS B. T HE WL HIERARCHY

Original: Following the terminology introduced in Maron et al.
Revised: Following the terminology introduced in Maron et al.

Original: (2019a)
Revised: (2019a)

Original: , we describe the so-called Folklore
Revised: , we describe the so-called Folklore

Original: WL family ( k -FWL) .
Revised: WL family ( k -FWL) .

Original: Note that, in the majority of papers on GNN expressivity (Morris et al., 2019;
Revised: Note that, in the majority of papers on GNN expressivity (Morris et al., 2019;

Original: Maron et al., 2019a; Chen et al., 2020) another family of WL tests is discussed, under the terminology k -WL with expressive power equal to ( k − 1) -FWL.
Revised: Maron et al., 2019a; Chen et al., 2020) another family of WL tests is discussed, under the terminology k -WL with expressive power equal to ( k − 1) -FWL.

Original: In contrast, in most graph theory papers on graph isomorphism (Cai et al., 1992; F¨urer, 2017; Arvind et al., 2019) the k -WL term is used to describe the algorithms referred to as k -FWL in GNN papers.
Revised: In contrast, in most graph theory papers on graph isomorphism (Cai et al., 1992; F ¨ urer, 2017; Arvind et al., 2019) the k -WL term is used to describe the algorithms referred to as k -FWL in GNN papers.

Original: Here, we follow the k -FWL convention to align with the work mostly related to ours.
Revised: Here, we follow the k -FWL convention to align with the work mostly related to ours.

Original: The k -FWL operates on k -tuples of nodes v = ( v 1 , v 2 , . .
Revised: The k -FWL operates on k -tuples of nodes v = ( v 1 , v 2 , . .

Original: , v k ) to which an initial colour c 0 ( v ) is assigned based on their isomorphism types (see section B.3), which can loosely be thought of as a generalisation of isomorphism that also preserves the ordering of the nodes in the tuple.
Revised: , v k ) to which an initial colour c 0 ( v ) is assigned based on their isomorphism types (see section B.3), which can loosely be thought of as a generalisation of isomorphism that also preserves the ordering of the nodes in the tuple.

Original: Then, at each iteration the colour is reﬁned as follows:
Revised: Then, at each iteration the colour is reﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where v u,j = ( v 1 , v 2 , . .
Revised: where v u,j = ( v 1 , v 2 , . .

Original: . , v j − 1 , u, v j +1 , . . .
Revised: . , v j − 1 , u, v j +1 , . . .

Original: , v k ) .
Revised: , v k ) .

Original: The multiset (cid:72) (cid:0) c t ( v u, 1 ) , c t ( v u, 2 ) , . . . ,
Revised: The multiset (cid:72) (cid:0) c t ( v u, 1 ) , c t ( v u, 2 ) , . . . ,

Original: c t ( v u,k ) (cid:1) (cid:73) u ∈V can be perceived as a form of generalised neighbourhood.
Revised: c t ( v u,k ) (cid:1) (cid:73) u ∈V can be perceived as a form of generalised neighbourhood.

Original: Note here, that information is saved in all possible tuples in the graph, thus each k -tuple receives information from the entire graph , contrary to the local nature of the 1-WL test.
Revised: Note here, that information is saved in all possible tuples in the graph, thus each k -tuple receives information from the entire graph , contrary to the local nature of the 1-WL test.

Original: B. W HEN CAN GSN BE STRONGER THAN 2-FWL?
Revised: B. W HEN CAN GSN BE STRONGER THAN 2-FWL?

Original: In order to give further intuition in the limitations of the 2-FWL, in the following section we describe a speciﬁc family of graphs which 2-FWL is known to be unable to disambiguate.
Revised: In order to give further intuition in the limitations of the 2-FWL, in the following section we describe a speciﬁc family of graphs which 2-FWL is known to be unable to disambiguate.

Original: Deﬁnition B.1 (Strongly regular graph) .
Revised: Deﬁnition B.1 (Strongly regular graph) .

Original: A SR( n , d , λ , µ )-graph is a regular graph with n nodes and degree d , where every two adjacent vertices have always λ mutual neighbours, while every two non-adjacent vertices have always µ mutual neighbours.
Revised: A SR( n , d , λ , µ )-graph is a regular graph with n nodes and degree d , where every two adjacent vertices have always λ mutual neighbours, while every two non-adjacent vertices have always µ mutual neighbours.

Original: 2-FWL will always decide that a pair of SR graphs with the same parameters n, d, λ, µ are isomorphic, since all 2-tuples of the same isomorphism type have the exact same generalised neighbourhood, leading 2-FWL to converge to its solution at initialisation.
Revised: 2-FWL will always decide that a pair of SR graphs with the same parameters n, d, λ, µ are isomorphic, since all 2-tuples of the same isomorphism type have the exact same generalised neighbourhood, leading 2-FWL to converge to its solution at initialisation.

Original: On the other hand, as we show in section 5 of the main paper, GSN can tell strongly regular graphs apart by using small-sized substructures.
Revised: On the other hand, as we show in section 5 of the main paper, GSN can tell strongly regular graphs apart by using small-sized substructures.

Original: Although it is not clear if there exists a certain substructure collection that results in GSNs that align with the WL hierarchy, we stress that this is not a necessary condition in order to design more powerful GNNs.
Revised: Although it is not clear if there exists a certain substructure collection that results in GSNs that align with the WL hierarchy, we stress that this is not a necessary condition in order to design more powerful GNNs.

Original: In particular, the advantages offered by k-WL might not be able to outweigh the disadvantage of the larger computational complexity introduced.
Revised: In particular, the advantages offered by k-WL might not be able to outweigh the disadvantage of the larger computational complexity introduced.

Original: For example, a 2-FWL equivalent
Revised: For example, a 2-FWL equivalent

Original: GNN will still fail to count 4-cliques (a frequent pattern in social and biological networks) orcycles (a common ring structure in organic molecules).
Revised: GNN will still fail to count 4-cliques (a frequent pattern in social and biological networks) orcycles (a common ring structure in organic molecules).

Original: Therefore, we suggest that it might be more appropriate to design powerful GNNs based on the distinct characteristics of the task at hand.
Revised: Therefore, we suggest that it might be more appropriate to design powerful GNNs based on the distinct characteristics of the task at hand.

Original: B. 
Revised: B. W HY DOES 2-FWL FAIL ON STRONGLY REGULAR GRAPHS ?

Original: W HY DOES 2-FWL FAIL ON STRONGLY REGULAR GRAPHS ?

Original: Proof.
Revised: Proof.

Original: We ﬁrst rigorously describe what an isomorphism type is.
Revised: We ﬁrst rigorously describe what an isomorphism type is.

Original: Two k-tuples v a = { v a 1 , v a 2 , . .
Revised: Two k-tuples v a = { v a 1 , v a 2 , . .

Original: . , v ak } , v b = { v b 1 , v b 2 , . . .
Revised: . , v ak } , v b = { v b 1 , v b 2 , . . .

Original: , v bk } will have the same isomorphism type iff:
Revised: , v bk } will have the same isomorphism type iff:

Original: • ∀ i, j ∈ { 0 , 1 , . . .
Revised: • ∀ i, j ∈ { 0 , 1 , . . .

Original: , k } , v ai = v aj ⇔ v bi = v bj • ∀ 
Revised: , k } , [Equation] • ∀ i, j ∈ { 0 , 1 , . . . , k } ,

Original: i, j ∈ { 0 , 1 , . . . , k } , v ai ∼ v aj ⇔ v bi ∼ v bj , where ∼ means that the vertices are adjacent.
Revised: [Equation] ∼ [Equation] , where ∼ means that the vertices are adjacent.

Original: Note that this is a stronger condition than isomorphism, since the mapping between the vertices of the two tuples needs to preserve order.
Revised: Note that this is a stronger condition than isomorphism, since the mapping between the vertices of the two tuples needs to preserve order.

Original: In case the graph is employed with edge and vertex features, they need to be preserved as well (see Chen et al.
Revised: In case the graph is employed with edge and vertex features, they need to be preserved as well (see Chen et al.

Original: for the extended case).
Revised: for the extended case).

Original: For the 2-FWL test, when working with simple undirected graphs without self-loops, we have the following 2-tuple isomorphism types:
Revised: For the 2-FWL test, when working with simple undirected graphs without self-loops, we have the following 2-tuple isomorphism types:

Original: • v = { v 1 , v 1 } : vertex type .
Revised: • v = { v 1 , v 1 } : vertex type .

Original: Mapped to the colour c (0)
Revised: Mapped to the colour c (0)

Original: =
Revised: =

Original: c α • v = { v 1 , v 2 } and v 1 (cid:54)∼ v 2 : non-edge type .
Revised: c α • v = { v 1 , v 2 } and v 1 (cid:54)∼ v 2 : non-edge type .

Original: Mapped to the colour c (0)
Revised: Mapped to the colour c (0)

Original: =
Revised: =

Original: c β • v = { v 1 , v 2 } and v 1 ∼ v 2 : edge type .
Revised: c β • v = { v 1 , v 2 } and v 1 ∼ v 2 : edge type .

Original: Mapped to the colour c (0)
Revised: Mapped to the colour c (0)

Original: =
Revised: =

Original: c γ
Revised: c γ

Original: For each 2-tuple v = { v 1 , v 2 } , a generalised “neighbour” is the following tuple: ( v u, 1 , v u, 2 ) = (cid:0) ( u, v 2 ) , ( v 1 , u ) (cid:1) , where u is an arbitrary vertex in the graph.
Revised: For each 2-tuple v = { v 1 , v 2 } , a generalised “neighbour” is the following tuple: ( v u, 1 , v u, 2 ) = (cid:0) ( u, v 2 ) , ( v 1 , u ) (cid:1) , where u is an arbitrary vertex in the graph.

Original: Now, let us consider a strongly regular graph SR( n , d , λ , µ ).
Revised: Now, let us consider a strongly regular graph SR( n , d , λ , µ ).

Original: We have the following cases:
Revised: We have the following cases:

Original: • generalised neighbour of a vertex type tuple: ( v u, 1 , v u, 2 ) = (cid:0) ( u, v 1 ) , ( v 1 , u ) (cid:1) 
Revised: • generalised neighbour of a vertex type tuple: ( v u, 1 , v u, 2 ) = (cid:0) ( u, v 1 ) , ( v 1 , u ) (cid:1) .

Original: The corresponding neighbour colour tuples are: – ( c α , c α ) if v 1 = u , – ( c β , c β ) if v 1 (cid:54)∼ u , – ( c γ , c γ ) if v 1 ∼ u .
Revised: The corresponding neighbour colour tuples are: – ( c α , c α ) if v 1 = u , – ( c β , c β ) if v 1 (cid:54)∼ u , – ( c γ , c γ ) if v 1 ∼ u .

Original: The update of the 2-FWL is: c (1) ( v ) =
Revised: The update of the 2-FWL is: c [Equation] HASH [Equation]

Original: HASH (cid:16) c α , (cid:72) ( c α , c α ) (cid:124) (cid:123)(cid:122) (cid:125) 1 time , ( c β , c β ) (cid:124) (cid:123)(cid:122) (cid:125) n − 1 − d times , ( c γ , c γ ) (cid:124) (cid:123)(cid:122) (cid:125)
Revised: (cid:72) ( [Equation] (cid:124) (cid:123)(cid:122) (cid:125) 1 time , ( c [Equation]

Revised: c

Revised: [Equation]

Revised: ) (cid:124) (cid:123)(cid:122) (cid:125) n − 1 − d times , ( c γ , c γ ) (cid:124) (cid:123)(cid:122) (cid:125)

Original: d times (cid:73) (cid:17)
Revised: d times (cid:73) (cid:17)

Original: same for all vertex type 2-tuples.
Revised: same for all vertex type 2-tuples.

Original: • generalised neighbour of a non-edge type tuple: ( v u, 1 , v u, 2 ) =
Revised: • generalised neighbour of a non-edge type tuple: ( v u, 1 , v u, 2 ) =

Original: (cid:0) ( u, v 2 ) , ( v 1 , u ) (cid:1) .
Revised: (cid:0) ( u, v 2 ) , ( v 1 , u ) (cid:1) .

Original: The corresponding neighbour colour tuples are: – ( c α , c β ) if v 2 = u , – ( c β , c α ) if v 1 = u , – ( c γ , c β ) if v 2 ∼ u and v 1 (cid:54)∼ u , – ( c β , c γ ) if v 1 ∼ u and v 2 (cid:54)∼ u , – ( c β , c β ) if v 1 (cid:54)∼ u and v 2 (cid:54)∼ u , – ( c γ , c γ ) if v 1 ∼ u and v 2 ∼ u .
Revised: The corresponding neighbour colour tuples are: – ( c α , c β ) if v 2 = u , – ( c β , c α ) if v 1 = u , – ( c γ , c β ) if v 2 ∼ u and v 1 (cid:54)∼ u , – ( c β , c γ ) if v 1 ∼ u and v 2 (cid:54)∼ u , – ( c β , c β ) if v 1 (cid:54)∼ u and v 2 (cid:54)∼ u , – ( c γ , c γ ) if v 1 ∼ u and v 2 ∼ u .

Original: The update of the 2-FWL is:
Revised: The update of the 2-FWL is:

Original: [Equation]
Revised: [Equation]

Original: (cid:72)

Original: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125) 1 time

Original: [Equation]

Original: ( c

Original: [Equation]

Original: c α ) (cid:124) (cid:123)(cid:122) (cid:125) 1 time , ( c γ , c β ) (cid:124) (cid:123)(cid:122) (cid:125)

Original: d − µ times , ( c β , c γ ) (cid:124) (cid:123)(cid:122)

Original: [Table] [Table] same for all non-edge type 2-tuples. • generalised neighbour of an edge type tuple: – ( c α , c γ ) if v 2 = u , – ( c γ , c α ) if v 1 = u , – ( c γ , c β ) if v 2 ∼ u and v 1 (cid:54)∼ u , – ( c β , c γ ) if v 1 ∼ u and v 2 (cid:54)∼ u , – ( c β , c β ) if v 1 (cid:54)∼ u and v 2 (cid:54)∼ u , – ( c γ , c γ ) if v 1 ∼ u and v 2 ∼ u .
Revised: • generalised neighbour of an edge type tuple: – ( c α , c γ ) if v 2 = u , – ( c γ , c α ) if v 1 = u , – ( c γ , c β ) if v 2 ∼ u and v 1 (cid:54)∼ u , – ( c β , c γ ) if v 1 ∼ u and v 2 (cid:54)∼ u , – ( c β , c β ) if v 1 (cid:54)∼ u and v 2 (cid:54)∼ u , – ( c γ , c γ ) if v 1 ∼ u and v 2 ∼ u .

Original: The update of the 2-FWL is:
Revised: The update of the 2-FWL is:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: same for all edge type 2-tuples.
Revised: same for all edge type 2-tuples.

Original: From the analysis above, it is clear that all 2-tuples in the graph of the same initial type are assigned the same colour in the 1st iteration of 2-FWL.
Revised: From the analysis above, it is clear that all 2-tuples in the graph of the same initial type are assigned the same colour in the 1st iteration of 2-FWL.

Original: In other words, the vertices cannot be further partitioned, so the algorithm terminates.
Revised: In other words, the vertices cannot be further partitioned, so the algorithm terminates.

Original: Therefore, if two SR graphs have the same parameters n , d , λ , µ then 2-FWL will yield the same colour distribution and thus the graphs will be deemed isomorphic.
Revised: Therefore, if two SR graphs have the same parameters n , d , λ , µ then 2-FWL will yield the same colour distribution and thus the graphs will be deemed isomorphic.

Original: C E XPERIMENTAL S ETTINGS - A DDITIONAL D ETAILS
Revised: C E XPERIMENTAL S ETTINGS - A DDITIONAL D ETAILS

Original: In this section, we provide additional implementation details of our experiments.
Revised: In this section, we provide additional implementation details of our experiments.

Original: All experiments were performed on a server equipped with 8 Tesla V100 16 GB GPUs, except for the Collab dataset where a Tesla V100 GPU with 32 GB RAM was used due to larger memory requirements (a large percentage of Collab graphs are dense or even nearly complete in some cases).
Revised: All experiments were performed on a server equipped with 8 Tesla V100 16 GB GPUs, except for the Collab dataset where a Tesla V100 GPU with 32 GB RAM was used due to larger memory requirements (a large percentage of Collab graphs are dense or even nearly complete in some cases).

Original: Experimental tracking and hyperparameter optimisation were done via the Weights & Biases platform (wandb) (Biewald, 2020).
Revised: Experimental tracking and hyperparameter optimisation were done via the Weights & Biases platform (wandb) (Biewald, 2020).

Original: Our implementation is based on native PyTorch sparse operations (Paszke et al., 2019) in order to ensure complete reproducibility of the results.
Revised: Our implementation is based on native PyTorch sparse operations (Paszke et al., 2019) in order to ensure complete reproducibility of the results.

Original: PyTorch Geometric (Fey & Lenssen, 2019) was used for additional operations (such as preprocessing and data loading).
Revised: PyTorch Geometric (Fey & Lenssen, 2019) was used for additional operations (such as preprocessing and data loading).

Original: In each one of the different experiments we aim to show that structural identiﬁers can be used off-the-shelf and are independent of the architecture.
Revised: In each one of the different experiments we aim to show that structural identiﬁers can be used off-the-shelf and are independent of the architecture.

Original: At the same time we aim to suppress the effect of other confounding factors in the model performance, thus wherever possible we build our model on top of a baseline architecture.
Revised: At the same time we aim to suppress the effect of other confounding factors in the model performance, thus wherever possible we build our model on top of a baseline architecture.

Original: More details in the relevant subsections.
Revised: More details in the relevant subsections.

Original: Interestingly, we observed that in most of the cases it was sufﬁcient to replace only the ﬁrst layer of the baseline architecture with a GSN layer, in order to obtain a boost in performance.
Revised: Interestingly, we observed that in most of the cases it was sufﬁcient to replace only the ﬁrst layer of the baseline architecture with a GSN layer, in order to obtain a boost in performance.

Original: This can be understood by considering that if the update and message functions are sufﬁciently expressive, then they should be able to learn to preserve the input information in their output (in the hidden states of the vertices with regards to vertex counts, or in the hidden states of the endpoints of the edges with regards to edge counts).
Revised: This can be understood by considering that if the update and message functions are sufﬁciently expressive, then they should be able to learn to preserve the input information in their output (in the hidden states of the vertices with regards to vertex counts, or in the hidden states of the endpoints of the edges with regards to edge counts).

Original: Throughout the experimental evaluation the structural identiﬁers x V ( v ) and x E ( { u, v } ) are onehot encoded, by taking into account the unique count values present in the dataset.
Revised: Throughout the experimental evaluation the structural identiﬁers x V ( v ) and x E ( { u, v } ) are onehot encoded, by taking into account the unique count values present in the dataset.

Original: Other more sophisticated methods can be used, e.g. transformation to continuous features via a normalisation scheme or binning.
Revised: Other more sophisticated methods can be used, e.g. transformation to continuous features via a normalisation scheme or binning.

Original: However, we found that the number of unique values in our datasets were usually relatively small (which is a good indication of recurrent structural roles) and thus such methods were not necessary.
Revised: However, we found that the number of unique values in our datasets were usually relatively small (which is a good indication of recurrent structural roles) and thus such methods were not necessary.

Original: C. G RAPH R EGRESSION ON ZINC
Revised: C. G RAPH R EGRESSION ON ZINC

Original: Experimental Details: The ZINC dataset includes 12k molecular graphs of which 10k form the training set and the remaining 2k are equally split between validation and test (splits obtained from https://github.com/graphdeeplearning/benchmarking-gnns ).
Revised: Experimental Details: The ZINC dataset includes 12k molecular graphs of which 10k form the training set and the remaining 2k are equally split between validation and test (splits obtained from https://github.com/graphdeeplearning/benchmarking-gnns ).

Original: Molecule sizes range from 9 to 37 nodes/atoms.
Revised: Molecule sizes range from 9 to 37 nodes/atoms.

Original: Node features encode the type of atoms and edge features the chemical bonds between them.
Revised: Node features encode the type of atoms and edge features the chemical bonds between them.

Original: Again, here node and edge features are one-hot encoded.
Revised: Again, here node and edge features are one-hot encoded.

Original: We re-implemented the GIN baseline ( GIN † in Table 2 of the main paper).
Revised: We re-implemented the GIN baseline ( GIN † in Table 2 of the main paper).

Original: We extended GIN with structural identiﬁers as in Eq.
Revised: We extended GIN with structural identiﬁers as in Eq.

Original: 8 and 9 ( GSN model in Table 2).
Revised: 11 and 12 ( GSN model in Table 2).

Original: Our stronger baseline ( MPNN-EF model in Table 2) updates node representations as follows:
Revised: Our stronger baseline ( MPNN-EF model in Table 2) updates node representations as follows:

Original: h t +1 ( v ) =
Revised: h t +1 ( v ) =

Original: UP t +1 ( h t ( v ) ,MSG t +1 ( v )) ,
Revised: UP t +1 ( [Equation] t ( v ) ,

Original: [Equation]
Revised: [Equation]

Revised: t +

Original: [Equation]
Revised: [Equation]

Original:  where UP t and M t functions are MLPs and e ( { u, v } ) are edge features.
Revised: v )) , [Equation] where UP t and M t functions are MLPs and e ( { u, v } ) are edge features.

Original: Our extension with structural identiﬁers ( GSN-EF model in Table 2) is precisely the model of Eq.
Revised: Our extension with structural identiﬁers ( GSN-EF model in Table 2) is precisely the model of Eq.

Original: 3 or 4 of the main paper.
Revised: A.3 or 4 of the main paper.

Original: Observe that, probably due to the fact that the ZINC dataset is larger and more stable, the general MPNN-based formulation performs better than the GIN-based counterpart.
Revised: Observe that, probably due to the fact that the ZINC dataset is larger and more stable, the general MPNN-based formulation performs better than the GIN-based counterpart.

Original: Following the same rationale as before, the network conﬁguration is minimally modiﬁed w.r.t.
Revised: Following the same rationale as before, the network conﬁguration is minimally modiﬁed w.r.t.

Original: the baseline of Dwivedi et al.
Revised: the baseline of Dwivedi et al.

Original: (2020), while here no hyperparameter tuning is done, since the best performing hyperparameters are provided by the authors.
Revised: (2020), while here no hyperparameter tuning is done, since the best performing hyperparameters are provided by the authors.

Original: In particular, the parameters are the following: 4 message passing layers, no jumping knowledge, transformation of the output of the last layer by a MLP, readout: sum, batch size: 128, dropout: 0.0, network width: 128, learning rate: 0.001.
Revised: In particular, the parameters are the following: 4 message passing layers, no jumping knowledge, transformation of the output of the last layer by a MLP, readout: sum, batch size: 128, dropout: 0.0, network width: 128, learning rate: 0.001.

Original: The learning rate is reduced by 0.5 (decay rate) after 5 epochs (patience) without improvement in the validation loss.
Revised: The learning rate is reduced by 0.5 (decay rate) after 5 epochs (patience) without improvement in the validation loss.

Original: Training is stopped when the learning rate reaches the minimum learning rate value of 10 − 5 .
Revised: Training is stopped when the learning rate reaches the minimum learning rate value of 10 − 5 .

Original: Validation and test metrics are inferred using the model at the last training epoch.
Revised: Validation and test metrics are inferred using the model at the last training epoch.

Original: We select our best performing substructure related parameters based on the performance in the validation set in the last epoch.
Revised: We select our best performing substructure related parameters based on the performance in the validation set in the last epoch.

Original: We search cycles with k = 3 , . .
Revised: We search cycles with k = 3 , . .

Original: , 10 , graphlets against motifs, and
Revised: , 10 , graphlets against motifs, and

Original: GSN-v against GSN-e. The chosen hyperparameters for GSN are: GSN-e, cycle graphlets ofnodes and for GSN -EF : GSN-v, cycle motifs of 8 nodes .
Revised: GSN-v against GSN-e. The chosen hyperparameters for GSN are: GSN-e, cycle graphlets ofnodes and for GSN -EF : GSN-v, cycle motifs of 8 nodes .

Original: Once the model is chosen, we repeat the experiment 10 times with different seeds and report the mean and standard deviation of the test MAE in the last epoch.
Revised: Once the model is chosen, we repeat the experiment 10 times with different seeds and report the mean and standard deviation of the test MAE in the last epoch.

Original: This is performed for all 4 models that we implemented and compared ( GIN † ,
Revised: This is performed for all 4 models that we implemented and compared ( GIN † ,

Original: MPNN-EF, GSN, GSN-EF ).
Revised: MPNN-EF, GSN, GSN-EF ).

Original: Fixed parameter budget:
Revised: Fixed parameter budget:

Original: In Table 4 we include an additional comparison between the main methods, where we maintain an approximately constant parameter budget (approx.
Revised: In Table 5 we include an additional comparison between the main methods, where we maintain an approximately constant parameter budget (approx.

Original: 100k parameters; we adjust network width to keep this number consistent across different models), as suggested in Dwivedi et al.
Revised: 100k parameters; we adjust network width to keep this number consistent across different models), as suggested in Dwivedi et al.

Original: Again here, GSN signiﬁcantly outperforms the baselines, thus comparative performance is insensitive to the parameter budget.
Revised: Again here, GSN signiﬁcantly outperforms the baselines, hence the improvement is insensitive to the absolute number of the parameters.

Original: [Table]
Revised: [Table]

Original: Generalisation: empirical observations: We repeat the experimental evaluation on ZINC using different fractions of the training set.
Revised: Generalisation: empirical observations: We repeat the experimental evaluation on ZINC using different fractions of the training set.

Original: We compare our best baseline model (MPNN -EF in Table 2) against GSN with the best performing substructure (GSN -EF in Table 2).
Revised: We compare our best baseline model (MPNN -EF in Table 2) against GSN with the best performing substructure (GSN -EF in Table 2).

Original: In Figure 5, we plot the training and test errors of both methods.
Revised: In Figure 6, we plot the training and test errors of both methods.

Original: Regarding the training error, GSN consistently performs better, following our theoretical analysis on its expressive power.
Revised: Regarding the training error, GSN consistently performs better, following our theoretical analysis on its expressive power.

Original: More importantly, GSN manages to generalise much better even with a small fraction of the training dataset.
Revised: More importantly, GSN manages to generalise much better even with a small fraction of the training dataset.

Original: Observe that GSN requires only 20% of the samples to achieve approximately the same test error that MPNN achieves when trained on the entire training set.[Figure]
Revised: Observe that GSN requires only 20% of the samples to achieve approximately the same test error that MPNN achieves when trained on the entire training set.

Original: Disambiguation Scores:
Revised: Disambiguation Scores:

Original: In Table 5, we provide the disambiguation scores δ as deﬁned in section 5 of the main paper for the different types of substructures.
Revised: In Table 6, we provide the disambiguation scores δ as deﬁned in section 5 of the main paper for the different types of substructures.

Original: Note that these are computed based on vertex structural identiﬁers (GSN-v).k
Revised: Note that these are computed based on vertex structural identiﬁers (GSN-v).

Original: Cycles
Revised: [Figure]

Original: Paths
Revised: [Table]

Original: Trees 0.

Original: C. S YNTHETIC E XPERIMENT
Revised: C. S YNTHETIC E XPERIMENT

Original: For the Strongly Regular graphs dataset (available from http://users.cecs.anu.edu.au/ ˜bdm/data/graphs.html )
Revised: For the Strongly Regular graphs dataset (available from http://users.cecs.anu.edu.au/ ˜bdm/data/graphs.html )

Original: we use all the available families of graphs with size of at mostnodes:
Revised: we use all the available families of graphs with size of at mostnodes:

Original: • SR(16,6,2,2): 2 graphs • SR(25,12,5,6): 15 graphs • SR(26,10,3,4): 10 graphs • SR(28,12,6,4): 4 graphs • SR(29,14,6,7): 41 graphs • SR(35,16,6,8): 3854 graphs • SR(35,18,9,9): 227 graphs
Revised: • SR(16,6,2,2): 2 graphs • SR(25,12,5,6): 15 graphs • SR(26,10,3,4): 10 graphs • SR(28,12,6,4): 4 graphs • SR(29,14,6,7): 41 graphs • SR(35,16,6,8): 3854 graphs • SR(35,18,9,9): 227 graphs

Original: The total number of non-isomorphic pairs of the same size is ≈ 7 ∗ 10 7 .
Revised: The total number of non-isomorphic pairs of the same size is ≈ 7 ∗ 10 7 .

Original: We used a simple 2-layer architecture with width 64.
Revised: We used a simple 2-layer architecture with width 64.

Original: The message aggregation was performed as in the general formulation of
Revised: The message aggregation was performed as in the general formulation of

Original: Eq.
Revised: Eq.

Original: 5 and 6 of the main paper, where the update and the message functions are MLPs.
Revised: 5 and 6 of the main paper, where the update and the message functions are MLPs.

Original: The prediction is inferred by applying a sum readout function in in the last layer (i.e. a graph-level representation is obtained by summing the node-level representations) and then passing the output through a MLP (we did not use jumping knowledge from intermediate layers (Xu et al., 2018)).
Revised: The prediction is inferred by applying a sum readout function in in the last layer (i.e. a graph-level representation is obtained by summing the node-level representations) and then passing the output through a MLP (we did not use jumping knowledge from intermediate layers (Xu et al., 2018)).

Original: Regarding the substructures, we use graphlet counting, as certain motifs (e.g. cycles of length up to 7) are known to be unable to distinguish strongly regular graphs (since they can be counted by the 2-FWL (F¨urer, 2017; Arvind et al., 2019)).
Revised: Regarding the substructures, we use graphlet counting, as certain motifs (e.g. cycles of length up to 7) are known to be unable to distinguish strongly regular graphs (since they can be counted by the 2-FWL (F ¨ urer, 2017; Arvind et al., 2019)).

Original: Given the adversities that strongly regular graphs pose in graph isomorphism testing, it would be interesting to see how this method can perform in other categories of hard instances, such as the classical CFI counter-examples for k-WL proposed in Cai et al.
Revised: Given the adversities that strongly regular graphs pose in graph isomorphism testing, it would be interesting to see how this method can perform in other categories of hard instances, such as the classical CFI counter-examples for k-WL proposed in Cai et al.

Original: (1992), and explore further its expressive power and combinatorial properties.
Revised: (1992), and explore further its expressive power and combinatorial properties.

Original: We leave this direction to future work. 
Revised: We leave this direction to future work. [Table]

Original: C. TUD G RAPH C LASSIFICATION B ENCHMARKS
Revised: C. TUD G RAPH C LASSIFICATION B ENCHMARKS

Original: For this family of experiments, due to the usually small size of the datasets, we choose a parameterefﬁcient architecture, in order to reduce the risk of overﬁtting.
Revised: For this family of experiments, due to the usually small size of the datasets, we choose a parameterefﬁcient architecture, in order to reduce the risk of overﬁtting.

Original: In particular, we follow the simple
Revised: In particular, we follow the simple

Original: GIN architecture (Xu et al., 2019) and we concatenate structural identiﬁers to node or edge features depending on the variant.
Revised: GIN architecture (Xu et al., 2019) and we concatenate structural identiﬁers to node or edge features depending on the variant.

Original: Then for GSN-v, the hidden representation is updated as follows:
Revised: Then for GSN-v, the hidden representation is updated as follows:

Original: [Equation]
Revised: [Equation]

Original: and for GSN-e:
Revised: and for GSN-e:

Original: [Equation]
Revised: [Equation]

Original: where x E ( { v, v } ) is a dummy variable (also one-hot encoded) used to distinguish self-loops from edges.
Revised: where x E ( { v, v } ) is a dummy variable (also one-hot encoded) used to distinguish self-loops from edges.

Original: Empirically, we did not ﬁnd training the (cid:15) parameter used in GIN to make a difference.
Revised: Empirically, we did not ﬁnd training the (cid:15) parameter used in GIN to make a difference.

Original: Note that this architecture is less expressive than our general formulation.
Revised: Note that this architecture is less expressive than our general formulation.

Original: However, we found it to work well in practice for the TUD datasets, possibly due to its simplicity and small number of parameters.
Revised: However, we found it to work well in practice for the TUD datasets, possibly due to its simplicity and small number of parameters.

Original: We implement an architecture similar to GIN (Xu et al., 2019), i.e. 4 message passing layers, jumping knowledge from all the layers (Xu et al., 2018)
Revised: We implement an architecture similar to GIN (Xu et al., 2019), i.e. 4 message passing layers, jumping knowledge from all the layers (Xu et al., 2018)

Original: (including the input), transformation of each intermediate graph-level representation by a linear layer, sum readout for biological and mean readout for social networks.
Revised: (including the input), transformation of each intermediate graph-level representation by a linear layer, sum readout for biological and mean readout for social networks.

Original: Node features are one-hot encodings of the categorical node labels.
Revised: Node features are one-hot encodings of the categorical node labels.

Original: Similarly to the baseline, the hyperparameters search space is the following: batch size in { 32, 128 } (except for
Revised: Similarly to the baseline, the hyperparameters search space is the following: batch size in { 32, 128 } (except for

Original: Collab where only 32 was searched due to GPU memory limits), dropout in { 0,0.5 } , network width in { 16,32 } for biological networks, 64 for social networks, learning rate in { 0.01, 0.001 } , decay rate in { 0.5,0.9 } and decay steps in { 10,50 } (number of epochs after which the learning rate is reduced by multiplying with the decay rate).
Revised: Collab where only 32 was searched due to GPU memory limits), dropout in { 0,0.5 } , network width in { 16,32 } for biological networks, 64 for social networks, learning rate in { 0.01, 0.001 } , decay rate in { 0.5,0.9 } and decay steps in { 10,50 } (number of epochs after which the learning rate is reduced by multiplying with the decay rate).

Original: For social networks, since they are not attributed graphs, we also experimented with using the degree as a node feature, but in most cases the structural identiﬁers were sufﬁcient.
Revised: For social networks, since they are not attributed graphs, we also experimented with using the degree as a node feature, but in most cases the structural identiﬁers were sufﬁcient.

Original: Model selection is done in two stages.
Revised: Model selection is done in two stages.

Original: First, we choose a substructure that we perceive as promising based on indications from the speciﬁc domain: triangles for social networks and Proteins, and 6-cycles (motifs) for molecules.
Revised: First, we choose a substructure that we perceive as promising based on indications from the speciﬁc domain: triangles for social networks and Proteins, and 6-cycles (motifs) for molecules.

Original: Under this setting we tune model hyperparameters for a GSN-e model.
Revised: Under this setting we tune model hyperparameters for a GSN-e model.

Original: Then, we extend our search to the parameters related to the substructure collection: i.e. the maximum size k and motifs against graphlets.
Revised: Then, we extend our search to the parameters related to the substructure collection: i.e. the maximum size k and motifs against graphlets.

Original: In all the molecular datasets we search cycles with  k = 3 , . ..
Revised: In all the molecular datasets we search cycles with [Table] k = 3 , . .

Original: , 12 , except for NCI1, where we also consider larger sizes due to the presence of large rings in the dataset ( macrocycles (Liu et al., 2017)).
Revised: 12 , except for NCI1, where we also consider larger sizes due to the presence of large rings in the dataset ( macrocycles (Liu et al., 2017)).

Original: For social networks, we searched cliques with[Table] k = 3 , 4 , 5 .
Revised: For social networks, we searched cliques with k = 3 , 4 , 5 .

Original: In Table 7 we report the hyperparameters chosen by our model selection procedure, including the best performing substructures.
Revised: In Table 8 we report the hyperparameters chosen by our model selection procedure, including the best performing substructures.

Original: The seven datasets 3 we chose are the intersection of the datasets used by the authors of our main baselines: the Graph Isomorphism Network (GIN)
Revised: The seven datasets 3 we chose are the intersection of the datasets used by the authors of our main baselines: the Graph Isomorphism Network (GIN)

Original: (Xu et al., 2019), a simple, yet powerful GNN with expressive power equal to the 1-WL test, and the Provably Powerful Graph Network (PPGN) (Maron et al., 2019a), a polynomial alternative to the Invariant Graph Network (Maron et al., 2019b), that increases its expressive power to match the 2-FWL.
Revised: (Xu et al., 2019), a simple, yet powerful GNN with expressive power equal to the 1-WL test, and the Provably Powerful Graph Network (PPGN) (Maron et al., 2019a), a polynomial alternative to the Invariant Graph Network (Maron et al., 2019b), that increases its expressive power to match the 2-FWL.

Original: We also compare our results to other GNNs as well as Graph Kernel approaches.
Revised: We also compare our results to other GNNs as well as Graph Kernel approaches.

Original: Our main baseline from the GK family is the Graph Neural
Revised: Our main baseline from the GK family is the Graph Neural

Original: Tangent Kernel (GNTK) (Du et al., 2019a), which is a kernel obtained from a GNN of inﬁnite width.
Revised: Tangent Kernel (GNTK) (Du et al., 2019a), which is a kernel obtained from a GNN of inﬁnite width.

Original: This operates in the Neural Tangent Kernel regime (Jacot et al., 2018; Allen-Zhu et al., 2019; Du et al., 2019b).
Revised: This operates in the Neural Tangent Kernel regime (Jacot et al., 2018; Allen-Zhu et al., 2019; Du et al., 2019b).

Original: Table 6 is an extended version of Table 1 of the main paper, where the most prominent methods are reported, regardless of the splits they were evaluated on.
Revised: Table 7 is an extended version of Table 1 of the main paper, where the most prominent methods are reported, regardless of the splits they were evaluated on.

Original: For DGK (best variant) (Yanardag & Vishwanathan, 2015), FSGD (Verma & Zhang, 2017), AWE (Ivanov & Burnaev, 2018), ECC (Simonovsky & Komodakis, 2017), PSCN (Niepert et al., 2016), DiffPool (Ying et al., 2018b), CCN (Kondor et al., 2018) (slightly different setting since they perform a train, validation, test split), 1-2GNN (Morris et al., 2019) and GNTK (Du et al., 2019a), we obtain the results from the original papers.
Revised: For DGK (best variant) (Yanardag & Vishwanathan, 2015), FSGD (Verma & Zhang, 2017), AWE (Ivanov & Burnaev, 2018), ECC (Simonovsky & Komodakis, 2017), PSCN (Niepert et al., 2016), DiffPool (Ying et al., 2018b), CCN (Kondor et al., 2018) (slightly different setting since they perform a train, validation, test split), 1-2GNN (Morris et al., 2019) and GNTK (Du et al., 2019a), we obtain the results from the original papers.

Original: For RWK (G¨artner et al., 2003), GK (Shervashidze et al., 2009), PK (Neumann et al., 2016),
Revised: For RWK (G¨artner et al., 2003), GK (Shervashidze et al., 2009), PK (Neumann et al., 2016),

Original: DCNN (Atwood & Towsley, 2016) and DGCNN (Zhang et al., 2018), we obtain the results from the
Revised: DCNN (Atwood & Towsley, 2016) and DGCNN (Zhang et al., 2018), we obtain the results from the

Original: DGCNN paper, where the authors reimplemented these methods and evaluated them with the same split.
Revised: DGCNN paper, where the authors reimplemented these methods and evaluated them with the same split.

Original: Similarly, we obtain the WLK (Shervashidze et al., 2011) and
Revised: Similarly, we obtain the WLK (Shervashidze et al., 2011) and

Original: GIN (Xu et al., 2019) results from the GIN paper, and IGN (Maron et al., 2019b) and PPGN (Maron et al., 2019a) results from the
Revised: GIN (Xu et al., 2019) results from the GIN paper, and IGN (Maron et al., 2019b) and PPGN (Maron et al., 2019a) results from the

Original: PPGN paper.
Revised: PPGN paper.

Original: C. G RAPH C LASSIFICATION ON OGBG - MOLHIV
Revised: C. G RAPH C LASSIFICATION ON OGBG - MOLHIV

Original: The ogbg-molhiv dataset contains ≈ 41K graphs, with 25.5 nodes and 27.5 edges on average.
Revised: The ogbg-molhiv dataset contains ≈ 41K graphs, with 25.5 nodes and 27.5 edges on average.

Original: As most molecular graphs, the average degree is small (2.2) and they exhibit a tree-like structure (average clustering coefﬁcient 0.002).
Revised: As most molecular graphs, the average degree is small (2.2) and they exhibit a tree-like structure (average clustering coefﬁcient 0.002).

Original: The average diameter is 12 (more details in Hu et al.
Revised: The average diameter is 12 (more details in Hu et al.

Original: We follow the design choices of the authors of Hu et al.
Revised: We follow the design choices of the authors of Hu et al.

Original: and extend their architectures to include structural identiﬁers.
Revised: and extend their architectures to include structural identiﬁers.

Original: Initial node features and edge features are multi-hot encodings passed through linear layers that project them in the same embedding space, i.e. h 0 ( v ) =
Revised: Initial node features and edge features are multi-hot encodings passed through linear layers that project them in the same embedding space, i.e. h 0 ( v ) =

Original: W 0 h · h in ( v ) ,
Revised: W 0 h · h in ( v ) ,

Original: [Table]
Revised: [Table]

Original: [Table]

Original: e t ( { v, u } ) =
Revised: e t ( { v, u } ) =

Original: W te · e in ( { u, v } ) .
Revised: W te · e in ( { u, v } ) .

Original: The baseline model is a modiﬁcation of GIN that allows for edge features: for each neighbour, the hidden representation is added to an embedding of its associated edge feature.
Revised: The baseline model is a modiﬁcation of GIN that allows for edge features: for each neighbour, the hidden representation is added to an embedding of its associated edge feature.

Original: Then the result is passed through a ReLU non-linearity which produces the neighbour’s message.
Revised: Then the result is passed through a ReLU non-linearity which produces the neighbour’s message.

Original: Formally, the aggregation is as follows:
Revised: Formally, the aggregation is as follows:

Original: [Equation]
Revised: [Equation]

Original: A stronger baseline is also proposed by the authors: in order to allow global information to be broadcasted to the nodes, a virtual node takes part in the message passing ( -VN setting in Table 3 of the main paper).
Revised: A stronger baseline is also proposed by the authors: in order to allow global information to be broadcasted to the nodes, a virtual node takes part in the message passing ( -VN setting in Table 3 of the main paper).

Original: The virtual node representation, denoted as G t , is initialised as a zero vector Gand then Message Passing proceeds as follows: 
Revised: The virtual node representation, denoted as G t , is initialised as a zero vector Gand then Message Passing proceeds as follows: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We modify this model, as follows:
Revised: We modify this model, as follows:

Original: ﬁrst the substructure counts are embedded into the same embedding space as the rest of the features.
Revised: ﬁrst the substructure counts are embedded into the same embedding space as the rest of the features.

Original: Then, for GSN-v, they are added to the corresponding node embeddings: ´h t ( v [Equation] =
Revised: Then, for GSN-v, they are added to the corresponding node embeddings: ´h t ( v ) =

Original: [Equation] tV · x V ( v ) , or for GSN-e, they are added to the edge embeddings ´e t ( { v, u } ) =
Revised: h t ( v ) + W tV · x V ( v ) , or for GSN-e, they are added to the edge embeddings ´e t ( { v, u } ) =

Original: e t ( { v, u } ) + W tE · x E ( { u, v } ) .
Revised: e t ( { v, u } ) + W tE · x E ( { u, v } ) .

Original: Interestingly, even with this simple modiﬁcation we obtain a considerable improvement in the performance of the model in all splits, thus demonstrating strong generalisation capabilities.
Revised: Interestingly, even with this simple modiﬁcation we obtain a considerable improvement in the performance of the model in all splits, thus demonstrating strong generalisation capabilities.

Original: We use the same hyperparameters as the ones provided by the authors, i.e. 5 message passing layers, no jumping knowledge, mean readout, a linear layer applied after the readout, batch size: 32, dropout: 0.5, network width/embedding dimension: 300 (in the ogb implementation the hidden layer of each
Revised: We use the same hyperparameters as the ones provided by the authors, i.e. 5 message passing layers, no jumping knowledge, mean readout, a linear layer applied after the readout, batch size: 32, dropout: 0.5, network width/embedding dimension: 300 (in the ogb implementation the hidden layer of each

Original: MLP has dimensions equal to 2*network width, contrary to the rest of the experiments where network width and MLP hidden dimensions are equal), learning rate: 0.001.
Revised: MLP has dimensions equal to 2*network width, contrary to the rest of the experiments where network width and MLP hidden dimensions are equal), learning rate: 0.001.

Original: We select our best performing substructure related parameters based on the highest validation ROCAUC (choosing the best scoring epoch as in Hu et al.
Revised: We select our best performing substructure related parameters based on the highest validation ROCAUC (choosing the best scoring epoch as in Hu et al.

Original: We search cycles with k = 3 , . .
Revised: We search cycles with k = 3 , . .

Original: . , 12 , graphlets against motifs, and GSN-v against GSN-e (see Table 8 for the chosen parameters).
Revised: . , 12 , graphlets against motifs, and GSN-v against GSN-e (see Table 9 for the chosen parameters).

Original: We repeat the experiment 10 times with different seeds and report the mean and standard deviation of the train, validation and test ROC-AUC, again by choosing the best scoring epoch w.r.t the validation set.
Revised: We repeat the experiment 10 times with different seeds and report the mean and standard deviation of the train, validation and test ROC-AUC, again by choosing the best scoring epoch w.r.t the validation set.

Original: We repeat the process for all 3 settings independently ( GSN-VN, GSN-AF, GSN-VN-AF ).
Revised: We repeat the process for all 3 settings independently ( GSN-VN, GSN-AF, GSN-VN-AF ).

Revised: Impact of the ‘Scaffold-based’ dataset split: Motivated by the differences between the train, validation and test distributions in the ogbg-molhiv dataset (the authors of Hu et al.

Revised: (2020) implement a “scaffold splitting” procedure, i.e. molecules with different 2D-structure are assigned to a different partition of the dataset), we studied how different substructure sizes affect the performance on different sets.

Revised: In particular, as shown in Table 10, we observed that substructures that yielded the best test score did not always attain best validation performance as well, this conﬁrming the characteristic discrepancy between the test and validation distributions.

Revised: Lastly, we note that these results were obtained with a GSN model using the structural features only at its input.

Revised: In fact, it is possible to ’inject’ the structural features at each message passing layer of the architecture, effectively accounting for a skip connection acting on structural identiﬁers.

Revised: Although we observe performance improvements, the typical validation/test distribution discrepancy persists.

Revised: Results are reported in Table 10 (“GSN - skip”).

Revised: [Table]

Revised: S

Revised: [Table]

Revised: F EATURES & M ESSAGE P

Revised: [Table]

Revised: The baseline architecture treats the input node and edge features, along with the structural identiﬁers, as a set .

Revised: In particular, we consider each graph as a set of independent edges ( v, u ) endowed with the features of the endpoint nodes h ( v ) , h ( u ) , the structural identiﬁers x V ( v ) , x V ( u ) and the edge features e ( v, u )

Revised: [Table]

Revised: and we implement a DeepSets universal set function approximator Zaheer et al.

Revised: (2017) to learn a prediction function f

Revised: [Table]

Revised: [Equation]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: , with E G the edge set of the graph and ρ, φ , modelled as MLPs.

Revised: This baseline is naturally extended to the case where we consider edge structural identiﬁers by replacing ( x V ( v ) , x V ( u )) with x E ( v, u ) .

Revised: For fairness of evaluation, we follow the exact same parameter tuning procedure as the one we followed for our GSN models for each benchmark, i.e. for the TUD datasets we ﬁrst tune network and optimisation hyperaparameters (network width was set to be either equal to the ones we tuned for GSN, or such that the absolute number of learnable parameters was equal to those used by GSN; depth of the MLPs was set to 2) and subsequently we choose the substructure related parameters based on the evaluation protocol of Xu et al.

Revised: For

Revised: ZINC and ogbg-molhiv we perform only substructures selection, based on the performance on the validation set.

Revised: Using the same widths as in GSN leads to smaller baseline models w.r.t the absolute number of parameters, and we interestingly observed this to lead to particularly strong performance in some cases, especially Proteins and MUTAG, where our DeepSets implementation attains state-of-art results.

Revised: This ﬁnding motivated us to explore ‘smaller’ GSNs (with either reduced layer width or a single message passing layer).

Revised: These GSN variants exhibited a similar trend, i.e. to perform better than their ‘larger’ counterparts over these two datasets.

Revised: We hypothesise this phenomenon to be mostly due to the small size of these datasets, which encourages overﬁtting when using architectures with larger capacity.

Revised: In Table 4 in the main paper, we report the result for the best performing architectures, along with the number of learnable parameters.