A N EW P ERSPECTIVE ON "H OW G RAPH N EURAL N ETWORKS G O B EYOND W EISFEILER -L EHMAN ?"
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a new perspective on designing powerful Graph Neural Networks (GNNs).
In a nutshell, this enables a general solution to inject structural properties of graphs into a message-passing aggregation scheme of GNNs.
As a theoretical basis, we develop a new hierarchy of local isomorphism on neighborhood subgraphs.
Then, we theoretically characterize how message-passing GNNs can be designed to be more expressive than the Weisfeiler Lehman test.
To elaborate this characterization, we propose a novel neural model, called GraphSNN , and prove that this model is strictly more expressive than the Weisfeiler Lehman test in distinguishing graph structures.
We empirically verify the strength of our model on different graph learning tasks.
It is shown that our model consistently improves the state-of-the-art methods on the benchmark tasks without sacriﬁcing computational simplicity and efﬁciency.
I NTRODUCTION
Many Graph Neural Networks (GNNs) employ a message-passing aggregation scheme to learn low-dimensional vector space representations for nodes in a graph (Kipf & Welling, 2017;
Veli ˇ ckovi ´ c et al., 2017; Hamilton et al., 2017; Gilmer et al., 2017; Sato, 2020; Loukas, 2020; de Haan et al., 2020).
Let G =
( V, E ) be a graph.
For each node v ∈ V , a message-passing aggregation scheme recursively aggregates the feature vectors of nodes in the neighborhood of v and combines the aggregated information with the feature vector of v itself to obtain a representation.
Since there is no natural ordering on nodes, such message-passing aggregation schemes are usually required to be permutation-invariant (Maron et al., 2018; Keriven & Peyré, 2019; Garg et al., 2020).
Despite advances of GNNs in various graph learning tasks such as node classiﬁcation (Kipf & Welling, 2017; Xu et al., 2018), graph classiﬁcation (Xu et al., 2019; Wu et al., 2019) and link prediction (Zhang & Chen, 2017), there is still a lack of theoretical understanding of how to design powerful and practically useful GNNs that can capture rich structural information of graphs.
Recent studies (Xu et al., 2019; Morris et al., 2019) have explored the connections between GNNs and the Weisfeiler-Lehman (WL) test (Weisfeiler & Leman, 1968).
By representing a neighborhood as a multiset of feature vectors and treating the neighborhood aggregation as an aggregation function over multisets, Xu et al.
(2019) showed that message-passing GNNs are at most as powerful as the
WL test in distinguishing graph structures.
However, many simple graph structures still cannot be distinguished by the WL test, e.g., G 1 and G 2 shown in Figure 1.
A question is: how to design expressive yet simple GNNs that can go beyond the WL test with a theoretically provable guarantee?
Recently, there have been three main directions of extending GNNs beyond WL: (1) building GNNs for higher-order WL (i.e. k -WL with k ≥ 3 ) or variants (Maron et al., 2019; Morris et al., 2020; 2019); (2) counting on pre-deﬁned substructures as additional features (Bouritsas et al., 2020); (3) augmenting node identiﬁers or random features into GNNs (You et al., 2021; Vignac et al., 2020;
Sato et al., 2021).
Unlike these works, we aim to introduce a general solution upon which GNNs can be enhanced to capture structural properties of graphs.
This solution enables GNNs to provably be more expressive than the Weisfeiler-Lehman test, but still computationally efﬁcient .
It overcomes the following limitations of existing works.
Compared with higher-order WL methods in (1) which require high computational overhead and are impractical, our method goes beyond the WL test but is still computationally efﬁcient.
Compared with the methods on counting substructures in (2), our method does not require to handcraft substructures.
Compared with the methods of augmenting node
Graphs
Neighborhood
Overlap
Subgraphs
Subgraphs (a) (b)
Neighborhood Subgraphs
Overlap Subgraphs Graphs u u v vuuvuuvuuvuu 4 G
Gh
u h v 1.5 v1. v1.
v v1.67 vv u
S v
S u
S vvv 4 v 3 v
S v
S v S v
GMP Framework
A GGREGATE N ( · )
A GGREGATE I ( · )
C OMBINE ( · )
EmbeddingSpace (a) (b)      1.
Neighborhood Subgraphs
Overlap Subgraphs Graphs u u v v vuvuvuvuvuvuvvuu 4 G
G
Embedding Space
h
u h v identiﬁers or random features in (3), our method can ﬂexibly quantify local structures (see examples in Figure 3) and also capture different classes of local structures w.r.t.
different graph learning tasks.
Our work is grounded in three observations: (i) Treating a neighborhood as a multiset of feature vectors ignores the rich structure information among vertices in the neighborhood, thereby limiting the representational capacity of the model.
Thus, we represent a neighborhood as a neighborhood subgraph in which vertices are structurally related, and show that the WL test is only as powerful as distinguishing neighborhood subgraphs in terms of their subtree structures in the neighborhood.
(ii) There exists a natural class of isomorphic graphs, which strictly lies in between neighborhood subgraph isomorphism and neighborhood subtree isomorphism . We call it overlap (subgraph) isomorphism .
The notion of overlap subgraph enables us to characterize structural interactions of vertices and inject them into a message-passing aggregation scheme for GNNs.
(iii)
By designing a proper function for quantifying structural interactions of vertices and preserving the injectiveness of a message-passing aggregation scheme, more expressive GNNs can be developed.
We propose a new GNN model that is strictly more expressive than the WL test to demonstrate an instance of this kind.
Contributions.
In summary, the main contributions of this work are as follows:
• We introduce a new hierarchy of local isomorphism to characterise different classes of local structures in neighborhood subgraphs, and discuss its connections with the WL test and GNNs (Section 2 and Theorems 1-2).
• We develop a simple yet powerful framework to inject structural properties into a messagepassing aggregation scheme, and theoretically characterize how GNNs can be designed to be more expressive beyond the WL test (Section 3 and Theorem 3).
• We propose a novel neural model for graph learning, called GraphSNN, and prove that GraphSNN is strictly more expressive than the the WL test in distinguishing graph structures (Section 4 and Theorem 4).
• We show that, due to the way of injecting structural properties into a structured-messagepassing aggregation scheme, GraphSNN can overcome the oversmoothing issue (Chen et al., 2020a; Zhao & Akoglu, 2019; Li et al., 2018)
(Section 5.4).
We have conducted experiments on benchmark tasks (Hu et al., 2020).
The experimental results show that our model is highly efﬁcient and can signiﬁcantly improve the state-of-the-art methods without sacriﬁcing computational simplicity.
Related work.
Weisfeiler-Lehman (WL) hierarchy is a well-established framework for graph isomorphism tests (Grohe, 2017).
Introduced by Weisfeiler and Lehman (Weisfeiler & Leman, 1968), the Weisfeiler-Lehman algorithm (also called 1-WL or color reﬁnement) is a computationally efﬁcient heuristic for testing graph isomorphism (Babai & Kucera, 1979).
It is known that 1-WL and 2-WL have the same power in distinguishing non-isomorphic graphs; when k ≥ 3, k-WL is strictly more powerful than (k-1)-WL (Cai et al., 1992; Grohe, 2017).
Message-passing GNNs are typically considered as a differentiable neural generalization of the
Weisfeiler-Lehman algorithms on graphs.
It has been reported (Xu et al., 2019) that some popular
GNNs such as GCN (Kipf & Welling, 2017) and GraphSAGE (Hamilton et al., 2017) are at most powerful as 1-WL in distinguishing graph structures.
Xu et al.
(2019) has shown that Graph
Isomorphism Network (GIN) can be as powerful as 1-WL.
At its core, GIN provides an injective aggregation scheme that is deﬁned as a function over multisets of feature vectors, and thus GIN has the representational power to map any two different multisets of feature vectors to different representations in an embedding space.
A considerable amount of efforts has been devoted to improve the expressive power of GNNs beyond 1-WL.
Generally, there are three directions: (1) Several works proposed higher-order variants of
GNNs that are as powerful as k-WL with k ≥ 3 (Azizian & Lelarge, 2020).
For example, Morris et al.
(2019) introduced k-order graph networks that are expressive as a set-based variant of k-WL, Maron et al.
(2019) proposed a reduced 2-order graph network that is as expressive as 3-WL, and
Morris et al.
(2020) proposed a local version of k-WL which considers only a subset of vertices in a neighborhood.
However, these more expressive GNNs are impractical to use due to their inherent high computational costs and sophisticated design.
(2) Some works attempted to incorporate inductive biases based on isomorphism counting on pre-deﬁned topological features such as triangles, cliques, and rings (Bouritsas et al., 2020;
Liu et al., 2020; Monti et al., 2018), similar to the traditional ideas of graph kernels (Yanardag & Vishwanathan, 2015).
However, pre-deﬁning topological features requires domain-speciﬁc expertise, which is often not readily available.
(3) Most recently, several works explored the ideas of augmenting GNNs using node identiﬁers or random features.
For example,
Vignac et al.
(2020) proposed a method that maintains a “local context" for each node based on manipulating node identiﬁers in a permutation equivariant way.
You et al.
(2021) developed ID-GNNs by taking into account the identity information of vertices.
Chen et al.
(2020b)
and Murphy et al.
(2019) assigned one-hot IDs to vertices based on the ideas of relational pooling.
Sato et al.
(2021) added a random feature to each node to improve the representational capability of GNNs.
Our work is fundamentally different from existing models by injecting properties of structural interactions among vertices based on a natural class of isomorphic graphs in the local neighborhood (i.e., overlap subgraph isomorphism) into a message-passing aggregation scheme of GNNs.
A N EW H IERARCHY OF L OCAL I SOMORPHISM
In this section, we characterize a hierarchy of graph isomorphism based on local neighborhood subgraphs and explore its connections to 1-WL.
Let G =
( V, E ) be a simple, undirected graph with a set V of vertices and a set E of edges.
The set of neighbors of a vertex v is denoted by N ( v ) = { u ∈ V | ( v, u ) ∈ E } .
The neighborhood subgraph of a vertex v , denoted by S v , is the subgraph induced in G by ˜ N ( v ) =
N ( v ) ∪ { v } , which contains all edges in E that have both endpoints in ˜ N ( v ) .
For two adjacent vertices v and u , i.e., ( v, u ) ∈ E , the overlap subgraph S vu between v and u is deﬁned as S vu = S v ∩ S u .
Let S i and S
j be the neighborhood subgraphs of two vertices i and j that are not necessarily adjacent, and h v be the feature vector of a vertex v ∈ V .
In the following, we deﬁne three notions of isomorphism, which correspond to different classes of local structures in neighborhood subgraphs.
Deﬁnition 1.
S i and S j are subgraph-isomorphic , denoted as S i (cid:39)
subgraph S j , if there exists a bijective mapping g : ˜ N ( i ) → ˜ N ( j ) such that g ( i ) =
j and for any two vertices v 1 , v 2 ∈ ˜ N
( i ) , vand v 2 are adjacent in S
i iff g ( v 1 ) and g ( v 2 ) are adjacent in S j , and h v 1 =
h g ( v 1 ) and h v 2 =
h g ( v 2 ) .
Deﬁnition 2.
S i and S j are overlap-isomorphic , denoted as S i (cid:39) overlap S j , if there exists a bijective mapping g :
˜ N ( i ) →
˜ N ( j ) such that g ( i ) =
j and for any v (cid:48) ∈
N ( i ) and g ( v (cid:48) )
=
u (cid:48) , S iv (cid:48) and S ju (cid:48) are subgraph-isomorphic.
Deﬁnition 3.
S i and S j are subtree-isomorphic , denoted as S i (cid:39) subtree S j , if there exists a bijective mapping g : ˜ N ( i )
→ ˜ N ( j ) such that g ( i ) =
j and for any v (cid:48) ∈
˜ N ( i ) and g ( v (cid:48) )
=
u (cid:48) , h v (cid:48)
=
h u (cid:48) .
Theorem 1 states that there is a hierarchy among these notions of local isomorphism on neighborhood subgraphs, where subgraph-isomorphism is the strongest one, subtree-isomorphism is the weakest,
11 u u v v 1 uu 1 v 2 uu 2 v 3 uu 3 v 4 uu 4 Gvv 4 v vvv v 1 vv 3 v {+
 S+}
[Figure]
[Figure] and overlap-isomorphism lies in between.
Figure 2 shows two groups of graphs: one is distinguishable w.r.t.
subgraph-isomorphism but not overlap-isomorphism, while the other is distinguishable by overlap-isomorphism but not subtree-isomorphism.
Theorem 1.
The following statements are true: (a) If S i (cid:39) subgraph S j , then S i (cid:39) overlap S j ; but not vice versa; (b) If S i (cid:39) overlap S j , then S i (cid:39) subtree S j ; but not vice versa.
Let S =
{ S v | v ∈ V } and ζ : S → R d mapping each neighborhood subgraph in S into a node embedding in R d .
The following theorem states that GNNs that are as powerful as 1-WL can distinguish two neighborhood subgraphs only w.r.t.
subtree-isomorphism at each layer.
Theorem 2.
Let M be a GNN.
M is as powerful as 1-WL in distinguishing non-isomorphic graphs if M has a sufﬁcient number of layers and each layer can map any S i and S j in S into two different embeddings (i.e., ζ ( S i ) (cid:54) =
ζ ( S j ) )
if and only if S i (cid:54)(cid:39) subtree S j .
The complete proofs of these theorems are provided in Appendix C.
A G ENERALISED M ESSAGE -P ASSING F RAMEWORK
In this section, we present a generalised message-passing framework (GMP) which enables to inject local structure into an aggregation scheme, in light of overlap subgraphs.
We theoretically characterize how GNNs can be designed to be more expressive than 1-WL in this framework.
Let S ∗ =
{ S vu | ( v, u ) ∈ E } be the set of overlap subgraphs in G .
We deﬁne structural coefﬁcients for each vertex v and its neighbors, i.e., ω : S×S ∗ → R
such that A vu = ω
( S v , S vu ) .
A question arising is: what are the desirable properties of such a function ω ?
Ideally, it should quantify how a vertex v structurally interacts with its neighbor u in the local neighborhood.
Thus, given S vu =
( V vu , E vu ) and S vu (cid:48)
=
( V vu (cid:48) , E vu (cid:48) ) , a carefully designed ω should exhibit the following properties:
(1) Local closeness: ω ( S v , S vu ) > ω ( S v , S vu (cid:48) ) if S vu and S vu (cid:48) are complete graphs with S vu
=
K i , S vu (cid:48) =
K j , and i > j , where K i refers to a complete graph on i vertices.
(2) Local denseness: ω ( S v , S vu ) > ω
( S v , S vu (cid:48) )
if S vu and S vu (cid:48) have the same number of vertices but differ in the number of edges s.t.
| V vu | = | V vu (cid:48)
| and | E vu | > | E vu (cid:48) | .
(3) Isomorphic invariant: ω ( S v , S vu ) =
ω ( S v , S vu (cid:48) )
if S vu and S vu (cid:48) are isomorphic.
Figure 3 illustrates the ﬁrst two properties.
Let {{·}} denote a multiset, ˜
A =
( ˜ A vu ) v,u ∈ V
where ˜
A vu is a normalised value of A vu , and X ∈ R | V |× f be a matrix of input feature vectors where x v ∈ R f associates each v ∈ V .
We denote the feature vector of v at the t-th layer by h ( t ) v and h (0) v = x v .
Then, the (t+1)-th layer of an aggregation scheme can be deﬁned as:
[Equation]
A GGREGATE N ( · ) and A GGREGATE I ( · ) are two possibly different parameterized functions.
Here, m ( t ) a is a message aggregated from the neighbors of v and their structural coefﬁcients, and m ( t ) v is an
11 Neighborhood Subgraphs
Overlap Subgraphs Graphs u u v v 1 uu 1 v 2 uu 2 v 3 uu 3 v 4 uu 4 G 2 Gh
u h v 1.5 v1.5 v 4 1.
v v1.67 v 3 (a) (b) i j
S i
S j
S v
S v
S v S v
[Figure]
“adjusted” message from v after performing an element-wise multiplication between A GGREGATE I ( · ) and h ( t ) v to account for structural effects from its neighbors.
Then, m ( t ) v and m ( t ) a are combined by
C OMBINE ( · ) to obtain the feature vector h ( t +1) v .
The following theorem states that a GNN can be more expressive than 1-WL if ω is powerful enough to distinguish structure beyond neighborhood subtrees and the neighborhood aggregation function Φ is injective under a sufﬁcient number of layers.
The proof is provided in Appendix C.
Theorem 3.
Let M be a GNN whose aggregation scheme Φ is deﬁned by Eq.
1-Eq.
M is strictly more expressive than 1-WL in distinguishing non-isomorphic graphs if M has a sufﬁcient number of layers and also satisﬁes the following conditions:
(1) M can distinguish at least two neighborhood subgraphs S i and S j with S i (cid:39) subtree S j , S i (cid:54)(cid:39) subgraph S j and {{ ˜ A iv (cid:48) | v (cid:
) ∈
N ( i ) }} (cid:54) = {{ ˜ A ju (cid:48) | u (cid:48)
∈ N ( j ) }} ; (2) Φ (cid:16) h ( t ) v , {{ h ( t ) u | u ∈ N ( v ) }} , {{ ( ˜ A vu , h ( t ) u ) | u ∈ N ( v ) }} (cid:17) is injective.
G RAPH SNN
Generally, there are many different ways of designing ω and Φ functions, leading to GNNs with different expressive powers.
To elaborate this, we propose a novel GNN model, named GraphSNN, whose aggregation scheme is an instantiation of our generalised message-passing framework.
We prove that the expressive power of GraphSNN goes beyond 1-WL.
Model design.
In the following, we provide a deﬁnition of ω that satisﬁes the properties of local closeness, local denseness, and isomorphic invariant. One key idea behind this deﬁnition is to make it capable of being generalized to support different graph learning tasks, controlled by λ > 0 (will be further discussed in Section 5.3):
[Equation]
This deﬁnition allows us to formulate a weighted adjacency matrix A
=
( A vu ) v,u ∈ V for GraphSNN.
To compare structural coefﬁcients across different nodes, we normalize A to ˜ A by ˜ A vu
=
A
[Equation]
vu .
Alternatively, A can be normalized using Softmax or other normalization techniques.
For each vertex v ∈ V , the feature vector at the (t+1)-th layer is generated by
[Equation]
where γ ( t ) is a learnable scalar parameter.
Since N ( v ) refers to one-hop neighbors of v , one can stack multiple layers to handle more than one-hop neighborhood.
Note that, to ensure the injectivity in the feature aggregation in the presence of structural coefﬁcients, we add 1 into the ﬁrst and second terms in Eq.
This design is critical for guaranteeing the expressiveness of GraphSNN beyond 1-WL, as will be discussed in the proofs of the lemmas and Theorem 4 later.
Expressiveness analysis.
We ﬁrst generalise the result of universal functions over multisets (Xu et al., 2019) to universal functions over pairs of multisets since Eq.
5 involves not only node features but
[Table]
also structural coefﬁcients.
Assume that H , A and W are countable sets where H is a node feature space, A is a structural coefﬁcient space, and
W =
{ A ij h i |
A ij ∈ A , h i ∈ H} .
Let H and W be two multisets containing elements from H and W , respectively, and | H | = | W | .
We can prove Lemma 1,
Lemma 2 and Theorem 4 below, where the proof details are provided in Appendix C.
Lemma 1.
There exists a function f s.t.
π ( H, W ) =
(cid:80) h ∈ H,w ∈ W f ( h, w ) is unique for any distinct pair of multisets ( H , W ).
Then, the injectiveness of π ( H, W ) can be extended to π (cid:48)
( a, H, W ) as in the lemma below.
Lemma 2.
There exists a function f s.t.
π (cid:48)
( h v , H, W ) =
γf ( h v , | H | h v ) + (cid:80) h ∈ H,w ∈ W f ( h, w ) is unique for any distinct ( h v , H, W ) , where h v ∈ H , | H | h v ∈ W , and γ can be an irrational number.
Since any function over ( h v , H, W ) can be decomposed as g ( γf ( h v , | H | h v )+ (cid:80) h ∈ H,w ∈ W f ( h, w )) , similar to Xu et al.
(2019), we use a parameterized multi-layer perceptron (M LP ) to learn f and g .
The following theorem characterises the expressive power of GraphSNN.
Theorem 4.
GraphSNN is more expressive than 1-WL in testing non-isomorphic graphs.
Since GIN is as powerful as 1-WL
(Xu et al., 2019), this theorem implies that GraphSNN is more expressive than GIN, i.e., GraphSNN can map at least two different neighborhood subgraphs that correspond to the same multiset of feature vectors to different representations.
Complexity analysis.
Similar to GCN and GIN, GraphSNN is computationally efﬁcient.
The time complexity and memory complexity are linear w.r.t.
the number of edges in a graph.
Further, due to the locality of GraphSNN, the computation of aggregating feature vectors from neighborhood subgraphs at each layer can be parallelized across all vertices.
Structural coefﬁcients can be precomputed with the time
complexity O ( ml ) , where m is the number of edges and l is the maximum degree of vertices in a graph, and this computation can also be parallelized across all edges.
Table 9 in Appendix A summarizes the time and space complexities of several popular message-passing
GNNs in comparison with GraphSNN.
N UMERICAL E XPERIMENTS
In this section, we evaluate our models on node classiﬁcation and graph classiﬁcation benchmark tasks.
All the results of our models are statistically signiﬁcant at 0.05 level of signiﬁcance.
5. N ODE C LASSIFICATION
Datasets.
We use ﬁve datasets: three citation network datasets Cora, Citeseer, and Pubmed (Sen et al., 2008) for semi-supervised document classiﬁcation, one knowledge graph dataset NELL
(Carlson et al., 2010) for semi-supervised entity classiﬁcation, and one OGB dataset ogbn-arxiv from (Hu et al., 2020).
Table 10 in Appendix B contains statistics for these datasets.
Baseline methods.
We consider the popular message-passing GNNs: GCN (Kipf & Welling, 2017),
GAT (Veliˇckovi´c et al., 2017), GIN (Xu et al., 2019), and GraphSAGE (Hamilton et al., 2017).
For each of these baselines, we construct a GraphSNN M model by replacing its aggregation scheme by our aggregation scheme, which is detailed in Appendix A.
The purpose of this setup is to evaluate
[Table]
how effectively our aggregation scheme with structural coefﬁcients can learn representations for vertices, compared with the standard message-passing aggregation scheme.
Experimental setup.
We use the Adam optimizer (Kingma & Ba, 2015) and λ = 1 .
For ogbnarxiv, our models are trained for 500 epochs with the learning rate 0.01, dropout 0.5, hidden units 256, and γ = 0 .
For the other datasets, we use 200 epochs with the learning rate 0.001, and choose the best values for weight decay from { 0 . 001 , 0 . 002 , ..., 0 . 009 } and hidden units from { 64 , 128 , 256 , 512 } .
For γ and dropout at each layer, the best value for each model in each dataset is selected from { 0 . 1 , 0 . 2 , ..., 0 . 6 } .
GraphSNN GAT uses the attention dropout 0.6 and 8 multi-attention heads.
GraphSNN GraphSAGE uses the neighborhood sample size 25 with the mean aggregation.
We consider two settings of data splits for all datasets except for ogbn-arxiv: (1) the standard splits in
Kipf & Welling (2017), i.e., 20 nodes from each class for training, 500 nodes for validation andnodes for testing, for which the results are presented in Table 1; (2) the random splits in Pei et al.
(2020), i.e., randomly splitting nodes into 60%, 20% and 20% for training, validation and testing, respectively, for which the results are presented in Table 13 in Appendix B. For ogbn-arxiv, we follow
Hu et al.
(2020) to use a time-based data split based on publication dates.
5. G RAPH C LASSIFICATION
We evaluate GraphSNN from three aspects: (1) small standard graph datasets, (2) large graph datasets and (3) comparison with GNNs that are go beyond 1-WL.
Experiments on small graphs.
We use eight datasets from two categories: (1) bioinformatics datasets: MUTAG, PTC-MR, COX2, BZR, PROTEINS, and D&D (Debnath et al., 1991;
Kriege et al., 2016; Wale et al., 2008; Shervashidze et al., 2011; Sutherland et al., 2003; Borgwardt & Kriegel, 2005); (2) social network datasets: IMDB-B and RDT-M5K (Yanardag & Vishwanathan, 2015).
Table 11 in Appendix B contains statistics for these small graph datasets.
We compare against 12 baselines: (1) Graph kernel based methods: WL subtree kernel (Shervashidze et al., 2011), RetGK (Zhang et al., 2018b), GNTK (Du et al., 2019), P-WL (Rieck et al., 2019),
WL-PM (Nikolentzos et al., 2017), WWL (Togninalli et al., 2019) and FGW (Titouan et al., 2019); (2) GNN based methods: PATCHY-SAN (Niepert et al., 2016), DGCNN (Zhang et al., 2018a),
CapsGNN (Xinyi & Chen, 2018), GIN (Xu et al., 2019), and GraphSAGE (Hamilton et al., 2017).
Both the standard stratiﬁed splits (Xu et al., 2019) and the random splits are considered.
We use 10-fold cross validation with 90% training and 10 % testing, and report the best mean accuracy.
For both settings, we use the Adam optimizer (Kingma & Ba, 2015), batch size 64, hidden dimension 64, weight decay of 0 . 009 , a 2-layer MLP with batch normalization, 500 epochs and dropout of 0 . 6 , and γ = 0 .
1 over all datasets.
The readout function as in (Xu et al., 2019) is used which concatenates representations of all layers to obtain a ﬁnal graph representation.
For the standard stratiﬁed splits, we use the learning rate 0 .
009 over all datasets.
For the random splits, we use the learning rate 0 .for
MUTAG and RDT-M5K, and 0 .
007 for the other datasets.
Table 2 presents the results.
[Table] [Table]
Experiments on large graphs.
We use ﬁve large graph datasets from Open Graph Benchmark (OGB)
Hu et al.
(2020), including four molecular graph datasets (ogbg-molhiv, ogbg-moltox21, ogbg-moltoxcast and ogb-molpcba) and one protein-protein association network (ogbg-ppa).
Table 12 in Appendix B contains statistics for these large graph datasets.
We compare against the following methods that have reported the results on the above OGB datasets:
GIN and GIN+VN (Hu et al., 2020), GSN (Bouritsas et al., 2020), PNA (Corso et al., 2020), ID-GNNs (You et al., 2021) and Deep LRP (Chen et al., 2020b).
In addition to the original model of GraphSNN, we also consider a variant, denoted as GraphSNN+VN, which performs the message passing over augmented graphs with virtual nodes in GraphSNN (Hu et al., 2020; Ishiguro et al., 2019).
We follow the same experiment setup as in Hu et al.
We use the Adam optimizer with learning rate 0.001, batch size 32, dropout 0.5 and 100 epochs for all datasets.
GraphSNN uses a 8-layer MLP with embedding dimension 512 for ogbg-moltoxcast and ogbg-moltox21, while GraphSNN+VN has the embedding dimensions 300 and 256, and 8-layer and 5-layer MLPs for ogbg-moltoxcast and ogbg-moltox21, respectively.
For ogbg-molhiv, ogbg-molpcba and ogbg-ppa, both GraphSNN and
GraphSNN+VN use a 5-layer MLP and embedding dimension 200.
Table 3 shows the results for the classiﬁcation accuracy.
Table 15 in Appendix B shows the results for the running time of the prepocessing step.
Comparison with GNNs beyond 1-WL.
We compare GraphSNN with the other GNNs that are more expressive than 1-WL, including: GSN (Bouritsas et al., 2020), ID-GNNs (You et al., 2021) and k-WL GNN (Morris et al., 2019).
We use the same experimental setup as in (Xu et al., 2019;
Bouritsas et al., 2020; Maron et al., 2019).
Table 4 shows the results.
5. A BLATION S TUDY
We perform an ablation study to analyze the effect of λ values on model performance.
Tables 5 andshow that λ
= 1 yields the highest performance for node classiﬁcation, while λ = 2 is the best for graph classiﬁcation.
This reﬂects a critical point - different classes of structure information are needed by different graph learning tasks.
λ = 1 captures local density, e.g., two overlap subgraphs may considerably vary in the number of vertices but their local density can be very close. Our experiments
[Table] [Table]
show that injecting such local density helps improve the performance of node classiﬁcation.
λ =captures local similarity, i [Table] e., how similar two overlap subgraphs are.
Two overlap subgraphs that considerably differ in the number of vertices would have very different structural coefﬁcients.
Since graph classiﬁcation requires to compare the similarity of two graphs, λ = 2 is thus the best.
5. O VERSMOOTHING A NALYSIS
We analyse the impact of model depth (number of layers) on node classiﬁcation performance.
In addition to GCN and GraphSNN GCN , we also compare these models with a residual connection (i.e., GCN+residual and GraphSNN GCN +residual).
We evaluate all the models on Cora dataset using the standard splits and same hyperparameters as in Section 5.1.
Table 7 shows the results.
When increasing the model depth, GraphSNN GCN performs consistently better than GCN at each layer.
This is because structural coefﬁcients capture structural connectivity between a target vertex and its neighbors.
Thus, a neighbor whose structural connectivity is weak would pass little messages to the target vertex, whereas a neighbor whose structural connectivity is strong would pass a strong message to the target vertex.
GraphSNN helps alleviate the oversmoothing issue even in the presence of residual connections.
Further results of the oversmoothing analysis are provided in Appendix B
[Table]
C ONCLUSIONS
In this paper, we have introduced a GNN framework, which enables a general way of injecting structural information into a message-passing aggregation scheme.
We have also introduced a novel
GNN model, GraphSNN, for graph learning [Table] and prove that GraphSNN is more expressive than 1- WL in distinguishing graph structures.
It is shown that GraphSNN consistently outperforms all the state-of-the-art approaches in both node classiﬁcation and graph classiﬁcation benchmark tasks.
A PPENDIX A. C ONNECTIONS TO P REVIOUS W ORK
In the following, we discuss how our framework generalizes the existing message-passing GNNs in the literature such as GCN (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017), GAT (Veliˇckovi´c et al., 2017) and
GIN (Xu et al., 2019) as special cases.
Table 8 presents the local aggregation schemes used by these existing GNN models.
They differ from each other w.r.t.
the way of aggregating feature vectors in a neighborhood and how they are combined with the current vertex’s feature itself, i.e., summation or concatenation.
Here, α vu is an attention coefﬁcient capturing the importance of a neighbor in GAT, (cid:15) is a learnable or ﬁxed scalar parameter used in GIN, W is a learnable weight matrix and σ is a non-linear activation function, such as ReLU.
[Table]
[Equation]
GraphSAGE
[Equation]
GAT
[Equation]
GIN
[Equation]
C OMPLEXITY A NALYSIS
Table 9 summarizes the time and space complexities of several popular message-passing GNNs and
GraphSNN, where n and m are the numbers of vertices and edges in a graph, respectively, k refers to the number of layers, f and d are the dimensions of input and output feature vectors, respectively, a is the number of attention heads used in GAT, and s is the number of neighbors sampled for each node at each layer in GraphSAGE.
[Table]
F ORMULATION OF G RAPH GNN M
For each of these message-passing GNNs, denoted as M , we construct a variant GraphSNN M by replacing its existing aggregation scheme by our aggregation scheme with structural coefﬁcients as formulated in Eq.
These variants are used in our experiments for node classiﬁcation benchmark tasks (see Section 5.1) in order to evaluate how our aggregation scheme with structural coefﬁcients can improve performance, compared with their standard message-passing aggregation schemes.
Below are the details of these variants.
GCN and GraphSNN GCN
Graph Convolutional Network (GCN) (Kipf & Welling, 2017) applies a normalized mean aggregation to combine the feature vector of a node v with the feature vectors in its neighborhood N ( v ) :
[Equation]
(cid:112)
[Equation] [Equation]
|N ( u ) ||N ( v ) | is a normalization constant for the edge ( v, u ) , which originates from the normalized adjacency matrix D − 1 / 2 AD − 1 / 2 .
W ( t ) is a trainable weight matrix and σ is a non-linear activation function such as ReLU.
We generalise GCN to a model under the GMP framework, namely
GraphSNN GCN , to improve the expressive power of GCN.
We ﬁrst construct a normalized structural coefﬁcient matrix ˜ A .
Formally, each neural layer of GraphSNN GCN may then be expressed as:
[Equation]
GraphSAGE and GraphSNN GraphSAGE
GraphSAGE (Hamilton et al., 2017) learns aggregation functions to induce new node feature vectors by sampling and aggregating features from a node’s local neighborhood.
GraphSAGE has considered three different aggregation functions such as mean aggregator, LSTM aggregator and pooling aggregator.
In our work, we mainly focus on the mean aggregator that, for each vertex v , takes the mean of the feature vectors of the nodes in its neighborhood and concatenates it with the feature vector of v as shown below:
[Equation]
(8) where W ( t ) is a learnable weight matrix, and σ represents a non-linear activation function.
We also generalise GraphSNN to a model under the GMP framework, namely GraphSNN GraphSAGE .
This model ﬁrst takes a mean aggregation of the feature vectors in the neighborhood N ( v ) and then concatenates it with the feature vector of v itself in the following manner:
[Equation]
GAT and GraphSNN GAT
Graph Attention Network (GAT) (Veliˇckovi´c et al., 2017) linearly transforms the input feature vectors and performs a weighted sum of the feature vectors for vertices in a neighborhood after the transformation.
GAT computes attention weights α ( t ) vu using an attention mechanism and aggregates the feature vectors in a neighborhood as follows:
[Equation]
(10) where W ( t ) is a trainable weight matrix and σ represents a non-linear activation function.
We generalise GAT to a model, called GraphSNN GAT , in the GMP framework.
Firstly, we aggregate the feature vectors based on structural coefﬁcients in our aggregation scheme, i.e., we compute
[Equation]
and
[Equation]
We then construct attention coefﬁcients α ( t ) vu on these aggregated feature vectors as follows:
[Equation]
[Equation]
where || represents the concatenation, W ( t ) is a learnabe weight matrix and a is a learnable weight vector.
After that, we aggregate the neighborhood features as follows using attention coefﬁcients.
[Equation]
where W ( t ) is a learnable weight matrix, and σ represents a non-linear activation function.
We use multi-head attention as stated in the original work Veliˇckovi´c et al.
GIN and GraphSNN GIN
Graph Isomorphism Network (GIN) (Xu et al., 2019) takes the sum aggregation over a neighborhood, followed by a 2-layer MLP.
The (cid:15) ( t +1) is a learnable parameter or ﬁxed scalar.
Each neural layer is expressed as:
[Equation]
Here, we consider one of GIN variants employed in the original paper, where the learnable parameter (cid:15) = 0 , and generalise it to GraphSNN GIN as deﬁned bwlow:
[Equation]
B. E XPERIMENTS D ATASETS
Table 10 contains the statistics for the ﬁve datasets used in our experiments for node classiﬁcation in
Section 5.1.
[Table]
#Features
[Table]
Table 11 below contains the statistics for the datasets used in our experiments on small graph classiﬁcation in Section 5.2, as well as the datasets used in an additional experiment for graph classiﬁcation following the data splits and experimental setup in (Errica et al., 2020).
The results of this additional experiment are reported under Section “Graph Classiﬁcation using Setup (Errica et al., 2020)” in Appendix B.
[Table]
Table 12 contains the statistics for the ﬁve large graph datasets from from Open Graph Benchmark (OGB)
Hu et al.
(2020), used in our experiments for large graph classiﬁcation in Section 5.2.
[Table]
N ODE C LASSIFICATION USING R ANDOM S PLITS
Following the work Pei et al.
(2020), we randomly split graph nodes into 60%, 20% and 20% for training, validation and testing, respectively.
The other hyperparameter settings are the same as in Section 5.1.
Table 13 shows the results.
We see that our models consistently outperform all of the baseline methods on all benchmark datasets.
Speciﬁcally, GraphSN GCN improves upon GCN by a margin of 1.5%, 1.7%, 1.6% and 2.4% on Cora, Citeseer, Pubmed and NELL, respectively.
GraphSN GAT improves upon GAT by 1.3%, 1.6% and 2.0% on Cora, Citeseer and Pubmed, respectively.
GraphSN
GIN improves upon GIN by 3.8%, 1.7%, 1.8% and 1.6% on Cora, Citeseer, Pubmed and NELL, respectively.
GraphSN GraphSAGE improves upon GraphSAGE by 1.3%, 1.7%, 1.1% and 2.3% on Cora, Citeseer, Pubmed and NELL, respectively.
[Table]
G RAPH C LASSIFICATION USING S ETUP (E RRICA ET AL ., 2020)
Following the data splits and experiment setup introduced in (Errica et al., 2020)
[Table]
we further evaluate our method.
The setup in (Errica et al., 2020) provides a fair performance comparison process on
GNN methods.
The evaluation process has two different phases: (1) model selection on the validation set, (2) model assessment on the test set.
Here, we have conducted experiments on four bioinformatics datasets (NCI1, PROTEINS, ENZYMES and D&D) and three social network datasets (COLLAB,
IMDB-B and REDDIT-5k) with node features.
The results of the baseline, DGCNN and GIN are taken from the paper (Errica et al., 2020).
Table 14 shows the results.
[Table]
G RAPH C LASSIFICATION ON OGB G RAPH D ATASETS
Table 15 shows the results for the running time of the prepocessing step in our method GraphSNN for large graph datasets (averaged over 5 runs).
Note that the preprocessing step can be parallellized efﬁciently at the node level.
The CPU time shows the total preprocessing time of a dataset in which
[Table]
O VERSMOOTHING A NALYSIS
We have also conducted further experiments to analyze the effectiveness of our method in alleviating the over-smoothing issue.
We compare GIN (i
[Table]
e., a spatial GNN), DFNets (Wijesinghe & Wang, 2019) (i.e., a spectral GNN), GraphSNN GIN and GraphSNN GCN .
For a fair comparison, we remove the dense-net architecture of DFNets and use the same hyperparameters as in the original paper.
We evaluate all models over the cora dataset using the standard splits.
The classiﬁcation accuracy is averaged over 10 runs on node-classiﬁcation.
[Table]
GraphSNN can alleviate over-smoothing is because structural coefﬁcients capture structural connectivity between a target vertex and its neighbors.
Thus, a neighbor whose structural connectivity is weak would pass little message to the target vertex, whereas a neighbor whose structural connectivity is strong would pass strong message to the target vertex.
Figure 4 shows the results of GCN and GraphSNN GCN on the datasets Cora, Citeseer and Pubmed, in terms of classiﬁcation accuracy averaged over 10 runs in the setting of standard splits.
[Figure]
A BLATION S TUDY WITH A UGMENTED N ODE F EATURES
We consider an experimental evaluation setup called BL, which serves as the baseline for all experiments in this ablation study.
In the setting of BL, the AGGREGATE I in GraphSNN is set to 1.
Then, different variants of BL consider different local substructure counts as additional node features.
This allows us to analyse what types of local substructures our proposed architecture can distinguish
[Figure] [Table] [Table] There are ﬁve variants of BL being considered in the ablation study:
(1) BL SC : Setting AGGREGATION I of GraphSNN to 1 and keeping structural coefﬁcients for neighbors.
(2) BL cliqueNF : Setting AGGREGATION I of GraphSNN to 1, removing structural coefﬁcients for neighbors, and adding additional node features (triangle and 4-clique counts) into the original feature vectors.
(3) BL cliqueSC + NF : Setting AGGREGATION I of GraphSNN to 1, keeping structural coefﬁcients for neighbors, and adding additional node features (triangle and 4-clique counts) into the original feature vectors.
(4) BL cycleNF : Setting AGGREGATION I of GraphSNN to 1, removing structural coefﬁcients for neighbors, and adding additional node features (cycle counts) into the original feature vectors.
(5) BL cycleSC + NF : Setting AGGREGATION I of GraphSNN to 1, keeping structural coefﬁcients for neighbors, and adding additional node features (cycle counts) into the original feature vectors.
We compare GraphSNN with GSN-v (Bouritsas et al., 2020), BL cliqueNF , BL SC , and BL cliqueSC + NF to analyze how our proposed architecture relates to the models with triangle and 4 clique counts as additional node features.
Similarly, we compare GraphSNN with ID-GNNs (You et al., 2021),
BL cycleNF , BL SC , and BL cycleSC + NF to analyze how our proposed architecture relates to the models with cycle counts as additional node features.
We concatenate the counts of cycles with lengthto 4 starting and ending at the given source node with its original feature vector as in (You et al., 2021).
Table 17 and Table 18 show the experimental results.
As AGGREGATE I is set to 1 in the setting of BL, the performance gap between BL NF and BL SC + NF reﬂects the effectiveness of structural coefﬁcients on enhancing relational inference between a target vertex and its neighbors.
The performance gap between BL SC and GraphSNN above shows the effectiveness of AGGREGATE I in our proposed model GraphSNN.
Furthermore, BL SC + NF consistently performs best since we incorporate both extra node features and structural coefﬁcients into the feature aggregation.
There is a small performance gap between BL SC + NF and GraphSNN due to augmented node features that can capture additional structural information that cannot be captured using structural coefﬁcients.
C. P ROOFS FOR L EMMAS AND T HEOREMS Proof for Theorem
Theorem 1.
The following statements are true: (a) If S i (cid:39) subgraph S j , then S i (cid:39) overlap S j ; but not vice versa; (b) If S i (cid:39) overlap S j , then S i (cid:39) subtree S j ; but not vice versa.
Proof.
In the following, we prove the statements in this theorem one by one.
For Statement (a), by S i (cid:39) subgraph S j and Deﬁnition 1, we know that there exists a bijective mapping g (cid:48) :
˜ N ( i ) →
˜ N ( j ) such that for the vertex i and any vertex v (cid:48) ∈
N ( i ) , i and v (cid:48) are adjacent in S
i iff j = g
( i ) and u (cid:48) =
g ( v (cid:48) ) are adjacent in S j , and
h i =
h j and h v (cid:48)
=
h u (cid:48) , where g is a bijective mapping between S i and S j as deﬁned by Deﬁnition 1.
Then for each pair of overlap subgraphs S iv (cid:48) and S ju (cid:48) , we can further extend g (cid:48) along g on S iv (cid:48) and S ju (cid:48) .
That is, g (cid:48) ( v ) =
u iff g ( v ) =
u .
If v in
S iv (cid:48) , by the deﬁnition of overlap subgraph, v must either be i or a neighbor of i .
Hence u = g (cid:48) ( v ) in this case must be either j or a neighbor of j .
By the deﬁnition of g and the fact that g (cid:48) ( v ) =
u iff g ( v ) =
u , we know that for any two vertices v 1 and v 2 in S iv (cid:48) , they are adjacent in S iv (cid:48)
iff their corresponding vertices g (cid:48) ( v 1 ) and g (cid:48) ( v 2 ) are adjacent in S ju (cid:48) and their corresponding feature vectors are indistinguishable, i.e, S iv (cid:48)
(cid:39) subgraph S ju (cid:48) for any v (cid:48) ∈
N ( i ) and g ( v (cid:48) )
=
u (cid:48) .
Conversely, if
S i (cid:39) overlap S j , then it is possible that S i (cid:54)(cid:39) subgraph S j as shown by the two graphs in Figure 2(a).
For Statement (b), if S i (cid:39) overlap S j , then to prove S i (cid:39) subtree S j we need to show that there exists a bijective mapping g : ˜ N ( i ) →
˜ N ( j ) such that g ( i ) =
j and, for any v (cid:48) ∈
˜ N ( i ) and g ( v (cid:48) )
=
u (cid:48) , the feature vectors of v (cid:48) and u (cid:48) are indistinguishable, i.e., h v (cid:48)
=
h u (cid:48) .
By Def. 2, we can ﬁnd a bijective mapping g (cid:48) :
˜ N ( i ) →
˜ N ( j ) such that g (cid:48)
( i ) =
j and, for any v (cid:48) ∈
N ( i ) and g (cid:48) ( v (cid:48) )
=
u (cid:48) , S iv (cid:48) and
S ju (cid:48) are subgraph-isomorphic.
This implies that g (cid:48) cannot distinguish the feature vectors of v (cid:48) and u (cid:48) for any v (cid:
) ∈
˜ N ( i ) and g ( v (cid:48) )
=
u (cid:48) .
Similarly, the converse does not necessarily hold and one counterexample is the set of graphs as shown in Figure 2(b) which are subtree-isomorphic but not overlap-isomorphic.
Proof for Theorem
Theorem 2.
Let M be a GNN.
M is as powerful as 1-WL in distinguishing non-isomorphic graphs if M has a sufﬁcient number of layers and each layer can map any S i and S j in S into two different embeddings (i.e., ζ ( S i ) (cid:54) =
ζ ( S j ) )
if and only if S i (cid:54)(cid:39) subtree S j .
Proof.
We ﬁrst show that, for any two graphs G 1 and G 2 , if they can be distinguished by 1-WL, then they must be distinguishable by such a GNN M as well.
Suppose that 1-WL takes k iterations to distinguish G 1 and G 2 , i.e., 1-WL yields the same multiset of node labels on G 1 and G 2 in the iterations from 0 to k -1, but two different multisets of node labels on G 1 and G 2 in the k-th iteration.
To derive a contradiction, we assume that a GNN M that satisﬁes the above two conditions cannot distinguish G 1 and G 2 in the iterations from 0 to k .
Since 1-WL can distinguish G 1 and G 2 in the k-th iteration, it means that there must exist two neighborhood subgraphs, say S i and S j , which correspond to two different multisets of node labels on G 1 and G 2 at the k-th iteration.
These two different multisets of node labels correspond to two different multisets of feature vectors in their neighborhoods, i.e., {{ h v | v ∈ N ( i ) }} (cid:54) =
{{ h u | u ∈ N ( j ) }} .
By Def. 3, we know that S
i (cid:54)(cid:39) subtree S j .
Then this means that ζ ( S i ) (cid:54) = ζ ( S j ) , which contradicts the assumption that M cannot distinguish Gand G 2 in the iteration k .
Now, we show the other direction that, for any two graphs G
=
( V 1 , E 1 ) and G 2 =
( V 2 , E 2 ) , if they can be distinguished by such a GNN M , then they must be distinguishable by 1-WL.
Similarly, suppose that at the k-th iteration, M maps the neighborhood subgraphs of these two graphs into two different multisets of node embeddings, i.e., {{ ζ ( S v ) | v ∈ V 1 }} (cid:54) = {{ ζ ( S u ) | v ∈ V 2 }} .
This is means that we can ﬁnd
at least two different neighborhood subgraphs S i and S j such that ζ ( S i ) (cid:54) =
ζ ( S j ) .
For such neighborhood subgraphs S i and S j , we know that S
i (cid:54)(cid:39) subtree S j .
Then this means that
S i and S j correspond to either h
i (cid:54) =
h j or {{ h v | v ∈ N ( i ) }} (cid:54) =
{{ h u | u ∈ N ( j ) }} , which can be relabeled by 1-WL into two different new labels.
Thus, 1-WL can also distinguish such neighborhood subgraphs, and accordingly distinguish G 1 and G 2 .
The proof is completed.
Proof for Theorem
Theorem 3.
Let M be a GNN whose aggregation scheme Φ is deﬁned by Eq.
1-Eq.
M is strictly more expressive than 1-WL in distinguishing non-isomorphic graphs if M has a sufﬁcient number of layers and also satisﬁes the following conditions:
(1) M can distinguish at least two neighborhood subgraphs S i and S j with S i (cid:39) subtree S j , S i (cid:54)(cid:39) subgraph S j and {{ ˜ A iv (cid:48) | v (cid:
) ∈
N ( i ) }} (cid:54) = {{ ˜ A ju (cid:48) | u (cid:48)
∈ N ( j ) }} ; (2) Φ (cid:16) h ( t ) v , {{ h ( t ) u | u ∈ N ( v ) }} , {{ ( ˜ A vu , h ( t ) u ) | u ∈ N ( v ) }} (cid:17) is injective.
Proof.
We prove this theorem in two steps.
First, we prove that a GNN M satisfying the above conditions can distinguish any two graphs that are distinguishable by 1-WL by contradiction.
Assume that there exist two graphs G 1 and G 2 which can be distinguished by 1-WL but cannot be distinguished by M .
Further, suppose that 1-WL cannot distinguish these two graphs in the iterations from 0 to k -1, but can distinguish them in the k-th iteration.
Then, there must exist two neighborhood subgraphs
S i and S j whose neighboring nodes correspond to two different multisets of node labels at the k-th iteration, i.e., {{ h ( k ) v | v ∈ N ( i ) }} (cid:54) =
{{ h ( k ) u | u ∈ N ( j ) }} .
By the above condition (2), we know that Φ is injective.
Thus, for S i and S j , Φ would yield two different feature vectors at the k-th iteration.
This means that M can also distinguish G 1 and G 2 , which contradicts the assumption.
Our proof in the ﬁrst step is done.
For the second step, we can prove that there exist at least two graphs that can be distinguished by M but cannot be distinguished by 1-WL.
Figure 1 presents two of such graphs.
Proof for Theorem
We consider that, for each vertex in a graph, its node features are from a countable set; similarly, for each pair of adjacent vertices in a graph, its structural coefﬁcient is also from a countable set.
Assume that H , A and W are countable sets where H is a node feature space, A is a structural coefﬁcient space, and
W =
{ A ij h i |
A ij ∈ A , h i ∈ H} .
Let H and W be two multisets containing elements from H and W , respectively, and | H | = | W | .
Lemma 1.
There exists a function f s.t.
π ( H, W ) =
(cid:80) h ∈ H,w ∈ W f ( h, w ) is unique for any distinct pair of multisets ( H , W ).
Proof.
Since H and W are countable, there must exist two functions ψ 1 : H → N odd mapping h ∈ H to odd natural numbers and ψ 2 : W → N even mapping w ∈ W to even natural numbers.
Further, for any pair of multisets ( H , W ), since the cardinality of H and W is bounded, there must exist a number N ∈ N such that | H | < N and | W | < N .
Thus, we can ﬁnd a prime number P > 2 N .
Then we have a mapping f as f ( h, w ) =
P − ψ 1 ( h ) + P − ψ 2 ( w ) such that (cid:80) h ∈ H,w ∈ W f ( h, w ) is unique for each distinct pair of ( H, W ) .
Lemma 2.
There exists a function f s.t.
π (cid:48)
( h v , H, W ) =
γf ( h v , | H | h v ) + (cid:80) h ∈ H,w ∈ W f ( h, w ) is unique for any distinct ( h v , H, W ) , where h v ∈ H , | H | h v ∈ W , and γ can be an irrational number.
Proof.
As h v ∈ H and | H | h v ∈ W , we may have f ( h v , | H | h v ) =
P − ψ 1 ( h v ) + P − ψ 1 ( | H | h v ) where ψ 1 : H → N odd and ψ 2 : W → N even as deﬁned in the proof for Lemma 1.
Let ( h v 1 , H 1 , W 1 ) and ( h v 2 , H 2 , W 2 ) be two different tuples.
Then, there are two cases:
(1) When h v 1 = h v 2 but ( H 1 , W 1 ) (cid:54) =
( H 2 , W 2 ) , by Lemma 1, we know that (cid:80) h ∈ H 1 ,w ∈ W 1 f
( h, w ) (cid:54) =
(cid:80) h ∈ H 2 ,w ∈ W 2 f
( h, w ) .
Thus, π (cid:48) ( h v 1 , H 1 , W 1 ) (cid:54) =
π (cid:48)
( h v 2 , H 2 , W 2 ) .
(2) When h v 1 (cid:54) =
h v 2 , we prove π (cid:48)
( h v 1 , H 1 , W 1 ) (cid:54) =
π (cid:48)
( h v 2 , H 2 , W 2 ) by contradiction.
Assume that π (cid:48) ( h v 1 , H 1 , W 1 ) =
π (cid:48)
( h v 2 , H 2 , W 2 ) .
Then, we have:
[Equation]
[Equation]
[Equation]
This gives us the following equation:
[Equation]
When γ is an irrational number, L.H.S. of the above equation is irrational but R.H.S. is rational.
There is a contradiction.
Thus, π (cid:48) ( h v 1 , H 1 , W 1 ) (cid:54) =
π (cid:48)
( h v 2 , H 2 , W 2 ) .
Based on Lemma 1 and Lemma 2, we can prove the following theorem.
Theorem 4.
GraphSNN is more expressive than 1-WL in testing non-isomorphic graphs.
Proof.
We prove this theorem by showing that GraphSNN is a GNN satisfying the conditions stated in Theorem 3.
For the ﬁrst condition, consider the two graphs shown in Figure 1.
GraphSNN can distinguish these two neighborhood subgraphs S i and S j with {{ ˜ A iv
[Equation]
N
[Equation] [Equation]
ju
[Equation]
|
[Equation]
∈
N ( j ) }} .
For the second condition, by Lemmas 1 and 2 as well as the fact that MLP as a universal approximator (Xu et al., 2019) can be used to model and learn the functions f and g , we know that
GraphSNN also satisﬁes this condition.