A Practical, Progressively-Expressive GNN
Abstract
Message passing neural networks (MPNNs) have become a dominant flavor of graph neural networks (GNNs) in recent years.
Yet, MPNNs come with notable limitations; namely, they are at most as powerful as the 1-dimensional
Weisfeiler-Leman (1-WL) test in distinguishing graphs in a graph isomorphism testing framework.
To this end, researchers have drawn inspiration from the k -WL hierarchy to develop more expressive GNNs.
However, current k -WLequivalent GNNs are not practical for even small values of k , as k -WL becomes combinatorially more complex as k grows.
At the same time, several works have found great empirical success in graph learning tasks without highly expressive models, implying that chasing expressiveness with a “coarse-grained ruler” of expressivity like k -WL is often unneeded in practical tasks.
To truly understand the expressiveness-complexity tradeoff, one desires a more “fine-grained ruler,” which can more gradually increase expressiveness.
Our work puts forth such a proposal: Namely, we first propose the ( k, c )( ≤ ) -S ET WL hierarchy with greatly reduced complexity from k -WL, achieved by moving from k -tuples of nodes to sets with ≤ k nodes defined over ≤ c connected components in the induced original graph.
We show favorable theoretical results for this model in relation to k -WL, and concretize it via ( k, c )( ≤ ) -S ET GNN, which is as expressive as ( k, c )( ≤ ) -S ET WL.
Our model is practical and progressively-expressive , increasing in power with k and c .
We demonstrate effectiveness on several benchmark datasets, achieving several state-of-the-art results with runtime and mem
- ory usage applicable to practical graphs.
We open source our implementation at https://github.com/LingxiaoShawn/KCSetGNN.
Introduction
In recent years, graph neural networks (GNNs) have gained considerable attention [53, 55] for their ability to tackle various node-level [30, 51], link-level [61, 46] and graph-level [8, 35] learning tasks, given their ability to learn rich representations for complex graph-structured data.
The common template for designing GNNs follows the message passing paradigm; these so-called message-passing neural networks (MPNNs) are built by stacking layers which encompass feature transformation and aggregation operations over the input graph [34, 22].
Despite their advantages, MPNNs have several limitations including oversmoothing [62, 11, 42], oversquashing [2, 50], inability to distinguish node identities [57] and positions [58], and expressive power [56].
Since Xu et al.’s
[56] seminal work showed that MPNNs are at most as powerful as the first-order
Weisfeiler-Leman (1-WL) test in the graph isomorphism (GI) testing framework, there have been several follow-up works on improving the understanding of GNN expressiveness [3, 13].
In re-
1, 2, 2, 3, 3, 1, 1, 1, 2, 4, k=2sets k=1sets k=3sets 1,2, 1,2,      1,2, 1,3, 1,3, 2,3, 2,3, 3,4, 2,4,5 c=1-connected sets c=2-
[Figure]
sets edges in (3,1) super-graph edges in (3,2) super-graph  3 e.g. graph extract k-node subgraphs 
GNN
GNN GNN
GNN init.
“color”vectors e.g., (k=3, c=2)-bipartite super-graph
B i d i r ec t i ona l b i p a r t i t e G
NN l a y e r …
B i d i r ec t i ona l b i p a r t i t e G
NN l a y e r forward msg.
pass.
backward msg.
pass.
graphemb.
community proposed many GNN models to overcome such limitations [47, 4, 63]
Several of these aim to design powerful higher-order GNNs which are increasingly expressive [40, 36, 28, 8] by inspiration from the k -WL hierarchy [49].
A key limitation of such higher-order models that reach beyond 1-WL is their poor scalability; in fact, these models can only be applied to small graphs with small k in practice [35, 36, 40] due to combinatorial growth in complexity.
On the other hand lies an open question on whether practical graph learning tasks indeed need such complex, and extremely expressive GNN models.
Historically, Babai et al.
[5] showed that almost all graphs on n nodes can be distinguished by 1WL.
In other contexts like node classification, researchers have encountered superior generalization performance with graph-augmented multi-layer perceptron (GA-MLP) models [45, 32] compared to MPNNs, despite the former being strictly less expressive than the latter [12].
Considering that increased model complexity has negative implications in terms of overfitting and generalization [25], it is worth re-evaluating continued efforts to pursue maximally expressive GNNs in practice.
Ideally, one could study these models’ generalization performance on various real-world tasks by increasing k (expressiveness) in the k -WL hierarchy.
Yet, given the impracticality of this too-coarse “ruler” of expressiveness, which becomes infeasible beyond k> 3 , one desires a more fine-grained “ruler”, that is, a new hierarchy whose expressiveness grows more gradually.
Such a hierarchy could enable us to gradually build more expressive models which admit improved scaling, and avoid unnecessary leaps in model complexity for tasks which do not require them, guarding against overfitting.
Present Work.
In this paper, we propose such a hierarchy, and an associated practical progressivelyexpressive GNN model , called ( k, c )( ≤ ) -S ET GNN, whose expressiveness can be modulated through k and c .
In a nutshell, we take inspiration from k -WL, yet achieve practicality through three design ideas which simplify key bottlenecks of scaling k -WL:
First , we move away from k -tuples of the original k -WL to k -multisets (unordered), and then to k ( ≤ ) -sets.
We demonstrate that these steps drastically reduce the number nodes in the k -WL graph while retaining significant expressiveness.
(Sec.s 3.2 − 3.3)
Second , by considering the underlying sparsity of an input graph, we reduce scope to k, c ( ≤ ) -sets that consist of ≤ k nodes whose induced subgraph is comprised of ≤ c ≤ k connected components.
This also yields massive reduction in the number of nodes while improving practicality; i.e. small values of c on sparse graphs can allow one to increase k well beyond 3 in practice.
(Sec. 3.4) Third , we design a super-graph architecture that consists of a sequence of k − 1 bipartite graphs over which we learn embeddings for our k, c ( ≤ ) -sets, using bidirectional message passing.
These embeddings can be pooled to yield a final graph embedding.. (Sec.s 4.1 − 4.2)
We also speed up initializing “colors” for the k, c ( ≤ ) -sets, for c > 1 , substantially reducing computation and memory.
(Sec. 4.3) Fig.
1 overviews of our proposed framework.
Experimentally, our ( k, c )( ≤ ) -S ET GNN outperforms existing state-of-the-art GNNs on simulated expressiveness as well as real-world graph-level tasks, achieving new bests on substructure counting and ZINC-12K, respectively.
We show that generalization performance reflects increasing expressiveness by k and c .
Our proposed scalable designs allow us to train models with e.g. k
=10 or c =3 with practical running time and memory requirements.
Related Work
MPNN limitations.
Given the understanding that MPNNs have expressiveness upper-bounded by 1-WL [56], several researchers investigated what else MPNNs cannot learn.
To this end, Chen et al.
[13] showed that MPNNs cannot count induced connected substructures of 3 or more nodes, while along with [13], Arvind et al.
[3] showed that MPNNs can only count star-shaped patterns.
Loukas [33] further proved several results regarding decision problems on graphs (e.g. subgraph verification, cycle detection), finding that MPNNs cannot solve these problems unless strict conditions of depth and width are met.
Moreover, Garg et al.
[19] showed that many standard MPNNs cannot compute properties such as longest or shortest cycles, diameters or existence of cliques.
Improving expressivity.
Several works aim to improve expressiveness limitations of MPNNs.
One approach is to inject features into the MPNN aggregation, motivated by Loukas [33] who showed that MPNNs can be universal approximators when nodes are sufficiently distinguishable.
Sato et al.
[48] show that injecting random features can better enable MPNNs to solve problems like minimum dominating set and maximum matching.
You et al.
[57] inject cycle counts as node features, motivated by the limitation of MPNNs not being able to count cycles.
Others proposed utilizing subgraph isomorphism counting to empower MPNNs with substructure information they cannot learn [9, 8].
Earlier work by You et al.
[58] adds positional features to distinguish node which naïve MPNN embeddings would not.
Recently, several works also propose utilizing subgraph aggregation; Zhang et al.
[60], Zhao et al.
[63] and Bevilacqua et al.
[7] propose subgraph variants of the WL test which are no less powerful than 3-WL.
Recently a following up work [18] shows that rooted subgraph based extension with 1-WL as kernel is bounded by 3-WL.
The community is yet exploring several distinct avenues in overcoming limitations: Murphy et al.
[41] propose a relational pooling mechanism which sums over all permutations of a permutation-sensitive function to achieve above-1-WL expressiveness.
Balcilar et al.
[6] generalizes spatial and spectral MPNNs and shows that instances of spectral MPNNs are more powerful than 1-WL.
Azizian et al.
[4] and Geerts et al.
[21] unify expressivity and approximation ability results for existing GNNs.
k -WL-inspired GNNs.
The k -WL test captures higher-order interactions in graph data by considering all k -tuples, i.e., size k ordered multisets defined over the set of nodes.
While highly expressive, it does not scale to practical graphs beyond a very small k ( k = 3 pushes the practical limit even for small graphs).
However, several works propose designs which can achieve k -WL in theory and strive to make them practical.
Maron et al.
[36] proposes a general class of invariant graph networks k -IGNs having exact k -WL expressivity [20], while being not scalable.
Morris et al.
have several work on k -WL and its variants like k -LWL [37], k -GNN [40], and δ - k -WL-GNN [39].
Both k -LWL and k -GNN use a variant of k -WL that only considers k -sets and are strictly weaker than k -WL but much more practical.
In our paper we claim that k ( ≤ ) -sets should be used with additional designs to keep the best expressivity while remaining practical.
The δ - k -WL-GNN works with k -tuples but sparsifies the connections among tuples by considering locality.
Qian et al.
[43] extends the k-tuple to subgraph network but has same scalability bottleneck as k-WL.
A recent concurrent work SpeqNet [38] proposes to reduce number of tuples by restricting the number of connected components of tuples’ induced subgraphs.
The idea is independently explored in our paper in Sec.3.4.
All four variants proposed by Morris et al.
do not have realizations beyond k > 3 in their experiments while we manage to reach k = 10 .
Interestingly, a recent work [29] links graph transformer with k -IGN that having better (or equal) expressivity, however is still not scalable.
A practical progressively-expressive isomorphism test: ( k, c )( ≤ ) -S ET WL
We first motivate our method ( k, c )( ≤ ) -S ET GNN from the GI testing perspective by introducing ( k, c )( ≤ ) -S ET WL.
We first introduce notation and background of k -WL (Sec. 3.1).
Next, we show how to increase the practicality of k -WL without reducing much of its expressivity, by removing node ordering (Sec. 3.2), node repetitions (Sec. 3.3), and leveraging graph sparsity (Sec. 3.4).
We then give the complexity analysis (Sec. 3.5).
We close the section with extending the idea to k -FWL which is as expressive as ( k + 1) -WL
(Sec. 3.6).
All proofs are included in Appendix A.4.
Notation: Let G =
( V ( G ) , E ( G ) , l G ) be an undirected, colored graph with nodes V ( G ) , edges
E ( G ) , and a color-labeling function l G : V ( G ) → C where C = { c 1 , ..., c d } denotes a set of d distinct colors.
Let [ n ] = { 1 , 2 , 3 , ..., n } .
Let ( · ) denote a tuple (ordered multiset), {{·}} de- note a multiset (set which allows repetition), and {·} denote a set .
We define −→ v = ( v 1 , ..., v k ) as a k -tuple, ˜ v = {{ v 1 , ..., v k }} as a k -multiset, and ˆ v = { v 1 , ..., v k } as a k -set.
Let −→ v [ x/i ] =
( v 1 , ..., v i − 1 , x, v i +1 , ..., v k ) denote replacing the i -th element in −→ v with x , and analogously for ˜ v [ x/i ] and ˆ v [ x/i ] (assume mutlisets and sets are represented with the canonical ordering).
When v i ∈ V ( G ) , let G [ −→ v ] , G [˜ v ] , G [ˆ v ] denote the induced subgraph on G with nodes inside −→ v , ˜ v , ˆ v respectively (keeping repetitions).
An isomorphism between two graphs G and G ′ is a bijective mapping p : V ( G ) → V ( G ′ ) which satisfies ∀ u, v ∈ V ( G ) , (
u, v ) ∈ E ( G )
⇐⇒ ( p ( u ) , p ( v ))
∈ E ( G ′ ) and ∀ u ∈ V ( G ) , l G ( u ) =
l G ′ ( p ( u )) .
Two graphs G, G ′ are isomorphic if there exists an isomorphism between them, which we denote as G ∼ = G ′ .
3.1 Preliminaries: the k -Weisfeiler-Leman ( k -WL) Graph Isomorphism Test
The 1-dimensional Weisfeiler-Leman ( 1 -WL) test, also known as color refinement [44] algorithm, is a widely celebrated approach to (approximately) test GI.
Although extremely fast and effective for most graphs ( 1 -WL can provide canonical forms for all but n − 1 / 7 fraction of n -vertex graphs [5]), it fails to distinguish members of many important graph families, such as regular graphs.
The more powerful k -WL algorithm first appeared in [52], and extends coloring of vertices (or 1-tuples) to that of k -tuples.
k -WL is progressively expressive with increasing k , and can distinguish any finite set of graphs given a sufficiently large k .
k -WL has many interesting connections to logic, games, and linear equations [10, 24].
Another variant of k -WL is called k -FWL (Folklore WL), such that ( k + 1) -WL is equally expressive as k -FWL [10, 23].
Our work focuses on k -WL.
k -WL iteratively recolors all n k k -tuples defined on a graph G with n nodes.
At iteration 0, each k -tuple −→ v = ( v 1 , ...,
v k ) ∈
V ( G ) k is initialized with a color as its atomic type at k ( G, −→ v ) .
Assume
G has d colors, then at k ( G, −→ v ) ∈ { 0 , 1 } 2 ( k 2 ) + kd is an ordered vector encoding.
The first (cid:0) k 2 (cid:1) entries indicate whether v i = v j , ∀ i, j, 1 ≤ i<j ≤ k , which is the node repetition information.
The second (cid:0) k 2 (cid:1) entries indicate whether ( v i ,
v j ) ∈ E ( G ) .
The last kd entries one-hot encode the initial color of v i , ∀ i, 1 ≤ i ≤ k .
Importantly, at k ( G, −→ v )
= at k ( G ′ , −→ v ′ )
if and only if v i
(cid:55)→ v ′
i is an isomorphism from G [ −→ v ] to G ′
[ −→ v ′ ] .
Let wl ( t ) k
( G, −→ v ) denote the color of k -tuple −→ v on graph G at t -th iteration of k -WL, where colors are initialized with wl (0)
k
( G, −→ v )
= at k ( G, −→ v ) .
At the t -th iteration, k -WL updates the color of each −→ v ∈ V ( G ) k according to wl
[Equation]
t +1)
k
[Equation]
Let gwl ( t )
k
( G ) denote the encoding of G at t -th iteration of k -WL.
Then,
[Equation]
Two k -tuples −→ v and −→ u are connected by an edge if |{
i ∈
[ k ] | v i =
u i }| = k − 1 , or informally if they share ( k − 1) entries.
Then, k -WL defines a super-graph S k -wl ( G ) with its nodes being all k -tuples in G , and edges defined as above.
Eq.
(1) defines the rule of color refinement on S k -wl ( G ) .
Intuitively, k -WL is akin to (but more powerful as it orders subgroups of neighbors) running 1-WL algorithm on the supergraph S k -wl ( G ) .
As t → ∞ , the color wl ( t +1) k
( G, −→ v ) converges to a stable value, denoted as wl ( ∞ ) k
( G, −→ v ) and the corresponding stable graph color denoted as gwl ( ∞ ) k
( G ) .
For two non-isomorphic graphs G, G ′ , k -WL can successfully distinguish them if gwl ( ∞ ) k ( G )
̸
= gwl ( ∞ ) k ( G ′ ) .
The expressivity of wl ( t ) k can be exactly characterized by first-order logic with counting quantifiers.
Let C tk denote all first-order formulas with at most k variables and t -depth counting quantifiers, then wl ( t ) k
( G, −→ v ) =
wl ( t ) k
( G ′ , −→ v ′ )
⇐⇒ ∀ ϕ ∈ C tk , ϕ ( G, −→ v ) =
ϕ ( G ′ , −→ v ′ ) .
Additionally, there is a t -step bijective pebble game that are equivalent to t -iteration k -WL in expressivity.
See
Appendix.A.2 for the pebble game characterization of k -WL.
Despite its power, k -WL uses all n k tuples and has O ( kn k ) complexity at each iteration.
3.2 From k -WL to k -M ULTISET WL : Removing Ordering
Our first proposed adaptation to k -WL is to remove ordering in each k -tuple, i.e. changing k -tuples to k -multisets.
This greatly reduces the number of supernodes to consider by O ( k !) times.
Let ˜ v = {{ v 1 , ..., v k }} be the corresponding multiset of tuple −→ v = ( v 1 , ..., v k ) .
We introduce a canonical order function on G , o G : V ( G ) →
[ n ] , and a corresponding indexing function o − 1 G (˜ v , i ) , which returns the i -th element of v 1 , ..., v k sorted according to o G .
Let mwl ( t ) k ( G, ˜ v ) denote the color of the k -multiset ˜ v at t -th iteration of k -M ULTISET WL, formally defined next.
At t = 0 , we initialize the color of ˜ v as mwl (0) k
( G, ˜ v ) =
HASH (cid:0) {{ at k ( G, p (˜ v ))
| p ∈ perm[k] }} (cid:1) where perm[k] 1 denotes the set of all permutation mappings of k elements.
It can be shown that mwl (0) k
( G, ˜ v ) =
mwl (0) k
( G ′ , ˜ v ′ )
if and only if G [˜ v ] and G ′ [˜ v ′ ] are isomorphic.
At t -th iteration, k -M ULTISET WL updates the color of every k -multiset by
[Equation]
Where ˜ v [ x/o − 1 G (˜ v , i )]) denotes replacing the i -th (ordered by o G ) element of the multiset with x ∈ V ( G ) .
Let S k -mwl ( G ) denote the super-graph defined by k -M ULTISET WL.
Similar to Eq.
(2), the graph level encoding is gmwl ( t ) k
( G ) =
HASH ( {{ mwl ( t ) k (cid:0) G, ˜ v (cid:1)(cid:12)(cid:12) ∀ ˜ v ∈ V ( S k -mwl ( G )) }} (cid:1) .
Interestingly, although k -M ULTISET WL has significantly fewer number of node groups than k -WL, we show it is no less powerful than k - 1 -WL in terms of distinguishing graphs, while being upper bounded by k -WL in distingushing both node groups and graphs.
Theorem 1.
Let k ≥ 1 and wl ( t ) k
( G, ˜ v ) :
= {{ wl ( t ) k
( G, p (˜ v ))
| p ∈ perm[k] }} .
For all t ∈ N and all graphs G, G ′ :
k -M ULTISET WL is upper bounded by k -WL in distinguishing multisets G, ˜ v
and
G ′ , ˜ v ′ at t -th iteration, i.e. wl ( t ) k ( G, [Equation] =
[Equation]
Theorem 2.
k -M ULTISET WL is no less powerful than ( k - 1 )-WL in distinguishing graphs: for any k ≥ 3 there exists graphs that can be distinguished by k -M ULTISET WL but not by ( k - 1 )-WL.
Theorem 2 is proved by using a variant of a series of CFI [10] graphs which cannot be distinguished by k -WL.
This theorem shows that k -M ULTISET WL is indeed very powerful and finding counter example of k -WL distinguishable graphs that cannot be distinguished by k -M ULTISET WL is very hard.
Hence we conjecture that k -M ULTISET WL may have the same expressivity as k -WL in distinguishing undirected graphs.
Additionally, the theorem also implies that k -M ULTISET WL is strictly more powerful than ( k − 1 )
-M ULTISET WL.
We next give a pebble game characterization of k -M ULTISET WL, which is named doubly bijective k -pebble game presented in Appendix.
A.3.
The game is used in the proof of Theorem 2.
Theorem 3.
k -M ULTISET WL has the same expressivity as the doubly bijective k -pebble game.
3.3 From k -M ULTISET
WL
to k ( ≤ ) -S ET WL :
Removing Repetition
Next, we propose further removing repetition inside any k -multiset, i.e. transforming k -multisets to k ( ≤ ) -sets.
We assume elements of k -multiset ˜ v and k ( ≤ ) -set ˆ v in G are sorted based on the ordering function o G , and omit o G for clarity.
Let s ( · ) transform a multiset to set by removing repeats, and let r ( · ) return a tuple with the number of repeats for each distinct element in a multiset.
Specifically, let ˆ v = s
(˜ v ) and ˆ n = r
(˜ v ) , then m :=
| ˆ v | =
| ˆ n | ∈ [ k ] denotes the number of distinct elements in k -multiset ˜ v , and ∀ i ∈ [ m ] , ˆ v
i is the i -th distinct element with ˆ n i repetitions.
Clearly there is an injective mapping between ˜ v and (ˆ v , ˆ n ) ; let f be the inverse mapping such that ˜ v = f
( s (˜ v ) , r (˜ v )) .
Equivalently, each m -set ˆ v can be mapped with a multiset of k -multisets:
ˆ v ↔ {{ ˜ v = f
(ˆ v , ˆ n ) | (cid:80) mi =
ˆ n i = k, ∀ i ˆ n i ≥ 1 }} .
Based on this relationship, we extend the connection among k -multisets to k ( ≤ ) -sets: given m 1 , m 2 ∈ [ k ] , a m 1 -set ˆ v is connected to a m 2 -set ˆ u if and only if ∃ ˆ n v , ˆ n u , f
(ˆ v , ˆ n v ) is connected with f (ˆ u , ˆ n u ) in k -M ULTISET WL.
Let S k -swl ( G ) denote the defined super-graph on G by k ( ≤ ) -S ET WL.
It can be shown that this is equivalent to either (1) (
| m 1 − m 2 | = 1) ∧
( | ˆ v ∩ ˆ u | = min( m 1 , m 2 )) or (2) ( m
=
m 2 ) ∧ ( | ˆ v ∩ ˆ u | = m 1 − 1) is true.
Notice that S k -swl ( G ) contains a sequence of k − 1 bipartite graphs with each reflecting the connections among the ( m − 1 )-sets and the m -sets.
It also contains k − 1 subgraphs, i.e. the connections among the m -sets for m = 2 , ..., k .
Later on we will show that these k − 1 subgraphs can be ignored without affecting k ( ≤ ) -S ET WL.
Let swl ( t ) k
( G, ˆ v ) denote the color of m -set ˆ v at t -th iteration of k ( ≤ ) -S ET WL.
Now we formally define k ( ≤ ) -S ET WL.
At t = 0 , we initialize the color of a m -set ˆ v ( m ∈
[ k ] ) as:
[Equation]
Clearly swl (0) k
( G, ˆ v )
= swl (0) k
( G ′ , ˆ v ′ )
if and only if G [ˆ v ] and G ′ [ˆ v ′ ] are isomorphic.
At t -th iteration, k ( ≤ ) -S ET WL updates the color of every m -set ˆ v by
[Equation]
Notice that when m = 1 and m = k , the third and second part of the hashing input is an empty multiset, respectively.
Similar to Eq.
(2), we formulate the graph level encoding as gswl ( t ) k
( G ) =
HASH ( {{ swl (
[Equation]
To characterize the relationship of their expressivity, we first extend k -M ULTISET WL on sets by defining the color of a m -set ˆ v on k -M ULTISET WL as mwl ( t ) k
( G, ˆ v ) :
= {{ mwl ( t ) k
( G, f (ˆ v , ˆ n )) (cid:12)(cid:12) (cid:80) mi
=1 ˆ v
i =
k, ∀ i ˆ n i ≥ 1 }} .
We prove that k -M ULTISET WL is at least as expressive as k ( ≤ ) -S ET WL in terms of separating node sets and graphs.
Theorem 4.
Let k ≥ 1 , then ∀ t ∈ N and all graphs G, G ′ : mwl ( t ) k
( G, ˆ v ) =
mwl ( t ) k
( G ′ , ˆ v ′ )
=
⇒ swl ( t ) k
( G, ˆ v )
=
swl ( t ) k
( G ′ , ˆ v ′ ) .
We also conjecture that k -M ULTISET WL and k ( ≤ ) -S ET WL could be equally expressive, and leave it to future work.
As a single m -set corresponds to (cid:0) k − 1 m − 1 (cid:1) k -multisets, moving from kM ULTISET WL to k ( ≤ ) -S ET WL further reduces the computational cost greatly.
3.4 From k ( ≤ ) -S ET WL to ( k, c )( ≤ ) -S ET WL : Accounting for Sparsity
Notice that for two arbitrary graphs G and G ′ with equal number of nodes, the number of k ( ≤ ) - sets and the connections among all k ( ≤ ) -sets in k ( ≤ ) -S ET WL are exactly the same, regardless of whether they are dense or sparse.
We next propose to account for the sparsity of a graph G to further reduce the complexity of k ( ≤ ) -S ET WL.
As the graph structure is encoded inside every mset, when the graph becomes sparser, there would be more sparse m -sets with a potentially large number of disconnected components.
Based on the hypothesis that the induced subgraph over a set (of nodes) with fewer disconnected components naturally contains more structural information, we propose to restrict the k ( ≤ ) -sets to be ( k, c )( ≤ ) -sets: all sets with at most k nodes and at most c connected components in its induced subgraph.
Let S k,c -swl ( G ) denote the super-graph defined by ( k, c )( ≤ ) -S ET WL, then S k,c -swl ( G ) =
S k -swl
( G )
[ { ˆ v | #components ( G [ˆ v ]) ≤ c } ] , which is the induced subgraph on the super-graph defined by k ( ≤ ) -S ET WL.
Fortunately, S k,c -swl ( G ) can be efficiently and recursively constructed based on S ( k − 1 ,c ) -swl
( G ) , and we include the construction algorithm in the Appendix A.5.
( k, c )( ≤ ) -S ET WL can be defined similarly to k ( ≤ ) -S ET WL (Eq.
(4) and Eq.
(5)), however while removing all colors of sets that do not exist on S k,c -swl ( G ) .
( k, c )( ≤ ) -S ET WL is progressively expressive with increasing k and c , and when c = k , ( k, c )( ≤ ) -
S ET WL becomes the same as k ( ≤ ) -S ET WL, as all k ( ≤ ) -sets are then considered.
Let swl ( t ) k,c ( G, ˆ v ) denote the color of a ( k, c )( ≤ ) -set
ˆ v on t -th iteration of ( k, c )( ≤ ) -S ET WL, then gswl ( t ) k,c
( G ) =
HASH ( {{ swl ( t ) k,c (cid:0) G, ˆ v (cid:1)(cid:12)(cid:12) ∀ ˆ v ∈ S k,c -swl ( G ) (cid:1) }} .
Theorem 5.
Let k ≥ 1 , then ∀ t ∈ N and all graphs G, G ′ :
(1) when 1 ≤ c 1 < c 2 ≤ k , if G, G ′ cannot be distinguished by ( k, c 2 )( ≤ ) -S ET WL , they cannot be distinguished by ( k, c 1 )( ≤ ) -S ET WL (2) when k 1 < k 2 , ∀ c ≤ k 1 , if G, G ′ cannot be distinguished by ( k 2 , c )( ≤ ) -S ET WL , they cannot be distinguished by ( k 1 , c )( ≤ ) -S ET WL
3.5 Complexity Analysis
All color refinement algorithms described above run on a super-graph; thus, their complexity at each iteration is linear to the number of supernodes and number of edges of the super-graph.
Instead of using big O notation that ignores constant factors, we compare the exact number of supernodes and edges.
Let G be the input graph with n nodes and average degree d .
For k -WL, there are n k supernodes and each has n ∗ k number of neighbors, hence S k -wl ( G ) has n k supernodes and
kn k +1 /edges.
For m ∈ [ k ] , there are (cid:0) nm (cid:1) m -sets and each connects to m number of ( m − 1) -sets.
So S k -swl has (cid:80) ki =1 (cid:0) ni (cid:1) ≤ (cid:0) nk (cid:1) n − k +1 n − 2 k +1 supernodes and (cid:80) ki
=
i (cid:0) ni (cid:1) = n (cid:80) k − 1 i =1 (cid:0) n − 1 i (cid:1) ≤ n (cid:0) n − 1 k − 1 (cid:1) n − k +1 n − 2 k +2 edges (derivation in Appendix).
Here we ignore edges within m -sets for any m ∈ [ k ] as they can be reconstructed from the bipartite connections among ( m − 1) -sets and m -sets, described in detail in
Sec. 4.1.
Consider e.g., n = 30 , k = 5 ; we get | V ( S k -wl ( G ))
| | V ( S k -swl ( G ))
| = 139 , | E ( S k -wl ( G )) | | E ( S k -swl ( G )) | = 2182 .
Directly analyzing the savings by restricting number of components is not possible without assuming a graph family.
In Sec.
5.3 we measured the scalability for ( k, c )( ≤ ) -S ET GNN with different number of components directly on sparse graphs, where ( k, c )( ≤ ) -S ET WL shares similar scalability.
3.6 Set version of k -FWL
k -FWL is a stronger GI algorithm and it has the same expressivity as k + 1 -WL [23], in this section we also demonstrate how to extend the set to k -FWL to get k ( ≤ ) -S ET FWL.
Let fwl ( t ) k
( G, −→ v ) denote the color of k -tuple −→ v at t -th iteration of k -FWL.
Then k -FWL is initialized the same as the k -WL, i.e. fwl (0) k
( G, −→ v ) =
wl (0) k
( G, −→ v ) .
At t -th iteration, k -FWL updates colors with
[Equation]
Let sfwl ( t ) k
( G, ˆ v ) denote the color of m -set ˆ v at t -th iteration of k ( ≤ ) -S ET FWL.
Then at t -th iteration it updates with
[Equation]
[Equation]
The k ( ≤ ) -S ET FWL should have better expressivity than k ( ≤ ) -S ET WL.
We show in the next section that k ( ≤ ) -S ET WL can be further improved with less computation through an intermediate step while this is nontrivial for k ( ≤ ) -S ET FWL.
We leave it to future work of studying k ( ≤ ) -S ET FWL.
A practical progressively-expressive GNN: ( k, c )( ≤ ) -S ET GNN
In this section we transform ( k, c )( ≤ ) -S ET WL to a GNN model by replacing the HASH function in
Eq.
(5) with a combination of MLP and DeepSet [59], given they are universal function approximators for vectors and sets, respectively [26, 59].
After the transformation, we propose two additional improvements (Sec. 4.2 and Sec. 4.3) to further improve scalability.
We work on vector-attributed graphs.
Let G =
( V ( G ) , E ( G ) , X ) be an undirected graph with node features x i ∈ R d ,
∀ i ∈ V ( G ) .
4.1 From
( k c )( ≤ ) -SetWL to ( k, c )( ≤ ) -SetGNN
( k, c )( ≤ ) -S ET WL defines a super-graph S k,c -swl , which aligns with Eq.
We first rewrite Eq.
(5) to reflect its connection to S k,c -swl .
For a supernode ˆ v in S k,c -swl , let N G left (ˆ v )
= { ˆ u | ˆ u ∈
S k,c -swl , ˆ u ↔ ˆ v and | ˆ u | = | ˆ v | − 1 } , and N G right ( ˆ v ) =
{ ˆ u | ˆ u ∈ S k,c -swl , ˆ u ↔ ˆ v and | ˆ u | = | ˆ v | + 1 } .
Then we can rewrite Eq.
(5) for ( k, c )( ≤ ) -S ET WL as
[Equation]
G left (ˆ v ) }} , {{ swl ( t +12) k,c
( G, ˆ u ) | ˆ u ∈ N G left (ˆ v ) }} (cid:19) (8) where swl ( t + 12 ) k,c
( G, ˆ v ) :
= {{ swl ( t ) k,c
( G, ˆ u ) | ˆ u ∈ N G right (ˆ v ) }} .
Notice that we omit HASH and apply it implicitly.
Eq.
(8) essentially splits the computation of Eq.
(5) into two steps and avoids repeated computation via caching the explicit t + 12 step.
It also implies that the connection among m -sets for any m ∈ [ k ] can be reconstructed from the bipartite graph among m -sets and ( m − 1) -sets.
Next we formulate ( k, c )( ≤ ) -S ET GNN formally.
Let h ( t ) (ˆ v ) ∈ R d t denote the vector representation of supernode ˆ v on the t -th iteration of ( k, c )( ≤ ) -S ET GNN.
For any input graph G , it initializes representations of supernodes by
[Equation]
where the BaseGNN can be any GNN model.
Theoretically the BaseGNN should be chosen to encode non-isomorphic induced subgraphs distinctly, mimicking HASH.
Based on empirical tests of several GNNs on all (11,117) possible 8 -node non-isomorphic graphs [6], and given GIN [56] is simple and fast with nearly 100% separation rate, we use GIN as our BaseGNN in experiments.
( k, c )( ≤ ) -S ET GNN iteratively updates representations of all supernodes by
[Equation]
[Table]
[Equation]
Then after T iterations, we compute the graph level encoding as
[Equation] [Equation]
where POOL can be chosen as summation.
We visualize the steps in Figure 1.
Under mild conditions, ( k, c )( ≤ ) -S ET GNN has the same expressivity as ( k, c )( ≤ ) -S ET WL.
Theorem 6.
When ( i ) BaseGNN can distinguish any non-isomorhpic graphs with at most k nodes, ( ii ) all MLPs have sufficient depth and width, and ( iii ) POOL is an injective function, then for any t ∈ N , t -layer ( k, c )( ≤ ) -S ET GNN is as expressive as ( k, c )( ≤ ) -S ET WL at the t -th iteration.
The following facts can be derived easily from Theorem 5 and Theorem 6.
Corollary 6.1.
( k, c )( ≤ ) -S ET GNN is progressively-expressive with increasing k
and
c , that is
, (1) when c 1 > c 2 , ( k, c 1 )( ≤ ) -S ET GNN is more expressive than ( k, c 2 )( ≤ ) -S ET GNN , and (2)
when k 1 > k 2 , ( k 1 , c )( ≤ ) -S ET GNN is more expressive than ( k 2 , c )( ≤ ) -S ET GNN .
4.2 Bidirectional Sequential Message Passing
The t -th layer of ( k, c )( ≤ ) -S ET GNN (Eq.
(10) and Eq.
(11)) are essentially propagating information back and forth on the super-graph S k,c -swl ( G ) , which is a sequence of k − 1 bipartite graphs, in parallel for all supernodes.
We propose to change it to bidirectional blockwise sequential message passing, which we call ( k, c )( ≤ ) -S ET GNN ∗ , defined as follows.
[Equation]
Notice that ( k, c )( ≤ ) -S ET GNN ∗ has lower memory usage, as ( k, c )( ≤ ) -S ET GNN load the complete supergraph ( k − 1 bipartites) while ( k, c )( ≤ ) -S ET GNN ∗ loads 1 out of k − 1 bipartites at a time, which is beneficial for limited-size GPUs.
What is more, for a small, finite t it is even more expressive than ( k, c )( ≤ ) -S ET GNN.
We provide both implementation of parallel and sequential message passing in the official github repository, while only report the performance of ( k, c )( ≤ ) -S ET GNN ∗ given its efficiency and better expressivity.
Theorem 7.
For any t ∈ N , the t -layer ( k, c )( ≤ ) -S ET GNN ∗ is more expressive than the t -layer ( k, c )( ≤ ) -S ET GNN .
As lim t →∞ , ( k, c )( ≤ ) -S ET GNN is as expressive as ( k, c )( ≤ ) -S ET GNN ∗ .
4.3 Improving Supernode Initialization
Next we describe how to improve the supernode initialization (Eq.
(9)) and extensively reduce computational and memory overhead for c > 1 without losing any expressivity.
We achieve this by the fact that a graph with c components can be viewed as a set of c connected components.
Formally,
Theorem 8.
Let G be a graph with c connected components C 1 , ..., C c , and G ′ be a graph also with c connected components C ′ 1 , ..., C ′ c , then G and G ′ are isomorphic if and only if ∃ p : [ c ] → [ c ] , s.t.
∀ i ∈
[ c ] , C i and C ′ p ( i ) are isomorphic.
The theorem implies that we only need to apply Eq.
(9) to all supernodes with a single component, and for any c -components ˆ v with components ˆ u 1 , ..., ˆ u c , we can get the encoding by h (0) (ˆ v ) =
DeepSet ( { h (0) (ˆ u 1 ) , ..., h (0) (ˆ u c ) } ) without passing its induced subgraph to BaseGNN.
This eliminates the heavy computation of passing a large number of induced subgraphs.
We give the algorithm of building connection between ˆ v to its single components in Appendix A.6.
[Table] [Table]
Experiments
We design experiments to answer the following questions.
Q1.
Performance:
How does ( k, c )( ≤ ) -
S ET GNN ∗ compare to SOTA expressive GNNs?
Q2.
Varying k and c : How does the progressively increasing expressiveness reflect on generalization performance?
Q3.
Computational requirements: Is ( k, c )( ≤ ) -S
ET GNN feasible on practical graphs w.r.t.
running time and memory usage?
5.1 Setup
Datasets.
To inspect the expressive power, we use four different types of simulation datasets: 1) EXP [1] contains 600 pairs of 1&2 -WL failed graphs which we split into two where graphs in each pair is assigned to two different classes; 2) SR25 [6] has 15 strongly regular graphs ( 3WL failed) with 25 nodes each, which we transform to a 15-way classification task; 3) Substructure counting (i.e. triangle, tailed triangle, star and 4-cycle) tasks on random graph dataset [13]; 4) Graph property regression (i.e. connectedness, diameter, radius) tasks on random graph dataset [14].
We also evaluate performance on two real world graph learning tasks: 5) ZINC-12K [16], and 6) QM [54] for molecular property regression.
See Table 6 in Appendix A.7 for detailed dataset statistics.
Baselines.
We use GCN [31], GIN [56], PNA ∗ [14], PPGN [35], PF-GNN [15], and GNN-AK [63] as baselines on the simulation datasets.
On ZINC-12K we also reference CIN [8] directly from literature.
Most baselines results are taken from [63].
Finally, we compare to GINE [27] on QM9.
Hyperparameter and model configurations are described in Appendix A.8.
5.2 Results
Theorem 2 shows that k -M ULTISET WL is able to distinguish CFI(k) graphs.
It also holds for k ( ≤ ) -
S ET WL as the proof doesn’t use repetitions.
We implemented the construction of CFI(k) graphs for any k. Empirically we found that ( k, c )( ≤ ) -S ET GNN ∗ is able to distinguish CFI(k) for k = 3, 4,(k>6 out of memory) .
This empirically verifies its theoretical expressivity.
Table 1 shows the performance results on all the simulation datasets.
The low expressive models such as GCN, GIN and PNA ∗ underperform on EXP and SR25, while 3-WL equivalent PPGN excels only on EXP.
Notably, ( k, c )( ≤ ) -S ET GNN ∗ achieves 100% discriminating power with any k larger than 2 and 3, and any c larger than 1 and 0, resp.
for EXP and SR25.
( k, c )( ≤ ) -S ET GNN ∗ also outperforms the baselines on all substructure counting tasks, as well as on two out of three graph property prediction tasks (except Radius) with significant gap, for relatively small values of k and c .
In Table 2, we show the train and test MAEs on substructure counting tasks for individual values of k and c .
As expected, performances improve for increasing k when c is fixed, and vice versa.
It is notable that orders of magnitude improvements on test error occur moving from k =2 to 3 for the triangle tasks as well as the star task, while a similarly large magnitude drop is obtained at k =4 for the 4-cycle task, which is expected as triangle and 4-cycle have 3 and 4 nodes respectively.
Similar observations hold for graph property tasks as well.
(See Appendix A.9)
[Table]
In addition to simulation datasets, we evaluate our ( k, c )( ≤ ) -
S ET GNN ∗ on real-world data; Table 3 shows our performance on
ZINC-12K. Our method achieves a new state-of-the-art performance, with a mean absolute error (MAE) of 0.0750, using k =5 and c =2 .
In Table 4 we show the test and validation MAE along with training loss for varying k and c for ZINC-12K. For a fixed c , validation MAE and training loss both follow a first decaying and later increasing trend with increasing k , potentially owing to the difficulty in fitting with too many sets (i.e. supernodes) and edges in the super-graph.
Similar results are shown for QM9 in Table 5.
For comparison we also show GINE performances using both 4 and 6 layers, both of which are significantly lower than ( k, c )( ≤ ) -S ET GNN ∗ .
Test MAE .
[Table] [Table]
[Figure]
[Table]
5.3 Computational requirements
[Figure]
We next investigate how increasing k and c change the computational footprint of ( k, c )( ≤ ) -S ET GNN ∗ in practice.
Fig.
2 shows that increasing these parameters expectedly increases both memory consumption (in MB in (a)) as well as runtime (in seconds per epoch in (b)).
Notably, since larger k and c increases our model’s expressivity, we observe that suitable choices for k and c allow us to practically realize these increasingly expressive models on commodity hardware.
With conservative values of c (e.g. c =1 ), we are able to consider passing messages between sets of k (e.g. 10) nodes far larger than k -WL-style higher order models can achieve ( ≤ 3).
Conclusion
Our work is motivated by the impracticality of higher-order GNN models based on the k -WL hierarchy, which make it challenging to study how much expressiveness real-world tasks truly necessitate.
To this end, we proposed ( k, c )( ≤ ) -SetWL, a more practical and progressively-expressive hierarchy with theoretical connections to k -WL and drastically lowered complexity.
We also designed and implemented a practical model ( k, c )( ≤ ) -SetGNN ( ∗ ) , expressiveness of which is gradually increased by larger k and c .
Our model achieves strong performance, including several new best results on graph-level tasks like ZINC-12K and expressiveness-relevant tasks like substructure counting, while being practically trainable on commodity hardware.
A Appendix A.1 Visualization Supergraph Connection of 2-SetWL
All nodes in 2-WL 12 13 14 15 23 24 25 34 35
All nodes in 2-SetWL                42    35    2
A Subgraph (1-egonet of node 23 & node 32) in 2-WL's graph (for offdiagnoal nodes)
A Subgraph (1-egonet of node 23) in 2SetWL's graph (for offdiagnoal nodes)
Assume the original graph has 6 nodes: [1,2,3,4,5,6], focus on 2-WL and 2-SetWL
Neighbors of
Neighbors of2-WL 2-SetWL
[Figure]
The pebble game characterization of k-FWL is appeared in [10].
We use the pebble game defined in [24] for k-WL.
Let G, G ′ be graphs and −→ v 0 , −→ v ′
be k -tuple.
Then the bijective k -pebble game
BP k ( G, −→ v 0 , G ′ , −→ v ′ 0 ) is defined as the follows.
The game is played by two players Player I and
Player II.
The game proceeds in rounds starting from initial position ( −→ v 0 , −→ v ′ 0 ) and continues to new position ( −→ v t , −→ v ′ t ) as long as after t rounds −→ v
t
[ i ] (cid:55)→ −→ v ′ t
[ i ] defines an isomorphism between G [ −→ v t ] and G ′
[ −→ v ′ t ] .
If the game hasn’t end after t rounds, Player II wins the t -step BP k
( G, −→ v 0 , G ′ , −→ v ′ 0 ), otherwise Player I wins.
At t -th each round of the game,
Player I picks up the i -th pair of pebbles with i ∈
[ k ] 2.
Player II chooses bijection f : V ( G ) → V ( G ′ ) 3.
Player I chooses x ∈ V ( G ) 4.
The new position is ( −→ v t [ x/i ] , −→ v ′ t
[ f ( x ) /i ] ) if it is a local isomorphism between.
Otherwise the game ends and Player II loses.
The game BP k ( G, −→ v 0 , G ′ , −→ v ′ 0 ) has the same expressivity as k -WL in distinguishing ( G, −→ v 0 ) and ( G ′ , −→ v ′ 0 ).
Theorem 9.
k -WL cannot distinguish ( G, −→ v 0 ) and ( G ′ , −→ v ′ 0 ) at step t , i.e. wl ( t ) k ( G, −→ v 0 )
=
wl ( t ) k
( G ′ , −→ v ′ 0 ) , if and only if Player II has a winning strategy for t -step BP k
( G, −→ v 0 , G ′ , −→ v ′ 0 ).
There is an extension of the pebble game, BP k
( G, G ′ ), without specifying the starting position.
Specifically, at the beginning of the game, Player II is first asked to provide a bijection g : V ( G ) k
→ V ( G ′ )
k .
Then Player I chooses −→ v 0 ∈ V ( G ) k .
Then Player I and Player II start to play BP k ( G, −→ v 0 , G ′ , g ( −→ v 0 ) ).
Theorem 10.
k -WL cannot distinguish G and G ′ at step t , i.e. gwl ( t )
k
( G ) =
gwl ( t ) k
( G ′ ) , if and only if Player II has a winning strategy for t -step BP k
( G, G ′ ) .
The proof of Theorem 9 and Theorem 10 can be found in Grohe’s work [24].
A.3 Doubly Bijective k -Pebble Game for k -MultisetWL
Let G, G ′ be graphs and ˜ v 0 , ˜ v ′ 0 be k -multisets.
We adapt the BP k game for k -M ULTISET WL and call it doubly bijective k -pebble game, i.e. DBP k
( G, ˜ v 0 , G ′ , ˜ v ′ 0 ) .
The DBP k ( G, ˜ v 0 , G ′ , ˜ v ′ 0 ) is defined as the follows.
The game starts at the position ( ˜ v 0 , ˜ v ′ 0 ).
Let the current position be (˜ v t , ˜ v ′ t ) .
The t -th round is played as follows.
Player II chooses a bijection h : ˜ v t → ˜ v ′ t 2.
Player I chooses the y ∈ ˜ v t 3.
Player II chooses bijection f : V ( G ) → V ( G ′ ) 4.
Player I chooses x ∈ V ( G ) 5.
The new position is (˜ v t [ x/ idx ˜ v t ( y )] , ˜ v ′ t [ f ( x ) / idx ˜ v ′ t ( h ( y ))]) and the game continues, if G [˜ v t [ x/ idx ˜ v t ( y )]] and G ′
[˜ v ′ t [ f ( x ) / idx ˜ v ′ t ( h ( y ))]] are isomorphic.
Otherwise the game ends and Player II loses.
To extend the game to the setting of no starting position, we define the corresponding game
DBP k ( G, G ′ ) .
Simliar to BP k
( G, G ′ ) , at the beginning Player II is asked to pick up a bijection g : Mlutiset ( V ( G ) k )
→ Mlutiset ( V ( G ′ ) k ) .
Then Player I picks up a ˜ v 0 and the game
DBP k ( G, ˜ v 0 , G ′ , g ( ˜ v 0 )) starts.
Theorem 3.
k -M ULTISET WL has the same expressivity as the doubly bijective k -pebble game.
Proof.
Specifically, given graph G with k -multiset ˜ v and graph G’ with k -multiset ˜ v ′ , we are going to prove that mwl ( t ) k
( G, ˜ v ) =
mwl ( t ) k ( G ′ , ˜ v ′ )
⇐⇒ Player II has a winning strategy for t -step game
DBP k ( G, ˜ v , G ′ , ˜ v ′ ) .
We now prove it by induction on t .
When t = 0 , it’s obvious that the statement is correct, as mwl (0)
k
( G, ˜ v ) =
mwl (0) k
( G ′ , ˜ v ′ ) is equivalent to G [˜ v ] and G ′
[˜ v ′ ] being isomorphic, which implies that Player II can start the game without losing.
Now assume that for t ≤ l the statement is correct.
For step t = l + 1 , let’s first prove left =
⇒
right.
mwl ( l +1) k
( G, ˜ v ) =
mwl ( l +
) k
( G ′ , ˜ v ′ ) .
By Eq.(3) we know this is equivalent to
(1) mwl ( l ) k
( G, ˜ v ) =
mwl ( l ) k ( G ′ , ˜ v ′ )
(2) ∃ bijective h : ˜ v → ˜ v ′ , ∀ y ∈ ˜ v , {{ mwl ( l ) k
( G, ˜ v [ x/ idx ˜ v ( y )]) (cid:12)(cid:12) x
∈
[Equation]
( G ) }}
=
{{ mwl ( l ) k ( G ′ , ˜ v ′
[ x/ idx ˜ v ′ ( h ( y ))]) (cid:12)(cid:12) x ∈ V ( G ′ ) }} .
Now let’s start the DBP k game at position ( ˜ v , ˜ v ′ ).
By (2) we know that there exist a h with (2) hold.
Then at the first round, we as Player II picks the h as the bijection.
Next the Player I will choose a y ∈ ˜ v .
According to (2), for the h and y we have {{ mwl ( l ) k
( G, ˜ v [ x/ idx ˜ v ( y )]) (cid:12)(cid:12) x ∈
V ( G ) }} = {{ mwl ( l ) k
( G ′ , ˜ v ′
[ x/ idx ˜ v ′ ( h ( y ))]) (cid:12)(cid:12) x ∈ V ( G ′ ) }} .
This implies that there exists a bijection f : V ( G ) → V ( G ′ )
such that ∀ x ∈ V ( G ) , mwl ( l ) k
( G, ˜ v [ x/ idx ˜ v ( y )])
=
mwl ( l ) k
( G, ˜ v ′
[ f ( x ) / idx ˜ v ′ ( h ( y ))]) .
Hence let Player II picks the f .
Now player I will choose a x ∈ V ( G ) .
Then mwl ( l ) k ( G, ˜ v [ x/ idx ˜ v ( y )]) =
mwl ( l ) k
( G, ˜ v ′
[ f ( x ) / idx ˜ v ′ ( h ( y ))]) implies mwl (0)
k
( G, ˜ v [ x/ idx ˜ v ( y )])
=
mwl (0) k
( G, ˜ v ′
[ f ( x ) / idx ˜ v ′ ( h ( y ))]) hence G [˜ v t [ x/ idx ˜ v t ( y )]] and
G ′
[˜ v ′ t [ f ( x ) / idx ˜ v ′ t ( h ( y ))]] are isomorphic.
So the game doesn’t end.
At next round, the game start at the position (˜ v [ x/ idx ˜ v ( y )] , ˜ v ′
[ f ( x ) / idx ˜ v ′ ( h ( y ))]) , and we know mwl ( l ) k
( G, ˜ v [ x/ idx ˜ v ( y )])
=
mwl ( l ) k
( G, ˜ v ′
[ f ( x ) / idx ˜ v ′ ( h ( y ))]) .
By inductive hypothesis, the Player II has a strategy to play
DBP k at the new position for l round.
Hence the Player II has a winning strategy to play DBP k at original position ( ˜ v , ˜ v ′ ) for l + 1 rounds.
Next we prove right =
⇒ left by showing that mwl ( l +1) k
[Equation]
k
[Equation]
G
[Equation]
˜ v
[Equation]
) =
⇒ Player
I has a winning strategy for l + 1 -step game DBP k
( G, ˜ v , G ′ , ˜ v ′ ) .
mwl
[Equation]
G
[Equation] [Equation]
implies:
[Equation]
or (2) for any bijection h : ˜ v → ˜ v ′ , ∃ y ∈ ˜ v , such that {{ mwl ( l ) k
( G, ˜ v [ x/ idx ˜ v ( y )]) (cid:12)(cid:12) x ∈ V ( G ) }} ̸ = {{
[Equation]
If (1) holds, then by induction we know that Player I has a winning strategy within l -steps hence Player II also has a winning strategy within l + 1 steps.
If (2) holds, then at the first round after Player II pickup a bijection h , the Player I can choose the specific y ∈ ˜ v with {{ mwl ( l ) k ( G, ˜ v [ x/ idx ˜ v ( y )])
(cid:12)(cid:12) x ∈ V ( G ) }} ̸ = {{ mwl ( l )
k
( G ′ , ˜ v ′
[ x/ idx ˜ v ′ ( h ( y ))]) (cid:12)(cid:12) x ∈ V ( G ′ ) }} .
Then no matter which bijection f : V ( G ) → V ( G ′ )
the Player II choose, the Player I can always choose a x ∈ V ( G ) such that mwl ( l ) k
( G, ˜ v [ x/ idx ˜ v ( y )])
̸ = mwl ( l ) k
( G ′ , ˜ v ′
[ x/ idx ˜ v ′ ( h ( y ))]) .
Then by induction, the Player I has a winning strategy within l -steps for the DBP k starts at position (˜ v [ x/ idx ˜ v ( y )] , ˜ v ′
[ x/ idx ˜ v ′ ( h ( y ))]) .
Hence even if the Player I doesn’t win in the first round, he/she can still win in l + 1 round.
Combining both sides we know that the equivalence between t -step DBP k and t -step kM ULTISET WL holds for any t and any k -multisets.
Theorem 11.
k -M ULTISET WL cannot distinguish G and G ′ at step t , i.e. gmwl ( t ) k
( G ) = gmwl ( t ) k
( G ′ ) , if and only if Player II has a winning strategy for t -step DBP k
( G, G ′ ) . Proof. The proof is strict forward with using the proof inside Theorem 3. We just need to show that the pooling of all k -multisets representations is equivalent to Player II finding a bijection g :
Mlutiset ( V ( G ) k ) → Mlutiset ( V ( G ′ )
k ) at the first step of the game.
We omit that given its simplicity.
A.4 Proofs of Theorems A.4.1 Bound of Summation of Binomial Coefficients
Derivation of:
[Equation]
(cid:88)
[Equation]
(cid:18)
[Equation]
Proof.
[Equation]
A.4.2 Proof of Theorem
Theorem 1.
Let k ≥ 1 and wl ( t ) k
( G, ˜ v ) :
= {{ wl ( t ) k
( G, p (˜ v ))
| p ∈ perm[k] }} .
For all t ∈ N and all graphs G, G ′ :
k -M ULTISET WL is upper bounded by k -WL in distinguishing multisets G, ˜ v
and
G ′ , ˜ v ′ at t -th iteration, i.e. wl ( t ) k
( G
[Equation]
˜ v )
= wl
[Equation]
t )
[Equation]
=
[Equation]
k ( G
[Equation]
, ˜ v
[Equation]
Proof.
By induction on t .
It’s obvious that when t =
the above statement hold, as both side are equivalent to G [˜ v ] and G ′
[˜ v ′ ] being isomorphic to each other.
Assume ≤ t the above statement is true.
For t + 1 case, by definition the left side is equivalent to {{ wl ( t +1) k
( G, p (˜ v ))
| p ∈ perm[k] }} = {{ wl ( t +1) k
( G ′ , p (˜ v ′ ))
| p ∈ perm[k] }} .
Let −→ v be the ordered version of ˜ v following canonical ordering over G , then there exists a bijective mapping f between ˜ v and ˜ v ′ , such that wl ( t +1) k
[Equation]
( G ′ , f ( −→ v )) , where f ( −→ v ) :
= ( f ( −→ v 1 ) , ..., f ( −→ v k )) .
By [10]’s Theorem 5.2, for any t , wl ( t ) k
( G, −→ v ) =
wl ( t ) k
( G ′ , f ( −→ v )) is equivalent to that player II has a winner strategy for t -step pebble game with initial configuration G, −→ v
and G ′ ,
f ( −→ v ) (please refer to [10] for the description of pebble game).
Notice that applying any permutation to the pebble game’s initial configuration won’t change having winner strategy for player II, hence we know that ∀ p ∈ perm[k], wl ( t +1) k
( G, p ( −→ v ))
=
wl ( t +1) k
( G ′ , p ( f ( −→ v ))) .
Now applying Eq.(1), we know that (1) wl ( t ) k
( G, p ( −→ v ))
=
wl ( t ) k
( G ′ , p ( f ( −→ v ))) , and (2) ∀ i ∈
[ k ] , {{ wl ( t ) k
( G, p ( −→ v [ x/i ]))
| x ∈ V ( G ) }}
= {{ wl ( t ) k
( G ′ , p ( f ( −→ v )
[ x/i ]))
| x ∈ V ( G ′ ) }} .
We rewrite (2) as, ∀ i ∈
[ k ] , ∃ bijective h i :
V ( G ) → V ( G ′ ) , ∀ x ∈ V ( G ) , wl ( t ) k
( G, p ( −→ v [ x/i ]))
=
wl ( t ) k
( G ′ , p ( f ( −→ v )
[ h i ( x ) /i ])) .
As (1) and (2) hold for any p ∈ perm[k] , now applying induction hypothesis to both (1) and (2), we can get (a) mwl ( t ) k ( G, ˜ v ) =
mwl ( t ) k ( G ′ , ˜ v ′ ) , and (b) ∀ i ∈
[ k ] , ∃ bijective h
i : V ( G ) → V ( G ′ ) , ∀ x ∈ V ( G ) , mwl ( t ) k
( G, ˜ v [ x/i ])
=
mwl ( t ) k
( G ′ , ˜ v ′
[ h i ( x ) /g
( i )]) , where g : [ k ] → [ k ] is the index mapping function corresponding to f .
Now we rewrite (b) as ∃ g : [ k ] →
[ k ] , ∀ i ∈
[ k ] , {{ mwl ( t ) k
( G, ˜ v [ x/i ]) | x ∈ V ( G ) }} = {{ mwl ( t ) k
( G ′ , ˜ v ′
[ x/g ( i )]) | x ∈ V ( G ′ ) }} .
Combining (a) and (b), using Eq.(3) we can get mwl ( t +1) k
( G, ˜ v ) =
mwl ( t +1) k
( G ′ , ˜ v ′ ) .
Thus for any t the above statement is correct.
A.4.3 Proof of TheoremCFI Graphs and Their Properties
Cai, Furer and Immerman [10] designed a construction of a series of pairs of graphs CFI(k) such that for any k , k -WL cannot distinguish the pair of graphs CFI ( k ) .
We will use the CFI graph construction to prove the theorem.
Here we use a variant of CFI construction that is proposed in
Grohe et al.
’s
work [24].
Let K he a complete graph with k nodes.
We first construct an enlarged graph X ( K ) from the base graph K by mapping each node and edge to a group of vertices and connecting all (
[Equation]
) | ) groups of vertices following certain rules.
Notice that we use node for base graph and use vertex for the enlarged graph for a distinction.
Let vw denotes the edge connecting node v and node w .
For a node v ∈ V ( K ) , let E ( v ) :
= (cid:8) vw | vw ∈ E ( K ) (cid:9) denotes the set of all adjacent edges of v in the graph K .
{1} {2} {1,2} {1,3} {} {} {1} {3} {} {2} {3} {2,3} 1^ 1^ 3^ 3^ 2^ 2^ {1,2} {1,3} {} {} {} {2,3} 1^ 1^ 3^ 3^ 2^ 2^ {1} {2} {1,3} {} {} {2,3} 1^ 1^ 3^ 3^ 2^ 2^
For every node v ∈ V ( K ) , we map node v to a group of vertices S v :=
{ v X | X ⊆ E ( v ) } with size | S v | = 2 deg ( v ) = 2 k − 1 .
For every edge e ∈ E ( K ) , the construction maps e to two vertices
S e := { e 0 , e 1 } .
Hence there are 2 ∗| E ( K ) | + | V ( K ) |∗ 2 k − 1 =
k ( k − 1)+ k (2 k − 1 ) number of vertices in the enlarged graph
X ( K ) with V ( X ( K ))
= ( ∪ v ∈ V ( K ) S v )
∪ ( ∪ e ∈ E ( K ) S e ) .
Let V ∗ :
= ∪ v ∈ V ( K ) S v and E ∗ := ∪ e ∈ E ( K ) S e .
Now edges inside X ( K ) are defined as the follows
[Equation]
[Equation] [Equation] [Equation]
[Equation]
What’s more, we also color the vertices such that all vertices inside S v have color v for every v ∈ V ( K ) and all vertices inside S e have color e for every e ∈ E ( K ) .
See Figure.4 top right for the visualization of transforming from base graph
K to X ( K ) with k = 3 .
There are several important properties about the automorphisms of X ( K ) .
Let h ∈ Aut ( X ( K )) be an automorphism, then
h ( S v ) =
S v
and h ( S e ) =
S e for all v ∈ V ( K ) and e ∈ E ( K ) .
For every subset F ⊆ E ( K ) , there is exactly one automorphism h F that flips precisely all edges in F, i
e. h F ( e 0 )
=
e 1 and h F ( e 1 ) =
e 0 if and only if e ∈ F .
More specifically, • h F ( e i ) =
h F ( e 1 − i ) ∀ e ∈ F • h F ( e i ) =
h F ( e i ) ∀
e ̸∈ F • h F ( v X ) =
v Y , ∀ v ∈ E ( K ) , X
⊆ E ( v ) where Y :
=
X
△
(cid:0)
[Equation]
Proof.
These properties are not hard to prove.
First for property 1, it is true based on the coloring rules of vertices in X ( K ) .
Now for the second property.
As h F is flips precisely all edges in F , we have h F ( e i )
=
h F ( e 1 − i ) ∀ e ∈ F and h F ( e i )
=
h F ( e i ) ∀
e ̸∈ F .
Now let h F ( v X )
= v Y ∀ v ∈ E ( K ) ,
X ⊆ E ( v ) , we need to figure out Y ’s formulation.
Let’s focus on node v without losing generality, and we can partition the set E ( v ) to two parts: E ( v ) ∩ F and E [ v ] \ F .
Then for any e ∈ E ( v ) ∩ F , we know that h F ( e i ) =
e 1 − i .
Let v ↔ w denotes v and w are connected in X ( K ) .
Then ∀ X ⊆
E ( v ) , e ∈ X ⇐⇒ e 0 ↔ v X ⇐⇒ h F ( e 0 ) ↔ h F ( v X )
⇐⇒ e 1 ↔ v Y ⇐⇒
e ̸∈ Y .
And similarly we have ∀ X ⊆
E ( v ) , e ̸∈ X
⇐⇒ e ∈ Y .
Hence ∀ e ∈ E ( v ) ∩ F, ∀ X ⊆
E ( v ) , e ∈ X △ Y .
Then implies that E ( v ) ∩ F ⊆ X △ Y .
Following the same logic we can also get ∀ e ∈ E ( v ) \ F, ∀ X
⊆ E ( v ) , e ∈ E ( v ) \ ( X △ Y ) , which is equivalent to E ( v )
\ F ⊆ E ( v ) \ ( X △ Y ) , which further implies that E ( v ) ∩ F ⊇ X △ Y as X △ Y ⊆
E ( v ) .
Then combining both side we know that ∀ v ∈ E ( K ) , ∀ X ⊆
E ( v ) , E ( v ) ∩ F = X △ Y .
Hence we get Y = X △ ( X △ Y ) =
X △ (cid:0) F ∩ E ( v ) (cid:1) .
In the proof we can also know another important property.
That is ∀ v, X
⊆
E ( v ) , X △ h F ( X ) =
E ( v ) ∩ F is constant for any input X ∈ E ( v ) .
Now we are ready to construct variants of graphs that are not isomorphic from the enlarged graph
X ( K ) .
Now let T be a subset of V ( K ) .
Now we define an induced subgraph X T ( K ) of the enlarged graph
X ( K ) .
Specifically, we define the new node group as the follows
[Equation]
Then the induced subgraph is defined as X T ( K ) :
= X ( K )[( ∪ v ∈ V ( K ) S Tv ) ∪ E ∗ ] .
In Figure 4 we show the X ∅ ( K ) in bottom left (labeled with X 0 ( K ) ) and X { v 1 } ( K ) in the bottom right (labeled with
X 1 ( K ) ).
Lemma 1.
For all T, U ⊆ V ( K ) , X T ( K ) ∼ =
X U ( K )
if and only if | T | ≡ | U | mod 2 .
And if they are isomorphic, the only isomorphism between
X T ( K ) and X U ( K ) is h F with F = E ( K [ T △ U ]) . The proof of the lemma can be found in [10] and [24]. With this lemma we know that X { v 1 } ( K ) and
X ∅ ( K ) are not isomorphic.
In next part we show that X ∅ ( K ) and X { v 1 } ( K ) cannot be distinguished by ( k − 1) -WL but can be distinguished by k -M ULTISET WL, which means Theorem 2 is True.
Main Proof
Theorem 2.
k -M ULTISET WL is no less powerful than ( k - 1 )-WL in distinguishing graphs: for k ≥ 3 there is a pair of graphs that can be distinguished by k -M ULTISET WL but not by ( k - 1 )-WL.
Proof. A.4.4 Proof of Theorem
See Appendix.
A.3.
A.4.5 Proof of Theorem
Theorem 4.
Let k ≥ 1 , then ∀ t ∈ N and all graphs G, G ′ : mwl ( t ) k
( G, ˆ v ) =
mwl ( t ) k
( G ′ , ˆ v ′ )
=
⇒ swl ( t ) k
( G, ˆ v )
=
swl ( t ) k
( G ′ , ˆ v ′ ) .
Proof.
Notation: Let s ( · ) transform a multiset to set by removing repeats, and let r ( · ) return a tuple with the number of repeats for each distinct element in a multiset.
Let f be the inverse mapping such that ˜ v = f
( s (˜ v ) , r (˜ v )) .
Define F ( t +1) ( G, G ′ , ˜ v , ˜ v ′ ) :
= { injective f : ˜ v → ˜ v ′ | f ∈ F ( t ) ( G, G ′ , ˜ v , ˜ v ′ ) , AND , ∀ y ∈ ˜ v , ∃ h
y : V ( G ) → V ( G ′ ) , ∀ x, f ∈ F ( t ) ( G, G ′ , ˜ v [ x/ idx ˜ v ( y )] , ˜ v ′
[ h y ( x ) / idx ˜ v ′ ( f ( y ))]) } .
Let
F (0) ( G, G ′ , ˜ v , ˜ v ′ ) :
= { f | f is an isomorphism from G [˜ v ] to G ′
[˜ v ′ ] } .
Lemma 2.
∀ t ,
[Equation]
, ∀ ˆ n with ( (cid:80) mi =1 ˆ n i =
k, ∀ i ˆ n i ≥ 1) , mwl ( t ) k
( G, f ( s (˜ v ) , ˆ n )) = mwl ( t ) k
( G ′ , f ( h ( s (˜ v )) , ˆ n )) .
When t = 0 , by the definition of swl (0) k , the statement is true.
Now hypothesize that the statement is true for ≤ t case.
For t + 1 case, the left side implies that existing a ˜ v with s (˜ v ) =
ˆ v and a ˜ v ′ with s (˜ v ′ )
=
ˆ v ′ , such that mwl ( t +1) k
( G, ˜ v ) =
mwl ( t +1) k
( G ′ , ˜ v ′ ) .
And for a mapping h ∈
F ( t +1) ( G, G ′ , ˜ v , ˜ v ′ ) , we have ∀ ˆ n with ( (cid:80) mi
=1 ˆ n i = k, ∀ i ˆ n i ≥ 1) , mwl ( t +1) k [Equation] [Equation] = mwl
[Equation] k ( G [Equation] f (
[Equation] s
[Equation] [Equation] n [Equation]
For a specific ˆ n , define ˜ u
= f ( s (˜ v ) , ˆ n ) and ˜ u ′ =
f ( h ( s (˜ v )) , ˆ n ) =
h (˜ u ) .
By Eq.(3):
[Equation]
As h ∈ F ( t +1)
( G, G ′ , ˜ v , ˜ v ′ )
=
F ( t +1) ( G, G ′ , ˜ u , ˜ u ′ ) , we can change (2) by choosing f as h .
Hence we update it as:
[Equation]
V ( G ′ ) }} And (2) can be split into two parts:
[Equation]
and
[Equation]
Combining the Lemma.2 and induction hypothesis
[Equation]
also knowing that s (˜ u ) = ˆ v and s (˜ u ′ )
=
ˆ v ′
[Equation]
we get:
[Equation]
[Equation]
} , this is derived with picking ˆ n with ˆ n [ y ] = 1 , such that ˜ u only has 1 y and ˜ u
′ only has 1 h ( y ) .
(c) ∀
y ∈
ˆ v , {{ swl ( t ) k
( G, ˆ v ∪ { x } ) (cid:12)(cid:12) x ∈ V ( G ) \
ˆ v }} = {{ swl ( t ) k
( G ′ , ˆ v ′ ∪ { x } ) (cid:12)(cid:12) x ∈ V ( G ′ )
\ ˆ v ′ }} , this is derived with picking ˆ n with ˆ n [ y ] > 1 .
(d) ∀
y ∈
ˆ v , {{ swl ( t ) k
( G, ˆ v \ y ∪ { x } ) (cid:12)(cid:12) x ∈ V ( G ) \
ˆ v }} = {{ swl ( t ) k
( G ′ , ˆ v
′ \ h
( y ) ∪ { x } ) (cid:12)(cid:12) x ∈ V
( G ′ ) \ ˆ v ′ }} , this is derived with picking ˆ n with ˆ n [ y ] = 1 , such that ˜ u only has 1 y and ˜ u
′ only has 1 h ( y )
Now combining (a) (b) (c) (d) and using Eq.(5), we can get that swl ( t +1) k
( G, ˆ v ) =
swl ( t +1) k ( G ′ , ˆ v ′ ) , and we proved the statement.
A.4.6 Proof of Theorem
Theorem 5.
Let k ≥ 1 , then ∀ t ∈ N and all graphs G, G ′ :
(1) when 1 ≤ c 1 < c 2 ≤ k , if G, G ′ cannot be distinguished by ( k, c 2 )( ≤ ) -S ET WL , they cannot be distinguished by ( k, c 1 )( ≤ ) -S ET WL (2) when k 1 < k 2 , ∀ c ≤ k 1 , if G, G ′ cannot be distinguished by ( k 2 , c )( ≤ ) -S ET WL , they cannot be distinguished by ( k 1 , c )( ≤ ) -S ET WL
Proof.
We will prove (2) first, and the proof for (
[Equation]
Lemma 3.
For k 1 < k 2 , for any t , for any ˆ v and ˆ v ′ with ≤ k 1 nodes inside, swl ( t ) k 2 ,c ( G, ˆ v ) = swl ( t ) k 2 ,c ( G ′ , ˆ v ′ ) = ⇒ swl ( t ) k 1 ,c
( G, ˆ v ) =
swl ( t ) k 1 ,c ( G ′ , ˆ v ′ ) .
Proof of Lemma.3:
Proof.
We prove it by induction.
As the color initialization stage of ( k 1 , c )( ≤ ) -S ET WL and ( k 2 , c )( ≤ ) -S ET WL are the same, when t =
the statement of the Lemma is correct.
Assume it holds correct for ≤ t .
When t + 1 , swl ( t +1) k 2 ,c ( G, ˆ v )
=
swl ( t +1)
k
[Equation]
c ( G ′ , ˆ v ′ ) implies:
[Equation]
(1) + induction hypothesis =
⇒ (a) swl
[Equation] [Equation]
(2) has two situations: | ˆ v | = | ˆ v ′
| < k 1 and | ˆ v | =
| ˆ v ′ |
=
k 1 .
For the first situation, N Gk 2 ,c, right (ˆ v ) =
N Gk 1 ,c, right (ˆ v ) and N G ′ k 2 ,c, right (ˆ v ′ )
=
N G ′ k 1 ,c, right (ˆ v ′ ) , ∃ a bijective mapping b between N G ′ k 2 ,c, right (ˆ v ′ ) and N Gk 2 ,c, right (ˆ v ) such that ∀ ˆ u ∈ N Gk 2 ,c, right (ˆ v ) , swl ( t ) k 2 ,c
( G, ˆ u ) =
swl ( t ) k 2 ,c ( G ′ , b (ˆ u )) and by induction swl ( t ) k 1 ,c ( G, ˆ u ) = swl ( t ) k 1 ,c ( G ′ , b (ˆ u )) , hence swl ( t +1 / 2) k 1 ,c
( G, ˆ v )
=
swl ( t +1 / 2) k 1 ,c ( G ′ , ˆ v ′ ) .
For the second situation, N G ′ k 2 ,c, right (ˆ v [Equation] ) elements while N G ′ k 1 ,c, right (ˆ v ′ )
= ∅ , then clearly (b) swl ( t +1 / 2) k 1 ,c
( G, [Equation]
= swl [Equation] + [Equation]
[Equation] [Equation] ( G ′ , ˆ v ′ )
(both are empty multisets).
(3) follows the same argument step in (2)’s first situation, with induction hypothesis we can get (c) {{ swl [Equation] t ) k 1 ,c [Equation]
G, [Equation]
[Equation] 1 ,c,
[Equation]
} (4) implies that ∃ a mapping b between N G ′ k 2 ,c, left (ˆ v ′ ) and N Gk 2 ,c, left (ˆ v ) such that for any ˆ u ∈ N G ′ k 2 ,c, left (ˆ v ′ ) , swl ( t +1 / 2) k 2 ,c (
[Equation]
[Equation]
( G ′ , b (ˆ u ′ )) .
Using the same argument in (2) and induction hypothesis, this implies that for any ˆ u ∈ N G ′ k 1 ,c, left (ˆ v ′ ) , swl ( t +1 / 2) k 1 ,c ( G, ˆ u ) = swl ( t +1 / 2) k 1 ,c
( G ′ , b (ˆ u ′ )) .
Hence we get (d) {{ swl ( t +1 / 2) k 1 ,c [Equation] [Equation] left
[Equation]
) }} = {{ swl ( t +1 / 2) k 1 ,c ( G
[Equation]
u
[Equation]
) | ˆ u
[Equation]
∈ N G
[Equation]
k [Equation] left ( ˆ v ′ ) }} . Combining (a) (b) (c) (d) we get swl ( t +1) k 1 ,c
( G, ˆ v )
=
swl ( t +1) k 1 ,c ( G ′ , ˆ v ′ ) .
With Lemma.3 we are ready to prove (2) in Theorem 3.
When two graphs G and G ′ cannot be distinguished by ( k 2 , c )( ≤ ) -S ET WL, we have {{ swl ( t ) k 2 ,c ( G, ˆ v ) | ˆ v ∈ V ( S k 2 ,c
-swl
( G )) }}
= {{ swl ( t ) k 2 ,c
( G ′ , ˆ v ′ )
| ˆ v ′ ∈ V ( S k 2 ,c
-swl
( G ′ )) }} .
When ˆ v and ˆ u have different k (number of nodes) and c (number of components of its induced subgraph), their color cannot be the same (as t =already be different).
Hence {{ swl ( t ) k 2 ,c ( G, ˆ v ) | ˆ v ∈ V ( S k 2 ,c
-swl
( G )) }}
= {{ swl ( t ) k 2 ,c
( G ′ , ˆ v ′ )
| ˆ v ′ ∈
V ( S k 2 ,c -swl ( G ′ )) }} is equivalent to ∀ k ≤ k 2 , cc ≤ c , {{ swl ( t ) k 2 ,c
( G, ˆ v ) | ˆ v ∈ V k,cc ( S k 2 ,c
-swl
( G )) }}
= {{ swl ( t ) k 2 ,c
( G ′ , ˆ v ′ )
| ˆ v ′ ∈
V k,cc ( S k 2 ,c -swl ( G ′ )) }} .
With Lemma.3 we can get that ∀ k ≤ k 1 , cc ≤ c , {{ swl ( t ) k 1 ,c
( G, ˆ v )
| ˆ v ∈ V k,cc ( S k 1 ,c -swl ( G )) }} = {{ swl ( t )
k 1 ,c ( G ′ , ˆ v ′ )
| ˆ v ′ ∈
V k,cc ( S k 1 ,c
-swl
( G ′ )) }} .
Hence two graphs G and G ′ cannot be distinguished by ( k 1 , c )( ≤ ) -S ET WL.
A.4.7 Proof of Theorem
Theorem 6.
When ( i ) BaseGNN can distinguish any non-isomorhpic graphs with at most k nodes, ( ii ) all MLPs have sufficient depth and width, and ( iii ) POOL is an injective function, then for any t ∈ N , t -layer ( k, c )( ≤ ) -S ET GNN is as expressive as ( k, c )( ≤ ) -S ET WL at the t -th iteration.
Proof.
For any G, ˆ v , let swl ( t ) k,c
( G, ˆ v ) denotes the color of ˆ v at t -iteration ( k, c )( ≤ ) -S ET WL and h ( t ) k,c
( G, ˆ v ) denotes the embedding of ˆ v at t -th layer ( k, c )( ≤ ) -S ET GNN.
We prove the above theorem by showing that swl ( t ) k,c ( G, ˆ v ) =
swl ( t ) k,c ( G ′ , ˆ v ′ )
⇐⇒ h
( t ) k,c ( G, ˆ v ) =
h ( t ) k,c ( G ′ , ˆ v ′ ) .
We remove the subscript k, c when possible without introducing confusion.
For easier reference, recall the updating formulation for t -iteration ( k, c )( ≤ ) -S ET WL is
[Equation]
[Equation]
[Equation]
And the updating formulation for ( k, c )( ≤ ) -S ET GNN is
[Equation]
At t = 0 , given powerful enough BaseGNN with condition (i) in the theorem, h (0) ( G, ˆ v ) =
h (0) ( G ′ , ˆ v ′ )
⇐⇒ G
[ˆ v ] and G ′
[ˆ v ′ ] are isomorphic ⇐⇒ swl (0)
( G, ˆ v )
= swl (0) ( G ′ , ˆ v ′ ) .
Now assume for ≤ t iterations the claim swl ( t ) ( G, ˆ v )
=
swl ( t ) ( G ′ , ˆ v ′ )
⇐⇒ h
( t ) ( G, ˆ v )
=
h ( t ) ( G ′ , ˆ v ′ ) holds (for any ˆ v and ˆ v ′ ).
We prove it holds for t + 1 iteration.
We first prove the forward direction.
swl ( t +1)
( G, ˆ v ) =
swl ( t +1)
( G ′ , ˆ v ′ ) imples that
(1) swl ( t ) ( G, ˆ v ) = swl ( t ) ( G ′ , ˆ v ′ )
(2) swl ( t +1 / 2) ( G, ˆ v ) =
swl ( t +1 / 2) ( G ′ , ˆ v ′ )
(3) {{ swl ( t ) ( G, ˆ u ) | ˆ u ∈ N G left (ˆ v ) }} = {{ swl ( t ) ( G ′ , ˆ u ′ )
| ˆ u ′ ∈ N G ′ left (ˆ v ′ ) }} (
[Equation]
} (1) and (3) can be directly transformed to relationship between h using the inductive hypothesis.
Formally, we have
(a) h ( t ) ( G, ˆ v ) = h ( t ) ( G ′ , ˆ v ′ ) and (c) {{ h ( t ) ( G, ˆ u ) | ˆ u ∈ N G left (ˆ v ) }} = {{ h ( t ) ( G ′ , ˆ u ′ ) | ˆ u ′ ∈ N G ′ left (ˆ v ′ ) }}
(2) and (4) need one additional process.
Notice that (2) is equivalent to {{ swl ( t ) ( G, ˆ u ) | ˆ u ∈
N
G right (ˆ v ) }} = {{ swl ( t ) ( G ′ , ˆ u ′ )
| ˆ u ∈ N G ′ right
(ˆ v ′ ) }} and by inductive hypothesis we know {{ h ( t ) ( G, ˆ u ) | ˆ u ∈ N G right
(ˆ v ) }} = {{ h ( t ) ( G ′ , ˆ u ′ )
| ˆ u ∈ N G ′ right
(ˆ v ′ ) }} .
As the formulation Eq.(29) applies a MLP with summation, which is permutation invariant to ordering, to the multiset {{ h ( t ) ( G, ˆ u ) | ˆ u ∈ N G right (ˆ v ) }} , the same multiset leads to the same output.
Hence we know
(b) h ( t +1 / 2) (
G,
ˆ v ) =
h ( t +
G ′ , ˆ
v ′
For (4) we know that there is a bijective mapping g between N G left (ˆ v ) and N G left (ˆ v ′ )
such that swl ( t +1 / 2) ( G, ˆ u ) = swl ( t +1 / 2) ( G ′ , g (ˆ u )) .
Then using the same argument as (2) =>(b) we can get h ( t +1 / 2) ( G, ˆ u )
=
h ( t +1 / 2) ( G ′ , g (ˆ u )) for any ˆ u ∈ N G left (ˆ v ) , which is equivalent to
[Equation]
Combining (a)(b)(c)(d) and using the permutation invariant property of MLP with summation, we can derive that h ( t +1)
( G, ˆ v ) =
h ( t +1) ( G ′ , ˆ v ′ ) .
Now for the backward direction.
We first characterize the property of MLP with summation from
DeepSet [59] and GIN [56]’s Lemma 5.
Lemma 4 (Lemma 5 of [56]) .
Assume X is countable.
There exists a function f : X → R n so that h ( X )
= (cid:80) x ∈ X f
( x ) is unique for each multiset X ∈ X of bounded size.
Using this Lemma, we know that given enough depth and width of a MLP, there exist a MLP that (cid:80) x ∈ X MLP ( x )
= (cid:80)
y ∈
Y MLP ( y )
⇐⇒ X
= Y or in other words two multisets X and Y are equivalent.
Now by h ( t +1) ( G, ˆ v ) =
h ( t +1) ( G ′ , ˆ v ′ ) and using the Eq.(31), we know there exists
MLPs inside Eq.(29) and Eq.(31), such that
(1) h ( t ) ( G, ˆ v ) =
h ( t ) ( G ′ , ˆ v ′ )
(2) h ( t +1 / 2) ( G, ˆ v ) =
h ( t +1 / 2) ( G ′ , ˆ v ′ ) (3) {{ h ( t ) ( G, ˆ u ) | ˆ u ∈ N G left (ˆ v ) }} = {
[Equation]
Where (3) and (4) are derived using the provided lemma.
Hence following the same argument as the forward process, we can prove that swl ( t +1)
( G, ˆ v )
=
swl ( t +1)
( G ′ , ˆ v ′ ) .
Combining the forward and backward direction, we have swl ( t +1)
( G, ˆ v )
=
swl ( t +1)
( G ′ , ˆ v ′ )
⇐⇒ h
( t +1)
( G, ˆ v ) =
h ( t +1) ( G ′ , ˆ v ′ ) .
Hence by induction we proved for any step t and any ˆ v , ˆ v ′ , the above statement is true.
This shows that ( k, c )( ≤ ) -S ET GNN and ( k, c )( ≤ ) -S ET WL have same expressivity.
A.4.8 Proof of Theorem
Theorem 7.
For any t ∈ N , the t -layer ( k, c )( ≤ ) -S ET GNN ∗ is more expressive than the t -layer ( k, c )( ≤ ) -S ET GNN .
As lim t →∞ , ( k, c )( ≤ ) -S ET GNN is as expressive as ( k, c )( ≤ ) -S ET GNN ∗ .
Proof.
We first prove that t -layer ( k, c )( ≤ ) -S ET GNN ∗ is more expressive than t -layer ( k, c )( ≤ ) -
S ET GNN, by showing that if a m -set ˆ u has the same representation to another m -set ˆ v in t -layer ( k, c )( ≤ ) -S ET GNN ∗ , then they also have the same representation in t -layer ( k, c )( ≤ ) -S ET GNN.
Let h be the representation inside ( k, c )( ≤ ) -S ET GNN ∗ , and g be the representation inside ( k, c )( ≤ ) -S ET GNN.
To simplify the proof, we first simplify the formulations of ( k, c )( ≤ ) -S ET GNN ∗ Eq.13 and Eq.
by removing unnecessary super and under script of MLP without introducing ambiguity.
We then add another superscript to embeddings to indicate which step inside the one-layer bidirectional propagation.
Notice that for one-layer bidirectional propagation, there are 2 k − 2 intermediate sequential steps.
The proof assumes that all MLPs are injective.
We rewrite Eq.13 and Eq.14 as follows ∀
[Equation]
∀ m -set
[Equation]
Where we have boundary case with h ( t + 12 , 0) (ˆ u )
= h ( t ) (ˆ u ) and h ( t +1 , 0)
(ˆ u ) =
h ( t + 12 ) (ˆ u ) .
h ( t + 12 ) (ˆ v ) :
= h ( t + 12 ,k − m ) (ˆ v ) represents that the representation is calculated at k − m step for t + 12 layer.
We prove the theorem by induction on t .
Specifically, we want to prove that for any t , ˆ v , ˆ u , h ( t ) (ˆ u )
= h ( t ) (ˆ v ) =
⇒ g
( t ) (ˆ u ) =
g ( t ) (ˆ v ) and h ( t − 12 ) (ˆ u )
=
h ( t − 12 ) (ˆ v ) =
⇒ g ( t − 12 ) (ˆ u )
=
g ( t − 12 ) (ˆ v ) .
The base case is easy to verify as the definition of initialization step is the same.
Now assume that for t ≤ l we have for any t , ˆ v , ˆ u , h ( t ) (ˆ u )
= h ( t ) (ˆ v ) =
⇒ g
( t ) (ˆ u ) =
g ( t ) (ˆ v ) and h ( t − 12 ) (ˆ u )
=
h ( t − 12 ) (ˆ v ) =
⇒ g ( t − 12 ) (ˆ u )
=
g ( t − 12 ) (ˆ v ) .
We first prove that for t = l + 1 , h ( l + 12 ) (ˆ u )
=
h ( l + 12 ) (ˆ v ) =
⇒ g
( l + 12 ) (ˆ u ) =
g ( l + 12 ) (ˆ v ) .
First, Eq.32 can be rewrite as
[Equation]
[Equation] Then h ( l + 12 ) (ˆ v ) = h ( l + 12 ) (
[Equation]
Hence we can find a bijective mapping f between N G right (ˆ v ) and N G right (ˆ u ) such that MLP ( h ( l ) ( ˆ w )) =
MLP ( h ( l ) ( f ( ˆ w ))) .
Then by inductive hyphothesis, MLP ( g ( l ) ( ˆ w )) =
MLP ( g ( l ) ( f ( ˆ w ))) for all ˆ w ∈ N G right (ˆ v ) .
This implies that (cid:80) ˆ w ∈N G right [Equation] g [Equation]
l
[Equation]
( ˆ [Equation] = (cid:80) ˆ w ∈N
[Equation] MLP ( g [Equation] l ) ( ˆ w )) or equivalently g ( l + 12 ) (ˆ v ) =
g ( l + 12 ) (ˆ u ) .
Now we can assume that for t ≤ m we have for any t , ˆ v , ˆ u , h ( t ) (ˆ u )
= h ( t ) (ˆ v ) =
⇒ g
( t ) (ˆ u ) =
g ( t ) (ˆ v ) and for t ≤ l + 1 h
( t − 12 ) (ˆ u ) =
h ( t − 12 ) (ˆ v ) =
⇒ g ( t − 12 ) (ˆ u )
=
g ( t − 12 ) (ˆ v ) .
We prove that for t = l + 1 , h ( l +1)
(ˆ u ) =
h ( l +1)
(ˆ v ) =
⇒ g
( l +1)
(ˆ u ) =
g ( l +1) (ˆ v ) .
We first rewrite Eq.33 as
[Equation]
=
[Equation]
Then h ( l +1) (ˆ u ) =
h ( l +1) (ˆ v ) implies
1) h ( t + 12 ,k − m ) (ˆ v ) =
h ( t + 12 ,k − m ) (ˆ u ) or equivalently
h ( t + 12 ) (ˆ v ) = h ( t + 12 ) (ˆ u ) 2) {{ h ( t + 12 ) ( ˆ w ) | ˆ w ∈ N G left (ˆ v ) }} = {{ h ( t + 12 ) ( ˆ w ) | ˆ w ∈ N G left (ˆ u ) }}
Also by {+Eq.
+} we know that h ( t + 12 ) (ˆ v ) =
h ( t + 12 ) (ˆ u ) implies 3) h t
ˆ v ) =
h ( t ) ( ˆ u
Combining with 2) and 3) we know
[Equation]
Now using our inductive hypothesis, we can transform 1) 2) 3) 4) by replace h with g .
Then based on the equation in 11, we know g ( l +1)
(ˆ u ) =
g ( l +1) (ˆ v ) .
Combining the two proved inductive hypothesis and applying them alternately, we know that for any t , ˆ v , ˆ u , h ( t ) (ˆ u )
=
h ( t ) (ˆ v ) =
⇒ g
( t ) (ˆ u ) =
g ( t ) (ˆ v ) and h ( t − 12 ) (ˆ u )
=
h ( t − 12 ) (ˆ v ) =
⇒ g ( t − 12 ) (ˆ u )
=
g ( t − 12 ) (ˆ v ) .
Hence t -layer ( k, c )( ≤ ) -S ET GNN ∗ is more expressive than t -layer ( k, c )( ≤ ) -S ET GNN.
Now let’s considering t to infinity.
Then all representations will become stable with h ( t ) (ˆ v )
=
h ( t ) (ˆ u ) ⇐⇒ h
( t +1) (ˆ v )
=
h ( t +1) (ˆ u ) and h ( t − 1 / 2) (ˆ v )
=
h ( t − 1 / 2) (ˆ u ) ⇐⇒ h
( t +1 / 2) (ˆ v ) =
h ( t +1 / 2) (ˆ u ) .
Notice that for a single set ˆ v , the information used from its neighbors are the same in both ( k, c )( ≤ ) -S ET GNN and ( k, c )( ≤ ) -S ET GNN ∗ .
Hence the equilibrium equations for set ˆ v should be the same in both ( k, c )( ≤ ) -S ET GNN and ( k, c )( ≤ ) -S ET GNN ∗ .
Then they will have the same stable representations at the end.
A.4.9 Proof of Theorem
Theorem 8.
Let G be a graph with c connected components C 1 , ..., C c , and G ′ be a graph also with c connected components C ′ 1 , ..., C ′ c , then G and G ′ are isomorphic if and only if ∃ p : [ c ] → [ c ] , s.t.
∀ i ∈
[ c ] , C i and C ′ p ( i ) are isomorphic.
Proof.
Right =
⇒ Left .
Let h i : V ( C i )
→ V ( C ′ p ( i ) ) be one isomorphism from C i to C ′ p
( i ) for i ∈ [ c ] .
Then we can create a new mapping h : V ( G )
→ V ( G ′ ) , such that for any x ∈ V ( C ) , it first locates which component x inside, for example i , then apply h i to x .
Clearly h is a isomorphism between G and G ′ , hence G and G ′ are isomorphic.
Left = ⇒
Right .
G and G ′ are isomorphic, then there exists an isomorphism h from G to G ′ .
Now for a specific component C i in G , h maps V ( C i ) to a nodes set inside V ( G ′ ) , and name it as B i .
For i ̸
= j , B i ∩ B j = ∅
and there is not edges e ∈ E ( G ′ ) between B i and B j otherwise there must be an corresponding edge (apply h − 1 ) between V ( C i ) and V ( C j ) which is impossible.
Hence { G ′
[ B i ] } ci =1 are disconnected subgraphs.
As h also preserves connections, for any i , C i and G ′
[ B i ] are isomorphic.
Hence we proved the right side.
A.4.10 Conjecture that k-MWL is equvialent to k-WL
Proof
[Equation]
Proof.
We first take a closer look at the condition of mwl ( t +1) k
[Equation]
˜ v ) =
mwl ( t +1) k
( G ′ , ˜ v ′ ) .
By Eq.(3) we know this is equivalent to (1) mwl ( t ) k
( G, ˜ v ) =
mwl ( t ) k ( G ′ , ˜ v ′ ) and (2) (cid:26)(cid:26) {{ mwl ( t ) k
( G, ˜ v [ x/o − 1 G (˜ v , i )]) (cid:12)(cid:12) x ∈ V ( G ) }} | i = 1 , ..., k (cid:27)(cid:27) =
(cid:26)(cid:26) {{ mwl ( t ) k
( G ′ , ˜ v ′
[ x/o − 1 G (˜ v ′ , i )]) (cid:12)(cid:12) x ∈ V ( G ′ ) }} | i = 1 , ..., k (cid:27)(cid:27) .
(2) can be rewrite as (cid:26)(cid:26)
[Equation]
y
[Equation]
(cid:27)(cid:27) [Equation] (cid:26)(cid:26) {{ mwl
[Equation]
( G
[Equation]
, ˜
[Equation]
[ x/
[Equation]
( y [Equation] [Equation] (cid:12)(cid:12) x ∈
V ( G ′ ) }} | y ∈ ˜ v ′ (cid:27)(cid:27) , which is equivalent to (3) ∃ bijective f : V ( G ) → V ( G ′ ) , ∀ y ∈
˜
v , {{ mwl ( t ) k
( G, ˜ v [ x/ idx ˜ v ( y )]) (cid:12)(cid:12) x ∈ V ( G ) }} = {{ mwl ( t ) k
( G ′ , ˜ v ′
[ x/ idx ˜ v ′ ( f ( y ))]) (cid:12)(cid:12) x ∈ V ( G ′ ) }} .
Define F ( t +1) ( G, G ′ , ˜ v , ˜ v ′ ) :
= { injective f : ˜ v → ˜ v ′ | f ∈ F ( t ) ( G, G ′ , ˜ v , ˜ v ′ ) , AND , ∀ y ∈ ˜ v , ∃ h
y : V ( G ) → V ( G ′ ) , ∀ x, f ∈ F ( t ) ( G, G ′ , ˜ v [ x/ idx ˜ v ( y )] , ˜ v ′
[ h y ( x ) / idx ˜ v ′ ( f ( y ))]) } .
Let
F (0) ( G, G ′ , ˜ v , ˜ v ′ ) :
= { f | f is an isomorphism from G [˜ v ] to G ′
[˜ v ′ ] } .
Lemma 5.
∀ t , mwl ( t ) k
( G, ˜ v ) =
mwl ( t ) k ( G ′ , ˜ v ′ )
if and only if F ( t ) ( G, G ′ , ˜ v , ˜ v ′ ) ̸
= ∅ .
Notice that this Lemma is conjectured to be true.
This needs additional proof.
Lemma 6.
∀ t, mwl ( t ) k
( G, ˜ v )
[Equation]
[Equation]
) k
( G, f
[Equation]
v ))
Proof of Lemma.6:
We prove it by induction on t .
When t = 0 , F (0) ( G, G ′ , ˜ v , ˜ v ′ ) contains all isomorphisms between G [˜ v ] and G ′
[˜ v ′ ] , hence the right side is correct.
Assume the statement is correct for ≤ t .
For t + 1 case, the left side implies (1) mwl ( t ) k
( G, ˜ v ) =
mwl ( t ) k ( G ′ , ˜ v ′ ) and (2) F ( t +1)
( G, G ′ , ˜ v , ˜ v ′ )
̸
= ∅ , ∀ f ∈ F ( t +1) ( G, G ′ , ˜ v , ˜ v ′ ) , ∀ y ∈ ˜ v , ∃ h y :
V ( G ) →
V ( G ′ ) , ∀ x ∈
V ( G ) , mwl ( t ) k
( G, ˜ v [ x/ idx ˜ v ( y )])
=
mwl ( t ) k ( G ′ , ˜ v ′
[ h y ( x ) / idx ˜ v ′ ( f ( y ))]) .
By induction hypothesis, (1) and F ( t +1)
( G, G ′ , ˜ v , ˜ v ′ ) ⊆
F ( t ) ( G, G ′ , ˜ v , ˜ v ′ )
imply (a) ∀ f ∈ F ( t +1)
( G, G ′ , ˜ v , ˜ v ′ ) , wl ( t ) k
( G, −→ v ) =
wl ( t ) k
( G ′ , f ( −→ v )) ; (2) and ∀ y ∈ ˜ v , f ∈ F ( t ) ( G, G ′ , ˜ v [ x/ idx ˜ v ( y )] , ˜ v ′
[ h y ( x ) / idx ˜ v ′ ( f ( y ))]) imply (b) ∃ h , ∀ x ∈ V ( G ) , wl ( t ) k
( G, −→ v [ x/ idx −→ v ( y )])
=
wl ( t ) k
( G ′ , f ( −→ v )
[ h ( x ) / idx f ( −→ v ) ( f ( y ))]) , which can be rewritten as ∀ y ∈ ˜ v , {{ wl ( t ) k
( G, −→ v [ x/ idx −→ v ( y )] | x ∈ V ( G )) }} = {{ wl ( t ) k
( G ′ , f ( −→ v )[ x/ idx f ( −→ v ) ( f ( y ))] | x ∈
V ( G )) }} .
Now applying Eq.(1) with (a) and (b), we can derive that ∀ f ∈ F ( t +1)
( G, G ′ , ˜ v , ˜ v ′ ) , wl ( t +1) k
( G, −→ v ) =
wl ( t +1) k
( G, f ( −→ v )) .
Hence the statement is correct for any t .
Using Lemma.6 and the conclusion ( ∀ p ∈ perm[k], wl ( t +1)
k [Equation]
proved in the first part of the proof of Theorem.1, we proved ∀ t, mwl ( t ) k
( G [Equation]
˜ v ) =
[Equation]
A.5 Algorithm of extracting
( k, c )( ≤ ) sets and constructing the super-graph of ( k, c )( ≤ ) -S ET WL
Here we present the
Input: Input graph G , the list O t, ≤ c containing all supernodes with t nodes and ≤ c number of components, the list N t, ≤ c with N t, ≤ c
[ i ] be the number of components of G [ O t, ≤ c [ i ]] .
Output: O t +1 , ≤ c , N t +1 , ≤ c , B t containing edges between O t, ≤ c and O t +1 , ≤ c O t +1 , ≤ c , N t +1 , ≤ c , B t ← [] , [] , [] for all ˆ s , n in zip( O t, ≤ c , N t, ≤ c ) do
N 1 ← 1 -st hop neighbors of G [ˆ s ]
N > 1 ← ( V ( G ) \
ˆ s ) \ N 1 for all m in
N 1 do ▷ Number of components doesn’t change.
ˆ q ← ˆ s + { m } Append ˆ q into O t +1 , ≤ c , n into N t +1 , ≤ c
Append edge (ˆ s , ˆ q ) into B t
▷ We change the edge to (ˆ s , ˆ q , m ) for Algorithm.
end for if n < c then ▷ Creating an additional component.
for all m in N > 1 do ˆ q ← ˆ s + { m } Append ˆ q into O t +1 , ≤ c , n + 1 into N t +1 , ≤ c
Append edge (ˆ s , ˆ q ) into B t
▷ We change the edge to (ˆ s , ˆ q , m ) for Algorithm.
end for end if end for
Remove repeated elements inside O t +1 , ≤ c , with corresponding mask M
Apply mask M to N t +1 , ≤ c return O t +1 , ≤ c , N t +1 , ≤ c , B t
To get the full supergraph S k,c -swl , we need to apply Algorithm.1 k − 1 times sequentially.
A.6 Algorithm of connecting supernodes to their connected components
In Sec.4.3 we proved that for a supernode with multi-component induced subgraph (call it multicomponent supernode), the color initialization can be greatly sped up given knowing its each component’s corresponding supernode.
Hence we need to build the bipartite graph (call it components graph) between single-component supernodes and multi-component supernodes.
We present the algorithm of constructing these kind of connections sequentially. That is, given knowing the connections between single-component supernodes and all multi-component supernodes with ≤ t number of nodes, to build the connections between single-component supernodes and all multi-component supernodes with ≤ t +1 {+number of nodes. To build the full bipartite graph for ( k, c )( ≤ ) sets, we need to conduct the algorithm k − 1 times sequentially. Algorithm 
+}
Constructing the bipartite between single multi + Input: { B i } ti =1 , { O i, ≤ c } t +1 i =1 , { N i, ≤ c } t +1 i =1 from Algorithm.1 (with blue lines applied), dictionary D ≤ t with each key being a multi-component supernode with ≤ t nodes, and value being a list of its corresponding single-component supernodes.
Output: D ≤ t +1 D ≤ t +1 ← D ≤ t for all ˆ q in O t +1 , ≤ c do
Get all edges T = { (ˆ s i , ˆ q ) , m i } in B t with ˆ q inside, and let j ← arg max i N t, ≤ c [ˆ s i ] if N t, ≤ c [ˆ s j ] =
= N t +1 , ≤ c [ˆ q ] then ▷
No increasing of components
D ≤ t +1 [ˆ q ] ← [] for all ˆ p in D ≤ t [ˆ s j ] do
Assume k
= | ˆ p | , i.e. the number of nodes inside.
Traverse B k to find the supernode (ˆ p , m j ) if Found it and N k +1 , ≤ c
[(ˆ p , m j )]
=
= 1 then
Append (ˆ p , m j ) to D ≤ t +1 [ˆ q ] else
Append ˆ p to D ≤ t +1 [ˆ q ] end if end for else ▷ N t, ≤ c [ˆ s j ] = N t +1 , ≤ c [ˆ q ] − 1 D ≤ t +1 [ˆ q ] ← D ≤ t [ˆ s j ] + [ m j ] end if end for return D ≤ t +
For the clear of presentation we use for-loops inside Algorithm.2, while in practice we use matrix operations to eliminate for-loops.
A.7 Dataset Details
[Table]
A.8 Experimental Setup
Due to limited time and resource, we highly restrict the hyperparameters and fix most of hyperparameters the same across all models and baselines to ensure a fair comparison.
This means the performance of ( k, c )( ≤ ) -S ET GNN ∗ reported in the paper is not the best performance given that we didn’t tune much hyperparameters.
Nevertheless the performance still reflects the theory and designs proposed in the paper, and we postpone studying the SOTA performance of ( k, c )( ≤ ) -S ET GNN ∗ to future work.
To be clear, we fix batch size to 128, the hidden size to 128, the number of layers of Base GNN to 4, and the number of layers of ( k, c )( ≤ ) -S ET GNN ∗ (the number of iterations of ( k, c )( ≤ ) -S ET WL) to be 2 (we will do ablation study over it later).
This hyperparameters configuration is used for all datasets.
We have run the baseline GINE over many datasets, and we tune the number of layers from [4,6] and keep other hyperparameters the same as ( k, c )( ≤ ) -S ET GNN ∗ .
For all other baselines, we took the performance reported in [63].
For all datasets except QM9, we follow the same configuration used in [63].
For QM9, we use the dataset from PyG and conduct regression over all 19 targets simultaneously.
To balance the scale of each target, we preprocess the dataset by standardizing every target to a Gaussian distribution with mean 0 and standard derivation 1.
The dataset is randomly split with ratio 80%/10%/10% to train/validation/test sets (with a fixed random state so that all runs and models use the same split).
For every graph in QM9, it contains 3d positional coordinates for all nodes, and we use them to augment edge features by using the absolute difference between the coordinates of two nodes on an edge for all models.
Notice that our goal is not to achieve SOTA performance on QM9 but mainly to verify our theory and effectiveness of designs.
We use Batch Normalization and ReLU activation in all models.
We use Adam optimizer with learning rate 0.001 in all experiments for optimization.
We repeat all experiments three times (for random initialization) to calculate mean and standard derivation.
All experiments are conducted on
V100 and RTX-A6000 GPUs.
A.9 Graph Property Dataset Results
[Table]
A.10 Ablation Study over Number of Layers
[Table]
The table shows that increasing the number of bidirectional message passing ( t in Eq.(13) and
Eq.(14) ) always increase the performance, which is aligning with the fact that increasing number of layers always increases expressivity.
A.11 Computational Footprint on QM
[Figure]
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] In Appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Will add later into the URL (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In Appendix (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
In Appendix 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]