Beyond 1-WL with Local Ego-Network Encodings
Abstract
Identifying similar network structures is key to capture graph isomorphisms and learn representations that exploit structural information encoded in graph data.
This work shows that ego-networks can produce a structural encoding scheme for arbitrary graphs with greater expressivity than the Weisfeiler-Lehman (1-WL) test.
We introduce I GEL , a preprocessing step to produce features that augment node representations by encoding ego-networks into sparse vectors that enrich
Message Passing (MP) Graph Neural Networks (GNNs) beyond 1-WL expressivity.
We describe formally the relation between I GEL and 1-WL, and characterize its expressive power and limitations.
Experiments show that I GEL matches the empirical expressivity of state-of-the-art methods on isomorphism detection while improving performance on seven GNN architectures.
Introduction
Novel approaches for representation learning on graph structured data have appeared in recent years [1].
Graph neural networks can efficiently learn representations that depend both on the graph structure and node and edge features from large-scale graph datasets.
The most popular choice of architecture is the Message Passing Graph Neural Network (MP-GNN).
In MP-GNNs, a node is represented by iteratively aggregating local feature ‘messages’ from its neighbors.
Despite being succesfully applied in a wide variety of domains [2–6], there is a limit on the representational power of MP-GNNs provided by the computationally efficient Weisfeiler-Lehman (1-WL) test for checking graph isomorphism [7, 8].
Establishing this connection has lead to a better theoretical understanding of the performance of MP-GNNs and many possible generalizations [9–13].
To improve the expressivity of MP-GNNs, recent methods have extended the vanilla messagepassing mechanism is various ways.
For example, using higher order k -vertex tuples [8] leading to k -WL generalizations, introducing relative positioning information for network vertices [14], propagating messages beyond direct neighborhoods [15], using concepts from algebraic topology [16], or combining sub-graph information in different ways [17–25].
All aforementioned approaches (which we review in more detail in Appendix A) improve expressivity by extending MP-GNNs architectures, often evaluating on standarized benchmarks [26–29].
However, identifying the optimal approach on novel domains remains unclear and requires costly architecture search.
In this work, we show that incorporating simple ego-network encodings already boosts the expressive power of MP-GNNs beyond the 1-WL test, while keeping the benefits of efficiency and simplicity.
We present I GEL , an I nductive G raph E ncoding of L ocal information, which in its basic form extends node attributes with histograms of node degrees at different distances.
The I GEL encodings can be computed as a pre-processing step irrespective of model architecture.
Theoretically, we formally prove that the I GEL encoding is no less expressive than the 1-WL test, and provide examples that show that it is more expressive than 1-WL.
We also identify expressivity upper-bounds for graphs that are indistinguishable using state of the art methods.
Experimentally, we asses the performance of seven model architectures enriched with I GEL encodings on five tasks and ten graph data sets, and find that it consistently improves downstream model performance.
I GEL : Ego-Networks As Sparse Inductive Representations
Given a graph G =
( V, E ) , we define n = | V | and m = | E | , d G ( v ) is the degree of a node v in G and d max is the maximum degree.
For u, v ∈ V , l G ( u, v ) is their shortest distance, and diam ( G ) = max( l G ( u, v ) | u, v ∈ V ) is the diameter of G .
Double brackets {{·}} denote a lexicographicallyordered multi-set, E αv ⊆
G is the α -depth ego-network centered on v , and N αG ( v ) is the set of neighbors of v in G up to distance α , i.e., N αG ( v ) =
{ u | u ∈ V ∧ l G ( u, v ) ≤ α } .
Algorithm 1 shows the 1-WL test, where hash maps a multi-set to an equivalence class shared by all nodes with matching multi-set encodings after a 1-WL iteration.
The output of 1-WL is N n —mapping each node to a color, bounded by n distinct colors if each node is uniquely colored.
k -higher order variants of the WL test (denoted k -WL) operate on k -tuples of vertices, such that colors are assigned to k -vertex tuples.
If two graphs G 1 , G 2 are not distinguishable by the k -WL test (that is, their coloring histograms match), they are k -WL equivalent—denoted G 1 ≡ k − WL G 2 .
Due to the hashing step, 1-WL does not preserve distance information in the encoding, and minor changes in the structure of the network (removing one edge) may dramatically change node-level representations.
I GEL addresses both limitations, improving expressivity in the process.
2.1 The I GEL Algorithm
Intuitively, I GEL encodes a vertex v with the multi-set of ordered degree sequences at each distance within E αv .
As such, I GEL is a variant of the 1-WL algorithm shown in Algorithm 1, executed for α steps with two modifications.
First, the hashing step is removed and replaced by computing the union of multi-sets across steps ( ∪ ); second, the iteration number is explicitly introduced in the representation—with the output multi-set e αv shown in Algorithm 2.
be used as vertex features, the multi-set be represented as a sparse vector I GEL vec ( v ) , where the frequency of a pair of distance and degree δ is contained on index i
= λ · ( d max + 1) +
Degrees greater than d max are capped to d max , with the resulting vector shown in Figure 1:
[Equation] [Equation]
, for λ · ( d max + 1) + δ = i.
G 1 =
( V 1 , E 1 ) and G 2 =
( V 1 , E 1 ) are I GELequivalent for α if the sorted multi-set containing node representations is the same for G 1 and G 2 :
G
[Equation]
≡
[Equation]
2 ⇐⇒ {{ e αv 1 : ∀ v 1 ∈ V 1 }} = {{ e αv 2 : ∀ v 2 ∈ V 2 }} .
Input: G =
( V, E ) 1: c 0 v :=
hash ( {{ d G ( v ) }} ) ∀ v ∈ V 2: do 3:
c i +1 v :=
hash ( {{ c iu : ∀ u ̸ = v u ∈ N 1 G ( v ) }} ) 4: while c iv ̸
= c i − 1 v
Output: c iv : V → N
Input: G =
( V, E ) , α : N 1: e 0 v := {{ (0 , d G ( v )) }} ∀ v ∈ V 2: for i := 1 ; i += 1 until i =
α do 3: e iv :=
(cid:83) ( e i − 1 v , 4: {{ ( i, d E αG ( v ) ( u )) 5: ∀ u ∈ N αG ( v ) | l G ( u, v ) = i }} ) 6: end for
Output: e αv : V → {{ ( N , N ) }}
Space complexity.
I GEL ’s worst case space complexity is O
( α · n · d max ) , conservatively assuming that every node will require d max parameters at every α depth from the center of the ego-network.
Time complexity.
For I GEL , each vertex has d max neighbors where the α iterations imply traversing through geometrically larger ego-networks with ( d max ) α vertices, upper bounded by m .
Thus
I GEL ’s
time complexity follows O ( n · min( m, ( d max ) α )) , with O ( n · m ) when α ≥ diam ( G ) , when implemented as BFS, for which we provide further details in Appendix F.
Theoretical and Experimental Findings
First, we analyze I GEL ’s expressive power with respect to 1-WL and recent improvements.
Second, we measure the impact of I GEL as an additional input to enrich existing MP-GNN architectures.
3.1 Expressivity: Which Graphs are I GEL -Distinguishable?
In this section, we discuss the increased expressivity of I GEL with respect to 1-WL, and identify expressivity upper-bounds for graphs that are indistinguishable under M ATLANG and the 3-WL test.
b a b a a a a a b b c a c d a d c c d d 0   0 1(a) (b) 0 1 1 (c) (a)
(d)
— Relationship to 1-WL.
I GEL is more powerful than 1-WL following Lemma 1 (as shown in Appendix C) and Lemma 2 (as shown below):
Lemma 1.
I GEL is at least as expressive as 1WL.
That is, for any pair of graphs G 1 , G 2 such that G 1 ̸≡ 1-WL G 2 are distinguished by 1-WL in k iterations, G 1 ̸≡ α
I GEL G 2 for α = k + 1 .
Lemma 2.
There exist graphs that I GEL can distinguish but that 1-WL cannot distinguish.
Proof.
For example, any two d -regular graphs with equal cardinality are indistinguishable by 1-WL (as shown in Appendix B), but I GEL can distinguish some of them.
A graph is d -regular if all nodes have degree d .
Figure 2 shows two d -regular graphs where 1-WL (Algorithm 1) assigns the same color for all nodes, stabilizing after one iteration.
In contrast, I GEL ( α = 1)
counts different frequencies for four structures
hence distinguishing between
both graphs.
— Expressivity upper bounds.
We identify an expressivity upper bound for I GEL , which fails to distinguish S trongly R egular G raphs with equal parameters (Theorem 1, see Appendix E for details):
Definition 1.
A n -vertex d -regular graph is strongly regular—denoted SRG ( n, d, β, γ ) —if adjacent vertices have β vertices in common, and non-adjacent vertices have γ vertices in common.
Theorem 1.
I GEL cannot distinguish SRG s when n , d , and β are the same, and between any value of γ (same or otherwise).
I GEL when α = 1 can only distinguish SRGs with different values of n , d , and β , while I GEL when α = 2 can only distinguish SRGs with different values of n and d .
Our findings show that I GEL is a powerful permutation-equivariant representation (see Lemma 3), capable of distinguishing 1-WL equivalent graphs as shown in Figure 2—which as cospectral graphs, are known to be distinguishable in strictly more powerful M ATLANG sub-languages than 1-WL [12].
Additionally, the upper bound on SRG s is a hard ceiling on expressivity since SRG s are known to be indistinguishable by 3-WL [31].
I GEL shares the experimental upper-bound of expressivity of methods like GNNML3 [20].
Furthermore, I GEL can provably reach comparable expressivity on
SRG s with respect to sub-graph methods implemented within MP-GNN architectures (see Appendix E, subsection E.1), such as Nested GNNs [19] and GNN-AK [23], which are known to be not less powerful than 3-WL, and ESAN when using ego-networks with root-node flags as subgraph sampling policy (EGO+) [24], which is as powerful as the 3-WL test on SRG s (see [24], Prop. 3).
3.2 Experimental Evaluation
We evaluate I GEL α vec ( v ) to produce architecture-agnostic vertex features on five tasks: graph classification, isomorphism detection, graphlet counting, link prediction, and node classification.
Experimental Setup.
We introduce I GEL on graph classification, isomorphism and graphlet counting, comparing the performance of adding/removing I GEL on six GNN architectures following [20].
We also evaluate I GEL on link prediction against transductive baselines, and on node classification as additional feature in MLPs without message-passing.
Appendix G describes experimentation details.
Notation.
The following formatting denotes significant (as per paired t-tests) positive , negative , and insignificant differences after introducing I GEL , with the best results per task / dataset underlined.
[Table] [Table]
— Graph Classification.
Table 1 shows graph classification results on the TU molecule data sets [27].
We evaluate differences in mean accuracy between 10 runs with (left) / without (right) I GEL .
We do not tune network hyper-parameters and establish statistical significance through paired t-tests, with p < 0 .
(*) and p < 0 . 0001 ( ⋄ ).
Our results show that I GEL in the Mutag and Proteins data sets improves the performance of all MP-GNN models, including GNNML3.
On the Enzymes and PTC data sets, results are mixed: excluding GNNML3, I GEL either significantly improves accuracy (on
MLPNet, GCN, and GIN on Enzymes), or does not have a negative impact on performance.
Table 2 compares I GEL results from Table 1 with reported results for state-of-the-art 1-WL expressive
MP-GNNs.
Results are comparable to I GEL except where highlighted in color.
Overall, when comparing I GEL and best performing baselines, only differences with ID-GNN on Proteins are statistically significant (using p -value threshold p < 0 .
01 , where ID-GNN shows p = 0 . 009 ).
— Isomorphism Detection & Graphlet Counting.
Adding I GEL to the sixmodels in Table 1 on the
EXP [32] isomorphism detection yields significant improvements: all GNN models distinguish all non-isomorphic yet 1-WL equivalent EXP graph pairs with I GEL vs. 50% accuracy without I GEL (i.e. random guessing).
Additionally, I GEL significantly improves GNN graphlet-counting performance on three graphlet types in the RandomGraph data set [33].
We provide further details in Appendix H.
— Link Prediction & Node Classification.
We test I GEL on edge / node level tasks to assess its use as a baseline in non-GNN settings.
On a transductive link prediction task, we train DeepWalk [34] style embeddings of I GEL encodings rather than node identities on the Facebook and CA-AstroPh graphs [35].
I GEL -derived embeddings outperform transductive baselines on link prediction as an edge-level binary classification task, measuring 0.976 vs. 0.968 (Facebook) and 0.984 vs. 0.937 (CAAstroPh) AUC comparing I GEL vs. node2vec [36].
On multi-label node classification on PPI [37], we train an MLP (e.g. no message passing) with node features and I GEL encodings.
Our MLP shows better micro-F1 (0.850) when α = 1 than MP-GNN architectures such as GraphSAGE (0.768, as reported in [38]), but underperforms compared to a 3-layer GAT ( 0.973 micro-F1 from [38]).
— Experimental Summary.
Introducing I GEL yields comparable performance to state-of-the-art methods without architectural modifications—including when compared to strong baseline models focused on WL expressivity such as GNNML3, GSN, Nested GNNs, ID-GNN, GNN-AK or ESAN.
Furthermore, I GEL achieves this at a lower computational cost, in comparison for instance with
GNNML3, which requires a O ( n 3 ) eigen-decomposition step to introduce spectral channels.
Finally,
I GEL can also be used in transductive settings (link prediction) as well as node-level tasks (node classification) and outperform strong transductive baselines or enhance models without messagepassing, such as MLPs.
As such, we believe I GEL is an attractive baseline with a clear relationship to the 1-WL test that improves MP-GNN expressivity without the need for costly architecture search.
Conclusions
We presented I GEL , a novel vertex representation algorithm on unattributed graphs allowing MPGNN architectures to go beyond 1-WL expressivity.
We showed that I GEL is related and more expressive than the 1-WL test, and formally proved an expressivity upper bound on certain families of Strongly Regular Graphs.
Finally, our experimental results indicate that introducing I GEL in existing MP-GNN architectures yield comparable performance to state-of-the-art methods, without architectural modifications and at lower computational costs than other approaches.
Author Contributions
Nurudin Alvarez-Gonzalez: Conceptualization, Methodology, Software, Investigation, Formal analysis, Writing - Original Draft; Andreas Kaltenbrunner: Validation, Supervision, Writing - Review &
Editing; Vicenç Gómez: Resources, Validation, Supervision, Writing - Review & Editing.
Acknowledgements
Vicenç Gómez has received funding from “la Caixa” Foundation (ID 100010434), under the agreement
LCF/PR/PR16/51110009.
Andreas Kaltenbrunner acknowledges support from Intesa Sanpaolo
Innovation Center.
The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
A Relation with Previous Works
In the past few years, many different approaches have been developed for improving the expressivity of MP-GNNs.
Here we review the works that are more related to I GEL .
For a more detailed overview augmented message-passing methods for graph representation learning, see [39].
In k -hop MP-GNNs ( k -hop)
[15] the authors propose to propagate messages beyond immediate vertex neighbors, effectively using ego-network information in the vertex representation.
Their proposed algorithm requires to extract neighborhood sub-graphs and to perform message-passing on each sub-graph, which has an exponential cost on the number of hops k both at pre-processing and at each iteration (epoch).
In contrast, I GEL only requires a single pre-processing step that can cached once computed.
Distance Encoding GNNs ( DE-GNN ) [25] also propose to improve MP-GNN by using extra node features by encoding distances to a subset of p nodes.
The features obtained by DE-GNN are similar to IGEL when conditioning the subset to size p = 1 and using a distance encoding function with k = α .
However, these features are not strictly equivalent to the I GEL features, since within the ego-network the node degrees can be smaller than the actual degrees, and they are more expensive to compute.
DE-GNN needs to compute power iterations of the entire adjacency matrix, which is more expensive and does not exploit network sparsity.
Graph Substructure Networks ( GSNs ) [18] incorporate hand-crafted topological features by counting local substructures (such as the presence of cliques or cycles).
GSNs require expert knowledge on what features are relevant for a given task and depart from the original MP-GNN in their architecture.
We show that I GEL reaches comparable performance using a general encoding for ego-networks and without altering the original message-passing mechanism.
GNNML3 [20] proposes a way to perform message passing in spectral-domain with a custom frequency profile.
While this approach achieves good performance on graph classification, it requires an expensive preprocessing step for computing the eigendecomposition of the graph Laplacian and
O ( k ) -order tensors to achieve k -WL expressiveness, which does not scale to large graphs.
More recently, a series of methods formulate the problem of representing vertices or graphs as aggregations over sub-graphs.
The sub-graph information is pooled or introduced during messagepassing at an additional cost that varies depending on each architecture.
Consequently, they require generating the subgraphs (or effectively replicating the nodes of every subgraph of interest) and pay an additional overhead due to the aggregation.
These approaches include Ego-GNNs [22], Nested
GNNs ( NGNNs ) [19], GNN-as-Kernel ( GNN-AK ) [23], Identity-aware GNNs ( ID-GNNs ) [21].
Ego-GNNs perform message-passing over the ego-graphs of all the nodes in a graph, and subsequently perform aggregation.
They provide empirical evidence of a superior expressive power than the classical 1-WL.
ID-GNNs embed each node incorporating identity information in the GNN and apply rounds of heterogeneous message passing; NGNNs perform a two-level GNN using rooted sub-graphs and consider a graph as a bag of sub-graphs; GNN-AK uses a very similar idea, but as the authors describe, it sets the number of layers to the number of iterations of 1-WL;
Compared to all these methods I GEL only relies on an initial pre-processing step based on distances and degrees without having to run additional message passing iterations.
Despite its simplicity, I GEL performs competitively, as we show in Table 2.
Equivariant Subgraph Aggregation Networks ( ESAN ) [24] also propose to encode bags of subgraphs and show that such an encoding can lead to a better expressive power.
In the case of the egonetworks policy (EGO), ESAN is strongly related with I GEL .
Interestingly, as described in concurrent work [40], the implicit encoding of the pairwise distance between nodes, plus the degree information which can be extracted via aggregation are fundamental to provide a theoretical justification of ESAN .
In this work, we directly consider distances and degrees in the ego-network, explicitly providing the structural information encoded by more expressive GNN architectures.
These similarities may explain why the performance of both methods is comparable, as shown in Table 2.
B 1-WL Expressivity and Regular Graphs
Remark 1 shows that 1-WL, as defined in Algorithm 1, is unable of distinguishing d -regular graphs:
Remark 1.
Let G 1 and G 2 be two d -regular graphs such that | V 1 | = | V 2 | .
Tracing Algorithm 1, all vertices in V 1 , V 2 share the same initial color due to d -regularity: ∀ v ∈ V 1 (cid:83) V 2 ; c
0 v = hash
( {{ d }} ) .
After the first color refinement iteration, consider the colorings of G 1 and G 2 :
[Equation]
Since ∀ v 1 ∈ V 1 , v 2 ∈ V 2 ; d = |N 1 G 1 ( v 1 ) |
= |N 1 G 2 ( v 2 ) | , substituting c 1 v 1 , c 1 v 2 in the next iteration step yields {{ hash ( c 1 v 1 ) : ∀ v 1 ∈ V 1 }} = {{ hash ( c 1 v 2 ) : ∀ v 2 ∈ V 2 }} .
Thus, on any pair of d -regular graphs with equal cardinality, 1-WL stabilizes after one iteration produces equal colorings for all nodes on both graphs—regardless of whether G 1 and G 2 are isomorphic, as Figure 2 shows.
C I GEL is At Least As Powerful as 1-WL
In this section we formally prove Lemma 1, i.e. that I GEL is at least as expressive as 1-WL.
For this, we consider a variant of 1-WL which removes the hashing step.
This modification can only increase the expressive power of 1-WL but makes it possible to directly compare with the encodings generated by I GEL .
Intuitively, after k color refinement iterations, 1-WL considers nodes at k hops from each node, which is equivalent to running I GEL with α = k
+ 1 , so that the ego-networks include the information of all nodes that 1-WL would visit.
Lemma 1.
I GEL is at least as expressive than 1-WL.
That is, for any pair of graphs G 1 , G 2 such that G 1 ̸≡ 1-WL G 2 are distinguished by 1-WL in k iterations, G 1 ̸≡ α
I GEL G 2 for α = k + 1 .
Proof of Lemma 1: For convenience, let c i +1 v = {{ c iv ; c iu ∀ u ∈ N 1 G ( v ) | u ̸ = v }} be a recursive definition of Algorithm 1 where hashing is removed and c 0 v = {{ d G ( v ) }} .
Since the hash is no longer computed, the nested multi-sets contain strictly the same or more information as in the traditional 1-WL algorithm.
For I GEL to be less expressive than 1-WL, it must hold that there exist two graphs G 1 =
( V 1 , E 1 ) and G 2 =
( V 2 , E 2 ) such that G 1 ̸≡ 1-WL G 2 while G 1 ≡ α I GEL G 2 .
Let k be the minimum number of color refinement iterations such that ∃ v 1 ∈ V 1 and ∀ v 2 ∈ V 2 , c kv 1 ̸ = c kv 2 .
We define an equally or more expressive variant of the 1-WL test 1-WL * where hashing is removed, such that c kv 1 = {{{{ ... {{ d G ( v 1 ) }} , {{ d G ( u ) ∀ u ∈ N 1 G 1 ( v 1 ) }} ... }}}} , nested up to depth k .
To avoid nesting, the multi-set of nested degree multi-sets can be rewritten as the union of degree multi-sets by introducing an indicator variable for the iteration number where a degree is found:
[Equation]
(cid:26)(cid:26) (0 ,
[Equation]
[Equation]
(cid:27)(cid:27) (cid:91) (cid:26)(cid:26) (1 , d G ( v 1 )); (1 , d G ( u )) ∀ u
[Equation]
N 1 G ( v 1 ) (cid:27)(cid:27) (cid:91) (cid:26)(cid:26) (2 , d G ( v 1 ));
(2 , d G ( u )) ∀ u ∈ N 1 G ( v 1 ); (2 , d G ( w ))
∀ w ∈ N 1 G ( u ) (cid:27)(cid:27) (cid:91) ...
At each step i , we introduce information about nodes up to distance i of v 1 .
Furthermore, by construction, nodes will be visited on every subsequent iteration—i.e. for c 2 v 1 , we will observe (2 , d G ( v 1 ))
exactly d G ( v 1 ) + 1 times, as all its d G ( v 1 ) neighbors u ∈ N 1 G ( v ) encode the degree of v 1 in c 1 u .
The flattened representation provided by 1-WL * is still equally or more expressive than 1-WL, as it removes hashing and keeps track of the iteration at which a degree is found.
Let I GEL -W be a less expressive version of I GEL that does not include edges between nodes at k +hops of the ego-network center.
Now consider the case in which c kv 1 ̸ = c kv 2 from 1-WL * , and let α = k + 1 so that I GEL -W considers degrees by counting edges found at k to k + 1 hops of v 1 and v 2 .
Assume that G 1 ≡ α I GEL -W
G 2 .
By construction, this means that {{ e α v 1 : ∀ v 1 ∈ V 1 }} = {{ e α v 2 : ∀ v 2 ∈ V 2 }} .
This implies that all degrees and iteration counts match as per the distance indicator variable at which the degrees are found, so c kv 1 = c kv 2 which contradicts the assumption c kv [-
-]1 ̸ = c kv 2 and therefore implies that also G 1 ≡ 1-WL* G 2 .
Thus, G 1 ≡ α
I GEL -W
G 2 ⇒ G 1 ≡ 1-WL* G 2 for α = k
+ 1 and also G 1 ̸≡ 1-WL*
G 2 ⇒ G 1 ̸≡ α
I GEL -W
G 2 . Therefore by extension I GEL is at least as expressive as 1-WL.
D I GEL is Permutation Equivariant
Lemma 3.
Given any v ∈ V for G =
( V, E ) and given a permuted graph G ′
=
( V ′ , E ′ ) of
G produced by a permutation of node labels π : V → V ′ such that ∀ v ∈ V ⇔ π ( v ) ∈ V ′ , ∀ ( u, v )
∈
E ⇔ ( π ( u ) , π ( v ))
∈ E ′ .
The I GEL representation is permutation equivariant at the graph level
[Equation]
The I GEL representation is permutation invariant at the node level
[Equation]
Proof.
Note that e αv in Algorithm 2 can be expressed recursively as:
[Equation]
Since I GEL only relies on node distances l G ( · , · ) and degree nodes d G ( · ) , and both l G ( · , · ) and d G ( · ) are permutation invariant (and the node level) and equivariant (at the graph level) functions, the I GEL representation is permutation equivariant at the graph level, and permutation invariant at the node level.
E Proof of Theorem
In this appendix, we provide proof for Theorem 1, showing that I GEL cannot distinguish certain pairs of SRG s with equal parameters of n (cardinality), d (degree), β (shared edges between adjacent nodes), and γ (shared edges between non-adjacent nodes).
Let {{·}} d denote a repeated multi-set with d -times the cardinality of the items in the multi-set, and let e αG = {{ e αv : ∀ v ∈ V }} be short-hand notation for the I GEL encoding of G , defined as the sorted multi-set containing I
GEL encodings of all nodes in G .
Lemma 4.
For any G = SRG ( n, d, β, γ ) , diam ( G ) ≤ 2 .
Note that by definition of SRGs, n affects cardinality while d and β control adjacent vertex connectivity at 1-hop.
For γ , we have to consider two cases: when γ ≥ 1 and when γ = 0 : — Let γ ≥ 1 : by definition, ∀ u, v ∈ V s.t.
( u, v ) / ∈ E, ∃ w ∈ V s.t.
( u, w ) ∈ E ∧ ( v, w ) ∈ E .
Thus, ∀ ( u, v ) ∈ E, l G ( u, v ) = 1 and ∀ ( u, v ) / ∈ E, l G ( u, v )
= 2 .
— Let γ = 0 : ∀ u, v ∈ V , if ( u, v ) / ∈ E then ∄ w ∈ V s.t.
( u, w ) ∈ E ∧ ( v, w )
∈ E as w is in common between u and v .
Then, ∀ u, v, w ∈ V s.t.
( u, v ) ∈ E, ( u, w )
∈
E ⇔ ( v, w ) ∈ E —hence, only nodes and their neighbors can be in common.
Thus: ∀ u, v ∈ V s.t.u ̸ = v, l G ( u, v ) = 1 .
Given both scenarios, we can conclude that for any γ ∈ N , ∀ u, v ∈ V, l G ( u, v ) ≤ 2 and thus diam ( G ) ≤ 2 .
Lemma 5.
For any finite graph G , there is a finite range of α ∈ N where I GEL encodings distinguish between different values of α .
For values of α larger than the diameter of the graph (that is, α ≥ diam ( G ) ), it holds that e αv =
e α +1 v as E αv =
E
α +1 v = G .
Proof.
Per Lemma 4 and Lemma 5, SRG s have a maximum diameter of two, and I GEL encodings are equal for all α ≥ diam ( G ) .
Thus, given G = SRG ( n, d, β, γ ) , only α ∈ { 1 , 2 } produce different encodings of G .
It can be shown that e αv can only distinguish different values of n , d and β , and
I GEL 2 enc can only distinguish values of n and d :
— Let α = 1 : ∀ v ∈ V, E 1 v =
( V
, E ′ )
s.t. V ′
=
N
G ( v ) .
Since G is d -regular, v is the center of E 1 v
and has
d
-neighbors.
By SRG ’s
definition
, the
d
neighbors
of v
have
β
shared
neighbors
with
v
each, plus an edge with v .
Thus, for any SRG s G 1 , G 2 where n 1 = n 2 ,
d
1 =
d
2 , and β 1 = β 2 ,
e 1 G
= e
1 G
produce equal encodings by expanding e 1 v in Algorithm 2:
[Equation]
— Let α = 2 : ∀ v ∈ V, E 2 v =
G as ∀ u ∈ V, u ∈ N 2 G ( v ) when diam ( G ) ≤ 2 .
G is d -regular, so ∀ v ∈ V, d = d E 2 v ( v ) =
d G ( v ) .
Thus, for any SRG s G 1 , G 2 s.t. n 1 = n 2 and d 1 = d 2 , e 2 G 1 =
e 2 G 1 , containing n equal e 2 v encodings by expanding Algorithm 2:
[Equation]
Thus, I GEL cannot distinguish pairs of SRG s when n , d , and β are the same, and between any value of γ (equal or different between the pair).
I GEL when α = 1 can only distinguish SRG s with different values of n , d , and β , while I GEL when α = 2 can only distinguish SRG s with different values of n and d .
We note that it is straightforward to extend I GEL so that different values of γ can be distinguished.
We explore one possible extension in subsection E.1.
E.1 Improving Expressivity on the γ Parameter
I GEL as presented is unable to distinguish between any values of γ in SRG s.
However, I GEL can be trivially extended to distinguish between pairs of SRG s, bringing parity with methods such as the
EGO+ policy in ESAN, NGNNs and GNN-AK.
Intuitively, I GEL is unable to distinguish γ because its ( λ, δ ) tuples are unable to represent relationships between vertices at different distances (i.e. the γ parameter).
The structural feature definition may be extended to compute the degree between ‘distance layers’ in the sub-graphs, addressing this pitfall.
This means modifying e iv in Algorithm 2:
[Equation]
where:
[Equation]
and d pG ( u, v ) generalizes d G ( u ) to count edges of u at a relative distance p of v in G =
( V, E ) :
[Equation]
It can be shown that this definition of e iv is strictly more powerful distinguishing at SRG s following an expansion of Algorithm 2 with α = 2 :
[Equation]
Proof.
For any G = SRG ( n, d, β, γ ) , ∀ v ∈ V , l E 2 v ( v, v ) =
and there are d edges towards its neighbors—thus the root is encoded as (0 , 0 , d ) .
Each neighbor is at l E 2 v ( u, v ) = 1 , with β edges among each other, and γ with vertices not adjacent to v —thus (1 , β, γ ) , where d = 1 + β + γ .
By definition, every vertex w ∈ V s.t.
( u, w ) / ∈ E has γ neighbors shared with v , and d neighbors overall.
Per Lemma 4, the maximum diameter of G is two, hence l E 2 v ( v, w ) = 2 and for any w , the representation is (2 , d − γ, 0) .
F Implementing I GEL through Breadth-First Search
The idea behind the I GEL encoding is to represent each vertex v by compactly encoding its corresponding ego-network E αv at depth α .
The choice of encoding consists of a histogram of vertex degrees at distance d ≤ α , for each vertex in E αv .
Essentially, I GEL runs a Breadth-First Traversal up to depth α , counting the number of times the same degree appears at distance d ≤ α .
The algorithm shown in Algorithm 2 showcases I GEL and its relationship to the 1-WL test.
However, in a practical setting, it might be preferable to implement I GEL through Breadth-First Search (BFS).
In Algorithm 3, we show one such implementation that fits the time and space complexity described in section 2:
Input: v ∈ V, α ∈ N 1: toVisit :=
[ ] ▷ Queue of nodes to visit.
2: degrees := { } ▷ Mapping of nodes to their degrees.
3: distances :
= { v : 0 } ▷ Mapping of nodes to their distance to v 4: while toVisit ̸ = ∅ do 5: u :=
toVisit.dequeue () 6: currentDistance := distances [ u ] 7: currentDegree :=8: for w ∈ u .neighbors () do 9:
if w / ∈ distances then 10: distances [ w ] := currentDistance + 1 ▷ w is a new node 1-hop further from v .
11: end if 12: if distances [ w ] ≤ α then 13: currentDegree := currentDegree +▷ Count edges only within α -hops.
14: if w / ∈ degrees then ▷ Enqueue if w has not been visited.
15: toVisit.append ( w ) 16: end if 17: end if 18: end for 19: degrees [ u ] := currentDegree ▷ u is now visited: we know its degree and distance to v .
20: end while 21: e αv = {{ ( distances [ u ] , degrees [ u ]) ∀ u ∈ degrees.keys () }} ▷ Produce the multi-set of (distance, degree) pairs for all visited nodes.
Output: e αv
( N , N )
N
Due to how we structure BFS to count degrees and distances in a single pass, each edge is processed twice—once for each node at end of the edge.
It must be noted that when processing every v ∈ V , the time complexity is O ( n · min( m, ( d max ) α )) .
However, the BFS implementation is also embarrassingly parallel, which means that it can be distributed over p processors with O ( n · min( m, ( d max ) α ) /p ) time complexity.
G Experimental Settings And Procedures
In this section, we provide additional details of our experimental setting.
We summarize our datasets and tasks in Table 6.
On graph-level tasks, we introduce I GEL encodings concatenated to existing vertex features into the best performing model configurations found by [20] without any hyper-parameter tuning (e.g. number of layers, hidden units, choice pooling and activation functions).
We evaluate performance differences with and without I GEL on each task, data set and model on 10 independent runs, measuring statistical significance of the differences through paired t-tests.
On vertex and edge-level tasks, we report best performing configurations after hyper-parameter search.
Each configuration is evaluated on 5 independent runs.
We provide a breakdown of the best performing hyper-parameters in the section below.
G.1 Hyper-parameters and Experiment Details Graph Level Experiments
We reproduce the benchmark of [20] without modifying model hyper-parameters for the tasks of
Graph Classification, Graph Isomorphism Detection, and Graphlet Counting.
For classification tasks, the 6 models in Table 2 are trained on binary / categorical cross-entropy objectives depending on the task.
For Graph Isomorphism Detection, we train GNNs as binary classification models on the binary classification task on EXP [32], and identify isomorphisms by counting the number of graph pairs
[Table]
for which randomly initialized MP-GNN models produce equivalent outputs on Graph8c 23 .
For the graphlet counting regression task on the RandomGraph data set [33], we train models to minimize
Mean Squared Error (MSE) on the normalized graphlet counts 4 for five types of graphlets.
On all tasks, we experiment with α ∈ { 1 , 2 } and optionally introduce a preliminary linear transformation layer to reduce the dimensionality of I GEL encodings.
For every setup, we execute the same configuration 10 times with different seeds and compare runs introducing I GEL or not by measuring whether differences on the target metric
(e.g. accuracy or MSE) are statistically significant as shown in Table 1 and Table 2.
In Table 3, we provide the value of α that was used in our experimental results.
Our results show that the choice of α depends on both the task and model type.
We believe these results may be applicable to subgraph-based MP-GNNs, and will explore how different settings, graph sizes, and downstream models interact with α in future work.
Reproducibility – We provide an additional repository with our changes to the original benchmark, including our modelling scripts, metadata, and experimental
results 5 . Vertex and Edge-level Experiments
In this section we break down the best performing hyper-parameters on the Edge (link prediction) and Vertex-level (node classification) experiments.
Link Prediction – The best performing hyperparameter configuration on the Facebook graph including α = 2 , learning t = 256 component vectors with e = 10 walks per node, each of length s = 150 and p = 8 negative samples per positive for the self-supervised negative sampling.
Respectively on the arXiv citation graph, we find the best configuration at α = 2 , t = 256 , e = 2 , s = 100 and p = 9 .
Node Classification – We analyze both encoding distances α ∈ { 1 , 2 } .
Other I GEL hyper-parameters are fixed after a small greedy search based on the best configurations in the link prediction experiments.
For the MLP model, we perform greedy architecture search, including number of hidden units, activation functions and depth.
Our results show scores averaged over five different seeded runs with the same configuration obtained from hyperparameter search.
The best performing hyperparameter configuration on the node classification is found with α =on t = 256 length embedding vectors, concatenated with node features as the input layer forepochs in a 3-layer MLP using ELU activations with a learning rate of 0.005.
Additionally, we apply 100 epoch patience for early stopping, monitoring the F1-score on the validation set.
Reproducibility – We provide a replication folder in the code repository for the exact configurations used to run the experiments 6 .
H Extended Results on Isomorphism Detection and Graphlet Counting
In this section we summarize additional results on isomorphism detection and graphlet counting.
H.1 Isomorphism Detection
We provide a detailed breakdown of isomorphism detection performance after introducing I GEL in Table 4, complimenting our summary on subsection 3.2.
[Table]
— Graph8c.
On the Graph8c dataset, introducing
I GEL significantly reduces the amount of graph pairs erroneously identified as isomorphic for all MP-GNN models, as shown in Table 4.
Furthermore, I GEL allows a linear baseline employing a sum readout function over input feature vectors, then projecting onto a 10-component space, to identify all butnon-isomorphic pairs compared to the erroneous pairs
GCNs (4196 errors) or GATs (1827 errors) can identify without I GEL .
Additionally, we find that all Graph8c graphs can be distinguished if the I GEL encodings for α = 1 and α = 2 are concatenated.
We do not explore the expressivity of combinations of α in this work, but hypothesize that concatenated encodings of α may be more expressive.
— Empirical Results on Strongly
Regular Graphs.
We also evaluate I GEL on SR25 7 , which contains
Strongly Regular graphs with 25 vertices, known to be indistinguishable by 3-WL.
With SR25, we validate Theorem 1.
[20] showed that no models in our benchmark distinguish any of the 105 non-isomorphic graph pairs in SR25.
As expected from Theorem 1,
I GEL does not improve distinguishability.
H.2 Graphlet Counting
We evaluate I GEL on a (regression) graphlet 8 counting task.
We minimize Mean Squared Error (MSE) on normalized graphlet counts 9 .
Table 5 shows the results of introducing I GEL in 5 graphlet counting tasks on the RandomGraph data set [33].
Stat sig.
differences ( p < 0 . 0001 ) shown in bold green , with best (lowest MSE) per-graphlet results underlined.
[Table]
Introducing I GEL improves counting performance on triangles, tailed triangles and the custom 1-WL graphlets proposed by [20].
Star graphlets can be identified by all baselines
[Table]
and
I GEL only produces statistically significant improvements for the Linear baseline.
Notably, the Linear baseline plus
I GEL outperforms MP-GNNs without
I GEL for star, triangle, tailed triangle and custom 1-WL graphlets.
By introducing I GEL on the MLP baseline, it outperforms all other models including GNNML3 on the triangle, tailedtriangle and custom 1-WL graphlets.
[Table]
Since Linear and MLP baselines do not use message passing, we believe raw
I GEL encodings may be sufficient to identify certain graph structures even with simple linear models.
For all graphlets except 4-cycles, introducing I GEL yields performance similar to GNNML3 at lower pre-processing and model training/inference costs, as I GEL obviates the need for costly eigen-decomposition and can be used in simple models only performing graph-level readouts without message passing.
T a b l e 6 :
O v e r v
i e w o f t
h e g r a ph s u s e d i n t
h
e
e xp e r
i m e n t s .
W e s ho w t
h e a v e r a g e nu m b e r
o f v e r ti ce s (
A vg . n ) , e dg e s (
A vg .
m ) , nu m b e r o f g r a ph s , t a r g e t t a s k , ou t pu t s h a p e , a nd s p l it s ( w h e n a pp li ca b l e ) .
A vg .
n
A vg .
m
N u m .
G r a ph s
T a s k
O u t pu t
Sh a p e
Sp li t s ( T r a i n /
V a li d /
T e s t)
E n z y m e s 32 .62 .
M u lti - c l a ss
G r a ph
C l a ss .
6 ( m u lti - c l a s s p r ob a b iliti e s ) 9 -f o l d /f o l d ( G r a ph s ,
T r a i n / E v a l )
M u t ag 17 .39 .
B i n a r y
G r a ph
C l a ss .
2 ( b i n a r y c l a ss p r ob a b iliti e s ) 9 -f o l d /f o l d ( G r a ph s ,
T r a i n / E v a l )
P r o t e i n s 39 .72 .
B i n a r y
G r a ph
C l a ss .
2 ( b i n a r y c l a ss p r ob a b iliti e s ) 9 -f o l d /f o l d ( G r a ph s ,
T r a i n / E v a l )
P T
C 25 .51 .
B i n a r y
G r a ph
C l a ss .
2 ( b i n a r y c l a ss p r ob a b iliti e s ) 9 -f o l d /f o l d ( G r a ph s ,
T r a i n / E v a l )
G r a ph 8 c 8 .28 .
N on - i s o m o r ph
i s m
D e t ec ti on
N / A N / A
E X
P C l a ss i f y 44 .111 .
B i n a r y
C l a ss .
( p a i r w i s e g r a ph d
i s ti
ngu
i s h a b ilit y )
( non - i s o m o r p
h
i c g r a ph p
a
i r p r ob a b ilit y )
G r a ph p a i r s / /
S R
N on - i s o m o r ph
i s m
D e t ec ti on
N / A N / A
R a nd o m G r a ph 18 .62 .
R e g r e ss i on ( G r a ph l e t
C oun ti ng ) 1 ( g r a ph l e t c ou n t s )
G r a ph s / /
A r X i v
A S T
R OP H
B i n a r y
C l a ss .
( L i nk
P r e d i c ti on ) 1 ( e dg e p r ob a b i lit y )
R a ndo m
l y
s a m p l e d e dg e s 50 % t
r a i n /% t e s t
F a ce b oo k
B i n a r y
C l a ss .
( L i nk
P r e d i c ti on ) 1 ( e dg e p r ob a b i lit y )
R a ndo m
l y
s a m p l e d e dg e s 50 % t
r a i n /% t e s t
PP I  .
M u lti - l a b e l
V e r t e x C l a ss .
( b i n a r y c l a ss p r ob a b iliti e s )
G r a ph s 20 / 2 /