Weisfeiler and Leman Go Walking: Random Walk Kernels Revisited
Abstract
Random walk kernels have been introduced in seminal work on graph learning and were later largely superseded by kernels based on the Weisfeiler-Leman test for graph isomorphism.
We give a unified view on both classes of graph kernels.
We study walk-based node refinement methods and formally relate them to several widelyused techniques, including Morgan’s algorithm for molecule canonization and the
Weisfeiler-Leman test.
We define corresponding walk-based kernels on nodes that allow fine-grained parameterized neighborhood comparison, reach Weisfeiler-Leman expressiveness, and are computed using the kernel trick.
From this we show that classical random walk kernels with only minor modifications regarding definition and computation are as expressive as the widely-used Weisfeiler-Leman subtree kernel but support non-strict neighborhood comparison.
We verify experimentally that walk-based kernels reach or even surpass the accuracy of Weisfeiler-Leman kernels in real-world classification tasks.
Introduction
Machine learning with graph-structured data has various applications, from bioinformatics to social network analysis to drug discovery and has become an established research field.
Graph kernels [26, 5] and graph neural networks (GNNs) [44] are two widely-used techniques for learning with graphs, the latter of which has recently received significant research interest.
Technically, various methods of both categories exploit the link between graph data and linear algebra by representing graphs by their (normalized) adjacency matrix.
Such methods are often defined or can be interpreted in terms of walks.
On the other hand, the Weisfeiler-Leman heuristic for graph isomorphism testing has attracted great interest in machine learning [33, 34].
This classical graph algorithm has been studied extensively in structural graph theory and logic, and its expressive power, i e., its ability to distinguish non-isomorphic graphs, is well understood [3].
The Weisfeiler-Leman method turned out to be suitable to derive powerful and efficient graph kernels [40].
Moreover, it is closely related to
GNNs [45, 31, 14], allowing to establish links, e.g., to results from logic [15].
Several results have paved the way for these insights.
The seminal work by Kersting et al.
[22] links algebraic methods and the combinatorial Weisfeiler-Leman algorithm showing that it can be understood in terms of walks and simulated by iterated matrix products.
A comprehensive view of the expressive power of linear algebra on graphs was recently given by Geerts [13].
GNNs are closely related to algebraic methods but involve activation functions and learnable weights.
Morris et al.
[31] has shown that (i) the expressive power of GNNs is limited by the Weisfeiler-Leman algorithm, and (ii) that GNN architectures exist that reach this expressive power for suitable weights.
Independently, Xu et al.
[45] obtained the same result by using injective set functions computable by multilayer perceptrons [46].
Most recently, Geerts et al.
[14] has proven that already the early GCN layer-based architectures [23] achieve the expressive power of the Weisfeiler-Leman test with only a minor modification already mentioned in the original publication.
Random walk kernels [12, 21] have been the starting point of a long line of research in graph kernels, see [26, 5] for details.
An extension of random walk kernels is based on so-called tree patterns, which may contain repeated nodes just like walks [37, 28].
A cornerstone in the development of graph kernels is the Weisfeiler-Leman graph kernel [40], which in contrast to tree pattern kernels, relies on entire node neighborhoods instead of all subsets.
This restriction allowed manageable feature vectors and led to a significant improvement in terms of both running time and classification accuracy.
Several other kernels based on the Weisfeiler-Leman test have been proposed, e.g., combining it with optimal assignments [24] or the Wasserstein distance [42].
Graph kernels inspired the development of neural modules based on random walks and Weisfeiler-Leman labels [27].
Chen et al.
[7] introduced multilayer-kernels conceptually similar to GNN layers and relates random walk and Weisfeiler-Leman kernels in this framework for graphs, where the node labels induce a unique bijection between neighbors.
Dell et al.
[8] recently established the equivalence in expressive power between tree patterns formalized as tree homomorphism counts and the Weisfeiler-Leman method.
On this basis, homomorphism counts were quickly adapted for graph classification [35].
The classical Weisfeiler-Leman kernel [40] as well as the works [27, 7, 35] enumerate graph features and generate (approximate) feature vectors.
In contrast, the classical random walk kernel is computed using the kernel trick and thus operates implicitly in a high dimensional features space.
The advantages and disadvantages of fixed-size graph embeddings are subject to recent research [25, 2].
Nowadays, random walk kernels are widely abandoned, while
Weisfeiler-Leman kernels remain an important baseline method performing competitively on many real-world datasets [26, 5].
Our contribution.
We formally relate random walks in labeled graphs and the Weisfeiler-Leman method and link random walk kernels and the Weisfeiler-Leman subtree kernel.
Starting from a combinatorial perspective, we consider walk-based label refinement and relate it to the WeisfeilerLeman method.
We extend the concept to fine-grained pairwise node similarities that satisfy the kernel properties and are computed via algebraic techniques using the kernel trick similarly to random walk kernels.
From the node similarities, we derive a node-centric walk kernel on graphs.
This kernel generalizes the classical random walk kernel [12] and allows grouping of walks by their start node.
We show that grouping significantly improves the expressive power of random walk kernels and, with a minor modification, reaches the expressive power of the Weisfeiler-Leman method.
Kernel parameters control the grouping of walks and allow to interpolate between random walk and Weisfeiler-Leman type kernels.
We verify our theoretical results on real-world graphs for which our approach reaches high accuracies, surpassing the Weisfeiler-Leman subtree kernel in some cases.
Fundamentals
We aim at establishing formal links between random walks, Weisfeiler-Leman refinement, and corresponding kernels.
We review the basics of these methods and refer the reader to recent surveys for further details [26, 34].
We proceed by introducing the notation used.
2.1 Definitions and notation
An (undirected) graph G is a pair ( V, E ) with nodes V and edges E ⊆ V 2 , where ( u, v ) ∈
E ⇔ ( v, u ) ∈ E .
A graph may be endowed with a (node) labeling σ : V → Σ .
The labels Σ can be arbitrary structures such as (multi)sets but are typically represented by or mapped to integers.
We denote sets by {·} and multisets allowing multiple instances of the same element by {{·}} .
We refer to the neighbors of a node u by N ( u ) = { v ∈ V | ( u, v ) ∈ E } .
Two graphs G =
( V, E ) and H =
( V ′ , E ′ ) are isomorphic , written G ≃ H , if there is a bijection ψ : V → V ′
such that ( u, v ) ∈
E ⇔ ( ψ ( u ) , ψ ( v ))
∈
E ′ for all u, v in V .
For labeled graphs, additionally σ ( v )
=
σ ( ψ ( v )) must hold for all v ∈ V ; for two graphs with roots r ∈ V and r ′ ∈ V ′ , ψ ( r ) =
r ′ must be satisfied.
The map ψ is called isomorphism .
An isomorphism of a graph to itself is called automorphism .
A labeling σ is said to refine a labeling η , denoted by σ ⊑ η , if for all nodes u and v , σ ( u )
=
σ ( v ) implies η ( u )
=
η ( v ) ; we write σ ≡ η if σ ⊑ η and η ⊑ σ .
A label refinement is a sequence of labelings ( σ (0) , σ (1) , . . . )
such that σ ( i ) refines σ ( i − 1) for all i > 0 .
A function k : X × X
→ R is a kernel on
X , if there is a Hilbert space ( H , ⟨· , ·⟩ ) and a map ϕ : X → H , such that k ( x, y ) =
⟨ ϕ ( x ) , ϕ ( y ) ⟩ for all x, y ∈ X .
A kernel on the set of graphs is a graph kernel .
We denote by k δ the Dirac kernel with k δ ( x, y )
=
if x = y and 0 otherwise.
We write vectors and matrices in bold, using capital letters for the latter, and denote the column vector of ones by 1 .
2.2 Random walk kernels
The classical random walk kernel proposed by Gärtner et al.
[12] compares two graphs with discrete labels by counting their common walks , i.e., the pairs of walks with the same label sequence.
Let W i ( G ) be the set of all walks of length i in a graph G .
For a walk w = ( v 1 , v 2 , . . . ) let σ ( w )
= ( σ ( v 1 ) , σ ( v 2 ) , . . . )
denote its label sequence.
Definition 2.1 ( ℓ -step random walk kernel) .
Let λ
i ∈
R ≥ 0 for i ∈ { 0 , . .
. , ℓ } be a sequence of weights, the ℓ -step random walk kernel is
[Equation]
Random walk kernels can be computed based on a product graph using the kernel trick.
Definition 2.2 (Direct product graph) .
For two labeled graphs G = ( V, E ) and H =
( V ′ , E ′ )
the direct product graph G × H is the graph ( V , E ) with V =
{ ( v, v ′ )
∈
V × V ′ | σ ( v ) =
σ ( v ′ ) } and
E = (cid:8) (( u, u ′ ) , ( v, v ′ ))
∈
V 2 | ( u, v ) ∈
E ∧ ( u ′ , v ′ )
∈
E ′ (cid:9) .
Lemma 2.3.
There is a bijection between W i ( G × H ) and { ( w, w ′ )
∈
W i ( G ) × W i ( H )
| σ ( w ) =
σ ( w ′ ) } , ∀ i ≥ 0 .
Using Lemma 2.3, the ℓ -step random walk kernel is computed from the adjacency matrix A × of
G × H by matrix products or iterated matrix-vector multiplication as
[Equation]
ℓ (cid:88) k
= λ k
A k × (cid:35) ij = |V| (cid:88)
i = (cid:34) ℓ (cid:88) k
= λ k w ( k ) (cid:35) i with w ( k ) =
A × w ( k − 1) and w (0) = 1 .
The random walk kernel K × is defined as the limit
K ∞× , where λ = ( λ 0 , λ 1 , . . . ) is chosen such that the sum converges [12, 41].
A well-studied instantiation is the geometric random walk kernel using weights
λ i
= γ i , i ∈ N , where γ < 1 a with a ≥ min { ∆ − , ∆ + } and ∆ + / − the maximum in- and outdegree of G × H guarantees convergence [12].
For this choice, the kernel can be computed in polynomial-time applying an analytical expression based on matrix inversion.
Efficient methods of computation have been studied extensively [43, 20].
However, theoretical and empirical results suggest that the benefit of using walks of infinite length as in the geometric random walk kernel is limited [41, 25].
Random walk kernels can be extended to score the similarity of walks instead of requiring their labels to match exactly, making them suitable for graphs with arbitrary node and edge attributes compared by dedicated kernels [17, 43, 25].
The accuracy and running time of random walk kernels can be improved by refining vertex labels in a preprocessing step leading to sparse product graphs [29, 25, 19].
2.3 Weisfeiler-Leman refinement and graph kernels
The Weisfeiler-Leman method, often referred to as color refinement , initially assigns labels wl (0) ( v )
=
σ ( v ) to all nodes v (or uniform labels if graphs are unlabeled) and then iteratively computes new labels for all v in V as
[Equation]
Convergence is reached when wl ( i ) ( u ) =
wl ( i ) ( v ) ⇐⇒ wl
( i +1) ( u ) = wl
( i +1)
( v ) holds for all u, v ∈ V
and we denote the corresponding stable labeling by wl ( ∞ ) .
In practice, nested multisets are avoided by applying an injective mapping to integers after every iteration.
For all j ≥ i , wl ( j ) ⊑ wl ( i ) holds.
The stable partition wl ( ∞ ) is refined by the orbit labeling assigning two nodes u , v the same label if and only if there is an automorphism ψ with ψ ( u ) =
ψ ( v ) .
In graph isomorphism testing, only the final stable labeling wl ( ∞ ) is of interest, while in graph kernels, only the first few iterations are used.
The Weisfeiler-Leman subtree kernel [40] K ℓ WL computes the first ℓ iterations and maps graphs to feature vectors counting the number of nodes in the graph for every label, which is equal to
[Equation]
a f b d i j g h e c a d j f i b f f f c h
j e i g e e e
T [2 , f ]
T [2 , e ]
G (a) Graph and unfolding trees
[Figure]
Comparing nodes by walks
We propose to compare the nodes of one or multiple graphs according to the walks originating at them.
We start with discrete walk labelings and characterize their ability to distinguish nodes relating the approach to classical methods and results for unlabeled graphs, namely Morgan’s algorithm, walk partitions and Weisfeiler-Leman refinement.
Then we focus on the pairwise node comparison and propose walk kernels on nodes, which are extended to reach the expressiveness of the Weisfeiler-Leman method.
3.1 Walk labelings and their properties
We consider graphs with discrete labels.
For a node v in a graph G , we define W i ( v ) as the set of walks of length i in G starting in v .
We associate with every node v a label
rw ( i ) ( v ) representing the label sequences of length i walks originating at v , i.e., rw ( i ) ( v ) = {{ σ ( w ) | w ∈ W
i ( v ) }} .
This does not yield a label refinement, since rw ( i +1) not necessarily refines rw ( i ) .
For a counterexample consider nodes f and g of the graph G in Figure 1(a), for which rw (1) ( f ) ̸
= rw (1) ( g ) , but rw (2) ( f ) =
rw (2) ( g ) .
To guarantee that two nodes once labeled different will obtain different labels in all subsequent iterations, we consider all walks up to a given length ℓ and define W + ℓ ( v ) =
(cid:83) ℓi =
W i ( v ) .
Definition 3.1 ( ℓ -walk label) .
The ℓ -walk label of a node v is rw ( ℓ )+ ( v ) =
{{ σ ( w ) | w ∈ W + ℓ ( v ) }} .
Since rw ( i )+ ( v ) ⊆
rw ( i +1)+ ( v ) for all v in V and walks of different length have different label sequences, it immediately follows that rw ( i +1)+ ⊑
rw ( i )+ .
We say that two nodes u and v are ℓ -walk indistinguishable if rw ( ℓ )+ ( u ) = rw ( ℓ )+ ( v ) .
We call u and v walk indistinguishable if they are ∞ -walk indistinguishable.
3.1.1 Relation to Morgan’s algorithm
Morgan [30] proposed a method to generate canonical representations for molecular graphs.
To reduce ambiguities each node v is endowed with its extended connectivity ec ( v ) .
Let G =
( V, E ) be a (molecular) graph, initially we assign ec (1) ( v ) =
deg( v ) to every node v in V , where deg( v ) is the degree of v .
Then, the values ec ( i ) ( v ) are computed iteratively for i ≥ 2 and all nodes v in V as
[Equation]
until |{ ec ( i ) ( v ) | v ∈ V }| ≤ |{ ec ( i +1) ( v ) | v ∈ V }| .
Then, ec ( i ) ( v ) is the final extended connectivity of the node v .
For all i ≥ 1 the extended connectivity values ec ( i ) are equivalent to the row (or column) sums of the i th power of the adjacency matrix, i.e., A i 1 , which gives the number of walks of length i starting at each node [38, 10].
We relate the extended connectivity to walk labelings.
Proposition 3.2.
For i ≥ 1 and all graphs, rw ( i ) ⊑ ec ( i ) holds with rw ( i ) ≡ ec ( i ) in the case of unlabeled graphs.
Proof.
From the definition of walk labelings we conclude | rw ( i ) ( u ) | = ec ( i ) ( u ) for all u in V .
Hence, rw ( i ) ( u ) = rw ( i ) ( v ) =
⇒ ec ( i ) ( u ) = ec ( i ) ( v ) .
Vice versa, ec ( i ) ( u ) = ec ( i ) ( v ) is a necessary condition for rw ( i ) ( u )
=
rw ( i ) ( v ) .
For unlabeled graphs, all walks have the same label sequence and rw ( i ) contains multiple instance of a single label sequence for all nodes, proving the equivalence.
3.1.2 Relation to walk partitions
Powers and Sulaiman [36] studied the walk partition of a graph with adjacency matrix A , which is obtained from the n × ℓ matrix W ( ℓ ) =
[ 1 , A 1 , A 2 1 , . . .
, A ℓ 1 ] .
We define a node labeling wp , where the i th node is assigned its row vector, i.e., wp ( i ) =
W ( ℓ ) i, · .
With Proposition 3.2 we directly obtain the following result.
Proposition 3.3.
For i ≥ 0 and all graphs, rw ( i )+ ⊑ wp
( i ) holds with rw ( i )+
≡ wp ( i ) in the case of unlabeled graphs.
For unlabeled graphs, it is known that wl ( ∞ ) ⊑ wp ( ∞ ) and both coincide for many graphs [36].
3.1.3 Relation to the Weisfeiler-Leman method
The Weisfeiler-Leman labels encode neighborhood patterns.
The unfolding tree T [ n, v ] of depth n at the node v is defined recursively as the tree with root v and children N ( v ) .
Each child w ∈ N ( v ) is the root of the unfolding tree T [ n − 1 , w ] and T [0 , w ]
=
( { w } , ∅ ) .
The labels of the original graph are preserved in the unfolding tree.
Lemma 3.4 (Folklore) .
For ℓ ≥ 0 and nodes u and v , wl ( ℓ ) ( u ) = wl ( ℓ ) ( v ) ⇐⇒ T [ ℓ, u ] ≃ T [ ℓ, v ] .
As unfolding trees also encode walks we can relate Weisfeiler-Leman and walk labelings.
For an unfolding tree T [ ℓ, v ] , let PL ( T [ ℓ, v ]) denote the set of unique paths from the root v to a leaf.
Lemma 3.5.
Let ℓ ≥ 0 and v a node, then PL ( T [ ℓ, v ])
=
W ℓ ( v ) .
Proof.
For a walk ( v = v 0 , . . . , v ℓ ) we have v i +1 ∈ N ( v i ) for all 0 ≤ i <
ℓ and hence a path in
PL ( T [ ℓ, v ]) exists.
Vice versa, every path in PL ( T [ ℓ, v ]) is a walk of length ℓ starting at v .
Since T [ ℓ, u ] ≃ T [ ℓ, v ] implies PL ( T [ ℓ, u ]) =
PL ( T [ ℓ, v ]) for all i ∈ { 0 , . .
. , ℓ } we conclude.
Proposition 3.6.
Let ℓ ∈ N 0 , then wl ( ℓ ) ⊑
rw ( ℓ )+ ⊑ rw ( ℓ ) .
Vice versa, for ℓ = 0 or ℓ = 1 , PL ( T [ ℓ, u ]) =
PL ( T [ ℓ, v ]) implies T [ ℓ, u ] ≃ T [ ℓ, v ] , since the unfolding tree consists of a single node or is a star graph, respectively.
Proposition 3.7.
It holds wl (0) ≡ rw (0) + ≡ rw (0) and wl (1) ≡ rw (1) + ≡ rw (1) .
Figure 1 illustrates the relations and shows an example, where the refinement relation is strict.
The nodes e and f are walk indistinguishable but obtain different Weisfeiler-Leman labels after only two refinement steps.
In this sense the walk labeling is weaker than the Weisfeiler-Leman labeling.
This does not necessarily mean that it is less suitable for learning tasks.
We study the difference of the two techniques on common benchmark datasets in Section 5.
3.2 Pairwise node comparison
In the previous section, two walks were considered equal if they exhibit the same label sequence and two nodes have the same walk label if the multisets of walks originating at them are equal.
We define the functions k ℓ and k + ℓ between nodes that measure the similarity based on walks in a more fine-grained manner as
[Equation]
u, v ) with
[Equation]
u
[Equation]
where w = ( u = u 0 , u 1 , . .
. , u ℓ ) , w ′ =
( v = v 0 , v 1 , . .
. , v ℓ ) and k V is a user-defined function measuring the similarity of nodes, e.g., by taking continuous attributes into account.
Assuming that k V is a kernel on nodes, it follows from the concept of convolution kernels and basic closure properties [39]
that k + ℓ again is a kernel.
We denote its feature map by ϕ + ℓ and refer the reader to [25] for the tools to construct feature vectors.
If we assume that k V ( u, v ) =
k δ ( σ ( u ) , σ ( v )) , then a possible feature map ϕ + ℓ takes v to a vector having a component for every possible label sequence s of length at most ℓ and counts the number of walks w starting at v with σ ( w ) =
s , i.e., it is the characteristic vector of the multiset rw ( ℓ )+ ( v ) .
We obtain a new parameterized kernel that allows controlling the strictness of neighborhood comparison.
Definition 3.8 (Generalized ℓ -walk node kernel) .
Let α ∈ R ≥ 0 , the generalized ℓ -walk node kernel is
[Equation]
This is a Gaussian kernel, where the Euclidean distance is substituted by the kernel distance associated with k + ℓ , which yields a valid kernel [16].
The kernel distance can be computed using the kernel trick according to Equation (3).
We proceed by relating the generalized ℓ -walk node kernel to walk labelings.
Proposition 3.9.
For α > 0 and k V = k δ , the equality ˆ k + ℓ ( u, v ; α ) = 1 holds if and only if u and v are ℓ -walk indistinguishable, i.e., rw ( ℓ )+ ( u ) =
rw ( ℓ )+ ( v ) .
Proof.
For α ̸ = 0 , the Gaussian kernel is one if and only if the Euclidean distance is zero, i.e., ϕ + ℓ ( u ) = ϕ
+ ℓ ( v ) .
With the feature map of k + ℓ that maps v to the characteristic vector of the multiset rw ( ℓ )+ ( v ) the result follows immediately.
Other normalized kernels ˜ k satisfying ˜ k
( x, y ) =
if and only if x = y are known [1, 11], but cannot be controlled with a parameter α in a similar way.
For α = 0 , we have ˆ k + ℓ ( u, v ) = 1 .
For α → ∞ , we have ˆ k ℓ ( u, v ) =
unless u and v are ℓ -walk indistinguishable.
In particular, we can relate the generalized ℓ -walk node kernel to walk labelings.
Corollary 3.10.
For α → ∞ , the image of ˆ k + ℓ is { 0 , 1 } and ˆ k + ℓ ( u, v ) =
1 ⇐⇒ rw ( ℓ )+ ( u ) = rw ( ℓ )+ ( v ) .
3.2.1 Computation
Computing walk labelings for labeled graphs by enumerating walks is prohibitive.
We employ a technique based on direct product graphs introduced for the computation of random walk kernels, see Section 2.2.
Using Lemma 2.3 we can compute the generalized ℓ -walk node kernel by counting the walks in the direct product graph with the techniques described above.
Algorithm 1 realizes this approach and computes the kernel for all pairs of nodes having a non-zero value.
The vectors w and w + each have one entry for each node ( u, v ) in the direct product graph, which after the i th iteration stores the value of k i ( u, v ) and k + i ( u, v ) , respectively.
In Line 8 the generalized ℓ -walk node kernel is computed according to Definition 3.8 based on the three values
k
+ i ( u, v ) , k
+ i ( u, u ) and k
+ i ( v, v ) using the kernel trick.
The symmetry of kernels is reflected by symmetries in the direct product graph G × G which can be exploited to speed-up computation.
Line 9 is optional and increases expressivity as discussed in Section 3.2.2.
The method of computation can be extended to incorporate vertex and edge kernels by using a weighted adjacency matrix and initializing w (0) according to the vertex kernel, see [25] for details.
Moreover, the limit ℓ → ∞ can be computed by introducing suitable weights and well-known techniques proposed for random walk kernels, see Section 2.2.
3.2.2 Weisfeiler-Leman expressiveness
We can modify Algorithm 1 to perform a re-encoding in every iteration to achieve Weisfeiler-Leman expressiveness for α → ∞ by adding Line 9 (highlighted in blue).
Proposition 3.11.
For ℓ > 0 and α → ∞ , Algorithm 1 including Line 9 computes ˆ k + WL ℓ with image { 0 , 1 } and ˆ k + WL ℓ
[Equation]
[Figure]
w
[Equation]
w + ← w 3 for
← 1 to ℓ do
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation] [Figure]
[Equation]
[Figure] [Equation] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Proof.
The construction of the direct product graph and Corollary 3.10 with Proposition 3.7 guarantee that the statement is satisfied initially for i = 1 .
For α → ∞ , ˆ K ( i ) uv = 1 if and only if w + uu = w + vv = w + uv in iteration i , and zero otherwise.
Assume that the equivalence holds in iteration i , then the vector w contains ones for the node pairs ( u, v ) with wl ( i ) ( u ) =
wl ( i ) ( v ) and w + uu = w + vv = w + uv .
Hence, in iteration i + 1 after Line 4, w counts for each node pair the number of matching neighbors regarding wl ( i ) .
After adding these values to w + , it directly follows that, if wl ( i +1) ( u ) =
wl
( i +1) ( v ) , then w + uu = w + vv = w + uv and ˆ K
( i +1) uv
= 1 in Line 8.
Vice versa, ˆ K ( i +1) uv
=
if and only if w + uu = w + vv = w + uv , which implies w uu = w vv = w uv .
In combination with σ ( u ) = σ ( v ) from
Definition 2.2, we conclude wl ( i +1) ( u ) = wl
( i +1) ( v ) .
Comparing graphs by walks
We define a graph kernel based on the generalized ℓ -walk node kernel obtained from the Cartesian product of the node sets of the two input graphs.
Its parameters allow controlling the strictness of neighborhood comparison and the importance of walk counts.
Definition 4.1 (Node-centric ℓ -walk graph kernel) .
Given α, β ∈ R ≥ 0 , the node-centric ℓ -walk graph kernel is defined as
[Equation]
The node-centric ℓ -walk graph kernel combines the generalized ℓ -walk node kernel of Definition 3.
with a polynomial of the kernel k i .
It can incorporate continuous attributes via a dedicated node kernel k V , cf.
Equation (1).
We show in the following that Definition 4.1 resembles two widely-used graph kernels from the literature for certain parameter choices.
4.1 Relation to random walk and Weisfeiler-Leman subtree kernels
Random walk kernels, see Section 2.2, are closely related to the node-centric ℓ -walk graph kernel.
Proposition 4.2.
The node-centric ℓ -walk graph kernel for α = 0 and β = 1 is equal to the ℓ -step random walk kernel with λ i = 1 for i ∈ { 0 , . .
. , ℓ } , i.e., K + ℓ ( G, H ; 0 , 1) =
K ℓ × ( G, H ) .
Proof.
The parameter choice guarantees that ˆ k
+ i ( u, v ; α )
k
βi ( u, v ) =
k i ( u, v ) .
Hence, this kernel sums over all pairs of walks ( w, w ′ ) of length less or equal to ℓ .
Since k δ ( σ ( w ) , σ ( w ′ )) = (cid:81)
j k δ ( σ ( u j ) , σ ( v j )) , where w = ( u 0 , u 1 , . . . ) , w ′ =
( v 0 , v 1 , . . . )
, this corresponds to Definition 2.1.
This result also holds for the limit ℓ → ∞ if we assume that walks are adequately down-weighted by their length to guarantee convergence.
In random walk kernels a pair of vertices, that both have a large neighborhood, have a great ability to contribute strongly to the total kernel value, since a large number of walks originates at them.
In the Weisfeiler-Leman subtree kernel the same pair contributes a value bounded by ℓ depending on whether their neighborhood matches exactly.
We hypothesize that this conceptional difference is crucial and allows the Weisfeiler-Leman kernel to outperform previous walk-based kernels.
By setting α sufficiently high and β = 0 , it follows from Corollary 3.10 that the node-centric ℓ -walk graph kernel resembles the Weisfeiler-Leman kernel in this respect, but uses walk labelings instead.
Our discussion of walk labelings and their relation to Weisfeiler-Leman labels suggest that these are only slightly less expressive.
We verify our hypothesis experimentally in Section 5.
Moreover, by applying the iterated re-encoding described in Section 3.2.2, we obtain exactly the same value as the Weisfeiler-Leman subtree kernel for α → ∞ and β = 0 .
Other parameter choices allow for a less strict comparison, which is not possible with the Weisfeiler-Leman subtree kernel.
4.2 Computation
We can compute the node-centric ℓ -walk graph kernel for two graphs G and H by applying Algorithmto their union G ∪ H and plugging the results stored in the two kernel matrices K ( i ) and ˆ K ( i ) into Equation (4).
This can be optimized by exploiting the structural properties of the direct product graph.
Proposition 4.3.
Let G and H be two graphs, then the direct product graph of their union P = G ∪ H is P × P =
( G × G ) ∪ ( G × H )
∪ ( H × G ) ∪
( H × H ) .
Proof.
The result follows from Definition 2.2 with the fact that the vertex and edge sets of G and H are disjoint.
As there are no walks between disconnected components, we can run Algorithm 1 separately on the product graphs G × H , G × G and H × H , ignoring H × G due to symmetry.
Moreover, if we compute the kernel for all pairs of graphs in a dataset D of graphs, we can compute all node self-similarities from product graphs G × G for all G ∈ D once in a preprocessing step.
In this case, the overhead compared to the standard ℓ -step random walk kernel is only minor and is mainly attributed to exponentiation.
Direct computation via the directed product graph is not suitable for large-scale graphs [25].
However, existing acceleration techniques [43, 20, 19] remain applicable.
In particular, for unlabeled graphs (or k V ( u, v ) = 1 for all nodes u, v ) the result can be obtained from the walk counts of the input graphs without generating a product graph.
Experimental evaluation
We experimentally verify the hypotheses that have originated from our theoretical results.
In particular, we aim to answer the following research questions.
Q1 Is the Weisfeiler-Leman subtree kernel more expressive than walk-based kernels regarding their ability to distinguish non-isomorphic graphs on common benchmark datasets?
Q2 Are our walk-based kernels competitive to the state-of-the-art regarding classification accuracy?
Q
Which level of strictness in walk-based neighborhood comparison is most suitable?
5.1 Experimental setup
We describe the kernels under comparison, their parameters and the used datasets.
All experiments were performed on an Intel Xeon E5-2690v4 machine at 2.6GHz with 384 GB of RAM.
5.1.1 Kernels
We implemented the node-centric ℓ -walk kernel (NCW) of Definition 4.1 with ˆ k + i
and
k i computed by
Algorithm 1 without the blue Line 9 and the variant with WL expressiveness including the blue line (NCWWL).
For unlabeled graphs we computed NCW without creating the direct product graph.
[Table] As a baseline we used the node label kernel (VL) and edge label kernel (EL), which are the dot products on node and edge label histograms, respectively, see [41, 26].
For the Weisfeiler-Leman subtree kernel (WL), the ℓ -step random walk kernel (RW) as well as NCW and NCWWL we chose the iteration number and walk length from { 0 , . . .
, 5 } by cross-validation.
For RW, λ i = 1 for i ∈ { 0 , . . . , ℓ } was used.
For NCW and NCWWL, we selected α from { 0 . 01 , 0 . 1 , 1 , 1000 } and β from { 0 , 0 . 5 , 1 } .
We have not included extensions of the WL such as [24, 42], which could also be applied similarly to the node-centric ℓ -walk graph kernel.
In addition we used a graphlet kernel (GL3), the shortest-path kernel (SP) [4], and the Graph Hopper kernel (GH) [9].
GL3 is based on connected subgraphs with three nodes taking labels into account similar to the approach used by Shervashidze et al.
For
SP and GH we used the Dirac kernel to compare path lengths and node labels.
We implemented the node-centric ℓ -walk graph kernel as well as all baselines in Java.
1 We performed classification experiments with the C -SVM implementation LIBSVM [6].
We report mean prediction accuracies and standard deviations obtained by 10 -fold nested cross-validation repeated 10 times with random fold assignment.
Within each fold all necessary parameters were selected by cross-validation based on the training set.
This includes the regularization parameter C and kernel parameters.
5.1.2 Datasets
We tested on widely-used graph classification benchmarks datasets of the T U D ATASETS repository [32] representing graphs from different domains.
M UTAG , NCI1, NCI109 and P TC -F
M represent small molecules, E NZYMES and P ROTEINS are derived from macromolecules, and
C OLLAB and I MDB B IN are social networks.
The datasets define binary graph classification experiments with exception of
E NZYMES and C OLLAB , which are divided into six and three classes, respectively.
All graphs have node labels with exception of the social network graphs.
We removed edge labels, if present, since they are not supported by all graph kernel implementations.
5.2 Results
We discuss our results and research questions.
Q1:
Expressiveness.
We investigate the expressiveness of the ℓ -step random walk kernel (RW ℓ ), the node-centric ℓ -walk graph kernel (NCW ℓ ) with α = 1000 and β = 0 , and the Weisfeiler-Leman subtree kernel (WL ℓ ).
Since the used graph datasets contain duplicates [18], we filtered them such that for each isomorphism class a single representative remains.
We computed the completeness ratio introduced by Kriege et al.
[26], i.e., the fraction of graphs that can be distinguished from all other graphs in the dataset.
Figure 2 confirms that with increasing parameter ℓ all methods become more expressive.
For all datasets we observe that NCW ℓ is clearly more expressive than RW ℓ showing that grouping the walks according to their start node increases the expressive power.
Moreover, NCW ℓ achieves the same values as WL ℓ for all ℓ .
Although we have shown that walk labelings are weaker than
Weisfeiler-Leman labelings, cf.
Figure 1, this does not affect the ability to distinguish the considered real-world graphs.
This confirms that walks provide expressive features when grouped at a node level.
239 151 73 54 1677 32 15 87 15α β
Total 0.0 0.5 1.
M UTAG 0.0 0.5 1.
N CI0.0 0.5 1. N CI0.0 0.5 1.
P TC -F M 0.0 0.5 1.
E NZYMES 0.0 0.5 1.
P ROTEINS 0.0 0.5 1.
I MDB B IN 0.0 0.5 1.
C OLLAB
Q2: Accuracy.
Table 1 shows the resulting accuracies and standard deviations.
For several datasets
WL outperforms RW as expected, e.g., N CI 1, N CI 109 and E NZYMES .
In these cases NCW provides an accuracy close to WL and outperforms RW by a large margin, although both are based on walks and
NCW is obtained with only a minor modification of RW.
This clearly shows that node-centric walks are suitable for obtaining high accuracies.
For E NZYMES , NCW reaches an accuracy higher than WL indicating that a higher expressiveness does not necessarily lead to better generalization.
This suggests that a less strict comparison of neighborhoods can be beneficial for some of the datasets.
NCWWL overall performs comparable to NCW but reaches slight improvements for several datasets.
Q3:
Strict neighborhood comparison.
NCW supports non-strict neighborhood comparison by tuning the parameter α and incorporates walk counts for increasing β leading to a kernel similar to WL and the ℓ -step random walk kernel for the extreme cases.
We investigate the selection of these parameters in the above classification experiments.
For each dataset 10-fold cross-validation was repeated 10 times leading to 100 classification experiments in total, for each of which the hyperparameters were optimized.
The distribution of the selected values for α and β is shown in
Figure 3.
We observe that the choices vary between datasets.
For the social network datasets there is a clear preference for β = 0 , which explains the worse performance of the ℓ -step random walk kernels on these datasets.
For N CI 1 and N CI 109, α ≥ 1 and β ≤ 0 . 5 were preferred leading to kernel values closer to WL, which is in accordance with the accuracy reached.
Of particular interest is the dataset
E NZYMES for which NCW and NCWWL provide the highest accuracies.
Figure 3 shows that for this dataset, in most cases non-strict neighborhood comparison ( α ≤ 1 ) with only a minor influence of walk counts ( β ≤ 0 . 5 ) was selected, a choice not possible with previous kernels.
Further investigation shows that the parameter ℓ = 2 was selected in the majority of the cases for WL, while for NCW, ℓ = 4 with α = 0 .
1 or α = 0 .
01 was selected most frequently.
This indicates that for this dataset it is preferable to take larger neighborhoods into account but to compare them less strictly.
Conclusion and future work
We have contributed to the understanding of walk and Weisfeiler-Leman based methods showing that classical random walk kernels can be lifted to obtain the expressive power of the Weifeiler-Leman subtree kernel, drastically increasing classification accuracy.
While the direct product graph based approach is less efficient than the Weisfeiler-Leman method, its advantage is that we can control the strictness of neighborhood comparison and incorporate node and edge similarities for attributed graphs.
Improving the running time and the application to attributed graphs remains future work.
Our results have implications beyond graphs kernels.
In graph neural networks, the bottleneck problem [2] refers to the observation that node neighborhoods grow exponentially with increasing radius and thus cannot be represented accurately by a fixed-sized feature vector.
This is a possible explanation for the weakness of GNNs in long-range tasks.
The product graph based approach works with infinite-dimensional node embeddings using the kernel trick.
Investigating the bottleneck problem in product graph based GNNs might shed more light into this phenomenon.
Acknowledgements
I would like to thank Floris Geerts for pointing me to the paper on walk partitions by Powers and
Sulaiman [36].
I am grateful to the Vienna Science and Technology Fund (WWTF) for supporting me through project VRG19-009.