E XPRESSIVENESS AND A PPROXIMATION P ROPERTIES OF G RAPH N EURAL N ETWORKS
A BSTRACT
Characterizing the separation power of graph neural networks ( GNN s) provides an understanding of their limitations for graph learning tasks.
Results regarding separation power are, however, usually geared at specific GNN architectures, and tools for understanding arbitrary GNN architectures are generally lacking.
We provide an elegant way to easily obtain bounds on the separation power of GNN s in terms of the Weisfeiler-Leman ( WL ) tests, which have become the yardstick to measure the separation power of GNN s. The crux is to view GNN s as expressions in a procedural tensor language describing the computations in the layers of the
GNN s. Then, by a simple analysis of the obtained expressions, in terms of the number of indexes and the nesting depth of summations, bounds on the separation power in terms of the WL -tests readily follow.
We use tensor language to define Higher-Order Message-Passing Neural Networks (or k - MPNN s), a natural extension of MPNN s. Furthermore, the tensor language point of view allows for the derivation of universality results for classes of GNN s in a natural way.
Our approach provides a toolbox with which GNN architecture designers can analyze the separation power of their GNN s, without needing to know the intricacies of the WL -tests.
We also provide insights in what is needed to boost the separation power of GNN s.
I NTRODUCTION
Graph Neural Networks ( GNN s) (Merkwirth & Lengauer, 2005; Scarselli et al., 2009) cover many popular deep learning methods for graph learning tasks (see Hamilton (2020) for a recent overview).
These methods typically compute vector embeddings of vertices or graphs by relying on the underlying adjacency information.
Invariance (for graph embeddings) and equivariance (for vertex embeddings) of GNN s ensure that these methods are oblivious to the precise representation of the graphs.
Separation power.
Our primary focus is on the separation power of GNN architectures, i.e., on their ability to separate vertices or graphs by means of the computed embeddings.
It has become standard to characterize GNN architectures in terms of the separation power of graph algorithms such as color refinement ( CR ) and k -dimensional Weisfeiler-Leman tests ( k - WL ) , as initiated in Xu et al.
and Morris et al.
Unfortunately, understanding the separation power of any given GNN architecture requires complex proofs, geared at the specifics of the architecture.
We provide a tensor language-based technique to analyze the separation power of general GNN s.
Tensor languages.
Matrix query languages (Brijder et al., 2019; Geerts et al., 2021b) are defined to assess the expressive power of linear algebra.
Balcilar et al.
(2021a) observe that, by casting various GNN s into the MATLANG (Brijder et al., 2019) matrix query language, one can use existing separation results (Geerts, 2021) to obtain upper bounds on the separation power of GNN s in terms of 1 - WL and 2 - WL .
In this paper, we considerably extend this approach by defining, and studying, a new general-purpose tensor language specifically designed for modeling GNN s. As in Balcilar et al.
(2021a), our focus on tensor languages allows us to obtain new insights about GNN architectures.
First, since tensor languages can only define invariant and equivariant graph functions, any GNN that can be cast in our tensor language inherits these desired properties.
More importantly, the separation power of our tensor language is as closely related to CR and k - WL as GNN s are.
Loosely speaking, if tensor language expressions use k + 1 indices, then their separation power is bounded by k - WL .
Furthermore, if the maximum nesting of summations in the expression is t , then t rounds of k - WL are needed to obtain an upper bound on the separation power.
A similar connection is obtained for
CR and a fragment of tensor language that we call “guarded” tensor language.
We thus reduce problem of assessing the separation power of any specific GNN architecture to the problem of specifying it in our tensor language, analyzing the number of indices used and counting their summation depth.
This is usually much easier than dealing with intricacies of CR and k - WL , as casting GNN s in our tensor language is often as simple as writing down their layer-based definition.
We believe that this provides a nice toolbox for GNN designers to assess the separation power of their architecture.
We use this toolbox to recover known results about the separation power of specific GNN architectures such as GIN s (Xu et al., 2019), GCN s (Kipf & Welling, 2017),
Folklore GNN s (Maron et al., 2019b), k - GNN s (Morris et al., 2019), and several others.
We also derive new results: we answer an open problem posed by Maron et al.
(2019a)
by showing that the separation power of Invariant Graph Networks ( k - IGN s ) , introduced by Maron et al.
(2019b), is bounded by ( k − 1) - WL .
In addition, we revisit the analysis by Balcilar et al.
(2021b) of ChebNet (Defferrard et al., 2016), and show that CayleyNet (Levie et al., 2019) is bounded by 2 - WL .
When writing down GNN s in our tensor language, the less indices needed, the stronger the bounds in terms of k - WL we obtain.
After all, ( k − 1) - WL is known to be strictly less separating than k - WL (Otto, 2017).
Thus, it is important to minimize the number of indices used in tensor language expressions.
We connect this number to the notion of treewidth : expressions of treewidth k can be translated into expressions using k + 1 indices.
This corresponds to optimizing expressions, as done in many areas in machine learning, by reordering the summations (a.k.a.
variable elimination).
Approximation and universality.
We also consider the ability of GNN s to approximate general invariant or equivariant graph functions.
Once more, instead of focusing on specific architectures, we use our tensor languages to obtain general approximation results, which naturally translate to universality results for GNN s. We show: ( k + 1) -index tensor language expressions suffice to approximate any (invariant/equivariant) graph function whose separating power is bounded by k - WL , and we can further refine this by comparing the number of rounds in k - WL with the summation depth of the expressions.
These results provide a finer picture than the one obtained by Azizian & Lelarge (2021).
Furthermore, focusing on “guarded” tensor expressions yields a similar universality result for CR , a result that, to our knowledge, was not known before.
We also provide the link between approximation results for tensor expressions and GNN s, enabling us to transfer our insights into universality properties of GNN s. As an example, we show that k - IGN s can approximate any graph function that is less separating than ( k − 1) - WL .
This case was left open in Azizian & Lelarge (2021).
In summary, we draw new and interesting connections between tensor languages, GNN architectures and classic graph algorithms.
We provide a general recipe to bound the separation power of GNN s, optimize them, and understand their approximation power.
We show the usefulness of our method by recovering several recent results, as well as new results, some of them left open in previous work.
Related work.
Separation power has been studied for specific classes of GNN s (Morris et al., 2019; Xu et al., 2019; Maron et al., 2019b; Chen et al., 2019; Morris et al., 2020; Azizian & Lelarge, 2021).
A first general result concerns the bounds in terms of CR and 1 - WL of Message-Passing
Neural Networks (Gilmer et al., 2017; Morris et al., 2019; Xu et al., 2019).
Balcilar et al.
(2021a) use the MATLANG matrix query language to obtain upper bounds on the separation power of various
GNN s. MATLANG can only be used to obtain bounds up to 2 - WL and is limited to matrices.
Our tensor language is more general and flexible and allows for reasoning over the number of indices, treewidth, and summation depth of expressions.
These are all crucial for our main results.
The tensor language introduced resembles sum - MATLANG (Geerts et al., 2021b), but with the added ability to represent tensors.
Neither separation power nor guarded fragments were considered in Geerts et al.
(2021b).
See Section A in the supplementary material for more details.
For universality, Azizian & Lelarge (2021) is closest in spirit.
Our approach provides an elegant way to recover and extend their results.
Azizian & Lelarge (2021) describe how their work (and hence also ours) encompasses previous works (Keriven & Peyr´e, 2019; Maron et al., 2019c; Chen et al., 2019).
Our results use connections between k - WL and logics (Immerman & Lander, 1990; Cai et al., 1992), and CR and guarded logics (Barcel´o et al., 2020).
The optimization of algebraic computations and the use of treewidth relates to the approaches by Aji & McEliece (2000) and Abo Khamis et al. (2016).
B ACKGROUND
We denote sets by {} and multisets by {{}} .
For n ∈ N , n > 0 ,
[ n ] := { 1 , . .
. , n } .
Vectors are denoted by v , w , . .
, matrices by A , B , . . .
and tensors by S , T , . .
Furthermore, v i is the i -th entry of vector v , A ij is the ( i, j ) -th entry of matrix A and S
i denotes the i
=
( i 1 , . .
, i k ) -th entry of a tensor S .
If certain dimensions are unspecified, then this is denoted by a “ : ”.
For example, A i : and
A : j denote the i -th row and j -th column of matrix A , respectively.
Similarly for slices of tensors.
We consider undirected simple graphs G =
( V G , E G , col G ) equipped with a vertex-labelling col G :
V G → R ℓ .
We assume that graphs have size n , so V G consists of n vertices and we often identify
V G with [ n ] .
For a vertex v ∈ V G , N G ( v ) :
= { u ∈ V G | vu ∈ E G } .
We let G be the set of all graphs of size n and let G s be the set of pairs ( G, v ) with G ∈ G and v ∈ V sG .
Note that G = G 0 .
The color refinement algorithm ( CR ) (Morgan, 1965) iteratively computes vertex labellings based on neighboring vertices, as follows.
For a graph G and vertex v ∈ V G , cr ( 0 ) ( G, v ) := col G ( v ) .
Then, for t ≥ 0 , cr ( t +1 ) ( G, v ) :
= ( cr ( t ) ( G, v ) , {{ cr ( t ) ( G, u ) | u ∈ N G ( v ) }} ) .
We collect all vertex labels to obtain a label for the entire graph by defining gcr ( t ) ( G ) :
= {{ cr ( t ) ( G, v ) | v ∈ V G }} .
The k -dimensional Weisfeiler-Leman algorithm ( k - WL ) (Cai et al., 1992) iteratively computes labellings of k -tuples of vertices .
For a k -tuple v , its atomic type in G , denoted by atp k ( G, v ) , is a vector in
R 2 ( k 2 ) + kℓ .
The first (cid:0) k 2 (cid:1) entries are 0 / 1 -values encoding the equality type of v , i.e., whether v i = v j for 1 ≤ i < j ≤ k .
The second (cid:0) k 2 (cid:1) entries are 0 / 1 -values encoding adjacency information, i.e., whether v i v j ∈ E G for 1 ≤ i < j ≤ k .
The last kℓ real-valued entries correspond to col G ( v i ) ∈ R ℓ for 1 ≤ i ≤ k .
Initially, for a graph G and v ∈ V kG , k - WL assigns the label wl ( 0 ) k
( G, v ) :
= atp k ( G, v ) .
For t ≥ 0 , k - WL revises the label according to wl ( t +1 ) k
( G, v ) :
= ( wl ( t ) k
( G, v ) , M ) with
M := (cid:8)(cid:8)(cid:0) atp k +1 ( G, v u ) , wl ( t ) k
( G, v [ u/ 1]) , . .
, wl ( t ) k
( G, v [ u/k ]) (cid:1) | u ∈ V G (cid:9)(cid:9) , where v [ u/i ] := ( v 1 , . .
, v i − 1 , u, v i +1 , . .
, v k ) .
We use k - WL to assign labels to vertices and graphs by defining: vwl ( t ) k
( G, v ) :
= wl
[Equation]
) k
( G, (
[Equation]
. , v )) , for vertex-labellings, and gwl ( t )
k := {{ wl ( t )
k
( G, v ) | v ∈ V kG }} , for graph-labellings.
We use cr ( ∞ ) , gcr ( ∞ ) , vwl ( ∞ ) k , and gwl ( ∞ ) k to denote the stable labellings produced by the corresponding algorithm over an arbitrary number of rounds.
Our version of 1 - WL differs from CR in that 1 - WL also uses information from non-adjacent vertices; this distinction only matters for vertex embeddings (Grohe, 2021).
We use the “folklore” k - WL of Cai et al.
(1992), except Cai et al.
use 1 - WL to refer to CR .
While equivalent to “oblivious” ( k + 1) - WL (Grohe, 2021), used in some other works on GNN s, care is needed when comparing to our work.
Let G be a graph with V G = [ n ] and let σ be a permutation of [ n ] .
We denote by σ ⋆ G the isomorphic copy of G obtained by applying the permutation σ .
Similarly, for v ∈ V k G , σ ⋆ v is the permuted version of v .
Let F be some feature space.
A function f : G 0 → F is called invariant if f ( G ) =
f ( σ ⋆ G ) for any permutation π .
More generally, f : G s → F is equivariant if f ( σ ⋆ G, σ ⋆ v ) =
f ( G, v ) for any permutation σ .
The functions cr ( t ) : G 1 → F and vwl ( t ) k : G 1 → F are equivariant, whereas gcr ( t ) : G 0 → F and gwl ( t ) k : G 0 → F are invariant, for any t ≥ 0 and k ≥ 1 .
S PECIFYING GNN S
Many GNN s use linear algebra computations on vectors, matrices or tensors, interleaved with the application of activation functions or MLP s. To understand the separation power of GNN s, we introduce a specification language, TL , for tensor language, that allows us to specify any algebraic computation in a procedural way by explicitly stating how each entry is to be computed.
We gauge the separation power of GNN s by specifying them as TL expressions, and syntactically analyzing the components of such TL expressions.
This technique gives rise to Higher-Order Message-Passing
Neural Networks (or k - MPNN s), a natural extension of MPNN s (Gilmer et al., 2017).
For simplicity, we present TL using summation aggregation only but arbitrary aggregation functions on multisets of real values can be used as well (Section C.5 in the supplementary material).
To introduce TL , consider a typical layer in a GNN of the form F ′
=
σ ( A · F · W ) , where A ∈ R n × n is an adjacency matrix, F ∈ R n × ℓ are vertex features such that F i : ∈ R ℓ is the feature vector of vertex i , σ is a non-linear activation function, and W ∈ R ℓ × ℓ is a weight matrix.
By exposing the indices in the matrices and vectors we can equivalently write: for i ∈ [ n ] and s ∈ [ ℓ ] :
[Equation]
In TL , we do not work with specific matrices or indices ranging over [ n ] , but focus instead on expressions applicable to any matrix.
We use index variables x 1 and x 2 instead of i and j , replace
A ij with a placeholder E ( x 1 , x 2 ) and F jt with placeholders P t ( x 2 ) , for t ∈ [ ℓ ] .
We then represent the above computation in TL by ℓ expressions ψ s ( x 1 ) , one for each feature column, as follows:
[Equation]
These are pure syntactical expressions.
To give them a semantics, we assign to E a matrix
A ∈ R n × n , to P t column vectors F : t ∈ R n × 1 , for t ∈ [ ℓ ] , and to x 1 an index i ∈ [ n ] .
By letting the variable x 2 under the summation range over 1 , 2 , . . . , n , the TL expression ψ s ( i ) evaluates to F ′ is .
As such, F ′ =
σ ( A · F · W ) can be represented as a specific instance of the above TL expressions.
Throughout the paper we reason about expressions in TL rather than specific instances thereof.
Importantly, by showing that certain properties hold for expressions in TL , these properties are inherited by all of its instances.
We use TL to enable a theoretical analysis of the separating power of GNN s; It is not intended as a practical programming language for GNN s.
Syntax.
We first give the syntax of TL expressions.
We have a binary predicate E , to represent adjacency matrices, and unary vertex predicates P s , s ∈ [ ℓ ] , to represent column vectors encoding the ℓ -dimensional vertex labels.
In addition, we have a (possibly infinite) set Ω of functions, such as activation functions or MLP s.
Then, TL (Ω) expressions are defined by the following grammar:
[Equation]
where op ∈ { = , ̸ = } , x, y are index variables that specify entries in tensors, s ∈ [ ℓ ] , a ∈ R , and f ∈ Ω .
Summation aggregation is captured by (cid:80) x φ .
1 We sometimes make explicit which functions are used in expressions in TL (Ω) by writing TL ( f 1 , f 2 , . . . )
for f 1 , f 2 , . .
in Ω .
For example, the expressions ψ s ( x 1 ) described earlier are in TL ( σ ) .
The set of free index variables of an expression φ , denoted by free ( φ ) , determines the order of the tensor represented by φ .
It is defined inductively: free ( 1 x op y )
=
free ( E ( x, y )) :
= { x, y } , free ( P s ( x )) =
{ x } , free ( φ 1 · φ 2 ) = free ( φ 1 + φ 2 ) :
= free ( φ 1 ) ∪ free ( φ 2 ) , free ( a · φ 1 ) :
= free ( φ 1 ) , free ( f ( φ 1 , . .
. , φ p )) :
= ∪ i ∈
[ p ] free ( φ i ) , and free ( (cid:80) x φ 1 ) :
= free ( φ 1 ) \ { x } .
We sometimes explicitly write the free indices.
In our example expressions ψ s ( x 1 ) , x 1 is the free index variable.
An important class of expressions are those that only use index variables { x 1 , . . .
, x k } .
We denote by TL k (Ω) the k -index variable fragment of TL (Ω) .
The expressions ψ s ( x 1 ) are in TL 2 ( σ ) .
Semantics.
We next define the semantics of expressions in TL (Ω) .
Let G =
( V G , E G , col G ) be a vertex-labelled graph.
We start by defining the interpretation [[ · , ν ]] G of the predicates E , P s and the (dis)equality predicates, relative to G and a valuation ν assigning a vertex to each index variable:
[Equation]
if ν
[Equation]
op ν ( y ) then 1 else
[Equation]
In other words, E is interpreted as the adjacency matrix of G and the P s ’s interpret the vertexlabelling col G .
Furthermore, we lift interpretations to arbitrary expressions in TL (Ω) , as follows:
[Equation]
ν ]] G ) where, ν [ x (cid:55)→ v ] is the valuation ν
but which now maps the index x to the vertex v ∈ V G .
For simplicity, we identify valuations with their images.
For example, [[ φ ( x ) , v ]] G denotes [[ φ ( x ) , x (cid:55)→ v ]] G .
To illustrate the semantics, for each v ∈ V G , our example expressions satisfy [[ ψ s , v ]] G = F ′ vs for F ′ =
σ ( A · F · W ) when A is the adjacency matrix of G and F represents the vertex labels.
k -MPNNs.
Consider a function f : G s → R ℓ : ( G, v ) (cid:55)→ f
( G, v ) ∈ R ℓ for some ℓ ∈ N .
We say that the function f can be represented in TL (Ω) if there exists ℓ expressions φ 1 ( x 1 , . . .
, x s ) , . .
. , φ ℓ ( x 1 , . .
, x s ) in TL (Ω) such that for each graph G and each s -tuple v ∈ V sG :
[Equation]
Of particular interest are k th-order MPNN s (or k - MPNN s) which refers to the class of functions that can be represented in TL k +
(Ω) .
We can regard GNN s as functions f : G s → R ℓ .
Hence, a
GNN is a k - MPNN if its corresponding functions are k - MPNN s.
For example, we can interpret
F ′ =
σ ( A · F · W ) as a function f : G 1 → R ℓ such that f ( G, v ) :
= F ′ v : .
We have seen that for each s ∈ [ ℓ ] , [[ ψ s , v ]] G = F ′ vs with ψ s ∈ TL 2 ( σ ) .
Hence, f ( G, v ) =
([[ ψ 1 , v ]] G , . . .
, [[ ψ ℓ , v ]] G ) and thus f belongs to 1 - MPNN s and our example GNN is a 1 - MPNN .
TL represents equivariant or invariant functions.
We make a simple observation which follows from the type of operators allowed in expressions in TL (Ω) .
Proposition 3.1.
Any function f : G s → R ℓ represented in TL (Ω) is equivariant (invariant if s = 0 ).
An immediate consequence is that when a GNN is a k - MPNN , it is automatically invariant or equivariant, depending on whether graph or vertex tuple embeddings are considered.
S EPARATION P OWER OF T ENSOR L ANGUAGES
Our first main results concern the characterization of the separation power of tensor languages in terms of the color refinement and k -dimensional Weisfeiler-Leman algorithms.
We provide a finegrained characterization by taking the number of rounds of these algorithms into account.
This will allow for measuring the separation power of classes of GNN s in terms of their number of layers.
4. S EPARATION P OWER
We define the separation power of graph functions in terms of an equivalence relation, based on the definition from Azizian & Lelarge (2021), hereby first focusing on their ability to separate vertices.
Definition 1.
Let F be a set of functions f : G 1 → R ℓ f .
The equivalence relation ρ 1 ( F ) is defined by F on G 1 as follows: (cid:0) ( G, v ) , ( H, w )
[Equation]
∈ ρ 1 ( F ) ⇐⇒
[Equation]
f ∈ F , f (
[Equation]
=
[Equation]
H
[Equation]
In other words, when (( G, v ) , ( H, w )) ∈
ρ 1 ( F ) , no function in F can separate v in G from w in
H .
For example, we can view cr ( t ) and vwl ( t ) k as functions from G 1 to some R ℓ .
As such ρ 1 ( cr ( t ) ) and ρ 1 ( vwl ( t ) k ) measure the separation power of these algorithms.
The following strict inclusions are known: for all k ≥ 1 , ρ 1 ( vwl ( t ) k +1 )
⊂ ρ 1 ( vwl ( t ) k ) and ρ 1 ( vwl ( t ) 1 ) ⊂
ρ 1 ( cr ( t ) )
(Otto, 2017; Grohe, 2021).
It is also known that more rounds ( t ) increase the separation power of these algorithms (F ¨ urer, 2001).
For a fragment L of TL (Ω) expressions, we define ρ 1 ( L ) as the equivalence relation associated with all functions f : G 1 → R ℓ f that can be represented in L .
By definition, we here thus consider expressions in TL (Ω) with one free index variable resulting in vertex embeddings.
4. M AIN R ESULTS
We first provide a link between k - WL and tensor language expressions using k + 1 index variables:
Theorem 4.1.
For each k ≥ 1 and any collection Ω of functions, ρ 1 (cid:0) vwl ( ∞ ) k (cid:1)
=
ρ 1 (cid:0)
TL k +
(Ω) (cid:1) .
This theorem gives us new insights: if we wish to understand how a new GNN architecture compares against the k - WL algorithms, all we need to do is to show that such an architecture can be represented in TL k +
(Ω) , i.e., is a k - MPNN , an arguably much easier endeavor.
As an example of how to use this result, it is well known that triangles can be detected by 2 - WL but not by 1 - WL .
Thus, in order to design GNN s that can detect triangles, layer definitions in TL 3 rather than TL 2 should be used.
We can do much more, relating the rounds of k - WL to the notion of summation depth of TL (Ω) expressions.
We also present present similar results for functions computing graph embeddings.
The summation depth sd ( φ ) of a TL (Ω) expression φ measures the nesting depth of the summations (cid:80) x in the expression.
It is defined inductively: sd ( 1 x op y )
=
sd ( E ( x, y ))
=
sd ( P s ( x )) :
= 0 , sd ( φ 1 · φ 2 ) =
sd ( φ 1 + φ 2 ) :
= max { sd ( φ 1 ) , sd ( φ 2 ) } , sd ( a · φ 1 ) :
= sd ( φ 1 ) , sd ( f ( φ 1 , . .
. , φ p )) :
= max { sd ( φ i ) | i ∈
[ p ] } , and sd ( (cid:80) x φ 1 ) :
= sd ( φ 1 ) + 1 .
For example, expressions ψ s ( x 1 ) above have summation depth one.
We write TL ( t )
k
+1 (Ω) for the class of expressions in TL k +1 (Ω) of summation depth at most t , and use k - MPNN ( t ) for the corresponding class of k - MPNN s. We can now refine
Theorem 4.1, taking into account the number of rounds used in k - WL .
Theorem 4.2.
For all t ≥ 0 , k ≥ 1 and any collection Ω of functions, ρ 1 (cid:0) vwl ( t ) k (cid:1) = ρ 1 (cid:0) TL ( t )
k
+1 (Ω) (cid:1) .
Guarded TL and color refinement.
As noted by Barcel´o et al.
, the separation power of vertex embeddings of simple GNN s, which propagate information only through neighboring vertices, is usually weaker than that of 1 - WL .
For these types of architectures, Barcel´o et al.
(2020) provide a relation with the weaker color refinement algorithm, but only in the special case of firstorder classifiers.
We can recover and extend this result in our general setting, with a guarded version of TL which, as we will show, has the same separation power as color refinement.
The guarded fragment GTL (Ω) of TL 2 (Ω) is inspired by the use of adjacency matrices in simple
GNN s. In GTL (Ω) only equality predicates 1 x i =
x i (constant 1 ) and 1 x i ̸
= x
i (constant 0 ) are allowed, addition and multiplication require the component expressions to have the same (single) free index, and summation must occur in a guarded form (cid:80) x j (cid:0) E ( x i , x j ) · φ ( x j ) (cid:1) , for i, j ∈ [2] .
Guardedness means that summation only happens over neighbors.
In GTL (Ω) , all expressions have a single free variable and thus only functions from G 1 can be represented.
Our example expressions ψ s ( x 1 ) are guarded.
The fragment GTL ( t ) (Ω) consists of expressions in GTL (Ω) of summation depth at most t .
We denote by MPNN s and MPNN s ( t ) the corresponding “guarded” classes of 1 - MPNN s.
Theorem 4.3.
For all t ≥ 0 and any collection Ω of functions: ρ 1 (cid:0) cr ( t ) (cid:1) =
ρ 1 (cid:0) GTL ( t ) (Ω) (cid:1) .
As an application of this theorem, to detect the existence of paths of length t , the number of guarded layers in GNN s should account for a representation in GTL (Ω) of summation depth of at least t .
We recall that ρ 1 ( vwl ( t ) 1 ) ⊂
ρ 1 ( cr ( t ) ) which, combined with our previous results, implies that TL ( t ) 2 (Ω) (resp., 1 - MPNN s) is strictly more separating than GTL ( t ) (Ω) (resp., MPNN s).
Graph embeddings.
We next establish connections between the graph versions of k - WL and CR , and TL expressions without free index variables.
To this aim, we use ρ 0 ( F ) , for a set F of functions f : G → R ℓ f , as the equivalence relation over G defined in analogy to ρ 1 : ( G, H ) ∈ ρ 0 ( F ) ⇐⇒ ∀ f ∈ F , f ( G ) =
f ( H ) .
We thus consider separation power on the graph level.
For example, we can consider ρ
( gcr ( t ) ) and ρ 0 ( gwl ( t ) k ) for any t ≥ 0 and k ≥ 1 .
Also here, ρ
( gwl ( t ) k +1 )
⊂ ρ
( gwl ( t ) k ) but different from vertex embeddings, ρ 0 ( gcr ( t ) )
=
ρ
( gwl ( t ) 1 ) (Grohe, 2021).
We define ρ 0 ( L ) for a fragment L of TL (Ω) by considering expressions without free index variables.
The connection between the number of index variables in expressions and k - WL remains to hold.
Apart from k = 1 , no clean relationship exists between summation depth and rounds, however.
Theorem 4.4.
For all t ≥ 0 , k ≥ 1 and any collection Ω of functions, we have that:
[Equation]
Intuitively, in (1) the increase in summation depth by one is incurred by the additional aggregation needed to collect all vertex labels computed by gwl ( t ) 1 .
Optimality of number of indices.
Our results so far tell that graph functions represented in
TL k +1 (Ω) are at most as separating as k - WL .
What is left unaddressed is whether all k + 1 index variables are needed for the graph functions under consideration.
It may well be, for example, that there exists an equivalent expression using less index variables.
This would imply a stronger upper bound on the separation power by ℓ - WL for ℓ < k .
We next identify a large class of TL (Ω) expressions, those of treewidth k , for which the number of index variables can be reduced to k + 1 .
Proposition 4.5.
Expressions in TL (Ω) of treewidth k are equivalent to expressions in TL k +1 (Ω) .
Treewidth is defined in the supplementary material (Section G) and a treewidth of k implies that the computation of tensor language expressions can be decomposed, by reordering summations, such that each local computation requires at most k + 1 indices (see also Aji & McEliece (2000)).
As a simple example, consider θ ( x 1 ) =
(cid:80) x 2 (cid:80) x 3 E ( x 1 , x 2 ) · E ( x 2 , x 3 ) in TL ( 2 ) 3 such that [[ θ, v ]] G counts the number of paths of length two starting from v .
This expression has a treewidth of one.
And indeed, it is equivalent to the expression ˜ θ ( x 1 ) =
(cid:80) x 2 E ( x 1 , x 2 ) · (cid:0)(cid:80) x 1 E ( x 2 , x 1 ) (cid:1) in TL ( 2 ) 2 (and in fact in GTL ( 2 ) ).
As a consequence, no more vertices can be separated by θ ( x 1 ) than by cr ( 2 ) , rather than vwl 22 as the original expression in TL ( 2 ) 3 suggests.
On the impact of functions.
All separation results for TL (Ω) and fragments thereof hold irregardless of the chosen functions in Ω , including when no functions are present at all.
Function applications hence do not add expressive power.
While this may seem counter-intuitive, it is due to the presence of summation and multiplication in TL that are enough to separate graphs or vertices.
C ONSEQUENCES FOR GNN S
We next interpret the general results on the separation power from Section 4 in the context of GNN s.
The separation power of any vertex embedding GNN architecture which is an MPNN ( t ) is bounded by the power of t rounds of color refinement.
We
consider the Graph Isomorphism Networks ( GIN s ) (Xu et al., 2019) and show that these are
MPNN s. To do so, we represent them in GTL (Ω) .
Let gin be such a network; it updates vertex embeddings as follows.
Initially, gin (0) : G 1 → R ℓ 0 :
( G, v ) (cid:55)→ F ( 0 ) v : := col G ( v ) ∈ R ℓ 0 .
For layer t > 0 , gin ( t ) : G 1 → R ℓ t is given by: ( G, v ) (cid:55)→ F ( t ) v : := mlp ( t ) (cid:0) F ( t − 1 ) v : , (cid:80) u ∈ N G ( v ) F ( t − 1 ) u : (cid:1) , with
F ( t ) ∈ R n × ℓ t and mlp ( t ) =
( mlp ( t ) 1 , . . .
, mlp ( t ) ℓ t ) : R 2 ℓ t − 1 → R ℓ t is an MLP .
We denote by GIN ( t ) the class of GIN s consisting t layers.
Clearly, gin ( 0 ) can be represented in GTL ( 0 ) by considering the expressions φ ( 0 ) i ( x 1 ) :
= P i ( x 1 ) for each i ∈ [ ℓ 0 ] .
To represent gin ( t ) , assume that we have ℓ t −expressions φ ( t − 1 ) i ( x 1 ) in GTL ( t − 1 ) (Ω) representing gin ( t − 1 ) .
That is, we have [[ φ ( t − 1 ) i , v ]] G = F ( t − 1 ) vi for each vertex v and i ∈
[ ℓ t − 1 ] .
Then gin ( t ) is represented by ℓ t expressions φ ( t ) i ( x 1 ) defined as:
[Equation]
which are now expressions in GTL ( t ) (Ω) where Ω consists of MLP s. We have [[ φ ( t ) i , v ]] G = F ( t ) v,
i for each v ∈ V G and i ∈ [ ℓ t ] , as desired.
Hence, Theorem 4.3 tells that t -layered GIN s cannot be more separating than t rounds of color refinement, in accordance with known results (Xu et al., 2019; Morris et al., 2019).
We thus simply cast GIN s in GTL (Ω) to obtain an upper bound on their separation power.
In the supplementary material (Section D) we give similar analyses for GraphSage GNN s with various aggregation functions (Hamilton et al., 2017), GCN s (Kipf & Welling, 2017), simplified GCN s ( SGC s) (Wu et al., 2019), Principled Neighborbood Aggregation ( PNA s) (Corso et al., 2020), and revisit the analysis of ChebNet (Defferrard et al., 2016) given in Balcilar et al.
(2021a).
The separation power of any vertex embedding GNN architecture which is an k - MPNN ( t ) is bounded by the power of t rounds of k - WL .
For k = 1 , we consider extended Graph Isomorphism Networks ( eGIN s) (Barcel´o et al., 2020).
For an egin ∈ eGIN , egin ( 0 ) : G 1 → R ℓ 0 is defined as for GIN s, but for layer t > 0 , egin ( t ) :
G 1 → R ℓ t is defined by ( G, v ) (cid:55)→ F ( t ) v : :=
mlp ( t ) (cid:0) F ( t − 1 ) v : , (cid:80) u ∈ N G ( v ) F ( t − 1 ) u : , (cid:80) u ∈ V G F ( t − 1 ) u : (cid:1) , where mlp ( t ) is now an MLP from R 3 ℓ t − 1 → R ℓ t .
The difference with GIN s is the use of (cid:80) u ∈ V G F ( t − 1 ) u : which corresponds to the unguarded summation (cid:80) x 1 φ ( t − 1 ) ( x 1 ) .
This implies that TL rather than GTL needs to be used.
In a similar way as for GIN s, we can represent eGIN layers in TL ( t ) 2 (Ω) .
That is, each eGIN ( t ) is an 1 - MPNN ( t ) .
Theorem 4.2 tells that t rounds of 1 - WL bound the separation power of t -layered extended GIN s, conform to Barcel´o et al.
More generally, any GNN looking to go beyond CR must use non-guarded aggregations.
For k ≥ 2 , it is straightforward to show that t -layered “folklore” GNN s ( k - FGNN s) (Maron et al., 2019b) are k - MPNN ( t ) and thus, by Theorem 4.2, t rounds of k - WL bound their separation power.
One merely needs to cast the layer definitions in TL (Ω) and observe that k +1 indices and summation depth t are needed.
We thus refine and recover the k - WL bound for k - FGNN s by Azizian & Lelarge (2021).
We also show that the separation power of ( k +1)
-Invariant
Graph Networks ( ( k +1) - IGN s ) (Maron et al., 2019b) are bounded by k - WL , albeit with an increase in the required rounds.
Theorem 5.1.
For any k ≥ 1 , the separation power of a t -layered ( k + 1) - IGN s is bounded by the separation power of tk rounds of k - WL .
We hereby answer open problem 1 in Maron et al. (2019a).
The case k = 1 was solved in Chen et al.
by analyzing properties of 1 - WL .
By contrast, Theorem 4.2 shows that one can focus on expressing ( k +1) - IGN s in
TL k +
(Ω)
and analyzing the summation depth of expressions.
The proof of Theorem 5.1 requires non-trivial manipulations of tensor language expressions; it is a simplified proof of Geerts (2020).
The additional rounds ( tk ) are needed because ( k + 1) - IGN s aggregate information in one layer that becomes accessible to k - WL in k rounds.
We defer detail to Section E in the supplementary material, where we also identify a simple class of t -layered ( k + 1) - IGN s that are as powerful as ( k + 1) - IGN s but whose separation power is bounded by t rounds of k - WL .
We also consider “augmented” GNN s, which are combined with a preprocessing step in which higher-order graph information is computed.
In the supplementary material (Section D.3) we show how TL encodes the preprocessing step, and how this leads to separation bounds in terms of k - WL , where k depends on the treewidth of the graph information used.
Finally, our approach can also be used to show that the spectral CayleyNet s (Levie et al., 2019) are bounded in separation power by 2 - WL .
This result complements the spectral analysis of CayleyNet s given in Balcilar et al.
(2021b).
The separation power of any graph embedding GNN architecture which is a k - MPNN is bounded by the power of k - WL
Graph embedding methods are commonly obtained from vertex (tuple) embeddings methods by including a readout layer in which all vertex (tuple) embeddings are aggregated.
For example, mlp ( (cid:80) v ∈ V egin ( t ) ( G, v )) is a typical readout layer for eGIN s .
Since egin ( t ) can be represented in
TL ( t ) 2 (Ω) , the readout layer can be represented in TL ( t +1 )(Ω) , using an extra summation.
So they are 1 - MPNN s. Hence, their separation power is bounded by gwl ( t ) 1 , in accordance with Theorem 4.4.
This holds more generally.
If vertex embedding methods are k - MPNN s, then so are their graph versions, which are then bounded by gwl ( ∞ ) k by our Theorem 4.4.
To go beyond the separation power of k - WL , it is necessary to use GNN s whose layers are represented by expressions of treewidth > k .
Hence, to design expressive GNN s one needs to define the layers such that treewidth of the resulting
TL expressions is large enough.
For example, to go beyond 1 - WL , TL 3 representable linear algebra operations should be used.
Treewidth also sheds light on the open problem from Maron et al. (2019a) where it was asked whether polynomial layers (in A ) increase the separation power.
Indeed, consider a layer of the form σ ( A 3 · F · W ) , which raises the adjacency matrix A to the power three.
Translated in TL (Ω) , layer expressions resemble (cid:80) x 2 (cid:80) x 3 (cid:80) x 4 E ( x 1 , x 2 ) · E ( x 2 , x 3 ) · E ( x 3 , x 4 ) , of treewidth one.
Proposition 4.5 tells that the layer is bounded by wl ( 3 ) 1 (and in fact by cr ( 3 ) ) in separation power.
If instead, the layer is of the form σ ( C · F · W ) where C ij holds the number of cliques containing the edge ij .
Then, in TL (Ω) we get expressions containing (cid:80) x 2 (cid:80) x
3 E ( x 1 , x 2 ) · E ( x 1 , x 3 ) · E ( x 2 , x 3 ) .
The variables form a 3 -clique resulting in expressions of treewidth two.
As a consequence, the separation power will be bounded by wl ( 2 ) 2 .
These examples show that it is not the number of multiplications (in both cases two) that gives power, it is how variables are connected to each other.
F UNCTION A PPROXIMATION
We next provide characterizations of functions that can be approximated by TL expressions, when interpreted as functions.
We recover and extend results from Azizian & Lelarge (2021) by taking the number of layers of GNN s into account.
We also provide new results related to color refinement.
6. G ENERAL TL A PPROXIMATION R ESULTS
We assume that G s is a compact space by requiring that vertex labels come from a compact set K ⊆ R ℓ 0 .
Let F be a set of functions f : G s → R ℓ f and define its closure F as all functions h from G s for which there exists a sequence f 1 , f 2 , . . .
∈ F such that lim
i →∞ sup G, v ∥ f
i ( G, v ) − h
( G, v ) ∥
=for some norm ∥ .
∥ .
We assume F to satisfy two properties.
First, F is concatenation-closed : if f 1 : G s → R p and f 2 : G s → R q are in F , then g :=
( f 1 , f 2 ) :
G s → R p + q : ( G, v ) (cid:55)→ ( f 1 ( G, v ) , f 2 ( G, v )) is also in F .
Second, F is function-closed , for a fixed ℓ ∈ N : for any f ∈ F such that f : G s → R p , also g ◦ f : G s → R ℓ is in F for any continuous function g : R p
→ R ℓ .
For such F , we let F ℓ be the subset of functions in F from G s to R ℓ .
Our next result is based on a generalized Stone-Weierstrass Theorem (Timofte, 2005), also used in Azizian & Lelarge (2021).
Theorem 6.1.
For any ℓ , and any set F of functions, concatenation and function closed for ℓ , we have: F ℓ = { f : G s → R ℓ | ρ s ( F ) ⊆
ρ s ( f ) } .
This result gives us insight on which functions can be approximated by, for example, a set F of functions originating from a class of GNN s. In this case, F ℓ represent all functions approximated by instances of such a class and Theorem 6.1 tells us that this set corresponds precisely to the set of all functions that are equally or less separating than the GNN s in this class.
If, in addition, F ℓ is more separating that CR or k - WL , then we can say more.
Let alg ∈ { cr ( t ) , gcr ( t ) , vwl ( t ) k , gwl ( ∞ ) k } .
Corollary 6.2.
Under the assumptions of Theorem 6.1 and if ρ ( F ℓ ) = ρ ( alg ) , then F ℓ = { f : G s → R ℓ | ρ ( alg ) ⊆
ρ ( f ) } .
The properties of being concatenation and function-closed are satisfied for sets of functions representable in our tensor languages, if Ω contains all continuous functions
g : R p
→
R ℓ , for any p , or alternatively, all MLP s (by Lemma 32 in Azizian & Lelarge (2021)).
Together with our results in
Section 4, the corollary implies that MPNN s ( t ) , 1 - MPNN s ( t ) , k - MPNN s ( t ) or k - MPNN s can approximate all functions with equal or less separation power than cr ( t ) , gcr ( t ) , vwl ( t ) k or gwl ( ∞ ) k , respectively.
Prop. 3.1 also tells that the closure consists of invariant ( s = 0 ) and equivariant ( s > 0 ) functions.
6. C ONSEQUENCES FOR GNN S
All our results combined provide a recipe to guarantee that a given function can be approximated by GNN architectures.
Indeed, suppose that your class of GNN s is an MPNN ( t ) (respectively, 1 - MPNN ( t ) , k - MPNN ( t ) or k - MPNN , for some k ≥ 1 ).
Then, since most classes of GNN s are concatenation-closed and allow the application of arbitrary MLP s, this implies that your GNN s can only approximate functions f that are no more separating than cr ( t ) (respectively, gcr ( t ) , vwl ( t ) k or gwl ( ∞ ) k ).
To guarantee that that these functions can indeed be approximated, one additionally has to show that your class of GNN s matches the corresponding labeling algorithm in separation power.
For example, GNN s in GIN ( t ) ℓ are MPNN s ( t ) , and thus GIN ( t ) ℓ contains any function f : G 1 → R ℓ satisfying ρ 1 ( cr ( t ) ) ⊆
ρ 1 ( f ) .
Similarly, eGIN ( t ) ℓ s are 1 - MPNN s ( t ) , so eGIN ( t ) ℓ contains any function satisfying ρ 1 ( wl ( t ) 1 ) ⊆
ρ 1 ( f ) ; and when extended with a readout layer, their closures consist of functions f : G 0 → R ℓ satisfying ρ 0 ( gcr ( t ) )
=
ρ
( vwl ( t ) 1 ) ⊆
ρ 0 ( f ) .
Finally, k - FGNN ( t ) ℓ s are kMPNN s ( t ) , so k - FGNN ( t ) ℓ consist of functions f such that ρ 1 ( vwl ( t ) k ) ⊆
ρ 1 ( f ) .
We thus recover and extend results by Azizian & Lelarge (2021) by including layer information ( t ) and by treating color refinement separately from 1 - WL for vertex embeddings.
Furthermore, Theorem 5.1 implies that ( k + 1) - IGN ℓ consists of functions f satisfying ρ 1 ( vwl ( ∞ ) k ) ⊆
ρ 1 ( f ) and ρ 0 ( gwl ( ∞ ) k ) ⊆
ρ 0 ( f ) , a case left open in Azizian & Lelarge (2021).
These results follow from Corollary 6.2, that the respective classes of GNN s can simulate CR or k - WL on either graphs with discrete (Xu et al., 2019; Barcel´o et al., 2020) or continuous labels (Maron et al., 2019b), and that they are k - MPNN s of the appropriate form.
C ONCLUSION
Connecting GNN s and tensor languages allows us to use our analysis of tensor languages to understand the separation and approximation power of GNN s. The number of indices and summation depth needed to represent the layers in GNN s determine their separation power in terms of color refinement and Weisfeiler-Leman tests.
The framework of k - MPNN s provides a handy toolbox to understand existing and new GNN architectures, and we demonstrate this by recovering several results about the power of GNN s presented recently in the literature, as well as proving new results.
A KNOWLEDGEMENTS & D ISCLOSURE F UNDING
This work is partially funded by ANID–Millennium Science Initiative Program–Code ICN17 002,
Chile.
E THICS S TATEMENT
The results in this paper do not include misleading claims; their correctness is theoretically verified.
Related work is accurately represented.
S UPPLEMENTARY M ATERIAL A R ELATED W ORK C NT ’ D
We provide additional details on how the tensor language TL (Ω) considered in this paper relates to recent work on other matrix query languages.
Closest to TL (Ω) is the matrix query language sum - MATLANG (Geerts et al., 2021b) whose syntax is close to that of TL (Ω) .
There are, however, key differences.
First, although sum - MATLANG uses index variables (called vector variables), they all must occur under a summation.
In other words, the concept of free index variables is missing, which implies that no general tensors can be represented.
In TL (Ω) , we can represent arbitrary tensors and the presence of free index variables is crucial to define vertex, or more generally, k -tuple embeddings in the context of GNN s. Furthermore, no notion of summation depth was introduced for sum - MATLANG .
In TL (Ω) , the summation depth is crucial to assess the separation power in terms of the number of rounds of color refinement and k - WL .
And in fact, the separation power of sumMATLANG was not considered before, and neither are finite variable fragments of sum - MATLANG and connections to color refinement and k - WL studied before.
Finally, no other aggregation functions were considered for sum - MATLANG .
We detail in Section C.5 that TL (Ω) can be gracefully extended to TL (Ω , Θ) for some arbitrary set Θ of aggregation functions.
Connections to 1 - WL and 2 - WL and the separation power of another matrix query language,
MATLANG (Brijder et al., 2019) were established in Geerts (2021).
Yet, the design of MATLANG is completely different in spirit than that of TL (Ω) .
Indeed, MATLANG does not have index variables or explicit summation aggregation.
Instead, it only supports matrix multiplication, matrix transposition, function applications, and turning a vector into a diagonal matrix.
As such, MATLANG can be shown to be included in TL 3 (Ω) .
Similarly as for sum - MATLANG , MATLANG cannot represent general tensors, has no (free) index variables and summation depth is not considered (in view of the absence of an explicit summation).
We also emphasize that neither for MATLANG nor for sum - MATLANG a guarded fragment was considered.
The guarded fragment is crucial to make connections to color refinement (Theorem 4.3).
Furthermore, the analysis in terms of the number of index variables, summation depth and treewidth (Theorems 4.1,4.2 and Proposition 4.5), were not considered before in the matrix query language literature.
For none of these matrix query languages, approximation results were considered (Section 6.1).
Matrix query languages are used to assess the expressive power of linear algebra.
Balcilar et al.
(2021a) use MATLANG and the above mentioned connections to 1 - WL and 2 - WL , to assess the separation power of GNN s. More specifically, similar to our work, they show that several GNN architectures can be represented in MATLANG , or fragments thereof.
As a consequence, bounds on their separation power easily follow.
Furthermore, Balcilar et al.
(2021a) propose new architectures inspired by special operators in MATLANG .
The use of TL (Ω) can thus been seen as a continuation of their approach.
We note, however, that TL (Ω) is more general than MATLANG (which is included in TL 3 (Ω) ), allows to represent more complex linear algebra computations by means summation (or other) aggregation, and finally, provides insights in the number of iterations needed for color refinement and k - WL .
The connection between the number of variables (or treewidth) and k - WL is not present in the work by Balcilar et al.
(2021a), neither is the notion of guarded fragment, needed to connect to color refinement.
We believe that it is precisely these latter two insights that make the tensor language approach valuable for any GNN designer who wishes to upper bound their GNN architecture.
B D ETAILS OF S ECTIONB. P ROOF OF P ROPOSITION 3.
Let G =
( V, E, col ) be a graph and let σ be a permutation of V .
As usual, we define σ ⋆ G =
( V σ , E σ , col σ ) as the graph with vertex set V σ :=
V , edge set vw ∈ E
σ if and only if σ − 1 ( v ) σ − 1 ( w ) ∈ E , and col σ ( v ) :
= col ( σ − 1 ( v )) .
We need to show that for any expression φ ( x ) in TL (Ω) either [[ φ, σ ⋆ v ]] σ⋆G =
[[ φ, v ]] G , or when φ has no free index variables, [[ φ ]] σ⋆G =
[[ φ ]] G .
We verify this by a simple induction on the structure of expressions in TL (Ω) .
• If φ ( x i , x j ) =
1 x i op x j , then for a valuation ν mapping x i to v i and x j to v j in V :
[Equation]
where we used that σ is a permutation.
• If φ ( x i ) =
P ℓ ( x i ) , then for a valuation µ mapping x i to v i in V :
[Equation]
G
[Equation]
where we used the definition of col σ .
• Similarly, if φ ( x i , x j ) =
E ( x i , x j ) , then for a valuation ν
assigning x i to v i and x j to v j :
[Equation]
where we used the definition of E σ .
• If φ ( x ) =
φ 1 ( x 1 ) · φ 2 ( x 2 ) , then for a valuation ν from x to V :
[Equation]
φ,
[Equation]
, ν ]]
[Equation]
, ν ]
[Equation]
ν
[Equation]
φ
[Equation] [Equation]
ν ]]
[Equation]
= [[ φ, σ ⋆ ν ]] σ⋆G , where we used the induction hypothesis for φ 1 and φ 2 .
The cases φ ( x ) =
φ 1 ( x 1 ) + φ 2 ( x 2 ) and φ ( x ) = a · φ 1 ( x ) are dealt with in a similar way.
• If φ ( x ) =
f
( φ 1 ( x 1 ) , . . . , φ p ( x p )) , then [[ φ, ν ]] G = f
([[ φ 1 , ν ]] G , . . .
, [[ φ p , ν ]] G
[Equation]
where we used again the induction hypothesis for φ 1 , . .
. , φ p .
• Finally, if φ ( x ) =
(cid:80) y
φ 1 ( x , y )
then for a valuation ν of x to V :
[Equation]
where we used the induction hypothesis for φ 1 and that V σ = V because σ is a permutation.
We remark that when φ does not contain free index variables, then [[ φ, ν ]] G =
[[ φ ]] G for any valuation ν , from which invariance follows from the previous arguments.
This concludes the proof of
Proposition 3.1.
C D ETAILS OF S ECTION
In the following sections we prove Theorem 4.1, 4.2, 4.3 and 4.4.
More specifically, we start by showing these results in the setting that TL (Ω) only supports summation aggregation ( (cid:80) x e ) and in which the vertex-labellings in graphs take values in { 0 , 1 } ℓ .
In this context, we introduce classical logics in Section C.1 and recall and extend connections between the separation power of these logics and the separation power of color refinement and k - WL in Section C.2.
We connect TL (Ω) and logics in Section C.3, to finally obtain the desired proofs in Section C.4.
We then show how these results can be generalized in the presence of general aggregation operators in Section C.5, and to the setting where vertex-labellings take values in R ℓ in Section C.6.
C. C LASSICAL L OGICS
In what follows, we consider graphs G =
( V G , E G , col G ) with col G : V G → { 0 , 1 } ℓ .
We start by defining the k -variable fragment C k of first-order logic with counting quantifiers, followed by the definition of the guarded fragment GC of C 2 .
Formulae φ in C k are defined over the set { x 1 , . . .
, x k } of variables and are formed by the following grammar:
[Equation]
where i, j ∈ [ k ] , E is a binary predicate, P s for s ∈ [ ℓ ] are unary predicates for some ℓ ∈ N , and m ∈ N .
The semantics of formulae in C k is defined in terms of interpretations relative to a given graph G and a (partial) valuation µ : { x 1 , . . .
, x k } → V G .
Such an interpretation maps formulae, graphs and valuations to Boolean values B := {⊥ , ⊤} , in a similar way as we did for tensor language expressions.
More precisely, given a graph G =
( V G , E G , col G ) and partial valuation µ : { x 1 , . . .
, x k } → V G , we define [[ φ, µ ]] B G ∈ B for valuations defined on the free variables in φ .
That is, we define:
[Equation]
In the last expression, µ [ x i (cid:55)→ v ] denotes the valuation µ modified such that it maps x i to vertex v .
We will also need the guarded fragment GC of C 2 in which we only allow equality conditions of the form x i
= x
i , component expressions of conjunction and disjunction should have the same single free variable, and counting quantifiers can only occur in guarded form: ∃ ≥ m x 2 ( E ( x 1 , x 2 )
∧ φ ( x 2 )) or ∃ ≥ m x 1 ( E ( x 2 , x 1 ) ∧ φ ( x 1 )) .
The semantics of formulae in GC is inherited from formulae in C 2 .
Finally, we will also consider C k ∞ ω , that is, the logic C k extended with infinitary disjunctions and conjunctions .
More precisely, we add to the grammar of formulae the following constructs:
[Equation]
where the index set A can be arbitrary, even containing uncountably many indices.
We define
GC ∞ ω in the same way by relaxing the finite variable conditions.
The semantics is, as expected:
[[ (cid:87)
α ∈
A φ α , µ ]] B G = ⊤ if for at least one α ∈ A , [[ φ α , µ ]] B G = ⊤ , and [[ (cid:86) α ∈
A φ α , µ ]] B G = ⊤ if for all α ∈ A , [[ φ α , µ ]] B G = ⊤ .
We define the free variables of formulae just as for TL , and similarly, quantifier rank is defined as summation depth (only existential quantifications increase the quantifier rank).
For any of the above logics L we define L ( t ) as the set of formulae in L of quantifier rank at most t .
To capture the separation power of logics , we define ρ 1 (cid:0) L ( t ) (cid:1) as the equivalence relation on Gdefined by
[Equation]
where µ v is any valuation such that µ ( x ) = v , and likewise for w .
The relation ρ 0 is defined in a similar way, except that now the relation is only over pairs of graphs, and the characterization is over all formulae with no free variables (also called sentences).
Finally, we also use, and define, the relation ρ s , which relates pairs from G s : consisting of a graph and an s -tuple of vertices.
The relation is defined as
[Equation]
where x consist of s free variables and µ v is a valuation assigning the i -th variable of x to the i -th value of v , for any i ∈ [ s ] .
C. C HARACTERIZATION OF S EPARATION P OWER OF L OGICS
We first connect the separation power of the color refinement and k -dimensional Weisfeiler-Leman algorithms to the separation power of the logics we just introduced.
Although most of these connections are known, we present them in a bit of a more fine-grained way.
That is, we connect the number of rounds used in the algorithms to the quantifier rank of formulae in the above logics.
Proposition C.1.
For any t ≥ 0 , we have the following identities:
(1) ρ 1 (cid:0) cr ( t ) (cid:1) =
ρ 1 (cid:0) GC ( t ) (cid:1) and ρ 0 (cid:0) gcr ( t ) (cid:1) =
ρ 0 (cid:0) gwl ( t ) 1 (cid:1)
=
ρ 0 (cid:0) C 2 , ( t +1) (cid:1) ; (2) For k ≥ 1 , ρ 1 (cid:0) vwl ( t ) k (cid:1)
=
ρ 1 (cid:0) C k +1 , ( t ) (cid:1) and
[Equation]
As a consequence, ρ
[Equation]
Proof.
For (1), the identity ρ 1 (cid:0) cr ( t ) (cid:1) =
ρ 1 (cid:0) GC ( t ) (cid:1) is known and can be found, for example, in
Theorem V.10 in Grohe (2021).
The identity ρ 0 (cid:0) gcr ( t ) (cid:1) =
ρ 0 (cid:0) gwl ( t ) 1 (cid:1) can be found in Proposition
V.4 in Grohe (2021).
The identity ρ 0 (cid:0) gwl ( t ) 1 (cid:1) =
ρ 0 (cid:0) C 2 , ( t +1) (cid:1) is a consequence of the inclusion shown in (2) for k = 1 .
For (2), we use that ρ k (cid:0) wl ( t ) k (cid:1) =
ρ k (cid:0) C k +1 , ( t ) (cid:1) , see e.g., Theorem V.8 in Grohe (2021).
We argue that this identity holds for ρ 1 (cid:0) vwl ( t ) k (cid:1)
=
ρ 1 (cid:0) C k +1 , ( t ) (cid:1) .
Indeed, suppose that ( G, v ) and ( H, w ) are not in ρ 1 (cid:0) C k +1 , ( t ) (cid:1) .
Let φ ( x 1 ) be a formula in C k +1 , ( t ) such that [[ φ, v ]] B G ̸ =
[[ φ, w ]] B H .
Consider the formula φ + ( x 1 , . . .
, x k ) =
φ ( x 1 ) ∧ (cid:86) k
i =1 ( x 1 = x i ) .
Then, [[ φ + , ( v, . . .
, v )]] B G ̸ =
[[ φ + , ( w, . . . , w )]] B H , and hence ( G, ( v, . .
. , v )) and ( H, ( w, . . .
, w )) are not in ρ k (cid:0) C k +1 , ( t ) (cid:1) either.
This implies that wl ( t ) k ( G, ( v, . .
. , v ))
̸ = wl ( t )
k
( H, ( w, . . . , w )) , and thus, by definition, vwl ( t ) k
( G, v ) ̸
= vwl ( t ) k
( H, w ) .
In other words, ( G, v ) and ( H, w ) are not in ρ 1 (cid:0) vwl ( t ) k (cid:1) , from which the inclusion ρ 1 (cid:0) vwl ( t ) k (cid:1) ⊆
ρ 1 (cid:0) C k +1 , ( t ) (cid:1) follows.
Conversely, if ( G, v ) and ( H, w ) are not in ρ 1 (cid:0) vwl ( t ) k (cid:1) , then wl ( t ) k
( G, ( v, . . .
, v ))
̸ = wl ( t )
k
( H, ( w, . . . , w )) .
As a consequence, ( G, ( v, . . .
, v )) and ( H, ( w, . . .
, w )) are not in ρ k (cid:0) C k +1 , ( t ) (cid:1) either.
Let φ ( x 1 , . . .
, x k ) be a formula in C k +1 , ( t ) such that [[ φ, ( v, . .
. , v )]] B G ̸ =
[[ φ, ( w, . . . , w )]] B H .
Then it is readily shown that we can convert φ ( x 1 , . . .
, x k ) into a formula φ − ( x 1 ) in C k +1 , ( t ) such that [[ φ − , v ]] B G ̸ =
[[ φ − , w ]] B H , and thus ( G, v ) and ( H, w ) are not in ρ
1 (cid:0) C k
+1 , ( t ) (cid:1) .
Hence, we also have the inclusion ρ 1 (cid:0) vwl ( t ) k (cid:1) ⊇
ρ 1 (cid:0) C k
+1 , ( t ) (cid:1) , form which the first identity in (2) follows.
It remains to show ρ 0 (cid:0) C k +1 , ( t + k ) (cid:1) ⊆
ρ 0 (cid:0) gwl ( t ) k (cid:1) ⊆
ρ 0 (cid:0) C k
+1 , ( t +1) (cid:1) .
Clearly, if ( G, H ) is not in ρ 0 (cid:0) gwl ( t ) k (cid:1) then the multisets of labels wl ( t ) k
( G, v ) and wl ( t ) k
( H, w ) differ.
It is known that with each label c one can associate a formula φ c in C k +1 , ( t ) such that [[ φ c , v ]] B G = ⊤
if and only if wl ( t ) k ( G, v ) =
c .
So, if the multisets are different, there must be a c that occurs more often in one multiset than in the other one.
This can be detected by a fomulae of the form ∃ = m ( x 1 , . . .
, x k ) φ c ( x 1 , . . .
, x k ) which is satisfied if there are m tuples v with label c .
It is now easily verified that the latter formula can be converted into a formula in C k +1 , ( t + k ) .
Hence, the inclusion ρ 0 (cid:0) C k +1 , ( t + k ) (cid:1) ⊆
ρ 0 (cid:0) gwl ( t ) k
(cid:1) follows.
For ρ 0 (cid:0) gwl ( t ) k
(cid:1) ⊆
ρ 0 (cid:0) C k
+1 , ( t +1) (cid:1)
, we show that if ( G, H ) is in ρ 0 (cid:0) gwl ( t ) k (cid:1) , then this implies that [[ φ, µ ]] B G =
[[ φ, µ ]] B H for all formulae in C k +1 , ( t +1) and any valuation µ (notice that µ is superfluous in this definition when formulas have no free variables).
Assume that ( G, H ) is in ρ 0 ( gwl ( t ) k ) .
Since any formula of quantifier rank t +1 is a Boolean combination of formulas of less rank or a formula of the form φ = ∃ ≥ m x i ψ where ψ is of quantifier rank t , without loss of generality consider a formula of the latter form, and assume for the sake of contradiction that [[ φ, µ ]] B G = ⊤
but [[ φ, µ ]] B H = ⊥ .
Since [[ φ, µ ]] B G = ⊤ , there must be at least m elements satisfying ψ .
More precisely, let v 1 , . .
. , v p in G be all vertices in G such that for each valuation µ [ x (cid:55)→ v i ] it holds that [[ ψ, µ [ x (cid:55)→ v i ]] B G = ⊤ .
As mentioned, it must be that p is at least m .
Using again the fact that ρ k (cid:0) wl ( t ) k (cid:1) =
ρ k (cid:0) C k +1 , ( t ) (cid:1) , we infer that the color wl ( t − 1) k
( G, ( v i , . . .
, v i )) is the same, for each such v i .
Now since gwl ( t − 1) k ( G ) =
gwl ( t − 1) k
( H ) , it is not difficult to see that there must be exactly p vertices w 1 , . . .
, w p in H such that wl ( t − 1) k
( G, ( v i , . . .
, v i ))
=
wl ( t − 1) k
( H, ( w i , . . . , w i )) .
Otherwise, it would simply not be the case that the aggregation step of the colors, assigned by k - WL is the same in G and H .
By the connection to logic, we again know that for valuation µ [ x (cid:55)→ w i ] it holds that [[ ψ, µ [ x (cid:55)→ w i ]] B H = ⊤ .
It then follows that [[ φ, µ ]] B H =
⊤ for any valuation µ , which was to be shown.
Finally, we remark that ρ 0 (cid:0) gwl ( ∞ ) k (cid:1) =
ρ 0 (cid:0)
C k +1 (cid:1) follows from the preceding inclusions in (2).
Before moving to tensor languages, where we will use infinitary logics to simulate expressions in
TL k (Ω) and GTL (Ω) , we recall that, when considering the separation power of logics, we can freely move between the logics and their infinitary counterparts:
Theorem C.2.
The following identities hold for any t ≥ 0 , k ≥ 2 and s ≥ 0 :
[Equation]
Proof.
For identity (1), notice that we only need to prove that ρ 1 (cid:0) GC ( t ) (cid:1) ⊆
ρ 1 (cid:0) GC ( t ) ∞ ω (cid:1) , the other direction follows directly from the definition.
We point out the well-known fact that two tuples ( G, v ) and ( H, w ) belong to ρ 1 (cid:0) GC ( t ) (cid:1) if and only if the unravelling of G rooted at v up to depth t is isomorphic to the unravelling of H rooted at w up to root t .
Here the unravelling is the infinite tree whose root is the root node, and whose children are the neighbors of the root node (see e.g. Barcel´o et al.
; Otto (2019).
Now for the connection with infinitary logic.
Assume that the unravellings of G rooted at v and of H rooted at w up to level t are isomorphic, but assume for the sake of contradiction that there is a formula φ ( x ) in GC ( t ) ∞
ω such that [[ φ, µ v ]] B G ̸ =
[[ φ, µ w ]] B H , where µ v and µ w are any valuation mapping variable x to v and w , respectively.
Now since G and H are finite graphs, one can construct, from formula ϕ , a formula ϕ ′ in GC ( t ) such that [[ ψ, µ v ]] B G ̸ =
[[ ψ, µ w ]] B H .
Notice that this is in contradiction with our assumption that unravellings where isomorphic and therefore indistinguishable by formulae in GC ( t ) .
To construct ψ , consider an infinitary disjunction (cid:87)
a ∈
A α a .
Since G and H have a finite number of vertices, and the formulae have a finite number of variables, the number of different valuations from the variables to the vertices in G or H is also finite.
Thus, one can replace any extra copy of α a , α a ′ such that their value is the same in G and H .
The final result is a finite disjunction, and the truth value over G and H is equivalent to the original infinitary disjunction.
For identity (2) we refer to Corollary 2.4 in Otto (2017).
C. F ROM TL (Ω) TO C k ∞ ω AND GC ∞ ω
We are now finally ready to make the connection between expressions in TL (Ω) and the infinitary logics introduced earlier.
Proposition C.3.
For any expression φ ( x ) in TL k (Ω) and c ∈ R , there exists an expression ˜ φ c ( x ) in C k ∞ ω
such that [[ φ, v ]] G = c
if and only if [[ ˜ φ c , v ]] B G = ⊤ for any graph G =
( V G , E G , col G ) in G and v ∈ V kG .
Furthermore, if φ ( x ) ∈ GTL (Ω) then ˜ φ c ∈ GC ∞ ω .
Finally, if φ has summation depth t then ˜ φ c has quantifier rank t .
Proof.
We define ˜ φ c inductively on the structure of expressions in TL k (Ω) .
• φ ( x i , x j ) :
= 1 x i op x j .
Assume
first that op
is “
= ”.
We distinguish between (a) i ̸
=
j and
(b) i = j .
For case a),
if
c
=
1 , then we
define ˜ φ 1 ( x i , x j ) :
= ( x i = x j ) , if c = 0 , then we define ˜ φ 0 ( x i , x j ) :
= ¬ ( x i = x j ) , and if c ̸ = 0 , 1 , then we define ˜ φ c ( x i , x j ) :
= x i ̸
= x i .
For case (b), if c = 1 , then we define ˜ φ 1 ( x i , x j ) :
= ( x i = x i ) , and for any c ̸ = 1 , we define ˜ φ c ( x i , x j ) :
= ¬ ( x i = x i ) .
The case when op is “ ̸ = ” is treated analogously.
• φ ( x i ) :
= P ℓ ( x i ) .
If c = 1 , then we define ˜ φ 1 ( x i ) :
= P ℓ ( x i ) , if c = 0 , then we define ˜ φ 0 ( x i ) :
= ¬ P j ( x i ) .
For all other c , we define ˜ φ c ( x i , x j ) :
= ¬ ( x i = x i ) .
• φ ( x i , x j ) :
=
E ( x i , x j ) .
If c = 1 , then we define ˜ φ 1 ( x i , x j ) :
= E ( x i , x j ) , if c = 0 , then we define ˜ φ 0 ( x i , x j ) :
=
¬ E ( x i , x j ) .
For all other c , we define ˜ φ c ( x i , x j ) :
= ¬ ( x i = x i ) .
• φ :=
φ 1 + φ 2 .
We observe that [[ φ, v ]] G = c if and only if there are c 1 , c 2 ∈ R such that [[ φ 1 , v ]] G = c 1 and [[ φ 2 , v ]] G = c 2 and c
= c 1 + c 2 .
Hence, it suffices to define
[Equation]
where ˜ φ c 1 1 and ˜ φ c 2 2 are the expressions such that [[ φ 1 , v ]]
G = c
if and only if [[ ˜ φ c 1 1 , v ]] B G = ⊤ and [[ φ 2 , v ]]
G = c
if and only if [[ ˜ φ c 2 2 , v ]] B G = ⊤ , which exist by induction.
• φ := φ 1 · φ 2 .
This is case is analogous to the previous one.
Indeed, [[ φ, v ]]
G = c
if and only if there are c 1 , c 2 ∈ R such that [[ φ 1 , v ]] G = c 1 and [[ φ 2 , v ]] G = c 2 and c = c 1 · c 2 .
Hence, it suffices to define
[Equation]
• φ := a · φ 1 .
This is case is again dealt with in a similar way.
Indeed, [[ φ, v ]]
G = c if and only if there is a c 1 ∈ R
such that [[ φ 1 , v ]] G = c 1 and c = a · c 1 .
Hence, it suffices to define
[Equation]
• φ := f ( φ 1 , . .
. , φ p ) with f : R p
→ R .
We observe that [[ φ, v ]] G = c if and only if there are c 1 , . . .
, c p ∈ R such that c
= f ( c 1 , . .
c p ) and [[ φ i , v ]]
G = c i for i ∈
[ p ] .
Hence, it suffices to define
[Equation]
• φ := (cid:80) x
i φ 1 .
We observe that [
[ φ, µ ]] G = c implies that we can partition V G into ℓ parts
V 1 , . .
. , V ℓ , of sizes m 1 , . . .
, m ℓ , respectively, such that [[ φ 1 , µ [ x i → v ]]]
G = c i for each v ∈ V i , and such that all c
i ’s are pairwise distinct and c = (cid:80) ℓi
=1 c i · m i .
It now suffices to consider the following formula
[Equation]
where ∃
= m i x i ψ is shorthand notation for ∃ ≥
m i x i ψ ∧ ¬∃ ≥ m i +1 x i ψ , and ∀ x
i ψ denotes ¬∃ ≥ 1 x i ¬ ψ .
This concludes the construction of ˜ φ c .
We observe that we only introduce a quantifiers when φ = (cid:80) x
i φ 1 and hence if we assume by induction that summation depth and quantifier rank are in sync, then if φ 1 has summation depth t − 1 and thus ˜ φ c 1 has quantifier rank t − 1 for any c ∈ R , then φ has summation depth t , and as can be seen from the definition of ˜ φ c , this formula has quantifier rank t , as desired.
It remains to verify the claim about guarded expressions.
This is again verified by induction.
The only case requiring some attention is φ ( x 1 ) :
= (cid:80) x 2 E ( x 1 , x 2 )
∧ φ 1 ( x 2 ) for which we can define
[Equation]
∃
[Equation]
which is a formula in GC again only adding one to the quantifier rank of the formulae ˜ φ c 1 for c ∈ R .
So also here, we have the one-to-one correspondence between summation depth and quantifier rank.
C. P ROOF OF T HEOREM 4.1, 4.2, 4.3 AND 4.
Proposition C.4.
We have the following inclusions:
For any t ≥ 0 and any collection Ω of functions:
• ρ 1 (cid:0) cr ( t ) (cid:1) ⊆
ρ 1 (cid:0) GTL ( t ) (Ω) (cid:1) ;
• ρ 1 (cid:0) vwl ( t ) k (cid:1) ⊆
ρ 1 (cid:0) TL ( t )
k +1 (Ω) (cid:1) ; and • ρ 0 (cid:0) gwl ( t ) k (cid:1) ⊆
ρ 0 (cid:0) TL ( t +1)
k +1 (Ω) (cid:1) .
Proof.
We first show the second bullet by contraposition.
That is, we show that if ( G, v ) and ( H, w ) are not in ρ 1 (cid:0) TL ( t )
k
+1 (Ω) (cid:1) , then neither are they in ρ 1 (cid:0) vwl ( t ) k (cid:1) .
Indeed, suppose that there exists an expression φ ( x 1 ) in TL ( t )
k +1 (Ω) such that [[ φ, v ]]
G = c ̸ = c ′ =
[[ φ, w ]] H .
From Proposition C.3 we know that there exists a formula ˜ φ c in C k +1 , ( t ) ∞
ω such that [[ ˜ φ c , v ]] B G = ⊤ and [[ ˜ φ c , w ]] B H = ⊥ .
Hence, ( G, v ) and ( H, w ) do no belong to ρ 1 (cid:0) C k +1 , ( t ) ∞ ω (cid:1) .
Theorem C.2 implies that ( G, v ) and ( H, w ) also do not belong to ρ 1 (cid:0) C k +1 , ( t ) (cid:1) .
Finally, Proposition C.1 implies that ( G, v ) and ( H, w ) do not belong to ρ 1 ( vwl ( t ) k (cid:1) , as desired.
The third bullet is shown in precisely the same, but using the identities for ρ 0 rather than ρ 1 , and gwl ( t ) k rather than vwl ( t ) k .
Also the first bullet is shown in the same way, using the connection between GTL ( t ) (Ω) , GC 2 , ( t ) ∞ ω ,
GC ( t ) and cr ( t ) , as given by Proposition C.1, Theorem C.2, and Proposition C.3.
We next show that our tensor languages are also more separating than the color refinement and k -dimensional Weisfeiler-Leman algorithms.
Proposition C.5.
We have the following inclusions:
For any t ≥ 0 and any collection Ω of functions:
• ρ 1 (cid:0) GTL ( t ) (Ω) (cid:1) ⊆
ρ 1 (cid:0) cr
( t ) (cid:1) ; • ρ 1 (cid:0) TL ( t )
k +1 (Ω) (cid:1) ⊆
ρ 1 (cid:0) vwl ( t ) k (cid:1) ; and • ρ 0 (cid:0) TL ( t + k ) k
+1 (Ω) (cid:1) ⊆
ρ 0 (cid:0) gwl ( t ) k (cid:1) .
Proof.
For any of these inclusions to hold, for any Ω , we need to show the inclusion without the use of any functions.
We again use the connections between the color refinement and k -dimensional
Weisfeiler-Leman algorithms and finite variable logics as stated in Proposition C.1.
More precisely, we show for any formula φ ( x ) ∈ C k, ( t ) there exists an expression ˆ φ ( x ) ∈ TL ( t ) k
such that for any graph G in G , [[ φ, v ]] B G = ⊤ implies [[ ˆ φ, v ]] G = 1 and [[ φ, v ]] B G = ⊥ implies [[ ˆ φ, v ]] G = 0 .
By appropriately selecting k and t and by observing that when φ ( x ) ∈ GC then ˆ φ ( x ) ∈ GTL , the inclusions follow.
The construction of ˆ φ ( x ) is by induction on the structure of formulae in C k .
• φ :=
( x i = x j ) .
Then, we define ˆ φ := 1 x i = x j . •
φ := P ℓ ( x i ) .
Then, we define ˆ φ := P ℓ ( x i ) .
• φ :=
E ( x i , x j ) .
Then, we define ˆ φ := E ( x i , x j ) .
• φ := ¬ φ 1 .
Then, we define ˆ φ := 1 x i =
x i − ˆ φ 1 .
• φ := φ 1 ∧ φ 2 .
Then, we define ˆ φ :
= ˆ φ 1 · ˆ φ 2 .
• φ := ∃ ≥ m x i φ 1 .
Consider a polynomial p ( x ) :
= (cid:80)
j a j x j such that p ( x ) = 0 for x ∈ { 0 , 1 , . .
, m − 1 } and p ( x ) = 1 for x ∈ { m, m + 1 , . . . , n } .
Such a polynomial exists by interpolation.
Then, we define ˆ φ := (cid:80) j a j (cid:0)(cid:80) x i ˆ φ 1 (cid:1) j .
We remark that we here crucially rely on the assumption that G contains graphs of fixed size n and that TL k is closed under linear combinations and product.
Clearly, if φ ∈ GC , then the above translations results in an expression ˆ φ ∈ GTL (Ω) .
Furthermore, the quantifier rank of φ is in oneto-one correspondence to the summation depth of ˆ φ .
We can now apply Proposition C.1.
That is, if ( G, v ) and ( H, w ) are not in ρ 1 (cid:0) cr ( t ) (cid:1) then by Proposition C.1, there exists a formula φ ( x ) in GC ( t ) such that [[ φ, v ]] B G = ⊤ ̸
= [[ φ, w ]] B H = ⊥ .
We have just shown when we consider ˜ φ , in GTL ( t ) , also [[ ˜ φ, v ]] G ̸ =
[[ ˜ φ, w ]] H holds.
Hence, ( G, v ) and ( H, w ) are not in ρ 1 (cid:0) GTL ( t ) (Ω) (cid:1) either, for any Ω .
Hence, ρ 1 (cid:0) GTL ( t ) (Ω) (cid:1) ⊆
ρ 1 (cid:0) cr
( t ) (cid:1) holds.
The other bullets are shown in the same way, again by relying on Proposition C.1 and using that we can move from vwl ( t ) k and gwl ( t ) k to logical formulae, and to expressions in TL ( t )
k +1 and TL ( t + k )
k +1 , respectively, to separate ( G, v ) from ( H, w ) or G from H , respectively.
Theorems 4.1, 4.2, 4.3 and 4.4 now follow directly from Propositions C.4 and C.5.
C. O THER AGGREGATION FUNCTIONS
As is mentioned in the main paper, our upper bound results on the separation power of tensor languages (and hence also of GNN s represented in those languages) generalize easily when other aggregation functions than summation are used in TL expressions.
To clarify what we understand by an aggregation function, let us first recall the semantics of summation aggregation.
Let φ := (cid:80)
x
i φ 1 , where (cid:80) x i represents summation aggregation, let
G =
( V G , E G , col G ) be a graph, and let ν be a valuation assigning index variables to vertices in V G .
The semantics is then given by:
[Equation]
as explained in Section 3.
Semantically, we can alternatively view (cid:80) x
i φ 1 as a function which takes the sum of the elements in the following multiset of real values:
[Equation]
One can now consider, more generally, an aggregation function F as a function which assigns to any multiset of values in R a single real value.
For example, F could be max , min , mean , . . . .
Let Θ be such a collection of aggregation functions.
We next incorporate general aggregation function in tensor language.
First, we extend the syntax of expressions in TL (Ω) by generalizing the construct (cid:80) x i φ in the grammar of TL (Ω) expression.
More precisely, we define TL (Ω , Θ) as the class of expressions, formed just like tensor language expressions, but in which two additional constructs, unconditional and conditional aggregation , are allowed.
For an aggregation function F we define:
[Equation]
where in the latter construct (conditional aggregation) the expression φ ( x j ) represents a TL (Ω , Θ) expression whose only free variable is x j .
The intuition behind these constructs is that unconditional aggregation aggr
Fx j ( φ ) allows for aggregating, using aggregate function F , over the values of φ where x j ranges unconditionally over all vertices in the graph.
In contrast, for conditional aggregation aggr Fx j (cid:0) φ ( x j ) | E ( x i , x j ) (cid:1) , aggregation by F of the values of φ ( x j ) is conditioned on the neighbors of the vertex assigned to x i .
That is, the vertices for x j range only among the neighbors of the vertex assigned to x i .
More specifically, the semantics of the aggregation constructs is defined as follows:
[Equation]
We remark that we can also consider aggregations functions F over multisets of values in R ℓ for some ℓ ∈ N .
This requires extending the syntax with aggr Fx j ( φ 1 , . .
. , φ ℓ ) for unconditional aggregation and with aggr Fx j (cid:0)
φ 1 ( x j ) , . .
. , φ ℓ ( x j ) | E ( x i , x j ) (cid:1) for conditional aggregation.
The semantics is as expected: F ( {{ (([[ φ 1 , ν
[ x j (cid:55)→ v ]]] G , . .
, [[ φ ℓ , ν [ x j (cid:55)→ v ]]] G ) | v ∈ V G }} ) ∈ R and
F ( {{ (([[ φ 1 , ν [ x j (cid:55)→ v ]]] G , . . .
, [[ φ ℓ , ν [ x j (cid:55)→ v ]]] G ) | v ∈ V G , ( ν ( x i ) , v ) ∈
E G }} ) ∈ R .
The need for considering conditional and unconditional aggregation separately is due to the use of arbitrary aggregation functions.
Indeed, suppose that one uses an aggregation function F for which 0 ∈ R is a neutral value .
That is, for any multiset X of real values, the equality F ( X )
=
F ( X ⊎{ 0 } ) holds.
For example, the summation aggregation function satisfies this property.
We then observe:
[Equation] [Equation]
In other words, unconditional aggregation can simulate conditional aggregation.
In contrast, when 0 is not a neutral value of the aggregation function F , conditional and unconditional aggregation behave differently.
Indeed, in such cases aggr Fx j (cid:0) φ ( x j ) | E ( x i , x j ) (cid:1) and aggr Fx j ( φ ( x j ) · E ( x i , x j )) may evaluate to different values, as illustrated in the following example.
As aggregation function F we take the average avg ( X ) :
= 1 | X | (cid:80) x ∈ X x for multisets X of real values.
We remark that 0 ’s in X contribute to the size of X and hence 0 is not a neutral element of avg .
Now, let us consider the expressions φ
[Equation]
x i ) :
=
aggr avg x j ( 1 x j = x j · E ( x i , x j )) and φ 2 ( x i ) :
=
aggr avg x j ( 1 x j = x j | E ( x i , x j )) .
Let ν be such that ν ( x i )
= v .
Then, [[ φ 1 , ν ]] G results in applying the average to the multiset {{ 1 w = w ·
E ( v, w ) | w ∈ V G }} which includes the value 1 for every w ∈ N G ( v ) and a 0 for every nonneighbor w ̸∈ N G ( v ) .
In other words, [[ φ 1 , ν ]] G results in | N G ( v ) | / | V G | .
In contrast, [[ φ 2 , ν ]] G results in applying the average to the multiset {{ 1 w = w | w ∈ V G , ( v, w )
∈
E G }} .
In other words, this multiset only contains the value 1 for each w ∈ N G ( v ) , ignoring any information about the non-neighbors of v .
In other words, [[ φ 2 , ν ]] G results in | N G ( v ) | / | N G ( v ) | = 1 .
Hence, conditional and unconditional aggregation behave differently for the average aggregation function.
This said, one could alternative use a more general variant of conditional aggregation of the form aggr Fx j ( φ | ψ ) with as semantics [[ aggr Fx j ( φ | ψ ) , ν ]] G := F (cid:0) {{ [[ φ, ν [ x j → v ]]] G | v ∈ V G , [[ ψ, ν [ x j → v ]] G ̸ = 0 }} (cid:1) where one creates a multiset only for those valuations ν [ x j → v ] for which the condition ψ evaluates to a non-zero value.
This general form of aggregation includes conditional aggregation, by replacing ψ with E ( x i , x j ) and restricting φ , and unconditional aggregation, by replacing ψ with the constant function 1 , e.g., 1 x j = x j .
In order not to overload the syntax of TL expressions, we will not discuss this general form of aggregation further.
The notion of free index variables for expressions in TL (Ω , Θ) is defined as before, where now free ( aggr Fx j ( φ )) :
= free ( φ ) \ { x j } , and where free ( aggr Fx j (cid:0) φ ( x j ) | E ( x i , x j )) :
= { x i } (recall that free ( φ ( x j ))
=
{ x j } in conditional aggregation).
Moreover, summation depth is replaced by the notion of aggregation depth , agd ( φ ) , defined in the same way as summation depth except that agd ( aggr Fx j ( φ )) :
= agd ( φ ) + 1 and agd ( aggr Fx j ( φ ( x j ) | E ( x i , x j )) :
= agd ( φ ) + 1 .
Similarly, the fragments TL k (Ω , Θ) and its aggregation depth restricted fragment TL ( t ) k
(Ω , Θ) are defined as before, using aggregation depth rather than summation depth.
For the guarded fragment, GTL (Ω , Θ) , expressions are now restricted such that aggregations must occur only in the form aggr
Fx j ( φ ( x j ) | E ( x i , x j )) , for i, j ∈ [2] .
In other words, aggregation only happens on multisets of values obtained from neighboring vertices.
We now argue that our upper bound results on the separation power remain valid for the extension
TL (Ω , Θ) of TL (Ω) with arbitrary aggregation functions Θ .
Proposition C.6.
We have the following inclusions: For any t ≥ 0 , any collection Ω of functions and any collection Θ of aggregation functions:
• ρ 1 (cid:0) cr ( t ) (cid:1) ⊆
ρ 1 (cid:0) GTL ( t ) (Ω , Θ) (cid:1) ; • ρ 1 (cid:0) vwl ( t ) k (cid:1) ⊆
ρ 1 (cid:0) TL ( t )
k +1 (Ω , Θ) (cid:1) ; and • ρ 0 (cid:0) gwl ( t ) k (cid:1) ⊆
ρ 0 (cid:0) TL ( t +1)
k +1 (Ω , Θ) (cid:1) .
Proof.
It suffices to show that Proposition C.3 also holds for expressions in the fragments of
TL (Ω , Θ) considered.
In particular, we only need to revise the case of summation aggregation (that is, φ := (cid:80) x
i φ 1 ) in the proof of Proposition C.3.
Indeed, let us consider the more general case when one of the two aggregating functions are used.
• φ :=
aggr
Fx i ( φ 1 ) .
We then define
[Equation]
where C ( m 1 , . . .
, m ℓ , F ) now consists of all ( c, c 1 , . .
. , c ℓ ) ∈ R ℓ +1 such that
[Equation]
• φ :=
aggr
Fx i ( φ 1 ( x i ) | E ( x j , x i )) .
We then define
[Equation]
where C ( m 1 , . . .
, m ℓ , F ) again consists of all ( c, c 1 , . . .
, c ℓ ) ∈ R ℓ +1 such that
[Equation]
[Table]
m
[Table]
It is readily verified that [[ aggr Fx i ( φ ) , v ]] G = c iff
[[ ˜ φ c , v ]] B G = ⊤ , and [[ aggr Fx i ( φ ( x i ) |
E ( x j , x i )) , v ]] G = c iff [[ ˜ φ c , v ]] B G = ⊤ , as desired.
For the guarded case, we note that the expression ˜ φ c above yields a guarded expression as long conditional aggregation is used of the form aggr Fx i ( φ ( x i ) | E ( x j , x i )) with i, j ∈ [2] , so we can reuse the argument in the proof of Proposition C.3 for the guarded case.
We will illustrate later on (Section D) that this generalization allows for assessing the separation power of GNN s that use a variety of aggregation functions.
The choice of supported aggregation functions has, of course, an impact on the ability of TL (Ω , Θ) to match color refinement or the k - WL procedures in separation power.
The same holds for GNN s, as shown by Xu et al.
And indeed, the proof of Proposition C.5 relies on the presence of summation aggregation.
We note that most lower bounds on the separation power of GNN s in terms of color refinement or the k - WL procedures assume summation aggregation since summation suffices to construct injective sum-decomposable functions on multisets (Xu et al., 2019; Zaheer et al., 2017), which are used to simulate color refinement and k - WL .
A more in-depth analysis of lower bounding GNN s with less expressive aggregation functions, possibly using weaker versions of color refinement and k - WL is left as future work.
C. G ENERALIZATION TO G RAPHS WITH REAL - VALUED VERTEX LABELS
We next consider the more general setting in which col G : V G → R ℓ for some ℓ ∈ N .
That is, vertices in a graph can carry real-valued vectors.
We remark that no changes to neither the syntax nor the semantics of TL expressions are needed, yet note that [[ P s ( x ) , ν ]] G := col G ( ν ) s is now an element in R rather than 0 or 1 , for each s ∈ [ ℓ ] .
A first observation is that the color refinement and k - WL procedures treat each real value as a separate label.
That is, two values that differ only by any small ϵ > 0 , are considered different.
The proofs of Theorem 4.1, 4.2, 4.3 and 4.4 rely on connections between color refinement and k - WL and the finite variable logics GC and C k +1 , respectively.
In the discrete context, the unary predicates
P s ( x ) used in the logical formulas indicate which label vertices have.
That is, [[ P s , v ]] B G =
⊤ iff col G ( v ) s = 1 .
To accommodate for real values in the context of separation power, these logics now need to be able to differentiate between different labels, that is, different real numbers.
We therefore extend the unary predicates allowed in formulas.
More precisely, for each dimension s ∈ [ ℓ ] , we now have uncountably many predicates of the form P s,r , one for each r ∈ R .
In any formula in GC or C k +1 only a finite number of such predicates may occur.
The Boolean semantics of these new predicates is as expected:
[Equation]
In other words, in our logics, we can now detect which real-valued labels vertices have.
Although, in general, the introduction of infinite predicates may cause problems, we here consider a specific setting in which the vertices in a graph have a unique label.
This is commonly assumed in graph learning.
Given this, it is easily verified that all results in Section C.2 carry over, where all logics involved now use the unary predicates P s,r with s ∈ [ ℓ ] and r ∈ R .
The connection between TL and logics also carries over.
First, for Proposition C.3 we now need to connect TL expressions, that use a finite number of predicates P s , for s ∈ [ ℓ ] , with the extended logics having uncountably many predicates P s,r , for s ∈ [ ℓ ] and r ∈ R , at their disposal.
It suffices to reconsider the case φ ( x i ) =
P s ( x i ) in the proof of Proposition C.3.
More precisely, [[ P s ( x i ) , ν ]] G can now be an arbitrary value c ∈ R .
We now simply define ˜ φ c ( x i ) :
=
P s,c ( x i ) .
By definition [[ P s ( x i ) , ν ]]
G = c if and only if [[ P s,c ( x i ) , ν ]] B G = ⊤ , as desired.
The proof for the extended version of proposition C.5 now needs a slightly different strategy, where we build the relevant TL formula after we construct the contrapositive of the Proposition.
Let us first show how to construct a TL formula that is equivalent to a logical formula on any graph using only labels in a specific (finite) set R of real numbers.
In other words, given a set R of real values, we show that for any formula φ ( x ) ∈ C k, ( t ) using unary predicates P s,r such that r ∈ R , we can construct the desired ˆ φ .
As mentioned, we only need to reconsider the case φ ( x i ) :
= P s,r ( x i ) .
We define
[Equation]
Then, [[ ˆ φ, ν ]] G evaluates to (cid:81)
[Equation]
Indeed, if [[ P s,r , ν ]] = ⊤ , then col G ( v ) s
= r
and hence [[ P s , v ]] G = r , resulting in the same nominator and denominator in the above fraction.
If [[ P s,r , ν ]] = ⊥ , then col G ( v ) s = r ′ for some value r ′ ∈ R with r ̸ = r ′ .
In this case, the nominator in the above fraction becomes zero.
We remark that this revised construction still results in a guarded TL expression, when the input logical formula is guarded as well.
Coming back to the proof of the extended version of Proposition C.5, let us show the proof for the the fact that ρ 1 (cid:0) GTL ( t ) (Ω) (cid:1) ⊆
ρ 1 (cid:0) cr
( t ) (cid:1) , the other two items being analogous.
Assume that there is a pair ( G, v ) and ( H, w ) which is not in ρ 1 (cid:0) cr ( t ) (cid:1) .
Then, by Proposition C.1, applied on graphs with real-valued labels, there exists a formula φ ( x ) in GC ( t ) such that [[ φ, v ]] B G = ⊤ ̸
= [[ φ, w ]] B H = ⊥ .
We remark that φ ( x ) uses finitely many P s,r predicates.
Let R be the set of real values used in both G and H (and φ ( x ) ).
We note that R is finite.
We invoke the construction sketched above, and obtain a formula ˆ φ in GTL ( t ) such that [[ ˜ φ, v ]] G ̸ =
[[ ˜ φ, w ]] H .
Hence, ( G, v ) and ( H, w ) is not in ρ 1 (cid:0) GTL ( t ) (Ω) (cid:1) either, for any Ω , which was to be shown.
D D ETAILS OF S ECTION
We here provide some additional details on the encoding of layers of GNN s in our tensor languages, and how, as a consequence of our results from Section 4, one obtains a bound on their separation power.
This section showcases that it is relatively straightforward to represent GNN s in our tensor languages.
Indeed, often, a direct translation of the layers, as defined in the literature, suffices.
D. COLOR R EFINEMENT
We start with GNN architectures related to color refinement, or in other words, architectures which can be represented in our guarded tensor language.
GraphSage.
We first consider a “basic” GNN , that is, an instance of GraphSage (Hamilton et al., 2017) in which sum aggregation is used.
The initial features are given by F ( 0 ) =
( f ( 0 ) 1 , . .
f ( 0 ) d 0 )
where f ( 0 ) i ∈ R n × 1 is a hot-one encoding of the i th vertex label in G .
We can represent the initial embedding easily in GTL ( 0 ) , without the use of any summation.
Indeed, it suffices to define φ ( 0 ) i ( x 1 ) :
= P i ( x 1 ) for i ∈ [ d 0 ] .
We have F ( 0 ) vj
=
[[ φ ( 0 ) j , v ]] G for j ∈ [ d 0 ] , and thus the initial features can be represented by simple expressions in GTL ( 0 ) .
Assume now, by induction, that we can also represent the features computed by a basic GNN in layer t − 1 .
That is, let F ( t − 1 ) ∈ R n × d t − 1 be those features and for each i ∈
[ d t − 1 ] let φ ( t − 1 ) i
( x 1 ) be expressions in GTL ( t − 1 )
( σ ) representing them.
We assume that, for each i ∈
[ d t − 1 ] , F ( t − 1 ) vi =
[[ φ ( t − 1 ) i , v ]] G .
We remark that we assume that a summation depth of t − 1 is needed for layer t − 1 .
Then, in layer t , a basic GNN computes the next features as
[Equation]
where A ∈ R n × n is the adjacency matrix of G , V ( t ) and W ( t ) are weight matrices in R d t − 1 × d t ,
B ( t ) ∈
R n × d t is a (constant) bias matrix consist of n copies of b ( t ) ∈ R d t , and σ is some activation function.
We can simply use the following expressions φ ( t )
j ( x 1 ) , for j ∈ [ d t ] :
[Equation]
Here, W ( t ) ij , V ( t ) ij and b ( t )
j are real values corresponding the weight matrices and bias vector in layer t .
These are expressions in GTL ( t ) ( σ ) since the additional summation is guarded, and combined with the summation depth of t − 1 of φ ( t − 1 )
i , this results in a summation depth of t for layer t .
Furthermore, F ( t ) vi =
[[ φ ( t ) i , v ]] G , as desired.
If we denote by bGNN ( t ) the class of t -layered basic
GNN s, then our results imply
[Equation]
and thus the separation power of basic GNN s is bounded by the separation power of color refinement.
We thus recover known results by Xu et al.
and Morris et al.
Furthermore, if one uses a readout layer in basic GNN s to obtain a graph embedding, one typically applies a function ro : R d t → R d t in the form of ro (cid:0)(cid:80) v ∈ V G F ( t ) v (cid:1) , in which aggregation takes places over all vertices of the graph.
This corresponds to an expression in TL ( t +1 )
( σ, ro ) : φ
[Equation]
ro j (cid:0)(cid:80) x 1 φ ( t − 1 ) j ( x 1 ) (cid:1) , where ro j is the projection of the readout function on the j the coordinate.
We note that this is indeed not a guarded expression anymore, and thus our results tell that
[Equation]
More generally, GraphSage allows for the use of general aggregation functions F on the multiset of features of neighboring vertices.
To cast the corresponding layers in TL (Ω) , we need to consider the extension TL (Ω , Θ) with an appropriate set Θ of aggregation functions, as described in Section C.5.
In this way, we can represent layer t by means of the following expressions φ ( t )
j ( x 1 ) , for j ∈ [ d t ] .
[Equation]
which is now an expression in GTL ( t ) ( { σ } , Θ) and hence the bound in terms of t iterations of color refinement carries over by Proposition C.6.
Here, Θ simply consists of the aggregation functions used in the layers in GraphSage.
GCNs.
Graph Convolution Networks ( GCN s) (Kipf & Welling, 2017) operate alike basic GNN s except that a normalized Laplacian D − 1 / 2 ( I + A ) D − 1 / 2 is used to aggregate features, instead of the adjacency matrix A .
Here, D − 1 / 2 is the diagonal matrix consisting of reciprocal of the square root of the vertex degrees in G plus 1 .
The initial embedding F ( 0 ) is just as before.
We use again d t to denote the number of features in layer t .
In layer t > 0 , a GCN computes F ( t ) :
= σ ( D − 1 / 2 ( I + A ) D − 1 / 2 · F ( t − 1 ) W ( t ) + B ( t ) ) .
If, in addition to the activation function σ we add the function
[Equation]
1 √ x +1 to Ω , we can represent the GCN layer, as follows.
For j ∈ [ d t ] , we define the GTL ( t +1 )
( σ, 1 √ x +1 ) expressions
[Equation]
where we omitted the bias vector for simplicity.
We again observe that only guarded summations are needed.
However, we remark that in every layer we now add two the overall summation depth, since we need an extra summation to compute the degrees.
In other words, a t -layered GCN correspond to expressions in GTL ( 2 t ) ( σ, 1 √ x +1 ) .
If we denote by GCN ( t ) the class of t -layered GCN s, then our results imply
[Equation]
We remark that another representation can be provided, in which the degree computation is factored out (Geerts et al., 2021a), resulting in a better upper bound ρ 1 (cid:0) cr ( t +1) (cid:1) ⊆
ρ 1 (cid:0) GCN ( t ) (cid:1) .
In a similar way as for basic GNN s, we also have ρ 0 (cid:0) gcr ( t +1) (cid:1) ⊆
ρ 0 (cid:0) GCN ( t ) + readout (cid:1) .
SGCs.
As an other example, we consider a variation of Simple Graph Convolutions ( SGC s) (Wu et al., 2019), which use powers the adjacency matrix and only apply a non-linear activation function at the end.
That is, F :=
σ ( A p · F ( 0 ) · W ) for some p ∈ N and W ∈ R d 0 × d 1 .
We remark that SGC s actually use powers of the normalized Laplacian, that is, F := σ (cid:0) ( D − 1 /
( I + A G ) D − 1 / 2 ))
p ·
F ( 0 ) · W (cid:1) but this only incurs an additional summation depth as for GCN s. We focus here on our simpler version.
It should be clear that we can represent the architecture in TL ( p )
p
+
(Ω) by means of the expressions:
[Equation]
for j ∈ [ d 1 ] .
A naive application of our results would imply an upper bound on their separation power by p - WL .
We can, however, use Proposition 4.5.
Indeed, it is readily verified that these expressions have a treewidth of one, because the variables form a path.
And indeed, when for example, p = 3 , we can equivalently write φ ( t ) j
( x 1 ) as
[Equation]
by reordering the summations and reusing index variables.
This holds for arbitrary p .
We thus obtain guarded expressions in GTL ( p ) ( σ ) and our results tell that t -layered SGC s are bounded by cr ( p ) for vertex embeddings, and by gcr ( p ) for SGC s + readout .
Principal Neighbourhood Aggregation.
Our next example is a GNN in which different aggregation functions are used: Principal Neighborhood Aggregation ( PNA ) is an architecture proposed by
Corso et al.
(2020) in which aggregation over neighboring vertices is done by means of mean , stdv , max and min , and this in parallel.
In addition, after aggregation, three different scalers are applied.
Scalers are diagonal matrices whose diagonal entries are a function of the vertex degrees.
Given the features for each vertex v computed in layer t − 1 , that is, F ( t − 1 ) v : [Equation] PNA computes v ’s new features in layer t in the following way (see layer definition (8) in (Corso et al., 2020)).
First, vectors G ( t ) v : ∈ R 1 × 4 ℓ are computed such that
G ( t ) vj =   mean (cid:0) {{ mlp j ( F ( t − 1 ) w : ) | w ∈ N G ( v ) }} (cid:1) for 1 ≤ j ≤ ℓ stdv (cid:0) {{ mlp j ( F ( t − 1 ) w : ) | w ∈ N G ( v ) }} (cid:1) for ℓ + 1 ≤ j ≤ 2 ℓ max (cid:0) {{ mlp j ( F ( t − 1 ) w : ) | w ∈ N G ( v ) }} (cid:1) for 2 ℓ + 1 ≤ j ≤ 3 ℓ min (cid:0) {{ mlp j ( F ( t − 1 ) w : ) | w ∈ N G ( v ) }} (cid:1) for 3 ℓ + 1 ≤ j ≤ 4 ℓ, where mlp j : R ℓ → R is the projection of an MLP mlp : R ℓ → R ℓ on the j th coordinate.
Then, three different scalers are applied.
The first scaler is simply the identity, the second two scalers sand s 2 depend on the vertex degrees.
As such, vectors H ( t ) v : ∈
R 12 ℓ are constructed as follows:
[Equation]
1 ≤ j
[Equation] where s 1 and s 2 are functions from R → R (see (Corso et al., 2020) for details).
Finally, the new vertex embedding is obtained as
[Equation]
for some MLP mlp ′ :
R 12 ℓ → R ℓ .
The above layer definition translates naturally into expressions in TL (Ω , Θ) , the extension of TL (Ω) with aggregate functions (Section C.5).
Indeed, suppose that for each j ∈ [ ℓ ] we have TL (Ω , Θ) expressions φ ( t − 1 )
j ( x 1 ) such that [[ φ ( t − 1 ) j , v ]] G = F ( t − 1 ) vj for any vertex v .
Then, G ( t ) vj simply corresponds to the guarded expressions
[Equation]
x
[Equation]
|
[Equation] x [Equation] x 2 )) , for 1 ≤ j ≤ ℓ , and similarly for the other components of G ( t ) v : using the respective aggregation functions, stdv , max and min .
Then, H ( t ) vj corresponds to
[Equation]
[Equation]
for
[Equation]
where we use summation aggregation to compute the degree information used in the functions in the scalers s 1 and s 2 .
And finally,
[Equation]
represents F ( t ) vj .
We see that all expressions only use two index variables and aggregation is applied in a guarded way.
Furthermore, in each layer, the aggregation depth increases with one.
As such, a t -layered PNA can be represented in GTL ( t ) (Ω , Θ) , where Ω consists of the MLP s and functions used in scalers, and Θ consists of sum (for computing vertex degrees), and mean , stdv , max and min .
Proposition C.6 then implies a bound on the separation power by cr ( t ) .
Other example.
In the same way, one can also easily analyze GAT s (Velickovic et al., 2018) and show that these can be represented in GTL (Ω) as well, and thus bounds by color refinement can be obtained.
D. k - DIMENSIONAL W EISFEILER -L EMAN TESTS
We next discuss architectures related to the k -dimensional Weisfeiler-Leman algorithms.
For k = 1 , we discussed the extended GIN s in the main paper.
We here focus on arbitrary k ≥ 2 .
Folklore GNN s. We first consider the “Folklore” GNN s or k - FGNN s for short (Maron et al., 2019b).
For k ≥ 2 , k - FGNN s computes a tensors.
In particular, the initial tensor F ( 0 ) encodes atp k ( G, v ) for each v ∈ V kG .
We can represent this tensor by the following k 2 ( ℓ + 2) expressions in TL ( 0 ) k : φ ( 0 ) r,s,j ( x 1 , . . .
, x k ) :
=   1 x r = x s · P j ( x r ) for j ∈ [ ℓ ]
E ( x r , x s ) for j = ℓ +1 x r = x s for j = ℓ +, for r, s ∈ [ k ] and j ∈ [ ℓ + 2] .
We note: [[ φ ( 0 ) r,s,j , ( v 1 , . . .
, v k )]] G = F ( 0 ) v 1 ,...,v k ,r,s,j for all ( r, s, j ) ∈
[ k ] 2 × [ ℓ + 2] , as desired.
We let τ 0 :=
[ k ] 2 × [ ℓ + 2] and set d 0 = k 2 × ( ℓ + 2) .
Then, in layer t , a k - FGNN computes a tensor
[Equation]
where mlp ( t ) s : R d t − 1 → R d ′ t , for s ∈ [ k ] , and and mlp ( t ) 0 : R d t − 1 × d ′ t → R d t are MLP s.
We here use • to denote combinations of indices in τ d for F ( t ) and in τ d − 1 for F ( t − 1 ) .
Let F ( t − 1 ) ∈
R n k × d t − 1 be the tensor computed by an k - FGNN in layer t − 1 .
Assume that for each tuple of elements j in τ d t − 1 we have an expression φ ( t − 1 )
j ( x 1 , . . .
, x k ) satisfying [[ φ ( t − 1 ) j , ( v 1 , . .
, v k )]] G = F ( t − 1 ) v 1 ,...,v k , j and such that it is an expression in TL ( t − 1 )
k
+1 (Ω) .
That is, we need k + 1 index variables and a summation depth of t − 1 to represent layer t − 1 .
Then, for layer t , for each j ∈ τ d t , it suffices to consider the expression
[Equation]
where mlp ( t ) o, j and mlp ( t ) s, j are the projections of the MLP s on the j -coordinates.
We remark that we need k + 1 index variables, and one extra summation is needed.
We thus obtain expressions in
TL ( t ) k
+1 (Ω) for the t th layer, as desired.
We remark that the expressions are simple translations of the defining layer definitions.
Also, in this case, Ω consists of all MLP s. When a k - FGNN is used for vertex embeddings, we now simply add to each expression a factor (cid:81) ks =1 1 x
= x s .
As an immediate consequence of our results, if we denote by k - FGNN ( t ) the class of t -layered k - FGNN s, then for vertex embeddings:
[Equation]
in accordance with the known results from Azizian & Lelarge (2021).
When used for graph embeddings, an aggregation layer over all k -tuples of vertices is added, followed by the application of an
MLP .
This results in expressions with no free index variables, and of summation depth t + k , where the increase with k stems from the aggregation process over all k -tuples.
In view of our results, for graph embeddings:
[Equation]
in accordance again with Azizian & Lelarge (2021).
We here emphasize that the upper bounds in terms of k - WL are obtained without the need to know how k - WL works.
Indeed, one can really just focus on casting layers in the right tensor language!
We remark that Azizian & Lelarge (2021) define vertex embedding k - FGNN s in a different way.
Indeed, for a vertex v , its embedding is obtained by aggregating of all ( k − 1) tuples in the remaining coordinates of the tensors.
They define vwl k accordingly.
From the tensor language point of view, this corresponds to the addition of k − 1 to the summation depth.
Our results indicate that we loose the connection between rounds and layers, as in Azizian & Lelarge (2021).
This is the reason why we defined vertex embedding k - FGNN s in a different way and can ensure a correspondence between rounds and layers for vertex embeddings.
Other higher-order examples.
It is readily verified that t -layered k - GNN s (Morris et al., 2019) can be represented in TL ( t )
k +1 (Ω) , recovering the known upper bound by vwl ( t ) k
(Morris et al., 2019).
It is an equally easy exercise to show that 2 - WL -convolutions (Damke et al., 2020) and Ring- GNN s (Chen et al., 2019) are bounded by 2 - WL , by simply writing their layers in TL 3 (Ω) .
The invariant graph networks ( k - IGN s) (Maron et al., 2019b) will be treated in Section E, as their representation in TL k +
(Ω) requires some work.
D. A UGMENTED GNN S
Higher-order GNN architectures such as k - GNN s, k - FGNN s and k - IGN s, incur a substantial cost in terms of memory and computation (Morris et al., 2020).
Some recent proposals infuse more efficient GNN s with higher-order information by means of some pre-processing step.
We next show that the tensor language approach also enables to obtain upper bounds on the separation power of such “augmented” GNN s.
We first consider F - MPNN s (Barcel´o et al., 2021) in which the initial vertex features are augmented with homomorphism counts of rooted graph patterns.
More precisely, let P r be a connected rooted graph (with root vertex r ), and consider a graph G =
( V G , E G , col G ) and vertex v ∈ V G .
Then, hom ( P r , G v ) denotes the number of homomorphism from P to G , mapping r to v .
We recall that a homomorphism is an edge-preserving mapping between vertex sets.
Given a collection F =
{ P r 1 , . .
. , P rℓ } of rooted patterns, an F - MPNN runs an MPNN on the augmented initial vertex features:
[Equation]
Now, take any GNN architecture that can be cast in GTL (Ω) or TL 2 (Ω) and assume, for simplicity of exposition, that a t -layer GNN corresponds to expressions in GTL ( t ) (Ω) or TL ( t )
(Ω) .
In order to analyze the impact of the augmented features, one only needs to revise the expressions φ ( 0 )
j ( x 1 ) that represent the initial features.
In the absence of graph patterns, φ ( 0 ) j ( x 1 ) :
=
P j ( x 1 ) , as we have seen before.
By contrast, to represent ˜ F ( 0 ) vj we need to cast the computation of hom ( P ri , G v ) in TL .
Assume that the graph pattern P i consists of p vertices and let us identify the vertex set with [ p ] .
Furthermore, without of loss generality, we assume that vertex “ 1 ” is the root vertex in P i .
To obtain hom ( P ri , G v ) we need to create an indicator function for the graph pattern P i and then count how many times this indicator value is equal to one in G .
The indicator function for P i is simply given by the expression (cid:81)
uv ∈ E Pi E ( x u , x v ) .
Then, counting just pours down to summation over all index variables except the one for the root vertex.
More precisely, if we define
[Equation]
x p (cid:89) uv ∈ E Pi
E ( x u , x v ) , then [[ φ P i , v ]] G = hom ( P ri , G v ) .
This encoding results in an expression in TL p .
However, it is well-known that we can equivalently write φ P i ( x 1 ) as an expression ˜ φ P i ( x 1 ) in TL
k +1 where k is the treewidth of the graph P i .
As such, our results imply that F - MPNN s are bounded in separation power by k - WL where k is the maximal treewidth of graphs in F .
We thus recover the known upper bound as given in Barcel ´ o et al.
(2021) using our tensor language approach.
Another example of augmented GNN architectures are the Graph Substructure Networks ( GSN s ) (Bouritsas et al., 2020).
By contrast to F - MPNN s, subgraph isomorphism counts rather than homomorphism counts are used to augment the initial features.
At the core of a GSN thus lies the computation of sub ( P r , G v ) , the number of subgraphs H in G isomorphic to P (and such that the isomorphisms map r to v ).
In a similar way as for homomorphisms counts, we can either directly cast the computation of sub ( P r , G v ) in TL resulting again in the use of p index variables.
A possible reduction in terms of index variables, however, can be obtained by relying on the result (Theorem 1.1.) by Curticapean et al.
(2017) in which it shown that sub ( P r , G v ) can be computed in terms of homomorphism counts of graph patterns derived from P r .
More precisely, Curticapean et al.
(2017) define spasm ( P r ) as the set of graphs consisting of all possible homomorphic images of
P r .
It is then readily verified that if the maximal treewidth of the graphs in spasm ( P r ) is k , then sub ( P r , G v ) can be cast as an expression in TL
k
+1 .
Hence, GSN s using a pattern collection F can be represented in TL k +1 , where k is the maximal treewidth of graphs in any of the spams of patterns in F , and thus are bounded in separation power k - WL in accordance to the results by Barcel ´ o et al.
As a final example, we consider the recently introduced Message Passing Simplicial Networks ( MPSN s) (Bodnar et al., 2021).
In a nutshell, MPSN s are run on simplicial complexes of graphs instead of on the original graphs.
We sketch how our tensor language approach can be used to assess the separation power of MPSN s on clique complexes .
We use the simplified version of MPSN s which have the same expressive power as the full version of MPSN s (Theorem 6 in Bodnar et al.
We recall some definitions.
Let Cliques ( G ) denote the set of all cliques in G .
Given two cliques c and c ′ in Cliques ( G ) , define c ≺ c ′ if c ⊂ c ′ and there exists no c ′′ in Cliques ( G ) , such that c ⊂ c ′′ ⊂ c ′ .
We define Boundary ( c, G ) :
= {
c ′ ∈
Cliques ( G )
| c ′ ≺ c } and Upper ( c, G ) :
= { c ′ ∈
Cliques ( G ) | ∃ c ′′ ∈ Cliques ( G ) ,
c ′ ≺ c ′′ and c ≺ c ′′ } .
For each c in Cliques ( G ) we have an initial feature vector F ( 0 ) c : ∈ R 1 × ℓ .
Bodnar et al.
(2021) initialize all initial features with the same value.
Then, in layer t , for each c ∈ Cliques ( G ) , features are updated as follows:
[Equation]
[Equation] where F B and F U are aggregation functions and mlp B , mlp U and mlp are MLP s. With some effort, one can represent these computations by expressions in TL p (Ω , Θ) where p is largest clique in G .
As such, the separation power of clique-complex MPSN s on graphs of clique size at most p is bounded by p − 1 - WL .
And indeed, Bodnar et al.
(2021) consider Rook’s 4 × 4 graph, which contains a 4 -clique, and the Shirkhande graph, which does not contain a 4 -clique.
As such, the analysis above implies that clique-complex MPSN s are bounded by 2 - WL on the Shrikhande graph, and by 3 - WL on Rook’s graph, consistent with the observation in Bodnar et al.
A more detailed analysis of MPSN s in terms of summation depth and for other simplicial complexes is left as future work.
This illustrates again that our approach can be used to assess the separation power of a variety of GNN architectures in terms of k - WL , by simply writing them as tensor language expressions.
Furthermore, bounds in terms of k - WL can be used for augmented GNN s which form a more efficient way of incorporating higher-order graph structural information than higher-order GNN s.
D. S PECTRAL GNN S
In general, spectral GNN s are defined in terms of eigenvectors and eigenvalues of the (normalized) graph Laplacian (Bruna et al., 2014; Defferrard et al., 2016; Levie et al., 2019; Balcilar et al., 2021b)).
The diagonalization of the graph Laplacian is, however, avoided in practice, due to its excessive cost.
Instead, by relying on approximation results in spectral graph analysis (Hammond et al., 2011), the layers of practical spectral GNN s are defined in term propagation matrices consisting of functions, which operate directly on the graph Laplacian.
This viewpoint allows for a spectral analysis of spectral and “spatial” GNN s in a uniform way, as shown by Balcilar et al.
(2021b).
In this section, we consider two specific instances of spectral GNN s: ChebNet
(Defferrard et al., 2016) and CayleyNet (Levie et al., 2019), and assess their separation power in terms of tensor logic.
Our general results then provide bounds on their separation power in terms color refinement and 2 - WL , respectively.
Chebnet.
The separation power of ChebNet (Defferrard et al., 2016) was already analyzed in
Balcilar et al.
(2021a)
by representing them in the MATLANG matrix query language (Brijder et al., 2019).
It was shown (Theorem 2 (Balcilar et al., 2021a)) that it is only the maximal eigenvalue λ max of the graph Laplacian used in the layers of ChebNet that may result in the separation power of
ChebNet to go beyond 1 - WL .
We here revisit and refine this result by showing that, when ignoring the use of λ max , the separation power of Chebnet is bounded already by color refinement (which, as mentioned in Section 2, is weaker than 1 - WL for vertex embeddings).
In a nutshell, the layers of a ChebNet are defined in terms of Chebyshev polynomials of the normalized Laplacian L norm =
I − D − 1 / 2 · A · D − 1 / 2 and these polynomials can be easily represented in GTL (Ω) .
One can alternatively use the graph Laplacian L = D − A in a ChebNet , which allows for a similar analysis.
The distinction between the choice of L norm and L only shows in the needed summation depth (in as in similar way as for the GCN s described earlier).
We only consider the normalized Laplacian here.
More precisely, following Balcilar et al.
(2021a;b), in layer t , vertex embeddings are updated in a
ChebNet according to:
[Equation]
with
C
[Equation]
, for s ≥ 3 , and where λ max denotes the maximum eigenvalue of L norm .
We next use a similar analysis as in Balcilar et al.
(2021a).
That is, we ignore for the moment the maximal eigenvalue λ max and redefine C (2) as c L norm − I for some constant c .
We thus see that each C ( s ) is a polynomial of the form p s ( c, L norm ) :
= (cid:80) q s i =
a ( s ) i
( c ) · ( L norm )
i with scalar functions a ( s ) i : R → R and where we interpret ( L norm )
= I .
To upper bound the separation power using our tensor language approach, we can thus shift our attention entirely to representing ( L norm )
i · F ( t − 1) · W ( t − 1 ,s ) for powers i ∈ N .
Furthermore, since ( L norm ) i is again a polynomial of the form q
i
( D − 1 / 2 · A · D − 1 / 2 ) :
= (cid:80) r i j =0 b ij · ( D − 1 / 2 · A · D − 1 / 2 ) j , we can further narrow down the problem to represent
[Equation]
D − 1 / 2 ·
[Equation] [Equation]
− 1 /
[Equation]
in GTL (Ω) , for powers j ∈ N .
And indeed, combining our analysis for GCN s and SGC s results in expressions in GTL (Ω) .
As an example let us consider ( D − 1 / 2 · A · D − 1 / 2 ) 2 · F ( t − 1) · W ( t − 1) , that is we use a power of two.
It then suffices to define, for each output dimension j , the expressions:
[Equation]
where the φ ( t − 1) i ( x 1 ) are expressions representing layer t − 1 .
It is then readily verified that we can use ψ 2 j ( x 1 ) to cast layer t of a ChebNet in GTL (Ω) with Ω consisting of f 1 / √ x : R → R : x (cid:55)→ 1 √ x , f 1 /x :
R → R : x (cid:55)→ 1 x , and the used activation function σ .
We thus recover (and slightly refine)
Theorem 2 in Balcilar et al.
(2021a):
Corollary D.1.
On graphs sharing the same λ max values, the separation power of ChebNet is bounded by color refinement, both for graph and vertex embeddings.
A more fine-grained analysis of the expressions is needed when interested in bounding the summation depth and thus of the number of rounds needed for color refinement.
Moreover, as shown by Balcilar et al.
(2021a), when graphs have non-regular components with different λ max values,
ChebNet can distinguish them, whilst 1 - WL cannot.
To our knowledge, λ max cannot be computed in TL k (Ω) for any k .
This implies that it not clear whether an upper bound on the separation power can be obtained for ChebNet taking λ max into account.
It is an interesting open question whether there are two graphs G and H which cannot be distinguished by k - WL but can be distinguished based on λ max .
A positive answer would imply that the computation of λ max is beyond reach for
TL (Ω) and other techniques are needed.
CayleyNet.
We next show how the separation power of CayleyNet (Levie et al., 2019) can be analyzed.
To our knowledge, this analysis is new.
We show that the separation power of CayleyNet is bounded by 2 - WL .
Following Levie et al.
(2019) and Balcilar et al.
(2021b), in each layer t ,
a
CayleyNet updates features as follows:
[Equation]
with
[Equation]
where h is a constant, ı is the imaginary unit, and Re : C → C maps a complex number to its real part.
We immediately observe that a CayleyNet requires the use of complex numbers and matrix inversion.
So far, we considered real numbers only, but when our separation results are concerned, the choice between real or complex numbers is insignificant.
In fact, only the proof of Proposition C.3 requires a minor modification when working on complex numbers: the infinite disjunctions used in the proof now need to range over complex numbers.
For matrix inversion, when dealing with separation power, one can use different expressions in TL (Ω) for computing the matrix inverse, depending on the input size.
And indeed, it is well-known (see e.g., Csanky (1976))
that based on the characteristic polynomial of A , A − 1 for any matrix A ∈ R n × n can be computed as a polynomial − 1 c n (cid:80) n − 1 i =1 c i A n − 1 − i if c n ̸ = 0 and where each coefficient c i is a polynomial in tr ( A j ) , for various j .
Here, tr ( · ) is the trace of a matrix.
As a consequence, layers in CayleyNet can be viewed as polynomials in h L − ı
I with coefficients polynomials in tr (( h L − ı I ) j ) .
One now needs three index variables to represent the trace computations tr (( h L − ı I ) j ) .
Indeed, let φ 0 ( x 1 , x 2 ) be the TL 2 expression representing h L − ı I .
Then, for example, ( h L − ı I )
j can be computed in TLusing
[Equation]
and hence tr (( h L − ı I ) j ) is represented by (cid:80) x 1 (cid:80) x
2 φ j ( x 1 , x 2 ) · 1 x 1 = x 2 . .
In other words, we obtain expressions in TL 3 .
The polynomials in h L − ı I can be represented in TL 2 just as for ChebNet .
This implies that each layer in CayleyNet can be represented, on graphs of fixed size, by TL 3 (Ω) expressions, where Ω includes the activation function σ and the function Re .
This suffices to use our general results and conclude that CayleyNet s are bounded in separation power by 2 - WL .
An interesting question is to find graphs that can be separated by a CayleyNet but not by 1 - WL .
We leave this as an open problem.
E P ROOF OF T HEOREM 5.
We here consider another higher-order GNN proposal: the invariant graph networks or k - IGN s of
Maron et al. (2019b).
By contrast to k - FGNN s, k - IGN s are linear architectures.
If we denote by k - IGN ( t ) the class of t layered k - IGN s, then following inclusions are known (Maron et al., 2019b)
[Equation]
The reverse inclusions were posed as open problems in Maron et al.
(2019a)
and were shown to hold by Chen et al.
(2020) for k = 2 , by means of an extensive case analysis and by relying on properties of 1 - WL .
In this section, we show that the separation power of k - IGN s is bounded by that of ( k − 1) - WL , for arbitrary k ≥ 2 .
Theorem 4.2 tells that we can entirely shift our attention to showing that the layers of k - IGN s can be represented in TL k (Ω) .
In other words, we only need to show that k index variables are needed for the layers.
As we will see below, this requires a bit of work since a naive representation of the layers of k - IGN s use 2 k index variables.
Nevertheless, we show that this can be reduced to k index variables only.
By inspecting the expressions needed to represent the layers of k - IGN s in TL k (Ω) , we obtain that a t layer k - IGN ( t ) require expressions of summation depth of tk .
In other words, the correspondence between layers and summation depth is precisely in sync.
This implies, by Theorem 4.2:
[Equation]
where we ignore the number of layers.
We similarly obtain that ρ [Equation] k [Equation]
IGN
[Equation] = ρ [Equation] gwl
[Equation] 1 (cid:1) , hereby answering the open problem posed in Maron et al. (2019a).
Finally, we observe that the k - IGN s used in Maron et al.
(2019b) to show the inclusion ρ 1 (cid:0) k - IGN ( t ) (cid:1) ⊆
ρ 1 (cid:0) vwl ( t ) k − 1 (cid:1) are of very simple form.
By defining a simple class of k - IGN s, denoted by k - GIN s, we obtain
[Equation]
hereby recovering the layer/round connections.
We start with the following lemma:
Lemma E.1.
For any k ≥ 2 , a t layer k - IGN s can be represented in TL ( tk ) k
(Ω) .
Before proving this lemma, we recall k - IGN s. These are architectures that consist of linear equivariant layers.
Such linear layers allow for an explicit description.
Indeed, following Maron et al.
(2019c), let ∼ ℓ be the equality pattern equivalence relation on [ n ] ℓ such that for a , b ∈ [ n ] ℓ , a ∼ ℓ b if and only if a i = a j ⇔ b i =
b j for all j ∈ [ ℓ ] .
We denote by [ n ] ℓ / ∼ ℓ the equivalence classes induced by ∼ ℓ .
Let us denote by F ( t − 1 )
∈
R n k × d t − 1 the tensor computed by an k - IGN in layer t − 1 .
Then, in layer t , a new tensor in R n k × d t is computed, as follows.
For j ∈ [ d t ] and v 1 , . . .
, v k ∈ [ n ] k :
[Equation]
for activation function σ , constants c γ,i,j and b µ,j in R and where 1 ( v , w ) ∈ γ and 1 v ∈ µ are indicator functions for the 2 k -tuple ( v , w ) to be in the equivalence class γ ∈ [ n ] 2 k / ∼ 2 k and the k -tuple v to be in class µ ∈ [ n ] k / ∼ k .
As initial tensor F ( 0 ) one defines F ( 0 ) v 1 ,...,v k ,j := atp k ( G, v )
∈ R d 0 , with d 0 = 2( k 2 ) + kℓ where ℓ is the number of initial vertex labels, just as for k - FGNN s.
We remark that the need for having a summation depth of tk in the expressions in TL k (Ω) , or equivalently for requiring tk rounds of ( k − 1) - WL , can intuitively be explained that each layer of a k - IGN aggregates more information from “neighbouring” k -tuples than ( k − 1) - WL does.
Indeed, in each layer, an k - IGN can use previous tuple embeddings of all possible k -tuples.
In a single round of ( k − 1) - WL only previous tuple embeddings from specific sets of k -tuples are used.
It is only after an additional k − 1 rounds, that k - WL gets to the information about arbitrary k -tuples, whereas this information is available in a k - IGN in one layer directly.
Proof of Lemma E.1.
We have seen how F ( 0 ) can be represented in TL k (Ω) when dealing with k - FGNN s. We assume now that also the t − 1 th layer F ( t − 1 ) can be represented by d t − 1 expressions in TL ( ( t − 1) k ) k and show the the layer.
We first represent F ( t ) in TL 2 k (Ω) , based on the explicit description given earlier.
The expressions use index variables x 1 , . .
, x k and y 1 , . . .
, y k .
More specifically, for j ∈ [ d t ] we consider the expressions:
[Equation]
where ψ µ ( x 1 , . . . , x k ) is a product of expressions of the form 1 x i op x
j encoding the equality pattern µ , and similarly, ψ γ ( x 1 , . .
, x k , y 1 , . .
, y k ) is a product of expressions of the form 1 x i op x j , 1 y i op
y j and 1 x i op
y
j encoding the equality pattern γ .
These expressions are indicator functions for the their corresponding equality patterns.
That is,
[Equation]
We remark that in the expressions φ ( t )
j we have two kinds of summations: those ranging over a fixed number of elements (over equality patterns, feature dimension), and those ranging over the index variables y 1 , . . .
, y k .
The latter are the only ones contributing the summation depth.
The former are just concise representations of a long summation over a fixed number of expressions.
We now only need to show that we can equivalently write φ ( t ) j ( x 1 , . . .
, x k ) as expressions in TL k (Ω) , that is, using only indices x 1 , . . .
, x k .
As such, we can already ignore the term (cid:80) µ ∈ [ n ] k / ∼ k b µ,j · ψ µ ( x 1 , . . .
, x k ) since this is already in TL k (Ω) .
Furthermore, this expressions does not affect the summation depth.
Furthermore, as just mentioned, we can expand expression φ ( t ) j into linear combinations of other simpler expressions.
As such, it suffices to show that k index variables suffice for each expression of the form:
[Equation]
obtained by fixing µ and i in expression (2).
To reduce the number of variables, as a first step we eliminate any disequality using the inclusion-exclusion principle.
More precisely, we observe that ψ γ ( x , y ) can be written as:
[Equation]
[Equation]
4) for some sets I , J and K of pairs of indices in [ k ] 2 , and where ¯ I =
[ k ] 2 \ I , ¯ J =
[ k ] 2 \ J and ¯
K =
[ k ] 2 \ K .
Here we use that 1 x i ̸
= x j = 1 − 1 x i = x j , 1 y i ̸
= y j
= 1 − 1 y i =
y j and 1 x i ̸ =
y j
= 1 − 1 y i =
y j and use the inclusion-exclusion principle to obtain a polynomial in equality conditions only.
In view of expression (4), we can push the summations over y 1 , . . .
, y k in expression (3) to the subexpressions that actually use y 1 , . . .
, y k .
That is, we can rewrite expression (3) into the equivalent expression:
[Equation]
By fixing A , B and C , it now suffices to argue that
[Equation]
[Equation]
can be equivalently expressed in TL k (Ω) .
Since our aim is to reduced the number of index variables from 2 k to k , it is important to known which variables are the same.
In expression (6), some equalities that hold between the variables may not be explicitly mentioned.
For this reason, we expand I ∪ A , J ∪ B and K ∪ C with their implied equalities.
That is, 1 x i = x
j is added to I
∪ A , if for any ( v , w ) such that
[Equation]
holds.
Similar implied equalities 1 y i =
y j and 1 x i
= y j are added to J ∪ B and K ∪ C , respectively.
let us denoted by I ′ , J ′ and K ′ .
It should be clear that we can add these implied equalities to expression (6) without changing its semantics.
In other words, expression (6) can be equivalently represented by
[Equation]
There now two types of index variables among the y 1 , . . .
, y k : those that are equal to some x i , and those that are not.
Now suppose that ( j, j ′ ) ∈ J ′ , and thus y j = y j ′ , and that also ( i, j ) ∈ K ′ , and thus x i = y j .
Since we included the implied equalities, we also have ( i, j ′ )
∈ K ′ , and thus x i =
y j ′ .
There is no reason to keep ( j,
j ′ ) ∈
J ′ as it is implied by ( i, j ) and ( i, j ′ )
∈ K ′ .
We can thus safely remove all pairs ( j, j ′ ) from J ′ such that ( i, j ) ∈ K ′ (and thus also ( i, j ′ )
∈ K ′ ).
We denote by J ′′ be the reduced set of pairs of indices obtained from J ′ in this way.
We have that expression (7) can be equivalently written as
[Equation]
where we also switched the order of equalities in J ′′ and K ′ .
Our construction of J ′′ and K ′ ensures that none of the variables y j with j belonging to a pair in J ′′ is equal to some x i .
By contrast, the variable y j occurring in ( i, j ) ∈ K ′ are equal to x i .
We observe, however, that also certain equalities among the variables { x 1 , . . .
, x k } hold, as represented by the pairs in I ′ .
let
I ′ ( i ) :
= { i ′ | ( i, i ′ )
∈
I ′ } and define ˆ ı as a unique representative element in I ′
( i ) .
For example, one can take ˆ i to be smallest index in I ′
( i ) .
We use this representative index (and corresponding x -variable) to simplify K ′ .
More precisely, we replace each pair ( i, j ) ∈ K ′ with the pair (ˆ ı, j ) .
In terms of variables, we replace x i = y j with x ˆ i =
y j .
Let K ′′ be the set K ′′ modified in that way.
Expression (8) can thus be equivalently written as
[Equation]
where the free index variables of the subexpression
[Equation]
are precisely the index variables x ˆ ı for (ˆ ı, j ) ∈ K ′′ .
Recall that our aim is to reduce the variables from 2 k to k .
We are now finally ready to do this.
More specifically, we consider a bijection β : { y 1 , . . .
, y k } → { x 1 , . . .
, x k } in which ensure that for each ˆ ı there is a j such that (ˆ i, j ) ∈ K ′′ and β ( y j ) =
x ˆ ı .
Furthermore, among the summations (cid:80) y 1 · · · (cid:80) y k we can ignore those for which β ( y j ) =
x ˆ ı holds.
After all, they only contribute for a given x ˆ ı value.
Let Y be those indices in [ k ] such that β ( y j ) ̸
= x ˆ ı for some ˆ ı .
Then, we can equivalently write expression (9) as
[Equation]
) ∈
I ′ 1 x i = x j · (cid:32) (cid:88) β ( y i ) ,i ∈ Y (cid:89) (ˆ ı,j ) ∈ K ′ 1 x ˆ ı = β
( y j ) · (cid:89) ( i,
j ) ∈ J ′′ 1 β ( y i ) = β ( y j ) · β ( φ ( t − 1 ) i ( y 1 , . . . , y k )) (cid:33) , (11) where β ( φ ( t − 1 ) i ( y 1 , . . . , y k )) denotes the expression obtained by renaming of variables y 1 , . . .
, y j in φ ( t − 1) i ( y 1 , . . .
, y k ) into x -variables according to β .
This is our desired expression in TL k (Ω) .
If we analyze the summation depth of this expression, we have by induction that the summation depth of φ ( t − 1 ) i is at most ( t − 1) k .
In the above expression, we are increasing the summation depth with at most | Y | .
The largest size of Y is k , which occurs when none of the y -variables are equal to any of the x -variables.
As a consequence, we obtained an expression of summation depth at most tk , as desired.
As a consequence, when using k - IGN s ( t ) for vertex embeddings, using ( G, v ) → F ( t ) v,...,v, : one simply pads the layer expression with (cid:81) i ∈
[ k ] 1 x 1 = x i which does not affect the number of variables or summation depth.
When using k - IGN s ( t ) of graph embeddings, an additional invariant layer is added to obtain an embedding from G → R d t .
Such invariant layers have a similar (simpler) representation as given in equation 1 (Maron et al., 2019c), and allow for a similar analysis.
One can verify that expressions in TL ( ( t +1) k ) k (Ω) are needed when such an invariant layer is added to previous t layers.
Based on this, Theorem 4.2, Lemma E.1 and Theorem 1 in Maron et al. (2019b), imply that ρ 1 (cid:0) k - IGN (cid:1) =
ρ 1 (cid:0) vwl ( ∞ ) k − 1 (cid:1) and ρ 0 (cid:0) k - IGN (cid:1) = ρ 0 (cid:0) gwl ( ∞ ) k − 1 (cid:1) hold.
k -dimensional GINs.
We can recover a layer-based characterization for k - IGN s that compute vertex embeddings by considering a special subset of k - IGN s.
Indeed, the k - IGN s used in Maron et al.
(2019b) to show ρ 1 ( wl ( t ) k − 1 ) ⊆
ρ 1 ( k - IGN ( t ) ) are of a very special form.
We extract the essence of these special k - IGN s in the form of k -dimensional GIN s. That is, we define the class k - GIN s to consist of layers defined as follows.
The initial layers are just as for k - IGN s. Then, for t ≥ 1 :
[Equation]
, where F ( t − 1 ) v 1 ,v 2 ,...,v k , : ∈ R d t − 1 , mlp ( t ) 1 : R d t − 1 → R b t and mlp ( t ) 1 : R d t − 1 + kb t → R d t are MLP s.
It is now an easy exercise to show that k - GIN ( t ) can be represented in TL ( t )
k
(Ω) (remark that the summations used increase the summation depth with one only in each layer).
Combined with Theorem 4.
and by inspecting the proof of Theorem 1 in Maron et al.
(2019b), we obtain:
Proposition E.2.
For any k ≥ 2 and any t ≥ 0 : ρ 1 ( k - GIN ( t ) )
=
ρ 1 ( vwl ( t ) k − 1 ) .
We can define the invariant version of k - IGN s by adding a simple readout layer of the form [Equation] as is used in Maron et al.
(2019b).
We obtain, ρ 0 ( k - GIN )
=
ρ
( gwl ( ∞ ) k − 1 ) , by simply rephrasing the readout layer in TL k (Ω) .
F D ETAILS OF S ECTION
Let C ( G s , R ℓ ) be the class of all continuous functions from G s to R ℓ .
We always assume that G s forms a compact space.
For example, when vertices are labeled with values in { 0 , 1 } ℓ 0 , G s is a finite set which we equip with the discrete topology.
When vertices carry labels in R ℓ 0 we assume that these labels come from a compact set K ⊂ R ℓ 0 .
In this case, one can represent graphs in G s by elements in ( R ℓ 0 ) 2 and the topology used is the one induced by some metric ∥ .
∥ on the reals.
Similarly, we equip R ℓ with the topology induced by some metric ∥ .
∥ .
Consider F ⊆ C ( G s , R ℓ ) and define F as the closure of F in C ( G s , R ℓ ) under the usual topology induced by f (cid:55)→ sup G, v ∥ f ( G, v ) ∥ .
In other words, a continuous function h : G s → R ℓ is in F if there exists a sequence of functions f 1 , f 2 , . . .
∈ F such that lim
i →∞ sup G, v ∥ f
i ( G, v ) − h
( G, v ) ∥
= 0 .
The following theorem provides a characterization of the closure of a set of functions.
We state it here modified to our setting.
Theorem F.1 ((Timofte, 2005)) .
Let F ⊆ C ( G s , R ℓ ) such that there exists a set S ⊆ C ( G s , R ) satisfying S · F ⊆ F and ρ ( S ) ⊆
ρ ( F ) .
Then,
[Equation]
where F ( G, v ) :
= { h ( G, v ) | h ∈ F} ⊆ R ℓ .
We can equivalently replace ρ ( F ) by ρ ( S ) in the expression for F .
We will use this theorem to show Theorem 6.1 in the setting that F consists of functions that can be represented in TL (Ω) , and more generally, sets of functions that satisfy two conditions, stated below.
We more generally allow F to consist of functions f : G s → R ℓ f , where the ℓ f ∈ N may depend on f .
We will require F to satisfy the following two conditions: concatenation-closed: If f 1 : G s → R p and f 2 : G s → R q are in F , then g :=
( f 1 , f 2 )
: G s →
R p
+ q : (
G v ) (cid:55)→ ( f 1 ( G, v , f 2 ( G, v )) is also in F .
function-closed: For a fixed ℓ ∈ N , for any f ∈ F such that f : G s → R p , also h ◦ f : G s → R ℓ is in F for any continuous function
h ∈ C ( R p , R ℓ ) .
We denote by F ℓ be the subset of F of functions from G s to R ℓ .
Theorem 6.1.
For any ℓ , and any set F of functions, concatenation and function closed for ℓ , we have: F ℓ = { f : G s → R ℓ | ρ s ( F ) ⊆
ρ s ( f ) } .
Proof.
The proof consist of (i) verifying the existence of a set S as mentioned Theorem F.1; and of (ii) eliminating the pointwise convergence condition “ ∀ ( G, v ) ∈
G s , f ( G, v ) ∈ F ℓ ( G, v ) in the closure characterization in Theorem F.1.
For showing (ii) we argue that F ℓ ( G, v ) =
R ℓ such that the conditions f
( G, v ) ∈ F ℓ ( G, v ) is automatically satisfied for any f ∈ C ( G s , R ℓ ) .
Indeed, take an arbitrary f ∈ G k
→ R ℓ and consider the constant functions
b i : R ℓ → R ℓ : x (cid:55)→ b i with b i ∈ R ℓ the i th basis vector.
Since F is functionclosed for ℓ , so is F ℓ .
Hence, b i :=
g i ◦ f ∈ F ℓ as well.
Furthermore, if s a : R ℓ → R ℓ : x (cid:55)→ a × x , for a ∈ R , then s a ◦ f ∈ F ℓ and thus F ℓ is closed under scalar multiplication.
Finally, consider + : R 2 ℓ → R ℓ : ( x , y )
(cid:55)→ x + y .
For f and g in F ℓ , h
=
( f, g ) ∈ F since F is concatenationclosed.
As a consequence, the function + ◦ h : G s → R ℓ is in F ℓ , showing that F ℓ is also closed under addition.
All combined, this shows that F ℓ is closed under taking linear combinations and since the basis vectors of R ℓ can be attained, F ℓ ( G, v ) :
= R ℓ , as desired.
For (i), we show the existence of a set S ⊆ C ( G s , R ) such that S · F ℓ ⊆ F ℓ and ρ s ( S ) ⊆
ρ s ( F ℓ ) hold.
Similarly as in Azizian & Lelarge (2021), we define
[Equation]
We remark that for s ∈ S and f ∈ F ℓ , s · f : G s → R ℓ : ( G, v ) (cid:55)→ s ( G, v ) ⊙ f ( G, v ) , with ⊙ being pointwise multiplication, is also in F ℓ .
Indeed, s · f = ⊙ ◦ ( s, f ) with ( s, f ) the concatenation of s and f and ⊙ : R 2 ℓ → R ℓ : ( x , y )
→ x ⊙ y being pointwise multiplication.
It remains to verify ρ s ( S ) ⊆
ρ s ( F ℓ ) .
Assume that ( G, v ) and ( H, w ) are not in ρ s ( F ℓ ) .
By definition, this implies the existence of a function ˆ f ∈ F ℓ such that ˆ f ( G, v ) =
a ̸ = b = ˆ f
( H, w ) with a , b ∈ R ℓ .
We argue that ( G, v ) and ( H, w ) are also not in ρ s ( S ) either.
Indeed, Propositionin Maron et al. (2019b) implies that there exists natural numbers ααα =
( α 1 , . . .
, α ℓ ) ∈ N ℓ
such that the mapping h ααα : R ℓ → R : x → (cid:81) ℓi
=1 x α
i
i satisfies h ααα ( a )
= a ̸ = b = h ααα ( b ) , with a, b ∈ R .
Since F (and thus also F ℓ ) is function-closed, h ααα ◦ f ∈ F ℓ for any f ∈ F ℓ .
In particular, g :=
h ααα ◦ ˆ f ∈ F ℓ and concatenation-closure implies that ( g, . . .
, g ) :
G s → R ℓ is in F ℓ too.
Hence, g ∈ S , by definition.
It now suffices to observe that g ( G, v )
=
h ααα ( ˆ f
( G, v ))
= a ̸ = b = h ααα ( ˆ f
( H, w ))
=
g ( H, w ) , and thus ( G, v ) and ( H, w ) are not in ρ s ( S ) , as desired.
When we know more about ρ s ( F ℓ ) we can say a bit more.
In the following, we let alg ∈ { cr ( t ) , gcr ( t ) , vwl ( t ) k , gwl ( ∞ ) k } and only consider the setting where s is either 0 (invariant graph functions) or s = 1 (equivariant graph/vertex functions).
Corollary 6.2.
Under the assumptions of Theorem 6.1 and if ρ ( F ℓ ) = ρ ( alg ) , then F ℓ = { f : G s → R ℓ | ρ ( alg ) ⊆
ρ ( f ) } .
Proof.
This is just a mere restatement of Theorem 6.1 in which ρ s ( F ℓ ) in the condition ρ s ( F ℓ ) ⊆
ρ s ( f ) is replaced by ρ s ( alg ) , where s = 1 for alg ∈ { cr ( t ) , vwl ( t ) k } and s = 0 for alg ∈ { gcr ( t ) , gwl ( ∞ ) k } .
To relate all this to functions representable by tensor languages, we make the following observations.
First, if we consider F to be the set of all functions that can be represented in GTL ( t ) (Ω) , TL ( t +1 )(Ω) ,
TL ( t ) k
+1 (Ω) or TL (Ω) , then F will be automatically concatenation and function-closed, provided that Ω consists of all functions in (cid:83)
p C
( R p , R ℓ ) .
Hence, Theorem 6.1 applies.
Furthermore, our results from Section 4 tell us that for all t ≥ 0 , and k ≥ 1 , ρ 1 (cid:0) cr ( t ) (cid:1) =
ρ 1 (cid:0) GTL ( t ) (Ω) (cid:1) , ρ 0 (cid:0) gcr ( t ) (cid:1)
=
ρ 0 (cid:0) TL ( t +1 )
(Ω) (cid:1) =
ρ 0 (cid:0) gwl ( t ) 1 (cid:1) , ρ 1 (cid:0) ewl ( t ) k (cid:1) =
ρ 1 (cid:0) TL ( t )
k
+1 (Ω) (cid:1) , and ρ 0 (cid:0) TL
k +1 (Ω) (cid:1) =
ρ 0 (cid:0) gwl ( ∞ ) k (cid:1) .
As a consequence, Corollary 6.2 applies as well.
We thus easily obtain the following characterizations:
Proposition F.2.
For any t ≥ 0 and k ≥ 1 :
• If F consists of all functions representable in GTL ( t ) (Ω) , then F ℓ = { f : G 1 → R ℓ | ρ 1 (cid:0) cr ( t ) (cid:1) ⊆
ρ 1 ( f ) } ; • If F consists of all functions representable in TL ( t )
k +1 (Ω) , then F ℓ = { f : G 1 → R ℓ | ρ 1 (cid:0) vwl ( t ) k (cid:1) ⊆
ρ 1 ( f ) } ; • If F consists of all functions representable in TL ( t +1 )(Ω) , then F ℓ = { f : G 0 → R ℓ | ρ 0 (cid:0) gwl ( t ) 1 (cid:1) ⊆
ρ 0 ( f ) } ; and finally, • If F consists of all functions representable in TL k +
(Ω) , then
F ℓ = { f : G 0 → R ℓ | ρ 0 (cid:0) gwl ( ∞ ) k
(cid:1) ⊆
ρ 0 ( f ) } , provided that Ω consists of all functions in (cid:83) p C
( R p , R ℓ ) .
In fact, Lemma 32 in Azizian & Lelarge (2021) implies that we can equivalently populate Ω with all MLP s instead of all continuous functions.
We can thus use MLP s and continuous functions interchangeably when considering the closure of functions.
At this point, we want to make a comparison with the results and techniques in Azizian & Lelarge (2021).
Our proof strategy is very similar and is also based on Theorem F.1.
The key distinguishing feature is that we consider functions f : G s → R ℓ f instead of functions from graphs alone.
This has as great advantage that no separate proofs are needed to deal with invariant or equivariant functions.
Equivariance incurs quite some complexity in the setting considered in Azizian & Lelarge (2021).
A second major difference is that, by considering functions representable in tensor languages, and based on our results from Section 4, we obtain a more fine-grained characterization.
Indeed, we obtain characterizations in terms of the number of rounds used in CR and k - WL .
In Azizian &
Lelarge (2021), t is always set to ∞ , that is, an unbounded number of rounds is considered.
Furthermore, when it concerns functions f : G 1 → R ℓ f , we recall that CR is different from 1 - WL .
Only 1 - WL is considered in Azizian & Lelarge (2021).
Finally, another difference is that we define the equivariant version vwl k in a different way than is done in Azizian & Lelarge (2021), because in this way, a tighter connection to logics and tensor languages can be made.
In fact, if we were to use the equivariant version of k - WL from Azizian & Lelarge (2021), then we necessarily have to consider an unbounded number of rounds (similarly as in our gwl k case).
We conclude this section by providing a little more details about the consequences of the above results for GNN s. As we already mentioned in Section 6.2, many common GNN architectures are concatenation and function-closed (using MLP s instead of continuous functions).
This holds, for example, for the classes
GIN ( t ) ℓ , eGIN ( t ) ℓ , k - FGNN ( t ) ℓ and k - GIN ( t ) ℓ and k - IGN ( t ) , as described in Section 5 and further detailed in Section E and D.
Here, the subscript ℓ refers to the dimension of the embedding space.
We now consider a function f
that is not more separating than cr ( t ) (respectively, gcr ( t ) , vwl ( t ) k or gwl ( ∞ ) k , for some k ≥ 1 ), and want to know whether f can be approximated by a class of GNN s.
Proposition F.2 tells that such f can be approximated by a class of GNN s as long as these are at least as separating as GTL ( t ) (respectively, TL ( t +1 ), TL ( t )
k +1 or TL ( ∞ ) k +1 ).
This, in turn, amounts showing that the GNN s can be represented in the corresponding tensor language fragment, and that they can match the corresponding labeling algorithm in separation power.
We illustrate this for the GNN architectures mentioned above.
• In Section 5 we showed that GIN ( t ) ℓ can be represented in GTL ( t ) (Ω) .
Theorem 4.3 then implies that ρ 1 ( cr ( t ) ) ⊆
ρ 1 ( GIN ( t ) ℓ ) .
Furthermore, Xu et al.
(2019) showed that ρ 1 ( GIN ( t ) ℓ ) ⊆
ρ 1 ( cr ( t ) ) .
As a consequence, ρ 1 ( GIN ( t ) ℓ ) =
ρ 1 ( cr ( t ) ) .
We note that the lower bound for GIN s only holds when graphs carry discrete labels.
The same restriction is imposed in Azizian & Lelarge (2021).
• In Section 5 we showed that eGIN ( t ) ℓ can be represented in TL ( t ) 2 (Ω) .
Theorem 4.2 then implies that ρ 1 ( vwl ( t ) 1 ) ⊆
ρ 1 ( eGIN ( t ) ℓ ) .
Furthermore, Barcel ´ o et al.
(2020) showed that ρ 1 ( eGIN ( t ) ℓ ) ⊆
ρ 1 ( vwl ( t ) 1 ) .
As a consequence, ρ 1 ( eGIN ( t ) ℓ ) =
ρ 1 ( vwl ( t ) 1 ) .
Again, the lower bound is only valid when graphs carry discrete labels.
• In Section 5 we mentioned (see details in Section D) that k - FGNN ( t ) ℓ can be represented in TL ( t )
k +1 (Ω) .
Theorem 4.2 then implies that ρ 1 ( vwl ( t ) k ) ⊆
ρ 1 ( k - FGNN ( t ) ℓ ) .
Furthermore, Maron et al.
(2019b) showed that ρ 1 ( k - FGNN ( t ) ℓ ) ⊆
ρ 1 ( vwl ( t ) k ) .
As a consequence, ρ 1 ( k - FGNN ( t ) ℓ ) ⊆
ρ 1 ( vwl ( t ) k ) .
Similarly, ρ 1 (( k + 1) - GIN ( t ) ℓ ) =
ρ 1 ( vwl ( t ) k ) for the special class of ( k + 1) - IGN s described in Section
E. No restrictions are in place for the lower bounds and hence real-valued vertex-labelled graphs can be considered.
• When GIN ( t ) ℓ or eGIN ( t ) ℓ are extended with a readout layer, we showed in Section 5 that these can be represented in TL ( t +1 )
(Ω) .
Theorem 4.4 and the results by Xu et al.
and Barcel´o et al.
(2020) then imply that ρ 0 ( vwl ( t ) 1 ) and ρ 0 ( gcr ( t ) ) coincide with the separation power of these architectures with a readout layer.
Here again, discrete labels need to be considered.
• Similarly, when k - FGNN or ( k +1) - IGN s are used for graph embeddings, we can represent these in TL k +
(Ω) resulting again that their separation power coincides with that of gwl ( ∞ ) k .
No restrictions are again in place on the vertex labels.
So for all these architectures, Corollary 6.2 applies and we can characterize the closures of these architectures in terms of functions that not more separating than their corresponding versions of cr or k - WL , as described in the main paper.
In summary,
Proposition F.3.
For any t ≥ 0 :
[Equation]
and when extended with a readout layer:
[Equation]
Furthermore, for any k ≥k - FGNN (
[Equation]
) ℓ = k - GIN
[Equation]
and when converted into graph embeddings:
[Equation]
where the closures of the tensor languages are interpreted as the closure of the graph or graph/vertex functions that they can represent.
For results involving GIN s or eGIN s, the graphs considered should have discretely labeled vertices.
As a side note, we remark that in order to simulate CR on graphs with real-valued labels, one can use a GNN architecture of the form F ( t ) v : = (cid:0) F ( t − 1 ) v : , (cid:80) u ∈ N G ( v ) mlp ( F ( t − 1 ) u : ) (cid:1) , which translates in
GTL ( t ) (Ω) as expressions of the form
[Equation]
The upper bound in terms of CR follows from our main results.
To show that CR can be simulated, it suffices to observe that one can approximate the function used in Proposition 1 in Maron et al.
(2019b) to injectively encode multisets of real vectors by means of MLP s.
As such, a continuous version of the first bullet in the previous proposition can be obtained.
G D ETAILS ON T REEWIDTH AND P ROPOSITION 4.
As an extension of our main results in Section 4, we enrich the class of tensor language expressions for which connections to k - WL exist.
More precisely, instead of requiring expressions to belong to
TL k +1 (Ω) , that is to only use k + 1 index variables, we investigate when expressions in TL (Ω) are semantically equivalent to an expression using k + 1 variables.
Proposition 4.5 identifies a large class of such expressions, those of treewidth k .
As a consequence, even when representing GNN architectures may require more than k + 1 index variables, sometimes this number can be reduced.
As a consequence of our results, this implies that their separation power is in fact upper bounded by ℓ - WL for a smaller ℓ < k .
Stated otherwise, to boost the separation power of GNN s, the treewidth of the expressions representing the layers of the GNN s must have large treewidth.
We next introduce some concepts related to treewidth.
We here closely follow the exposition given in Abo Khamis et al.
(2016) for introducing treewidth by means variable elimination sequences of hypergraphs.
In this section, we restrict ourselves to summation aggregation.
G. E LIMINATION SEQUENCES
We first define elimination sequences for hypergraphs.
Later on, we show how to associate such hypergraphs to expressions in tensor languages, allowing us to define elimination sequences for tensor language expressions.
With a multi-hypergraph H =
( V , E ) we simply mean a multiset E of subsets of vertices V .
An elimination hypergraph sequences is a vertex ordering σ = v 1 , . .
. , v n of the vertices of H .
With such a sequence σ , we can associate for j = n, n − 1 , n − 2 , . .
, 1 a sequence of n multi-hypergraphs
H σn , H σn − 1 , . . .
, H σ 1 as follows.
We define
[Equation]
F. and for j = n − 1 , n − 2 , . .
[Equation]
[Equation]
, v [Equation]
[Equation]
The induced width on H by σ is defined as max i ∈ [ n ] | U i | − 1 .
We further consider the setting in which H has some distinguished vertices.
As we will see shortly, these distinguished vertices correspond to the free index variables of tensor language expressions.
Without loss of generality, we assume that the distinguished vertices are v 1 , v 2 , . .
, v f .
When such distinguished vertices are present, an elimination sequence is just as before, except that the distinguished vertices come first in the sequence.
If v 1 , . .
. , v f are the distinguished vertices, then we define the induced width of the sequence as f + max f +1 ≤ i ≤ n | U i \ { v 1 , . .
. , v f }| − 1 .
In other words, we count the number of distinguished vertices, and then augment it with the induced width of the sequence, starting from v f + to to v n , hereby ignoring the distinguished variables in the U i ’s. One could, more generally, also try to reduce the number of free index variables but we assume that this number is fixed, similarly as how GNN s operate.
G. C ONJUNCTIVE TL EXPRESSIONS AND TREEWIDTH
We start by considering a special form of TL expressions, which we refer to as conjunctive TL expressions, in analogy to conjunctive queries in database research and logic.
A conjunctive TL expression is of the form
[Equation]
where x denote the free index variables, y contains all index variables under the scope of a summation, and finally, ψ ( x , y ) is a product of base predicates in TL .
That is, ψ ( x , y ) is a product of
E ( z i , z j ) and P ℓ ( z i ) with z i , z j variables in x or y .
With such a conjunctive TL expression, one can associate a multi-hypergraph in a canonical way (Abo Khamis et al., 2016).
More precisely, given a conjunctive TL expression φ ( x ) we define H φ as:
• V φ consist of all index variables in x and y ;
• E φ : for each atomic base predicate τ in ψ we have an edge F τ containing the indices occurring in the predicate; and • the vertices corresponding to the free index variables x form the distinguishing set of vertices.
We now define an elimination sequence for φ as an elimination sequence for H φ taking the distinguished vertices into account.
The following observation ties elimination sequences of φ to the number of variables needed to express φ .
Proposition G.1.
Let φ ( x ) be a conjunctive TL expression for which an elimination sequence of induced with k − 1 exists.
Then φ ( x ) is equivalent to an expression ˜ φ ( x ) in TL k .
Proof.
We show this by induction on the number of vertices in H φ which are not distinguished.
For the base case, all vertices are distinguished and hence φ ( x ) does not contain any summation and is an expression in TL k itself.
Suppose that in H φ there are p undistinguished vertices.
That is,
[Equation]
By assumption, we have an elimination sequence of the undistinguished vertices.
Assume that y p is first in this ordering.
Let us write
[Equation]
where ψ 1 is the product of predicates corresponding to the edges F ∈ E φ \ ∂ ( y p ) , that is, those not containing y p , and ψ 2 is the product of all predicates corresponding to the edges F ∈ ∂ ( y p ) , that is, those containing the predicate y p .
Note that, because of the induced width of k − 1 , (cid:80)
y p ψ 2 ( x , y ) contains all indices in U p which is of size ≤ k .
We now replace the previous expression with another expression
[Equation]
Where R p is regarded as an | U p | − 1 -ary predicate over the indices in U p \
y p .
It is now easily verified that H φ ′ is the hypergraph H p − 1 corresponding to the variable ordering σ .
We note that this is a hypergraph over p − 1 undistinguished vertices.
We can apply the induction hypothesis and replace φ ′ ( x ) with its equivalent expression ˜
φ ′ ( x ) in TL k .
To obtain the expression ˜ φ ( x ) of φ ( x ) , it now remains to replace the new predicate R p with its defining expression.
We note again that R p contains at most k − 1 indices, so it will occur in ˜
φ
( x ) in the form R p ( x , z ) where | z | ≤ k − 1 .
In other words, one of the variables in z is not used, say z s , and we can simply replace R p ( x , z ) by (cid:80)
z s ψ x ( x , z , z s )
As a consequence, one way of showing that a conjunctive expression φ ( x ) in TL is equivalently expressible in TL k , is to find an elimination sequence of induced width k − 1 .
This in turn is equivalent to H φ having a treewidth of k − 1 , as is shown, e.g., in Abo Khamis et al.
As usual, we define the treewidth of a conjunctive expression φ ( x ) in TL as the treewidth of its associated hypergraph H φ .
We recall the definition of treewidth (modified to our setting): A tree decomposition T
=
( V T , E T , ξ T ) of H φ with ξ T : V T → 2 V is such that
• For any F ∈ E , there is a t ∈ V T such that F ⊆ ξ T ( t ) ; and • For any v ∈ V corresponding to a non-distinguished index variable, the set { t | t ∈ V T , v ∈ ξ ( t ) } is not empty and forms a connected sub-tree of T .
The width of a tree decomposition T is given by max t ∈ V T | ξ T ( t )
| − 1 .
Now the treewidth of H φ , tw ( H ) is the minimum width of any of its tree decompositions.
We denote by tw ( φ ) the treewidth of
H φ .
Again, similar modifications are used when distinguished vertices are in place.
Referring again to Abo Khamis et al.
(2016), tw ( φ ) =
k − 1 is equivalent to having a variable elimination sequence for φ of an induced width of k − 1 .
Hence, combining this observation with Proposition G.1 results in:
Corollary G.2.
Let φ ( x ) be a conjunctive TL expression of treewidth k − 1 .
Then φ ( x ) is equivalent to an expression ˜ φ ( x ) in TL k .
That is, we have established Proposition 4.5 for conjunctive TL expressions.
We next lift this to arbitrary TL (Ω) expressions.
G. A RBITRARY TL (Ω) EXPRESSIONS
First, we observe that any expression in TL can be written as a linear combination of conjunctive expressions.
This readily follows from the linearity of the operations in TL and that equality and inequality predicates can be eliminated.
More specifically, we may assume that φ ( x ) in TL is of the form
[Equation]
with A finite set of indices and a α ∈ R , and ψ α ( x , y ) conjunctive TL expressions.
We now define
[Equation]
for expressions in TL .
To deal with expressions in TL (Ω) that may contain function application, we define tw ( φ ) as the maximum treewidth of the expressions: (i) φ nofun ( x ) ∈ TL obtained by replacing each top-level function application f ( φ 1 , . .
. , φ p ) by a new predicate R f with free indices free ( φ 1 ) ∪ · · · ∪ free ( φ p ) ; and (ii) all expressions φ 1 , . .
φ p occurring in a top-level function application f ( φ 1 , . .
. , φ p ) in φ .
We note that these expression either have no function applications (as in (i)) or have function applications of lower nesting depth (in φ , as in ( ii ) ).
In other words, applying this definition recursively, we end up with expressions with no function applications, for which treewidth was already defined.
With this notion of treewidth at hand, Proposition 4.5 now readily follows.
H H IGHER - ORDER MPNN S
We conclude the supplementary material by elaborating on k - MPNN s and by relating them to classical MPNN s (Gilmer et al., 2017).
As underlying tensor language we use TL
k +1 (Ω , Θ) which includes arbitrary functions ( Ω ) and aggregation functions ( Θ ), as defined in Section C.5.
We recall from Section 3 that k - MPNN s refer to the class of embeddings f : G s → R ℓ for some ℓ ∈ N that can be represented in TL
k +1 (Ω , Θ) .
When considering an embedding f : G s → R ℓ , the notion of being represented is defined in terms of the existence of ℓ expressions in TL
k +1 (Ω , Θ) , which together provide each of the ℓ components of the embedding in R ℓ .
We remark, however, that we can alternatively include concatenation in tensor language.
As such, we can concatenate ℓ separate expressions into a single expression.
As a positive side effect, for f : G s → R ℓ to be represented in tensor language, we can then simply define it by requiring the existence of a single expression, rather than ℓ separate ones.
This results in a slightly more succinct way of reasoning about k - MPNN s.
In order to reason about k - MPNN s as a class of embeddings, we can obtain an equivalent definition for the class of k - MPNN s by inductively stating how new embeddings are computed out of old embeddings.
Let X
= { x 1 , . . .
, x k +1 } be a set of k + 1 distinct variables.
In the following, v denotes a tuple of vertices that have at least as many components as the highest index of variables used in expressions.
Intuitively, variable x j refers to the j th component in v .
We also denote the image of a graph G and tuple v by an expression φ , i.e., the semantics of φ given G and v , as φ ( G, v ) rather than by [[ φ, v ]] G .
We further simply refer to embeddings rather than expressions.
We first define “atomic” k - MPNN embeddings which extract basic information from the graph G and the given tuple v of vertices.
• Label embeddings of the form φ ( x i ) :
=
P s ( x i ) , with x i ∈ X , and defined by φ ( G, v ) :
= ( col G ( v i ))
s , are k - MPNN s; • Edge embeddings of the form φ ( x i , x j ) :
=
E ( x i , x j ) , with x i , x j ∈ X , and defined by φ ( G, v ) :
= (cid:26) 1 if v i v j ∈ E G 0 otherwise , are k - MPNN s; and • (Dis- )equality embeddings of the form φ ( x i , x j ) :
= 1 x i op x j , with x i , x j ∈ X , and defined by [Equation] 0 otherwise , are k - MPNN s.
We next inductively define new k - MPNN s from “old” k - MPNN s. That is, given k - MPNN s φ 1 ( x 1 ) , . .
. , φ ℓ ( x ℓ ) , the following are also k - MPNN s:
• Function applications of the form φ ( x ) :
= f ( φ 1 ( x 1 ) , . .
. , φ ℓ ( x ℓ ) are k - MPNN s, where x = x 1 ∪ · · · ∪ x ℓ , and defined by
[Equation]
Here, if φ i ( G, v | x i
) ∈
R d i , then f : R d 1 × · · · × R d ℓ → R d for some d ∈ N .
That is, φ generates an embedding in R d .
We remark that our function applications include concatenation.
• Unconditional aggregations of the form φ ( x ) :
=
agg F x j ( φ 1 ( x , x j )) are k - MPNN s, where x j ∈ X and x j ̸∈ x , and defined by
[Equation]
Here, if φ 1 generates an embedding in R d 1 , then F is an aggregation function assigning to multisets of vectors in R d 1 a vector in R d , for some d ∈ N .
So, φ generates an embedding in R d .
• Conditional aggregations of the form φ ( x i ) :
=
agg F x j ( φ 1 ( x i , x j ) | E ( x i , x j )) are kMPNN s, with x i , x j ∈ X , and defined by
[Equation]
As before, if φ 1 generates an embedding in R d 1 , then F is an aggregation function assigning to multisets of vectors in R d 1 a vector in R d , for some d ∈ N .
So again, φ generates an embedding in R d .
As defined in the main paper, we also consider the subclass k - MPNN s ( t ) by only considering kMPNN s defined in terms of expressions of aggregation depth at most t .
Our main results, phrased in terms of k - MPNN s are:
[Equation]
and [Equation] Hence, if the embeddings computed by GNN s are k - MPNN s, one obtains an upper bound on the separation power in terms of k - WL .
The classical MPNN s (Gilmer et al., 2017) are subclass of 1 - MPNN s in which no unconditional aggregation can be used and furthermore, function applications require input embeddings with the same single variable ( x 1 or x 2 ), and only 1 x i
= x
i and 1 x i ̸
= x i are allowed.
In other words, they correspond to guarded tensor language expressions (Section 4.2).
We denote this class of 1 - MPNN s by MPNN s and by MPNN s ( t ) when restrictions on aggregation depth are in place.
And indeed, the classical way of describing MPNN s as
[Equation]
correspond to 1 - MPNN s that satisfy the above mentioned restrictions.
Without readouts, MPNN s compute vertex embeddings and hence, our results imply
[Equation]
s ( t ) ) .
Furthermore, MPNN s with a readout function fall into the category of 1 - MPNN s:
[Equation]
where unconditional aggregation is used.
Hence,
[Equation]
We thus see that k - MPNN s gracefully extend MPNN s and can be used for obtaining upper bounds on the seperation power of classes of GNN s.