C OUNTING S UBSTRUCTURES WITH H IGHER -O RDER G RAPH N EURAL N ETWORKS : P OSSIBILITY AND I MPOSSIBILITY R ESULTS
Anonymous authors
Paper under double-blind review
A BSTRACT
While message passing Graph Neural Networks (GNNs) have become increasingly popular architectures for learning with graphs, recent works have revealed important shortcomings in their expressive power.
In response, several higherorder GNNs have been proposed that substantially increase the expressive power, albeit at a large computational cost.
Motivated by this gap, we explore alternative strategies and lower bounds.
In particular, we analyze a new recursive pooling technique of local neighborhoods that allows different tradeoffs of computational cost and expressive power.
First, we prove that this model can count subgraphs of size k , and thereby overcomes a known limitation of low-order GNNs.
Second, we show how recursive pooling can exploit sparsity to reduce the computational complexity compared to the existing higher-order GNNs.
More generally, we provide a (near) matching information-theoretic lower bound for counting subgraphs with graph representations that pool over representations of derived (sub-)graphs.
We also discuss lower bounds on time complexity.
I NTRODUCTION
Graph Neural Networks (GNNs) are powerful tools for graph representation learning (Scarselli et al., 2008; Kipf & Welling, 2017; Hamilton et al., 2017), and have been successfully applied to molecule property prediction, simulating physics, social network analysis, knowledge graphs, trafﬁc prediction and many other domains (Duvenaud et al., 2015; Defferrard et al., 2016;
Battaglia et al., 2016;
Jin et al., 2018).
The perhaps most widely used class of GNNs, Message Passing Graph Neural Networks (MPNNs)
(Gilmer et al., 2017; Kipf & Welling, 2017; Hamilton et al., 2017; Xu et al., 2019;
Scarselli et al., 2008), follow an iterative message passing scheme to compute a graph representation.
Despite the empirical success of MPNNs, their expressive power has been shown to be limited.
For example, their discriminative power, at best, corresponds to the one-dimensional Weisfeiler-Leman (1-WL) graph isomorphism test (Xu et al., 2019; Morris et al., 2019), so they cannot distinguish regular graphs, for instance.
Likewise, they cannot count any induced subgraph with at least three vertices (Chen et al., 2020), or learn structural graph parameters such as clique information, diameter, conjoint or shortest cycle (Garg et al., 2020).
Yet, in applications like computational chemistry, materials design or pharmacy (Elton et al., 2019; Sun et al., 2020; Jin et al., 2018), the functions we aim to learn often depend on the presence or count of speciﬁc substructures, e.g., functional groups.
The limitations of MPNNs result from their inability to distinguish individual nodes.
To resolve this issue, two routes have been studied: (1) using unique node identiﬁers (Loukas, 2019;
Sato et al., 2019; Abboud et al., 2021), and (2) higher-order GNNs that act on k -tuples of nodes.
Node IDs, if available, enable Turing completeness for sufﬁciently large MPNNs (Loukas, 2019).
Higher-order networks use an encoding of k -tuples and then apply message passing (Morris et al., 2019), or equivariant tensor operations (Maron et al., 2018).
The expressive power of MPNNs is often measured in terms of a hierarchy of graph isomorphism tests, speciﬁcally the k -Weisfeiler-Leman ( k -WL) hierarchy.
The k -order models in (Maron et al., 2018) and (Maron et al., 2019a) are equivalent to the k -WL and ( k +1 )
-WL “tests”, respectively, and are universal for the corresponding function classes (Azizian & Lelarge, 2020; Maron et al., 2019b;
Keriven & Peyr´e, 2019).
Yet, these models are computationally expensive, operating on Θ( n k ) tuples and according to current upper bounds requiring up to O ( n k ) iterations (Kiefer & McKay, 2020).
The necessary tradeoffs between expressive power and computational complexity are still an open question.
However, for speciﬁc classes of tasks this full universality may not be needed.
Here, we study such an example of practical interest: counting substructures, as proposed in (Chen et al., 2020).
In particular, we study if it is possible to count given substructures with a GNN whose complexity is between that of MPNNs and existing higher-order GNNs.
To this end, we study a generic scheme followed by many GNN architectures, including MPNNs and higher-order GNNs (Morris et al., 2019; Chen et al., 2020): select a collection of subgraphs of the input graph, encode these, and apply an aggregation function on this collection.
First, we study the power of pooling by itself , as a multi-set function over node features.
We prove that k recursive applications on each node’s neighborhood allow to count any substructure of size k .
This is in contrast to iterative MPNNs.
We call this technique Recursive neighborhood pooling (RNP) .
While subgraph pooling relates to the graph reconstruction conjecture, our strategy has important differences.
In particular, we show how the aggregation “augments” local encodings, if they play together and the subgraphs are selected appropriately, and this reasoning may be of interest for the design of other, even partially, expressive architectures.
Moreover, our results show that the complexity is adjustable to the counting task of interest and the sparsity of the graph.
The strategy of pooling subgraph encodings has previously been used for counting in Local Relational Pooling (LRP)
(Chen et al., 2020).
LRP relies on an isomorphic encoding of subgraphs, which is expensive – e.g., the relational pooling it uses requires O ( k !)
time for a subgraph of size k .
Other higher-order GNNs would be expensive, too, as high orders are needed for complete isomorphism power.
A major difference to our RNP is that our recursion uses subgraphs of varying sizes and structures, many of them much smaller – adapted to the graph structure and speciﬁc counting task.
Furthermore, we study lower bounds on GNNs that count motifs.
We show an information-theoretic lower bound on the number of subgraphs to encode, as a function of an encoding complexity.
We also transfer computational lower bounds that apply to any counting GNN.
The lower bounds show that the recursive pooling is close to tight.
In short, in this paper, we make the following contributions:
• We study the power of pooling encodings of subgraphs, and show that pooling, as an injective multi-set function, is sufﬁcient by itself for counting when applied recursively on appropriate subgraphs, remarkably without relying on other encoding techniques or node IDs.
This is different from any other strategy we are aware of in the literature.
• We analyze the complexity of recursive pooling, as a function of the task and input graph.
• We provide complexity lower bounds for pooling and general GNN architectures that count motifs.
For instance, we show a lower bound on the number of subgraphs that need to be encoded.
B ACKGROUND
Message Passing Graph Neural Networks.
Let G =
( V , E , X ) be an attributed graph with |V| = n nodes.
Here, X v ∈ X denotes the initial attribute of v ∈ V , where X ⊆
N is a (countable) domain.
A typical Message Passing Graph Neural Network (MPNN) ﬁrst computes a representation of each node, and then aggregates the node representations via a readout function into a representation of the entire graph G .
The representation h ( i ) v of each node v ∈ V is computed iteratively by aggregating the representations
h
( i − 1) u of the neighboring vertices u :
[Equation]
for any v ∈ V , for k iterations, and with h (0) v = X v .
The A GGREGATE /C
OMBINE functions are parametrized, and {{ . }} denotes a multi-set, i.e., a set with (possibly) repeating elements.
A graphlevel representation can be computed as h G = R EADOUT (cid:0) {{ h ( k ) v : v ∈ V}} (cid:1) , where R EADOUT is a learnable aggregation function.
For representational power, it is important that the learnable functions are injective (Xu et al., 2019).
Higher-Order GNNs.
To increase the representational power of GNNs, several higher-order
GNNs have been proposed.
In k -GNN , message passing is applied to k − tuples of nodes, inspired by k -WL (Morris et al., 2019).
At initialization, each k -tuple is labeled such that two k -tuples are labeled differently if their induced subgraphs have different isomorphism types (Maron et al., 2019a;
Cai et al., 1992).
As a result, k -GNNs can count (induced) substructures with at most k vertices even at initialization.
Another class of higher-order networks applies (linear) equivariant operations, interleaved with coordinate-wise nonlinearities, to order- k tensors consisting of the adjacency matrix and input node attributes (Maron et al., 2018; 2019a;b).
These GNNs are at least as powerful as k − GNNs, and hence they too can count substructures with at most k vertices.
All these methods need Ω( n k ) operations.
Local Relational Pooling (LRP) (Chen et al., 2020) was designed for counting and applies relational pooling (Murphy et al., 2019b;a) on local neighborhoods, i.e., one pools over evaluations of a permutation-sensitive function applied to all k !
permutations of the nodes in a k -size neighborhood of each node.
O THER R ELATED W ORKS
Expressive power.
Several other works have studied the expressive power of GNNs as function approximators (Azizian & Lelarge, 2020).
Scarselli et al.
(2009) extend universal approximation from feedforward networks to MPNNs, using the notion of unfolding equivalence , i.e., functions on computation trees.
Indeed, graph distinction and function approximation are closely related (Chen et al., 2019; Azizian & Lelarge, 2020; Keriven & Peyr ´ e, 2019).
Maron et al.
(2019b)
and Keriven &
Peyr ´ e (2019) show that higher-order, tensor-based GNNs provably achieve universal approximation of permutation-invariant functions on graphs, and Loukas (2019) analyzes expressive power under depth and width restrictions.
Studying GNNs from the perspective of local algorithms, Sato et al.
(2019) show that GNNs can approximate solutions to certain combinatorial optimization problems.
For counting substricutres, Arvind et al.
(2020); F ¨ urer (2017) show that 2-WL, which is equivalent to MPNNs, can count not necessarily induced cycles and paths of up to 7 vertices with O ( n 2 ) complexity,
Subgraphs and GNNs.
Having infromation about subgraphs can be quite helpful in various graph representation algorithms (Liu et al., 2019;
Monti et al., 2018;
Liu et al., 2020; Yu et al., 2020; Meng et al., 2018; Cotta et al., 2020;
Alsentzer et al., 2020; Huang & Zitnik, 2020).
For example, for graph comparison (i.e., testing whether a given (possibly large) subgraph exists in the given model), Ying et al.
(2020) compare the outputs of GNNs for small subgraphs of the two graphs.
To improve the expressive power of GNNs, Bouritsas et al.
(2020) use features that are counts of speciﬁc subgraphs of interest.
Another example is (Vignac et al., 2020), where an MPNN is strengthened by learning local context matrices around vertices.
Recent works have also developed GNNs that pass messages on ego-nets (You et al., 2021; Sandfelder et al., 2021).
With motivation from the reconstruction conjecture, Cotta et al.
(2021) process node-deleted subgraphs with individual MPNNs, and then pool them with a DeepSets model to get a representation of the original graph.
R ECURSIVE N EIGHBORHOOD P OOLING
Let G =
( V , E , X ) be an attributed input graph with |V| = n nodes, and let h (0) v
=
X v be the initial representation of each node v .
In this work, we study architectures that ﬁrst ﬁnd representations of a collection of m subgraphs G i and then aggregate (pool) over these representations with a multi-set function, i.e., A GGREGATE ( {{ ψ ( G i ) : i ∈ [ m ] }} ) , [ m ]
[Equation]
It is clear that if the ψ can count a subgraph structure H , then the entire model can count H .
In particular, we aim to apply this strategy to obtain the representations ψ , too.
To appreciate the challenges in doing so, recall two such examples.
First, MPNNs follow this pooling strategy, by iteratively aggregating over node neighborhoods, and then aggregating all node representations into a graph representation.
However, it is known that MPNNs can count at most star structures or edges (Arvind et al., 2020).
This is because they represent a local computation tree, which loses structural information about node identities and connectivity.
Second, this strategy is at the heart of the Graph Reconstruction Conjecture (Kelly, 1957), which conjectures that a graph G can be reconstructed from its subgraphs {{ G v = G \ { v } : v ∈ V ( G ) }} (Appendix E).
This, however, is unknown for general graphs.
Although the G v retain some structure, we lose information about their structural relationship .
In summary, encoding structural information is the key question.
[Equation]
[Figure] [Equation] [Figure] Hence, to represent the counting function ψ over potentially large neighborhoods, an MPNN does not sufﬁce.
But, aggregation over input node attributes, along with edge information, can count edge types, i e., tiny subgraphs.
Hence, we recursively apply aggregation on smaller sub-neighborhoods while remembering structural information, with node-wise aggregation as the base case.
For intuition, suppose we aim to count the occurrence of subgraph H in the r 1 -neighborhood N r 1 ( v ) of a node v .
To do so, we may count H v
=
H \ { v } in the smaller graph N r 1 ( v ) \ { v } .
But, to combine these counts with the presence of v to “complete” H , we must know how the H v are connected to v in the screened graph.
Hence, to retain structure information, we mark the neighbors of v .
We then recursively call neighborhood pooling to process smaller neighborhoods N r 2 ( u ) within
N r 1 ( v ) \ { v } .
This could, e.g., learn to count marked versions of H v .
The radii r of neighborhoods may differ in recursive calls.
In Section 5, we relate their size to H .
Recursive neighborhood pooling RNP-GNN ( G, { h in u } u ∈V ( G ) , ( r 1 , . .
, r τ )) takes an attributed graph along with a sequence of neighborhood radii for different recursions, and returns a set of node encodings { h v } v ∈V ( G ) .
For any v ∈ G , RNP-GNN ﬁrst constructs v ’s neighborhood, removes v and marks its neighbors 1 :
[Equation]
Here, we extracted r 1 − neighborhood of v , removed the central node v , and then added the new structural information to 1 − neighbors of v . Then we aggregate over subgraph representations, i.e., the representations of G v , v ∈ [ n ] when they considered as a set graphs.
If τ = 1 (base case), we use the input features:
[Equation]
In other words, in this case we only combine hidden representations in each G v without any iteration/recursion.
If τ > 1 , we recursively represent neighborhoods of nodes in G v :
[Equation]
[Equation]
Here, we ﬁrst apply an RNP-GNN on each G v to update the representations of nodes for each node in G v .
Then, we combine those representations to ﬁnd the representation of v .
The fact that G v has less nodes than G allows to deﬁne the algorithm in an inductive way; this ensures the algorithm does not have any logical loop.
For aggregation, we can use, e.g., the injective multi-set function from (Xu et al., 2019): [Equation]
[Equation]
where we use MLP modules with ReLU activation, and (cid:15) is an arbitrary irrational constant (Xu et al., 2019).
The ﬁnal readout aggregates over the ﬁnal node representations of the entire graph.
Figureillustrates an RNP-GNN with recursion parameters (2 , 2 , 1) , and Appendix F provides pseudocode.
One can optionally do message passing iterations to the representations h v , v ∈ [ n ] to make the model more expressive; though our results hold even without those iterations.
While MPNNs also encode a representation of a local neighborhood, the recursive representations differ as they take into account intersections of neighborhoods.
As a result, as we will see in Section 5, they retain more structural information and are more expressive than MPNNs; one can take r 1 = 1 to simulate one iteration of MPNNs (and then iterate).
Models like k -GNN and LRP also compute encodings of subgraphs, and then update the resulting representations via message passing.
We can do the same with the neighborhood representations computed by RNP-GNNs to encode more global information, although our representation results in Section 5 hold even without that.
E XPRESSIVE P OWER OF R ECURSIVE P OOLING
In this section, we analyze the expressive power of RNP-GNNs.
We notice that for full expressiveness, we indeed need the identiﬁers in Eq.
(3); their addition is an important insight.
Still,
RNP-GNN does maintain some expressive power without the augmented identiﬁers.
For instance, consider the graphs G 1 = two triangles and G 2 =
the 6-cycle, which are 1-WL equivalent and thus cannot be distinguished by MPNNs. An RNP-GNN with r 1 = 1 can distinguish these without the augmented identiﬁers. This is because on the ﬁrst recursion level, the 1-neighborhoods of G 1 are two nodes with an edge between them, and the 1-neighborhoods of G 2 consist of two isolated nodes. 5. C OUNTING (I NDUCED ) S UBSTRUCTURES
In contrast to MPNNs, which in general cannot count substructures of three vertices or more (Chen et al., 2020), in this section we prove that for any set of substructures, there is an RNP-GNN that provably counts them.
We begin with a few deﬁnitions.
Deﬁnition 1.
Let G, H be arbitrary, potentially attributed simple graphs, where V is the set of nodes in G .
Also, for any S ⊆ V , let G ( S ) denote the subgraph of G induced by S .
The induced subgraph count function is deﬁned as
[Equation]
i.e., the number of subgraphs of G isomorphic to H .
To relate the size of encoded neighborhoods to the substructure H , we will need a notion of covering sequences for graphs.
Deﬁnition 2.
Let H =
( V H , E H ) be a simple connected graph.
For any S ⊆ V H and v ∈ V H , deﬁne the covering distance of v from [Equation]
as
[Equation]
where d ( ., . ) is the shortest-path distance in H .
Deﬁnition 3.
Let H be a simple connected graph on τ + 1 vertices.
A permutation of vertices, such as ( v 1 , v 2 , . .
. , v τ +1 ) , is called a vertex covering sequence with respect to a sequence r = ( r 1 , r 2 , . . .
, r τ ) ∈
N τ called a covering sequence , if and only if
[Equation]
10) for any i ∈ [ τ + 1] = { 1 , 2 , . . .
, τ + 1 } , where S i
= { v i , v i +1 . .
. , v τ +1 } and H (cid:48) i
= H ( S i ) is the subgraph of H induced by the set of vertices
S i .
We also say that H admits the covering sequence r =
( r 1 , r 2 , . . .
, r τ ) ∈
N τ if there is a vertex covering sequence for H with respect to r .
In particular, in a covering sequence we ﬁrst consider the whole graph as a local neighborhood of one of its nodes with radius r 1 .
Then, we remove that node and compute the covering sequence of the remaining graph.
Figure 4 shows an example of covering sequence computation.
An important property, which holds by deﬁnition, is that if r is a covering sequence for H , then any r (cid:48) ≥ r (coordinate-wise) is also a covering sequence for H .
Note that any connected graph on k nodes admits at least one covering sequence, which is ( k − 1 , k − 2 , . .
To observe this fact, note that in a connected graph, there is at least one node that can be removed and the remaining graph still remains connected.
Therefore, we may take this node as the ﬁrst element of a vertex covering sequence, and inductively ﬁnd the other elements.
Since the diameter of a connected graph with k vertices is always bounded by k − 1 , we achieve the desired result.
However, we will see in the next section that, when using covering sequences to identify sufﬁciently powerful RNP-GNNs, it is desirable to have covering sequences with low r 1 , since the complexity of the resulting RNP-GNN depends on r 1 .
More generally, if H 1 and H 2 are (possibly attributed) simple graphs on k nodes and H 1 (cid:98) H 2 , i.e.,
H 1 is a subgraph of H 2 (not necessarily induced subgraph), then it follows from the deﬁnition that any covering sequence for H 1 is also a covering sequence for H 2 .
As a side remark, as illustrated in
Figure 2, covering sequences need not always to be decreasing.
Using covering sequences, we can show the following result.
Theorem 1.
Consider a set of (possibly attributed) graphs H on τ +1 vertices, such that any H ∈ H admits the covering sequence ( r 1 , r 2 , . .
, r τ ) .
Then, there is an RNP-GNN f ( · ; θ ) with recursion parameters ( r 1 , r 2 , . . .
, r τ ) that can count any H ∈ H .
In other words, for any two graphs G 1 , G 2 , if there exists H ∈ H such that C ( G 1 ; H ) (cid:54) =
C ( G 2 ; H ) , then f ( G 1 ; θ ) (cid:54) =
f ( G 2 ; θ ) .
The same result also holds for the non-induced subgraph count function.
Theorem 1 states that, with appropriate recursion parameters, any set of (possibly attributed) substructures can be counted by an RNP-GNN.
Interestingly, induced and non-induced subgraphs can be both counted in RNP-GNNs 2 .
Also, for a given covering sequence, we can simultaneously count a set of substructures that each admit it.
This means that one RNP-GNN is able to potentially count many substructures.
We prove Theorem 1 in Appendix A.2.
The main idea is to show that we can implement the intuition for recursive pooling outlined in Section 4 formally with the proposed architecture and multiset functions 3 .
The theorem holds for any covering sequence that is valid for all graphs in H .
For any graph, one can compute a covering sequence by computing a spanning tree, and sequentially pruning the leaves of the tree.
The resulting sequence of nodes is a vertex covering sequence, and the corresponding covering sequence can be obtained from the tree too (Appendix D).
A valid covering sequence for all the graphs in H is the coordinate-wise maximum of all these sequences.
For large substructures, the sequence ( r 1 , r 2 , . . .
, r τ ) can be long or include large numbers, and this will affect the computational complexity of RNP-GNNs.
For small, e.g., constant-size substructures, the recursion parameters are also small (i.e., r i
= O (1) for all i ), raising the hope to count these structures efﬁciently.
In particular, r 1 is an important parameter.
In Section 5.3, we analyze the complexity of RNP-GNNs in more detail.
5. A U NIVERSAL A PPROXIMATION R ESULT FOR L OCAL F UNCTIONS
Theorem 1 shows that RNP-GNNs can count substructures if their recursion parameters are chosen carefully.
Next, we provide a universal approximation result, which shows that they can represent any function related to local neighborhoods or small subgraphs in a graph.
First, we recall that for a graph G , G ( S ) denotes the subgraph of G induced by the set of nodes S .
Deﬁnition 4.
A function (cid:96) : G n → R d is called an r − local graph function if
[Equation]
where ψ : G r → R d (cid:48) is a function on graphs, φ is a multi-set function, and V denotes the set of all nodes.
In other words, a local function only depends on small substructures.
Theorem 2.
For any r − local graph function (cid:96) ( . ) , there exists an RNP-GNN f ( . ; θ ) with recursion parameters ( r − 1 , r − 2 , . .
. , 1) such that f ( G ; θ ) =
(cid:96) ( G ) for any G ∈ G n .
To prove the theorem, we use speciﬁc aggregation functions, but since MLPs with ReLU activation are universal approximators, we can approximate those aggregations and ﬁnd an RNP-GNN f ( G ; θ ) implemented by MLPs such that | f
( G ; θ ) − (cid:96) ( G ) | < (cid:15) , for arbitrary small (cid:15) .
As a result, we can provably approximate all the local information in a graph with an appropriate RNP-GNN.
Note that we still need recursions, because the function ψ ( . ) may be an arbitrarily difﬁcult graph function.
However, to achieve the full generality of such a universal approximation result, we need to consider large recursion parameters ( r 1 = r − 1 ) and injective aggregations in the RNP-GNN network.
For universal approximation, we may also need high dimensions if fully connected layers are used for aggregation (see the proof in Appendix B for more details).
As a remark, for r = n , achieving universal approximation on graphs implies solving the graph isomorphism problem.
But, in this extreme case, the computational complexity of RNP is in general not polynomial in n .
5. C OMPUTATIONAL C OMPLEXITY
vvvvvv
The computational complexity of RNP-GNNs is graphdependent.
For instance, we need to compute the set of local neighborhoods, which is cheaper for sparse graphs.
Moreover, in the recursions, we use intersections of neighborhoods which become smaller and sparser.
Theorem 3.
Let f ( . ; θ ) : G n → R d be an RNPGNN with recursion parameters ( r 1 , r 2 , . .
, r τ ) .
Assume that the observed graphs G 1 , G 2 , . .
, whose representations we compute, satisfy the following property: max v ∈ [ n ] |N r 1 ( v ) | ≤ c, for a constant c .
Then the number of node updates in the RNP-GNN is O ( nc τ ) .
In other words, if c = n o (1) and τ = O (1) , then RNPGNN requires relatively few updates (that is, n 1+ o (1) ).
If the maximum degree of the given graphs is ∆ , then c
= O ( r 1 ∆ r 1 ) .
Therefore, similarly, if ∆ = n o (1) then we can count with at most n 1+ o (1) updates.
Additional gains may arise from rapidly shrinking neighborhoods, which are not yet accounted for in Theorem 3.
To put this in context, the higher-order GNNs based on tensors or k -WL would operate on tensors of order n τ +1 .
[Table]
The above results show that when using RNP-GNNs with sparse graphs, we can represent functions of substructures with k nodes without requiring k − order tensors.
LRPs also encode neighborhoods of distance raround nodes.
In particular, all c !
permutations of the nodes in a neighborhood of size c are considered to obtain the representation.
As a result, LRP networks only have polynomial complexity if c = o (log( n )) .
Thus, RNP-GNNs can provide an exponential improvement in terms of the tolerable size c of neighborhoods with distance r 1 in the graph.
Moreover, Theorem 3 suggests to aim for small r 1 .
The other r i ’s may be larger than r 1 , as shown in Figure 2, but do not affect the upper bound on the complexity.
A N I NFORMATION -T HEORETIC L OWER B OUND
In this section, we provide a general information-theoretic lower bound for graph representations that encode a given graph G by ﬁrst encoding a number of (possibly small) graphs G 1 , G 2 , . . .
, G t and then aggregating the resulting representations.
The sequence of graphs G 1 , G 2 , . .
. , G t may be obtained in an arbitrary way from G .
For example, in an MPNN, G i can be the computation tree (rooted tree) at node i .
As another example, in LRP, G i is the local neighborhood around node i .
Formally, consider a graph representation f ( . ; θ ) :
G n → R d as
[Equation]
for any G ∈ G n , where A GGREGATE is a multi-set function, ( G 1 , G 2 , . . .
, G t ) =
Ξ( G ) where Ξ( . ) :
G n → (cid:0) (cid:83) ∞ m =1 G m (cid:1) t is a function from one graph to t graphs, and ψ : (cid:83) ∞ m =1 G m → [ s ] is a function on graphs taking s values.
In short, we encode t graphs, and each encoding takes one of s values.
We call this graph representation function an ( s, t ) -good graph representation.
Theorem 4.
Consider a parametrized class of ( s, t ) − good representations f ( . ; θ ) : G n → R d that is able to count any (not necessarily induced 4 ) substructure with k vertices.
More precisely, for any graph H with k vertices, there exists f ( . ; θ ) such that if C ( G 1 ; H ) (cid:54) =
C ( G 2 ; H ) , then f ( G 1 ; θ ) (cid:54) =
f ( G 2 ; θ ) .
Then 5 t = ˜Ω( n ks − 1 ) .
In particular, for any ( s, t ) − good graph representation with s = 2 , i.e., binary encoding functions, we need ˜Ω( n k ) encoded graphs.
This implies that, for s = 2 , enumerating all subgraphs and deciding for each whether it equals H is near optimal.
Moreover, if s ≤ k , then t = Θ( n ) small graphs would not sufﬁce to enable counting.
More interestingly, if k, s =
O (1) , then it is impossible to perform the substructure counting task with t =
O (log( n )) .
As a result, in this case, considering n encoded graphs (as is done in GNNs or
LRP networks) cannot be exponentially improved.
The lower bound in this section is information-theoretic and hence applies to any algorithm.
It may be possible to strengthen it by considering computational complexity, too.
For binary encodings, i.e., s = 2 , however, we know that the bound cannot be improved since manual counting of subgraphs matches the lower bound.
T IME C OMPLEXITY L OWER B OUNDS FOR C OUNTING S UBGRAPHS
In this section, we put our results in the context of known hardness results for subgraph counting.
In general, the subgraph isomorphism problem is known to be NP-complete.
Going further, the
Exponential Time Hypothesis (ETH) is a conjecture in complexity theory (Impagliazzo & Paturi, 2001), and states that several NP-complete problems cannot be solved in sub-exponential time.
ETH, as a stronger version of the P (cid:54) =
NP problem, is widely believed to hold.
Assuming that
ETH holds, the k − clique detection problem requires at least n Ω( k ) time (Chen et al., 2005).
This means that if a graph representation can count any subgraph H of size k , then computing it requires at least n Ω( k ) time.
Corollary 1.
Assuming the ETH conjecture holds, any graph representation that can count any substructure H on k vertices with appropriate parametrization needs n Ω( k ) time to compute.
The above bound matches the O ( n k ) complexity of the higher-order GNNs.
Comparing with Theorem 4 above, Corollary 1 is more general, while Theorem 4 has fewer assumptions and offers a reﬁned result for aggregation-based graph representations.
Given that Corollary 1 is a worst-case bound, a natural question is whether we can do better for subclasses of graphs.
Regarding H , even if H is a random Erd¨os-R´enyi graph, it can only be counted in n Ω( k/ log k ) time (Dalirrooyfard et al., 2019).
Regarding the input graph in which we count, consider two classes of sparse graphs: strongly sparse graphs have maximum degree ∆
=
O (1) , and weakly sparse graphs have average degree ¯∆
= O (1) .
We argued in Theorem 3 that RNP-GNNs achieve almost linear complexity for the class of strongly sparse graphs.
For weakly sparse graphs, in contrast, the complexity of RNP-GNNs is generally not linear, but still polynomial, and can be much better than O ( n k ) .
One may ask whether it is possible to achieve a learnable graph representation such that its complexity for weakly sparse graphs is still linear.
Recent results in complexity theory imply that this is impossible:
Corollary 2 (Gishboliner et al.
; Bera et al.
There is no graph representation algorithm that runs in linear time on weakly sparse graphs and is able to count any substructure H on k vertices (with appropriate parametrization) [Table]
Hence, RNP-GNNs are close to optimal for several cases of counting substructures with parametrized learnable functions.
E XPERIMENTS
In this section
[Table]
we validate our theoretical ﬁndings via numerical experiments.
Here
[Table]
we brieﬂy describe our experimental setup and results — further experimental details are given in Appendix H
[Table]
[Table]
[Table]
Counting substructures.
First, we follow the experimental setup of Chen et al.
(2020) on tasks for counting substructures.
In Table 2, we report results for learning induced subgraph count of triangles and non-induced subgraph count of 3 -stars.
We test on two datasets of 5000 graphs each: one of
Erd ˝ os-Renyi graphs and one of random regular graphs.
Our RNP-GNN model is consistently within the best performing models for these counting tasks, thus validating our theoretical results.
Based on the baseline results taken from (Chen et al., 2020), RNP-GNN tends to widely outperform MPNNs (GCN (Kipf & Welling, 2017),
GIN (Xu et al., 2019), GraphSAGE (Hamilton et al., 2017)) [Table] and other models not tailored for counting: spectral GNN (Chen et al., 2018) [Table]
and 2-IGN (Maron et al., 2018). Also, RNP-GNN often beats higher-order GNNs: PPGN (Maron et al., 2019a) and LRP-13 (Chen et al., 2020).
RNP-GNN is mostly comparable to Deep LRP-1-3, though Deep LRP-1outperforms it in a few cases.
Recall that Deep LRP-1-3 is a practical version of LRP — we leave further developments of practical variants of RNP-GNN to future work.
The best r parameters for
RNP-GNN were: r = (1 , 1 , 1 , 1) for triangles on Erd˝os-Renyi, r = (1 , 1) for stars on Erd˝os-Renyi, r = (1 , 1) for triangles on random regular and r = (1 , 1 , 1) for stars on random regular.
Satisﬁability of propositional formulas.
Second, we test the expressiveness of our model in distinguishing non-isomorphic graphs that 1-WL cannot distinguish.
The EXP dataset of 600 graphs (Abboud et al., 2021) for classifying whether certain propositional formulas are satisﬁable requires higher than 1-WL expressive power to achieve better than random accuracy.
As shown in Table 3, while our RNP-GNN with r 1 = 1 is unable to achieve better than random accuracy, our RNP-GNN with r 1 = 2 (we use r = (2 , 1 ) achieves near perfect accuracy — beating all other models based on results taken from (Abboud et al., 2021).
These other models include universal models with random node identiﬁers (GCN-RNI (Abboud et al., 2021)),
GNNs with 3-WL power (PPGN (Maron et al., 2019a)), and GNNs that imitate some (possibly weaker) version of 3-WL (1-2-3-GCN-L (Morris et al., 2019), 3-GCN (Abboud et al., 2021)).
Thus, our architecture, which is not developed within common frameworks for achieving k -WL expressiveness, is in fact powerful at distinguishing nonisomorphic graphs.
A P ROOF OF T HEOREMA. P RELIMINARIES
Let us ﬁrst state a few deﬁnitions about the graph functions.
Note that for any graph function f : G n → R d , we have f ( G ) =
f ( H ) for any G ∼ = H .
Deﬁnition 5.
Given two graph functions f, g : G n → R d , we write f (cid:119) g , if and only if for any
G 1 , G 2 ∈ G n ,
[Equation]
[Equation]
or, equivalently,
[Equation]
Proposition 1.
Consider graph functions f, g, h : G n → R d such that f (cid:119) g and g (cid:119) h .
Then, f (cid:119) h .
In other words, (cid:119) is transitive.
Proof.
The proposition holds by deﬁnition.
Proposition 2.
Consider graph functions f, g : G n → R d such that f (cid:119) g .
Then, there is a function ξ : R d → R d such that ξ ◦ f = g .
Proof.
Let G n =
(cid:116) i ∈ N F i be the partitioning induced by the equality relation with respect to the function f on G n .
Similarly deﬁne G i , i ∈ N for g .
Note that due to the deﬁnition, {F i : i ∈ N } is a reﬁnement for {G i : i ∈ N } .
Deﬁne ξ to be the unique mapping from {F i : i ∈ N } to {G i : i ∈ N } which respects the equality relation.
One can observe that such ξ satisﬁes the requirement in the proposition.
Deﬁnition 6.
An RNP-GNN is called maximally expressive, if and only if
• all the aggregate functions are injective as mappings from a multi-set on a countable ground set to their codomain.
• all the combine functions are injective mappings.
Proposition 3.
Consider two RNP-GNNs f, g with the same recursion parameters r = ( r 1 , r 2 , . .
, r τ ) where f is maximally expressive.
Then, f (cid:119) g .
Proof.
The proposition holds by deﬁnition.
Proposition 4.
Consider a sequence of graph functions f, g 1 , . .
. , g k .
If f (cid:119) g i for all i ∈
[ k ] , then
[Equation]
for any c i ∈ R , i ∈ N .
Proof.
Since f (cid:119) g
i , we have
[Equation]
[Equation]
G n :
[Equation]
(16) for all i ∈ [ k ] .
This means that for any G 1 , G 2 ∈ G n if f ( G 1 ) =
f ( G 2 ) then g i ( G 1 ) =
g i ( G 2 ) , i ∈ [ k ] , and consequently (cid:80) ki
=1 c i g i ( G 1 )
= (cid:80) ki
=1 c i g i ( G 2 ) .
Therefore, from the deﬁnition we conclude f (cid:119) (cid:80) ki =1 c i g i .
Note that the same proof also holds in the case of countable summations as long
Deﬁnition 7.
Let H =
( V H , E H , X H ) be a attributed connected simple graph with k nodes.
For any attributed graph G =
( V G , E G , X G ) ∈ G n , the induced subgraph count function C ( G ; H ) is deﬁned as
[Equation]
Also, let ¯ C
( G ; H ) denote the number of non-induced subgraphs of G which are isomorphic to H .
It can be deﬁned with the homomorphisms from H to G .
Formally, if n > k deﬁne
[Equation]
Otherwise, n = k , and we deﬁne
[Equation]
where
[Equation]
is deﬁned with respect to the graph isomorphism, and c ˜ H,H ∈ N denotes the number of subgraphs in H identical to ˜ H .
Note that ˜ H ( H ) is a ﬁnite set and (cid:99) denotes being a (not necessarily induced) subgraph.
Proposition 5.
Let H be a family of graphs.
If for any H ∈ H , there is an RNP-GNN f H ( . ; θ ) with recursion parameters ( r 1 , r 2 , . . .
, r τ ) such that f H (cid:119) C ( G ; H ) , then there exists an RNP-GNN f ( . ; θ ) with recursion parameters ( r 1 , r 2 , . . .
, r τ ) such that f (cid:119) (cid:80) H ∈H C ( G ; H ) .
Proof.
Let f ( . ; θ ) be a maximally expressive RNP-GNN.
Note that by the deﬁnition f (cid:119)
f H for any
H ∈ H .
Since (cid:119) is transitive, f (cid:119) C
( G ; H ) for all H ∈ H , and using Proposition 4, we conclude that f
[Equation]
The following proposition shows that it is sufﬁcient to address counting attributed graphs.
Proposition 6.
Let H 0 be an unattributed connected graph.
Assume that for any attributed graph
H , which is constructed by adding arbitrary attributes to H 0 , there exists an RNP-GNN f H ( . ; θ H ) such that f H (cid:119) C ( G ; H ) , then for its unattributed counterpart H 0 , there exists an RNP-GNN f ( . ; θ ) with the same recursion parameters as f H ( . ; θ H ) such that f (cid:119) C
( G ; H 0 ) .
Proof.
If there exists an RNP-GNN f H ( . ; θ H ) such that f H (cid:119) C ( G ; H ) , then for a maximally expressive RNP-GNN f ( . ; θ ) with the same recursion parameters as f H we also have f (cid:119) C ( G ; H ) .
Let H be the set of all attributed graphs H
= ( V , E , X ) ∈
G k up to graph isomorphism, where X ∈ X k for a countable set X .
Note that H
= { H 1 , H 2 , . . .
} is a countable set.
Now we write
[Equation]
[Equation]
S⊆
[Equation]
[Equation]
(cid:88)
S⊆ [ n ] |S| = k 1 { G ( S ) ∼ = H [Equation] Now using Proposition 4 we conclude that f (cid:119) C ( G ; H 0 ) since C ( G ; H 0 ) is always ﬁnite.
Deﬁnition 8.
Let H be a (possibly attributed) simple connected graph.
For any S ⊆ V H and v ∈ V H , deﬁne
[Equation]
Deﬁnition 9.
Let H be a (possibly attributed) connected simple graph with k = τ + 1 vertices.
A permutation of vertices, such as ( v 1 , v 2 , . .
. , v τ +1 ) , is called a vertex covering sequence, with respect to a sequence r = ( r 1 , r 2 , . . .
, r τ ) ∈
N τ , called a covering sequence, if and only if
[Equation]
i [Equation] 27) for i ∈ [ τ + 1] , where H (cid:48)
i = H ( S i ) and S
i = { v i , v i +1 , . .
. , v τ +1 } .
Let C H ( r ) denote the set of all vertex covering sequences with respect to the covering sequence r for H .
Proposition 7.
For any G, H ∈ G k , if G (cid:99) H (non-induced subgraph), then
[Equation]
for any sequence r .
Proof.
The proposition follows from the fact that the function ¯ d is decreasing with introducing new edges.
Proposition 8.
Assume that Theorem 1 holds for induced-subgraph count functions.
Then, it also holds for the non-induced subgraph count functions.
Proof.
Assume that for a connected (attributed or unattributed) graph H , there exists an RNP-GNN with appropriate recursion parameters f H ( . ; θ H ) such that f H (cid:119) C ( G ; H ) , then we prove there exists an RNP-GNN f ( . ; θ ) with the same recursion parameters as f H such that f (cid:119) ¯ C ( G ; H ) .
If there exists an RNP-GNN f H ( . ; θ H ) such that f H (cid:119) C ( G ; H ) , then for a maximally expressive
RNP-GNN f ( . ; θ ) with the same recursion parameters as f H we also have f (cid:119) C ( G ; H ) .
Note that
[Equation]
where ˜ H ( H ) =
{ H 1 , H 2 , . . . } .
Claim 1.
f (cid:119) C ( G, H i ) for any i .
Using Proposition 4 and Claim 1 we conclude that f (cid:119) ¯ C
( G ; H ) since ¯ C ( G ; H ) is ﬁnite and f (cid:119) C
( G, H i ) for any i , and the proof is complete.
The missing part which we must show here is that for any H
i the sequence ( r 1 , r 2 , . . .
, r t ) which covers H also covers H i .
This follows from
Proposition 7.
We are done.
At the end of this part, let us introduce an important notation.
For any attributed connected simple graph on k vertices G = ( V , E , X ) , let G ∗ v be the resulting induced graph obtained after removing v ∈ V from G with the new attributes deﬁned as
[Equation]
(33) for each u ∈ V \ { v } .
We may also use X ∗ vu for more clariﬁcation.
A. P ROOF OF T HEOREM
We utilize an inductive proof on τ , which is the length of the covering sequence of H .
Equivalently, due to the deﬁnition, τ = k − 1 , where k is the number of vertices in H .
First, we note that due to Proposition 8, without loss of generality, we can assume that H is a simple connected attributed graph and the goal is to achieve the induced-subgraph count function via an RNP-GNN with appropriate recursion parameters.
We also consider only maximally expressive networks here to prove the desired result.
Induction base.
For the induction base, i.e., τ = 1 , H is a two-node graph.
This means that we only need to count the number of a speciﬁc (attributed) edge in the given graph G .
Note that in this case we apply an RNP-GNN with recursion parameter r 1 ≥ 1 .
Denote the two attributes of the vertices in H by X H 1 , X H 2 ∈ X .
The output of an RNP-GNN f ( . ; θ ) is
[Equation]
where we assume that f ( . ; θ ) is maximally expressive.
The goal is to show that f (cid:119) C ( G ; H ) .
Using the transitivity of (cid:119) , we only need to choose appropriate φ, ψ, ϕ to achieve ˆ f = C ( G ; H ) as the ﬁnal representation.
Let
[Equation]
Then, a simple computation shows that
[Equation]
Since ˆ f ( . ; θ ) is an RNP-GNN with recursion parameter r 1 and for any maximally expressive RNPGNN f ( . ; θ ) with the same recursion parameter as ˆ f we have f (cid:119) ˆ f
and ˆ f (cid:119) C ( G ; H ) , we conclude that f (cid:119) C ( G ; H ) and this completes the proof.
Induction step.
Assume that the desired result holds for τ − 1 ( τ ≥ 2 ).
We show that it also holds for τ .
Let us ﬁrst deﬁne
[Equation]
where H ∗ v means the induced subgraph after removing a node, with new attributes (see A.1).
Note that H ∗ (cid:54) = ∅ by the assumption.
Let
[Equation] [Equation]
For all H 0 ∈ H ∗ , using the induction hypothesis, there is a (universal) RNP-GNN ˆ f ( . ; ˆ θ ) with recursion parameters ( r 2 , r 3 , . . .
, r τ )
such that ˆ f (cid:119) C
( G ; H 0 ) .
Using Proposition 4 we conclude
[Equation]
Deﬁne a maximally expressive RNP-GNN with the recursion parameters ( r 1 , r 2 , . . .
, r τ ) as follows:
[Equation]
Similar to the proof for τ = 1 , here we only need to propose a (not necessarily maximally expressive)
RNP-GNN which achieves the function C ( G ; H ) .
Let us deﬁne
[Equation]
where
[Equation]
and ξ ◦ ˆ f = C ( G ; H ∗ u ) .
Note that the existence of such function ξ is guaranteed due to Proposition 2.
Now we write
[Equation]
which means that
[Equation]
However, for a maximally expressive RNP-GNN f ( . ; θ ) we know that f (cid:119)
f H ∗ u for all H ∗ u ∈ H and this means that f (cid:119) C ( G ; H ) .
The proof is thus complete.
B P ROOF OF T HEOREM
For any attributed graph H on r nodes (not necessarily connected)
we claim that RNP-GNNs can count them.
Claim 2.
Let f ( . ; θ ) : G n → R d be a maximally expressive RNP-GNN with recursion parameters ( r − 1 , r − 2 , . .
Then, f (cid:119) C ( G ; H ) .
Now consider the function
[Equation]
We claim that f (cid:119) (cid:96) ( f is deﬁned in the previous claim) and this completes the proof according to
Proposition 2.
To prove the claim, assume that f ( G 1 ) =
f ( G 2 ) .
Then, we conclude that C ( G 1 ; H ) =
C ( G 2 ; H ) for any attributed H (not necessarily connected) with r vertices.
Now, we have
[Equation] [Equation]
the multiplicity of H
Proof of Claim 2.
To prove the claim, we use an induction on the number of connected components
c H of graph H .
If H is connected, i.e., c H = 1 , then according to Theorem 1, we know that f (cid:119) C ( G ; H ) .
Now assume that the claim holds for c H =
c − 1 ≥ 1 .
We show that it also holds for c H
= c .
Let
H 1 , H 2 , . .
, H c denote the connected components of H .
Also assume that H i (cid:54)∼
= H j for all i (cid:54) =
j .
We will relax this assumption later.
Let us deﬁne
[Equation]
Note that we can write
[Equation]
where H (cid:48) 1 , H (cid:48) 2 , .
are all non-isomorphic graphs obtained by adding edges (at least one edge) between c graphs H 1 , H 2 , . .
, H c , or contracting a number of vertices of them.
The constants c (cid:48)
j are just used to remove the effect of multiple counting due to the symmetry.
Now, since for any
H i , H (cid:48)
j the number of connected components is strictly less that c , using the induction, we have f (cid:119) C
( G ; H i ) and f (cid:119) C ( G ; H (cid:48)
j ) for all j and all i ∈ [ c ] .
According to Proposition 4, we conclude that f (cid:119) C ( G ; H ) and this completes the proof.
Also, if H i , i ∈ [ c ] , are not pairwise non-isomorphic, then we can use αC ( G ; H ) in above equation instead of C ( G ; H ) , where α >removes the effect of multiple counting by symmetry.
The proof is thus complete.
Remark 1.
As we explained in the above proof, one can modify Theorem 1 to hold for disconnected graphs.
To this end, we need to generalize the notion of covering sequence to hold for this class of graphs.
Since this special case is out of scope of this paper, we only restrict to a special, but more insightful case.
C P ROOF OF T HEOREM
To prove Theorem 3, we need to bound the number of node updates required for an RNP-GNN with recursion parameters ( r 1 , r 2 , . .
. , r t ) .
First of all, we have n variables used for the ﬁnal representations of vertices.
For each vertex v 1 ∈ V , we explore the local neighborhood N r 1 ( v 1 ) and apply a new RNP-GNN network to that neighborhood.
In other words, for the second step we need to update |N r 1 ( v 1 ) | nodes.
Similarly, for the i th step of the algorithm we have at most
[Equation]
[Equation]
updates.
Therefore, we can bound the number of node updates as
[Equation]
Since λ i is decreasing in i , the desired result holds.
D P ROOF OF T HEOREM
Let K k denote the complete graph on k vertices.
Claim 3.
For any k, n ∈ N , such that n is sufﬁciently large,
[Equation]
where c is a constant which does not depend on k, n .
In particular, we claim that the number of different values that C ( G ; K k ) can take is n k , up to poly-logarithmic factors.
To prove the theorem, we use the above claim.
Consider a class of ( s, t ) − good graph representations f ( . ; θ ) which can count any substructure on k vertices.
As a result, f (cid:119) C ( G ; K k ) for an appropriate parametrization θ .
By the deﬁnition, f ( . ) must take at least (cid:12)(cid:12)(cid:12) { C ( G ; K k ) :
G ∈ G n } (cid:12)(cid:12)(cid:12) different values, i.e.,
[Equation]
Also,
[Equation]
) where ( G 1 , G 2 , . .
. , G t ) =
Ξ( G ) .
But, ψ can take only s values.
Therefore, we have
[Equation]
As a result, ( t + 1) s − 1 = ˜Ω( n k ) or t = ˜Ω( n ks − 1 ) .
To complete the proof, we only need to prove the claim.
Proof of Claim 3.
Let p 1 , p 2 , . .
. , p m be distinct prime numbers less than n/k .
Using the prime number theorem, we know that lim n →∞ m n/ ( k log( n/k ))
= 1 .
In particular, we can choose n large enough to ensure cn/ ( k log( n/k )) <
m for any constant c < 1 .
For any B = { b 1 , b 2 , . . .
, b k } ⊆
[ m ] , deﬁne G B as a graph on n vertices such that V G B = V
(cid:116) ( (cid:116) i ∈
[ k ] V i ) , and |V i | =
p b i .
Also,
[Equation]
[Equation]
The graph G B is well-deﬁned since (cid:80) ki =1 p b i ≤ k × n/k = n .
Note that C ( G B ; K k ) =
(cid:81) ki
=1 p b i .
Also, since p i , i ∈ [ m ] , are prime numbers, there is a unique bijection [Equation]
[Equation]
Therefore,
[Equation]
E R ELATIONSHIP TO THE R ECONSTRUCTION C ONJECTURE
Theorem 2 provides a universality result for RNP-GNNs.
Here, we note that the proposed method is closely related to the reconstruction conjecture, an old open problem in graph theory.
This motivates us to explain their relationship/differences.
First, we need a deﬁnition for unattributed graphs.
Deﬁnition 10.
Let F n ⊆
G n be a set of graphs and let G v =
G ( V \ { v } ) for any ﬁnite simple graph G = (
V , E ) , and any v ∈ V .
Then, we say the set F is reconstructible if and only if there is a bijection [Equation] 77) for any G ∈ F n .
In other words, F n is reconstructible, if and only if the multi-set {{ G v : v ∈ V}} fully identiﬁes G for any G ∈ F n .
It is known that the class of disconnected graphs, trees, regular graphs, are reconstructible (Kelly, 1957; McKay, 1997).
The general case is still open; however it is widely believed that it is true.
Conjecture 1 (Kelly (1957)) .
G n is reconstructible.
For RNP-GNNs, the reconstruction from the subgraphs G ∗ v , v ∈ [ n ] is possible, since we relabel any subgraph (in the deﬁnition of X ∗ ) and this preserves the critical information for the recursion to the original graph.
In the reconstruction conjecture, this part of information is missing, and this makes the problem difﬁcult.
Nonetheless, since in RNP-GNNs we preserve the original node’s information in the subgraphs with relabeling, the reconstruction conjecture is not required to hold to show the universality results for RNP-GNNs, although that conjecture is a motivation for this paper.
Moreover, if it can be shown that the reconstruction conjecture it true, it may be also possible to ﬁnd a simple encoding of subgraphs to an original graph and this may lead to more powerful but less complex new GNNs.
F T HE RNP-GNN A LGORITHM
In this section, we provide pseudocode for RNP-GNNs.
The algorithm below computes node representations.
In the algorithms, we frequently use MLP modules with ReLU activation.
For a graph representation, we can aggregate them with a common readout, e.g., h G ← MLP (cid:16) (cid:80) v ∈V h ( k ) v (cid:17) .
Following (Xu et al., 2019), we use sum pooling here, to ensure that we can represent injective aggregation functions.
Input: G =
( V , E , { x v } v ∈V )
where V =
[ n ] , recursion parameters r 1 , r 2 , . . . , r τ ∈ N , (cid:15) ( i ) ∈ R , i ∈
[ τ ] , node features { x v } v ∈V .
Output: h v for all v ∈ V h in v ← x v for all v ∈ V if τ = 1 then h v ←
[Equation]
for all v ∈ V .
elsefor all v ∈ V do
G (cid:48) v ← G ( N r 1 ( v ) \ { v } ) , which has node attributes { ( h in u , 1 ( u, v ) ∈ E ) (cid:48) } u ∈N r 1 ( v ) \{ v } { ˆ
[Equation]
) h v ←
[Equation]
[Equation]
(cid:16)
[Equation]
+ (cid:15)
[Equation]
end for end if return { h v } v ∈V
With this algorithm, one can achieve the expressive power of RNP-GNNs if high dimensional MLPs are allowed (Xu et al., 2019;
Hornik et al., 1989; Hornik, 1991).
That said, in practice, smaller
MLPs may be acceptable (Xu et al., 2019).
G C OMPUTING A C OVERING S EQUENCE
As we explained in the context of Theorem 1, we need a covering sequence (or an upper bound to that) to design an RNP-GNN network that can count a given substructure.
A covering sequence can be constructed from a spanning tree of the graph.
For reducing complexity, it is desirable to have a covering sequence with minimum r 1 (Theorem 3).
Here, we suggest an algorithm for obtaining such a covering sequence, shown in Algorithm 2.
For obtaining merely an aribtrary covering sequence, one can compute any minimum spanning tree (MST), and then proceed as with the MST in Algorithm 2.
Given an MST, we build a vertex covering sequence by iteratively removing a leaf v i from the tree and adding the respective node v i to the sequence.
This ensures that, at any point, the remaining graph is connected.
At position i corresponding to v i , the covering sequence contains the maximum distance r i of v i to any node in the remaining graph, or an upper bound on that.
For efﬁciency, an upper bound on the distance can be computed in the tree.
To minimize r 1 = max u ∈V d ( u, v 1 ) , we need to ensure that a node in arg min v ∈V max u ∈V d ( u, v ) is a leaf in the spanning tree.
Hence, we ﬁrst compute max u ∈V d ( u, v ) for all nodes v , e.g., by running All-Pairs-Shortest-Paths (APSP) (Kleinberg & Tardos, 2006), and sort them in increasing order by this distance.
Going down this list, we try whether it is possible to use the respective node as v 1 , and stop when we ﬁnd one.
Say v ∗ is the current node in the list.
To compute a spanning tree where v ∗ is a leaf, we assign a large weight to all the edges adjacent to v ∗ , and a very low weight to all other edges.
If there exists such a tree, running an MST with the assigned weights will ﬁnd one.
Then, we use v ∗ as v 1 in the vertex covering sequence.
This algorithm runs in polynomial time.
Algorithm
Input: H =
( V , E , X ) where V
=
[ τ + 1] Output: A minimal covering sequence ( r 1 , r 2 . . .
, r τ ) , and its corresponding vertex covering sequence ( v 1 , v 2 , . .
. , v τ +1 )
For any u, v ∈ V , compute d ( u, v ) using APSP ( u 1 , u 2 , . . .
, u τ +1 ) ← all the vertices sorted increasingly in s ( v ) :
= max u ∈V d ( u, v ) for i = 1 to τ + 1 do
Set edge weights w ( u, v ) =
1 + τ × 1 { u = u
i ∨ v = u i } for all ( u, v ) ∈
E H
T ← the MST of H with weights w if u i is a leaf in H T then v 1 ← u i r← s ( u i ) break end if end for for i = 2 to τ + 1 do v
i ← one of the leaves of H T r i ← max u ∈V HT d ( u, v i )
H T ← H T after removing v i end for return ( r 1 , r 2 , . . .
, r τ ) and ( v 1 , v 2 , . .
. , v τ +1 )
H E XPERIMENTAL D ETAILS
[Table] H. D ATASET AND T ASK D ETAILS
For the counting experiments, we follow the setup of Chen et al.
There are two datasets: one consisting of 5000 Erd˝os-Renyi graphs (Erdos et al., 1960) and one consisting of 5000 noisy random regular graphs (Steger & Wormald, 1999).
Each Erd˝os-Renyi graph has 10 nodes, and each random regular graph has either 10, 15, 20, or 30 nodes.
Also, n random edges are deleted from each random regular graph, where n is the number of nodes.
For the experiments on distinguishing non-isomorphic graphs, we use the EXP dataset (Abboud et al., 2021).
This dataset consists of 600 pairs of graphs (so 1200 graphs in total), where each pair is 1-WL equivalent but distinguishable by 3-WL, and each pair contains one graph that represents a satisﬁable formula and one graph that represents an unsatisﬁable formula.
On average, each graph contains about 44 nodes and 110 edges (Balcilar et al., 2021).
We report the mean and standard deviation across 10 cross-validation folds.
Additionally, we report the runtimes in Table 4 for the counting experiments.
H. RNP-GNN I MPLEMENTATION D ETAILS
Here, we detail some speciﬁc design choices we make in implementing our RNP-GNN model.
Most embeddings are computed in R d for some ﬁxed hidden dimension d .
The input node features are ﬁrst embedded in R d by an initial linear layer.
Then RNP layers are applied to compute node representations.
Finally, a sum pooling across nodes followed by a ﬁnal MLP is used to compute a graph-level output.
An RNP layer for r = ( r 1 , . . .
, r τ ) is implemented as follows.
Note that the input node features to this layer are in R d due to our initial linear layer.
Also, note that we concatenate an extra feature dimension due to the augmented indicator feature at each recursion step.
To align these feature dimensions, for l ∈ [ τ ] , we parameterize the l -th GIN (Xu et al., 2019) by a feedforward neural network MLP ( l ) :
R d + l → R d + l − 1 .
For instance, the last GIN has a feedforward network MLP ( τ ) :
R d + τ → R d + τ − 1 , because after τ levels of recursion we have augmented τ features.
Dropout and nonlinear activation functions are only applied in the MLPs.
H. H YPERPARAMETERS
For all baseline models, we take the results from other papers.
Thus, for the counting experiments the conﬁgurations for the baseline models are from Chen et al.
(2020), while for the EXP experiments the conﬁgurations for the baseline models are from Abboud et al.
RNP-GNN hyperparameters.
For all experiments we ran random search over hyperparameters.
In all cases we used the Adam optimizer with initial learning rate in { . 01 , .
We train for 100 epochs with a batch size in { 16 , 32 , 128 } .
The number of stacked RNP-GNNs for computing node representations is in { 1 , 2 } .
We use a dropout ratio in { 0 , .
The recursion parameters used varies for each task.
We used two layers for each MLP used in the aggregation function.
Also, the graph-level output obtained after sum-pooling across nodes is computed by a two layer MLP.
Speciﬁcally for the counting experiments, the number of hidden dimensions is searched in { 16 , 32 , 64 } .
For all tasks we used r 1 = 1 .
In particular, the optimal r parameters for RNP-GNN were: r = (1 , 1 , 1 , 1) for triangles on Erd˝os-Renyi, r = (1 , 1) for stars on Erd˝os-Renyi, r = (1 , 1) for triangles on random regular and r = (1 , 1 , 1) for stars on random regular.
We use ReLU activations in the MLPs.
We either decay the learning rate by half every 25 , 50 , or ∞ epochs (where ∞ means never decaying).
For the EXP experiments, the number of hidden dimensions is searched in { 8 , 16 , 32 , 64 } .
We use either ELU or ReLU activations in the MLPs.
We decay the learning rate by half at the 50th epoch.
The recursion parameters are r = (2 , 1) . [Figure] [Figure] For this
( v v v v