W HAT G RAPH N EURAL N ETWORKS C ANNOT L EARN : D EPTH VS W IDTH
A BSTRACT
This paper studies the expressive power of graph neural networks falling within the message-passing framework ( GNN mp ).
Two results are presented.
First, GNN mp are shown to be Turing universal under sufﬁcient conditions on their depth, width, node attributes, and layer expressiveness.
Second, it is discovered that GNN mp can lose a signiﬁcant portion of their power when their depth and width is restricted.
The proposed impossibility statements stem from a new technique that enables the repurposing of seminal results from distributed computing and leads to lower bounds for an array of decision, optimization, and estimation problems involving graphs.
Strikingly, several of these problems are deemed impossible unless the product of a GNN mp ’s depth and width exceeds a polynomial of the graph size; this dependence remains signiﬁcant even for tasks that appear simple or when considering approximation.
I NTRODUCTION
A fundamental question in machine learning is to determine what a model can and cannot learn.
In deep learning, there has been signiﬁcant research effort in establishing expressivity results for feedforward (Cybenko, 1989; Hornik et al., 1989; Lu et al., 2017) and recurrent neural networks (Neto et al., 1997), as well as more recently for Transformers and Neural GPUs (P´erez et al., 2019).
We have also seen the ﬁrst results studying the universality of graph neural networks, i e., neural networks that take graphs as input.
Maron et al.
(2019b) derived a universal approximation theorem over invariant functions targeted towards deep networks whose layers are linear and equivariant to permutation of their input.
Universality was also shown for equivariant functions and a particular shallow architecture (Keriven & Peyr ´ e, 2019).
Universality statements allow us to grasp the expressive power of models in the limit.
In theory, given enough data and the right training procedure, a universal network will be able to solve any task that it is presented with.
Nevertheless, the insight brought by such results can also be limited.
Knowing that a sufﬁciently large network can be used to solve any problem does not reveal much about how neural networks should be designed in practice.
It also cannot guarantee that said network will be able to solve a given task given a particular training procedure, such as stochastic gradient descent.
On the other hand, it might be easier to obtain insights about models by studying their limitations.
After all, the knowledge of what cannot be computed (and thus learned) by a network of speciﬁc characteristics applies independently of the training procedure.
Further, by helping us comprehend the difﬁculty of a task in relation to a model, impossibility results can yield practical advice on how to select model hyperparameters.
Take, for instance, the problem of graph classiﬁcation.
Training a graph classiﬁer entails identifying what constitutes a class, i.e., ﬁnding properties shared by graphs in one class but not the other, and then deciding whether new graphs abide to said learned properties.
However, if the aforementioned decision problem is shown to be impossible by a graph neural network of certain depth then we can be certain that the same network will not learn how to classify a sufﬁciently diverse test set correctly, independently of which learning algorithm is employed.
We should, therefore, focus on networks deeper that the lower bound when performing experiments.
[Table]
1. M AIN RESULTS
This paper studies the expressive power of message-passing graph neural networks ( GNN mp )
(Gilmer et al., 2017).
This model encompasses several state-of-the-art networks, including GCN (Kipf &
Welling, 2016), gated graph neural networks (Li et al., 2015), molecular ﬁngerprints (Duvenaud et al., 2015), interaction networks (Battaglia et al., 2016), molecular convolutions (Kearnes et al., 2016), among many others.
Networks using a global state (Battaglia et al., 2018) or looking at multiple hops per layer (Morris et al., 2019; Liao et al., 2019; Isuﬁ et al., 2020) are not directly GNN mp , but they can often be re-expressed as such.
The provided contributions are two-fold:
I.
What GNN mp can compute.
Section 3 derives sufﬁcient conditions such that a GNN mp can compute any function on its input that is computable by a Turing machine.
This result compliments recent universality results (Maron et al., 2019b; Keriven & Peyr ´ e, 2019) that considered approximation (rather than computability) over speciﬁc classes of functions (permutation invariant and equivariant) and particular architectures.
The claim follows in a straightforward manner by establishing the equivalence of GNN mp with LOCAL (Angluin, 1980; Linial, 1992; Naor & Stockmeyer, 1993), a classical model in distributed computing that is itself Turing universal.
In a nutshell, GNN mp are shown to be universal if four strong conditions are met: there are enough layers of sufﬁcient expressiveness and width, and nodes can uniquely distinguish each other.
Since Turing universality is a strictly stronger property than universal approximation, Chen et al.
(2019)’s argument further implies that a Turing universal GNN mp can solve the graph isomorphism problem (a sufﬁciently deep and wide network can compute the isomorphism class of its input).
II.
What GNN mp cannot compute (and thus learn).
Section 4 analyses the implications of restricting the depth d and width w of GNN mp that do not use a readout function.
Speciﬁcally, it is proven that GNN mp lose a signiﬁcant portion of their power when the product dw , which I call capacity , is restricted.
The analysis relies on a new technique that enables repurposing impossibility results from the context of distributed computing to the graph neural network setting.
Speciﬁcally, lower bounds for the following problems are presented: (i) detecting whether a graph contains a cycle of speciﬁc length; (ii) verifying whether a given subgraph is connected, contains a cycle, is a spanning tree, is bipartite, is a simple path, corresponds to a cut or Hamiltonial cycle; (iii) approximating the shortest path between two nodes, the minimum cut, and the minimum spanning tree; (iv) ﬁnding a maximum independent set, a minimum vertex cover, or a perfect coloring; (v) computing or approximating the diameter and girth.
The bounds are summarized in Table 1 and the problem deﬁnitions can be found in Appendix A. Section 5 presents some empirical evidence of the theory.
Though formulated in a graph-theoretic sense, the above problems are intimately linked to machine learning on graphs.
Detection, veriﬁcation, and computation problems are relevant to classiﬁcation: knowing what properties of a graph a GNN mp cannot see informs us also about which features of a graph can it extract.
Further, there have been attempts to use GNN mp to devise heuristics for graph-based optimization problems (Khalil et al., 2017; Battaglia et al., 2018;
Li et al., 2018; Joshi et al., 2019; Bianchi et al., 2019), such as the ones discussed above.
The presented results can then be taken as a worst-case analysis for the efﬁciency of GNN mp in such endeavors.
1. D ISCUSSION
The results of this paper carry several intriguing implications.
To start with, it is shown that the capacity dw of a GNN mp plays a signiﬁcant role in determining its power.
Solving many problems is shown to be impossible unless dw = ˜Ω( n δ ) , where δ ∈ [ 1 / 2 , 2] , n is the number of nodes of the graph, and f ( n ) = ˜Ω
( g ( n )) is interpreted as f ( n ) being, up to logarithmic factors, larger than g ( n ) as n grows.
This reveals a direct trade-off between the depth and width of a graph neural network.
Counter-intuitively, the dependence on n can be signiﬁcant even if the problem appears local in nature or one only looks for approximate solutions.
For example, detecting whether G contains a short cycle of odd length cannot be done unless dw = ˜Ω( n ) .
Approximation helps, but only to a limited extent; computing the graph diameter requires dw = ˜Ω( n ) and this reduces to dw = ˜Ω( √ n ) for any 3 / 2 -factor approximation.
Further, it is impossible to approximate within any constant factor the shortest path, the minimum cut, and the minimum spanning tree, all three of which have polynomial-time solutions, unless d √ w = ˜Ω( √ n ) .
Finally, for truly hard problems, the capacity may even need to be super-linear on n .
Speciﬁcally, it is shown that, even if the layers of the GNN mp are allowed to take exponential time, solving certain NP-hard problems necessitates d = ˜Ω( n 2 ) depth for any constant-width network.
Relation to previous impossibility results.
In contrast to universality (Maron et al., 2019b; Keriven & Peyr´e, 2019), the limitations of GNN mp have been much less studied.
In particular, the bounds presented here are the ﬁrst impossibility results that (i) explicitly connect GNN mp properties (depth and width) with graph properties and that (ii) go beyond isomorphism by addressing decision, optimization, and estimation graph problems.
Three main directions of related work can be distinguished.
First, Dehmamy et al.
(2019) bounded the ability of graph convolutional networks (i.e., GNN mp w/o messaging functions) to compute speciﬁc polynomial functions of the adjacency matrix, referred to as graph moments by the authors.
Second, Xu et al.
and Morris et al.
(2019) established the equivalence of anonymous GNN mp (those that do not rely on node identiﬁcation) to the Weisfeiler-Lehman (WL) graph isomorphism test.
The equivalence implies that anonymous networks are blind to the many graph properties that WL cannot see: e.g., any two regular graphs with the same number of nodes are identical from the perspective of the WL test (Arvind et al., 2015; Kiefer et al., 2015).
Third, in parallel to this work, Sato et al.
(2019) utilized a connection to LOCAL to derive impossibility results for the ability of a class of novel partially-labeled GNN mp to ﬁnd good approximations for three NP-hard optimization problems.
Almost all of the above negative results occur due to nodes being unable to distinguish between neighbors at multiple hops (see Appendix D).
With discriminative attributes GNN mp become signiﬁcantly more powerful (without necessarily sacriﬁcing permutation in/equivariance).
Still, as this work shows, even in this setting certain problems remain impossible when the depth and width of the GNN mp is restricted.
For instance, though cycles can be detected (something impossible in anonymous networks (Xu et al., 2018; Morris et al., 2019)), even for short cycles one now needs dw = ˜Ω( n ) .
Further, in contrast to Sato et al.
(2019), an approximation ratio below 2 for the minimum vertex cover is not impossible, but necessitates dw = ˜Ω( n 2 ) .
Limitations.
First, all lower bounds are of a worst-case nature: a problem is deemed impossible if there exists a graph for which it cannot be solved.
The discovery of non worst-case capacity bounds remains an open problem.
Second, rather than taking into account speciﬁc parametric functions, each layer is assumed to be sufﬁciently powerful to compute any function of its input.
This strong assumption does not signiﬁcantly limit the applicability of the results, simply because all lower bounds that hold with universal layers also apply to those that are limited computationally.
Lastly, it will be assumed that nodes can uniquely identify each other.
Node identiﬁcation is compatible with permutation invariance/equivariance as long as the network output is asked to be invariant to the particular way the ids have been assigned.
In the literature, one-hot encoded node ids are occasionally useful (Kipf & Welling, 2016; Berg et al., 2017).
When attempting to learn functions across multiple graphs, ids should be ideally substituted by sufﬁciently discriminative node attributes (attributes that uniquely identify each node within each receptive ﬁeld it belongs to can serve as ids).
Nevertheless, similar to the unbounded computation assumption, if a problem cannot be solved by a graph neural network in the studied setting, it also cannot be solved without identiﬁers and discriminative attributes.
Thus, the presented lower bounds also apply to partially and fully anonymous networks.
Notation.
I consider connected graphs G =
( V , E ) consisting of n = |V| nodes.
The edge going from v j to v i is written as e i ← j and it is asserted that if e i ← j ∈ E then also e j ← i ∈ E .
The neighborhood
N i of a node v i ∈ V consists of all nodes v j for which e i ← j ∈ E .
The degree of v i is denoted by deg i , ∆ is the maximum degree of all nodes and the graph diameter δ G is the length of the longest shortest path between any two nodes.
In the self-loop graph G ∗ =
( V , E ∗ ) , the neighborhood set of v i is given by N ∗ i =
N i ∪ v i .
T HE GRAPH NEURAL NETWORK COMPUTATIONAL MODEL
Graph neural networks are parametric and differentiable learning machines.
Their input is usually an attributed graph G a =
( G, ( a i : v i ∈ V ) , ( a i ← j : e i ← j ∈ E )) , where vectors a 1 , . .
. , a n encode relevant node attributes and a i ← j are edge attributes, e.g., encoding edge direction.
Model 1 formalizes the graph neural network operation by placing it in the message passing model (Gilmer et al., 2017).
The computation proceeds in layers, within which a message m i ← j is passed along each directed edge e i ← j ∈ E going from v j to v i and each node updates its internal representation by aggregating its state with the messages sent by its incoming neighbors v j ∈ N i .
The network output can be either of two things: a vector x i for each node v i or a single vector x G obtained by combining the representations of all nodes using a readout function.
Vectors x i / x G could be scalars (node/graph regression), binary variables (node/graph classiﬁcation) or multi-dimensional (node/graph embedding).
I use the symbols GNN nmp and GNN gmp to distinguish between models that return a vector per node and one per graph, respectively.
[Table]
Initialization: Set x (0) i
= a i for all v i ∈ V .
for layer (cid:96) = 1 , . . .
, d do for every edge e i ← j ∈ E ∗ (in parallel) do
[Equation]
for every node v i ∈ V (in parallel) do
[Equation]
Set x i = x ( d ) i .
return Either x i for every v i ∈ V ( GNN nmp ) or x G = R EAD ( { x i : v i ∈ V} )
( GNN gmp ).
The operation of a GNN mp is primarily determined by the messaging , update , and readout functions.
I assume that M SG (cid:96) and U P (cid:96) are general functions that act on intermediate node representations and node ids (the notation is overloaded such that v i refers to both the i -th node as well as its unique id).
As is common in the literature (Lu et al., 2017; Battaglia et al., 2018), these functions are instantiated by feed-forward neural networks.
Thus, by the universal approximation theorem and its variants (Cybenko, 1989; Hornik et al., 1989), they can approximate any general function that maps vectors onto vectors, given sufﬁcient depth and/or width.
Function R EAD is useful when one needs to retrieve a representation that is invariant of the number of nodes.
The function takes as an input a multiset , i.e., a set with possibly repeating elements, and returns a vector.
Commonly,
R EAD is chosen to be a dimension squashing operator, such as a sum or a histogram, followed by a feed-forward neural network (Xu et al., 2018; Seo et al., 2019).
Depth and width.
The depth d is equal to the number of layers of the network.
Larger depth means that each node has the opportunity to learn more about the rest of the graph (i.e., it has a larger receptive ﬁeld).
The width w of a GNN mp is equal to the largest dimension of state x ( l ) i over all layers l and nodes v i ∈ V .
Since nodes need to be able to store their own unique ids, in the following it is assumed that each variable manipulated by the network is represented in ﬁnite-precision using p = Θ(log n ) bits (though this is not strictly necessary for the analysis).
S UFFICIENT CONDITIONS FOR T URING UNIVERSALITY
This section studies what graph neural networks can compute.
It is demonstrated that, even without readout function, a network is computationally universal 1 if it has enough layers of sufﬁcient 1 It can compute anything that a Turing machine can compute when given an attributed graph as input.
width, nodes can uniquely distinguish each other, and the functions computed within each layer are sufﬁciently expressive.
The derivation entails establishing that GNN nmp is equivalent to LOCAL , a classical model used in the study the distributed algorithms that is itself Turing universal.
3. T HE LOCAL COMPUTATIONAL MODEL
A fundamental question in theoretical computer science is determining what can and cannot be computed efﬁciently by a distributed algorithm.
The LOCAL model, initially studied by Angluin (1980), Linial (1992), and Naor & Stockmeyer (1993), provides a common framework for analyzing the effect of local decision.
Akin to GNN mp , in LOCAL a graph plays a double role: it is both the input of the system and captures the network topology of the distributed system that solves the problem.
In this spirit, the nodes of the graph are here both the machines where computation takes place as well as the variables of the graph-theoretic problem we wish to solve—similarly, edges model communication links between machines as well as relations between nodes.
Each node v i ∈ V is given a problem-speciﬁc local input and has to produce a local output.
The input contains necessary the information that speciﬁes the problem instance.
All nodes execute the same algorithm, they are fault-free, and they are provided with unique identiﬁers.
A pseudo-code description is given in Model 2.
Variables s ( l ) i and s ( l ) i ← j refer respectively to the state of v i in round l and to the message sent by v j to v i in the same round.
Both are represented as strings.
The computation starts simultaneously and unfolds in synchronous rounds l = 1 , . . .
, d .
Three things can occur within each round: each node receives a string of unbounded size from its incoming neighbors; each node updates its internal state by performing some local computation; and each node sends a string to every one of its outgoing neighbors.
Functions A LG 1 l and A LG 2 l are algorithms computed locally by a Turing machine running on node v i .
Before any computation is done, each node v i is aware of its own attribute a i as well as of all edge attributes { a i ← j : v j ∈ N ∗ i } .
Computational
Initialization: Set s (0) i ← i = ( a i , v i ) and s (0)
i ← j = ( a j , v j ) for all e i ← j ∈ E .
for round (cid:96) = 1 , . . .
, d do
Receive s ( (cid:96) − 1) i ← j from v j ∈ N ∗ i , compute s ( (cid:96) )
i =
A LG 1 (cid:96) (cid:16)(cid:110)(cid:16) s ( (cid:96) − 1)
i ← j , a i ← j (cid:17) : v j ∈ N ∗
i (cid:111) , v i (cid:17) , and send s ( (cid:96) )
j ← i =
A LG 2 (cid:96) (cid:16) s ( (cid:96) )
i , v i (cid:17) to v j ∈ N ∗ i .
return s ( d ) i
In LOCAL , there are no restrictions on how much information a node can send at every round.
Asserting that each message s ( (cid:96) )
i ← j is at most b bits yields the CONGEST model (Peleg, 2000).
3. T URING UNIVERSALITY
The reader might have observed that LOCAL resembles closely GNN nmp in its structure, with only a few minor differences: ﬁrstly, whereas a LOCAL algorithm A may utilize messages in any way it chooses, a GNN nmp network N always sums received messages before any local computation.
The two models also differ in the arguments of the messaging function and the choice of information representation (string versus vector).
Yet, as the following theorem shows, the differences between
GNN nmp and LOCAL are inconsequential when seen from the perspective of their expressive power:
Theorem 3.1 (Equivalence) .
Let N (cid:96) ( G a ) be the binary representation of the state ( x ( (cid:96) ) 1 , . . .
, x ( (cid:96) ) n ) of a GNN n mp network N and A (cid:96) ( G a ) =
( s ( (cid:96) ) 1 , . . .
, s ( (cid:96) ) n ) that of a LOCAL algorithm A .
If M SG (cid:96) and
U P (cid:96) are Turing complete functions, then, for any algorithm
A there exists N (resp.
for any N there exists A ) such that
[Equation]
layer
[Equation]
G
[Equation]
G
[Equation]
where G a is the set of all attributed graphs.
This equivalence enables us to reason about the power of GNN nmp by building on the well-studied properties of LOCAL .
In particular, it is well known in distributed computing that, as long as the number of rounds d of a distributed algorithm is larger than the graph diameter δ G , every node in a LOCAL can effectively make decisions based on the entire graph (Linial, 1992).
Together with
Theorem 3.1, the above imply that, if computation and memory are not an issue, one may construct a
GNN nmp that effectively computes any computable function w.r.t.
its input.
Corollary 3.1.
GNN nmp can compute any Turing computable function over connected attributed graphs if the following conditions are jointly met: each node is uniquely identiﬁed;
M SG l and U P l are
Turing-complete for every layer (cid:96) ; the depth is at least d ≥ δ G layers; and the width is unbounded.
Why is this result relevant?
From a cursory review, it might seem that universality is an abstract result with little implication to machine learning architects.
After all, the utility of a learning machine is usually determined not with regards to its expressive power but with its ability to generalize to unseen examples.
Nevertheless, it can be argued that universality is an essential property of a good learning model.
This is for two main reasons: First, universality guarantees that the learner does not have blind-spots in its hypothesis space.
No matter how good the optimization algorithm is, how rich the dataset, and how overparameterized the network is, there will always be functions which a non universal learner cannot learn.
Second, a universality result provides a glimpse on how the size of the learner’s hypothesis space is affected by different design choices.
For instance, Corollary 3. puts forth four necessary conditions for universality: the GNN nmp should be sufﬁciently deep and wide, nodes should be able to uniquely and consistently identify each other, and ﬁnally, the functions utilized in each layer should be sufﬁciently complex.
The following section delves further into the importance of two of these universality conditions.
It will be shown that GNN nmp lose a signiﬁcant portion of their power when the depth and width conditions are relaxed.
The universality of GNN gmp .
Though a universality result could also be easily derived for networks with a readout function, the latter is not included as it deviates from how graph neural networks are meant to function: given a sufﬁciently powerful readout function, a GNN gmp of d = 1 depth and O (∆) width can be used to compute any Turing computable function.
The nodes should simply gather one hop information about their neighbors; the readout function can then reconstruct the problem input based on the collective knowledge and apply any computation needed.
I MPOSSIBILITY RESULTS AS A FUNCTION OF DEPTH AND WIDTH
This section analyzes the effect of depth and width in the expressive power of GNN nmp .
Speciﬁcally, I will consider problems that cannot be solved by a network of a given depth and width.
To be able to reason in terms of width, it will be useful to also enforce that the message size in
LOCAL at each round is at most b bits.
This model goes by the name CONGEST in the distributed computing literature (Peleg, 2000).
In addition, it will be assumed that nodes do not have access to a random generator.
With this in place, the following theorem shows us how to translate impossibility results from CONGEST to GNN n mp :
Theorem 4.1.
If a problem P cannot be solved in less than d rounds in CONGEST using messages of at most b bits, then P cannot be solved by a GNN nmp of width w ≤ ( b − log 2 n ) /p
=
O ( b/ log n ) and depth d .
The p = Θ(log n ) factor corresponds to the length of the binary representation of every variable—the precision needs to depend logarithmically on n for the node ids to be unique.
With this result in place, the following sections re-state several known lower bounds in terms of a GNN nmp ’s depth and width.
4. I MPOSSIBILITY RESULTS FOR DECISION PROBLEMS
I ﬁrst consider problems where one needs to decide whether a given graph satisﬁes a certain property (Feuilloley & Fraigniaud, 2016).
Concretely, given a decision problem P and a graph G , the
GNN nmp should output x i ∈ { true , false } for all v i ∈ V .
The network then accepts the premise if the logical conjunction of { x 1 , . .
, x n } is true and rejects it otherwise.
Such problems are intimately connected to graph classiﬁcation: classifying a graph entails identifying what constitutes a class from some training set and using said learned deﬁnition to decide the label of graphs sampled from the test set.
Instead, I will suppose that the class deﬁnition is available to the classiﬁer and I will focus on the corresponding decision problem.
As a consequence, every lower bound presented below for a decision problem must also be respected by a GNN nmp classiﬁer that attains zero error on the corresponding graph classiﬁcation problem.
Subgraph detection.
In this type of problems, the objective is to decide whether G contains a subgraph belonging to a given family.
I focus speciﬁcally on detecting whether G contains a cycle
C k , i.e., a simple undirected graph of k nodes each having exactly two neighbors.
As the following result shows, even with ids, cycle detection remains relatively hard:
Corollary 4.1 (Repurposed from (Drucker et al., 2014; Korhonen & Rybicki, 2018)) .
There exists graph G on which every GNN nmp of width w requires depth at least d = Ω( √ n/ ( w log n )) and d = Ω( n/ ( w log n )) to detect if G contains a cycle C k for even k ≥ 4 and odd k ≥ 5 , respectively.
Whereas an anonymous GNN mp cannot detect cycles (e.g., distinguish between two C 3 vs one
C 6 (Maron et al., 2019a)), it seems that with ids the product of depth and width should exhibit an (at least) linear dependence on n .
The intuition behind this bound can be found in Appendix C and empirical evidence in support of the theory are presented in Section 5.
Subgraph veriﬁcation.
Suppose that the network is given a subgraph H
=
( V H , E H ) of G in its input.
This could, for instance, be achieved by selecting the attributes of each node and edge to be a one-hot encoding of their membership on V H and E H , respectively.
The question considered is whether the neural network can verify a certain property of H .
More concretely, does a graph neural network exist that can successfully verify H as belonging to a speciﬁc family of graphs w.r.t.
G ?
In contrast to the standard decision paradigm, here every node should reach the same decision—either accepting or rejecting the hypothesis.
The following result is a direct consequence of the seminal work by Sarma et al.
Corollary 4.2 (Repurposed from (Sarma et al., 2012)) .
There exists a graph G on which every
GNN nmp of width w requires depth at least d = Ω( (cid:113) n w log 2 n + δ G ) to verify if some subgraph H of G is connected, contains a cycle, forms a spanning tree of G , is bipartite, is a cut of G , or is an s - t cut of G .
Furthermore, the depth should be at least d = Ω (cid:16)(cid:16)
[Equation]
to verify if H is a Hamiltonian cycle or a simple path.
Therefore, even if one knows where to look in G , verifying whether a given subgraph meets a given property can be non-trivial, and this holds for several standard graph-theoretic properties.
For instance, if we constrain ourselves to networks of constant width, detecting whether a subgraph is connected can, up to logarithmic factors, require Ω( √ n ) depth in the worst case.
4. I MPOSSIBILITY RESULTS FOR OPTIMIZATION PROBLEMS
I turn my attention to the problems involving the exact or approximate optimization of some graphtheoretic objective function.
From a machine learning perspective, the considered problems can be interpreted as node/edge classiﬁcation problems: each node/edge is tasked with deciding whether it belongs to the optimal set or not.
Take, for instance, the maximum independent set, where one needs to ﬁnd the largest cardinality node set, such that no two of them are adjacent.
Given only information identifying nodes, GNN nmp will be asked to classify each node as being part of the maximum independent set or not.
Polynomial-time problems.
Let me ﬁrst consider three problems that possess known polynomialtime solutions.
To make things easier for the GNN nmp , I relax the objective and ask for an approximate solution rather than optimal.
An algorithm (or neural network) is said to attain an α -approximation if it produces a feasible output whose utility is within a factor α of the optimal.
Let OPT be the utility of the optimal solution and ALG that of the α -approximation algorithm.
Depending on whether the problem entails minimization or maximization, the ratio ALG/OPT is at most α and at least 1 /α , respectively.
According to the following corollary, it is non-trivial to ﬁnd good approximate solutions:
Corollary 4.3 (Repurposed from (Sarma et al., 2012; Ghaffari & Kuhn, 2013)) .
There exists graphs
G and G (cid:48) of diameter δ G = Θ(log n ) and δ G (cid:48)
=
O (1) on which every GNN n mp of width w requires depth at least d = Ω( (cid:113) n w log 2 n ) and d (cid:48)
=
Ω(( n w log n ) γ ) with γ = 12 − 1 2( δ G (cid:48) − 1) , respectively, to approximate within any constant factor: the minimum cut problem, the shortest s - t path problem, or the minimum spanning tree problem.
Thus, even for simple problems (complexity-wise), in the worst case a constant width GNN nmp should be almost Ω( √ n ) deep even if the graph diameter is exponentially smaller than n .
NP-hard problems.
So what about truly hard problems?
Clearly, one cannot expect a GNN mp to solve an NP-hard time in polynomial time 2 .
However, it might be interesting as a thought experiment to consider a network whose layers take exponential time on the input size—e.g., by selecting the
M SG l and U P l functions to be feed-forward networks of exponential depth and width.
Could one ever expect such a GNN nmp to arrive at the optimal solution?
The following corollary provides necessary conditions for three well-known NP-hard problems:
Corollary 4.4 (Repurposed from (Censor-Hillel et al., 2017)) .
There exists a graph G on which every
GNN nmp of width w = O (1) requires depth at least d = Ω( n 2 / log 2 n ) to solve: the minimum vertex cover problem; the maximum independent set problem; the perfect coloring problem.
Thus, even if each layer is allowed to take exponential time, the depth should be quadratically larger than the graph diameter δ G
= O ( n ) to have a chance of ﬁnding the optimal solution.
Perhaps disappointingly, the above result suggests that it may not be always possible to exploit the distributed decision making performed by GNN mp architectures to ﬁnd solutions faster than classical (centralized) computational paradigms.
4. I MPOSSIBILITY RESULTS FOR ESTIMATION PROBLEMS
Finally, I will consider problems that involve the computation or estimation of some real function that takes as an input the graph and attributes.
The following corollary concerns the computation of two well-known graph invariants: the diameter δ G and the girth .
The latter is deﬁned as the length of the shortest cycle and is inﬁnity if the graph has no cycles.
Corollary 4.5 (Repurposed from (Frischknecht et al., 2012)) .
There exists a graph G on which every
GNN nmp of width w requires depth at least d = Ω( n/ ( w log n ) + δ G ) to compute the graph diameter δ G and d = Ω( √ n/ ( w log n ) + δ G ) to approximate the graph diameter and girth within a factor of/ 2 and 2 , respectively.
Term δ G appears in the lower bounds because both estimation problems require global information.
Further, approximating the diameter within a 3 / 2 factor seems to be simpler than computing it.
Yet, in both cases, one cannot achieve this using a GNN nmp whose capacity is constant.
As a ﬁnal remark, the graphs giving rise to the lower bounds of Corollary 4.5 have constant diameter and Θ( n 2 ) edges.
However, similar bounds can be derived also for graphs with O ( n log n ) edges (Abboud et al., 2016).
For the case of exact computation, the lower bound is explained in Appendix C.
E MPIRICAL EVIDENCE
This section aims to empirically test the connection between the capacity dw of a GNN mp , the number of nodes n of its input, and its ability to solve a given task.
In particular, I considered the problem of 4-cycle classiﬁcation and tasked the neural network with classifying graphs based on whether they contained a cycle of length four.
Following the lower bound construction described in Appendix A, I generated ﬁve distributions over graphs with n ∈ (8 , 16 , 24 , 32 , 44) nodes and an average diameter of (4 , 6 , 8 , 9 , 11) , respectively (this was achieved by setting p ∈ (6 , 8 , 10 , 12 , 14) , see Figure 3a).
For each such distribution, I generated a training and test set consisting respectively of 1000 andexamples.
Both sets were exactly balanced, i.e., any example graph from the training and test set had exactly 50% chance of containing a 4-cycle.
The experiment aimed to evaluate how able were GNN mp of different capacities to attain high accuracy on the test set.
To this end, I performed grid search over the hyperparameters w ∈ (2 , 10 , 20) and 2 Unless P=NP.
[Figure]
[Figure] [Figure]
[Figure]
t r a i n i n g a cc u r a c y
[Figure]
d ∈ (5 , 10 , 20 , 15) .
To reduce the dependence on the initial conditions and training length, for each hyperparameter combination, I trained 4 networks independently (using Adam and learning rate decay) for 4000 epochs.
The GNN mp chosen was that proposed by Xu et al.
, with the addition of residual connections—this network outperformed all others that I experimented with.
It is important to stress that empirically verifying lower bounds for neural networks is challenging, because it involves searching over the space of all possible networks in order to ﬁnd the ones that perform the best.
For this reason, an experiment such as the one described above cannot be used to verify 3 the tightness of the bounds: we can never be certain whether the results obtained are the best possible or whether the optimization resulted in a local minimum.
In that view, the following results should be interpreted in a qualitative sense.
The question that I will ask is: to which extend do the trends uncovered match those predicted by the theory?
More speciﬁcally, does the ability of a network to detect 4-cycles depend on the relation between dw and n ?
To answer this question, Figure 1 depicts the training and test accuracy as a function of the capacity dw for all the 240 networks trained (5 distributions × 3 widths × 4 depths × 4 iterations).
The accuracy of the best performing networks with the smallest capacity is shown in Figures 1c and 1d.
It is important to stress that, based on Weisfeiler-Lehman analyses, anonymous GNN mp cannot solve the considered task.
However, as it seen in the ﬁgures, the impossibility is annulled when using node ids 4 .
Indeed, even small neural networks could consistently classify all test examples perfectly (i.e., achieving 100% test accuracy) when n ≤ 16 .
Moreover, as the theoretical results predicted, there is a strong correlation between the test accuracy, dw and n (recall that Corollary 4.1 predicts dw = ˜Ω( √ n ) ).
Figure 1d shows that networks of the same capacity were consistently less accurate on the test set as n increased (even though the cycle length remained 4 in all experiments).
It is also
[Figure] [Figure]
t r a i n i n g a cc u r a c y 1000 2000 3000 4000 epoch 0.
t e s t a cc u r a c y anonymous degree random unique id unique id
[Figure]
4 depth / critical     w i d t h / c
r
i t i c a l 0.
[Figure]
striking to observe that even the most powerful networks considered could not achieve a test accuracy above 95% for n > 16 ; for n = 40 their best accuracy was below 80%.
Effect of anonymity.
Figure 2a plots example training and test curves for GNN mp trained with four different node attributes: no attributes (anonymous), a one-hot encoding of the node degrees (degree), a one-hot encoding of node ids (unique id), and a one-hot encoding of node ids that changed across graphs (random unique id).
It can be clearly observed that there is a direct correlation between accuracy and the type of attributes used.
With non- or partially-discriminative attributes, the network could not detect cycles even in the training set.
The cycle detection problem was solved exactly with unique ids, but when the latter were inconsistently assigned, the network could not learn to generalize.
Exchangeability of depth and width.
Figure 2b examines further the relationship between depth, width, and test accuracy.
This time, networks were separated depending on their depth and width normalized by the square root of the “ critical capacity ”.
For each n , the critical capacity is the minimum dw of a network that was able to solve the task on a graph of n nodes—here, solving amounts to a test accuracy above 95%.
In this way, a network of depth d and width w tested on n nodes corresponds to a point positioned at x = d/ √ critical , y = w/ √ critical and no network positioned at xy < 1 can solve the task (non-highlighted region in the bottom left corner).
As seen, there is a crisp phase transition between the regime of under- and super-critical capacity: almost every network meeting the condition dw ≥ critical was able to solve the task, irrespective of whether the depth or width was larger.
Note that, the exchangeability of depth and width cannot be guaranteed by the proposed theory which asserts that the condition dw = ˜Ω( √ n ) is necessary—but not sufﬁcient.
The empirical results however do agree with the hypothesis that, for 4-cycle classiﬁcation, depth and width are indeed exchangeable.
C ONCLUSION
This work studied the expressive power of graph neural networks falling within the message-passing framework.
Two results were derived.
First, sufﬁcient conditions were provided such that GNN mp can compute any function computable by a Turing machine with the same connected graph as input.
Second, it was discovered that the product of a GNN mp ’s depth and width plays a prominent role in determining whether the network can solve various graph-theoretic problems.
Speciﬁcally, it was shown that GNN nmp with dw = ˜Ω( n δ ) and δ ∈ [0 . 5 , 2] cannot solve a range of decision, optimization, and estimation problems involving graphs.
Overall, the proposed results demonstrate that the power of graph neural networks depends critically on their capacity and illustrate the importance of using discriminative node attributes.
Acknowledgements.
I thank the Swiss National Science Foundation for supporting this work in the context of the project “ Deep Learning for Graph-Structured Data ” (grant number PZ00P2 179981).
A G RAPH THEORY DEFINITIONS
The main graph-theoretic terms encountered in this work are:
• k -cycle detection : a k -cycle is a subraph of G consisting of k nodes, each with degree two.
The k -cycle detection problem entails determining if G contains a k -cycle.
• Hamiltonian cycle : a cycle of length n • (minimum) spanning tree : a spanning tree is a tree subgraph of G consisting of n nodes.
The minimum spanning tree problem entails ﬁnding the spanning tree of G of minimum weight (the weight of a tree is equal to the sum of its edge weights).
• (minimum) cut : a cut is a subgraph of G that when deleted leaves G disconnected.
The minimum cut problem entails ﬁnding the cut of minimum weight (the weight of a cut is equal to the sum of its edge weights).
• s - t cut : a subgraph of G such that removing all subgraph edges from G will leave the nodes s and t of G disconnected.
• (shortest) path : a simple path is subgraph of G where all nodes have degree 2 except from the two endpoint nodes whose degree is one.
The shortest path problem entails ﬁnding the simple path of minimum weight that connects two given nodes (the weight of a path is equal to the sum of its edge weights).
• (maximum) independent set : an independent set is a set of nodes in a graph no two of which are adjacent.
The maximum independent set problem entails ﬁnding the independent set of maximum cardinality.
• (minimum) vertex cover : a vertex cover of G is a set of nodes such that each edge of G is incident to at least one node in the set.
The minimum vertex cover problem entails ﬁnding the vertex cover of minimum cardinality.
• (perfect) coloring : a coloring of G is a labeling of the nodes with distinct colors such that no two adjacent nodes are colored using same color.
The perfect coloring problem entails ﬁnding a coloring with the smallest number of colors.
• diameter : the diameter δ G of G equals the length of the longest shortest path.
• girth : the girth of G equals the length of the shortest cycle.
It is inﬁnity if no cycles are present.
B D EFERRED P ROOFS B. P ROOF OF T HEOREM 3.
The claim is proven by expressing the state of node v i in the two models in the same form.
It is not difﬁcult to see that for each layer of the GNN nmp one has
[Equation]
, (from (Xu et al., 2018, Lemma 5)) where A GG (cid:96) is an aggregation function , i.e., a map from the set of multisets onto some vector space.
In the last step, I used a result of Xu et al.
Xu et al.
(2018) stating that each aggregation function can be decomposed as an element-wise function over each element of the multiset, followed by summation of all elements, and then a ﬁnal function.
Similarly, one may write:
[Equation]
deﬁnition)
[Equation]
with the last step following by restructuring the input and deﬁning A LG (cid:96) as the Turning machine that simulates the action of both A LG 2 (cid:96) and A LG 1 (cid:96) − 1 .
Since one may encode any vector into a string and vice versa, w.l.o.g. one may assume that the state of each node in LOCAL is encoded as a vector x i .
Then, to complete the proof, one still needs to demonstrate that the functions
[Equation]
( { ( x i , x j , v i , v j , a i ← j ) : v j ∈ N ∗ i } ) and
A LG ( { ( x j , v i , v j , a i ← j ) : v j ∈ N ∗ i } ) are equivalent (in the interest of brevity the layer/round indices have been dropped).
If this holds then each layer of GNN nmp is equivalent to a round of LOCAL and the claim follows.
I ﬁrst note that, since its input is a multiset, A LG l is also an aggregation function.
To demonstrate equivalence, one thus needs to show that, despite not having identical inputs, each of the two aggregation functions can be used to replace the other.
For the forward direction, it sufﬁces to show that for every aggregation function A GG there exists A LG with the same output.
Indeed, one may always construct A LG =
A GG ◦ g , where g takes as input { ( x j , v i , v j , a i ← j ) : v j ∈ N ∗ i } , identiﬁes x i (by searching for v i , v i ) and appends it to each element of the multiset yielding { ( x i , x j , v i , v j , a i ← j ) : v j ∈ N ∗ i } .
The backward direction can also be proven with an elementary construction: given A LG , one sets A GG =
A LG ◦ h , where h deletes x i from each element of the multiset.
B. P ROOF OF C OROLLARY 3.
In the LOCAL model the reasoning is elementary (Linial, 1992; Fraigniaud et al., 2013; Seidel, 2015): suppose that the graph is represented by a set of edges and further consider that A LG l amounts to a union operation.
Then in d = δ G rounds, the state of each node will contain the entire graph.
The function A LG 1 d can then be used to make the ﬁnal computation.
This argument also trivially holds for node/edge attributes.
The universality of GNN nmp then follows by the equivalence of LOCAL and
GNN nmp .
[Equation]
ROOF OF T HEOREM
[Equation]
First note that, since the GNN nmp and LOCAL models are equivalent, if no further memory/width restrictions are placed, an impossibility for one implies also an impossibility for the other.
It can also be seen in Theorem 3.1 that there is a one to one mapping between the internal state of each node at each level between the two models (i.e., variables x ( l ) i and s ( l ) i ).
As such, impossibility results that rely on restrictions w.r.t.
state size (in terms of bits) also transfer between the models.
To proceed, I demonstrate that a depth lower bound in the CONGEST model (i.e., in the LOCAL model with bounded message size ) also implies the existence of a depth lower bound in the LOCAL model with a bounded state size —with this result in place, the proof of the main claim follows directly.
As in the statement of the theorem, one starts by assuming that P cannot be solved in less than d rounds when messages are bounded to be at most b bits.
Then, for the sake of contradiction, it is supposed that there exists an algorithm A ∈ LOCAL that can solve P in less than d rounds with a state of at most c bits, but unbounded message size.
I argue that the existence of this algorithm also implies the existence of a second algorithm A (cid:48) whose messages are bounded by c
+ log 2 n : since each message s ( l ) j ← i is the output of a universal Turing machine A LG 2 l that takes as input the tuple ( s ( l ) i , v i ) , algorithm A (cid:48) directly sends the input and relies on the universality of A LG 1 l +1 to simulate the action of A LG 2 l .
The message size bound follows by adding the size c of the state with that of representing the node id ( log 2 n bits sufﬁce for unique node ids).
This line of reasoning leads to a contradiction when c ≤ b − log 2 n , as it implies that there exists an algorithm (namely A (cid:48) ) that can solve P in less than d rounds while using messages of at most b bits.
Hence, no algorithm whose state is less than b − log 2 n bits can solve P in LOCAL , and the width of GNN nmp has to be at least ( b − log 2 n ) /p .
C A N EXPLANATION OF THE LOWER BOUNDS FOR CYCLE DETECTION AND DIAMETER ESTIMATION
A common technique for obtaining lower bounds in the CONGEST model is by reduction to the set-disjointness problem in two-player communication complexity: Suppose that Alice and Bob are each given some secret string ( s a and s b ) of q bits.
The two players use the string to construct a set by selecting the elements from the base set { 1 , 2 , . . .
, q } for which the corresponding bit is one.
It is known that Alice and Bob cannot determine whether their sets are disjoint or not without exchanging at least Ω( q ) bits (Kalyanasundaram & Schintger, 1992; Chor & Goldreich, 1988).
The reduction involves constructing a graph that is partially known by each player.
Usually, Alice and Bob start knowing half of the graph (red and green induced subgraphs in Figure 3).
The players then use their secret string to control some aspect of their private topology (subgraphs annotated in dark gray).
Let the resulting graph be G ( s a , s b ) and denote by cut the number of edges connecting the subgraphs controlled by Alice and Bob.
To derive a lower bound for some problem P , one needs to prove that a solution for P in G ( s a , s b ) would also reveal whether the two sets are disjoint or not.
Since each player can exchange at most O ( b · cut ) bits per round, at least Ω( q/ ( b · cut )) rounds are needed in total in CONGEST .
By Theorem 4.1, one then obtains a d = Ω( q/ ( w log n · cut )) depth lower bound for GNN nmp .
The two examples in Figure 3 illustrate the graphs G ( s a , s b ) giving rise to the lower bounds for even k -cycle detection and diameter estimation.
To reduce occlusion, only a subset of the edges are shown.
p p
[Figure]
(a) In the construction of Korhonen
Rybicki
2018), each player starts from a complete bipartite graph of p = √ q nodes (nodes annotated in dark grey) with nodes numbered from 1 to 2 p .
The nodes with the same id are connected yielding a cut of size 2 p .
Each player then uses its secret (there are as many bits as
bipartite
edges
) to decide which of
the
bipartite
edges will
be deleted (
corresponding
to
zero
bits)
Remaining edges are substituted by a
path of length k/ 2 − 1 .
This
happens in a
way
that
ensures
that
G ( s a , s
b
contains a cycle of length k
(half
known
by Alice and half by
Bob
) if and
only
if the
two sets
are disjoint
: the cycle will pass through nodes t and p + t of each player to signify that the t -th bits of s a
and s b are both one.
It
can then be shown that n = Θ( p 2 ) from which
it
follows that: CONGEST requires at least d = Ω( q/ ( b · cut ))
=
Ω( n/ ( b · p ))
=
Ω( √ n/b ) bits to decide if there is a cycle of length k ; and GNN nmp has to have d = Ω( √ n/ ( w log n )) depth to do the same.
(b) In the construction of Abboud et al. (2016), each string consists of q = Ω( n ) bits.
The strings are used to encode the connectivity of subgraphs annotated in dark gray: an edge exists between the red nodes i and q if and only if the i -th bit of s a is one (and similarly for green).
Due to the graph construction, the cut between Alice and Bob has O (log q ) edges.
Moreover, About et al.
proved that
G
s a
s b
) has diameter
at
least ﬁve if and only if the sets deﬁned by s a and s b are disjoint.
This implies that d = Ω( n/ ( w log 2 n )) depth is necessary to compute the graph diameter in GNN nmp .
D T HE COST OF ANONYMITY
There is a striking difference between the power of anonymous networks and those in which nodes have the ability to uniquely identify each other, e.g., based on ids or discriminative attributes (see the survey by Suomela (2013)).
To illustrate this phenomenon, I consider a thought experiment where a node is tasked with reconstructing the graph topology in the LOCAL model.
In the left, Figure 4 depicts the red node’s knowledge after two rounds (equivalent to a GNN nmp having d = 2 ) when each node has a unique identiﬁer (color).
At the end of the ﬁrst round, each node is aware of its neighbors and after two rounds the entire graph has been successfully reconstructed in red’s memory.
In the right subﬁgure, nodes do not possess ids (as in the analysis of (Xu et al., 2018; Morris et al., 2019)) and thus cannot distinguish which of their neighbors are themselves adjacent.
As such, the red node cannot tell whether the graph contains cycles: after two rounds there are at least two plausible topologies that could explain its observations.
[Figure]