U NDERSTANDING OVER - SQUASHING AND BOTTLENECKS ON GRAPHS VIA CURVATURE
A BSTRACT
Most graph neural networks (GNNs) use the message passing paradigm, in which node features are propagated on the input graph.
Recent works pointed to the distortion of information ﬂowing from distant nodes as a factor limiting the efﬁciency of message passing for tasks relying on long-distance interactions.
This phenomenon, referred to as ‘over-squashing’, has been heuristically attributed to graph bottlenecks where the number of k -hop neighbors grows rapidly with k .
We provide a precise description of the over-squashing phenomenon in GNNs and analyze how it arises from bottlenecks in the graph.
For this purpose, we introduce a new edge-based combinatorial curvature and prove that negatively curved edges are responsible for the over-squashing issue.
We also propose and experimentally test a curvature-based graph rewiring method to alleviate the over-squashing.
I NTRODUCTION
In the past few years, deep learning on graphs and in particular graph neural networks (GNNs) (Sperduti, 1994; Goller &
Kuchler, 1996; Sperduti & Starita, 1997;
Frasconi et al., 1998; Gori et al., 2005;
Scarselli et al., 2008; Bruna et al., 2014;
Defferrard et al., 2016; Kipf & Welling, 2017; Gilmer et al., 2017) have become very popular in the machine learning community due to their ability to deal with broad classes of systems of relations and interactions, ranging from social networks to particle physics (Shlomi et al., 2021).
The vast majority of GNNs follow the message passing paradigm (Gilmer et al., 2017), using learnable non-linear functions to diffuse information on the graph.
Multiple popular GNN architectures such as GCN (Kipf & Welling, 2017) and GAT (Veliˇckovi´c et al., 2018) can be posed as particular ﬂavors of this scheme and considered instances of a more general framework of geometric deep learning (Bronstein et al., 2021).
Some of the drawbacks of the message passing paradigm have now been identiﬁed and formalized, including the limits of expressive power (Xu et al., 2019; Morris et al., 2019; Maron et al., 2019) and the problem of over-smoothing (NT & Maehara, 2019; Oono & Suzuki, 2020).
On the other hand, much less is known about the phenomenon of over-squashing , consisting in the distortion of messages being propagated from distant nodes.
Alon & Yahav (2021) proposed rewiring the graph as a way of reducing the bottleneck , deﬁned as those topological properties in the graph leading to over-squashing.
This approach is in line with multiple other results e.g. using connectivity diffusion (Klicpera et al., 2019) as a preprocessing step to facilitate graph learning.
Yet, the exact understanding of the over-squashing and how it originates from the bottlenecks in the topology of the underlying graph are still elusive.
Consequently, there is currently no consensus on the right method (either based on graph rewiring or not) to address the bottleneck and hence alleviate the over-squashing.
In this paper, we address these questions using tools from differential geometry, which traditionally is concerned with the study of manifolds.
It offers an appealing framework to study the properties of graphs, in particular arguing that graphs, like manifolds, exhibit curvature that makes them more suitable to be realized in spaces with hyperbolic geometry (Liu et al., 2019; Chami et al., 2019;
Boguna et al., 2021).
One notion of curvature that has received attention for graph learning is Ricci curvature (Hamilton, 1988), also known in geometry for its use in Ricci ﬂow and the subsequent proof of the Poincaré conjecture (Perelman, 2003).
Certain graph analogues of the Ricci curvature (Forman, 2003; Ollivier, 2009; Sreejith et al., 2016) were used in Ni et al.
(2018) for a discrete version of Ricci ﬂow to construct a metric between graphs.
Graph Ricci ﬂow was also used in Ni et al.
for community detection.
Both of these methods use the edge weights as a substitute for the metric of a manifold, and do not change the topological structure of the graph.
Contributions and Outline.
This paper, to our knowledge, is the ﬁrst theoretical study of the bottleneck and over-squashing phenomena in message passing neural networks from a geometric perspective.
In Section 2, we propose the Jacobian of node representations as a formal way of measuring the over-squashing and we show that the graph topology may compromise message propagation in graph neural networks by creating a bottleneck.
In Section 3, we investigate how such a bottleneck is induced which leads to the over-squashing of information.
To this aim, we introduce a new combinatorial edge-based curvature called Balanced Forman curvature that constitutes a sharp lower bound to the standard Ollivier curvature on graphs, and prove that negatively curved edges are responsible for the formation of bottlenecks (and hence for over-squashing).
In Section 4, we present a new curvature-based method for graph rewiring called Stochastic Discrete Ricci Flow.
According to the theoretical results in Section 3, this rewiring method is suited to address the graph bottleneck and hence alleviate the over-squashing by surgically targeting the edges responsible for the issue.
By contrast, we rigorously show that a recently introduced diffusion-based rewiring scheme might generally fail to reduce the bottleneck.
Finally, in Section 5, we compare different rewiring strategies experimentally on several standard graph learning datasets.
A NALYSIS OF THE OVER - SQUASHING PHENOMENON 2. P RELIMINARIES
Let G =
( V, E ) be a simple, undirected, and connected graph, where ( i, j ) ∈
E iff i ∼ j .
We focus on the unweighted case, although the theory extends to the weighted setting as well.
We denote the adjacency matrix by A and let ˜ A
=
A
+ I be the adjacency matrix augmented with self-loops.
Similarly we let ˜ D
= D
+ I , with D the diagonal degree matrix, and let ˆ A =
˜ D − 12 ˜
A ˜ D − 12 be the normalized augmented adjacency matrix (self-loops are commonly included in GNN architecture, and in Section 2 we formally explain why GNNs are expected to propagate information more reliably when self-loops are taken into account).
Given i ∈ V , we denote its degree by d i and let
[Equation]
where d G is the standard shortest-path distance on the graph and r ∈ N .
The set B r ( i ) represents the receptive ﬁeld of an r -layer message passing neural network at node i .
Message passing neural networks (MPNNs).
Assume that the graph G is equipped with node features X ∈ R n × p 0 where x i ∈ R p 0 is the feature vector at node i = 1 , . . . , n = | V | .
We denote by h ( (cid:96) )
i ∈
R p (cid:96) the representation of node i at layer (cid:96) ≥ 0 , with h (0) i = x i .
Given a family of message functions ψ (cid:96) : R p (cid:96) × R p (cid:96) → R p (cid:48) (cid:96) and update functions φ (cid:96) : R p (cid:96) × R p (cid:48) (cid:96) → R p (cid:96) +1 , we can write the ( (cid:96) + 1) -st layer output of a generic MPNN as follows (Gilmer et al., 2017):
[Equation]
Here we have used the augmented normalized adjacency matrix to propagate messages from each node to its neighbors, which simply leads to a degree normalization of the message functions ψ (cid:96) .
To avoid heavy notations the node features and representations are assumed to be scalar from now on; these assumptions simplify the discussion and the vector case leads to analogous results.
2. T HE OVER - SQUASHING PROBLEM
Multiple recent papers observed that MPNNs tend to perform poorly in situations when the learned task requires long-range dependencies and at the same time the structure of the graph results in exponentially many long-range neighboring nodes.
We say that a graph learning problem has long-range dependencies when the output of a MPNN depends on representations of distant nodes interacting with each other.
If long-range dependencies are present, messages coming from nonadjacent nodes need to be propagated across the network without being too distorted.
In many cases however (e.g. in ‘small-world’ graphs such as social networks), the size of the receptive ﬁeld B r ( i ) grows exponentially with r .
If this occurs, representations of exponentially many neighboring nodes need to be compressed into ﬁxed-size vectors to propagate messages to node i , causing a phenomenon referred to as over-squashing of information (Alon & Yahav, 2021).
In line with Alon & Yahav (2021), we refer to those structural properties of the graph that lead to over-squashing as a bottleneck 1 .
Sensitivity analysis.
The hidden feature h ( (cid:96) ) i
= h ( (cid:96) )
i ( x 1 , . . .
, x n ) computed by an MPNN with (cid:96) layers as in equation 1 is a differentiable function of the input node features { x 1 , . . .
, x n } as long as the update and message functions φ (cid:96) and ψ (cid:96) are differentiable.
The over-squashing of information can then be understood in terms of one node representation h ( (cid:96) )
i failing to be affected by some input feature x s of node s at distance r from node i .
Hence, we propose the Jacobian ∂h ( r ) i /∂x s as an explicit and formal way of assessing the over-squashing effect 2 .
Lemma 1.
Assume an MPNN as in equation 1.
Let i, s ∈ V with
s ∈ S r +
( i ) .
If |∇ φ (cid:96) | ≤ α and |∇ ψ (cid:96) | ≤ β for 0 ≤ (cid:96) ≤ r , then
[Equation]
Lemma 1 states that if φ (cid:96) and ψ (cid:96) have bounded derivatives, then the propagation of messages is controlled by a suitable power of ˆ A .
For example, if d G ( i, s ) =
r + 1 and the sub-graph induced on
B r +
( i ) is a binary tree, then ( ˆ A r +1 )
is = 2 − 1 3 − r , which gives an exponential decay of the node dependence on input features at distance r , as also heuristically argued by Alon & Yahav (2021).
The sensitivity analysis in Lemma 1 relates the over-squashing – as measured by the Jacobian of the node representations – to the graph topology via powers of the augmented normalized adjacency matrix.
In the next section we explore this connection further by analyzing which local properties of the graph structure affect the right hand side in equation 2, hence causing the bottleneck.
We will address this problem by introducing a new combinatorial notion of edge-based curvature and showing that negatively curved edges are those responsible for the over-squashing phenomenon .
G RAPH CURVATURE AND BOTTLENECK
A natural object in Riemannian geometry is the Ricci curvature , a bilinear form determining the geodesic dispersion, i.e. whether geodesics starting at nearby points with ‘same’ velocity remain parallel ( Euclidean space ), converge ( spherical space ), or diverge ( hyperbolic space ).
To motivate the introduction of a Ricci curvature for graphs, we focus on these three cases.
Consider two nodes i ∼ j and two edges starting at i and j respectively.
In a discrete spherical geometry (Figure 2a), the edges would meet at k to form a triangle (complete graph).
In a discrete Euclidean geometry (Figure 2b), the edges would stay parallel and form a 4-cycle based at i ∼ j (orthogonal grid).
Finally, in a discrete hyperbolic geometry (Figure 2c), the mutual distance of the edge endpoints would have grown compared to that of i and j (tree).
Therefore, a Ricci curvature for graphs should provide us with more sophisticated tools than the degree to analyze the neighborhood of an edge.
[Figure]
Curvatures on graphs.
The main examples of edge-based curvature are the Forman curvature F ( i, j ) (Forman, 2003) and the Ollivier curvature κ ( i, j ) in Ollivier (2007; 2009) (see Appendix).
While F ( i, j ) is given in terms of combinatorial quantities (Sreejith et al., 2016), results are scarce and the deﬁnition is biased towards negative curvature.
The theory on κ ( i, j ) instead is richer (Lin et al., 2011; Münch, 2019) but its formulation makes it hard to control local quantities.
Balanced Forman curvature.
We propose a new curvature to address the shortcomings of the existing candidates.
We use the following deﬁnitions to describe the neighborhood of an edge i ∼ j
and we refer to the Appendix for a more complete discussion:
[Equation] [Equation] S
[Equation] i ) ∩ S 1 ( j ) are the triangles based at i ∼ j .
[Equation] ) \ S 1 ( j ) , k (cid:54) =
j [Equation]
∃ w [Equation] S
[Equation] [Equation] S [Equation]
)) \ S 1 ( i ) } are the neighbors of i forming a 4-cycle based at the edge i ∼ j
without
diagonals inside. (iii) γ max ( i, j ) is the maximal number of 4 − cycles based at i ∼ j traversing a common node (see Deﬁnition 4)
In line with the discussion about geodesic dispersion, one expects (cid:93) ∆ to be related to positive curvature (complete graph), (cid:93) i (cid:3) to zero curvature (grid), and the remaining outgoing edges to negative curvature (tree).
Our new curvature formulation reﬂects such an intuition and recovers the expected results in the classical cases.
In the example in Figure 3 we have (cid:93) 0 (cid:3) (0 , 1) = { 2 , 3 } while (cid:93) 1 (cid:3) (0 , 1) =
{ 5 } , both without 4 , 6 because of the triangle 1-6-0.
The degeneracy factor γ max (0 , 1) = 2 , as there exist two 4-cycles passing through node 5 .
Deﬁnition 1 ( Balanced Forman curvature).
For any edge i ∼ j in a simple, unweighted graph G , we let Ric ( i, j ) be zero if min { d i , d j } =
and otherwise
[Equation]
where the last term is set to be zero if | (cid:93)
i (cid:3) | (and hence | (cid:93) j (cid:3) | ) is zero.
In particular Ric ( i, j ) > − 2 .
The curvature is negative when i ∼ j behaves as a bridge between S 1 ( i ) and S 1 ( j ) , while it is positive when S 1 ( i ) and S 1 ( j ) stay connected after removing i ∼ j .
We refer to Ric as Balanced
Forman curvature .
We can relate the Balanced Forman curvature to the Jacobian of hidden features, while also extending many results valid for the Ollivier curvature κ ( i, j ) thanks to our next theorem.
Theorem 2.
Given an unweighted graph G , for any edge i ∼ j we have κ ( i, j ) ≥
Ric ( i, j ) .
Graph G
[Table]
Theorem 2 generalizes Jost & Liu (2014, Theorem 3) (see Appendix). We also note that the computational complexity for κ scales as O ( | E | d 3 max ) , while for our Ric we have O ( | E | d 2 max ) , with d max the maximal degree.
From Theorem 2 and Paeng (2012), we ﬁnd:
Corollary 3.
If Ric ( i, j ) ≥ k > 0 for any edge i ∼ j , then there exists a polynomial P such that
[Equation]
Therefore, when the curvature is positive everywhere, the bottleneck effect should not play a crucial role as the receptive ﬁeld of each node will be polynomial in the hop-distance.
This limit case shows how the curvature determines whether a learning task on a graph would suffer from over-squashing.
Curvature and over-squashing.
Thanks to our new combinatorial curvature and the sensitivity analysis in Lemma 1, we are able to relate local curvature properties to the Jacobian of the node representations.
This leads to one of the main results of this paper: negatively curved edges are those causing the graph bottleneck and thus leading to the over-squashing phenomenon:
Theorem 4.
Consider a MPNN as in equation 1.
Let i ∼ j with d i ≤ d j and assume that:
(i) |∇ φ (cid:96) | ≤ α and |∇ ψ (cid:96)
| ≤ β
for each 0 ≤ (cid:96) ≤ L − 1 , with L ≥ 2 the depth of the MPNN.
(ii) There exists δ s
t. 0 < δ < (max { d i , d j } ) − 12 , δ < γ − 1 max , and Ric ( i, j ) ≤ − 2 + δ .
Then there exists Q j ⊂ S 2 ( i ) satisfying | Q j | > δ − 1 and for 0 ≤ (cid:96)
≤ L − 2 we have
[Equation]
Condition (i) is always satisﬁed and allows us to control the message passing functions.
The requirement (ii) instead means that the curvature of ( i, j ) is negative enough when compared to the degrees of i and j (recall that
Ric ( i, j ) > − 2 ).
The further condition on γ max is to avoid pathological cases where we have a large number of degenerate 4-cycles passing through the same three nodes.
To understand the conclusions in Theorem 4, let us ﬁx (cid:96)
= 0 .
The equation 4 shows that negatively curved edges are the ones causing bottlenecks, interpreted as how the graph topology prevents a representation h (2) k to be affected by non -adjacent features h (0)
i =
x i .
Theorem 4 implies that if we have a negatively curved edge as in (ii), then there exist a large number of nodes k such that GNNson average - struggle to propagate messages from i to k in two layers despite these nodes k being at distance 2 from i .
In this case the over-squashing occurs as measured by the Jacobian in equationand hence the propagation of information suffers.
If the task at hand has long-range dependencies, then the over-squashing caused by the negatively curved edges may compromise the performance.
Bottleneck via Cheeger constant.
We now relate the previous discussion about bottlenecks and curvature to spectral properties of the graph.
In particular, since the spectral gap of a graph can be interpreted as a topological obstruction to the graph being partitioned into two communities, we argue below that this quantity is related to the graph bottleneck and should hence be controllable by the curvature.
We start with an intuitive explanation: suppose we are given a graph G with two communities separated by few edges.
In this case, we see that the graph can be easily disconnected.
This property is encoded in the classical notion of the Cheeger constant (Chung & Graham, 1997)
[Equation]
where ∂S = { ( i, j ) : i ∈ S, j ∈ V \ S } and vol ( S ) = (cid:80) i ∈ S
d i .
The main result about the Cheeger constant is the Cheeger inequality (Cheeger, 2015; Chung & Graham, 1997):
[Equation]
where λ 1 is the ﬁrst non-zero eigenvalue of the normalized graph Laplacian, often referred to as the spectral gap .
A graph with two tightly connected communities ( S and V \ S := ¯ S ) and few inter-community edges has a small Cheeger constant h G .
For nodes in different communities to interact with each other, all messages need to go through the same few bridges hence leading to the over-squashing of information (a similar intuition was explored in Alon & Yahav (2021)).
Therefore, h G can be interpreted as a rough measure of graph ‘bottleneckedness’, in the sense that the smaller its value, the more likely the over-squashing is to occur across inter-community edges.
Since Theoremimplies that negatively curved edges induce the bottleneck, we expect a relationship between h G and the curvature of the graph.
The next proposition follows from Theorem 2 and Lin et al.
Proposition 5.
If Ric ( i, j ) ≥ k > 0 for all i ∼ j , then λ 1 / 2 ≥ h G ≥ k 2 .
Therefore, a positive lower bound on the curvature gives us a control on h G and hence on the spectral gap of the graph.
In the next section, we show that diffusion-based graph-rewiring methods might fail to signiﬁcantly alter h G and hence correct the graph bottleneck potentially induced by inter-community edges.
This will lead us to propose an alternative curvature-based graph rewiring.
C URVATURE - BASED REWIRING METHODS
The traditional paradigm of message passing graph neural networks assumes that messages are propagated on the input graph (Gilmer et al., 2017).
More recently, there is a trend to decouple the input graph from the graph used for information propagation.
This can take the form of graph subsampling or resampling to deal with scalability (Hamilton et al., 2017) or topological noise (Zhang et al., 2019), using larger motif-based (Monti et al., 2018) or multi-hop ﬁlters (Rossi et al., 2020), or changing the graph either as a preprocessing step (Klicpera et al., 2019; Alon & Yahav, 2021) or adaptively for the downstream task (Wang et al., 2019; Kazi et al., 2020).
Such methods are often generically referred to as graph rewiring .
In the context of this paper, we assume that graph rewiring attempts to produce a new graph
G (cid:48)
=
( V, E (cid:48) ) with a different edge structure that reduces the bottleneck and hence potentially alleviates the over-squashing of information.
We propose a method that leverages the graph curvature to guide the rewiring steps in a surgical way by modifying the negatively-curved edges, so to decrease the bottleneck without signiﬁcantly compromising the statistical properties of the input graph.
We also rigorously show that a random-walk based rewiring method might generally fail to obtain an edge set E (cid:48) with a signiﬁcant improvement in its bottleneckedness as measured by the Cheeger constant.
Curvature-based graph rewiring.
Since according to Theorem 4 negatively curved edges induce a bottleneck and are hence responsible for over-squashing, a curvature-based rewiring method should attempt to alleviate a graph’s strongly-negatively curved edges.
To this end we implement a simple rewiring method called Stochastic Discrete Ricci Flow (SDRF), described in Algorithm 1.
upper-bound C +
Repeat
1) For edge i ∼ j with minimal Ricci curvature Ric ( i, j ) :
Calculate vector x where x kl =
Ric kl ( i, j ) − Ric
( i, j ) , the improvement to Ric ( i, j ) from adding edge k ∼ l where k ∈ B 1 ( i ) , l ∈ B 1 ( j ) ; Sample index k, l with probability softmax ( τ x ) kl and add edge k ∼ l to G . 2) Remove edge i ∼ j with maximal Ricci curvature Ric ( i, j ) if Ric ( i, j ) >
C + .
Until convergence, or max iterations reached;
At each iteration this preprocessing step adds an edge to ‘support’ the graph’s most negatively curved edge, and then removes the most positively curved edge.
The requirement on the added edge k ∼ l that k ∈ B 1 ( i ) and l ∈ B 1 ( j ) ensures that we’re adding either an extra 3- or 4-cycle around the negative edge i ∼ j so that this is a local modiﬁcation.
The graph edit distance between the original and preprocessed graph is bounded above by 2 × the max number of iterations. The temperature τ determines how stochastic the edge addition is, with τ = ∞ being fully deterministic (the best edge is always added).
At each step we remove the edge with most positive curvature to balance the distributions of curvature and node degrees.
We use Balanced Forman curvature as in equation 3 for Ric ( i, j ) .
C
+ can be chosen to stop the method skewing the curvature distribution negative, including C + = ∞ to not remove any edges.
The method is inspired by the continuous (backwards)
Ricci ﬂow with the aim of homogenizing edge curvatures.
This is different from more direct extensions of Ricci ﬂow on graphs where it becomes increasingly expensive to propagate messages across negatively curved edges (as in other applications such as Ni et al.
An example alongside its continuous analogue can be seen in Figure 1.
Can random-walk based rewiring address bottlenecks?
A good way of understanding the effectiveness of SDRF in reducing the graph bottleneck is through comparison with random-walk based rewiring strategies.
Recall that, as argued in Section 3, the Cheeger constant h G of a graph constitutes a rough measure of its bottleneckedness as induced by the inter-community edges (a small h G is indicative of a bottleneck).
Suppose we are given a graph G with a small h G and wish to rewire it into a graph G (cid:48) with a signiﬁcantly improved Cheeger constant in order to reduce the inter-community bottleneck.
A random-walk based rewiring method such as DIGL (Klicpera et al., 2019) acts by smoothing out the graph adjacency and hence tends to promote connections among nodes at short diffusion distance (Coifman & Lafon, 2006).
Accordingly, such a rewiring method might fail to correct structural features like the bottleneck, which is instead more prominent for nodes that are at long diffusion distance.
3 To emphasize this point, we consider a classic example: given α ∈ (0 , 1) , the Personalized Page Rank (PPR) matrix is deﬁned by (Brin & Page, 1998) as
[Equation]
Assume that we rewire the graph using R α as in Klicpera et al. (2019) with the PPR kernel, meaning that we replace the given adjacency A with R α .
Since R α is stochastic, the new Cheeger constant of the rewired graph can be computed as
[Equation]
By applying (Chung, 2007, Lemma 5), we show that we cannot improve the Cheeger constant (and hence the bottleneck) arbitrarily well (in contrast to a curvature-based approach).
We refer to
Proposition 17 and Remark 18 in Appendix E for results that are more tailored to the actual strategy adopted in Klicpera et al.
(2019) where we also take into account the effect of the sparsiﬁcation.
Theorem 6.
Let S ⊂ V with vol ( S ) ≤ vol ( G ) / 2 .
Then h S,α ≤ (cid:0) 1 − αα (cid:1)
d avg ( S ) d min ( S ) h S , where d avg ( S ) and d min ( S ) are the average and minimum degree on S , respectively.
The property that the new Cheeger constant is directly controlled by the old one stems from the fact that a random-walk approach like in Klicpera et al.
(2019) is meant to act more relevantly on intra-community edges rather than inter-community edges because it prioritizes short diffusion distance nodes.
This is also why this method performs well on high-homophily datasets, as discussed below.
In particular, for a ﬁxed α ∈ (0 , 1) , the bound in Theorem 6 can be very small.
As a speciﬁc example, consider two complete graphs K n joined by one bridge.
Then h G = ( n ( n − 1) + 1) − 1 , which means that the bound on the right hand side is O ( n − 2 ) .
Theorem 6 implies that a diffusion approach such as DIGL might fail to yield a new edge set E (cid:48) with a sufﬁciently improved bottleneck.
By contrast, from Theorem 4 and Proposition 5, we deduce that a curvature-based rewiring method such as SDRF properly addresses the edges that cause the bottleneck.
Graph structure preservation.
Although a graph-rewiring approach aims at providing a new edge set E (cid:48) potentially more beneﬁcial for the given learning task, it is still desirable to control how far
E (cid:48) is from E .
In this regard, we note that a curvature-based rewiring is surgical in nature and hence more likely to preserve the structure of the input graph better than a random-walk based approach.
Consider, for example, that we are given ρ > 0 and wish to rewire the graph such that the new edge set E (cid:48) is within graph-edit distance ρ from the original E .
Theorem 4 tells us how to do the rewiring under such constraints in order to best address the over-squashing: the topological modiﬁcations need to be localized around the most negatively-curved edges .
We can do this with SDRF, with the maximum number of iterations set to ρ/ 2 .
Secondly, we also point out that Ric ( i, j ) <
− 2 + δ implies that min { d i , d j } > 2 /δ .
Therefore, if we mostly modify the edge set at those nodes i, j joined by an edge with large negative curvature, then we are perturbing nodes with high degree where such a change is relatively insigniﬁcant, and thus overall statistical properties of the rewired graph such as degree distribution are likely to be better preserved.
Moreover, graph convolutional networks tend to be more stable to perturbations of high degree nodes (Zügner et al., 2020; Kenlay et al., 2021), making curvature-based rewiring more suitable for the downstream learning tasks with popular GNN architectures.
Homophily and bottleneck.
As a ﬁnal remark, note that the graph rewiring techniques considered in this paper (both DIGL and SDRF) are based purely on the topological structure of the graph and completely agnostic to the node features and to whether the dataset is homophilic (adjacent nodes have same labels) or heterophilic .
Nonetheless, the different nature of these rewiring methods allows us to draw a few broad conclusions about their suitability in each of these settings.
A random-walk
[Table]
approach such as DIGL tends to improve the connectivity among nodes that are at short diffusion distance; since for a high-homophily dataset these nodes often share the same label, a rewiring method like DIGL is likely to act as graph denoising and yield improved performance.
On the other hand, for datasets with low homophily, nodes at short diffusion distance are more likely to belong to different label classes, meaning that a diffusion-based rewiring might inject noise and hence compromise performance as also noted in Klicpera et al.
Conversely, on a low-homophily dataset, a curvature-based approach as SDRF modiﬁes the edge set mainly around the most negatively curved edges, meaning that it decreases the bottleneck without signiﬁcantly increasing the connectivity among nodes with different labels.
In fact, long-range dependencies are often more relevant in low-homophily settings, where nodes sharing the same labels are in general not neighbors.
This observation is largely conﬁrmed by experimental results reported in the next section.
E XPERIMENTAL R ESULTS
Experiment setup.
To demonstrate the theoretical results in this paper we ran a suite of semisupervised node classiﬁcation tasks comparing our curvature-based rewiring method SDRF to DIGL from Klicpera et al.
(GDC with the PPR kernel) and the +FA method from Alon & Yahav (2021), where the last layer of the GNN is made fully connected.
We evaluate the methods on nine datasets: Cornell, Texas and Wisconsin from the WebKB dataset 4 ; Chameleon and Squirrel (Rozemberczki et al., 2021) along with Actor (Tang et al., 2009); and Cora (McCallum et al., 2000),
Citeseer (Sen et al., 2008) and Pubmed (Namata et al., 2012).
Statistics for these datasets can be found in Appendix F.1.
Our base model is a GCN (Kipf & Welling, 2017).
Following Shchur et al.
(2018) and Klicpera et al.
we optimized hyperparameters for all dataset-preprocessing combinations separately by random search over 100 data splits.
Results are reported as average accuracies on a test set used once with 95% conﬁdence intervals calculated by bootstrapping.
We compared the performance on graphs with no preprocessing, making the graph undirected, +FA, DIGL, SDRF, and the given combinations.
For DIGL + Undirected we symmetrized the diffusion matrix as in Klicpera et al.
(2019), and for SDRF + Undirected we made the graph undirected before applying SDRF.
For more details on the experiments and datasets see Appendix F, and for the hyperparameters used for each model and preprocessing see Appendix F.4.
Node classiﬁcation results.
Table 2 shows the results of the experiments.
As well as reporting results we give a measure of homophily H ( G ) proposed by Pei et al.
(2019) (restated in Appendix F, equation 21), by which we can see our experiment set is diverse with respect to homophily.
We see that SDRF improves upon the baseline in all cases, and that the largest improvements are seen on the low-homophily datasets.
We also see that SDRF matches or outperforms DIGL and +FA on most datasets, supporting our argument that curvature-based rewiring is a viable candidate for improving
GNN performance.
Graph topology change.
Furthermore, SDRF preserves the graph topology to a far greater extent than DIGL due to its surgical nature.
Table 3 shows the number of edges added / removed by the two preprocessings on each dataset as a percentage of the original number of edges.
We see that for optimal performance DIGL makes the graph much denser, which signiﬁcantly affects the node degrees and may negatively impact the time and space complexity of the downstream GNN, which typically are O ( E ) .
In comparison, SDRF adds and removes a similar number of edges and approximately preserves the degree distribution.
The effect on the full degree distribution for three
[Figure]
[Table]
datasets is shown in Figure 4.
We see that the degree distribution following SDRF is close to, or in some cases indistinguishable from, the original distribution, whereas DIGL has a more noticeable effect.
We also compute the Wasserstein distance between the degree distributions, denoted by W 1 in the ﬁgure captions, to numerically conﬁrm this observation.
For this analysis extended to all nine datasets see Appendix F.2.
This difference is also visually evident from Figure 6 in Appendix F.3, where we again observe that SDRF largely preserves the topology of the
Cornell graph and that the curvature (encoded by the edge color) is homogenized across the graph.
We also see that the entries of the trained network’s Jacobian between a node’s prediction and the features of its 2-hop neighbors (encoded as node colors) are increased over the graph, which we may attribute to both DIGL and SDRF’s ability to alleviate the upper bound presented in Theorem 4 and thus reduce over-squashing.
C ONCLUSION
In this paper, we studied the graph bottleneck and the over-squashing phenomena limiting the performance of message passing graph neural networks from a geometric perspective.
We started with a Jacobian approach to determine how the over-squashing phenomenon is dictated by the graph topology as in equation 2.
We then investigated further how the topology induces the bottleneck and hence causes over-squashing.
We introduced a new notion of edge-based Ricci curvature called
Balanced Forman curvature, relating it to the classical Ollivier curvature (Theorem 2).
We then proved in Theorem 4 that negatively-curved edges are responsible for over-squashing, calling for a possibility of curvature-based rewiring of the graph in order to improve its bottleneckedness.
We show one such possibility (Algorithm 1), inspired by the classical Ricci ﬂow and comment on the advantages of surgical method such as this.
We show both theoretically and experimentally that the proposed method can be advantageous compared to a diffusion-based rewiring approach, opening the door for curvature-based rewiring methods for improving GNN performance going forward.
Limitations and future directions.
Our paper establishes a geometric perspective on the graph bottleneck and over-squashing, providing new tools to study and cope with these phenomena.
One limitation of our work is that the theoretical results presented here do not currently extend to multigraphs.
In addition, the current methodology is agnostic to information beyond the graph topology, such as node features.
In future works, we will develop a notion of the curvature and the corresponding rewiring method that can take into account such information.
Acknowledgements.
This research was supported in part by the EPSRC CDT in Modern Statistics and Statistical Machine Learning (EP/S023151/1) and the ERC Consolidator Grant No.(LEMAN).
X.D. gratefully acknowledges support from the Oxford-Man Institute of Quantitative
Finance and the EPSRC (EP/T023333/1).
A PPENDIX
The Appendix is structured as follows:
(i)
In Appendix A we prove Lemma 1 and a side-result about the role of self-loops.
We also summarize how to extend the analysis in Lemma 1 to message passing models with sum aggregations (not average), meaning those architectures where we do not normalize the adjacency matrix.
(ii)
In Appendix B we introduce and describe different quantities we use to characterize the neighbourhood of a given edge i ∼ j .
These objects are all essential to studying the new notion of balanced Forman curvature.
The focus is on how we can distinguish 4-cycles in a computationally tractable way without losing too much accuracy.
(iii)
In Appendix C we provide a brief literature review about existing curvature candidates.
In particular, we report the deﬁnitions of (modiﬁed) Ollivier curvature and Forman curvature.
(iv) In Appendix D we prove the statements in Section 3, i e. Theorem 2, Corollary 3, Theorem 4 and Proposition 5.
We also comment on the role of the assumptions and compare the bound in Theorem 2 with the existing literature.
Finally, we relate the classical notion of betweenness centrality to the over-squashing effect and the negatively curved edges in a graph.
(v) In Appendix E we prove the results in Section 4, namely Theorem 6 and an analogous result.
(vi) In Appendix F we describe more fully the experiments from Section 5, including a full analysis on degree distribution (Appendix F.2) and the hyperparameters used in our experiments (Appendix F.4).
(vii)
In Appendix G we comment on hardware speciﬁcations.
A P ROOFS OF RESULTS IN S ECTION
Lemma 1.
Assume an MPNN as in equation 1.
Let i, s ∈ V with
s ∈ S r +
( i ) .
If |∇ φ (cid:96) | ≤ α and |∇ ψ (cid:96) | ≤ β for 0 ≤ (cid:96) ≤ r , then
[Equation]
Proof.
Let i ∈ V and s ∈ S
r +1 ( i ) .
We recall that to ease the notations we assume that node features and hidden representations are scalar.
The proof in the more general higher-dimensional case follows without any modiﬁcation.
We compute
[Equation]
We can iterate the computation above and see that the right hand side can be expanded as
[Equation]
for some functions Z ij r ...k 1 j 0 of the input features obtained as products of r + 1 partial derivatives of the maps φ (cid:96) and r + 1 partial derivatives of the maps ψ (cid:96) .
Since H (0) =
X , we have
[Equation]
meaning that the previous sum becomes
[Equation]
Since d G ( i, s )
=
r + 1 , the only non-vanishing terms in the sum above are the minimal walks from i to s .
In fact, if there existed a different choice of coefﬁcients yielding a non-zero term then we would ﬁnd a walk joining i to s of length lesser than r + 1 , which is in contradiction with the deﬁnition of geodesic distance.
Since Z i...s ( X ) is a product of r
+ 1 -partial derivatives of the aggregation and update maps and by assumption their gradients are bounded by α and β respectively, we conclude that
[Equation]
which completes the proof of the Lemma.
As a byproduct of this analysis, we can also provide a rigorous motivation for the role of self-loops in
GNNs (see Appendix for details):
Corollary 7.
If h ( (cid:96) +1) i
= (cid:80) j ∼ i ψ (cid:96) ( h ( (cid:96) )
j ) , then h ( (cid:96) +1)
i only depends on nodes that can be reached via walks of length exactly (cid:96) + 1 .
By adding self-loops, the GNN also takes into account nodes that can be reached via walks of length r ≤ (cid:96) + 1 .
Proof.
If h ( (cid:96) +1) i
= (cid:80) j ∼ i ψ (cid:96) ( h ( (cid:96) )
j ) for each (cid:96) ∈ [0 , L − 1] , then we can argue as in the proof of
Lemma 1 and ﬁnd
[Equation]
The combinatorial coefﬁcient a ij (cid:96) . . .
a j 1 s is non-zero iff there exists a walk from i to s of length exactly (cid:96) +1 , since we are not taking into account the contribution coming from self-loops.
Conversely, if each term a ij (cid:96) was replaced by ˆ a ij (cid:96) then we would ﬁnd that ˆ a ij (cid:96) . . .
ˆ a j 1 s is non-zero iff there exists a walk from i to s of length at most r + 1 , since the diagonal entries are now positive.
Remark 8.
As a speciﬁc instance of Corollary 7, we note that if we do not include self-loops in the adjacency matrix, then the output of a 2-layer simpliﬁed graph neural network at node i is independent of the features of neighbours
k
that do not form a triangle with i .
Once again here the dependence is precisely measured via the Jacobian of the hidden features with respect to the input features.
Remark 9.
We note that the role of self-loops has also implicitly been noted in Xu et al.
(2018) where the analysis of the Jacobian of node representations on the graph augmented with self-loops has been related to lazy random-walks.
GNNs with different aggregations We note that similar conclusions extend to message passing architectures where the aggregations are sums and not averages meaning that we take the augmented adjacency without normalizing by the degree matrices.
Consistently with Lemma 1, we restrict to the setting where features and node representations at each layer are scalars to make the discussion simpler.
In line with the Xu et al.
we consider a GNN-model of the form
[Equation]
Note that the augmented neighbourhood ˜ N i is deﬁned as N i ∪ { i } .
Differently from the setting of
Theorem 1 in Xu et al.
(2018), the aggregation here is not an average but a simple sum.
Let us now take nodes i and s
such that s ∈ S
r +1 ( i ) as in the statement of Lemma 1.
In this case, instead of simply considering the quantity | ∂h ( r +1)
i /∂x s
| , we normalize the Jacobian entries - obtaining what [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
This of course represents now a relative importance of feature x s on the representation of node i at layer r + 1 .
If - similarly to Theorem 1 in Xu et al.
- we assume that all paths in the computational graph of the model are activated with the same probability, then we obtain that on average
[Equation]
where ˜ A =
A + I and vol ( S )
= (cid:80) j ∈ S d j .
In particular, we again ﬁnd that if we have a tree structure, then the right hand side decays exponentially as 2 − ( r +1) .
B P RELIMINARY ANALYSIS OF AN EDGE - NEIGHBORHOOD
Given an edge i ∼ j , we introduce the sets below:
(i) (cid:93) ∆ ( i, j ) :
= S 1 ( i ) ∩ S 1 ( j ) , the number of triangles based at the edge i ∼ j .
(ii) (cid:93) i (cid:3)
( i, j ) :
= { k ∈ S 1 ( i ) \ S 1 ( j ) , k (cid:54) =
j : ∃ w ∈ ( S 1 ( k ) ∩ S 1 ( j ))
\ S 1 ( i ) } , the number of nodes k ∈ S 1 ( i ) forming a 4-cycle based at i ∼ j without diagonals inside.
(iii) Q i ( j ) := S
( i ) \ ( { j } ∪ (cid:93) ∆ ( i, j ) ∪ (cid:93)
i (cid:3) ( i, j )) , simply the complement of the neighbours of i with respect to the sets introduced in (i) and (ii) once we also exclude j .
In the following we simply write (cid:93) ∆ , (cid:93) i (cid:3) and Q i when the edge i ∼ j is clear from the context.
4-cycle contributions.
In general the sets (cid:93) i (cid:3) and (cid:93) j (cid:3) may differ.
This may occur when there exists a node k belonging to (cid:93) i (cid:3) that admits multiple solutions w as in the deﬁnition of (cid:93) i (cid:3) .
This feature needs to be taken into account when comparing Ollivier’s Ricci curvature to the new notion we present below.
We ﬁrst introduce the following class to ease the notations.
Deﬁnition 2.
For any simple, undirected graph G =
( V, E ) , if U ⊂ V , then we set
[Equation]
We note that any ϕ ∈ D ( U ) is injective.
We may now deﬁne a quantity which measures the maximal number of 1-1 pairings that can be performed from (cid:93) i (cid:3) to (cid:93) j (cid:3) .
Deﬁnition 3.
For any edge i ∼ j we let
[Equation]
We often simply write (cid:93) m (cid:3) .
While the quantity (cid:93) m (cid:3) plays a role in the derivation of the Ollivier curvature of i ∼ j it is not computationally-friendly, as to determine (cid:93) m (cid:3) we need to identify and distinguish all possible 4-cycles based at i ∼ j
and then choose a maximal pairing map.
Accordingly, we consider a looser term which is easier to compute:
Deﬁnition 4.
For any pair of adjacent nodes i ∼ j we deﬁne
[Equation]
[Equation]
where A s denotes the s -th row of the adjacency matrix.
We usually simply write γ max .
Remark 10.
We note that given k ∈ S 1 ( i ) \ S 1 ( j ) the term ( A k · ( A j − A i (cid:12) A j ))
− 1 yields the number of nodes w forming a 4-cycle of the form i ∼ k ∼ w ∼ j ∼
i with no diagonals inside.
The value γ max measures the maximal degeneracy of edges forming 4-cycles, meaning that it is equal toiff for each k ∈ (cid:93)
i (cid:3) there exists a unique node w ∈ (cid:93) j (cid:3) such that i ∼ k ∼ w ∼ j ∼
i is a 4-cycle.
We now end the discussion about 4-cycle contributions by proving the following inequality, which allows us to avoid to compute directly the term (cid:93) m (cid:3) up to giving up some accuracy.
Lemma 11.
For any edge i ∼ j we have
[Equation]
Proof.
The proof is based on a combinatorial argument.
Let (cid:93) i (cid:3) =
{ k 1 , . . .
, k r } and let (cid:93) m (cid:3) =
{ k 1 , . . .
, k (cid:96) } , with (cid:96) < r .
By deﬁnition there exists ϕ : { k 1 , . . .
, k (cid:96) } → { w 1 , . . .
, w (cid:96) } , with k i ∼ w i and w i ∈ (cid:93) j (cid:3) , for 1 ≤ i ≤ (cid:96) .
Given k ∈ (cid:93) i (cid:3) \ { k 1 , . . .
, k (cid:96) } , then there are no w ∈ (cid:93)
j (cid:3) \ { w 1 , . .
. , w (cid:96) } such that k ∼ w , otherwise we could extend ϕ by setting k (cid:55)→ ϕ ( k ) :
= w and we would then get | (cid:93) m (cid:3) | = (cid:96) + 1 .
Accordingly, we have
[Equation]
which implies
[Equation]
C E XISTING CURVATURE CANDIDATES
Ollivier Ricci curvature For i ∈ V and α ∈ [0 , 1) we deﬁne a probability measure on B 1 ( i ) by:
[Equation]
Before we introduce the Ollivier curvature
[Equation]
we recall that the transportation distance between two ﬁnitely supported probability measures as above can be computed as
[Equation]
where d G ( · , · ) is the geodesic distance on the graph and the inﬁmum is taken over all matrices M satisfying the marginal constraints:
[Equation]
We are now ready to deﬁne the Ollivier Ricci curvature: the formulation below is due to Lin et al.
Deﬁnition 5.
Given i ∼ j we deﬁne the α - Ollivier curvature by
[Equation] [Equation]
Since κ α (1 − α ) − 1 is increasing and bounded the quantity below is well-deﬁned:
[Equation]
Forman Ricci curvature
In the following we report a formula for the augmented Forman Ricci curvature on unweighted graphs Samal et al.
We also note that Forman curvature on graphs has also been studied in Sreejith et al.
; Weber et al.
Deﬁnition 6.
For any edge i ∼ j the augmented Forman curvature is given by
[Equation]
We note that such formulation of curvature does not distinguish contributions coming from 4-cycles.
In fact, for the orthogonal grid with degree d ≥ 4 , Forman Ricci curvature is equal to 2(2 − d ) < 0 .
This does not reﬂect that the r -hop neighbourhood for such a graph grows polynomially in r .
We conclude this appendix by reporting a lower bound for the Ollivier Ricci curvature derived in Jost & Liu (2014).
We recall that κ α , with α ∈ [0 , 1) was deﬁned in equation 7.
Theorem 12 (Jost & Liu (2014)) .
For any edge i ∼ j , with d i ≤ d j , the following bound is satisﬁed:
[Equation]
D P ROOFS OF RESULTS IN S ECTION
We ﬁrst recall our deﬁnition of Balanced Forman:
Deﬁnition 7.
For any edge i ∼ j
we let Ric ( i, j ) be zero if min { d i , d j } = 1 , otherwise
Ric ( i, j ) :=
d i + 2 d j − 2 +| (cid:93) ∆ | max { d i , d j } + | (cid:93) ∆ | min { d i , d j
[Equation]
where the last term is set to be zero if | (cid:93)
i (cid:3) | (and hence | (cid:93) j (cid:3) | ) is zero.
We also extend the previous deﬁnition to the weighted case.
In this setting we let G =
( V, E, ω ) be a simple, locally ﬁnite, undirected graph with normalized weights.
We ﬁrst report the formula for the augmented Forman in the weighted case Samal et al.
[Equation]
where ω ∆ is taken to be the Heron formula for the area of a triangle while ω ( · ) denotes some weighting scheme for the nodes as well.
We propose a similar deﬁnition for the weighted case, which reduces to the one discussed above in the combinatorial setting.
We recall that W is the weighted adjacency matrix while A is the combinatorial one.
Moreover, we write A ij =
( A j − A i ) + and similarly for W ij .
Deﬁnition 8.
For any pair of adjacent nodes i, j ∈ V we deﬁne Ric ( i, j ) to be 0 if min {
| d i | , | d j |} = 1 , otherwise we set
[Equation]
k ∈ Q j (cid:114) ω ij ω
jk 
 + max { d i , d j } (cid:88) k ∈ S 1 ( i ) ∩ S 1 ( j ) ω
2 ij ω ∆ + (cid:88) k ∈ (cid:93) i (cid:3) ω ij ω (cid:3) (cid:18) ω ij ( γ max ) −max { d i , d j } − √ ω ij ν (cid:3)
d i (cid:19) + (cid:88) k ∈ (cid:93) j (cid:3) ω ij ω (cid:3) (cid:18) ω ij ( γ max ) −max { d i , d j } − √ ω ij ν (cid:3) d j (cid:19) , with
[Equation]
and, for a given k ∈ (cid:93) i (cid:3) ,
[Equation]
Important convention.
Without losing generality, in the following we always assume that 1 ≤ d i ≤ d j .
In particular, we write d i .
=
d and d j = d + s , for some s ≥ 0 , omitting to specify that both d and s are of course depending on i and j .
Moreover, from now on we only focus on the unweighted case.
We can now prove our main comparison theorem.
Theorem 2.
Given an unweighted graph G , for any edge i ∼ j we have κ ( i, j ) ≥
Ric ( i, j ) .
Proof.
We stick to the aforementioned convention: d i := d ≤ d j = d + s , s ≥ 0 .
The strategy of the proof amounts to ﬁnding a transportation plan providing an upper bound for W 1 ( µ αi , µ αj ) and hence a lower bound for the curvature κ
( i, j ) .
In particular, we consider plans moving the mass µ αi from
B 1 ( i ) to B 1 ( j ) .
If d = 1 , then the optimal transport plan consists of moving the mass α from i to j and the remaining mass 1 − α on j to S 1 ( j ) .
This yields a unit Wasserstein distance between µ αi and µ αj and hence zero
Ollivier curvature κ ( i, j ) , which coincides with the value of balanced Forman Ric ( i, j ) .
Assume now that d ≥ 2 .
A (possibly non-optimal) transport plan from µ αi to µ αj is given by:
(i) Move mass (1 − α ) /
( d + s ) from each node k ∈ (cid:93) m (cid:3) ⊂ (cid:93) i (cid:3) to its unique image in (cid:93) j (cid:3) under a bijection ϕ as per deﬁnition of (cid:93) m (cid:3) .
(ii)
The remaining mass on each node k ∈ (cid:93) m (cid:3) will need to travel by at most distance 3 to S 1 ( j ) .
(iii)
The extra-mass (1 − α )(1 /d − 1 / ( d + s )) on each common neighbour k ∈ (cid:93) ∆ will need to travel by at most distance 2 to S 1 ( j ) .
(iv) Move the mass (1 − α ) /d from j to S 1 ( j ) .
(v) Move the mass α from i to j .
This leaves left-over mass (1 − α ) /
( d + s ) at i from the distribution µ αj .
This mass can be compensated from mass in S 1 ( i ) which is at distance one.
(vi) Finally, we move the mass (1 − α )
/d of any untouched node in S 1 ( i ) to S 1 ( j ) along a path of length lesser or equal than three.
Note that the remaining mass is equal to (
[Equation]
[Equation]
[Equation]
( d + s ) , where the last terms comes from (v).
If we sum all the contributions we ﬁnd
[Equation]
d
[Equation]
Therefore we have
[Equation]
By using Lemma 11, we can bound the right hand side as κ
[Equation]
i,
[Equation]
≥
[Equation]
which completes the proof.
Remark 13.
By inspection Ric ( i, j ) ≥ Φ( i, j ) , with Φ( i, j ) as in Theorem 12.
We have three cases:
[Equation]
i
[Equation]
[Equation]
i, j ) , because Ric ( i, j ) takes into account the positive contribution of 4-cycles as well.
ii) Φ( i,
[Equation]
[Equation]
d + [Equation]
, which happens iff
[Equation]
and
[Equation]
From the previous inequalities we derive
[Equation]
(iii Φ [Equation] i,
[Equation]
[Equation]
|
[Equation]
| / ( d + s ) which is equivalent to
[Equation]
In this case we have
[Equation]
Corollary 3.
If Ric ( i, j ) ≥ k > 0 for any edge i ∼ j , then there exists a polynomial P such that
[Equation]
Proof.
This follows immediately from Theorem 2 and a Bishop-Gromov type of result for discrete
Ollivier curvature on graphs Paeng (2012).
To address the proof of Theorem 4, we ﬁrst need the Lemma below.
Lemma 14.
Given i ∼ j , with d i ≤ d j , if Ric ( i, j ) ≤ − 2 + δ , for some 0 < δ < (1 + γ max ) − 1 , then | Q j | | (cid:93) ∆ | + 1 > δ − 1 .
Proof.
According to our convention we let d i =
d and d j = d + s , for s ≥ 0 .
We also recall that
Q j = S 1 (
[Equation]
\ ( (cid:93) ∆ ∪ (cid:93) j (cid:3) ∪{ i } ) .
If we multiply equation 3 by d j = d + s , we see that
Ric ( i, j ) ≤ − 2+ δ iff
[Equation]
By assumption | (cid:93) j (cid:3) | ( γ − 1 max − δ ) ≥ 0 , meaning that
[Equation]
Therefore, we conclude
[Equation]
Theorem 4.
Consider a MPNN as in equation 1.
Let i ∼ j with d i ≤ d j and assume that:
(i) |∇ φ (cid:96) | ≤ α and |∇ ψ (cid:96) | ≤ β for each 0 ≤ (cid:96) ≤
L − 1 , with L ≥ 2 the depth of the MPNN.
(ii)
There exists δ s t. 0 < δ < (max { d i , d j } ) − 12 , δ < γ − 1 max , and Ric ( i, j ) ≤ − 2 + δ .
Then there exists Q j ⊂ S 2 ( i ) satisfying | Q j | > δ − 1 and for 0 ≤ (cid:96)
≤ L − 2 we have
[Equation]
Proof.
As usual we let d i :=
d and d j :=
d + s , for some s ≥ 0 .
We ﬁrst observe that from the requirement δ 2 ( d + s ) ≤ 1 in (ii), we derive Ric ( i, j ) ≤ −
+ δ iff
[Equation]
Therefore we have
[Equation]
meaning that
[Equation]
From now on we let Q j denote again the complement S 1 ( j ) \ ( S 1 ( i ) ∪ (cid:93) j (cid:3) ∪ { i } ) .
Without loss of generality we set (cid:96)
=
and hence h (0) i
=
x
i ; the very same proof applies to any other choice of (cid:96) 0 .
Given k ∈ Q j , since k ∈ S 2 ( i ) , we can apply Lemma 1 and derive
[Equation]
We may expand the power of the augmented normalized adjacency matrix as
[Equation]
If we introduce the set ˆ Q j = { k ∈ Q j : σ ik > 1 } , we can then write (cid:88)
[Equation]
where in the last equality we have again used the fact that k ∈ ˆ Q j iff there is w ∈ S 1 ( k ) ∩ S 1 ( i ) ∩ S 1 ( j ) .
To avoid heavy notations, we introduce V k :
= S 1 ( k ) ∩ S 1 ( i ) ∩ S 1 ( j ) .
Let us ﬁrst focus on the ﬁrst sum in equation 12.
We have
[Equation]
We now consider the second sum in equation 12.
We assume | (cid:93) ∆ | ≥ 1 , otherwise ˆ Q j = ∅ .
We let
[Equation]
for some C > 0 to be chosen below.
Then, the second sum in equation 12 can be split as
[Equation]
Since any w ∈ V k has degree at least three, we can bound the ﬁrst term in equation 14 as
[Equation]
We now observe that
[Equation]
Since d w ≤ (1 /C ) | ˆ Q j | / | (cid:93) ∆ | + 2 /C for any w ∈ Ω we see that the ﬁrst term in equation 14 can be bounded by (cid:88)
[Equation]
by deﬁnition of Ω .
We now bound the second term in equation 14 as
[Equation]
where we have used that d − 1 w ≤ C | (cid:93) ∆ | / | ˆ Q j | if w ∈ V k \ Ω .
Since
[Equation]
we see that (cid:88)
[Equation]
We are now ready to complete the proof of the theorem.
According to equation 11 it sufﬁces to show that
[Equation]
From equation 12 and equation 13 we derive that the left hand side of the equation above is bounded by
[Equation]
where in the last inequality we have used Lemma 14 to bound | Q j | − 1 by δ .
In particular we note that if (cid:93) ∆ = ∅ then ˆ Q j = ∅ , and the bound would be simply controlled by δ as claimed.
When | (cid:93) ∆ | > 0 , we can use equation 15 and equation 16 to estimate the second term from above by
[Equation]
By applying Lemma 14 we get
[Equation]
We now choose C = δ − 1 / 4 , so that the previous quantity can be bounded by
[Equation]
where in the last inequality we have used equation 10.
Therefore, we have shown that
[Equation]
where we have used that δ < 1 .
This completes the proof (once we absorb the extra factor 3 in the constant αβ in equation 11).
Remark 15.
The requirement δ (cid:112) max { d i , d j } < 1 can be replaced by a more general bound δ (cid:112) max { d i , d j } < r .
The argument above extends to this case up to renaming the constant αβ in the statement so to include an extra factor r .
We note that the condition δ max { d i , d j } < r would be stronger than the one appearing in (ii) of
Theorem 4.
In this regard, we recall that for a d -tree the curvature satisﬁes
Ric ( i, j ) = − 2 + 4 d .
We can also prove Proposition 5:
Proposition 5.
If Ric ( i, j ) ≥ k > 0 for all i ∼ j , then λ 1 / 2 ≥ h G ≥ k 2 .
Proof.
This follows as an immediate Corollary of Theorem 2 and (Lin et al., 2011, Theorem 4.2).
Betweenness centrality to measure bottleneck.
In equation 2 we have derived how the topology of the graph affects the dependence of the hidden node representation h ( r +1)
i on the input feature x s , for nodes i, s at distance r + 1 .
We note that in this case ˆ A r +1 is is exactly measuring the number of minimal paths from i to s .
If the receptive ﬁeld B r +1 ( i ) is a binary tree, then we have seen that the entry of the power matrix decays exponentially.
The reason for such decay stems from the existence of exponentially many nodes in the receptive ﬁeld combined with the lack of multiple minimal paths (shortcuts).
When such conditions hold, most of the minimal paths go through the same nodes, which is exactly what happens for the tree where each node is in the minimal paths between different branches.
Since the frequency in which a node appears in the minimal path of distinct pairs of nodes is measured by the betweenness centrality Freeman (1977), we propose a topological characterization of the ‘bottleneckedness’ of a graph as follows:
Deﬁnition 9 ( bottleneck).
The bottleneck-value of G is b G :=
1 n (cid:80) ni =1 b ( i ) , where b ( i ) denotes the betweenness centrality on node i .
From a standard combinatorial argument it follows that if G is connected, then
[Equation]
We note that b G = 0 iff G is the complete graph K n .
Therefore, b G determines how far the given topology is from K n , with the latter representing the limit case of a fully connected layer Alon &
Yahav (2021) where no bottleneck may occur as any pair of nodes would be neighbours.
This further supports our intuition that the betweenness centrality is a good topological candidate for providing a global measurement of bottleneckedness in the graph.
It also follows from equation 17 that any update to the graph topology consisting of edge additions would decrease b G and thus reduce the bottleneck.
The quantity b G is global in nature though and hence lacks robustness.
As a pedagogical example, consider a barbell G ( m, 2 r + 1) , with m the size of the two cliques joined by a path of length 2 r + 1 and focus on the edge i ∼ j in the middle of such path.
Nodes i and j are central to the graph, in the sense that most minimal paths go through them and indeed their betweenness centrality is b ( i )
=
b ( j ) =
( m + r )
+ ( m + r ) .
If now we add a single edge joining the two cliques K m , the values b ( i ) and b ( j ) decrease dramatically by Ω( m 2 + r ) .
Since the operation is non-local, we see that the representation h ( (cid:96) )
i of any MPNN is unaffected by the edge addition for any (cid:96) ∈ (0 , r ) , and similarly for j .
Eventually, if we keep adding edges, the receptive ﬁelds B s ( i ) will be affected for small values of s as well: the drawback of such approach is that the resulting adjacency may be signiﬁcantly different.
Conversely, the curvature provides a more precise, local and hence robust way of controlling the bottleneck and hence the over-squashing problem.
Nonetheless, we relate the betweenness centrality to the Jacobian of the hidden features.
Theorem 16.
Given i ∼ j , let Ω j := S 1 ( i ) ∩ S 1 ( j ) ∪ { j } .
If Ric ( i, j ) ≤ − 2 + δ , for 0 < δ < (1 + γ max ) − 1 , then
[Equation]
Proof.
We rewrite the quantity in the statement as
[Equation]
By deﬁnition, given a node k ∈ V , the betweenness centrality of k is given by
[Equation]
where σ st is the number of minimal paths between s and t while σ st ( k ) is the number of minimal paths from s to t passing through k .
For convenience, we introduce the set ˆ Q j ⊂ Q j deﬁned by
[Equation]
Equivalently, ˆ Q j consists of those nodes in S 1 ( j ) \ S 1 ( i ) which form a 4-cycle based at i ∼ j with a diagonal inside.
Indeed, if w ∈ Q j and σ iw > 1 , then there exists more than one minimal path between i and w , in addition to the one passing through node j .
For any such path there exists k ∈ S 1 ( i ) ∩ S 1 ( w ) .
Since w ∈ Q j and Q j ∩ (cid:93) j (cid:3) = ∅ , we derive that k ∈ S 1 ( j ) as well.
We then get (cid:88)
[Equation]
By summing σ iw ( k ) for all k ∈ (cid:93) ∆ we obtain all the 2-long minimal paths between i and w with the exception of the one passing through j :
[Equation]
On the other hand we also have
[Equation]
By combining equation 18 and equation 19 we ﬁnally get
[Equation]
where in the last equality we have used that by deﬁnition σ iz = 1 for all z ∈ Q j \ ˆ Q j .
By Lemmathe last quantity is larger than δ − 1 .
E P ROOFS OF RESULTS IN S ECTION
Theorem 6.
Let S ⊂ V with vol ( S ) ≤ vol ( G ) / 2 .
Then h S,α ≤ (cid:0) 1 − αα (cid:1)
d avg ( S ) d min ( S ) h S , where d avg ( S ) and d min ( S ) are the average and minimum degree on S , respectively.
Proof.
Given a signal f : V → R on the vertex set and U ⊂ V , analogously to Chung (2007), we introduce the notation
[Equation]
Let us rewrite the new Cheeger constant h S,α as follows:
[Equation]
with χ S the characteristic function of the subset S , i.e. χ S ( i )
= 1 iff i ∈ S .
Since the graph G is connected, we can bound h S,α from above as
[Equation]
It was proven in (Chung, 2007, Lemma 5) that
[Equation]
By applying equation equation 20 to the bound for the Cheeger constant h S,α we ﬁnally see that
[Equation]
We also report an equivalent result, again relying on (Chung, 2007, Lemma 5).
Proposition 17.
Let S ⊂ V with vol ( S ) ≤ vol ( G ) / 2 .
For any k ∈ N , there exists S k,α ⊂ S with vol ( S k,α ) ≥ vol ( S )(1 − (2 k ) − 1 ) such that
[Equation]
for all i ∈ S k,α .
Proof.
Let k ∈ N .
By modifying slightly the argument in (Chung, 2007, Lemma 5), we derive that
[Equation]
satisﬁes
[Equation]
Therefore, we obtain
[Equation]
We then conclude that the complement of S (cid:48) k,α has volume greater or equal than vol ( S )(1 − (2 k ) − 1 ) , which completes the proof.
Remark 18.
The previous proposition shows that after sparsifying the personalized page rank operator R α as suggested in Klicpera et al.
(2019) by setting entries below some threshold equal to zero, there will still be only few edges connecting different communities, once again highlighting that random-walk based methods are generally not suited to tackle the graph bottleneck.
F E XPERIMENTS
Our experiments in this paper are semi-supervised node classiﬁcation (semi-supervised in that the graph structure provides some unlabelled information) on nine common graph learning datasets.
Cornell, Texas and Wisconsin are small heterophilic datasets based on webpage networks from the WebKB dataset.
Chameleon and Squirrel (Rozemberczki et al., 2021) are medium heterophilic datasets based on Wikipedia networks, along with Actor, the actor-only induced subgraph of the ﬁlmdirector-actor-writer network (Tang et al., 2009).
Cora (McCallum et al., 2000), Citeseer (Sen et al., 2008) and Pubmed (Namata et al., 2012) are medium homophilic datasets based on citation networks.
As in Klicpera et al.
(2019), for all experiments we consider the largest connected component of the graph.
When splitting the data into train/validation/test sets, we ﬁrst separate the data into a development set and the test set.
This is done once to ensure the test set is not used for any training or hyperparameter ﬁtting before the ﬁnal evaluation.
For each of the 100 random splits the development set is divided randomly into a train set and a validation set, where we train models on the train set and evaluate on the validation set.
We ﬁt hyperparameters by random search, maximising the mean accuracy across the validation sets.
The accuracy then reported in Table 2 is the mean accuracy on the test set from models trained on the train sets with the chosen hyperparameters, along with a 95% conﬁdence interval calculated by bootstrapping the test set accuracies with 1000 samples.
For Cora, Citeseer and Pubmed the development set contains 1500 nodes with the rest kept for the test set, and for each random split the train set is chosen to contain 20 nodes of each class while the rest form the validation set.
As this is the same method as Klicpera et al.
and we use the same random seeds, we are using the same test set and expect to have comparable results.
For the remaining datasets we perform a 60/20/20 split, with 20% of nodes set aside as the test set and then for each random split the remaining 80% is split into 60% training, 20% validation.
The homophily index H ( G ) proposed by Pei et al.
(2019) is deﬁned as
[Equation]
v ∈ V
Number of v ’s neighbors who have the same label as v
Number of v ’s neighbors .
[Equation]
F. D ATASETS
For datasets with disconnected graphs, the statistics shown here are for the largest connected component.
[Table]
Edges
[Table]
F. D EGREE DISTRIBUTIONS
[Figure]
[Figure] [Figure]
( Original
[Figure]
SDRF )
[Figure]
g)
[Figure]
( Original
[Figure]
DIGL )
[Figure]
( Original
[Figure]
SDRF )
[Figure]
captions we see distance between
F H YPERPARAMETERS
[Table]
[Table] [Table] [Table]
0423 Squirrel
[Table] [Table] [Table]
Actor [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
G H ARDWARE SPECIFICATIONS
Our experiments were performed on a server with the following speciﬁcations:
[Table]