Expander Graph Propagation
Abstract
Deploying graph neural networks (GNNs) on whole-graph classification or regression tasks is known to be challenging: it often requires computing node features that are mindful of both local interactions in their neighbourhood and the global context of the graph structure.
GNN architectures that navigate this space need to avoid pathological behaviours, such as bottlenecks and oversquashing, while ideally having linear time and space complexity requirements.
In this work, we propose an elegant approach based on propagating information over expander graphs .
We provide an efficient method for constructing expander graphs of a given size, and use this insight to propose the EGP model.
We show that EGP is able to address all of the above concerns, while requiring minimal effort to set up, and provide evidence of its empirical utility on relevant datasets and baselines in the Open Graph Benchmark.
Importantly, using expander graphs as a template for message passing necessarily gives rise to negative curvature.
While this appears to be counterintuitive in light of recent related work on oversquashing, we theoretically demonstrate that negatively curved edges are likely to be required to obtain scalable message passing without bottlenecks.
To the best of our knowledge, this is a previously unstudied result in the context of graph representation learning, and we believe our analysis paves the way to a novel class of scalable methods to counter oversquashing in GNNs.
Introduction
Graph neural networks (GNNs) are a flexible class of models for learning representations over graph-structured data [1].
Their versatility [2–4] and generality [5, 6] has made them a very attractive approach, leading to considerable application in areas as diverse as virtual drug screening [7], traffic prediction [8], combinatorial chip design [9] and pure mathematics [10, 11].
Most GNNs rely on repeatedly propagating information between neighbouring nodes in the graph.
This is commonly expressed in the message passing [4] paradigm: nodes send vector-based messages to each other along the edges of the graph, and nodes update their representations by aggregating all the messages sent to them, in a permutation-invariant manner.
Under many industrially-relevant tasks (which require identifying node-level properties, often with homophily assumptions), this formalism is very well aligned, often allowing for highly scalable model variants [12–14].
However, in many areas of scientific interest, purely local interactions are likely to be insufficient.
Among the principal tasks over graphs, graph classification is perhaps most ripe with such situations: to meaningfully attach a label to a graph, in many cases it is insufficient to treat graphs as “bags of nodes”.
For example, when classifying a molecule for its potency as a candidate drug [7], the label is driven by complex substructure interactions in the molecule [15], rather than a na¨ıve sum of atom-level effects.
Accordingly, GNNs deployed in this regime need to update node features in a manner that is mindful of the global properties of the graph.
It quickly became apparent (as early as [2]) that it is often inadequate to merely stack more message passing layers over the input graph.
In fact, for many standard graph classification tasks, such approaches may be weaker than discarding the graph structure altogether [16, 17].
Now, it is wellunderstood that stacking many local layers leaves GNNs vulnerable to pathological behaviours such as oversquashing [18], wherein nodes close to bottlenecks in the graph would need to store quantities of information that are exponentially increasing with model depth.
Within this space, we are interested in proposing a method that satisfies four desirable criteria: ( C1 ) it is capable of propagating information globally in the graph; ( C2 ) it is resistant to the oversquashing effect and does not introduce bottlenecks; ( C3 ) its time and space complexity remain subquadratic (tighter than O p| V | 2 q for sparse graphs); and ( C4 ) it requires no dedicated preprocessing of the input.
Satisfying all four of these criteria simultaneously is challenging, and we will survey many of the popular approaches in the next section—demonstrating ways in which they fail to meet some of them.
In this paper, we identify expander graphs as very attractive objects in this regard.
Specifically, they offer a family of graph structures that are fundamentally sparse ( | E | “ O p| V |q ), while having low diameter : thus, any two nodes in an expander graph may reach each other in a short number of hops, eliminating bottlenecks and oversquashing (see Figure 1).
Further, we will demonstrate an efficient way to construct a family of expander graphs (leveraging known theoretical results on the special linear group , SL p 2 , Z n q ).
Once an expander graph of appropriate size is constructed, we can perform a certain number of GNN propagation steps over its structure to globally distribute the nodes’ features.
Accordingly, we name our method expander graph propagation ( EGP ).
Another important contribution of our work concerns extending the implications of prior art on oversquashing via curvature analysis [19].
According to [19], edges that are negatively curved are causing the oversquashing effect—yet, counterintuitively, the edges of the expander graphs we construct will always be negatively curved !
We prove, however, that our expanders can never be sufficiently negatively curved to trigger the conditions necessary for the results in [19] to be applicable, and show that the existence of negatively curved edges might in fact be required in order to have sparse communication without bottlenecks.
Related work
We begin with a survey of the many prior approaches to handling global context in graph representation learning, evaluating them carefully against our four desirable criteria ( C1 – C4 ; cf.
Table 1).
This list is by no means exhaustive, but should be indicative of the most important directions.
Stacking more layers.
As already highlighted, one way to achieve global information propagation is to have a deeper GNN.
In this case, we are capable of satisfying ( C1 ) and ( C4 )—no dedicated preprocessing is needed.
However, depending on the graph’s diameter, we may need up to O p| V |q layers to learning (Section 2).
“( ✓ )” indicates that a criterion may be satisfied, depending on the method’s tradeoffs.
Our proposal, the expander graph propagation (EGP) method, satisfies all four criteria.
[Table]
cover the graph, leading to quadratic complexity (violating ( C3 )) and introducing a vulnerability to bottlenecks ( C2 ), as theoretically and empirically demonstrated in [18].
Master nodes.
An attractive approach to introducing global context is to introduce a master node to the graph, and connect it to all of the graph’s nodes.
This can be done either explicitly [4] or implicitly, by storing a “global” vector [20].
It trivially reduces the graph’s diameter to 2 , introduces
O p 1 q new nodes and O p| V |q new edges, and requires no dedicated preprocessing, hence it satisfies ( C1 , C3 , C4 ).
However, these benefits come at the expense of introducing a bottleneck in the master node: it has a very challenging task (especially when graphs get larger) to continually incorporate information over a very large neighbourhood in a useful way.
Hence it fails to satisfy ( C2 ).
Fully connected graphs.
The converse approach is to make every node a master node: in this case, we make all pairs of nodes connected by an edge—this was initially proposed as a powerful method to alleviate oversquashing by [18].
This strategy proved highly popular in the recent surge of Graph
Transformers [22, 23, 25], and is common for GNNs used in physical simulation [21] or reasoning [24] tasks.
The graph’s diameter is reduced to 1, no bottlenecks remain, and the approach does not require any dedicated preprocessing.
Hence ( C1 , C2 , C4 ) are trivially satisfied.
The main downside of this approach is the introduction of O p| V | 2 q edges, which means ( C3 ) can never be satisfied—and this approach will hence be prohibitive even for modestly-sized graphs.
Feature augmentation.
An alternative approach is to provide additional features to the GNN which directly identify the structural role each node plays in the graph structure [26].
If done properly (i.e., if the computed features are directly relevant to the target task), this can drastically improve expressive power.
Hence, in theory, it is possible to satisfy ( C1 ) while not violating ( C2 , C3 ).
However, computing appropriate features requires either specific domain knowledge, or an appropriate pretraining procedure [27–31] to be applied, in order to obtain such embeddings.
Hence all of these gains come at the expense of failing to satisfy ( C4 ).
Graph rewiring.
Another promising line of research involves modifying the edges of the original graph, in order to alleviate bottlenecks.
Popular examples of this approach involve using diffusion [32]—which diffuse additional edges through the application of kernels such as the personalised
PageRank, and stochastic discrete Ricci flows [19]—which surgically modify a small quantity of edges to alleviate the oversquashing effect on the nodes with negative Ricci curvature.
If realised carefully, such approaches will not deviate too far from the original graph, while provably alleviating oversquashing; hence it is possible to satisfy ( C1 , C2 , C3 ).
However, this comes at a cost of having to examine the input graph structure, with methods that do not necessarily scale easily with the number of nodes.
As such, dedicated preprocessing is needed, failing to satisfy ( C4 ).
Hierarchical message passing.
Lastly, going beyond modifying the edges, it is also possible to introduce additional nodes in the graph—each of them responsible for a particular substructure in the graph 1 .
If done carefully, it has the potential to drastically reduce the graph’s diameter while not introducing bottlenecked nodes (hence, allowing us to satisfy ( C1 , C2 )).
However, in prior work, a cost has to be paid for this, usually in the need for dedicated preprocessing.
Prior proposals for hierarchical GNNs that remain scalable require a dedicated pre-processing step [33–35], sometimes coupled with domain knowledge [35]—thus failing to satisfy ( C4 ).
In addition, such methods may require adding prohibitively large numbers of substructures [36, 37] or expensive pre-computation, e.g. computing the graph Laplacian eigenvectors [38].
This might make even ( C3 ) hard to satisfy.
Theoretical background
We now dedicate our attention to the key theoretical results over expander graphs, which will allow
EGP to have favourable properties and be efficiently precomputable.
Definition 1.
For a finite connected graph G “ p V p G q , E p G qq , we consider functions f : V p G q Ñ R .
The Laplacian Lf : V p G q Ñ R of such a function is defined to be
[Equation]
where deg p v q is the degree of the vertex v .
The mapping L : R V p G q Ñ R V p G q sending a function f to its Laplacian Lf is a linear transformation.
It is not hard to show [39] that L is symmetric with respect to the standard basis for R V p G q and positive semi-definite and hence has non-negative real eigenvalues
[Equation]
The smallest eigenvalue is 0 and its associated eigenspace consists of the constant functions (assuming
G is connected).
The smallest positive eigenvalue, λ 1 p G q , is central to the definition of expander graphs, as the next definition shows.
Definition 2.
A collection t G i u of finite connected graphs is an expander family if there is a constant c ą 0 such that for all G i in the collection, λ 1 p G i q ě c .
Expander families [40–42] have many remarkable and useful properties, particularly when there is a uniform upper bound on the degree of the vertices of G i .
Definition 3.
Let G be a finite graph.
For A Ă V p G q , its boundary B A is the collection of edges with one endpoint in A and one endpoint not in A .
The Cheeger constant h p G q is defined to be
[Equation]
Thus, having a small Cheeger constant is equivalent to the graph having a ‘bottleneck’, in the sense that there is a collection of edges B A that, when removed, disconnects the vertices into two sets ( A and its complement, V p G qz A ), with the property that the sizes of A and its complement are significantly larger than the size of B A .
Expander families can be reinterpreted using Cheeger constants, as follows (see, e.g., [43–46]):
Theorem 4.
Let t G i u be a collection of finite connected graphs with a uniform upper bound on their vertex degrees.
Then the following are equivalent:
t G i u is an expander family; 2.
there is a constant ϵ ą 0 such that for all graphs in the collection, h p G i q ě ϵ .
Hence, expander graphs have higher Cheeger constants and will hence provably be bottleneck-free .
The following result is one of the many useful properties of expander families, and it concerns their diameter .
It was proved by Mohar [47, Theorem 2.3].
See also [44].
Theorem 5.
The diameter diam p G q of a graph G satisfies
[Equation]
where ∆ p G q is the maximal degree of any vertex of G .
Hence, if t G i u is an expander family of finite graphs with a uniform upper bound on their vertex degrees, then there is a constant k ą 0 such that for all graphs in the family,
[Equation]
Therefore, if we want to globally propagate information over an expander graph which has | V | nodes, we only need O p log | V |q propagation steps to do so—yielding subquadratic complexity.
We have now successfully shown that expander graphs are bottleneck-free, and have favourable propagation qualities.
What is missing is an efficient method of constructing an expander graph of (roughly) | V | nodes.
To demonstrate such a method, we leverage known results from group theory.
Definition 6.
A group p Γ , ˝q is a set Γ equipped with a composition operation
˝ : Γ ˆ Γ Ñ Γ (written concisely by omitting ˝ , i.e. g ˝ h “ gh , for g, h P Γ ), satisfying the following axioms:
• (Associativity) p gh q l “ g p hl q , for g, h, l P Γ .
• (Identity) There exists a unique e P Γ satisfying eg “ ge “ g for all g P Γ .
• (Inverse)
For every g P Γ there exists a unique g ´ 1 P Γ such that gg ´ 1 “ g ´ 1 g “ e .
Definition 7.
Let Γ be a group.
A subset S Ď Γ is a generating set for Γ if it can be used to “generate” all of Γ via composition.
Concretely, any element g P Γ can be expressed by composing elements in the generating set, or their inverses; that is, we can express g “ s ˘ 1 1 s ˘ 1 2 s ˘ 1 3 ¨ ¨ ¨ s ˘ 1 n ´ 1 s ˘ 1 n for s i P S .
Now we are ready to define a Cayley graph of a group w.r.t.
its generating set.
Definition 8.
Let Γ be a group with a finite generating set S .
Then the associated Cayley graph
Cay p Γ; S q has vertex set Γ and it has an edge g Ñ gs for each g P Γ and each s P S .
We say that s is the label on this edge.
This is a potentially non-simple graph, as it may have edges with both endpoints on the same vertex and it may have multiple edges between a pair of vertices.
In particular, when s has order 2 , then we view the edge g Ñ gs and the edge gs Ñ gs 2 “ g as distinct edges.
Note that the degree of each vertex of a Cayley graph Cay p Γ; S q is 2 | S | .
This is because each vertex g is joined by edges to gs and gs ´ 1 for each s P S .
Thus, we shall be particularly interested in the case where there is a uniform upper bound on | S | .
The specific group we use for EGP is as follows.
For each positive integer n , the special linear group SL p 2 , Z n q denotes the group of 2 ˆ 2 matrices with entries that are integers modulo n and with determinant 1 .
One of its generating sets is:
[Equation]
Central to our constructions is the following important result.
Theorem 9.
The family of Cayley graph Cay p SL p 2 , Z n q ; S n q forms an expander family.
The proof uses a result of Selberg [48] who showed that the smallest positive eigenvalue of the
Laplacian of certain hyperbolic surfaces is at least 3 { 16 .
One can use this to a produce a lower bound on the first eigenvalue of the Laplacian on Cay p SL p 2 , Z n q ; S n q .
Full proofs are given in [42, 41].
Lastly, it is useful to state a known result: the number of nodes of Cay p SL p 2 , Z n q ; S n q is:
[Equation]
hence, it is of the order of O p n 3 q .
We now study the local properties of Cayley graphs in detail.
Local structure of the Cayley graphs, and the utility of negative curvature
Recent work [19] has suggested that the local structure of the graph G underlying a GNN may play an important role in the way that information propagates around G .
In particular, various notions of ‘Ricci curvature’ such as Forman curvature [49], Ollivier curvature [50, 51] and balanced Forman curvature [19] have been examined.
These are all local quantities, in the sense that they depend on the structure of the graph within a small neighbourhood of each edge.
In this section, we will therefore examine the local structure of the Cayley graphs G n “ Cay p SL p 2 , Z n q ; S n q .
The various notions of curvature given above are defined for each e of the graph G .
Since, as defined by [19], the balanced Forman curvature of an edge depends only on local structures (i.e. triangles and squares) around that edge, they can be determined by only observing the immediate 2-hop surrounding of that edge.
Formally, for an edge e of a graph G , let N 2 p
e q be the induced subgraph with vertices that are at most two hops away from at least one endpoint of e .
Then the curvature of e only depends on the isomorphism type of N 2 p e q .
More specifically, if e and e 1 are edges in possibly distinct graphs, and there is a graph isomorphism between N 2 p e q and N 2 p e 1 q that sends e to e 1 , then this guarantees that the curvatures of e and e 1 are equal.
This situation arises prominently in the Cayley graphs that we are considering, as follows.
Proposition 11.
Let s be one of ˆ 10˙ [Equation] 1˙ .
Let n, n 1 ą 18 and let e and e 1 be s -labelled edges in G n and G n 1 .
Then there is a graph isomorphism between N 2 p e q and N 2 p e 1 q taking e to e 1 .
We prove Proposition 11 in Appendix A.
This immediately allows us to characterise the balanced
Forman curvature and Ollivier curvature for all of the Cayley graphs we generate:
Proposition 12.
The balanced Forman curvatures Ric p n q , and the Ollivier curvatures κ p n q of all edges of Cayley graphs G n are given by:
[Equation]
Proof.
Proposition 11 implies that the balanced Forman and Ollivier curvatures are all equal for n ą 18 .
Their values for 2 ď n ď 19 can all be empirically computed, and are given as above.
Prior work [19] suggests it is preferable for GNNs to operate on graphs with positive Ricci curvature, whereas our graphs G n p n ą 2 q all have negative Ricci curvature.
However, we contend that negative
Ricci curvature is not in itself an impediment to efficient propagation around a GNN.
Indeed, it was shown in [19, Theorem 4] that poor propagation arises when the balanced Forman curvature is close to ´ 2 , specifically if it is at most ´ 2 ` δ for some δ ą 0 .
Here, δ is required to satisfy certain inequalities.
But, with certainty, δ “ 1 can never be satisfied in the hypotheses of [19, Theorem 4].
Furthermore, positive Ricci curvature may have downsides when used for GNNs.
One significant downside to non-negative Ricci curvature can be derived using the main result of [52], which says that the three properties of expansion, sparsity and non-negative Ollivier curvature are incompatible, in the following sense.
Theorem 13.
For any δ ą 0 and ∆ ą 0 , there are only finitely many graphs with maximum vertex degree ∆ , Cheeger constant at least δ and non-negative Ollivier curvature.
We prove Theorem 13 in Appendix B. Furthermore, quoting directly from [52]: “The high-level message is that on large sparse graphs, non-negative curvature (in an even weak sense) induces extremely poor spectral expansion.
This stands in stark contrast with the traditional idea – quantified by a broad variety of functional inequalities over the past decade – that non-negative curvature is associated with good mixing behavior.”
In our view, it is highly desirable that the graphs used for GNNs have high Cheeger constants, in the sense of globally lacking bottlenecks.
Having bounded vertex degree is certainly useful too, since it implies that the graphs will be sparse, and the nodes will not have to handle ever-increasing neighbourhoods for message passing as graphs grow larger in size 2 .
As we have just shown, using the results from [52], non-negative Ollivier curvature is incompatible with these properties when the graph is sufficiently large.
The negative curvature of each edge in G n implies that they are locally ‘tree-like’.
In Appendix C, we make this statement precise by showing that G n is ‘tree-like’ up to scale c log p n q about each node, for c » p 1 { 2 qp log pp 1 ` ?
5 q{ 2 qq ´ 1 (see Figure 1 (Right) for a schematic view).
This tree-like structure might seem, at first, to be counter-productive for good propagation across the graphs G n .
Indeed, GNNs based on trees have been shown to have provably poor performance [18].
The reason for this seems to be two-fold.
On the one hand, trees have small Cheeger constant.
Indeed, any tree G on n vertices has a Cheeger constant 1 { t n { 2 u , since we may find an edge that, when removed, decomposes the graph into subgraphs with t n { 2 u and r n { 2 s vertices.
As discussed in Section 3 and in [19], when a graph has small Cheeger constant, its performance when used as a template for a GNN is likely to become poor.
Secondly, GNNs based on trees are susceptible to oversquashing.
For a k -regular infinite tree, there are k p k ´ 1 q r ´ 1 vertices at distance r from a given vertex.
Hence, if information is to be propagated at least distance r from a given vertex, then seemingly an exponential amount of information is required to be stored.
However, neither of these issues are problematic for a GNN based on the Cayley graph G n .
By
Theorem 9, their Cheeger constants are bounded away from 0 .
Secondly, although they are tree-like locally, this is only true up to scale O p log n q .
In fact, the r -neighbourhood of any vertex is the whole graph G n as soon as r ą C log n , for some constant C , by Theorem 5.
Being tree-like up to distance
O p log n q does not lead to a requirement to store too much information as the message propagates.
This is because k p k ´ 1 q r ´ 1 is polynomial in n when r ď O p log n q .
Beyond this scale, there exist many additional connections, which lead to many possible paths joining any pair of vertices.
Each of these paths can be a potential route of transfer of information from one vertex to another.
The perspective of information transfer also gives rise to another perspective in which expanders fare very favourably: the mixing time of their corresponding Markov chain.
We state several known facts about the favourable mixing times of expanders in Appendix D, to further supplement our claims on their efficient communication properties.
Expander graph propagation
Let an input to a graph neural network be a node feature matrix X P R | V |ˆ k , and an adjacency matrix
A P R | V |ˆ| V | .
This setup is such that the feature vector of node u , x u P R k , can be recovered by taking an appropriate row from X .
Note that the adjacency information can also be fed in an edge-list manner, which is desirable from a scalability perspective.
Further, each edge in the graph may be endowed with additional features rather than a single real scalar.
None of the above modifications would change the essence of our findings; we use a matrix formalism here purely for simplicity.
There exist many ways in which the computed Cayley graph Cay p SL p 2 , Z n q ; S n q can be leveraged for message propagation, and exploring these variations could be very useful for future work.
Here, we opt for a simple construction: interleave running a standard GNN over the given input structure, followed by running another GNN layer over the relevant Cayley graph.
If we let A Cay p n q be an adjacency matrix derived from Cay p SL p 2 , Z n q ; S n q , this implies:
[Equation]
Here, GNN refers to any preferred GNN layer, such as the graph isomorphism network [53, GIN]:
[Equation]
where N u is the neighbourhood of node u , i.e. in our setup, the set of all nodes v such that a vu ‰ 0 .
ϵ P R is a learnable scalar, and ϕ : R k Ñ R k 1 is a two-layer MLP.
This procedure is iterated for a certain number of steps, after which the computed node embeddings in H can be used for any downstream task of interest—such as node classification, link prediction or graph classification.
Note that, unlike [18], who apply their custom layer only at the tail of the architecture, we apply the expander graph immediately after each layer over the input graph.
We find that if the input graph given by A contains bottlenecks, applying the GNN over A Cay p n q only at the end may result in oversquashing occurring before any expander graph propagation can take place.
The setup so far assumed the number of nodes in our input graph to line up with the Cayley graph, that is, A Cay p n q P R | V |ˆ| V | .
However, there is no guarantee that we can find an appropriate n such that Cay p SL p 2 , Z n q ; S n q would have | V | nodes.
What we can do in practice, as an approximation, is choose the smallest n such that the number of nodes of Cay p SL p 2 , Z n q ; S n q is ě | V | , then consider
A Cay p n q 1: | V | , 1: | V | —i.e. only the subgraph containing the first | V | nodes in the Cayley graph.
There is a slight misalignment to our theory in this specific slicing choice—if the | V | vertices in this subgraph are chosen completely arbitrarily, we risk disconnecting the graph.
However, in all our experiments we construct the Cayley graph in a breadth-first manner, starting from the identity element as “node zero”.
Hence, the node at index i is always guaranteed to be reachable from the nodes at lower indices ( j ă i ), and the graph cannot be disconnected under this construction.
More interesting strategies for this step can also be considered in the future, in order to optimise the communicativity properties of this subgraph.
Note that we do not need to perform matching of the nodes in the original graph to the nodes of the
Cayley graph.
This is because, much like the fully connected graph used by [18], we interpret the
Cayley graph mainly as a template for global information propagation, in order to relieve bottlenecks in a scalable way.
Algorithm 1 summarises the steps of our proposed EGP model.
As direct corollaries of results we proved or demonstrated, we note that EGP satisfies all four of our desirable criteria: ( C1 ) by [-Theorem
-]Theorem 5 (so long as logarithmically many layers are applied), ( C2 ) by Theorem 4 (high Cheeger constant implies no bottlenecks), ( C3 ) by the fact our Cayley graphs are 4-regular and hence sparse, and ( C4 ) by the fact we can generate a Cayley graph of appropriate size without detailed analysis of the input—we may precompute a “bank” of Cayley graphs of various sizes to use in an ad-hoc manner. Algorithm
Inputs : Node features X P R | V |ˆ k , Adjacency matrix A P R | V |ˆ| V |
Output : Node embeddings H //
Choose the smallest Cayley graph from our family that has number of nodes equal to, or greater than, | V | n Ð argmin m P N | V p Cay p SL p 2 , Z m q ; S m qq| ě | V | ; // We can use Equation 10 to determine n
G Cay p n q Ð Cay p SL p 2 , Z n q ; S n q
A Cay p n q uv Ð " 1 p u, v q P E
p G Cay p n q q 0 otherwise ; //
Populate adjacency matrix of the Cayley graph
H p 0 q Ð X ; // Initialise GNN inputs for t P t 1 , . .
, T u do if t mod 2 “ 0 then
H p t q Ð GNN p t q p H p t ´ 1 q , A q ; // GNN layer over input graph; e.g. Equationendelse
H p t q Ð GNN Cay ´
H p t ´ 1 q , A Cay p n q 1: | V | , 1: | V | ¯ ; // GNN layer over Cayley graph; e.g. Equationend end return H p T q ; // Return final embeddings for downstream use
Empirical evaluation
Our work provides mainly a theoretical contribution: demonstrating a simple, theoretically-grounded approach to relieving bottlenecks and oversquashing in GNNs without requiring quadratic complexity or dedicated preprocessing.
Further, we prove several additional results which deepen our understanding of curvature-based analysis of GNNs, showing how our expanders can be favourable in spite of their negatively-curved edges.
We now provide several direct comparative experiments in order to ascertain that our EGP addition can directly help existing graph classification baselines, even without further hyperparameter tuning.
Datasets To show this, we leverage the established Open Graph Benchmark collection of tasks [54, OGB].
Specifically, we provide results on all of its graph classification datasets: ogbg-molhiv , ogbg-molpcba , ogbg-ppa and ogbg-code2 .
The first two are among the largest molecule property prediction datasets in the MoleculeNet benchmark [55].
The third dataset is concerned with classifying
[Table] [Table]
species into their taxa, from their protein-protein association networks [56, 57] given as input.
The fourth dataset is a code summarisation task: it requires predicting the tokens in the name of a Python method, given the abstract syntax tree (AST) of its implementation.
We provide a summary of important dataset statistics in Table 2; please see [54] for detailed information on the data.
These datasets are designed to span a wide variety of domains (virtual drug screening, molecular activity prediction, protein-protein interactions, code summarisation) and sizes (from small molecules to very large syntax trees—the largest graph in ogbg-code2 has 36 , 123 nodes).
Models In all four datasets, we want to directly evaluate the empirical gain of introducing an EGP layer and completely rule out any effects from parameter count, or similar architectural decisions.
To enable this, we take inspiration from the experimental setup of [18].
Our baseline model is the
GIN [53], with hyperparameters as given by [54].
We use the official publicly available model implementation from the OGB authors [54], and modify all even layers of the architecture to operate over the appropriately-sampled Cayley graph.
Note that our construction leaves both the parameter count and latent dimension of the model unchanged , hence any benefits coming from optimising those have been diminished.
Results
The results of our evaluation are presented in Table 3.
It can be observed that, in all four cases, propagating information over the Cayley graph yields improvements in mean performance—these improvements are most apparent on ogbg-molhiv , where our approach significantly outperforms even the “virtual node” version of GIN, which uses „ 1 . 8 ˆ more parameters and achieves 0 .
7707 ˘ 0 .[-
-]
AUC [54].
We believe that these results provide encouraging empirical evidence that propagating information over Cayley graphs is an elegant idea for alleviating bottlenecks.
Conclusion
In this paper, we have presented expander graph propagation (EGP), a novel and elegant approach to alleviating bottlenecks in graph representation learning, which provably supports global communication while not requiring quadratic complexity or dedicated preprocessing of the input.
To this end, we offered a detailed theoretical overview of Cayley graphs of special linear groups,
Cay p SL p 2 , Z n q ; S n q .
We cite proofs that these graphs have highly favourable properties for information propagation in graph neural networks: they are sparse and 4 -regular, they have logarithmic diameter, and they can be efficiently precomputed by a simple procedure that does not rely on the input structure.
We show that, in spite of having negatively curved edges, our findings do not violate any prior results on understanding oversquashing via curvature.
Even under a simple intervention— interleaving EGP layers inbetween standard GNN layers—we have been able to recover significant performance returns without changing the parameter count or latent space dimensionality.
We hope that our work serves as a foundation for further work on deploying Cayley graphs—or other expander families—within the context of GNNs.
A Proof of Proposition
Let s be one of ˆ 10˙ , ˆ 11˙ .
Let n, n 1 ą 18 and let e and e 1 be s -labelled edges in G n and G n 1 .
Then there is a graph isomorphism between N 2 p e q and N 2 p e 1 q taking e to e 1 .
Proof.
Note first that, by the homogeneity of the Cayley graphs G n and G n 1 , we may assume that e and e 1 emanate from the identity vertex of each graph.
Let G 8 be the Cayley graph of SL p 2 , Z q with respect to the generators
[Equation]
Let e 8 be the s -labelled edge emanating from the identity vertex of G 8 .
The quotient homomorphism
[Equation]
q induces a graph homomorphism G 8 Ñ G n sending e 8 to e .
We will show that it restricts to a graph isomorphism
[Equation]
As there is a similar graph isomorphism N 2 p e 8 q Ñ N 2 p e 1 q , the proposition will follow.
Note that two elements of SL p 2 , Z q map to the same element of SL p 2 , Z n q if and only if they differ by multiplication by an element of the kernel K n .
This is
[Equation]
The graph homomorphism sends edges to edges, and so it is distance non-increasing.
Hence it certainly sends N 2 p e 8 q to N 2 p e q .
It is also clearly surjective, because any element of N 2 p e q is reached from an endpoint of e by a path of length at most 2 , and there is a corresponding path in
N 2 p e 8 q .
We just need to show that this is an injection.
If not, then two distinct vertices g 1 and g 2 in N 2 p e 8 q map to the same vertex in N 2 p e q .
Note then that as elements of SL p 2 , Z q , g 2 “ g 1 k for some k P K n .
There are paths with length at most 3 joining the identity 1 to g 1 and g 2 respectively.
Hence, the distance in G 8 between g 1 and g 2 is at most 6 .
Therefore, the distance between 1 and g ´ 1 1 g 2 is at most 6 .
This element g ´ 1 1 g 2 lies in K n .
We will show that when n ą 18 , the only element of K n that has distance at most 6 from the identity is the identity itself.
This will imply that g ´ 1 1 g 2 “ 1 and hence g 1 “ g 2 .
But this contradicts the assumption that g 1 and g 2 are distinct vertices.
Our argument follows that of [58].
The operator norm ||
A || of a matrix A P SL p 2 , Z q is
[Equation]
This is submultiplicative: || AB || ď
||
A || || B
|| for matrices A and B .
It can be calculated as the square root of the largest eigenvalue of A t A .
In our case, the operator norms satisfy
[Equation]
Consider an element
[Equation]
of K n that is not the identity.
Since a ” d ” 1 modulo n and b ” c ” 0 modulo n , we deduce that at least one | a | , | b | , | c | and | d | is at least n ´ 1 .
Therefore, this matrix acts on one of the vectors p 1 , 0 q t or p 0 , 1 q t by scaling its length by at least n ´ 1 .
Therefore, || K || ě n ´ 1 . Suppose now that K has distance at most 6 from the identity.
Then K can be written as a word in the generators of SL p 2 , Z q with length at most 6 .
Therefore, we obtain the inequality
[Equation]
Hence, n ă 18 .
95 and therefore, as n is integral, n ď 18 .
B Proof of Theorem
For any δ ą 0 and ∆ ą 0 , there are only finitely many graphs with maximum vertex degree ∆ , Cheeger constant at least δ and non-negative Ollivier curvature.
Proof.
This is a consequence of the main result of Salez [52, Theorem 3].
This states if G n “ p V n , E n q is a sequence of graphs with the following properties:
[Equation]
then
[Equation]
Here, κ p e q is the Ollivier curvature of an edge e and
[Equation]
are the eigenvalues of the lazy random walk operator.
To prove the theorem, we suppose that on the contrary, there are infinitely many distinct graphs G n “ p V n , E n q with maximum vertex degree ∆ , Cheeger constant at least δ and non-negative Olliver curvature.
Then ÿ
[Equation]
and so condition 16 is satsfied.
Condition 17 is trivially satisfied because the Ollivier curvature of each graph is non-negative.
Thus, we deduce that the conclusion of Salez’ theorem holds.
Setting ρ “ 1 ´ p δ 2 { 4∆ 2 q , we deduce that a definite proportion of the eigenvalues of the lazy random walk operator are at least 1 ´ p
δ 2 { 4∆ 2 q .
In particular, µ 1 p G n q ě 1 ´ p δ 2 { 4∆ 2 q .
Denote the eigenvalues of the normalised Laplacian by
[Equation]
These are related to the eigenvalues of the lazy random walk operator by λ
[Equation]
Hence, λ 1 1 p G n q ď δ 2 {p 2∆ 2 q .
There is a variation of Cheeger’s inequality that relates λ 1 1 to the conductance of the graph.
To define this, one considers subsets A of the vertex set, and defines their volume to be vol p
A q “ ř v P
A deg p v q .
The conductance ϕ
p G q of a graph G is
[Equation]
Then, by Chung [39, Theorem 2.2],
[Equation]
Hence, in our case, ϕ p G n [Equation] ď
[Equation]
Consider any subset A n of the vertex set that realises ϕ p G n q .
Thus
ă vol p A n q ď vol p V n q{ 2 and |B
A n |{ vol p A n q “ ϕ p G n q ď δ { ∆ .
If A n is at most half the vertices of G n , then this implies that the
Cheeger constant h p G n q ď δ .
On the other hand, if A n is more than half the vertices of G n , we consider its complement A cn .
Its cardinality | A cn | satisfies
[Equation]
{ ∆ .
Hence,
[Equation]
In either case, we deduce that the Cheeger constant of G n is at most δ , contradicting one of our hypotheses.
Hence, there must have been only finitely many graphs satisfying the conditions of the theorem.
C Cayley graph at infinity is quasi-isometric to a tree
As all vertices of G n look the same, we focus attention on N r p 1 q , the r -neighbourhood of the identity vertex.
The proof of Proposition 11 immediately gives the following.
Proposition 18.
Let r be a positive integer satisfying
[Equation]
Then there is a graph isomorphism between the r -neighbourhood of the identity vertex in G n and the r -neighbourhood of the identity vertex in G 8 .
This isomorphism takes the identity vertex to the identity vertex.
Proof.
As shown in the proof of Proposition 11, there is a graph homomorphsm from N r p 1 q in Gto N r p 1 q in G n that is a surjection.
If it fails to be an injection, then there is a non-trivial element K in the kernel K n of SL p 2 , Z q Ñ SL p 2 , Z n q satisfying
[Equation]
But any non-trivial element K in K n satisfies
[Equation]
Rearranging gives the required inequality.
This raises the question of the local structure of G 8 .
The answer is well-known: it is ‘tree-like’.
Specifically, it is quasi-isometric to a tree.
The formal definition of quasi-isometry is as follows.
Definition 19.
A quasi-isometry between two metric spaces p X 1 , d 1 q and p X 2 , d 2 q is a function f : X 1 Ñ X 2 that satisfies the following two conditions:
there are constants c, C ą 0 such that, for every x, x 1 P Xc d 1 p x, x 1 q ´ c ď d 2 p f p x q , f p x 1 qq ď C d 1 p x, x 1 q ` C, 2.
there is a constant K ě 0 such that for every y P X 2 , there is an x P X 1 with d 2 p f p x q , y q ď K .
If there is such a quasi-isometry, we say that p X 1 , d 1 q and p X 2 , d 2 q are quasi-isometric .
This forms an equivalence relation on metric spaces.
When two metric spaces are quasi-isometric, they are viewed as being ‘essentially the same’ at large scales.
When S and S 1 are finite generating sets for a group Γ , the graphs Cay p Γ; S q and Cay p Γ; S 1 q are quasi-isometric.
Hence, the quasi-isometry type of a finitely generated group is well-defined, and this is the central object of study in geometric group theory.
The group SL p 2 , Z q has a finite-index subgroup that is a free group F [59].
If S 1 denotes a free generating set for F , then Cay p F ; S 1 q is a tree.
As passing to a finite-index subgroup preserves its quasi-isometry class, we deduce that the Cayley graph G 8 “ Cay p SL p 2 , Z q ; S 8 qq is indeed quasi-isometric to a tree, as claimed above.
D Mixing time properties of expander graphs
Expanders are well known to have small mixing time, in the following sense.
Let G be a graph.
We will consider probability distributions π on V p G q .
The lazy random walk operator M acts on probability distributions as follows.
We think of π p v q as being the probability of the random walk being at vertex v .
If the current location of the walk is at v , then at the next step of the walk, either we stay put with probability 1 { 2 or we move to one of its neighbours with equal probability.
Then Mπ is the new probability distribution.
In the case when G is k -regular, this takes a particular simple form.
The operator M is represented by the matrix p 1 { 2 q I ` p 1 { 2 k q A , where A is the adjacency matrix.
In that case, any initial distribution π converges under powers of M to the uniform distribution.
This is true for any reasonable notion of convergence, but we will use the } ¨ } 1 norm, where for two probability distributions π and π 1 ,
[Equation]
Definition 20.
The mixing time for a regular graph G is the minimum value of ℓ such that for any starting probability distribution π on the vertex set of G ,
[Equation]
Here, u is the uniform probability distribution on the vertex set, and M is the lazy random walk operator.
Expanders have small mixing times in the following very strong sense.
Theorem 21.
For any k ą 0 and δ ą 0 , there is a constant c ą 0 with the following property.
If G is a connected k -regular graph on n vertices with Cheeger constant at least δ ą 0 , then the mixing time for G is at most c log p n q .