Extended Abstract Track
Expander Graph Propagation
Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Abstract
Deploying graph neural networks (GNNs) on whole-graph classification or regression tasks is challenging, often requiring node features that are mindful of both local interactions and the graph global context.
GNN architectures need to avoid pathological behaviours, such as bottlenecks and oversquashing, while ideally having linear time and space complexity requirements.
In this work, we propose an elegant approach based on propagating information over expander graphs .
We provide an efficient method for constructing expander graphs of a given size, and use this insight to propose the EGP model.
We show that EGP is able to address all of the above concerns, while requiring minimal effort to set up, and provide evidence of its empirical utility on relevant datasets and baselines in the Open
Graph Benchmark.
Importantly, using expander graphs as a template for message passing necessarily gives rise to negative curvature.
While this appears to be counterintuitive in light of recent related work on oversquashing, we theoretically demonstrate that negatively curved edges are likely to be required to obtain scalable message passing without bottlenecks.
1. Introduction
Most GNNs rely on propagating information between neighbouring nodes in the graph (Bronstein et al., 2021).
However, in many areas of scientific interest, purely local interactions are likely to be insufficient, while merely stacking more message passing layers over the input graph is often inadequate, as it leaves GNNs vulnerable to pathological behaviours such as oversquashing (Alon and Yahav, 2020), wherein nodes close to bottlenecks in the graph would need to store quantities of information that are exponentially increasing with depth.
Precisely, we seek a method that satisfies four desirable criteria: ( C1 ) it is capable of propagating information globally in the graph; ( C2 ) it is resistant to the oversquashing effect and does not introduce bottlenecks; ( C3 ) its time and space complexity remain subquadratic ; and ( C4 ) it requires no dedicated preprocessing of the input.
We survey prior methods in detail within Appendix A, demonstrating they do not accomplish ( C1 – C4 ) simultaneously.
In this paper, we identify expander graphs as very attractive objects in this regard.
Specifically, they offer a family of graph structures that are fundamentally sparse , while having low diameter : thus, any two nodes in an expander graph may reach each other in a short number of hops, eliminating bottlenecks (Figure 1).
Further, we will demonstrate an efficient way to construct a family of expander graphs (leveraging known results on the special linear group , SL p 2 , Z n q ).
Once an expander graph of appropriate size is constructed, we perform a certain number of GNN propagation steps over its structure to globally distribute the nodes’ features.
Accordingly, we name our method expander graph propagation ( EGP ).
We also show that, in spite of their negative curvature, our expander graphs never trigger the conditions necessary for the oversquashing results in Topping et al.
(2021) to be applicable, and prove that the existence of negatively curved edges might in fact be required in order to have sparse communication without bottlenecks.
2. Theoretical background
Definition 1 A collection t G i u of finite connected graphs is an expander family if there is a constant c ą 0 such that for all G i in the collection, λ 1 p G i q ě c , where λ 1 p G i q is the second-smallest eigenvalue of the graph Laplacian of G i .
Definition 2 Let G be a finite graph.
For A Ă V p G q , its boundary B A is the collection of edges connecting a node in A to a node not in A .
The Cheeger constant h p G q is defined as
[Equation]
Having a small Cheeger constant is equivalent to the graph having a ‘bottleneck’.
Expander families can be reinterpreted using Cheeger constants, as follows (see, e.g., Alon (1986); Alon and Milman (1985); Dodziuk (1984); Tanner (1984)):
Theorem 3 Let t G i u be a collection of finite connected graphs with a uniform upper bound on their vertex degrees.
Then t G i u is an expander family iff there is a constant ϵ ą 0 such that for all graphs in the collection, h p G i q ě ϵ .
Hence, expanders have higher Cheeger constants and will hence be bottleneck-free .
We also show expanders have favourable diameter ; see Mohar (1991, Theorem 2.3) for a proof.
Extended Abstract Track
Theorem 4 The diameter diam p G q of a graph G satisfies
[Equation]
where ∆ p G q is the maximal degree of any vertex of G .
Hence, if t G i u is an expander family of finite graphs with a uniform upper bound on their vertex degrees, then there is a constant k ą 0 such that for all graphs in the family, diam p G i q ď k log V p G i q .
Therefore, globally propagating information over an expander with | V | nodes only requires
O p log | V |q propagation steps—yielding subquadratic complexity.
To efficiently construct expanders of (roughly) | V | nodes, we leverage results from group theory (Appendix B).
Definition 5 Let Γ be a group with a finite generating set S .
Then the associated Cayley graph Cay p Γ; S q has vertex set Γ and it has an edge g Ñ gs for each g P Γ and each s P S .
The degree of each vertex of a Cayley graph Cay p Γ; S q is 2 | S | .
This is because each vertex g is joined by edges to gs and gs ´ 1 for each s P S .
To preserve sparsity, we are interested in the case where there is a uniform upper bound on | S | ; the key example follows.
For n P N , the special linear group SL p 2 , Z n q denotes the group of 2 ˆ 2 matrices with entries that are integers modulo n and with determinant 1.
One of its generating sets is
S n “
[Equation]
ˆ
[Equation] [Equation] ˙* . Central to our constructions is the following result, for which proofs are given in Kowalski (2019);
Davidoff et al.
(2003), using result of Selberg (1965):
Theorem 6 The family of Cayley graph Cay p SL p 2 , Z n q ; S n q forms an expander family.
3. Local structure of Cayley graphs, and the utility of negative curvature
Recent work (Topping et al., 2021) suggested that the local structure of the graph G used by a GNN plays a major part in the oversquashing effect.
Specifically, negatively curved edges —using either the (balanced) Forman curvature (Forman, 2003) or Ollivier curvature (Ollivier, 2007, 2009)—were shown to be the culprits.
Firstly, and surprisingly, we prove in
Appendix C that edges in G n are never positively curved, with curvatures as low as ´ 1.
However, further qualifying the findings in Topping et al.
(2021), we contend that negative
Ricci curvature is not in itself an impediment to efficient propagation around a GNN.
It was shown in Topping et al.
(2021, Theorem 4) that poor propagation arises when the balanced
Forman curvature is close to ´ 2, specifically if it is at most ´ 2 ` δ ; however, with certainty, δ “ 1 can never be satisfied in the hypotheses of Topping et al.
(2021, Theorem 4).
Furthermore, positive Ricci curvature may have downsides when used for GNNs.
Using the main result of Salez (2021), we can show that the three properties of expansion, sparsity and non-negative Ollivier curvature are incompatible, in the following sense.
Theorem 7 For any δ ą 0 and ∆ ą 0 , there are only finitely many graphs with maximum vertex degree ∆ , Cheeger constant at least δ and non-negative Ollivier curvature. Extended Abstract Track [Table]
We prove Theorem 7 in Appendix E.
In our view, it is highly desirable that graphs used for GNNs have high Cheeger constants, in the sense of globally lacking bottlenecks.
Having bounded degree is useful too, as graphs will be sparse, and the nodes will not have to handle larger neighbourhoods as graphs grow larger in size.
As we have just shown, non-negative
Ollivier curvature is incompatible with these properties when the graph is sufficiently large.
4. Expander graph propagation
Our GNN input contains a node feature matrix X P R | V |ˆ k , and an adjacency matrix
A P R | V |ˆ| V | .
We ignore edge features for simplicity, without changing our findings.
The crux of our method is leveraging the computed Cayley graph Cay p SL p 2 , Z n q ; S n q for message propagation.
We opt for a simple construction: interleave running a standard
GNN over the given input structure, followed by running another GNN layer over the Cayley graph.
If A Cay p n q is the adjacency matrix derived from Cay p SL p 2 , Z n q ; S n q , this implies
H “ GNN p
GNN
p X [Equation] A q , A Cay p n q q .
Here, we use the GIN (Xu et al., 2018) as our GNN:
[Equation]
where N u is the neighbourhood of node u , i.e. in our setup, the set of all nodes v such that a vu ‰ 0.
ϵ P R is a learnable scalar, and ϕ : R k Ñ R k 1 is a two-layer MLP.
We assumed the number of nodes in our input graph to line up with the Cayley graph, that is, A Cay p n q P R | V |ˆ| V | .
However, there is no guarantee that n P N exists, such that
Cay p SL p 2 , Z n q ; S n q has | V | nodes.
As an approximation, we choose the smallest n such that the number of nodes of Cay p SL p 2 , Z n q ; S n q is ě | V | , then consider A Cay p n q 1: | V | , 1: | V | —i.e. only the subgraph containing the first | V | nodes in the Cayley graph.
Further, in all our experiments we construct the Cayley graph in a breadth-first manner, starting from the identity element as “node zero”—this guarantees that we do not disconnect the graph by taking this subgraph.
Appendix I summarises the steps of our proposed EGP model in pseudocode.
5. Empirical evaluation
We provide direct comparative experiments in order to supplement our theory, and ascertain that EGP can directly help existing graph classifiers, even without extensive tuning.
We leverage the Open Graph Benchmark (Hu et al., 2020, OGB) graph classification datasets: ogbg-molhiv , ogbg-molpcba , ogbg-ppa and ogbg-code2 .
In all four datasets, we want to directly evaluate the empirical gain of introducing an
EGP layer and completely rule out any effects from parameter count, or similar architectural Extended Abstract Track decisions.
Our baseline model is the GIN (Xu et al., 2018), with hyperparameters as given by (Hu et al., 2020).
We use the official publicly available model implementation from the
OGB authors (Hu et al., 2020), and modify all even layers of the architecture to operate over the appropriately-sampled Cayley graph.
Note that our construction leaves both the parameter count and latent dimension of the model unchanged , hence any benefits coming from optimising those have been diminished.
The results of our evaluation are presented in Table 1.
It can be observed that, in all four cases, propagating information over the Cayley graph yields improvements in mean performance—these improvements are most apparent on ogbg-molhiv .
We believe that these results provide encouraging empirical evidence that propagating information over
Cayley graphs is an elegant idea for alleviating bottlenecks
Appendix A. An in-depth analysis of prior art
In this appendix, we survey many of prior approaches to handling global context in graph representation learning, evaluating them carefully against our four desirable criteria ( C1 – C4 ; cf.
Table 2).
This list is by no means exhaustive, but should be indicative of the most important directions.
Stacking more layers.
As already highlighted, one way to achieve global information propagation is to have a deeper GNN.
In this case, we are capable of satisfying ( C1 ) and ( C4 )—no dedicated preprocessing is needed.
However, depending on the graph’s diameter, we may need up to O p| V |q layers to cover the graph, leading to quadratic complexity (violating ( C3 )) and introducing a vulnerability to bottlenecks ( C2 ), as theoretically and empirically demonstrated in (Alon and Yahav, 2020).
Master nodes.
An attractive approach to introducing global context is to introduce a master node to the graph, and connect it to all of the graph’s nodes.
This can be done either explicitly (Gilmer et al., 2017) or implicitly, by storing a “global” vector (Battaglia et al., 2018).
It trivially reduces the graph’s diameter to 2, introduces O p 1 q new nodes and
O p| V |q new edges, and requires no dedicated preprocessing, hence it satisfies ( C1 , C3 , C4 ).
However, these benefits come at the expense of introducing a bottleneck in the master node: it has a very challenging task (especially when graphs get larger) to continually incorporate information over a very large neighbourhood in a useful way.
Hence it fails to satisfy ( C2 ).
Fully connected graphs.
The converse approach is to make every node a master node: in this case, we make all pairs of nodes connected by an edge—this was initially proposed as a powerful method to alleviate oversquashing by (Alon and Yahav, 2020).
This strategy proved highly popular in the recent surge of Graph Transformers (Kreuzer et al., 2021; Mialon et al., 2021;
Ying et al., 2021), and is common for GNNs used in physical simulation (Battaglia et al., 2016) or reasoning (Santoro et al., 2017) tasks.
The graph’s diameter is reduced to 1, no bottlenecks remain, and the approach does not require any dedicated preprocessing.
Extended Abstract Track
Hence ( C1 , C2 , C4 ) are trivially satisfied.
The main downside of this approach is the introduction of O p| V | 2 q edges, which means ( C3 ) can never be satisfied—and this approach will hence be prohibitive even for modestly-sized graphs.
Feature augmentation.
An alternative approach is to provide additional features to the GNN which directly identify the structural role each node plays in the graph structure (Bouritsas et al., 2022).
If done properly (i.e., if the computed features are directly relevant to the target task), this can drastically improve expressive power.
Hence, in theory, it is possible to satisfy ( C1 ) while not violating ( C2 , C3 ).
However, computing appropriate features requires either specific domain knowledge, or an appropriate pre-training procedure (Grover and Leskovec, 2016; Perozzi et al., 2014; Tang et al., 2015; Thakoor et al., 2021;
Veliˇckovi´c et al., 2018) to be applied, in order to obtain such embeddings.
Hence all of these gains come at the expense of failing to satisfy ( C4 ).
Graph rewiring.
Another promising line of research involves modifying the edges of the original graph, in order to alleviate bottlenecks.
Popular examples of this approach involve using diffusion (Gasteiger et al.
, 2019)—which diffuse additional edges through the application of kernels such as the personalised PageRank, and stochastic discrete Ricci flows (Topping et al.
, 2021)—which surgically modify a small quantity of edges to alleviate the oversquashing effect on the nodes with negative Ricci curvature.
If realised carefully, such approaches will not deviate too far from the original graph, while provably alleviating oversquashing; hence it is possible to satisfy ( C1 , C2 , C3 ).
However, this comes at a cost of having to examine the input graph structure, with methods that do not necessarily scale easily with the number of nodes.
As such, dedicated preprocessing is needed, failing to satisfy ( C4 ).
Hierarchical message passing.
Lastly, going beyond modifying the edges, it is also possible to introduce additional nodes in the graph—each of them responsible for a particular substructure in the graph 1 .
If done carefully, it has the potential to drastically reduce the graph’s diameter while not introducing bottlenecked nodes (hence, allowing us to satisfy ( C1 , C2 )).
However, in prior work, a cost has to be paid for this, usually in the need for dedicated preprocessing.
Prior proposals for hierarchical GNNs that remain scalable require a dedicated pre-processing step (Bodnar et al., 2021a,b; Fey et al., 2020), sometimes coupled with domain knowledge (Fey et al., 2020)—thus failing to satisfy ( C4 ).
In addition, such methods may require adding prohibitively large numbers of substructures (Morris et al., 2020, 2019) or expensive pre-computation, e.g. computing the graph Laplacian eigenvectors (Stachenfeld et al., 2020).
This might make even ( C3 ) hard to satisfy.
We remark that our work is not the first to study expander graph-related topics in the context of GNNs.
Specifically, the ExpanderGNN (Lutzeyer et al., 2021) leverages expander graphs over neural network weights to sparsify the update step in GNNs, and the Cheeger constant has been previously used to quantify oversquashing in (Topping et al., 2021).
With respect to our contributions, neither of these cases discuss expander graphs in the context of the computational graph for a GNN, nor attempt to propagate messages over such a structure.
Further, neither of these proposals successfully satisfies all four of our desired criteria ( C1 – C4 ).
Extended
Appendix B. Extended theoretical background
Definition 8 For a finite connected graph G “ p V p G q , E p G qq , we consider functions f : V p G q Ñ R .
The Laplacian Lf : V p G q Ñ R of such a function is defined to be
Lf p v q “ deg p v q f p v q ´ ÿ vw P E p G q f p w q , where deg p v q is the degree of the vertex v .
The mapping L : R V p G q Ñ R V p G q sending a function f to its Laplacian Lf is a linear transformation.
It is not hard to show Chung (1997) that L is symmetric with respect to the standard basis for R V p G q and positive semi-definite and hence has non-negative real eigenvalues
[Equation]
The smallest eigenvalue is 0 and its associated eigenspace consists of the constant functions (assuming G is connected).
The smallest positive eigenvalue, λ 1 p G q , is central to the definition of expander graphs, as the next definition shows.
Definition 9 A group p Γ , ˝q is a set Γ equipped with a composition operation
˝ : Γ ˆ Γ Ñ Γ (written concisely by omitting ˝ , i.e. g ˝ h “ gh , for g, h P Γ ), satisfying the following axioms:
• (Associativity) p gh q l “ g p hl q , for g, h, l P Γ .
• (Identity)
There exists a unique e P Γ satisfying eg “ ge “ g for all g P Γ .
• (Inverse)
For every g P Γ there exists a unique g ´ 1 P Γ such that gg ´ 1 “ g ´ 1 g “ e .
Definition 10 Let Γ be a group.
A subset S Ď Γ is a generating set for Γ if it can be used to “generate” all of Γ via composition.
Concretely, any element g P Γ can be expressed by composing elements in the generating set, or their inverses; that is, we can express g “ s ˘ 1 1 s ˘ 1 2 s ˘
1 3 ¨ ¨ ¨ s ˘ 1 n ´ 1 s ˘ 1 n for s i P S .
Appendix C. Cayley graphs never have positive curvature
The various notions of curvature we discussed are defined for each e of the graph G .
Since, as defined by (Topping et al., 2021), the balanced Forman curvature of an edge depends only on local structures (i.e. triangles and squares) around that edge, they can be determined by only observing the immediate 2-hop surrounding of that edge.
Formally, for an edge e of a graph G , let N 2 p
e q be the induced subgraph with vertices that are at most two hops away from at least one endpoint of e .
Then the curvature of e only depends on the isomorphism type of N 2 p e q .
More specifically, if e and e 1 are edges in possibly distinct graphs, and there is a graph isomorphism between N 2 p e q and N 2 p e 1 q that sends e to e 1 , then this guarantees that the curvatures of e and e 1 are equal.
This situation arises prominently in the Cayley graphs that we are considering, as follows.
Extended Abstract Track
Proposition 11 Let s be one of ˆ {+
 [Equation] ˙+} , ˆ 11˙ .
Let n, n 1 ą 18 and let e and e 1 be s -labelled edges in G n and G n 1 .
Then there is a graph isomorphism between N 2 p e q and N 2 p
e 1 q taking e to e 1 .
We prove Proposition 11 in Appendix D. This immediately allows us to characterise the balanced Forman curvature and Ollivier curvature for all of the Cayley graphs we generate:
Proposition 12 The balanced Forman curvatures Ric p n q , and the Ollivier curvatures κ p n q of all edges of Cayley graphs G n are given by:
[Equation]
Proof Proposition 11 implies that the balanced Forman and Ollivier curvatures are all equal for n ą 18.
Their values for 2 ď n ď 19 can all be empirically computed, and are given as above.
Appendix D. Proof of Proposition
Let s be one of ˆ 10˙ , ˆ 11˙ .
Let n, n 1 ą 18 and let e and e 1 be s -labelled edges in G n and G n 1 .
Then there is a graph isomorphism between N 2 p e q and N 2 p
e 1 q taking e to e 1 .
Proof Note first that, by the homogeneity of the Cayley graphs G n and G n 1 , we may assume that e and e 1 emanate from the identity vertex of each graph.
Let G 8 be the Cayley graph of SL p 2 , Z q with respect to the generators
[Equation]
Let e 8 be the s -labelled edge emanating from the identity vertex of G 8 .
The quotient homomorphism
[Equation]
induces a graph homomorphism G 8 Ñ G n sending e 8 to e .
We will show that it restricts to a graph isomorphism
[Equation]
Extended Abstract Track
As there is a similar graph isomorphism N 2 p e 8 q Ñ N 2 p e 1 q , the proposition will follow.
Note that two elements of SL p 2 , Z q map to the same element of SL p 2 , Z n q if and only if they differ by multiplication by an element of the kernel K n .
This is
K n “ "ˆ
[Equation]
, Z
[Equation]
mod n and b
[Equation]
c
[Equation]
0 mod n * .
The graph homomorphism sends edges to edges, and so it is distance non-increasing.
Hence it certainly sends N 2 p e 8 q to N 2 p e q .
It is also clearly surjective, because any element of N 2 p e q is reached from an endpoint of e by a path of length at most 2, and there is a corresponding path in N 2 p e 8 q .
We just need to show that this is an injection.
If not, then two distinct vertices g 1 and g 2 in N 2 p e 8 q map to the same vertex in N 2 p e q .
Note then that as elements of SL p 2 , Z q , g 2 “ g 1 k for some k P K n .
There are paths with length at most 3 joining the identity 1 to gand g 2 respectively.
Hence, the distance in G 8 between g 1 and g 2 is at most 6.
Therefore, the distance between 1 and g ´ 1 1 g 2 is at most 6.
This element g ´ 1 1 g 2 lies in K n .
We will show that when n ą 18, the only element of K n that has distance at most 6 from the identity is the identity itself.
This will imply that g ´ 1 1 g 2 “ 1 and hence g 1 “ g 2 .
But this contradicts the assumption that g 1 and g 2 are distinct vertices.
Our argument follows that of Margulis (1982).
The operator norm ||
A || of a matrix A P SL p 2 , Z q is
[Equation]
This is submultiplicative:
[Equation]
B || for matrices A and B .
It can be calculated as the square root of the largest eigenvalue of A t A .
In our case, the operator norms satisfy
[Equation]
Consider an element
K
[Equation]
of K n
that is not the identity.
Since a ” d ” 1 modulo n and b ” c ” 0 modulo n , we deduce that at least one | a | , | b | , | c | and | d | is at least n ´ 1.
Therefore, this matrix acts on one of the vectors p 1 , 0 q t or p 0 , 1 q t by scaling its length by at least n ´ 1.
Therefore, || K || ě n ´ 1.
Suppose now that K has distance at most 6 from the identity.
Then K can be written as a word in the generators of SL p 2 , Z q with length at most 6.
Therefore, we obtain the inequality
[Equation]
Hence, n ă 18 .
95 and therefore, as n is integral, n ď 18.
Extended Abstract Track
Appendix E. Proof of Theorem
For any δ ą 0 and ∆ ą 0 , there are only finitely many graphs with maximum vertex degree ∆ , Cheeger constant at least δ and non-negative Ollivier curvature.
Proof This is a consequence of the main result of Salez (Salez, 2021, Theorem 3).
This states if G n “ p V n , E n q is a sequence of graphs with the following properties:
[Equation]
then
[Equation]
Here, κ p e q is the Ollivier curvature of an edge e and
[Equation]
are the eigenvalues of the lazy random walk operator.
To prove the theorem, we suppose that on the contrary, there are infinitely many distinct graphs G n “ p V n , E n q with maximum vertex degree ∆, Cheeger constant at least δ and non-negative Olliver curvature.
Then
[Equation]
and so condition 2 is satsfied.
Condition 3 is trivially satisfied because the Ollivier curvature of each graph is non-negative.
Thus, we deduce that the conclusion of Salez’ theorem holds.
Setting ρ “ 1 ´ p δ 2 { 4∆ 2 q , we deduce that a definite proportion of the eigenvalues of the lazy random walk operator are at least 1 ´ p
δ 2 { 4∆ 2 q .
In particular, µ 1 p G n q ě 1 ´ p δ 2 { 4∆ 2 q .
Denote the eigenvalues of the normalised Laplacian by
[Equation]
These are related to the eigenvalues of the lazy random walk operator by λ 1 i p G n q “ 2 ´ 2 µ i p G n q .
Hence, λ 1 1 p G n q ď δ 2 {p 2∆ 2 q .
There is a variation of Cheeger’s inequality that relates λ 1 1 to the conductance of the graph.
To define this, one considers subsets A of the vertex set, and defines their volume to be vol p
A q “ ř v P
A deg p v q .
The conductance ϕ
p G q of a graph G is
[Equation]
Then, by Chung (1997, Theorem 2.2),
[Equation]
Hence, in our case,
[Equation]
Extended Abstract Track
Consider any subset A n of the vertex set that realises ϕ p G n q .
Thus
ă vol p A n q ď vol p V n q{and |B
A n |{ vol p A n q “ ϕ p G n q ď δ { ∆.
If A n is at most half the vertices of G n , then this implies that the Cheeger constant h p G n q ď δ .
On the other hand, if A n is more than half the vertices of G n , we consider its complement A cn .
Its cardinality | A cn | satisfies
[Equation]
Hence,
[Equation]
In either case, we deduce that the Cheeger constant of G n is at most δ , contradicting one of our hypotheses.
Hence, there must have been only finitely many graphs satisfying the conditions of the theorem.
Appendix F. Cayley graphs are locally ‘tree-like’
The negative curvature of each edge in G n implies that they are locally ‘tree-like’.
In
Appendix G, we make this statement precise by showing that G n is ‘tree-like’ up to scale c log p n q about each node, for c » p 1 { 2 qp log pp 1 ` ?
5 q{ 2 qq ´ 1 (see Figure 1 (Right) for a schematic view).
This tree-like structure might seem, at first, to be counter-productive for good propagation across the graphs G n .
Indeed, GNNs based on trees have been shown to have provably poor performance (Alon and Yahav, 2020).
The reason for this seems to be two-fold.
On the one hand, trees have small Cheeger constant.
Indeed, any tree G on n vertices has a Cheeger constant 1 { t n { 2 u , since we may find an edge that, when removed, decomposes the graph into subgraphs with t n { 2 u and r n { 2 s vertices.
As discussed in Section 2 and in Topping et al.
(2021), when a graph has small Cheeger constant, its performance when used as a template for a GNN is likely to become poor.
Secondly, GNNs based on trees are susceptible to oversquashing.
For a k -regular infinite tree, there are k p k ´ 1 q r ´ 1 vertices at distance r from a given vertex.
Hence, if information is to be propagated at least distance r from a given vertex, then seemingly an exponential amount of information is required to be stored.
However, neither of these issues are problematic for a GNN based on the Cayley graph
G n .
By Theorem 6, their Cheeger constants are bounded away from 0.
Secondly, although they are tree-like locally, this is only true up to scale O p log n q .
In fact, the r -neighbourhood of any vertex is the whole graph G n as soon as r ą C log n , for some constant C , by Theorem 4.
Being tree-like up to distance O p log n q does not lead to a requirement to store too much information as the message propagates.
This is because k p k ´ 1 q r ´ 1 is polynomial in n when r ď O p log n q .
Appendix G. Cayley graph at infinity is quasi-isometric to a tree
As all vertices of G n look the same, we focus attention on N r p 1 q , the r -neighbourhood of the identity vertex.
The proof of Proposition 11 immediately gives the following.
Extended Abstract Track
Proposition 13 Let r be a positive integer satisfying
[Equation]
Then there is a graph isomorphism between the r -neighbourhood of the identity vertex in G n and the r -neighbourhood of the identity vertex in G 8 .
This isomorphism takes the identity vertex to the identity vertex.
Proof As shown in the proof of Proposition 11, there is a graph homomorphsm from N r p 1 q in G 8 to N r p 1 q in G n that is a surjection.
If it fails to be an injection, then there is a non-trivial element K in the kernel K n of SL p 2 , Z q Ñ SL p 2 , Z n q satisfying
[Equation]
But any non-trivial element K in K n satisfies
[Equation]
Rearranging gives the required inequality.
This raises the question of the local structure of G 8 .
The answer is well-known: it is ‘tree-like’.
Specifically, it is quasi-isometric to a tree.
The formal definition of quasi-isometry is as follows.
Definition 14 A quasi-isometry between two metric spaces p X 1 , d 1 q and p X 2 , d 2 q is a function f : X 1 Ñ X 2 that satisfies the following two conditions:
there are constants c, C ą 0 such that, for every x, x 1 P Xc d 1 p x, x 1 q ´ c ď d 2 p f p x q , f p x 1 qq ď C d 1 p x, x 1 q ` C, 2.
there is a constant K ě 0 such that for every y P X 2 , there is an x P X 1 with d 2 p f p x q , y q ď K .
If there is such a quasi-isometry, we say that p X 1 , d 1 q and p X 2 , d 2 q are quasi-isometric .
This forms an equivalence relation on metric spaces.
When two metric spaces are quasi-isometric, they are viewed as being ‘essentially the same’ at large scales.
When S and S 1 are finite generating sets for a group Γ, the graphs Cay p Γ; S q and
Cay p Γ; S 1 q are quasi-isometric.
Hence, the quasi-isometry type of a finitely generated group is well-defined, and this is the central object of study in geometric group theory.
The group SL p 2 , Z q has a finite-index subgroup that is a free group F (Trees, 2003).
If S 1 denotes a free generating set for F , then Cay p F ; S 1 q is a tree.
As passing to a finite-index subgroup preserves its quasi-isometry class, we deduce that the Cayley graph G 8 “ Cay p SL p 2 , Z q ; S 8 qq is indeed quasi-isometric to a tree, as claimed above.
Extended Abstract Track
Appendix H. Mixing time properties of expander graphs
Expanders are well known to have small mixing time, in the following sense.
Let G be a graph.
We will consider probability distributions π on V p G q .
The lazy random walk operator M acts on probability distributions as follows.
We think of π p v q as being the probability of the random walk being at vertex v .
If the current location of the walk is at v , then at the next step of the walk, either we stay put with probability 1 { 2 or we move to one of its neighbours with equal probability.
Then Mπ is the new probability distribution.
In the case when G is k -regular, this takes a particular simple form.
The operator M is represented by the matrix p 1 { 2 q I ` p 1 { 2 k q A , where A is the adjacency matrix.
In that case, any initial distribution π converges under powers of M to the uniform distribution.
This is true for any reasonable notion of convergence, but we will use the } ¨ } 1 norm, where for two probability distributions π and π 1 ,
[Equation]
Definition 15 The mixing time for a regular graph G is the minimum value of ℓ such that for any starting probability distribution π on the vertex set of G ,
[Equation]
Here, u is the uniform probability distribution on the vertex set, and M is the lazy random walk operator.
Expanders have small mixing times in the following very strong sense.
Theorem
For any k ą 0 and δ ą 0 , there is a constant c ą 0 with the following property.
If G is a connected k -regular graph on n vertices with Cheeger constant at least δ ą 0 , then the mixing time for G is at most c log p n q .
Appendix I. EGP forward pass pseudocode
In Algorithm 1, we provide the forward pass for our EGP layer, linking it back to previous equations.
Extended Abstract
Inputs : Node features X P R | V |ˆ k , Adjacency matrix A P R | V |ˆ| V |
Output : Node embeddings H //
Choose the smallest Cayley graph from our family that has number of nodes equal to, or greater than, | V | n Ð argmin m P N | V p Cay p SL p 2 , Z m q ; S m qq| ě | V |
G Cay p n q Ð Cay p SL p 2 , Z n q ; S n q
A Cay p n q uv Ð # p u, v q P E p G Cay p n q q otherwise ; //
Populate adjacency matrix of the Cayley graph
H p 0 q Ð X ; // Initialise GNN inputs for t P t 1 , . .
, T u do if t mod 2 “ 0 then
H p t q Ð GNN p t q p H p t ´ 1 q , A q ; // GNN layer over input graph; e.g.
Equation end else
H p t q Ð GNN Cay ´
H p t ´ 1 q , A Cay p n q 1: | V | , 1: | V | ¯ ; // GNN layer over Cayley graph; e.g.
Equationend end return H p T q ; //
Return final embeddings for downstream use
Extended Abstract Track