Recipe for a General, Powerful, Scalable Graph Transformer
Abstract
We propose a recipe on how to build a general, powerful, scalable (GPS) graph
Transformer with linear complexity and state-of-the-art results on a diverse set of benchmarks.
Graph Transformers (GTs) have gained popularity in the ﬁeld of graph representation learning with a variety of recent publications but they lack a common foundation about what constitutes a good positional or structural encoding, and what differentiates them.
In this paper, we summarize the different types of encodings with a clearer deﬁnition and categorize them as being local , global or relative .
The prior GTs are constrained to small graphs with a few hundred nodes, here we propose the ﬁrst architecture with a complexity linear in the number of nodes and edges O ( N + E ) by decoupling the local real-edge aggregation from the fully-connected Transformer.
We argue that this decoupling does not negatively affect the expressivity, with our architecture being a universal function approximator on graphs.
Our GPS recipe consists of choosing 3 main ingredients: (i) positional/structural encoding, (ii) local message-passing mechanism, and (iii) global attention mechanism.
We provide a modular framework G RAPH GPS 1 that supports multiple types of encodings and that provides efﬁciency and scalability both in small and large graphs.
We test our architecture on 16 benchmarks and show highly competitive results in all of them, show-casing the empirical beneﬁts gained by the modularity and the combination of different strategies.
Introduction
Graph Transformers (GTs) alleviate fundamental limitations pertaining to the sparse message passing mechanism, e.g., over-smoothing [47], over-squashing [1], and expressiveness bounds [61, 45], by allowing nodes to attend to all other nodes in a graph ( global attention ).
This beneﬁts several real-world applications, such as modeling chemical interactions beyond the covalent bonds [63], or graph-based robotic control [37].
Global attention, however, requires nodes to be better identiﬁable within the graph and its substructures [14].
This has led to a ﬂurry of recently proposed fullyconnected graph transformer models [14, 36, 63, 44, 31] as well as various positional encoding schemes leveraging spectral features [14, 36, 39] and graph features [16, 9].
Furthermore, standard global attention incurs quadratic computational costs O ( N 2 ) for a graph with N nodes and E edges, that limits GTs to small graphs with up to a few hundred nodes.
Whereas various GT models focus on particular node identiﬁability aspects, a principled framework for designing GTs is still missing.
In this work, we address this gap and propose a recipe for building general, powerful, and scalable (GPS) graph Transformers.
The recipe deﬁnes (i) embedding modules responsible for aggregating positional encodings (PE) and structural encodings (SE) with the node, edge, and graph level input features; (ii) processing modules that employ a combination of local message passing and global attention layers (see Figure 1).
The embedding modules organize multiple proposed PE and SE schemes into local and global levels serving as additional node features whereas positional and structural relative features contribute to edge features.
The processing modules deﬁne a computational graph that allows to balance between message-passing graph neural networks (MPNNs) and Transformer-like global attention, including attention mechanisms linear in the number of nodes O ( N ) .
To the best of our knowledge, application of efﬁcient attention models has not yet been thoroughly studied in the graph domain, e.g., only one work [11] explores the adaptation of Performer-style [12] attention approximation on small graphs.
Particular challenges emerge with explicit edge features that are incorporated as attention bias in fully-connected graph transformers [36, 63].
Linear transformers do not materialize the attention matrix directly, hence incorporating edge features becomes a nontrivial task.
In this work, we hypothesize that explicit edge features are not necessary for the global graph attention and adopt Performer [12] and BigBird [66] as exemplary linear attention mechanisms.
Our contributions are as follows.
(i) Provide a general, powerful, scalable (GPS) GT blueprint that incorporates positional and structural encodings with local message passing and global attention, visualized in Figure 1.
(ii) Provide a better deﬁnition of PEs and SEs and organize them into local, global , and relative categories.
(iii) Show that GPS with linear global attention, e.g., provided by
Performer [12] or BigBird [66], scales to graphs with several thousand nodes and demonstrates competitive results even without explicit edge features within the attention module, whereas existing fully-connected GTs [36, 63] are limited to graphs of up to few hundred nodes.
(iv) Conduct an extensive ablation study that evaluates contribution of PEs, local MPNN, and global attention components in perspective of several benchmarking datasets.
(v) Finally, following the success of
GraphGym [65] we implement the blueprint within a modular and performant G RAPH GPS package.
Related Work
Graph Transformers (GT).
Considering the great successes of Transformers in natural language processing (NLP) [55, 32] and recently also in computer vision [18, 25, 24], it is natural to study their applicability in the graph domain as well.
Particularly, they are expected to help alleviate the problems of over-smoothing and over-squashing [1, 54] in MPNNs, which are analogous to the vanishing gradients and lack of long-term dependencies in NLP.
Fully-connected Graph Transformer [14] was ﬁrst introduced together with rudimentary utilisation of eigenvectors of the graph Laplacian as the node positional encoding (PE), to provide the otherwise graph-unaware Transformer a sense of nodes’ location in the input graph.
Building on top of this work, SAN [36] implemented an invariant aggregation of Laplacian’s eigenvectors for the PE, alongside conditional attention for real and virtual edges of a graph, which jointly yielded signiﬁcant improvements.
Concurrently, Graphormer [63, 51] proposed using pair-wise graph distances (or 3D distances) to deﬁne relative positional encodings, with outstanding success on large molecular benchmarks.
Further, GraphiT [44] used relative PE derived from diffusion kernels to modulate the attention between nodes.
Finally, GraphTrans [31] proposed the ﬁrst hybrid architecture, ﬁrst using a stack of MPNN layers, before fully-connecting the graph.
Since, the ﬁeld has continued to propose alternative GTs: SAT [9], EGT [29], GRPE [48].
Positional and structural encodings.
There have been many recent works on PE and SE, notably on
Laplacian PE [14, 36, 3, 57, 39], shortest-path-distance [38, 63], node degree centrality [63], kernel distance [44], random-walk SE [16], structure-aware [9, 6, 5], and more.
Some works also propose dedicated networks to learn the PE/SE from an initial encoding [36, 16, 39, 9].
To better understand the different PE/SE and the contribution of each work, we categorize them in Table 1 and examine their effect in Section 3.2.
In most cases, PE/SE are used as soft bias, meaning they are simply provided as input features.
But in other cases, they can be used to direct the messages [3] or create bridges between distant nodes [35, 54].
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Structural encodings (SE)
[Table]
Graph features
[Table]
[Figure]
Linear Transformers.
The quadratic complexity of attention in the original Transformer architecture [55] motivated the search for more efﬁcient attention mechanisms that would scale linearly with the sequence length.
Most of such linear transformers are developed for language modeling tasks, e.g., Linformer [58], Reformer [34], Longformer [4], Performer [12], BigBird [66], and have a dedicated Long Range Arena benchmark [52] to study the limits of models against extremely long input sequences.
Pyraformer [40] is an example of a linear transformer for time series data, whereas
S4 [23] is a more general signal processing approach that employs the state space model theory without the attention mechanism.
In the graph domain, linear transformers are not well studied.
Choromanski et al.
[11] are the ﬁrst to adapt Performer-style attention kernelization to small graphs.
Methods
In this work we provide a general, powerful, scalable (GPS) architecture for graph Transformers, following our 3-part recipe presented in Figure 1.
We begin by categorization of existing positional (PE) and structural encodings (SE), a necessary ingredient for graph Transformers.
Next, we analyse how these encodings also increase expressive power of MPNNs.
The increased expressivity thus provides double beneﬁt to our hybrid MPNN+Transformer architecture, which we introduce in
Section 3.3.
Last but not least, we provide an extensible implementation of GPS in G RAPH GPS package, built on top of PyG [20] and GraphGym [65].
3.1 Modular positional and structural encodings
One of our contribution is to provide a modular framework for PE/SE.
It was shown in previous works that they are one of the most important factors in driving the performance of graph Transformers.
Thus, a better understanding and organization of the PE and SE will aid in building of a more modular architecture and in guiding of the future research.
We propose to organize the PE and SE into 3 categories: local , global and relative in order to facilitate the integration within the pipeline and facilitate new research directions.
They are presented visually in Figure 1, with more details in Table 1.
Although PE and SE can appear similar to some
Some encodings are assigned to multiple categories in order to show their multiple expected roles.
[Table]
extent, they are different yet complementary.
PE gives a notion of distance, while SE gives a notion of structural similarity.
One can always infer certain notions of distance from large structures, or certain notions of structure from short distances, but this is not a trivial task, and the objective of providing PE and SE remains distinct, as discussed in the following subsections.
Despite presenting a variety of possible functions, we focus our empirical evaluations on the global
PE , relative PE and local SE since they are known to yield signiﬁcant improvements.
We leave the empirical evaluation of other encodings for future work.
Positional encodings (PE) are meant to provide an idea of the position in space of a given node within the graph.
Hence, when two nodes are close to each other within a graph or subgraph, their PE should also be close.
A common approach is to compute the pair-wise distance between each pairs of nodes or their eigenvectors as proposed in [38, 63, 36, 57], but this is not compatible with linear
Transformers as it requires to materialize the full attention matrix [12].
Instead, we want the PE to either be features of the nodes or real edges of the graph, thus a better ﬁtting solution is to use the eigenvectors of the graph Laplacian or their gradient [15, 3, 36].
See Table 1 for more PE examples.
Structural encodings (SE) are meant to provide an embedding of the structure of graphs or subgraphs to help increase the expressivity and the generalizability of graph neural networks (GNN).
Hence, when two nodes share similar subgraphs, or when two graphs are similar, their SE should also be close.
Simple approaches are to identify pre-deﬁned patterns in the graphs as one-hot encodings, but they require expert knowledge of graphs [6, 5].
Instead, using the diagonal of the m -steps random-walk matrix encodes richer information into each node [16], such as for odd m it can indicate if a node is a part of an m -long cycle.
Structural encodings can also be used to deﬁne the global graph structure, for instance using the eigenvalues of the Laplacian, or as relative edge features to identify if nodes are contained within the same clusters, with more examples in Table 1.
3.2 Why do we need PE and SE in MPNN?
As reviewed earlier, several recent GNNs make use of positional encodings (PE) and structural encodings (SE) as soft biases to improve the model expressivity (summarized in Table 1), which also leads to better generalization.
In this section, we present an examination of PE and SE by showing how message-passing networks, despite operating on the graph structure, remain blind to the information encapsulated by the PE and SE.
1-Weisfeiler-Leman test (1-WL).
It is well known that standard MPNNs are as expressive as the 1-WL test, meaning that they fail to distinguish non-isomorphic graphs under a 1-hop aggregation.
We argue that the selected local , global and relative PE/SE allow MPNNs to become more expressive than the 1-WL test, thus making them fundamentally more expressive at distinguishing between nodes and graphs.
To this end, we study the following two types of graphs (Figure 2 and Appendix C.1).
[Table]
/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNWHNz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6reTfW6eV2p1/I4inAG53AJHtxCHe6hAS1ggPAMr/DmPDovzrvzsWwtOPnMKfyB8/kDxM2M5Q==</latexit
[Figure]
b <latexit sha1_
[Table]
="WbjOqWGUu6YMogqMdBIn23mRkDg=">AAAB6HicbVBNS8NAEJ3Ur1q/
[Table]
3azSbsboQS
[Table]
GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+
[Table]
+ [Table] [Figure]
[Table]
[Figure]
Circular Skip Link (CSL) graph.
In a CSL graph-pair [46], we want to be able to distinguish the two non-isomorphic graphs.
Since the 1-WL algorithm produces the same color for every node in both graphs, also every MPNN will fail to distinguish them.
However, using a global PE (e.g., Laplacian PE [15]) assigns each node a unique initial color and makes the CSL graph-pair distinguishable.
This demonstrates that an MPNN cannot learn such a PE from the graph structure alone.
Next, using a local SE (e.g., diagonals of m -steps random walk) can successfully capture the difference in the skip links of the two graphs [42], resulting in their different node coloring [16].
Decalin molecule.
In the bicyclic Decalin graph, Figures 2b and C.1b, the node a is isomorphic to node b , and so is the node c to node d .
A 1-WL coloring of the nodes, and analogously MPNN, would generate one color for the nodes a, b and another color for c, d .
The same applies to the aforementioned local SE [16].
In case of link prediction, this causes potential links ( a, d ) and ( b, d ) to be indistinguishable [67].
Using a distance-based relative PE on the edges or an eigenvectorbased global PE, however, would allow to differentiate the two links.
3.3 GPS layer: an MPNN+Transformer hybrid
In this section we introduce the GPS layer, which is a hybrid MPNN+Transformer layer.
First we argue how it alleviates the limitation of a closely related work.
Next, we list the layer update equations which can be instantiated with a variety of MPNN and Transformer layers.
Finally, we present its characteristics in terms of modularity, scalability and expressivity.
Preventing early smoothing.
Why not use an architecture like GraphTrans [31] comprising of a few layers of MPNNs before the Transformer?
Since MPNNs are limited by problems of over-smoothing, over-squashing, and low expressivity against the WL test [1, 54], these layers could irreparably fail to keep some information in the early stage.
Although they could make use of PE/SE or more expressive
MPNNs [3, 16], they are still likely to lose information.
An analogous 2-stage strategy was successful in computer vision [18, 24] thanks to the high expressivity of convolutional layers on grids, but we do not expect it to achieve the same success on graphs due to the limitations of message-passing.
Update function.
At each layer, the features are updated by aggregating the output of an MPNN layer with that of a global attention layer, as shown in Figures 1 and D.1, and described by the equations below.
Note that the edge features are only passed to the MPNN layer, and that residual connections with batch normalization [30] are omitted for clarity.
Both the MPNN and GlobalAttn layers are modular, i.e., MPNN can be any function that acts on a local neighborhood and GlobalAttn can be any fully-connected layer.
[Equation]
computed as [Equation] where A 2 R N ⇥ N is the adjacency matrix of a graph with N nodes and E edges; X ` 2 R N ⇥ d ` , E ` R E ⇥ d ` are the d ` -dimensional node and edge features, respectively; MPNN ` e and GlobalAttn ` are instances of an MPNN with edge features and of a global attention mechanism at the ` -th layer with their corresponding learnable parameters, respectively; MLP ` is a 2-layer MLP block.
Modularity is achieved by allowing drop-in replacement for a number of module choices, including the initial PE/SE types, the networks that processes those PE/SE, the MPNN and global attention layers that constitute a GPS layer, and the ﬁnal task-speciﬁc prediction head.
Further, as research advances in different directions, G RAPH GPS allows to easily implement new PE/SE and other layers.
Scalability is achieved by allowing for a computational complexity linear in both the number of nodes and edges O ( N + E ) ; excluding the potential precomputation step required for various PE, such as Laplacian eigen-decomposition.
By restricting the PE/SE to real nodes and edges, and by excluding the edge features from the global attention layer, we can avoid materializing the full quadratic attention matrix.
Therefore we can utilize a linear Transformer with O ( N ) complexity, while the complexity of an MPNN is O ( E ) .
For sparse graphs such as molecular graphs, regular graphs, and knowledge graphs, the edges are practically proportional to the nodes E = ⇥ ( N ) , meaning the entire complexity can be considered linear in the number of nodes O ( N ) .
Empirically, even on small molecular graphs, our architecture reduces computation time compared to other GT models, e.g., a model of ~6M parameters requires 196s per epoch on the ogbg-molpcba [27] dataset, compared to 883s for SAN [36] on the same GPU type.
Expressivity in terms of sub-structure identiﬁcation and the Weisfeiler-Leman (WL) test is achieved via providing a rich set of PE/SE, as proposed in various works [3, 36, 16, 5, 6] and detailed in Section 3.1.
Further, the Transformer allows to resolve the expressivity bottlenecks caused by over-smoothing [36] and over-squashing [1] by allowing information to spread across the graph via full-connectivity.
Finally, in Section 3.4, we demonstrate that, given the right components, the proposed architecture does not lose edge information and is a universal function approximator on graphs.
3.4 Theoretical expressivity
In this section, we ﬁrst discuss how the MPNN layer allows to propagate edge and neighbor information on the nodes.
Then, we show that the proposed model is a universal function approximator on graphs, similarly to the SAN architecture [36].
Preserving edge information in the Transformer layer.
Most GTs do not fully utilize edge features of the input graph.
The Graph Transformer [14], SAN [36] and Graphormer [63] only use edge features to condition the attention between a pair of nodes, that is, they inﬂuence the attention gating mechanism but are not explicitly involved in updating of the node representations.
GraphiT
[44] does not consider edge features at all.
Recent 2-step methods GraphTrans [31] and SAT [9] can use edge features in their ﬁrst MPNN step, however this step is applied only once and typically includes several k rounds of message passing.
Therefore this latter approach may suffer from initial oversmoothing, as k -hop neighborhoods together with the respective edge features need to be represented in a ﬁxed-sized node representation.
On the other hand, in GPS, interleaving one round of local neighborhood aggregation via an MPNN layer with global self-attention mechanism reduces the initial representation bottleneck and enables iterative local and global interactions.
In the attention, the key-query-value mechanism only explicitly depends on the node features, but assuming efﬁcient representation encoding by the MPNN, the node features can implicitly encode edge information, thus edges can play a role in either the key, query, or values.
In Appendix C.2, we give a more formal argument on how, following an MPNN layer, node features can encode edge features alongside information related to node-connectivity.
Universal function approximator on graphs.
Kreuzer et al.
[36][Sec. 3.5] demonstrated the universality of graph Transformers.
It was shown that, given the full set of Laplacian eigenvectors, the model was a universal function approximator on graphs and could provide an approximate solution to the isomorphism problem, making it more powerful than any Weisfeiler-Leman (WL) isomorphism test given enough parameters.
Here, we argue that the same holds for our architecture since we can also use the full set of eigenvectors, and since all edge information can be propagated to the nodes.
[Table] [Table]
Experiments
We perform ablation studies on 4 datasets to evaluate the contribution of the message-passing module, the global attention module, and the positional or structural encodings.
Then, we evaluate GPS on a diverse set of 11 benchmarking datasets, and show state-of-the-art (SOTA) results in many cases.
We test on datasets from different sources to ensure diversity, providing their detailed description in
Appendix A.1.
From the Benchmarking GNNs [15], we test on the ZINC, PATTERN, CLUSTER,
MNIST, CIFAR10.
From the open graph benchmark (OGB) [27], we test on all graph-level datasets: ogbg-molhiv, ogbg-molpcba, ogbg-code2, and ogbg-ppa, and from their large-scale challenge we test on the OGB-LSC PCQM4Mv2 [28].
Finally, we also select MalNet-Tiny [21] with 5000 graphs, each of up to 5000 nodes, since the number of nodes provide a scaling challenge for Transformers.
4.1 Ablation studies
In this section, we evaluate multiple options for the three main components of our architecture in order to gauge their contribution to predictive performance and to better guide dataset-speciﬁc hyper-parameter optimization.
First, we quantify beneﬁts of the considered global-attention modules in 4 tasks.
Then, we note that the MPNN layer is an essential part for high-performing models, and identify the layer type most likely to help.
Finally, we observe when different global PE or local SE provide signiﬁcant boost in the performance.
All ablation results are averaged over multiple random seeds and summarized in Table 2, with additional information available in Appendix B.
Global-Attention module.
Here we consider global attention implemented as O ( N 2 ) key-queryvalue Transformer attention or linear-time attention mechanisms of Performer or BigBird.
We notice in Table 2a that using a Transformer is always beneﬁcial, except for the ZINC dataset where no changes are observed.
This motivates our architecture and the hypothesis that long-range dependencies are generally important.
We further observe that Performer falls behind Transformer in terms of the predictive performance, although it provides a gain over the baseline and the ability to scale to very large graphs.
Finally, BigBird in our setting offers no signiﬁcant gain, while also being slower than
Performer (see Appendix B).
Having no gain on the ZINC dataset is expected since the task is a combination of the computed octanol-water partition coefﬁcient (cLogP) [60] and the synthetic accessibility score (SA-score) [19], both of which only count occurrences of local sub-structures.
Hence, there is no need for a global connectivity, but a strong need for structural encodings.
Message-passing module.
Next, we evaluate the effect of various message-passing architectures,
Table 2a.
It is apparent that they are fundamental to the success of our method: removing the layer leads to a signiﬁcant drop in performance across all datasets.
Indeed, without an MPNN, the edge
[Table]
features are not taken into consideration at all.
Additionally, without reinforcing of the local graph structure, the network can overﬁt to the PE/SE.
This reiterates ﬁndings of Kreuzer et al.
[36], where considerably larger weights were assigned to the local attention.
We also ﬁnd that although a vanilla PNA [13] generally outperforms GINE [26] and GatedGCN [7], adding the PE and SE results in major performance boost especially for the GatedGCN.
This is consistent with results of Dwivedi et al.
[16] and shows the importance of these encodings for gating.
Perhaps the necessity of a local message-passing module is due to the limited amount of graph data, and scaling to colossal datasets [49] that we encounter in language and vision could change that.
Indeed, the Graphormer architecture [63] was able to perform very well on the full PCQM4Mv dataset without a local module.
However, even large Transformer-based language models [8] and vision models [25] can beneﬁt from an added local aggregation and outperform pure Transformers.
Positional/Structural Encodings.
Finally, we evaluate the effects of various PE/SE schemes,
Table 2b.
We ﬁnd them generally beneﬁcial to downstream tasks, in concordance to the vast literature on the subject (see Table 1).
The beneﬁts of the different encodings are very dataset dependant, with the random-walk structural encoding (RWSE) being more beneﬁcial for molecular data and the
Laplacian eigenvectors encodings (LapPE) being more beneﬁcial in image superpixels.
However, using SignNet with DeepSets encoding [39] as an improved way of processing the LapPE seems to be consistently successful across tasks.
We hypothesize that SignNet can learn structural representation using the eigenvectors, for example, to generate local heat-kernels that approximate random walks [2].
Last but not least we evaluate PEG-layer design [57] with Laplacian eigenmap.
4.2 Benchmarking GPS
We compare GPS against a set of popular message-passing neural networks (GCN, GIN, GatedGCN,
PNA, etc.), graph transformers (SAN, Graphormer, etc.), and other recent graph neural networks with SOTA results (CIN, CRaWL, GIN-AK+, ExpC).
To ensure diverse benchmarking tasks, we use datasets from Benchmarking-GNNs [15], OGB [27] and its large-scale challenge [28], and
Long-Range Graph Benchmark [17], with more details given in Appendix A.1.
We report the mean and standard deviation over 10 random seeds if not explicitly stated otherwise.
Benchmarking GNNs [15].
We ﬁrst benchmark our method on 5 tasks from Benchmarking GNNs [15], namely ZINC, MNIST, CIFAR10, PATTERN, and CLUSTER, shown in Table 3.
We observe that our GPS gives SOTA results on ZINC and the second best in 3 more datasets, showcasing the ability to perform very well on a variety of synthetic tasks designed to test the model expressivity.
Open Graph Benchmark [27].
Next, we benchmark on all 4 graph-level tasks from OGB, namely molhiv, molpcba, ppa, and code2, Table 4.
On the molhiv dataset, we observed our model to suffer
[Table] [Table]
from overﬁtting, but to still outperform SAN, while other graph Transformers do not report results.
For the molpcba, ppa
[Table]
and code [Table] GPS always ranks among the top 3 models, highlighting again the versatility and expressiveness of the GPS approach.
Further, GPS outperforms every other GT on all 4 benchmarks, except SAT on code2.
OGB-LSC PCQM4Mv2 [28].
The large-scale PCQM4Mv2 dataset has been a popular benchmark for recent GTs, particularly due to Graphormer [63] winning the initial challenge.
We report the results in Table 5, observing signiﬁcant improvements over message-passing networks at comparable parameter budget.
GPS also outperforms GRPE [48], EGT [29], and Graphormer [63] with less than half their parameters, and with signiﬁcantly less overﬁtting on the training set.
Contrarily to Graphormer, we do not need to precompute spatial distances from approximate 3D molecular conformers [64], the RWSEs we utilize are graph-based only.
MalNet-Tiny.
The MalNet-Tiny [21] dataset consists of function call graphs with up to 5,000 nodes.
These graphs are considerably larger than previously considered inductive graph-learning benchmarks, which enables us to showcase scalability of GPS to much larger graphs than prior methods.
Our GPS reaches 92 .
72% ± 0 .
7pp test accuracy when using Performer global attention.
Interestingly, using
Transformer global attention leads to further improved GPS performance, 93 .
36% ± 0 .
6pp (based on 10 runs), albeit at the cost of doubled run-time.
In both cases, we used comparable architecture to
Freitas et al.
[21], with 5 layers and 64 dimensional hidden node representation, and outperform their best GIN model with 90% accuracy.
See Appendix B for GPS ablation study on MalNet-Tiny.
Long-Range Graph Benchmark [17].
Finally, we evaluate the GPS method on a recent Long-Range
Graph Benchmark (LRGB) suite of 5 datasets that are intended to test a method’s ability to capture long-range dependencies in the input graphs.
We abide to the ~500k model parameter budget and
[Table]
closely follow the experimental setup and hyperparameter choices of the graph Transformer baselines tested in LRGB [17].
We keep the same node/edge encoders and model depth (number of layers), deviating only in two aspects: i) we slightly decrease the size of hidden node representations to ﬁt within the parameter budget, ii) we employ cosine learning rate schedule as in our other experiments (Section A.3).
For each dataset we utilize LapPE positional encodings and GPS with GatedGCN [7] and Transformer [55] components.
GPS improves over all evaluated baselines in 4 out of 5 LRGB datasets (Table 6).
Additionally, we conducted GPS ablation studies on PascalVOC-SP and Peptides-func datasets in the same fashion as for 4 previous datasets in Table 2, presented in Tables B.5 and B.6, respectively.
For both datasets the global attention, in form of Transformer or Performer, is shown to be a critical component of the
GPS in outperforming MPNNs.
In the case of PascalVOC-SP, interestingly, the Laplacian PEs are not beneﬁcial, as without them the GPS scores even higher F1-score 0 .
3846 , and PEG [57] relative distance embeddings enable the highest score of 0 .
Conclusion
Our work is setting the foundation for a uniﬁed architecture of graph neural networks, with modular and scalable graph Transformers and a broader understanding of the role of graphs with positional and structural encodings.
In our ablation studies, we demonstrated the importance of each module: the
Transformer, ﬂexible message-passing, and rich positional and structural encodings all contributed to the success of GPS on a wide variety of benchmarks.
Indeed, considering 5 Benchmarking-GNN tasks [15], 5 OGB(-LSC) tasks [27, 28], 5 LRGB tasks [17] and MalNet-Tiny, we outperformed every graph Transformer on 11 out of 16 tasks while also achieving state-of-the-art on 8 of them.
We further showed that the model can scale to very large graphs of several thousand nodes, far beyond any previous graph Transformer.
By open-sourcing the G RAPH GPS package, we hope to accelerate the research in efﬁcient and expressive graph Transformers, and move the ﬁeld closer to a uniﬁed hybrid Transformer architecture for graphs.
Limitations.
We ﬁnd that graph transformers are sensitive to hyperparameters and there is no onesize-ﬁts-all solution for all datasets.
We also identify a lack of challenging graph datasets necessitating long-range dependencies where linear attention architectures could exhibit all scalability beneﬁts.
Societal Impact.
As a general graph representation learning method, we do not foresee immediate negative societal outcomes.
However, its particular application, e.g., in drug discovery or computational biology, will have to be thoroughly examined for trustworthiness or malicious usage.
Acknowledgments and Disclosure of Funding
This work was partially funded by IVADO (Institut de valorisation des données) grant PRF-20193583139727 and Canada CIFAR AI Chair [ G.W. ].
This research is supported by Nanyang Technological University, under SUG Grant (020724-00001)
[ V.P.D. ] and Samsung AI graph at Mila [ M.G. ].
The content provided here is solely the responsibility of the authors and does not necessarily represent the ofﬁcial views of the funding agencies.