A RE M ORE L AYERS B ENEFICIAL TO G RAPH T RANSFORMERS ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite that going deep has proven successful in many neural architectures, the existing graph transformers are relatively shallow.
In this work, we explore whether more layers are beneficial to graph transformers, and find that current graph transformers suffer from the bottleneck of improving performance by increasing depth.
Our further analysis reveals the reason is that deep graph transformers are limited by the vanishing capacity of global attention, restricting the graph transformer from focusing on the critical substructure and obtaining expressive features.
To this end, we propose a novel graph transformer model named DeepGraph that explicitly employs substructure tokens in the encoded representation, and applies local attention on related nodes to obtain substructure based attention encoding.
Our model enhances the ability of the global attention to focus on substructures and promotes the expressiveness of the representations, addressing the limitation of self-attention as the graph transformer deepens.
Experiments show that our method unblocks the depth limitation of graph transformers and results in stateof-the-art performance across various graph benchmarks with deeper models.
I NTRODUCTION
Transformers have recently gained rapid attention in modeling graph-structured data (Zhang et al., 2020; Dwivedi & Bresson, 2020; Maziarka et al., 2020;
Ying et al., 2021; Chen et al., 2022).
Compared to other graph neural networks, graph transformer implies global attention mechanism to enable information passing between every two nodes, which is advantageous to learn long-range dependency of the graph stuctures (Alon & Yahav, 2020).
In transformer, the graph structure information can be encoded into node feature (Kreuzer et al., 2021) or attentions (Ying et al., 2021) by a variant of methods flexibly with strong expressiveness, avoiding the inherent limitations of encoding paradigms that pass the information along graph edges.
Global attention (Bahdanau et al., 2015) also enables explicit focus on essential parts among the nodes to model crucial substructures in the graph.
Graph transformer in current studies is usually shallow, i.e., less than 12 layers.
Scaling depth is proven to increase the capacity of neural networks exponentially (Poole et al., 2016), and empirically improve transformer performance in nature language processing (Liu et al., 2020a;
Bachlechner et al., 2021) and computer vision (Zhou et al., 2021).
Graph neural network also benefits from more depth when properly designed (Chen et al., 2020a;
Liu et al., 2020b; Li et al., 2021).
However, it is still not clear whether the capability of graph transformers in graph tasks can be strengthened by increasing model depth.
So we conduct experiments and find that current graph transformers encounter the bottleneck of improving performance by increasing depth.
The further deepening will hurt performance when model exceeds 12 layers, which seems to be the upper limit of the current graph transformer depth, as Figure 1 shows.
In this work, we aim to answer why more self-attention layers become a disadvantage for graph transformers, as well as how to address these issues with the proper model design.
Self-attention mechanism (Bahdanau et al., 2015) makes a leap in model capacity by dynamically concentrating on critical parts of the input (Chen et al., 2022), i.e., substructures of a graph, and obtaining particular features.
Substructures are the basic intrinsic features of graph data widely used in data analysis (Yu et al., 2020) and machine learning (Shervashidze et al., 2009), as well as graph model interpretability (Miao et al., 2022).
Although the self-attention module appears to be very beneficial for automatically learning important substructure features in the graph, our analysis indicates that this ability vanishes as depth grows, restricting the deeper graph transformer from learning useful structure features.
Specifically, we focus on the influence of attention on different substructure patterns, which we found to decrease after each self-attention layer.
As a direct consequence, it is difficult for deep models to autonomously learn effective attention patterns of substructures and obtain expressive graph substructure features.
We further propose a graph transformer model named DeepGraph with a simple but effective method to enhance substructure encoding ability, enabling deeper graph transformer to achieve better performance.
The proposed model explicitly introduces local attention mechanism on substructures by employing additional substructure tokens in the model representation and applying local attention to nodes related to those substructures.
Our method not only introduces the substructure based attention to encourage the model to focus on substructure feature, but also enlarges the attention capacity theoretically, which improves the expressiveness of representations learned on substructures.
In summary, our contributions are as follows:
• We present the bottleneck of graph transformers’ performance when depth increases, illustrating the depth limitation of current graph transformers.
We theoretically study the bottleneck from the perspective of attention capacity’s decay with layers, and demonstrate the difficulty for deep models to learn effective attention patterns of substructures and obtain informative graph substructure features.
According to the theoretical finding, we propose a simple but effective substructure token based local attention mechanism in graph transformer, promoting focus on local substructure features of deeper graph transformer and improving the expressiveness of learned representations.
• Experiments show that our method unblocks the depth limitation of graph transformer and achieves state-of-the-art results on standard graph benchmarks with deeper models.
R ELATED W ORK
Graph transformers Transformer with the self-attention based encoding mechanism has been the mainstream method in nature language processing (Vaswani et al., 2017; Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020; Brown et al., 2020; Raffel et al., 2020), and is also proven competitive for image and video processing in computer vision (Dosovitskiy et al., 2020; Touvron et al., 2021;
Carion et al., 2020).
The pure transformer is a set mapping function with no relation information between tokens and requires additional position encoding to inform the model of structure information.
Recent works apply transformers in graph tasks by designing a variety of structure encoding techniques.
Some works embed structure information into graph nodes by methods including Laplacian vector, random walk, or other feature (Zhang et al., 2020; Dwivedi & Bresson, 2020; Kreuzer et al., 2021;
Kim et al., 2022; Wu et al., 2021).
Some other works introduce structure information into attention by graph distance, path embedding or feature encoded by GNN (Park et al., 2022; Maziarka et al., 2020;
Ying et al., 2021;
Chen et al., 2022; Mialon et al., 2021; Choromanski et al., 2022).
The two types of methods correspond to absolute position encoding or relative position encoding in transformers, respectively.
Other works use transformer as a module of the whole model (Bastos et al., 2022; Guo et al., 2022).
Deep neural networks There are many works focus on solving obstacles and release potential of deep neural networks in feed-forward network (FNN)
(Telgarsky, 2016; Yarotsky, 2017) and convolutional neural network (CNN)
(He et al., 2016a; Simonyan & Zisserman, 2014; Xiao et al., 2018;
He et al., 2016b).
For graph neural networks, early studies reveal severe performance degradation when stacking many layers (Li et al., 2019; Xu et al., 2018b; Li et al., 2018), caused by problems including over-smoothing and gradient vanishing (Oono & Suzuki, 2019; Zhao & Akoglu, 2019;
Rong et al., 2019; Nt & Maehara, 2019).
Recent works alleviate these problems by residual connec- tion, dropout, and other methods (Chen et al., 2020a; Li et al., 2020; 2021), and deepen GNN into hundreds and even thousands of layers with better performance.
Deep transformer is also proved powerful in nature language processing (Wang et al., 2022) and computational vision (Zhou et al., 2021), while better initialization (Zhang et al., 2019) or architecture (Wang et al., 2019; Liu et al., 2020a; Bachlechner et al., 2021) are proposed to solve optimization instability and over-smoothing (Shi et al., 2021;
Wang et al., 2021; Dong et al., 2021).
Graph substructure Substructure is one of the basic intrinsic features of graph data, which is widely used in both graph data analysis (Shervashidze et al., 2009; Yanardag & Vishwanathan, 2015;
Rossi et al., 2020) and deep graph neural models (Chen et al., 2020b; Bouritsas et al., 2022; Bodnar et al., 2021; Zhang & Li, 2021; Zhao et al., 2021).
In application, substructure related methods are widely used in various domain including computational chemistry (Murray & Rees, 2009; Jin et al., 2018; 2019; Duvenaud et al., 2015; Yu et al., 2020), computational biology (Koyut¨urk et al., 2004) and social network (Jiang et al., 2010).
Certain substructures can also be the pivotal feature for graph property prediction, which is a fundamental hypothesis in graph model interpretability studies (Ying et al., 2019; Miao et al., 2022), helping to understand how a graph model makes decisions.
P RELIMINARY 3. G RAPH AND S UBSTRUCTURE
We denote graph data as { G i , y i } , where a graph G =
( N, R, x, r ) includes nodes N = { 1 . . .
| G |} and corresponding edges R ⊂ N × N , while x and r are node features and edge features.
Label y can be graph-wise or node-wise, depending on the task definition.
Given graph G , a substructure is defined as G S =
{ N S , R S , x S , r S } , where N S ⊂ N, R S =
( N S × N S )
∩ R , i.e. nodes of G S form a subset of the graph G and edges are all the existing edges in G between nodes subset, which is also known as induced subgraph.
Because attention is only applied on graph nodes, attention on arbitrary subgraph is not well-defined.
Therefore, we only consider induced subgraph in this work.
3. T RANSFORMER
The core module of the transformer is self-attention.
Let H l
= [ h 1 , · · · , h n ] ⊤ ∈ R n × d h be the hidden representation in layer l , where n is the number of token, and d h is the dimension of hidden embedding of each token.
The self-attention mapping ˆAttn with parameter W V , W Q , W K ∈ R d h × d k and W O ∈ R
d
k × d h is
[Equation]
where W V O ≜ W V W O .
In practice, a complete transformer layer also contains a two-layer fullyconnected network FFN , and layer normalization with residual connection is also applied to both self-attention module and fully-connected network:
[Equation]
where Attn( H ) =
LN( AHW V O )) , FFN( H ′ )
=
LN( ReLU ( H ′ W F 1 + 1 b F 1 T ) W F 2 + 1 b F 2 T ) .
LN( f ( X ))
=
( X + f ( X ) − 1 b NT )
D is the layer normalization, where D is diagonal matrix with normalizing coefficients.
The vanilla transformer does not contain any structure information of graph, treating input tokens as a set.
Structure information can be encoded into token representations or attentions.
Our model adopts the distance and path-based relative position encoding methods as Ying et al.
(2021), which utilizes graph distance DIS and shortest path SP information to guide attention calculation:
[Equation]
We also employ deepnorm (Wang et al., 2022) residual connection method in out model, which adjust the residual connection in layer normalization by constant η : LN( f ( X ))
= ( ηX + f ( X ) − 1 b NT )
D to stabilize the optimization of deep transformers.
T HEORETICAL R ESULTS
In this section, we study the capacity of global attention to explain the depth bottleneck in graph transformers.
As stated above, substructures are fundamental graph features to represent graph structure and predict graph properties.
In the graph transformer, the global attention mechanism is expected to focus on important substructures and learn expressive substructure representations automatically.
However, we find that in graph transformer, attention’s influence decreases as depth grows.
We define the attention capacity first, then analyze how the attention capacity varies with depth.
Finally, we show that local attention on substructures is a good solution to the problem.
4. D EFINITION OF A TTENTION C APACITY
We define capacity of attention as the maximum difference between representations learned by different substructure attention patterns.
Let supp( e ) be all the non-zero dimensions of vector e .
We define a fixed attention pattern e of substructure G S as an attention vector that only focuses on nodes of G S , i.e., supp( e ) =
N S , where e ∈ [0 , 1] n , e T 1 = 1 .
Given a graph G with several important substructures G S 1 . . .
G Sm and corresponding attention patterns e 1 . . .
e m , denote E =
( e 1 , e 2 . . .
, e m ) where columns of E is the base vectors of attention patterns on substructures we are interested.
We consider the attention space spanned by attention patterns focusing on different substructures:
[Equation]
[Equation]
) and denote ∆ n S as matrix space with n columns all in space ∆ S .
We define attention capacity as the maximum difference of self-attention mapping results computed by different attention matrices from the attention space above.
Let ˆAttn A be the self-attention mapping where attention matrix equals A .
Definition 1 The attention capacity is defined as:
[Equation]
Attention capacity is an important property for graph transformer.
The larger capacity means paying attention to different substructures can yield very different features.
On the contrary, a small capacity means different substructures’ attention patterns do not have a large effect on learned representations.
It will be more difficult for the self-attention module with smaller capacity to learn attention patterns because the model is not sensitive to different substructures, and the learned representation is less informative with focused substructure.
4. A TTENTION C APACITY D ECREASES WITH D EPTH
Next, we illustrate that the stacked self-attention layers can cause decreased attention capacity for deeper layers.
We first demonstrate that the self-attention module decreases the capacity of attention, then discuss the case with fully-connected layers.
Theorem 1 (The stacked attention modules decrease the capacity of attention)
We analyze capacity of attention F H l for stacked only attention modules H
i +
=
Attn ( H i ) =
( H i
+
A
i H
i W V
O i − 1 b NiT )
D i .
Assume H
i W V
O
i
D i is full row rank for each layer i .
Due to the property of layer normalization, assume the output of attention module equals to its input, i.e., for input X , | Attn( X ) | F ≈ | X | F .
Then the attention capacity of layer l is upper bounded by
[Equation]
,where α i =
[Equation]
PH
[Equation]
D i | F < 1 with probability 1, and P ≜ ( I − 1 n 11 T ) , P m ≜ ( I − 1 m 11 T ) .
Proof is in Appendix 1.
The analysis above reveals that the self-attention module decreases the upper bound of attention capacity exponentially.
We next consider the case when fully-connected layers are also included in each layer, just like the architecture in practice.
Theorem 2 (The upper bound of attention capacity after stacked transformer layers) For transformer layer with self-attention H
′ i
=
Attn( H i ) =
( H i
+
A
i H
i W V
O i − 1 b NiT ) D i , and fullyconnected layer H
i +1 =
[Equation]
with the same assumption as the previous, attention capacity of layer l is upper bounded as follows:
[Equation]
,where α i =
[Equation]
1 with probability 1, and γ i = | D
[Equation]
i
[Equation]
|
[Equation]
|
[Equation]
| W F 2 i | 2 )
[Equation]
Proof is in Appendix 2.
The theory shows that the upper bound of the attention capacity is affected by the combined effect of the two coefficients when the fully-connected layer is considered.
Shi et al.
(2021) shows that there always exists a fraction of data with | D
′ i
| 2 < 1 , which dominants γ i .
In these cases, attention capacity vanishing happens at least for part of the data.
The decreasing attention capacity with depth reveals two problems in graph transformer.
First, it is hard for graph transformer to pay attention to useful substructures when depth grows.
Second, feature learned by attention mechanism loses expressiveness w.r.t substructures in deep layers.
Both of these problems are pivotal to graph representation learning because the structural features of graphs are essential components of graph data.
To address the first problem, it’s natural to introduce substructure-based local attention as inductive bias into graph transformer to make up for the deficiency of attention on substructures.
Furthermore,
We will next show that introducing substructures based local attention also solves the second problem.
An important reason for the reduction of attention capacity is the dense connection of global attention (Rong et al., 2019), which can attract representations into 1 -stretched space and increase denominator of α in Theorem 1 and 2.
Intuitively, the node information interacts too densely in global attention, squashing local information in features of the deeper layers.
Consequently, attention mechanism in the deep layers can no longer effectively distinguish the features at different positions, thereby weakening the expressive ability of the deep self-attention modules.
Following this line of thought, explicit substructure based local attention mechanism is a reasonable strategy to reduce the decay of attention capacity vanishing, where each node can only attend to other nodes that belong to the same substructure.
We next provide proof that the problem of capacity decay can be alleviated if local attention on substructures is applied in each layer.
Theorem 3 Define substructure based local attention as where each node only attends nodes that belong to the same substructures.
Assume that for each node, the number of nodes belonging to the same substructures is at most r , where r < n .
Let α s be the decay coefficient of this model in
Theorem 1, and α be the decay coefficient of the global attention model.
Denoted minimum of α s and α for all possible representation H i and model parameters by α s ∗ and α ∗ respectively.
Then we have α s ∗ > α ∗ .
The proof is in Appendix 3.
This theory illustrates that substructure based local attention not only introduces the substructure based attention to encourage the model to focus on substructure features but also enlarges the attention capacity of deep layers, which promotes the representation capacity of the model when it grows deep.
A PPROACH
As we analyzed above, explicit substructure based attention mechanism is a reasonable strategy to reduce the decay of attention capacity vanishing and promotes the representation capacity when model grows deep.
However, replacing global attention directly with local attention may lead to insufficient learning ability of the model for long-range dependencies.
To better introduce local attention to substructure in self-attention models, we propose a novel substructure token based local attention method, DeepGraph, as Figure 2 shows.
DeepGraph tokenizes the graph into node level and substructure level.
Local attention is applied to the substructure token and corresponding nodes, while the global attention between nodes is still preserved.
This provides a better solution for explicitly introducing attention to substructures in the model while combining global attention, to achieve a better global-local encoding balance.
Our method first samples substructures of the graph, then encodes substructures into tokens, and finally enforces local attention on substructures.
In this section, we introduce the construction and sampling of substructures, the encoding of substructures, and the form of local attention, respectively.
5. S UBSTRUCTURE S AMPLING
The diverse types of substructures in current studies can be divided into two classes: neighbors which contain local features, including k-hop neighbors and random walk neighbors, and geometric substructures representing graph topological features, like circles, pathes, stars and other special graphs.
In this study, we explore all of these substructures towards a general and flexible substructure based local attention framework.
In practice, these substructures can be pre-computed once and cached for reuse.
Details of substructures are in Appendix.
The number of substructures in a graph often far exceeds the number of nodes.
For the feasibility of the actual computation, we sample substructures in each computation to encode the graph.
Formally, at each time of encoding, for a graph G with the corresponding substructure set { G S } , we sample a subset { G S 1 , G S 2 . . .
G Sm } as input to our model:
[Equation]
Inspired by Zhao et al.
(2021), to reduce bias caused by uneven substructure density, we hope sampled substructures cover every node of the graph as evenly as possible.
Due to the divergent radios of different types of substructures in a graph, we also balance the substructure types in the sampling.
The details of SubstructureSampling are in Appendix.
5. S UBSTRUCTURE T OKEN E NCODING
The input embedding contains nodes tokens embedding { h 1 , h 2 , . .
h n } and substructure tokens embedding { h n +1 , . .
, h n + m } , encoded by node feature encoder g n and substructure encoder g s individually.
The token embedding of graph nodes is mapping from graph node feature x to feature vector h : h
i =
g n ( x i ) , i ∈ { 1 , 2 , . . .
n } .
For the substructure tokens, we apply permutation-based structure encoding (Murphy et al., 2019;
Chen et al., 2020b; Nikolentzos & Vazirgiannis, 2020) by encoding the substructure adjacency ma- trix
A Si directly.
The core idea of permutation-based encoding is pooling the encoding output of permuted graph information over all the possible permutations, which is permutation-invariant and has no information loss.
We further apply depth-first search (DFS) with the order of node degree to reduce the possible order of substructure nodes.
The details of DFS is in Appendix.
The formal definition of substructure token encoder is
[Equation]
In practice, sampling is applied instead of pooling.
We found that only one sample is enough during training for the model to learn the substructure stably.
5. L OCAL A TTENTION ON S UBSTRUCTURES
After adding substructure tokens, localized attention is applied to the substructure and its corresponding nodes.
Given input embedding H 1 = ( h 1 , h 2 , . . .
, h n + m ) , mask M is added in selfattention module
[Equation]
where M ij ∈ { 0 , −∞} , i.e., the elements of mask M can be 0 or −∞ , leading to a sparse attention matrix A with zero in position of −∞ in M .
In our model, the mask is defined as follows to induce local attention to substructure tokens and corresponding nodes: M ij =
−∞ if i + n ∈ { n + 1 , n + 2 , . . .
n + m } , j ̸∈ N Si or j + n ∈ { n + 1 , n + 2 , . . .
n + m } , i ̸∈ N Sj , and M ij =otherwise.
The substructure tokens only apply local attention to corresponding nodes.
The attention in our model is a combination of local attention on substructures and global attention on all nodes.
The substructure nodes serve as the core of local attention.
In each step, the substructure token only pays attention to its corresponding nodes, integrating the representation from the whole substructure.
Nodes belonging to the same substructure all share the message from the substructure token, which marks the nodes inside the substructure and increases the distinguishability between nodes from different substructures, promoting the representation capacity w.r.t substructure.
E XPERIMENTS
In this section, we aim to validate the performance of DeepGraph empirically.
Specifically, we attempt to answer the following questions: (i) How does DeepGraph perform compared to existing transformers on popular benchmarks?
(ii) Does DeepGraph’s performance improve when depth increases, and is it more effective compared to other methods of deepening transformer?
(iii) How much does each part of our model affect the model’s performance?
We first conduct experiments to evaluate our model on four datasets, including graph and node prediction tasks, compared with both state-of-art graph transformer models and their deeper versions augmented by recent methods toward deep transformers.
Then we validate the effect of different components of our architecture by ablation studies.
The codes of experiments will be released.
6. D ATASETS
Our method is validated on the tasks of the graph property prediction and node classification, specifically including PCQM4M-LSC (Hu et al., 2020), ZINC (Dwivedi et al., 2020), CLUSTER (Dwivedi et al., 2020) and PATTERN (Dwivedi et al., 2020), widely used in graph transformer studies.
PCQM4M-LSC is a large-scale graph-level prediction dataset with more than 3.8M molecules.
ZINC consists of 12,000 graphs with a molecule property to regression.
CLUSTER and PATTERN are challenging node classification tasks with graph sizes varying from dozens to hundreds, containing 14,000 and 12,000 graphs, respectively.
6. B ASELINES
We choose recent state-of-art graph transformer models: GT (Dwivedi & Bresson, 2020), SAN (Kreuzer et al., 2021), Graphormer (Ying et al., 2021) and SAT (Chen et al., 2022), covering various graph transformer methods including absolute and relative position embedding.
Graph neural
[Table]
networks baselines includes GCN (Kipf & Welling, 2016) and GIN (Xu et al., 2018a).
We first compare our model with the standard state-of-art models.
Then we compare our model with the deepened versions of the state-of-arts augmented by recent algorithms designed for deep transformers, including the fusion method (Shi et al., 2021), and reattention method (Zhou et al., 2021).
As we use deepnorm to stabilize our training process, we compare DeepGraph to naive deepnorm in the ablation study.
6. S ETTINGS
We implement the models with 12, 24, and 48 layers.
The hidden dimension is 80 for ZINC and
PATTERN, 48 for CLUSTER, and 768 for PCQM4M-LSC.
The models are trained by Adam optimizer, with warm-up and learning rate decay for learning rate adjusting.
Reported results are the average of over 4 seeds.
Our model uses both geometric substructure and neighbors as substructure patterns.
The geometric substructure includes k-path, k-star, and k-circle.
The neighbors include k-hop neighbor and k-step random walk neighbor.
We imply both geometric substructures and khop neighbors on ZINC and PCQM4M-LSC.
As for CLUSTER and PATTERN, only random walk neighbors are used due to the large scale and dense connection of these two datasets.
The effect of different substructure patterns is shown in the ablation study.
6. M AIN R ESULTS
Table 1 summarizes our experimental results on the graph regression and node classification tasks.
Data of the baseline models are copied from their papers.
The depth of our model varies from 12 to 48 layers.
As the results illustrated, our 12-layer-model outperforms baseline models on PCQM4MLSC, ZINC, and PATTERN, especially on ZINC and PATTERN, surpassing the previous best result reported significantly.
As model depth increase, our model achieves consistent improvement, especially on dataset ZINC.
Note that our model with 48 layers outperforms baseline models on all the datasets, proving the effectiveness of more stacked DeepGraph layers.
6. E FFECT OF D EEPENING
We next compare DeepGraph with other baseline models after deepening.
We choose Graphormer and SAT as baseline models to deepen by 2 and 4 times compared to the original version.
In addition to deepening naively, we also enhance the baseline model by reproducing fusion and reattention in the baseline models.
As shown in table 3, DeepGraph outperforms baselines with a significant margin.
Naive deepening decreases performance on all the datasets for both two base models.
Note that all the 4-time deeper
SAT fail to converge on CLUSTER due to the training difficulty, as well as Graphormer with reattention.
Fusion improves deeper Graphormer on PCQM4M-LSC, and reattention boosts 2x deep SAT on CLUSTER slightly, but none of them are consistent.
Our methods surpass fusion and reattention methods stably on all 4 datasets, supporting our theoretical claim that sensitivity to substructures is crucial in deep graph transformers.
[Table] [Table]
6. A BLATION AND S ENSITIVITY S TUDY
Finally
[Table]
we validate the effectiveness of each component of our model by ablation study.
We drop the components, including the local attention, the substructure encoding
[Table]
and deepnorm method.
Note that without local attention, the model is only a relative position based graph transformer with deepnorm residual connection.
Without substructure encoding, randomly initialized embedding is applied to each substructure token.
We conduct experiments on ZINC and CLUSTER, individually representing two types of tasks.
The result is in Table 2.
It can be found that without local attention, performance dropped a lot on both datasets, and the deeper model performs worse than the shallow ones, proving the effectiveness of the proposed methods.
The substructure encoding also plays an important role, emphasizing the importance of structural information.
Finally, deepnorm also contributes to the model performance by stabilizing the optimization, especially for the 48-layer models.
We validate the sensitivity of our model to different substructure types by testing the performance of DeepGraph with only geometric substructures or neighbors, as shown in Table 3.
While both types of substructures contribute to the performance, geometric substructures play a more critical role in the ZINC dataset.
This result is expected because specific structures like rings are important fundamental features for the molecule data.
At the same time, it can be further strengthened when combined with neighbor substructures.
Our model can flexibly use different forms of substructure to fully utilize prior knowledge of data.
Note that for CLUSTER and PATTERN, only neighbor type substructures are utilized, showing that the neighbor information also plays an important role in the complex graph.
C ONCLUSIONS
This work presents the bottleneck of graph transformers’ performance when depth increases.
We find that deep graph transformers are limited by the capacity bottleneck of graph attention by theoretical analysis.
We further propose a novel graph transformer model that applies substructure based local attention with additional substructure tokens substructure based attention encoding.
Our model enhances the ability of the global attention mechanism to focus on substructures and addresses the limitation of encoding expressive features as the model deepens.
Experiments show that our model breaks the depth bottleneck and achieves state-of-the-art performance on popular graph benchmarks.
A PPENDIX
, define P ≜ ( I − 1 n 11 T ) , and P m ≜ ( I − 1 m 11 T ) .
We first introduce an important property of P :
For any matrix B with left eigenvector B 1 = β 1 , we have PBC = PBPC, ∀ C .
To prove it, we can deompose C = PC + 1 n 11 T C .
Then
[Equation]
We next prove the main theorem by recursion.
By definition,
[Equation]
Note that ( C 1 − C 2 ) T 1 = 1 − 1 = 0 , so we can multiply P before it:
[Equation]
,where | ( C 1 − C 2 ) T | 2 ≤ (cid:112) | ( C 1 − C 2 ) T | 1 | ( C 1 − C 2 ) T | ∞ =
√ 2 m .
We further decompose | P m E T HW V O | F into
[Equation]
m
[Equation]
For layer l , we recursively compute | PH
i +1 | F by | PH i | F for layer
i =
l − 1 , l − 2 , . .
, 1 : |
[Equation]
i
[Equation]
= α
i λ i | PH i | F (14) ,where α i = | P ( PH i + A i PH i W VOi − 1 b NiT )
D i | F | ( PH i + A i PH i W VOi − 1 b NiT ) D i | F , and λ i = | ( PH i + A i PH i W VOi − 1 b NiT )
D i | F | PH i | F .
Note that P
= ( I − 1 n 11 T ) is a contraction mapping, so α i ≤ 1 , and equal to
if and only if A i PH
i W V
O
i
D i − 1 b NiT D i is orthogonal to the 1 -stretched space, which is usually impossible, because − 1 b NiT D i is in the 1 -stretched space, and the probability that projection of A i PH i W V
O
i
D i on 1 -stretched space equals to 1 b NiT D i is 0 due to the property of random matrix.
We further show that A i PH i W V O D i is full-rank with probability 1: As assumption,
H i
W V
O
i
D i is full-rank, and 1 -stretched space is the only subspace that PH
i W V
O
i
D i is orthogonal to, i.e. {+x T PH i W V O i D i = 
+}
⇔ x = β.
So if 1 T A i PH
i W V
O
i
D i = 0 , 1 T A i = β 1 T , i.e. A Ti 1 = β 1 , the probability of which is 0 for random parameters in transformer.
For coefficient λ i , due to the property of layer normalization and good parameter initialization that keeps norm of output equals to input, i.e. | ( X + A i XW V O
i
[Equation]
b [Equation] ) D i | F / | X | F ≈ 1 , ∀ X , we have λ ≈ 1 .
Based on the analysis above, we can bound F H l recursively:
[Equation]
) , where α i = | P ( PH i + A i PH i W VOi − 1 b NiT ) D i | F | ( PH [Equation] + A
[Equation]
PH
[Equation]
VOi
[Equation]
NiT
[Equation]
= (
[Equation]
VOi
[Equation] NiT [Equation] D i | F < 1 with probability 1.
Proof 2 As the same method in the previous proof, we need to recursively bound | PH
i +1 | F by | PH i | F for layer
i =
l − 1 , l − 2 , . .
Deote H
′ i
=
Attn( H i ) , and H i +1 =
FFN( H ′ i ) .
Similar to the previous proof, we have
[Equation]
i + A
i H
i W V
O i − 1 b NiT )
D i | F = | P ( PH i + A i PH
i W V
O i − 1 b NiT )
D i | F = | P ( PH i + A i PH
i W V
O i − 1 b NiT )
D i | F | ( PH i + A i PH
i W V
O i − 1 b NiT )
D i | F | ( PH i + A i PH
i W V
O i − 1 b NiT )
D i | F | PH i | F | PH i | F = α
i λ i
| PH i | F (16)
As for PH i +1 , we can bound it as
[Equation]
So the final conclusion is
[Equation]
, where α i = | P
[Equation]
1 with probability
[Equation]
and γ i = | D ′ i
| 2 (1 + | W F 1 i | 2 | W F 2 i | 2 )
[Equation]
Proof 3 We denote global attention and local attention by A and A s , respectively.
By definition, α i = ( | PH i + PA
i
H
i W VOi )
D i | F |
[Equation]
PH i
[Equation]
i PH i W VOi − 1 b NiT ) D i | F .
We analyse the worst case for α i :
[Equation]
Further analysis on | 1 n 11 T A i PH i W V
O
i
D i | F shows that
[Equation]
where PA Ti 1 =
A Ti 1 − 1 , the element of which is between [ − 1 , n − 1] , and the sum of all elements of which is 0.
The maximum of | PA Ti 1 | 2 is achieved when one element is n − 1 and other elements are − 1 .
However, for local attention A si that each node is only attended by at most r nodes, where r < n , the elements of | PA siT 1 | 2 is only between [ − 1 , r − 1] , and thus the maximum | PA siT 1 | 2 is less than | PA Ti 1 | 2 .
We further address that the two inequalities in 19 and 20 both can be achieved for proper H i and parameters.
So we can conclude that for the worst case of α i and α si , denoted by α ∗ i and α si ∗ ,
[Equation]
D ISCUSSION OF S UBSTRUCTURE B ASED A TTENTION S PACE
For a substructure G S , the fixed attention pattern e is an ideal attention pattern to help to concentrate on the information of this substructure, for example, a uniform attention vector on a benzene ring, or attention on a two-hop neighbor graph attenuating according to the distance to the central node.
Traditional GNN is also an example of such a fixed attention pattern, where each node takes uniform attention to its’ neighbor nodes.
Although we treat this pattern as fixed on the substructure, we denote that the definition is general because the value can be any learnable attention pattern.
The defined attention space construct attention patterns for all meaningful attention on substructures by containing the combination of all the elementary substructure pattern.
As this substructure attention patterns are considered important indicative bias, the graph transformer is expected to learn attention patterns in this space to utilize the substructure feature.
This definition is also general and expressive, as sufficient patterns can be added to include more general cases.
S UBSTRUCTURES
The sizes of the different substructures used in this study are 2-hop neighbor, 10-step random walk neighbor, 3-8 circle, 2-6 star, and 4-8 path.
For geometric substructures, we use the python package graph-tool for substructure matching.
It takes about 3 minutes to match all the geometric substructures of ZINC, and about 2 hours for PCQM4M-LSC.
We cache all the substructures for reuse.
S UBSTRUCTURE S AMPLING
A greedy sampling algorithm is used for the substructure sampling process.
In each step of the iteration, nodes with less coverage have a greater probability of being covered in the next iteration.
At the beginning of sampling, we randomly select n init substructures from all the substructures, where different types of substructures are balanced.
In each iteration, we first mark nodes that are covered less than threshold thre times by substructures that have been sampled already as set N left .
Then we calculate cnt i for each substructure left, which is the number of nodes in N left covered by substructure G Si .
We select n sample from substructures with top k cnt randomly.
The iteration ends when each node is covered at least thre times, or no substructure is left.
In the algorithm, n init , k , and n sample are determined by thre and nodes number to speed up the iteration and increase sampling variance, while retaining that nodes are covered as evenly as possible.
DFS
We use degree to sort nodes at the beginning of DFS and each step.
In the beginning, DFS start from node uniformly sampled from nodes with the least degree.
At each step, nodes are sorted according to degree, while nodes with the same degree are permuted randomly.
This method keeps the graph encoding permutation invariant, and reduces possible permutations a lot.
The drawback of permutation-based encoding is that the number of possible permutations becomes intractable when the graph is large.
Fortunately, the substructure size is usually not larger than 10.
R ELATED W ORK OF T OKEN U NIT
The most suitable basic unit for transformer as a token has been studied in many works in natural language processing and computational vision.
For nature language, sentences are separated into sub-words containing a variant number of primary characters (Kudo & Richardson, 2018).
In computational vision, while previous works apply transformer directly on image pixels (Parmar et al., 2018; Hu et al., 2019), recent works find it more beneficial to treat patches as tokens (Dosovitskiy et al., 2020), indicating the importance of proper segmentation of the input.
For graph transformers, tokens in most current studies are nodes.
We propose to encode substructures together with nodes in the graph as tokens in the transformer.
E MPIRICAL S TUDY ON A TTENTION C APACITY
We empirically explore the capacity of the model for the substructure, and validate our theoretical results.
We aim to answer the following three questions: (1) Do attention capacity really decreases in deeper graph transformer?
(2) Can our method, i.e., local attention with substructure tokens, help to increase model capacity of representing substructures?
(3) Can our method, i.e., local attention with substructure tokens, help to alleviate attention capacity decrease?
We compare the attention capacity on substructures of our method and Graphormer on ZINC dataset.
The substructures set is the same as substructures used in our model for fair comparison, while the base vectors E are defined as uniform distribution supported on corresponding substructures in computation.
We compute the attention capacity according to Definition 1 for a single output position, max c 1 ,c 2 ∈{ c | c ∈ [0 , 1] m ,c T 1 =1 } | c T 1 E T HW
V
O − c T 2 E T HW V O | 2 , and normalize it by the mean of l norm of all the value vector of nodes 1 n (cid:80)
i =1 ...n | h
i W V
O | 2 , to eliminate the influence of the representation norm.
Note that for our model, only node tokens are considered.
Additionally, because we explicitly tokenize substructures as tokens, we also directly compute the maximum difference between all the substructure tokens value vectors, max i,j ∈{ 1 ...m } | h
i + n W V
O − h j + n W V O | 2 , and normalize it by 1 m (cid:80) i =1 ...m |
h i + n W V
O | 2 , to illustrate the capacity of token representation of substructures.
We report the capacity of different layers for comparison.
The result of each layer is plotted in 4.
The results of 12th, 24th, 36th and 48th layer are as follows:
Layer
Graphormer 0.
[Table]
First, the result of Graphormer reveals that attention capacity of substructures decreases in deep layers.
Note that in the shallow layer the attention capacity increases, which is not contradicting our theory because we only claim that the upper bound of attention capacity decrease with depth.
However, after the 24th layer, attention capacity decreases a lot, revealing the problem of deepening graph transformer.
Second, the result of the third row indicates that local attention-based substructure tokens improve substructure representation capacity.
The difference between substructure tokens is much larger than the representation learned by attention on substructures, indicating learning substructure token representation by local attention can significantly improve the representation capacity of model, which supports our first motivation that introducing substructure-based local attention help model to encode substrutures better.
Finally, compared to Graphormer, attention capacity of our model remains high across all the layers.
Although it drops from 0.96 to 0.90 after the 24th layer, the dropping ratio is smaller than the vanilla graph transformer.
This supports our claim that substructure-based local attention helps alleviate attention capacity decrease.
Note that the capacity of substructure tokens also decreases slower.
P ARAMETER N UMBER C OMPARISON
We list parameter numbers of our model and baselines.
Note that parameter numbers of Graphormer and SAT are computed by running their official code directly, and other baselines are from papers.
The parameter number of SAT (*1) is different from reported in their papers, which may be due to code updates.
The parameter number of our base model is comparable to previous works
[Table]
and even less on CLUSTER.
The parameter number of our deeper model is also comparable to deeper baselines, while our performance surpasses them, as shown in Subsection 6.
[Table]
T IME C OMPLEXITY
The time complexity of our methods is mainly due to the larger input size, which increases the cost of transformer.
Given graph size N and sampled substructure number M , the complexity of transformer is O (( M + N ) 2 ) .
However, as we use sampling to reduce the substructure number, the substructure number is less than the graph size, i.e. M < N , so the complexity will not be more than four times of the standard transformer and remains O ( N 2 ) .
A BLATION S TUDY ON N EIGHBOR S IZE
We conduct ablation studies for random walk neighbors on CLUSTER and PATTERN datasets.
While we use size 10 neighbors in the paper, we compare the performance with sizes 5 and 15.
The result is as follows:
[Table]
The result shows that the performance remains robust to different substructure sizes, especially the deeper model.
Smaller substructures are easy for the model to learn its structure features, but they may not cover enough nodes and provide good graph structure information.
Larger substructures are more informative about graph feature, but is more difficult for stable learning, especially for a shallow model.