Hierarchical Graph Transformer with Adaptive Node Sampling
Anonymous Author s)
AffiliationAddress
Abstract
The Transformer architecture has achieved remarkable success in a number ofdomains including natural language processing and computer vision.
However,when it comes to graph-structured data, transformers have not achieved competitiveperformance, especially on large graphs.
In this paper, we identify the maindeficiencies of current graph transformers: (1) Existing node sampling strategiesin Graph Transformers are agnostic to the graph characteristics and the trainingprocess.
(2) Most sampling strategies only focus on local neighbors and neglectthe long-range dependencies in the graph.
We conduct experimental investigationson synthetic datasets to show that existing sampling strategies are sub-optimal.
Totackle the aforementioned problems, we formulate the optimization strategies ofnode sampling in Graph Transformer as an adversary bandit problem, where therewards are related to the attention weights and can vary in the training procedure.
Meanwhile, we propose a hierarchical attention scheme with graph coarseningto capture the long-range interactions while reducing computational complexity.
Finally, we conduct extensive experiments on real-world datasets to demonstratethe superiority of our method over existing graph transformers and popular GNNs.
Introduction
In recent years, the Transformer architecture [30] and its variants (e g., Bert [6] and ViT [7]) haveachieved unprecedented successes in natural language processing (NLP) and computer vision (CV).
In light of the superior performance of Transformer, some recent works [19, 36] attempt to generalize
Transformer for graph data by treating each node as a token and designing dedicated positionalencoding.
However, most of these works only focus on small graphs such as molecular graphswith tens of atoms [36].
For instance, Graphormer [36] achieves state-of-the-art performance onmolecular property prediction tasks.
When it comes to large graphs, the quadratic computationaland storage complexity of the vanilla Transformer with the number of nodes inhibits the practicalapplication.
Although some Sparse Transformer methods [28, 2, 17] can improve the efficiency ofthe vanilla Transformer, they have not exploited the unique characteristics of graph data and requirea quadratic or at least sub-quadratic space complexity, which is still unaffordable in most practicalcases.
Moreover, the full-attention mechanism potentially introduces noise from numerous irrelevantnodes in the full graph.
To generalize Transformer to large graphs, existing Transformer-based methods [8, 40, 6] on graphsexplicitly or implicitly restrict each node’s receptive field to reduce the computational and storagecomplexity.
For example, Graph-Bert [39] restricts the receptive field of each node to the nodeswith top- k intimacy scores such as Personalized PageRank (PPR).
GT-Sparse [8] only considers1-hop neighboring nodes.
We argue that existing Graph Transformers have the following deficiencies:
(1) The fixed node sampling strategies in existing Graph Transformers are ignorant of the graph
properties, which may sample uninformative nodes for attention.
Therefore, an adaptive nodesampling strategy aware of the graph properties is needed.
We conduct case studies in Section 4 tosupport our arguments.
(2) Though the sampling method enables scalability, most node samplingstrategies focus on local neighbors and neglect the long-range dependencies and global contexts ofgraphs.
Hence, incorporating complementary global information is necessary for Graph Transformer.
To solve the challenge (1), we propose A daptive N ode S ampling for G raph T ransformer (ANS-GT)and formulate the optimization strategy of node sampling in Graph Transformer as an adversarybandit problem.
Specifically in ANS-GT, we modify Exp4.P method [3] to adaptively assign weightsto several chosen sampling heuristics (e.g., 1-hop neighbors, 2-hop neighbors, PPR) and combinethese sampling strategies to sample informative nodes.
The reward is proportional to the attentionweights and the sampling probabilities of nodes, i.e. the reward to a certain sampling heuristic ishigher if the sampling probability distribution and the node attention weights distribution are moresimilar.
Then in the training process of Graph Transformer, the node sampling strategy is updatedsimultaneously to sample more informative nodes.
With more informative nodes input into theself-attention module, ANS-GT can achieve better performance.
To tackle the challenge (2), we propose a hierarchical attention scheme for Graph Transformer toencode both local and global information for each node.
The hierarchical attention scheme consistsof fine-grained local attention and coarse-grained global attention.
In the local attention, we use theaforementioned adaptive node sampling strategy to select informative local nodes for attention.
Asfor global attention, we first use graph coarsening algorithms [24] to pre-process the input graph andgenerate a coarse graph.
Such algorithms mimic a down-sampling of the original graph via groupingthe nodes into super-nodes while preserving global graph information as much as possible.
Thecenter nodes then interact with the sampled super-nodes.
Such coarse-grained global attention helpseach node capture long-distance dependencies while reducing the computational complexity of thevanilla Graph Transformers.
We conduct extensive experiments on real-world datasets to show the effectiveness of ANS-GT.
Our method outperforms all the existing Graph Transformer architectures and obtains state-of-the-art results on 6 benchmark datasets.
Detailed analysis and ablation studies further show the superiority of the adaptive node sampling module and the hierarchical attention scheme
In summary, we make the following contributions:
• We propose A daptive N ode S ampling for G raph T ransformer (ANS-GT), which modifies amulti-armed bandit algorithm to adaptively sample nodes for attention.•
In the hierarchical attention scheme, we introduce coarse-grained global attention with
graph coarsening, which helps graph transformer capture long-range dependencies whileincreasing efficiency.• We empirically evaluate our method on six benchmark datasets to show the advantage overexisting Graph Transformers and popular GNNs.
Related Work2.1 Transformers for Graph
Recently, Transformer [30] has shown its superiority in an increasing number of domains [6, 7, 38],e.g.
Bert [6] in NLP and ViT [7] in CV.
Existing works attempting to generalize Transformer to graphdata mainly focus on two problems: (1) How to design dedicated positional encoding for the nodes;(2) How to alleviate the quadratic computational complexity of the vanilla Transformer and scalethe Graph Transformer to large graphs.
As for the positional encoding, GT [8] firstly uses Laplacianeigenvectors to enhance node features.
Graph-Bert [39] studies employing Weisfeiler-Lehman code toencode structural information.
Graphormer [36] utilizes centrality encoding to enhance node featureswhile incorporating edge information with spatial (SPD-indexed attention bias) and edge encoding.
SAN [19] further replaces the static Laplacian eigenvectors with learnable positional encodings anddesigns an attention mechanism that distinguishes local connectivity.
For the scalability issue, oneimmediate idea is to restrict the number of attending nodes.
For example, GAT [31] and GT-Sparse[8] only consider the 1-hop neighboring nodes; Gophormer [40] uses GraphSAGE [10] samplingto uniformly sample ego-graphs with pre-defined maximum depth; Graph-Bert [39] restricts the receptive field of each node to the nodes with top-k intimacy scores (e.g., Katz and PPR).
However,these fixed node sampling strategies sacrifice the advantage of the Transformer architecture.
SAC[20] tries to use an LSTM edge predictor to predict edges for self-attention operations.
However, thefact that LSTM can hardly be parallelized reduces the computational efficiency of the Transformer.
2.2 Sparse Transformers
In parallel, many efforts have been devoted to reducing the computational complexity of the Transformer in the field of NLP [21] and CV [29].
In the domain of NLP, Longformer [2] applies block-wiseor strode patterns while only fixing on fixed neighbors.
Reformer [17] replaces dot-product attentionby using approximate attention computation based on locality-sensitive hashing.
Routing Transformer [28] employs online k-means clustering on the tokens.
Linformer [32] demonstrates that theself-attention mechanism can be approximated by a low-rank matrix and reduces the complexity from
O ( n 2 ) to O ( n ) .
As for vision transformers, Swin Transformer [22] proposes the shifted windowingscheme which brings greater efficiency by limiting self-attention computation to non-overlappinglocal windows while also allowing for cross-window connection.
Focal Transformer [35] presents anew mechanism incorporating both fine-grained local and coarse-grained global attention to captureshort- and long-range visual dependencies efficiently.
However, these sparse transformers do not takethe unique graph properties into consideration.
2.3 Graph Neural Networks and Node Sampling
Graph neural networks (GNNs) [16, 10, 33, 25, 11] follow a message-passing schema that iterativelyupdates the representation of a node by aggregating representations from neighboring nodes.
Whengeneralizing to large graphs, Graph Neural Networks face a similar scalability issue.
This is mainlydue to the uncontrollable neighborhood expansion in the aggregation stage of GNN.
Several nodesampling algorithms have been proposed to limit the neighborhood expansion, which mainly fallsinto node-wise sampling methods and layer-wise sampling methods.
In node-wise sampling, eachnode samples k neighbors from its sampling distribution, then the total number of nodes in the l -thlayer becomes O ( k l ) .
GraphSage [10] is one of the most well-known node-wise sampling methodswith the uniform sampling distribution.
GCN-BS [23] introduces a variance reduced sampler basedon multi-armed bandits.
To alleviate the exponential neighbor expansion O ( k l ) of the node-wisesamplers, layer-wise samplers define the sampling distribution as a probability of sampling nodesgiven a set of nodes in the upper layer [4, 14, 42].
From another perspective, these sampling methodscan also be categorized into fixed sampling strategies [10, 4, 42] and adaptive strategies [23, 37].
However, none of the above sampling methods in GNNs can be directly applied in Graph Transformeras Graph Transformer does not follow the message passing schema.
Preliminaries
3.1 Problem Definition
Let G =
( A, X ) denote the unweighted graph where A ∈ R n × n represents the symmetric adjacencymatrix with n nodes, and X ∈ R n × p is the attribute matrix of p attributes per node.
The element A ijin the adjacency matrix equals to 1 if there exists an edge between node v i and node v j , otherwise
A ij = 0 .
The label of node v i is y i .
In the node classification problem, the classifier has theknowledge of the labels of a subset of nodes V L .
The goal of semi-supervised node classification isto infer the labels of nodes in V \ V L by learning a classification function.
3.2 Transformer Architecture
The Transformer architecture consists of a series of Transformer layers [30].
Each Transformer layerhas two parts: a multi-head self-attention (MHA) module and a position-wise feed-forward network(FFN).
Let H =
[ h 1 , · · · , h m ] ⊤ ∈ R m ×
d denote the input to the self-attention module where d isthe hidden dimension, h i ∈ R
d × 1 is the hidden representation at position i , and m is the number ofpositions.
The MHA module firstly projects the input H to query-, key-, value-spaces, denoted as
Q , K , V , using three matrices W Q ∈ R d × d K ,
W K ∈ R d × d K and W V ∈ R
d × d V :
[Equation]
Then, in each head i ∈ { 1 , 2 , . . .
, B } ( B is the total number of heads), the scaled dot-productattention mechanism is applied to the corresponding { Q i , K i , V i } :
[Equation]
Finally, the outputs from different heads are further concatenated and transformed to obtain the finaloutput of MHA:
[Equation]
where W O ∈ R d × d .
In this work, we employ d K =
d V = d/B .
3.3 Graph Coarsening
The goal of Graph Coarsening [27, 24, 15] is to reduce the number of nodes in a graph by clusteringthem into super-nodes while preserving the global information of the graph as much as possible.
Given a graph G =
( V, E ) ( V is the node set and E is the edge set), the coarse graph is a smallerweighted graph G ′
=
( V ′ , E ′ ) .
G ′ is obtained from the original graph by first computing a partition{ C 1 , C 2 , · · · , C | V ′ | } of V , i.e., the clusters C 1 · · · C | V ′ | are disjoint and cover all the nodes in V .
Each cluster C i corresponds to a super-node in G ′ The partition can also be characterized by a matrixˆ P ∈ { 0 , {+1 } | V |×| V ′ | , with ˆ P ij = 
+}
if and only if node v i in G belongs to cluster C j .
Its normalizedversion can be defined by P ≜ ˆ PD − 12 , where D is a | V ′ | × | V ′ | diagonal matrix with | C i | as its i -thdiagonal entry. The feature matrix and weighted adjacency matrix of G ′ are defined by X ′ ≜ P T Xand A ′ ≜ P T AP .
After Graph Coarsening, the number of nodes/edges in G ′ is significantly smallerthan that of G .
The coarsening rate can be defined as c = | V ′ | | V | .
Motivating Observations
To generalize Transformer to large graphs, existing Graph Transformer models typically choose tosample a batch of nodes for attention.
However, real-world graph datasets exhibit different properties,which makes a fixed node sampling strategy unsuitable for all kinds of graphs.
Here, we present asimple yet intuitive case study to illustrate how the performance of Graph Transformer changes withdifferent node sampling strategies.
The main idea is to use four popular node sampling strategiesfor node classification: 1-hop neighbors, 2-hop neighbors, PPR, and KNN.
Then, we will check theperformance of Graph Transformer on graphs with different properties.
If the performance dropssharply when the property varies, it will indicate that graphs with different properties may requiredifferent node sampling strategies.
[Figure]
We conduct experiments on the Newman artificial networks [9] since it enable us to obtain networks withdifferent properties easily.
More detailed settings canbe found in Appendix A.
Here, we consider the property of homophily/heterophily as one example.
Following previous works [41], the degree of homophilyα can be defined as the fraction of edges in a network connecting nodes with the same class label, i.e.α ≜ | ( v i ,v
j ) ∈ E
∧ y i =
y j | | E | .
Graphs with α closer to 1 tendto have more edges connecting nodes within the sameclass, i.e. strong homophily; whereas networks withα closer to 0 have more edges connecting nodes indifferent classes, i.e. strong heterophily.
As shown in Figure 1, there is no sampling strategythat dominates other strategies across the spectrum ofα , which supports our claim that an adaptive samplingmethod is needed for graphs with different properties.
For graphs with strong homophily (e.g.,α = 1 . 0 ), it is easy to obtain high accuracy by sampling 1-hop neighbors or nodes with top PPRscores.
On the other hand, for graphs with strong heterophily (e.g., α = 0 . 05 ), the accuracy of using2-hop neighbors as neighborhoods (i.e., 73 . 1% ) is much higher than that of using 1-hop neighbors and PPR.
The probable reason is that the homophily ratio of 2-hop neighbors may rise with theincrease of inter-class edges.
Finally, Graph Transformer with KNN node sampling can get the mostconsistent results since KNN only calculates the similarity of attributes.
Thus, KNN achieves the bestperformance (i.e., 77 . 2% accuracy) when all the nodes are connected randomly, i.e. α = 0 .
25 for the
Newman network.
Hence, considering different graph properties (e.g., homophily/heterophily), Graph Transformershould adaptively sample the most informative nodes for attention.
These observations motivate us todesign the proposed hierarchical Graph Transformer with adaptive node sampling in Section 5.
The Proposed Method
In light of the limitations of existing Graph Transformers for large graphs and the motivatingobservation in the previous section, we propose two effective methods for Graph Transformer toadaptively sample informative nodes and capture the long-range coarse-grained dependencies in thissection.
We also show that our method has a computational complexity of O ( n ) .
The overview ofthe model framework is shown in Figure 2.
5.1 Adaptive Node Sampling
Our Adaptive Node Sampling module aims to adaptively choose the batch of most informative nodesby a multi-armed bandit mechanism.
In our setting, it is intuitive that the contributions of nodes tothe learning performance can be time-sensitive.
Meanwhile, the rewards which are associated withthe model training process are not independent random variables across the iterations.
The abovesituation can satisfy the adversarial setting in the multi-armed bandit problem [1].
To adaptivelychoose the most informative nodes with the designed sampling strategies, we adjust the method ALBLproposed in [13], which modifies the EXP4.P method [3].
EXP4.P possesses a strong theoreticalguarantee for the adversarial setting.
Formally, let w t =
( w t 1 , · · · , w tK ) be the adaptive weight vector in iteration t , where the k -th nonnegative element w tk is the weight corresponding to the k -th node sampling strategy.
The weightvector w t is then scaled to a probability vector p t =
( p t 1 , · · · , p tK ) where p tk ∈ [ p min , 1] withp min > 0 .
Our ANS-GT adaptively sample nodes based on the probability vector and then obtainsthe reward of the action.
For each center node, we consider the sampling probability matrix Q t ∈ R K × n , where K is thenumber of sampling heuristics and n is the number of nodes in the graph.
Specifically, Q tk,j denotesthe k -th sampling strategy’s preference on selecting node j in iteration t and Q t is normalized tosatisfy (cid:80) nj =1 Q tk,j = 1 .
Note that our ANS-GT is a general framework and is not restricted to acertain set of sampling heuristics.
In our work, we adopt four representative sampling heuristics:1-/2-hop neighbors : We adopt the normalized adjacency matrix (cid:101)
A
= ˆ D − 12 ˆ A
ˆ D − 12 for 1-hop neighbors and (cid:101)
A 2 for 2-hop neighbors, where ˆ A =
A
+ I is the adjacency matrix of the graph G with selfconnections added and ˆ D is a diagonal matrix with ˆ D ii
= (cid:80)
j ˆ A ij .
KNN : We adopts the cosine similarity of node attributes to measure the similarities of nodes.
Mathematically, the similarity score S ij between node i and j is calculated as S ij = x i · x j / ( | x i |·| x j | )where x i is the feature vector of node v i .
PPR : The Personalized PageRank [26] matrix S is calculated as:
S = c
( I − (1 − c ) A ) , where factorc ∈ [0 , 1] (set to 0.15 in our experiments).
A denotes the column-normalized adjacency matrix.
Finally, given the probability vector p t and the node sampling matrices Q t , the final node samplingprobability is:
[Equation]
We introduce a novel reward scheme based on the attention weights which is intrinsic in Transformer.
Formally, given an attention matrix A = Softmax( QK T ) / √ d , we use the first row of the attentionmatrix, i.e., A 1 ,i multiplying the magnitude of corresponding value V i to represent the significance
[Figure]
PPR
Linear
Linear
Linear Q
K
V
Matmul Matmul
Softmax
Scale
L ×
[Figure]
… Attribute
[Figure]
Input Graph
Proximity Encoding
Sampling Strategies
Graph Coarsening
Adaptive Node Sampling Module … Input Sequence
Attention of the center node over the other nodes
Score of each node:
Calculate reward:
Update
[Figure]
of each node to the center node: s i =
A 1 ,i × ∥V i ∥ .
In the situation of multiple heads and layersin the Transformer, we average the significance scores in multiple attention matrices.
The rewardto the k -th sampling strategy is: r k = (cid:80) Ni
=
s i Q tki ψ ti , where N is the number of sampled nodes foreach center node.
r k can be interpreted as the dot product between the significance score vectorand the normalized sampling probability vector.
The intuition behind the reward design is that thereward to a certain sampling heuristic is higher if the sampling probability distribution and the nodesignificance score distribution are closer.
Thus, exploiting such a sampling heuristic can help graphtransformer sample more informative nodes.
Finally, we update w t with the reward.
In experiments,for the efficiency and stability of training, we update the sampling weights and resample nodes every T epochs.
The pseudo-code of ANS-GT is listed in Algorithm 1.
5.2 Hierarchical Graph Attention
We argue that most node sampling strategies (e.g., 1-hop neighbors) focus on local informationand neglect long-range dependencies or global contexts.
Therefore, to efficiently capture both thelocal and global information in the graph, we propose a novel hierarchical graph attention schemeincluding fine-grained attention and coarse-grained attention.
Specifically, we use the proposedadaptive node sampling for local fined-grained attention.
On the other hand, we adopt the graphcoarsening algorithm [24] to generate the coarsened graph G ′ .
The sampled n s super-nodes from
G ′ are used to capture long-range dependencies.
Similar to [40], we also use n g global nodes withlearnable features to store global context information.
Finally, the hierarchical nodes are concatenatedwith the center nodes as the input sequences.
For the positional encoding, we use the proximity encoding in [40]: Φ m ( v i , v j ) =
(cid:101)
A m [ i, j ] , m ∈{ 0 , · · · , M − 1 } , where (cid:101)
A denotes the normalized adjacency matrices with self-loop.
Note thatour framework is agnostic to the positional encoding scheme and we left other positional encodingmethods such as Laplacian eigenvectors [8] for future exploration. We follow the Graphormer
[Equation]
framework to obtain the output of the l -th transformer layer, H ( l ) :
[Equation]
We apply the layer normalization (LN) before the multi-head self-attention (MHA) and the feedforward network (FFN).
5.3 Optimization and Inference
In the training and inference, we sample S input sequences for each center node and use the centernode representation from the final Transformer layer z ( s ) c for prediction.
Note that the computationalcomplexity is controllable by choosing suitable number of sampled nodes.
A MLP (Multi-Layer
Perceptron) is used to predict the node class:
[Equation]
where (cid:101) y c ∈ R C × 1 stands for the classification result, C stands for the number of classes.
In thetraining process, we optimize the average cross entropy loss of labeled training nodes V L :
[Equation]
where y i ∈ R C × 1 is the ground truth label of center node v i .
In the inference stage, we take abagging aggregation to improve accuracy and reduce variance:
[Equation]
5.4 Computational Complexity
Compared with existing Graph Transformers, ANS-GT requires extra computational costs on graphcoarsening and sampling weights update.
Here we want to show that the overall computationalcomplexity of ANS-GT is linear with the number of nodes n .
Hence, ANS-GT is scalable to largegraphs.
First, the computational complexity of graph coarsening is linear with n [24] and we onlyneed to do it once before training.
Second, the computational cost of self-attention calculation in oneepoch is O ( n S ( N + n s + n g ) 2 ) where the number of sampled nodes N , the number of sampledsuper-nodes n s , the number of global nodes n g , and the number of augmentations S are specifiedconstants.
Finally, the cost of updating sampling weights is linear to n , which is mainly attributed tothe computation of rewards.
Empirical efficiency analyses of ANS-GT are shown in the Appendix C.
[Table]
25±0.
89.56±0.
65.42±0.
40.10±1.
45.88±0.
      (SRFK      6 D P S OL QJ  : H L
JK W KRSKRS.1133 (a) Cora       (SRFK         6 D P S OL QJ  : H L JK W KRSKRS.1133 (b) Citeseer       (SRFK      6 D P S OL QJ  : H L
JK W KRSKRS.1133 (c) Actor       (SRFK       6 D P S OL QJ  : H L JK W KRSKRS.1133 (d) Squirrel
Experiments
6.1 Experimental Setup
Datasets.
To comprehensively evaluate the effectiveness of ANS-GT, we conduct experiments onthe six benchmark datasets including citation graphs Cora, CiteSeer, and PubMed [16]; Wikipediagraphs Chameleon, Squirrel, and the Actor co-occurrence graph [5].
We set the train-validation-testsplit as 60%/20%/20%.
The statistics of datasets are shown in the Appendix C.
Baselines.
To evaluate the effectiveness of ANS-GT on graph representation learning, we compareit with 9 baseline methods, including five popular GNN methods, i.e. GCN [16], GAT [31], GraphSAGE [10], JKNet [34], and APPNP [18], along with four state-of-the-art Graph Transformers, i.e.
GT [8], SAN [19], Graphormer [36], and Gophormer [40].
We use node classification accuracy asthe evaluation metric.
Implementation Details.
We adopt AdamW as the optimizer and set the hyper-parameter ϵ to 1e-8and ( β 1 , β 2 ) to (0.99,0.999).
The peak learning rate is set to 2e-4 with a 100 epochs warm-up stagefollowed by a linear decay learning rate scheduler.
We adopt the Variational Neighborhoods [24]with a coarsening rate of 0.01 as the default coarsening method.
All models were trained on one
NVIDIA Tesla V100 GPU.
Parameter Settings.
In the default setting, the dropout rate is set to 0.5, the end learning rate is setto 1e-9, the hidden dimension d is set to 128, the number of training epochs is set to 1,000, updateperiod T is set to 100, N is set to 20, M is set to 10, and the number of attention head H is set as8.
We tune other hyper-parameters on each dataset based on by grid search.
The searching spaceof batch size, number of data augmentation S , the number of layers L , number of sampled nodes,number of sampled super-nodes, number global nodes are { 8 , 16 , 32 } , { 4 , 8 , 16 , 32 } , { 2 , 3 , 4 , 5 , 6 } ,{ 10 , 15 , 20 , 25 } , { 0 , 3 , 6 , 9 } , { 1 , 2 , 3 } respectively.
6.2 Effectiveness of ANS-GTNode Classification Performance.
The node classification results are shown in Table 1.
Weapply 3 independent runs on random data splitting and report the means and standard deviations.
We have the following obervations: (1) Generally, we observe that ANS-GT overperforms all
Graph Transformer baselines and achieves state-of-the-art results on all 6 datasets, which demon- strates the effectiveness of our proposed model.
(2) We note that some Graph Transformer baselines achieve poor performance on node classification (e.g., GT only obtains 25 .
68% on Squirrel)
compared with graph neural network models.
This is probably due to the full graph attention mechanisms or the fixed node sampling schemes of existing Graph Transformers.
For instance, ANS-GT achieves an accuracy of 45.88 % on Squirrel while the best baseline has 43.15 % .
[Figure]
Effectiveness of Adaptive Node Sampling.
Our proposed adaptive node sampling module can adjust theweights for sampling based on the rewards as the training progresses.
To evaluate its effectiveness and givemore insights into the ANS module, we show the normalized sampling weights as a function of trainingepochs on four datasets in Figure 3.
Generally, weobserve that the sampling weights of different sampling strategies are time-sensitive and gradually stabilize with the increase of the number of epochs.
Interestingly, we find PPR and 1-hop neighbors achieveshigh weights on Cora while 2-hop neighbors dominateother sampling strategies on Squirrel.
This may be explained by the fact that Cora and Squirrel are stronghomophily/heterophily dataset respectively.
For Citeseer and Actor, the weights of KNN firstly goes up andgradually decreases.
This is probably due to the reasonthat nodes with similar attributes are most useful forthe training at the beginning stage; local neighbors such as nodes with high PPR scores are moreuseful at the fine-tuning stage.
Furthermore, Figure 4 shows the ablation studies of the adaptivenode sampling module.
We can observe that ANS-GT has a large advantage compared to its variantwithout the adaptive sampling module denoted as HGT (e.g., On Chameleon, ANS-GT
achieves65.42 % while HGT only has 57.20 % ).
[Table]
Graph Coarsening Methods.
In ANS-GT, weapply a hierarchical attention mechanism wherethe interactions between the center node and supernodes generated by graph coarsening are considered.
Here, we evaluate ANS-GT with differentcoarsening algorithms and different coarseningrates.
Specifically, the considered coarsening algorithms include Variation Neighborhoods (VN)[24], Variation Edges (VE) [24], and Algebraic(JC)
We vary the coarsening rate from 0.01to 0.50.
In Table 2, we can observe that there is nosignificant difference between different coarseningalgorithms, indicating the robustness of ANS-GTw.r.t.
them.
As for the coarsening rate, the resultsindicate that the coarsening rate of 0.01 to 0.10has the best performance.
Conclusion
Motivated by the obstacles to generalize Transformer to large graphs, we propose A daptive N ode
S ampling for G raph T ransformer (ANS-GT), which modifies a multi-armed bandit algorithm toadaptively sample nodes for attention in this paper.
To incorporate long-range dependencies andglobal contexts, we further design a hierarchical graph attention scheme in which coarse-grainedattention is achieved with graph coarsening.
We empirically evaluate our method on six benchmarkdatasets to show the advantage over existing Graph Transformers and popular GNNs.
The detailedanalysis demonstrates that the adaptive node sampling module could effectively adjust the samplingstrategies according to graph properties.
Finally, We hope our work can help Transformer generalizeto the graph domain and encourage the unified modeling of multi-modal data.
Checklist
For all authors...
Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
Did you describe the limitations of your work?
[Yes] Please see Appendix.
Did you discuss any potential negative social impacts of your work?
[Yes] Please seeAppendix.
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
Did you state the full set of assumptions of all theoretical results?
[N/A]
Did you include complete proofs of all theoretical results?
[N/A]If you ran experiments...
Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[No] The code will be released
once the paper is accepted.
Did you specify all the training details (e.g., data splits, hyperparameters, how they were
chosen)?
[Yes]
Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[Yes]
Did you include the total amount of computing and the type of resources used (e.g., type of
GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
If your work uses existing assets, did you cite the creators?
[Yes]
Did you mention the license of the assets?
[N/A]3.
Did you include any new assets either in the supplemental material or as a URL?
[N/A]4.
Did you discuss whether
and how consent was obtained from people whose data you’re
using/curating?
[N/A]
Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
Did you describe any potential participant risks, with links to Institutional Review Board(IRB) approvals, if applicable?
[N/A]3.
Did you include the estimated hourly wage paid to participants and the total amount spenton participant compensation?
[N/A]