A DAPTIVE S TRUCTURAL F INGERPRINTS FOR G RAPH A TTENTION N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Many real-world data sets are represented as graphs, such as citation links, social media, and biological interaction.
The volatile graph structure makes it non-trivial to employ convolutional neural networks (CNN’s) for graph data processing.
Recently, graph attention network (GAT) has proven a promising framework by combining graph neural networks with attention mechanism, so as to achieve massage passing in graphs with arbitrary structures.
However, the attention in GAT is computed mainly based on the similarity between the node content, while the structures of the graph remains largely unemployed (except in masking the attention out of one-hop neighbors).
In this paper, we propose an “ad aptive structural fingerprint” (ADSF) model to fully exploit both topological details of the graph and content features of the nodes.
The key idea is to contextualize each node with a weighted, learnable receptive ﬁeld encoding rich and diverse local graph structures.
By doing this, structural interactions between the nodes can be inferred accurately, thus improving subsequent attention layer as well as the convergence of learning.
Furthermore, our model provides a useful platform for different subspaces of node features and various scales of graph structures to “cross-talk” with each other through the learning of multi-head attention, being particularly useful in handling complex real-world data.
Encouraging performance is observed on a number of benchmark data sets in node classiﬁcation.
I NTRODUCTION
Many real-world data set are represented naturally as graphs.
For example, citation networks specify the citation links among scientiﬁc papers; social media often need to explore the signiﬁcant amount of connections between users; biological processes typically involve complex interactions such as protein-protein-interaction (PPI).
In these scenarios, the complex structures such as the graph topology or connectivities encode crucial domain-speciﬁc knowledge for the learning and prediction tasks.
Examples include node embedding or classiﬁcation, graph classiﬁcation, and so on.
The complexity of graph-structured data makes it non-trivial to employ traditional convolutional neural networks (CNN’s).
The CNN architecture was originally designed for images whose pixels are located on a uniform grids, and so the convolutional ﬁlters can be reused everywhere without having to accommodate local structure changes (LeCun & Kavukcuoglu, 2010) .
More recently,
CNN was used in natural language processing where the words of a sentence can be considered as a uniform chain, and showed great power in extracting useful semantic features (Kim, 2014).
However, extending CNN to deal with arbitrary structured graphs beyond uniform grids or chains can be quite non-trivial.
To solve this problem, graph neural networks (GNN) were early proposed by Gori et al.
and Sperduti (1997), which adopt an iterative process and propagate the state of each node, followed by a neural network module to generate the output of each node, until an equilibrium state is reached.
Recent development of GNN can be categorized into spectral and nonspectral approaches.
Spectral approaches employ the tools in signal processing and transform the convolutional operation in the graph domain to much simpler operations of the Laplacian spectrum (Bruna et al., 2014), and various approaches have been proposed to localize the convolution in either the graph or spectral domain (Henaff et al., 2015; Defferrard et al., 2016; Kipf & Welling, 2017).
Non-spectral approaches deﬁne convolutions directly on the graph within spatially close nodes.
As a result, varying node structures have to be accommodated through various processing steps such as ﬁxed-neighborhood size sampling (Hamilton et al., 2017), neighborhood normalization (Niepert et al., 2016), and learning a weight matrix for each node degree (Duvenaud et al., 2015) or neighborhood size (Hamilton et al., 2017).
Recently, graph attention network (GAT) proves a promising framework by combining graph neural networks with attention mechanism in handling graphs with arbitrary structures (Velickovic et al., 2017).
The attention mechanism allows dealing with variable sized input while focusing on the most relevant parts, and has been widely used in sequence modelling (Bahdanau et al., 2015; Devlin et al., 2019; Vaswani et al., 2017), machine translation (Luong et al., 2015), and visual processing (Xu et al., 2015).
The GAT model further introduces attention module into graphs, where the hidden representation of the nodes are computed by repeatedly attending over their neighbors’ features, and the weighting coefﬁcients are calculated inductively based on a self-attention strategy.
State-of-theart performance has been obtained on tasks of node embedding and classiﬁcation.
The attention in GAT is computed mainly based on the content of the nodes; the structures of the graph, on the other hand, are simply used to mask the attention, e.g., only one-hop neighbors will be attended.
However, we believe that rich structural information such as the topology or “shapes” of local edge connections should provide a more valuable guidance on learning node representations.
For example, in social networks or biological networks, a community or pathway is oftentimes composed of nodes that are densely inter-connected with each other but several hops away.
Therefore, it can be quite beneﬁcial if a node can attend high-order neighbors from the same community, even if they show no direct connections.
To achieve this, simply checking k-hop neighbors would seem insufﬁcient and a thorough exploration of structural landscapes of the graph becomes necessary.
In order to fully exploit rich, high-order structural details in graph attention networks, we propose a new model called “adaptive structural ﬁngerprints”.
The key idea is to contextualize each node within a local receptive ﬁeld composed of its high-order neighbors.
Each node in the neighborhood will be assigned a non-negative, closed-form weighting based on local information propagation procedures, and so the domain (or shape) of the receptive ﬁeld will adapt automatically to local graph structures and the learning task.
We call this weighted, tunable receptive ﬁeld for each node its “structural ﬁngerprint”.
We then deﬁne interactions between two structural ﬁngerprints, which will be used in conjunction with node feature similarities to compute a ﬁnal attention layer.
Furthermore, our approach provides a useful platform for different subspaces of the node features and various scales of local graph structures to coordinate with each other in learning multi-head attention, being particularly beneﬁcial in handling complex real-world graph data sets.
The rest of the paper is organized as follows.
In Section 2, we introduce the proposed method, including limitation of content-based graph attention, construction of the adaptive structural ﬁngerprints, and the whole algorithm workﬂow.
In Section 3, we discuss related work.
Section 4 reports empirical evaluations and the last section concludes the paper.
E XPLOITING G RAPH S TRUCTURAL D ETAILS IN A TTENTION 2. L IMITATIONS OF CONTENT - BASED ATTENTION
The “closeness” between two nodes should be determined from both their content and structure.
Here we illustrate the importance of detailed graph structures in determining node similarities.
In
Figure 1(a)
, suppose the feature similarity of node-pairs (A,B) and (A,C) are similar.
Namely, content-based attention will be similar for this two node pairs.
However, from structural point of view, the attention between (A,B) should be much stronger than that for (A,C).
This is because A and
B are located in a small, densely inter-connected community, and they share a signiﬁcant portion of common neighbors; while node A and C does not have any common neighbor.
Therefore we should anticipate a stronger attention for (A,B) considering this structural indicator.
In Figure 1 (b), node A and node B are not direct neighbors, and connecting them takes three edges.
As a result, their features will not directly affect each other in the message passing of GAT 1 .
However, both A and B are strongly connected to a dense community, and node B further connects to the community hub.
Therefore, it is reasonable for A and B to directly affect each other.
[Figure]
In these examples, feature based similarity alone (or even plus checking k -hop neighbors) can insufﬁcient in computing a faithful attention.
One may need to take into account structural details of higher-order neighbors and how they interact with each other.
In the literature, structural clues have long been exploited in solving problems of clustering, semi-supervised learning, and community detection .
For example, normalized cut minimizes edge connections between clusters to recover node grouping (Shi & Malik, 2000); mean-shift algorithm uses peaks of the density function to identify clusters in the data (Comaniciu & Meer, 2002); low-density separation (Chapelle & Zien, 2005) assumes that class boundaries pass through low-density regions in the feature space, leading to successful semi-supervised learning algorithms; in community detection, densely connected subgraphs are the main indicator of the existence of communities (Girvan & Newman, 2002).
In the following, we show how to build “adaptive structural ﬁngerprint” to extract informative structural clues to improve graph attention and henceforth node embedding and classiﬁcation.
2. A DAPTIVE S TRUCTURAL F INGERPRINTS
The key to exploiting the structural information is the construction of the so-called ”adaptive structural ﬁngerprints”, by placing each node in the context of the its local ”receptive ﬁeld”.
Figureprovides an illustration.
For a given node i , consider a spanning process that locates a local subgraph around i , for example, all the nodes within the k -hop neighbors of the center node i .
Denote the resultant subgraph as ( V i , E i ) , where V i and E i are the set of nodes (dark red) and edges (black) in this local neighborhood.
In the meantime, each node in this neighborhood will be assigned a non-negative weight, denoted by w i ∈ R n i × 1 .
The weight speciﬁes the importance of each node in their contribution to shaping the receptive ﬁled, and determines the effective “shape” and domain of the ﬁeld.
The structural ﬁngerprint of node i will be formally deﬁned as F i =
( V i , w i ) .
[Figure]
9 10      8 Local graph (structural fingerprint)
Gaussian decay: weights
Gaussian decay: contour 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
Randomwalk decay: weights
Randomwalk decay: contour 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
[Figure]
2. C ONSTRUCTION OF THE S TRUCTURAL F INGERPRINTS
Intuitively, the weight of the nodes should decay with their distance from the center of the ﬁngerprint.
A simple idea is to compute the weight of the node j using a Gaussian function of its distance from the center node i , i.e., w i ( j ) = exp( − dis ( i,j ) 2 2 h) , which we call Gaussian decay.
Alternatively, we can map the node distance levels [1 , 2 , ..., k ] to weight levels u
= [ u 1 , u 2 , ..., u k ] which are nonnegative and monotonic; we call nonparametric decay, which is more ﬂexible and will be used in our experiments.
In both cases, the decay parameter (Gaussian bandwidth h or nonparametric decay proﬁle u ) can be optimized, making the structural ﬁngerprint more adaptive to the learning process.
In practice, it is more desirable if node weights can be automatically adjusted by local graph structures (shapes or density of connections).
To achieve this, we propose to use Random Walk with
Restart (RWR).
Random walks were ﬁrst developed to explore global topology of a network by simulating a particle iteratively moving among neighboring nodes (Lovasz, 1993).
If the particle is forced to always restart in the same node (or set of “seed” nodes), random walk with restart (RWR) can then quantify the structural proximity between the seed(s) and all the other nodes in the graph, which has been widely used in information retrieval (Tong et al., 2006; Pan et al., 2004).
Consider a random walk with restart on a structural ﬁngerprint centered around node i , with altogether n i nodes and adjacency matrix E i .
The particle starts from the center node i and randomly walks to its neighbors in V i with a probability proportional to edge weights.
In each step, it also has a certain probability to return to the center node.
The iteration can be written as
[Equation]
where ˜ E is the transition probability matrix by normalizing columns of E i , c ∈ [0 , 1] is a tradeoff parameter between random walk and restart, and e i is a vector of all zeros except the entry corresponding to the center node i .
The converged solution can be written in closed form as
[Equation]
The w i quantiﬁes the proximity between the center node i and all other nodes of the ﬁngerprint, and in the meantime naturally reﬂects local structural details of the graph.
The c controls the decaying rate (effective size) of the ﬁngerprint: if c = 0 , w i will all be zeros except the i th node; if c = 1 , w i will be the stationary distribution of a standard random walk on graph ( V i , E i ).
In practice, c will be optimized so that the ﬁngerprint adapts naturally to both the graph structure and the learning task.
In Figure 3, we illustrate Gaussian-based and RWR-based ﬁngerprint weights.
We created a toy example of local receptive ﬁeld, which for convenience is a uniform grid but with a small denser
“patch” on the right bottom.
As can be expected, the Gaussian based weight decay has contours that are center-symmetric.
In comparison, the RWR automatically takes into account salient local structures and so the contours will be biased towards the dense subgraph.
This is particularly desirable in case the center node is close to (or residing in) a community; it will then be represented more closely by the nodes from community, achieving the effect of a “structural attractor”.
As can be seen, the structural attractor effect in building the ﬁngerprint using RWR coincides well with commonly used assumptions in clustering problems.
Namely, the structural ﬁngerprint of a node will emphasize more on densely inter-connected neighbors within predeﬁned range.
Since highly weighted nodes are more likely to come from the same cluster as the center node, the ﬁngerprint of each node is supposed to provide highly informative guidance on its structural identity, thus improving evaluation of node similarites through the ﬁngerprints, and ﬁnally the graph attention and classiﬁcation performance.
2. A LGORITHM D ESCRIPTION
Having constructed the structural ﬁngerprints for each node, we will exploit both the content and structural details of the graph in the GAT framework (Velickovic et al., 2017).
Our algorithm is illustrated in Figure 4.
[Figure]
Suppose we want to compute attention coefﬁcients between a pair of nodes i and j , each with their features and structural ﬁngerprints.
Content-wise, features of the two nodes will be used to compute their content similarity; structure-wise, structural ﬁngerprints of the two nodes will be used to evaluate their interaction.
Both scores will be incorporated in the attention layer, which will then be used in the message passing step to update node features.
Following Velickovic et al.
, we also apply a transform on the features, and apply multiple steps of message passing.
More speciﬁcally, given a graph of n nodes G =
( V, E ) where V is the set of the nodes and E is the set of the edges; let { h i } ni =1 be the d -dimensional input features for each node.
We will follow the basic structure of GAT algorithm (Velickovic et al., 2017) and describe our algorithms as follows.
• Step 1.
Evaluate the content similarity between node i and j as
[Equation]
where W ∈ R d × d is the transformation that maps the node features to a latent space, and function A fea ( · , · ) computes similarity (or interaction) between two feature h i and h j ,
[Equation]
• Step 2.
Evaluate structural interaction between the structural ﬁngerprints of node i and j ,
[Equation]
where A str ( F i , F j ) quantiﬁes the interaction between two ﬁngerprints.
Let w i and w j be the node weights of the ﬁngerprints for node i and j , as discussed in Section 2.3.
Then we can adopt the weighted Jacard similarity to evaluate the structural interactions, as
[Equation]
Here, with an abuse of notations, we have expanded w i and w j to all the nodes in V
i ∪ V i by ﬁlling zeros.
We can consider smooth version of the min/max function
[Equation]
or other smooth alternative 2 .
• Step 3.
Normalize (sparsify) feature similarities (2) and the structural interactions as (4)
[Equation]
and then combine them to compute the ﬁnal attention
[Equation]
Here α ( · ) and β ( · ) are transfer functions (such as Sigmoid) that adjust feature similarity and structure interaction scores before combining them.
For simplicity (and in our experiments), we use scalar α and β , which leads to a standard weighted average.
• Step
Perform message passing to update the features of each node as
[Equation]
Our algorithm has a particular advantage when multi-head attention is pursued.
Note that our model simultaneously calculates two attention scores: the content-based e ij and structure-based s ij , and combine them together.
Therefore each attention head will accommodate two sets of parameters: (1) those of the content-based attention, W and a (3), which explores the subspace of the node features, and (2) those of the structure-based attention, c (1), which explores the decay rate of the structural ﬁngerprint.
As a result, by learning an optimal mixture of the two attention (6), our model provides a ﬂexible platform for different subspaces of node features and various scales of local graph structures to “cross-talk” with each other, which can be quite useful in exploring complex real-world data.
Computationally, the local receptive ﬁeld will be conﬁned within a k -hop neighborhood around each node, and both structural attention and content-based attention will be considered when they distance is below a certain threshold k (cid:48) .
We usually choose k, k (cid:48) as small integers with k ≤ k (cid:48) , and use breadth-ﬁrst-search (BFS) to localize the neighborhood.
As a result, the complexity involved in structure exploration will be O ( n |N k (cid:48) | ) , where |N k (cid:48) | is the averaged k (cid:48) -hop-neighbor size.
C OMPARISONS WITH R ELATED W ORK
Note that in graph convolutional network (Kipf & Welling, 2017), the node representation is updated by h ( t +1)
i =
σ (cid:16) (cid:80) j ∈N A i
[ D − 1 / 2 AD − 1 / 2 ] ij h ( t ) j W (cid:17) , where A is the adjacency matrix, D is the degree matrix,
h i is representation of the i th node and W an embedding matrix.
Namely, the message
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Table]
passing is mainly determined by the (normalized) adjacency matrix.
The GAT method (Velickovic et al., 2017) replaces the ﬁxed adjacency matrix with an inductive, trainable attention function that relies instead on the node features within one-hop neighbors.
Our approach has a notable difference.
First, our message passing is determined by a mixed attention from both structure and content (6).
Second, the structural component of our attention is not simply based on the graph adjacency (Kipf & Welling, 2017), or one-hop neighbor (Velickovic et al., 2017), but instead relies on a local receptive ﬁeld whose “shapes” are optimized adaptively through learning (1).
Furthermore, our method fully exploits structural details (e.g. density & topology of local connections).
There are also a number of works that explore structures in graph classiﬁcation problems (Lee et al., 2018; Rossi et al., 2019).
There, attention is used to identify small but discriminative parts of the graph, also called “graphlets” or “motifs” (Morris et al., 2019), in order to perform classiﬁcation on the graph level (such as drug effect of molecules).
As can be seen, their goal is different from ours; another difference is that their node features are typically categorical, while we focus more on homogeneous nodes with rich features (such as bags of words feature for a document).
E XPERIMENTS
In this section, we report experimental results of the proposed method and state-of-the-art algorithms using graph-based benchmark data sets and transductive classiﬁcation problem.
Our codes can be downloaded from the anonymous Github link http://github.com/AvigdorZ .
4. E XPERIMENTAL S ETTING AND R ESULTS
We have reported results of the following baseline algorithms: Gaussian ﬁelds and harmonic function (Gaussian Fields) (Zhu et al., 2003), manifold regularization (Manifold Reg.)
(Belkin et al., 2006); Deep Semi-supervised learning (Deep-Semi) (Weston et al., 2012);
link-based classiﬁcation (Link-based)
Lu & Getoor.
(2003); skip-gram based graph embedding (Deep-Walk) (Perozzi et al., 2014); semi-supervised learning with graph embedding (Planetoid) (Yang et al., 2016); graph convolutional networks (GCN) (Kipf & Welling, 2017); high-order chebyshev ﬁlters with GCN (Chebyshev) (Defferrard et al., 2016), and the mixture model CNN (Mixture-CNN)
(Monti et al., 2016).
We have selected three benchmark graph-structured data set from (Sen et al., 2008), namely Cora,
Citeseer, and Pubmed.
The three data sets are all citation networks.
Here, each node denotes one document, and an edge will connect two nodes if there is citation link between the two document; the raw features of each document are bags-of-words representations.
Each node (document) will be associated with one label, and following the transductive setting in (Velickovic et al., 2017; Yang et al., 2016)
we only use 20 labeled samples for each class but with all the remaining, unlabelled data for training.
We split the data set into three parts: training, validation, and testing, as shown in table 1.
Algorithm performance will be evaluated on the classiﬁcation precision on the test split.
For algorithm using random initialization, averaged performance over 10 runs will be reported.
The network structures of our methods follow the GAT method (Velickovic et al., 2017), with the following details.
Altogether 8 attention head is used, and two layers of message passing are adopted in each attention head, sharing W ∈ R d × d feature transformation matrix across layers followed by a fully connected layer, and cross-entropy loss deﬁned on labeled nodes.
The number of parameters for each attention head will be d 2 + 2 d , where the second term is about the attention function (3); the prediction part involves a fully connected layer with hidden unit d (cid:48)
= 64 , and so the number of parameters is 8 d · d (cid:48)
· C , where d is the dimension of node features, and C is the number of classes.
Adam SGD is used for optimization, with learning rate λ
= 5 × 1 e − 4 .
[Table]
Experimental results are reported in table 2.
For the proposed method, we have 2 variations, including ADSF-Nonparametric, where we learn a non-parametric decay proﬁle w.r.t.
the node distance (up to k -hops); ADSF-RWR, where we use random-walk with re-start to build ﬁngerprints.
As can be seen, our approach consistently improves the performance on all benchmark data sets, and using random walk is slightly better.
The range of the structural ﬁngerprint is set to 3-hop neighbors.
Since Velickovic et al.
(2017) has performed an extensive set of evaluations in their work
[Figure]
we will use their reported results for all the baseline methods.
It is also worthwhile to note that our method only has a few more parameters compared with GAT method (e
[Figure]
g., non-parametric decay proﬁle u , c in RWR
[Figure]
and the mixing ratios in (6)), which is negligible w.r
[Figure]
t. the whole model size.
For example, for the Cora data set, GAT model has about 7 million parameters
[Figure]
while our method adds 24 extra parameters on top of it
[Figure]
In Figure 5, we plot evolution of the testing accuracy and loss function (on the test split) for GAT and our method (ADSF-RWR).
Since the loss functions of the two methods are the same, they are directly comparable.
The accuracy of GAT ﬂuctuates through iterations, while our approach is more stable and converges to a higher accuracy.
The objective value of our method also converges faster thanks to the utilization of higher-order neighborhood information through structural ﬁngerprints.
4. A BLATION S TUDIES
In this section
[Figure]
We further examined some interesting details of the proposed method, all using the Cora data set and the RWR ﬁngerprint construction scheme.
First, we study the impact of the size of the structural ﬁngerprint.
Intuitively, the structural ﬁngerprint should neither be too small or too large.
We have two steps to control the effective size of the ﬁngerprint.
In constructing the ﬁngerprint, we will ﬁrst choose neighboring nodes within a certain range, namely the k -hop neighbors; then we will ﬁne-tune the weight of each node in the ﬁngerprint, either through the nonparametric decay proﬁle or the c parameter in random walk.
Here, for comparison purposes, we have ﬁxed c = 0 .
5 and varied k as k = 1 , 2 , 3 so as to examine their respective accuracy in Figure 5(a) 3 .
As can be seen from Figure 6(a), the optimal order of neighborhood is two in this setting.
Note that in Cora data set, the averaged 1st-order neighborhood size is |N 1 | = 3 .
9 , while for 2nd-order neighbors |N 2 | = 42 . 5 .
In other words, if one only considers direct neighbors, a large portion of useful higher-order neighbors might be ignored, namely, systematic exploration of higher-order graph structures can make a signiﬁcant difference.
[Figure]
In Figure6(b) [Figure] we plot the learning performance w.r.t.
the choice of the c parameter in RWR when ﬁxing the neighborhood range k
= 2 .
As can be seen, the best performance is around c = 0 .
5 , meaning that the “random walking” and “restarting” should be given similar chances within 2-hop neighbors.
Empirically, we can always use back-propagation to optimize the choice of c , which can be more adaptive to the choice of the k -hop neighbors.
In Figure 6(c), we plot the non-parametric decay proﬁle u learned by our method, when setting the neighborhood orders to k = 3 .
As can be seen, the ﬁrst-order and second-order neighbors have higher weights, while the third-order neighbors almost have zero weights, meaning that they almost make no contributions to computing the structural attention.
This is consistent to our evaluations in
Figure 5(a), and demonstrates the power of our method in identifying useful high-order neighbors.
C ONCLUSION AND F UTURE W ORK
In this work, we proposed an adaptive structural ﬁngerprint model to encode complex topological and structural information of the graph to improve learning hidden representations of the nodes through attention.
There are a number of interesting future directions.
First, we will consider varying ﬁngerprint parameters (such as decay proﬁle) instead of sharing them across all the nodes; second, we will also consider applying the structural ﬁngerprints in problems of graph partitioning and community detection, where node features might be unavailable and graph structures will be the main information for decisions; third, we will extend our approach to challenging problems of graph-level classiﬁcation where node-types shall be taken into account in constructing structural ﬁngerprint; ﬁnally, on the theoretical side, we will borrow existing tools in semi-supervised learning and study the generalization performance of our approach on semi-supervised node embedding and classiﬁcation.
We highly appreciate the valuable comments from anonymous reviewers, which allow us to signiﬁcantly improve both the presentation and experimental design of our work S UPPLEMENTARY E XPERIMENTS Hop k =k =k =
Cora 83.0 ± 0.
15.2 ± 6.
13.3 ± 4.
Citeseer 72.5 ± 0.
68.4 ± 4.
65.7 ± 4.
Pubmed 79.0 ± 0.
78.7 ± 0.
67.2 ± 0.
As suggested by the anonymous reviewer, we perform an empirical study on the GAT performance when larger neighborhood is considered.
In the original GAT by Velickovic et al.
(2017), attention score is computed only for nodes that are direct neighbors with each other.
Here, we increase the domain of interaction to up to 2-hop and 3-hop neighbors, with all other components remaining the same.
As can be seen from Figure 7, the performance of GAT is the best when choosing only 1-hop neighbors to compute attention scores.
This is because although larger neighborhood provides more information, it may bring a signiﬁcant proportion of noisy vertices.
As a result, differentiating irrelevant neighbors against useful ones based on structural clues is particularly important in structural attention mechansim.