G RAPH J OINT A TTENTION N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph attention networks (GATs) have been recognized as powerful tools for learning in graph structured data.
However, how to enable the attention mechanisms in GATs to smoothly consider both structural and feature information is still very challenging.
In this paper, we propose Graph Joint Attention Networks (JATs) to address the aforementioned challenge.
Different from previous attention-based graph neural networks (GNNs), JATs adopt novel joint attention mechanisms which can automatically determine the relative signiﬁcance between node features and structural coefﬁcients learned from graph subspace, when computing the attention scores.
Therefore, representations concerning more structural properties can be inferred by JATs.
Besides, we theoretically analyze the expressive power of JATs and further propose an improved strategy for the joint attention mechanisms that enables JATs to reach the upper bound of expressive power which every message-passing GNN can ultimately achieve, i.e., 1-WL test.
JATs can thereby be seen as most powerful message-passing GNNs.
The proposed neural architecture has been extensively tested on widely used benchmarking datasets, including Cora, Cite, and Pubmed and has been compared with state-of-the-art
GNNs for node classiﬁcation tasks.
Experimental results show that JATs achieve state-of-the-art performance on all the testing datasets.
I NTRODUCTION
Many real-world data can be modeled as a graph, where a set of nodes (vertices), edges, and bag-ofwords features respectively represent data instances, instance-instance interrelationships, and contents characterizing the nodes.
For example, scientiﬁc articles in a research domain can be modeled as a graph, where nodes, edges, and node features respectively represent published articles, citations, and index information of the articles.
Besides, social network users and interacted biological units can also be similarly represented as graphs possessing different structural and descriptive information.
As graph data are widely available and they are related to various analytical tasks, learning in graphs has been a hot-spot in machine learning community.
There have been a number of approaches proposed to effectively learn in graph structured data.
Amongst them, graph convolutional networks (GCNs) have shown to be powerful in learning lowdimensional representations for various subsequent analytical tasks.
Different from those empirical convolutional neural networks (CNNs) which have achieved a great success in learning in image, vision, and natural language data (Krizhevsky et al., 2012; Xu et al., 2014), and whose convolution operators are always deﬁned to process a grid-like data structure, GCNs attempt to formulate convolution operators aggregating the node features according to the observed graph structure, and learn the information propagation through different neural architectures.
Meaningful representations which capture discriminative node features as well as intricate graph structure can thereby be learned by GCNs.
There have been several sophisticated GCNs proposed in the recent.
According to the ways through which GCNs make use of graph topology to deﬁne convolution operators for feature aggregation, GCNs can generally be categorized as spectral, and spatial ones (Wu et al., 2020).
Spectral GCNs deﬁne the convolutional layer for aggregating neighbor features based on the spectral representation of the graph.
For example, Spectral CNN (Bruna et al., 2013) constructs the convolution layer based on the eigen-decomposition of graph Laplacian in the Fourier domain.
However, such layer is computationally demanding.
Aiming to reduce such computational burden, several approaches adopting the convolution operators which are based on simpliﬁed/approximate spectral graph theory are proposed.
First, parameterized ﬁlters with smooth coefﬁcients are introduced for
Spectral CNN to allow it to consider spatially localized nodes in the graph (Henaff et al., 2015).
Chebyshev expansion is then introduced in (Defferrard et al., 2016) to approximate graph Laplacian rather than directly perform eigen-decomposition of it.
Finally, the graph convolution ﬁlter is further simpliﬁed by only considering connected neighbors of each node (Kipf & Welling, 2017)
so as to further make spectral GCNs computationally efﬁcient.
In contrast, spatial GCNs deﬁne the convolution operators for feature aggregation directly making use of local structural properties of the central node.
The key of spatial GCNs is consequently how to design an appropriate function for aggregating the effect brought by the features of candidate neighbors selected according to a proper sampling strategy.
To achieve this, it sometimes requires to learn a weight matrix according to node degree (Duvenaud et al., 2015), utilize the power of transition matrix to preserve the neighbor importance (Atwood & Towsley, 2016), extract the normalized neighbors (Niepert et al., 2016), or sample a ﬁxed number of neighbors (Hamilton et al., 2017; Zhang et al., 2020).
As a representative spatial GCN, Graph attention network (GAT) (Veliˇckovi´c et al., 2018; Zhang et al., 2020) has shown a promising performance in various graph learning tasks.
What makes GATs effective in learning graph representations is they adopt the attention mechanism, which has been successfully used in machine reading and translation (Cheng et al., 2016; Luong et al., 2015), and video processing (Xu et al., 2015), to compute the node-feature-based attention weights (attention scores) between a central node and its one-hop neighbors (including the central node itself).
Then,
GATs utilize the attention scores to obtain a weighted aggregation of node features which are propagated to the next layer.
As a result, those neighbors possessing similar features may impact more on the center node, and meaningful representations can be inferred by GATs.
Although GATs have been experimentally veriﬁed as powerful tools for various learning tasks in graph data, they still confront several challenges.
First, there are no appropriate attention mechanisms which can automatically identify the relative signiﬁcance between the latent structure and node features while computing the attention scores.
As a result, current attention mechanisms for
GATs cannot effectively capture the joint effect brought by the underlying graph structure and node features for seamlessly impacting the message-passing in the neural architecture.
Second, whether the expressive power of GNNs adopting the attention mechanisms which can effectively acquire the aforementioned joint effect may reach the upper bound of message-passing GNNs has not been theoretically investigated.
To address the mentioned challenges, in this paper, we propose novel attention-based GNNs, dubbed Graph Joint Attention Networks (JATs).
Different from previous works, the attention mechanisms adopted by JATs are able to automatically capture the relative signiﬁcance between structural coefﬁcient learned from graph subspace, and node features, so that higher attention scores may be learned by those neighbors which are topologically and contextually correlated.
JATs are consequently able to smoothly adjust attention scores according to the contemporary structure and node features, and truly capture the joint attention on structural and contextual information propagated in the neural network.
Besides, we theoretically analyze the expressive power of JATs and further propose an improved strategy which enables JATs to distinguish all distinct graph structures as 1-dimensional Weisfeiler-Lehman test (1-WL test) does.
This means JATs can reach the upper bound w.r.t.
expressive power which all message-passing GNNs can ultimately achieve.
JATs have been extensively tested on three widely used datasets, i.e., Cora, Citeseer, and
Pubmed, and have been compared with a number of strong baselines.
The experimental results show that JATs achieve the state-of-the-art performance.
The rest of the paper is organized as follows.
In Section 2, we elaborate the proposed JATs, and compare JATs with other GNNs.
In Section 3, we prove the limitation w.r.t.
expressive power of the joint attention mechanisms presented in Section 2.
A strategy is then proposed to improve JATs to reach the upper bound of expressive power which all message-passing GNNs can at most achieve.
The comprehensive experiments which are used to validate the effectiveness of JATs are presented in
Section 4.
Finally, we summarize the contributions of the paper and propose future works potentially improving JATs.
J OINT ATTENTION - BASED GRAPH NEURAL NETWORKS
In this section, we elaborate the proposed JATs.
Mathematical preliminaries and notations used in the paper are ﬁrstly illustrated.
How JATs learns the structural coefﬁcients which are used in the joint attention mechanisms is then introduced.
Following that, the joint attention layer, which is the cornerstone of JATs is elaborated.
At last, we compare the proposed JATs with their counterparts.
2. N OTATIONS AND PRELIMINARIES
Throughout this paper, we assume a graph G = { V, E } containing N nodes, | E | edges, where V and E respectively represent the node and edge set.
We use A ∈ { 0 , 1 } N × N and X ∈ R N × D to represent graph adjacency matrix and node feature matrix, respectively.
N i denotes the union of node i and its one-hop neighbors.
W l and { h li } i =1 ,...
N denote the weight matrix and features of node i at l th layer of JATs, respectively, and h 0 is set to be the input feature, i.e., X .
For the nodes in N i , their possible feature vectors form a multiset M i =
( S i , µ i ) , where S i = { s 1 , ...
s n } is the ground set of M i which contains the distinct elements existing in M i , and µ i : S i → N (cid:63) is the multiplicity function indicating the frequency of occurrence of each distinct s in M i .
2. L EARNING STRUCTURAL COEFFICIENTS FROM GRAPH SUBSPACE
It is well known that topology is the corner stone of the graph.
How to utilize such structural information to compute the attention scores naturally has a profound impact on the performance of attention-based GNNs.
Empirical attention based GNNs, e.g., GAT, compute the attention scores between connected neighbors only using node features, but overlook the structural correlation between pairwise nodes.
To allow attention-based GNNs to capture the higher-order structures in the graph, we propose JATs to learn the topological coefﬁcients from the graph subspace.
Inspired by subspace clustering (Elhamifar & Vidal, 2013), we may formulate the learning of structural coefﬁcients as follows.
Given N nodes in the graph drawn from multiple linear subspaces {S i } i =1 ,...
K , one can represent a node in a subpace as a linear combination of other nodes.
If each row in A is treated as the structural information of each node, one can simply represent it using other nodes (other rows in A ) as one equation, i.e., A i, : =
C i, : A , where C denotes the structural coefﬁcient matrix as A is used.
It has been shown in previous works (Ji et al., 2014) that under the assumption the subsapces are independent, by minimizing certain norm of C , C may have a block diagonal structure (after ﬁnite permutations).
In other words, C ij (cid:54) =
if and only if two nodes, v i and v j are in the same subspace.
So, we can utilize C to learn the structural correlations between neighbors in the graph.
And the above learning task can be formulated as the following optimization problem:
[Equation]
where (cid:107)·(cid:107) p stands for a certain matrix norm, and the zero constraint on the diagonal of C may prevent trivial solutions when (cid:107)·(cid:107) p is the norm considering sparsity.
To make the data corruption explainable, the equality constraint in Eq.
(1) is often relaxed as a regularization term and the learning of structural coefﬁcients can be reformulated as follows: [Equation] By rewriting Eq.
(2), we may reveal why subspace learning is effective in capturing the structural correlations between pairwise nodes.
Mathematically, Eq.
(2) is equivalent to the following nodewise minimization problem:
[Equation]
As CA i, : is equal to (cid:80) j C ij A j, : and C ii = 0 , one may easily ﬁnd that minimizing Eq.
(3) is equivalent to search for the optimal linear combination of other nodes that can be used to reconstruct the i th node in the graph.
As a result, C ij is high when A j, : is structurally similar to A i, : .
By minimizing Eq.
(2), one may identify those nodes which are in the same graph subspace, and the structural correlations between neighboring nodes can therefore be inferred.
As the above learning problem can be effectively solved via gradient descent, JATs can optimize Eq.
(2) together with the training of the neural architecture.
Also, we use l 1 norm for C to force JATs to learn sparse structural coefﬁcients.
2. J OINT ATTENTION LAYER
Having made the structural coefﬁcients available, we are now presenting joint attention layer, which is the core module for building JATs and will be used in our experiments.
Different from the attention layers utilized by other GNNs, the joint attention layer in the proposed framework adopts novel attention mechanisms, which may automatically identify the relative signiﬁcance between input features and structural information.
Appropriate attention scores between central node and its neighbors can be obtained and meaningful representations can be inferred by JATs.
Given a set of node features { h li } i =1 ,...N , each h li ∈ R F l , the joint attention layer of JATs is to map them into F l +1 dimensional space { h l +1 i } i =1 ,...
N , according to the correlations of input features and graph topology.
JAT ﬁrst attempts to compute the correlation between two connected nodes, say v i and v j .
To do so, we directly adopt the feature-based attention mechanism utilized by previous
GATs (Veli ˇ ckovi ´ c et al., 2018): [Equation]
[Equation]
) where ˜a ∈ R 2 F l +1 is a vector of parameters of the feedforward layer, (cid:107) stands for the concatenation function, and W l is F l +1 × F l variable matrix for feature mapping.
Based on Eq.
(4), JAT may capture the feature correlations between connected nodes (ﬁrst-order neighbors) by computing the similarities w.r.t.
features mapped to next layer.
However, determining the attention scores solely based on node features leads a GNN to overlook the structural information hidden in the graph.
To overcome this issue, we propose JAT to learn structural coefﬁcients as mentioned in Section 2.2 and utilize the novel joint attention mechanisms to appropriately compute the attention scores.
Given learnable structural coefﬁcient, C ij between two nodes, JAT obtains the structural correlation as follows:
[Equation]
Given f ij and s ij , JAT has two strategies (attention mechanisms) to compute the ﬁnal attention scores.
The ﬁrst mechanism is dubbed Implicit direction .
It aims at computing the attention scores whose relative signiﬁcance between structural and feature correlations can be automatically acquired.
To do so, JAT introduces two learnable parameters ( g f and g s ) used to determine the relative signiﬁcance and such signiﬁcance can be obtained as follows:
[Equation]
where r i represents the normalized signiﬁcance related to structure or node features.
Given r i , JAT is able to compute the attention score based on Implicit direction strategy:
[Equation]
Given the attention mechanism shown in Eq.
(7), α ij attempts to capture the weighted mean attention in terms of structural and feature correlations between neighbors.
Compared with the attention mechanism solely based on features of one-hop neighbors, α ij computed by Eq.
(7) may be softly adjusted according to the implicit impact brought by the structural coefﬁcients.
Moreover, the relative signiﬁcance r i can also be automatically inferred by JAT as it is involved into the back propagation process.
More smooth and appropriate attention scores can thereby be computed by JAT for learning meaningful representations.
To enhance the structural impact, JAT has another strategy, named as Explicit direction , to compute attention scores between neighbors.
Given f ij and s ij , the attention scores obtained via Explicit direction strategy is deﬁned as follows:
[Equation]
Compared with Eq.
(7), the structural coefﬁcient explicitly inﬂuences the magnitude of f ij , so that those node pairs which are structurally irrelevant are impossible to assign with high attention weights.
Utilizing Explicit direction strategy, JAT becomes more structure dependent when performing message passing in its neural architecture.
Having obtained the attention scores, JAT is now able to compute a linear combination of features corresponding to each node and its neighbors as the layer-wise output, which will be either propagated to the higher layer, or be used as the ﬁnal representations for subsequent learning tasks.
The mentioned output features can be computed as follows:
[Equation]
In Fig.
1, the joint attention mechanisms proposed in this paper are graphically illustrated.
And we may use a particular number of joint attention layers using the proposed attention mechanisms to construct JATs.
In practice, we also adopt the multi-head attention strategy (Vaswani et al., 2017) to stabilize the learning process.
JATs either concatenate the node features from multiple hidden layers as the input for next layer, or compute the average of node features obtained from multiple units of output layers as the ﬁnal node representations.
The details on how to implement multi-head attention in graph neural networks can be checked in (Veli ˇ ckovi ´ c et al., 2018).
2. C OMPARISON TO RELATED GRAPH NEURAL NETWORKS
Based on the structure of the introduced joint attention layer, it is found that Graph Joint Attention
Networks are quite different from previous related neural architectures and have several advantages.
When compared with spectral based GNNs, the proposed JATs provide a dynamic way to update layer-wise representations.
In each layer, the updating function for node representation in prevalent spectral GNNs, e.g., (Kipf & Welling, 2017) can generally be formulated as h (cid:48)
i =
σ ( (cid:80) j ∈N i
[ D − 12 AD − 12 ] ij h j W ) , where A and D are adjacency matrix and degree matrix, respectively, W is the learnable weight matrix for mapping the node features to the next layer, and h j represents the feature vector of j th node.
As it shows, the message passing in spectral GNNs is deﬁned according to the normalized adjacency matrix and it is static when spectral GNNs performs back propagation.
In contrast, JATs compute the attention score ( α ij ) for each pair of neighbors based on structural coefﬁcient and node features, which naturally leads each layer in the neural network to pay more attention to those node pairs sharing similar local structures and features.
Higher-order representations concerning such analogous neighbors in the graph can therefore be learned by JATs.
When compared with spatial GNNs, especially those attention based ones (Zhang et al., 2020;
Veliˇckovi´c et al., 2018), JATs adopt adaptive attention mechanisms which smoothly determine the relative signiﬁcance between graph structure and node features, to compute more appropriate attention scores for representation learning.
Compared with GATs (Veliˇckovi´c et al., 2018), JATs are able to compute attention scores for ﬁrst order neighbors according to both structural coefﬁcients and node features, so that the attention-based message passing considers more structural information.
JATs are also different from those spatial GNNs which update node representations via sampling a ﬁxed size of neighbors.
For example, to reduce the computational demand, GraphSAGE (Hamilton et al., 2017) performs the task of graph representation learning via max pooling.
Another attention based framework, ADSF (Zhang et al., 2020) also takes into consideration structural information when computing attention scores.
However, ADSF cannot infer the relative signiﬁcance between utilized topological information and node features, and ADSF considers the best k neighbors for computing attention scores.
Unlike JATs, such spatial GNNs sampling neighbors for representation updating cannot fully utilize the structural information provided by proximal nodes in the graph.
Due to the ﬂexibility of the proposed attention mechanisms, JATs can be readily combined with any paradigm which can learn the topological correlations, just like the structural coefﬁcients used in this paper.
This property enhances the applicability of the proposed JATs.
M ORE EXPRESSIVELY POWERFUL JAT S
Study on the expressive power of various GNNs has drawn much attention in the recent.
It concerns whether a given GNN can distinguish different structures where vertices possessing various vectorized features.
It has been found that what the neighborhood aggregation functions, or readout operators of all message-passing GNNs, including GCNs, GATs, and other related aim at are analogous to what 1-dimensional Weisfeiler-Lehman test (1-WL test), which is injective and iteratively operated in Weisfeiler-Lehman algorithm
(Weisfeiler & Leman, 1968; Xu et al., 2018; Zhang & Xie, 2020), does.
In other words, both aggregation/readout functions in message-passing GNNs and 1-WL test attempt to distinguish structures which are different in some ways.
As a result, all message-passing
GNNs are as most powerful as 1-WL test (Xu et al., 2018).
The theoretical validation of the expressive power of a given GNN thereby lies in whether those adopted aggreation/readout functions are homogeneous to 1-WL test.
Having the effective framework evaluating the expressive power of a given message-passing GNN, one may naturally be interested in whether the proposed JATs can distinguish all different graph structures as 1-WL test does.
As we mainly consider node classiﬁcation tasks, in this section, we investigate the expressive power of the neighborhood aggregation function concerning the joint attention mechanisms in Eqs.
(7) and (8).
We ﬁrstly show that the neighborhood aggregation function utilized by JATs still fails to discriminate some graph structures possessing certain topological properties.
Then, we propose a simple but effective strategy for the joint attention mechanisms to enable
JATs to successfully distinguish all those graph structures that previously cannot be discriminated.
For the neighborhood aggregation utilizing the attention mechanism shown in Eq.
(7), we have the following theorem pointing out the conditions under which the aggregation function fails to distinguish different structures.
Theorem 1 Assume the feature space X is countable and the aggregation function concerns the attention mechanism in Eq.
(7) is represented as h ( c, X ) =
(cid:80) x ∈
X α cx g ( x ) , where c is the feature of center node, X ∈ X is a multiset containing the feature vectors from nodes in N i , g ( · ) is a function for mapping input feature X , and α cx is the attention score between f ( c ) and f ( x ) .
For all g and the joint attention mechanism in Eq.
(7), h ( c 1 , X 1 )
=
h ( c 2 , X 2 ) if and only if c 1 = c 2 , X 1 = { S, µ 1 } ,
X 2 = { S, µ 2 } , and (cid:80) y
= x,y ∈
X 1 f c 1 y − (cid:80) y
= x,y ∈
X 2 f c 2 y =
q
[ (cid:80) y
= x,y ∈
X 2 s c 2 y − (cid:80) y
= x,y ∈
X 1 s c 1 y ] , for q = r s r f and x ∈ S .
In other words, h will map different multiset into the same embedding iff the multisets have same central node feature, same underlying set, and the difference in feature-based attention scores is proportional ( r s r f ) to the opposite of that in attention weights corresponding to structural coefﬁcients.
We leave the proof of all the theorems and corollaries in the appendix.
For the aggregation function utilizing the attention mechanism shown in Eq.
(8), we have the following theorem indicating the structures which cannot be correctly distinguished.
Theorem 2 Under the same assumptions shown in Theorem 1, for all g and the joint attention mechanism in Eq.
(8), h ( c 1 , X 1 ) =
h ( c 2 , X 2 ) if and only if c 1 = c 2 , X 1 = { S, µ 1 } , X 2 = { S, µ 2 } , and q · (cid:80) y
= x,y ∈
X 1 φ ( C c 1 x ) =
(cid:80) y
= x,y ∈
X 2 φ ( C c 2 y ) , for q > 0 and x ∈ S , where φ ( · ) is an function for mapping values to R + .
In other words, h will map different multiset into the same embedding iff the multisets have same central node feature, same node features whose corresponding mapped structural coefﬁcients are proportional.
Theorems 1 and 2 indicate that the joint attention mechanisms may still fail to distinguish some graph structures, although the aggregation functions adopting the joint attention mechanisms may be more expressively powerful than empirical GATs.
As node features and graph structure are heterogeneous, intuitively, sub-structures satisfying the mentioned conditions should be infrequent.
This may well explain why those attention-based GNNs concerning including structural factors into the computation of attention scores may experimentally perform better than GATs.
However, when distinct multisets with corresponding structural properties meet the conditions mentioned in
Theorems 1 and 2, the joint attention mechanisms in Eqs.
(7) and (8) cannot correctly distinguish such multisets.
Thus, JATs fail to reach the upper bound of expressive power of all message-passing
GNNs, i.e., 1-WL test.
However, we are able to readily improve the expressive power of JATs to meet the condition of 1WL test by slightly modifying the joint attention mechanisms.
The modiﬁed attention mechanisms are deﬁned as follows:
[Equation]
where α ij is the attention weight obtained by either Eq.
(7) or (8).
Then, the newly obtained attention scores can be used to aggregate the node features passed to the higher layers.
Next, we prove that such improved attention mechanisms reach the upper bound of message-passing GNNs via showing they can distinguish those structures possessing the properties mentioned in Theorems 1 and 2.
Corollary 1 Let T be the attention-based aggregator shown in Eq.
(9) that considers the attention mechanism in Eq.
(7) or (8) and operates on a multiset H ∈ H , where H is a node feature space mapped from the countable input feature space X .
A H exists so that utilizing α ij in Eq.
(10), T can distinguish all different multisets in aggregation that it previously cannot discriminate.
Based on Eq.
(10), one may use original JATs to perform different classiﬁcation tasks in graph data by setting (cid:15)
= 0 .
The expressive power of JATs can immediately reach the upper bound of message-passing GNNs when (cid:15) is set as a positive value.
Theoretically, the expressive power of JATs is stronger than state-of-the-art attention-based GNNs, e.g., GATs (Veli ˇ ckovi ´ c et al., 2018), and ADSF (Zhang et al., 2020).
E XPERIMENTS AND ANALYSIS
In this section, we evaluate the proposed Graph Joint Attention Networks (JATs) against a variety of prevalent baselines, on widely used network datasets.
4. E XPERIMENTAL SET - UP
To validate the effectiveness of JATs, we compare them with a number of state-of-the-art baselines, including Gaussian ﬁelds and harmonic functions (GF) (Zhu et al., 2003), Manifold regularization (Mani-reg) (Belkin et al., 2006), Deepwalk (Perozzi et al., 2014), Semi-supervised graph embedding (Planetoid) (Yang et al., 2016), Node2Vec (Grover & Leskovec, 2016), Graph convolutional networks (GCN) (Kipf & Welling, 2017), GCN with Chebyshev ﬁlters (Chebyshev) (Defferrard et al., 2016), GraphSAGE (Hamilton et al., 2017), mixture model CNN (MoNet) (Monti et al., 2017),
Graph attention networks (GAT) (Veliˇckovi´c et al., 2018), Bayesian GCN (BGCN) (Hasanzadeh et al., 2020), and Adaptive structural ﬁngerprints (ADSF) (Zhang et al., 2020).
Based on the experimental results reported in the previous works, these baselines may represent the most advanced techniques for learning in graph structured data.
Four widely-used document networks, which are Cora, Citeseer, Pubmed (Sen et al., 2008; Lu & Getoor, 2003), and OGBN-Arxiv (Hu et al., 2020), are used in our experiments. Cora, Citeseer, and Pubmed are three classical network datasets for validating the effectiveness of GNNs.
However, it is recently found that these three dataset sometimes may not effectively validate the predictive power of different graph learning approaches, due to the relatively small data size and data leakage (Hu et al., 2020; Shchur et al., 2018).
Thus, more massive datasets having better data quality have been proposed to evaluate the performance of different approaches (Hu et al., 2020; Dwivedi et al., 2020).
To effectively reveal the predictive power of JATs, we additionally use OGBN-Arxiv, which is available in the Open graph benchmark database, as one of the testing datasets.
The effectiveness of all methods is validated via allowing them to perform semi-supervised node classiﬁcation (transductive learning) in all the benchmarking sets and the classiﬁed nodes are evaluated using Accuracy [Table] [Table] ( Micro − F 1 ).
To compare JATs impartially with other baselines, we closely follow the experimental paradigms used in the related works (Hu et al., 2020; Veli ˇ ckovi ´ c et al., 2018; Kipf & Welling, 2017;
Yang et al., 2016).
We leave the details of testing datasets and experimental scenarios in appendix due to space limitation.
4. R ESULTS ON NODE CLASSIFICATION
We obtain the average classiﬁcation accuracy over 10 runs of JATs and compare it with other stateof-the-art approaches.
The corresponding results are summarized in Table 1 and 2.
As the tables show, JATs utilizing two different joint attention mechanisms outperform all the baselines on all the four testing datasets.
Speciﬁcally, the average Accuracy on node classiﬁcation obtained by JAT-I (JAT using Implicit direction strategy) achieves 85.8%, 74.3%, 82.8% and 72.9% on Cora, Citeseer, Pubmed, and OGBN-Arxiv, respectively.
The presented experimental results demonstrate that the proposed method is one of the most effective GNNs for learning in graph structured data.
4. A BLATION STUDY
Besides testing the performance of transductive learning in network data, we further investigate how different settings of JATs may impact their performance.
Speciﬁcally, we ﬁrst investigate how JATs perform when different values of (cid:15) are used.
The sensitivity test on (cid:15) =
[10 − 8 , 10 − 6 , 10 − 4 , 10 − 2 , 10 − 1 , 1 , 5 , 10] is plotted in Fig.
2 (a) and (b).
As the ﬁgures depict, both two versions of JATs perform robustly when (cid:15) ≤ 1 and the improved strategy shown in Eq.
(10) may boost the performance of JATs in some datasets, e.g., Pubmed.
Thus, we recommend to set (cid:15) ≤ 1 for both two versions of JATs.
Such setting may ensure JATs leverage the joint attention mechanisms as well as preserve the expressive power, when classifying nodes in graph structured data.
Then, to further investigate whether the strategy of automatically determining the relative signiﬁcance between feature and structural attention (Eqs.
(6) and (7)) may truly improve the performance on transductive learning, we compare JAT-I with GAT, ADSF, and JAT utilizing the attention strategy shown in Eq.
(7) but setting r f = r s = 0 .
5 (JAT-I w/o ad).
As Fig.
2 (c) shows, JAT-
I can Different versions of JATs [Table] outperform all the baselines on all the testing datasets as it is able to automatically determine which attention, i.e., structure or node features is more important when computing the ﬁnal attention scores for aggregating features for the central node.
The presented results indicate the effectiveness of the strategy of automatic determination adopted by JATs. Finally
[Table]
we further test the effectiveness of the proposed joint attention mechanisms by replacing the structural coefﬁcients ( C ) with other effective strategies.
Speciﬁcally, we use Cosine and Jaccard similarity measures to compute the structural correlations for connected node pairs and then let
JAT utilize them to compute attentions scores for the network training.
Besides, we also allow one version of the JATs (JAT-SSL) to only use the structural coefﬁcients learned via minimizing Eq.
(2) to compute the attention scores in the training stage.
The corresponding results are summarized in
Table 3.
As the table shows, JAT still performs robustly when using simple measures, e.g., Cosine and Jaccard similarity to obtain node-node structural correlations, although its performance is not better than original versions of JAT.
Given the results shown in Table 3, it is found that the proposed joint attention mechanisms are effective in learning node representations.
The JAT framework can therefore be readily combined with many effective approaches which can infer node-node structural correlations, such as network embedding (Grover & Leskovec, 2016) and deep structural learning (Monti et al., 2017) for effective representation learning.
[Figure]
C ONCLUSION
In this paper, we propose novel attention-based GNNs, dubbed Graph Joint Attention Networks (JATs).
Different from previous related approaches, JATs adopt novel joint attention mechanisms that can smoothly infer the relative signiﬁcance between graph structure and node features, so that structural properties and node features are appropriately preserved in node representations.
Besides, the expressive power of JATs is theoretically analyzed and the improved strategy to ensure JATs to be most powerful message-passing GNNs is also proposed.
In future, we will further improve the effectiveness of JATs by considering different structural properties hidden in the network data and explore JATs’ applicability by using them in multi-view and heterogeneous network data.
A P ROOF OF T HEOREM
To prove Theorem 1, we need to consider the two directions of the iff conditions (Zhang & Xie, 2020).
If given c 1 = c 2 , S 1 = S 2 , and (cid:80) y
= x,y ∈
X 1 f c 1 y − (cid:80) y
= x,y ∈
X 2 f c 2 y =
q
[ (cid:80) y
= x,y ∈
X 2 s c 2 y − (cid:80) y
= x,y ∈
X 1 s c 1 y ] , for q = r s r f , for the aggregation function utilizing the joint attention mechanism shown in Eq.
(7), we have:
[Equation]
where m c i x represents the feature similarity between c i and x .
Given Eq.
(11), we may directly derive h ( c 1 , X 1 ) and h ( c 2 , X 2 ) :
[Equation]
Considering c 1 = c 2 , S 1 = S 2 , and (cid:80) y
= x,y ∈
X 1 f c 1 y − (cid:80) y
=
[Equation]
y = x,y ∈
X 1 s c 1 y ] , for q =
r s r f , we directly derive h ( c 1 , X 1 )
=
h ( c 2 , X 2 ) .
If we are given h ( c 1 , X 1 ) =
h ( c 2 , X 2 ) , we are able to prove that the conditions mentioned in the theorem are necessary by showing contradictions occur when they are not satisﬁed.
If h ( c 1 , X 1 ) =
h ( c 2 , X 2 ) , we have:
[Equation]
Firstly, assuming S 1 (cid:54) =
S 2 , for any g ( · ) , we thereby have:
[Equation]
As Eq.
(14) holds for any g ( · ) , we may deﬁne another function g (cid:48) ( · ) as follows: g ( x ) = g (cid:48)
( x ) , for x ∈ S 1 ∩ Sg ( x ) =
g (cid:48) ( x ) − 1 , for x ∈ S 1 \ Sg ( x ) =
g (cid:48)
( x ) + 1 , for x ∈ S 2 \ S(15)
If Eq.
(14) holds, we also have:
[Equation]
As Eq.
(14) equals Eq.
(16), we have:
[Equation]
Obviously, the above equation does not hold as the terms in the summation operator are positive
[Equation]
Thus, S 1 (cid:54) =
S 2 is not true.
We may now assume S 1 = S 2 = S .
Eliminating the irrational terms in
Eq.
(14), we have:
[Equation]
Thus we know each term in the summation equals zero:
[Equation]
As structural coefﬁcients and node feature similarity are heterogeneous
[Equation]
we may assume (cid:80) y
[Equation]
y .
Eq.
(19) can be simpliﬁed and rewritten as:
[Equation]
It is obvious that LHS of Eq.
(20) is a rational number.
However, the RHS of Eq.
(20) can be an irrational number.
We may consider S =
{ s, s 0 } and assume c
= s 0 , c
2 = s .
We may also assume the feature similarity between central node and others as follows:
[Equation]
Consider x = s , we have:
[Equation]
where n stands for the number of s 0 in X 2 .
It is obvious that the above equality does not hold as the
RHS is an irrational number, while LHS is a rational number.
Thus c 1 (cid:54) =
c 2 is false.
Given c 1 = c 2 ,
Eq.
(19) can be rewritten as:
[Equation]
To ensure above equation holds, we have:
[Equation] [Equation]
Further denoting r s r f = q , we have (cid:80)
[Equation]
B P ROOF OF T HEOREM
To prove Theorem 2, we can follow the procedure which is used to prove Theorem 1.
If given c 1 = c 2 , S 1 = S 2 , and q · (cid:80) y
= x,y ∈
X 1 φ ( C c 1 y ) =
(cid:80) y
= x,y ∈
X 1 φ ( C c 2 y ) , for q > 0 , we may directly replace φ ( · ) using exp {·} .
For the aggregation function using the joint attention mechanism in Eq.
(8) we have:
[Equation]
where m c i x represents the feature similarity between c i and x .
Given Eq.
(25), we may directly derive h ( c 1 , X 1 ) and h ( c 2 , X 2 ) : [Equation] [Equation] (cid:88)
[Equation]
Given S 1 = S 2 and c 1 = c 2 , h ( c 2 , X 2 ) can be rewritten as: [Equation]
[Equation]
Given q · (cid:80) x
[Equation]
X 1 exp
[Equation] [Equation] [Equation]
y
[Equation]
x exp { C c 2 y } , Eq.
[Equation]
27) is equivalent to:
[Equation] [Equation]
Considering c 1 = c 2 , we have h ( c 1 , X 1 )
=
h ( c 2 , X 2 ) .
If we are given h ( c 1 , X 1 ) =
h ( c 2 , X 2 ) , we are able to prove the conditions mentioned in the theorem are necessary by showing contradictions occur when they are not satisﬁed.
If h ( c 1 , X 1 ) =
h ( c 2 , X 2 ) , we have:
[Equation]
Firstly, assuming S 1 (cid:54) =
S
[Equation]
for any g ( · )
[Equation]
we thereby have:
[Equation]
As Eq.
(30) holds for any g ( · ) , we may deﬁne another function g (cid:48) ( · ) as shown in Eq.
If Eq.
(30) holds, we also have:
[Equation]
As Eq.
(30) equals Eq
[Equation] [Equation]
we have:
[Equation]
Obviously
[Equation]
the above equation does not hold as softmax function is positive.
Thus, S 1 (cid:54) =
S 2 is not true.
We may now assume S 1 = S 2 = S .
Eliminating the irrational terms in Eq.
[Equation]
we have: (cid:88)
[Equation]
Thus we know each term in the summation equals zero:
[Equation]
Eq.
(34) is equivalent to:
[Equation]
We may consider S = { s, s 0 } and assume c
= s 0 , c
2 = s .
We may also assume the feature similarity between central node and others as follows:
[Equation]
Consider x = s , we have:
[Equation]
As the learning of C is independent of feature mapping, and the computation of attention coefﬁcients, exp ( C cx ) can be any positive value.
By setting the exponential values in the above equation as a , which is a positive value.
We have µ 1 ( s ) µ 2 ( s ) =
| X 1 | | X 2 |− n + ne .
Similar with Eq.
(22), c 1 (cid:54) =
c 2 is not true.
Since c 1 = c 2 = c , Eq.
(35) can be rewritten as:
[Equation]
By setting const as 1 q and exp ( C cy )
[Equation]
C cy ) , we have q (cid:80) y
= x,y
[Equation]
y = x,y ∈ X
[Equation]
C
[Equation]
[Equation]
C P ROOF OF C OROLLARY
For any two distinct multisets X 1 and X 2 that cannot be distinguished by T , according to Theorem 1 , we denote X 1 = ( S, µ 1 ) , X 2 = ( S, µ 2 ) , c ∈ S .
We also assume (cid:80) [Equation]
X
[Equation]
y ]
[Equation]
for q = r s r f .
When using the attention mechanism in Eq.
(7) for the aggregation function, it is easy to verify (cid:80) x
∈ X
α cx f ( x ) =
(cid:80) x ∈ X
2 α cx f ( x ) .
When using the improved mechanism shown in Eq.
(10), we have (cid:80) x ∈ X
α cx f ( x ) − (cid:80) x ∈ X
α cx f ( x ) =
(cid:15) ( 1 | X 1 | − 1 | X 2 | ) α cc f ( c ) , where | X 1 | =
|N 1 | , and | X 2 | =
|N 2 | .
Since | X 1 | (cid:54) =
| X 2 | , (cid:80) x ∈ X
α cx f ( x ) − (cid:80) x ∈ X
2 α cx f ( x ) (cid:54) = 0 , meaning the improved joint attention mechanism can distinguish those distinct multiset which T fails to distinguish.
Similarly, we can prove the aggregation function using the improved attention mechanism concerning Eq.
(8) also can distinguish those distinct multisets which T fails to distinguish.
[Equation]
D D ETAILED EXPERIMENTAL SET - UP AND ADDITIONAL TESTS
In this section, how the experiments used to validate the effectiveness of the proposed JATs are set up is introduced with more details.
D. D ATASET DESCRIPTION
Four document networks, which are Cora, Citeseer, Pubmed (Sen et al., 2008; Lu & Getoor, 2003), and Ogbn-Arxiv Hu et al.
(2020), are used in our experiments to validate the effectiveness of different GNNs.
In these four networks, vertices, edges, and vertex features represent the documents, citations between pairwise documents, and the bag-of-words representations of the documents, respectively.
Besides, each node in these three document networks has a class label.
[Table]
As graph transductive learning is mainly considered in our experiment to validate different approaches, we set up the experiment closely following the settings in the related works (Hu et al., 2020; Yang et al., 2016; Veliˇckovi´c et al., 2018). All the datasets are into three parts: training, validation, and testing. For datasets Cora, Citeseer, and Pubmed, we use only 20 nodes per class for training, but all the feature vectors. For dataset Ogbn-Arxiv, we use a practical split strategy, segmenting the nodes which represent the academic papers, according to publication years
Hu et al. Papers published up to 2017 are in the training split. While papers published in 2018 and 2019 are used for validation and test, respectively.
The performance of different approaches are evaluated on the test split.
The statistics of these benchmarking datasets are summarized in Table 4, where N , | E | , D , and C denote the number of vertices, edges, vertex features, and number of classes in each dataset, respectively.
D. E XPERIMENTAL SCENARIOS
The effectiveness of different methods is validated via allowing them to perform semi-supervised learning (transductive learning) in the aforementioned graph datasets.
To compare the proposed JATs impartially with other baselines, we closely follow the experimental paradigms used in previous works (Hu et al., 2020; Veli ˇ ckovi ´ c et al., 2018; Kipf & Welling, 2017; Yang et al., 2016).
Speciﬁcally, only partial nodes in each class are labeled in the training stage, and all the graph data, including vertex features and edges, can be accessible for the training of neural networks.
The predictive ability of the approaches is validated using the nodes in the test split.
Algorithm performance is evaluated using Accuracy ( Micro − F 1 ).
For the network structure of JAT, following GAT (Veli ˇ ckovi ´ c et al., 2018), we use a similar twolayer deep neural network to learn node representations in all the testing datasets.
The ﬁrst layer consists of multiple attention heads computing multiple features, followed by an exponential linear unit.
In our experiment, eight attention heads are used for Cora, Citeseer, and Pubmed datasets and three attention heads are used for Ogbn-Arxiv.
The second layer, composed of attention heads and a softmax activation is used for node classiﬁcation.
For Cora, Citeseer and Ogbn-Arxiv, a single attention head is used, while for Pubmed, eight output attention heads are used, to compute C features.
Regularization is applied within the model to cope with the small training set sizes.
During training, L 2 regularization with λ = 0 . 0005 and dropout with p = 0 .
6 are applied for both layers’ inputs when JAT is training with Cora, Citeseer, and Pubmed.
In Ogbn-Arxiv dataset, the setting of λ = 0 and dropout with p = 0 .
05 is used. All the networks are initialized using Glorot initialization and trained to minimize cross-entropy on the training nodes using the Adam SGD optimizer (Kingma & Ba, 2014).
A learning rate of 0. is used for datasets Cora, Citeseer, and Pubmed, while that of 0.002 is used in dataset Ogbn-Arxiv. As JAT requires structural coefﬁcients to compute attention scores, Eq. (2) needs optimizing. In our experiments, we use two different strategies to obtain optimal structural coefﬁcients.
For those datasets with small size, e.g., Cora, Citeseer, and Pubmed, Eq. (2) is optimized together with the training of the networks.
In other words, a loss function aggregating cross-entropy on the training nodes and structure reconstruction error Eq. (2) is used. Eq. (2) is iteratively optimized along with the training of JAT and the structural coefﬁcients obtained in each epoch can be used by JAT for the computation of attention scores.
For datasets possessing huge number of nodes, e.g., OgbnArxiv, Eq.
(2) is optimized ahead of training JATs for the purpose of dealing with limited computing resources.
Under such a case, the training efﬁciency of JATs can be signiﬁcantly improved.
[Figure]
D. S ENSITIVITY TEST ON β
For sensitivity test on β , which determines the relative signiﬁcance of graph self-expression (Eq.
(2)), we set β = [0 . 01 , 0 . 1 , 1 , 5 , 10 , 20 , 50 , 100] and let two versions of JAT to classify nodes on all the three datasets.
The corresponding results have been plotted in Fig.
3 (a) and (b).
As the ﬁgures depict, both two versions of JAT may obtain a robust performance when different settings of β are used.
Based on the the obtained results, we set β = 5 for JATs when they perform the tasks of semi-supervised node classiﬁcation on various datasets. E D ISCUSSIONS In this section, we perform some detailed discussions that may help one to better understand the proposed JATs.
E. M ODEL COMPLEXITY
Based on the descriptions in Section 2, we may analyze the computational complexity of the proposed JATs.
According to Eq.
(3), for each node in the graph, the complexity for computing all the structural coefﬁcients can be expressed as O ( N 2 + N ) for the worst case.
However, as real graphs are always very sparse, the complexity for learning structural coefﬁcients can be reduced to O ( e 2 + e ) , where e represents average degree of the nodes.
As the learning of each C i, : is independent of others,
Eq.
(2) can be efﬁciently solved via parallel or distributed optimization strategies.
{+For the computation of each attention head in the graph neural network, its complexity for learning F l +1 features for each node can be represented as O ( ND l D l +
+}
+( | E | + e ) D l +1 ) , which is same to the classical GAT.
{+If there are K attention heads used, the complexity becomes O ( K ( ND l D l +
+}
+ ( | E | + e ) D l +1 )) .
E. S TRUCTURAL INFORMATION FOR COMPUTING ATTENTION SCORES
In the proposed JATs, structural coefﬁcients learned from graph subspace are used for computing attention scores and the corresponding experimental performance is robust when compared with other state-of-the-art baselines.
The module for learning structural coefﬁcients is motivated by subspace clustering (Elhamifar & Vidal, 2013; Ji et al., 2014), which is an effective methodology for uncovering data clusters through performing the spectral analysis on the coefﬁcient matrix ( C ).
Such matrix is also named as self-expressiveness matrix as it contains the coefﬁcients which can construct a linear combination of each sample using others.
Given Eq.
(3), we know that the subspace learning module can directly infer the node-node correlations based on the graph data and those nodes sharing similar global structures (i.e., graph connectivity) may have higher structural coefﬁcients.
Rather than performing spectral analysis on C , JAT directly makes use of the coefﬁcients in it to compute attention scores for representation learning.
This may allow JAT to pay more attention on those nodes which are structurally correlated when aggregating features passed to the higher layers.
Though an effective method for structural coefﬁcients learning, the proposed subspace module is only one of alternatives that can be utilized by the generic framework JAT.
As shown in the manuscript, C learned by the subspace learning module is different from the one learned by those methods for structural network embedding, such as Deepwalk Perozzi et al.
(2014), Node2Vec
Grover & Leskovec (2016), Struc2vec (Ribeiro et al., 2017), and LINE Tang et al. (2015).
For such network embedding methods, the node-node correlation can be recovered via obtaining the inner product of node-wise representations.
Given the fact that these network embedding methods also can effectively capture the node-node correlations, they are also effective alternatives for learning C for the proposed JAT.
In the ablation study, we show that JAT may perform robustly even working with simple measures, e.g., Cosine and Jaccard similarity for obtaining node-node structural correlations, which indicates the proposed joint attention mechanisms are the main reasons boosting the performance of the attention-based GNNs.
Thus, any measures/methods, that may effectively capture structural relationship between pairwise nodes in the graph, including those network embedding approaches and deep structural learning approach such as MoNet (Monti et al., 2017) can be used by JAT to capture node-node structural correlations for computing attention scores.
E. R ELATION BETWEEN JAT, APPROACHES TO NETWORK EMBEDDING AND OTHERS
Concatenating the learned structural representations and corresponding node features, effective network embedding approaches, such as Deepwalk Perozzi et al.
Node2Vec Grover & Leskovec (2016), Struc2vec (Ribeiro et al., 2017), and LINE Tang et al.
(2015) can also consider both structural and feature information for the subsequent learning tasks.
Compared with them, the proposed
JAT still has its unique properties for learning graph representations.
Rather than directly concatenating the structural and feature vectors, JAT utilizes novel joint attention mechanisms which may automatically determine whether graph structure or node feature is more important for representation learning, to infer low-dimensional node representations via multi-layer feature aggregation.
As
C is used for computing attention scores, the node representations learned by JAT are automatically implanted with meaningful structural information hidden in the graph.
It is seen that the strategy for automatically determining the relative signiﬁcance between graph structure and vertex features and the veriﬁed expressive power distinguish the proposed JAT from other related works.
Besides, for attention-based GNNs, the proposed joint attention mechanisms are also novel attempts to effectively consider structural information for computing attention scores.
No previous mechanisms for
GNNs can consider learning the relative signiﬁcance between graph structure and node features for computing attentions scores.
The proposed JAT is also different from those GNNs considering structural information, e.g., diffusion (Klicpera et al., 2019; Busch et al., 2020) or GCNs for subspace clustering (Cai et al., 2020).
Those diffusion based GNNs mainly consider higher-order structural information among node pairs to formulate the functions for feature aggregation.
As for GNNs for subspace clustering, their mainly task is to learn the self-expressive matrix, i.e., C based on the deep neural networks and then data clusters are inferred via performing spectral analysis on the learned
C .
Compared with them, JAT directly makes use of C and node features to compute attention scores for aggregating the features passed to the higher layers.
C in the proposed JAT plays an important role in obtaining appropriate feature aggregations, rather than clustering.
Given the mentioned facts, the proposed JAT is novel compared with existing methods.
E. L IMITATIONS OF JAT
Though effective in predicting node labels in transductive learning tasks, the proposed JAT has some limitations that have overcome in the future.
First, the efﬁciency for learning C in the current version of JAT is somewhat unsatisfying.
Based on the analysis on computational complexity, JAT has to pre-train C for those massive datasets (e.g., OGBN-Arxiv) due to the limited computing resources.
More efﬁcient techniques for capturing the node-node structural correlations can therefore be considered by JAT.
Based on the results shown in the ablation studies, the proposed joint attention mechanisms ensure JAT to perform robustly as long as the used node-node correlations can effectively reveal the structural properties of the graph.
Thus, those network embedding approaches and other efﬁcient methods for deep structural learning can be combined with the generic
JAT.
Second, the expressive power might not always hold when JAT deals with graph data whose node features are in the uncountable space Corso et al.
Given Eq.
(10), it is known that the aggregated features for a central node equal the expectation of the features in the multiset plus a degree-related proportion of the original central node feature.
Obviously, for those graph whose features are in uncountable space, the proposed joint attention mechanisms are hard to always be injective as they may produce a same result of feature aggregation for different graphs.
Attention mechanisms which are also theoretically powerful in the space uncountable are therefore needed by
JAT and other attention-based GNNs.