H OW TO F IND Y OUR F RIENDLY N EIGHBORHOOD : G RAPH A TTENTION D ESIGN WITH S ELF -S UPERVISION
Anonymous authors
Paper under double-blind review
A BSTRACT
Attention mechanism in graph neural networks is designed to assign larger weights to important neighbor nodes for better representation.
However, what graph attention learns is not understood well, particularly when graphs are noisy.
In this paper, we propose a self-supervised graph attention network (SuperGAT), an improved graph attention model for noisy graphs.
Speciﬁcally, we exploit two attention forms compatible with a self-supervised task to predict edges, whose presence and absence contain the inherent information about the importance of the relationships between nodes.
By encoding edges, SuperGAT learns more expressive attention in distinguishing mislinked neighbors.
We ﬁnd two graph characteristics inﬂuence the effectiveness of attention forms and self-supervision: homophily and average degree.
Thus, our recipe provides guidance on which attention design to use when those two graph characteristics are known.
Our experiment on 17 real-world datasets demonstrates that our recipe generalizes acrossdatasets of them, and our models designed by recipe show improved performance over baselines.
I NTRODUCTION
Graphs are widely used in various domains, such as social networks, biology, and chemistry.
Since their patterns are complex and irregular, learning to represent graphs is challenging (Bruna et al., 2014; Henaff et al., 2015; Defferrard et al., 2016; Duvenaud et al., 2015; Atwood & Towsley, 2016).
Recently, graph neural networks (GNNs) have shown a signiﬁcant performance improvement by generating features of the center node by aggregating those of its neighbors (Zhou et al., 2018; Wu et al., 2020).
However, real-world graphs are often noisy with connections between unrelated nodes, and this causes GNNs to learn suboptimal representations.
Graph attention networks (GATs) (Veliˇckovi´c et al., 2018) adopt self-attention to alleviate this issue.
Similar to attention in sequential data (Luong et al., 2015; Bahdanau et al., 2015; Vaswani et al., 2017), graph attention captures the relational importance of a graph , in other words, the degree of importance of each of the neighbors to represent the center node.
GATs have shown performance improvements in node classiﬁcation, but they are inconsistent in the degree of improvement across datasets, and there is little understanding of what graph attention actually learns.
Hence, there is still room for graph attention to improve, and we start by assessing and learning the relational importance for each graph via self-supervised attention.
We leverage edges that explicitly encode information about the importance of relations provided by a graph.
If node i and j are linked, they are more relevant to each other than others, and if node i and j are not linked, they are not important to each other.
Although conventional attention is trained without direct supervision, if we have prior knowledge about what to attend, we can supervise attention using them (Knyazev et al., 2019; Yu et al., 2017).
Speciﬁcally, we exploit a self-supervised task, using the attention value as input to predict the likelihood that an edge exists between nodes.
To encode edges in graph attention, we ﬁrst analyze what graph attention learns and how it relates to the presence of edges.
In this analysis, we focus on two commonly used attention mechanisms,
GAT’s original single-layer neural network (GO) and dot-product (DP), as building blocks of our proposed model, self-supervised graph attention network (SuperGAT).
We observe that DP attention shows better performance than GO attention in the task to predict link with attention value.
On the other hand, GO attention outperforms DP attention in capturing label-agreement between a target node and its neighbors.
Based on our analysis, we propose two variants of SuperGAT, scaled dotproduct (SD) and mixed GO and DP (MX), to emphasize the strength of GO and DP.
Then, which graph attention models the relational importance best and produces the best node representations?
We ﬁnd that it depends on the average degree and homophily of the graph.
We generate synthetic graph datasets with various degrees and homophily, and analyze how the choice of attention affects node classiﬁcation performance.
Based on this result, we propose a recipe to design graph attention with edge self-supervision that works most effectively for given graph characteristics.
We conduct experiments on a total of 17 real-world datasets and demonstrate that our recipe can be generalized across them.
In addition, we show that models developed by our method improve performance over baselines on four standard benchmarks (Cora, CiteSeer, PubMed, and PPI).
We present the following contributions.
First, we present models with self-supervised attention using edge information.
Second, we analyze the classic attention forms GO and DP using label-agreement and link prediction tasks, and this analysis reveals that GO is better at label agreement and DP at link prediction.
Third, we propose recipes to design graph attention concerning homophily and average degree and conﬁrm its validity through experiments on real-world datasets.
We make our code available for replication and future research ( See supplementary materials ).
R ELATED W ORK
Deep neural networks are actively studied in modeling graphs, for example the graph convolutional networks (Kipf & Welling, 2017) which approximate spectral graph convolution (Bruna et al., 2014;
Defferrard et al., 2016).
A representative work in a non-spectral way is the graph attention networks (GATs) (Veli ˇ ckovi ´
c et al., 2018) which model relations in graphs using self-attention mechanism (Vaswani et al., 2017).
Similar to attention in sequence data (Bahdanau et al., 2015;
Luong et al., 2015; Vaswani et al., 2017), variants of attention in graph neural networks (Thekumparampil et al., 2018; Zhang et al., 2018;
Wang et al., 2019a; Gao & Ji, 2019; Zhang et al., 2020;
Hou et al., 2020) are trained without direct supervision.
Our work is motivated by studies that improve attention’s expressive power by giving direct supervision (Knyazev et al., 2019; Yu et al., 2017).
Speciﬁcally, we employ a self-supervised task to predict edge presence from attention value.
This is in line with two branches of recent GNN research: self-supervision and graph structure learning.
Recent studies about self-supervised learning for GNNs propose tasks leveraging the inherent information in the graph structure: clustering, partitioning, context prediction after node masking, and completion after attribute masking (Hu et al., 2020b;
Hui et al., 2020; Sun et al., 2020; You et al., 2020).
To the best of our knowledge, ours is the ﬁrst study to analyze self-supervised learning of graph attention with edge information.
Our self-supervised task is similar to link prediction (Liben-Nowell & Kleinberg, 2007), which is a well-studied problem and recently tackled by neural networks (Zhang & Chen, 2017; 2018).
Our DP attention to predict links is motivated by graph autoencoder (GAE) (Kipf & Welling, 2016) and its extensions (Pan et al., 2018; Park et al., 2019) reconstructing edges by applying a dot-product decoder to node representations.
Graph structure learning is an approach to learn the underlying graph structure while jointly learning downstream tasks (Jiang et al., 2019; Franceschi et al., 2019; Klicpera et al., 2019; Stretcu et al., 2019; Zheng et al., 2020).
Since real-world graphs often have noisy edges, encoding structure information contributes to learn better representation.
However, recent models with graph structure learning suffer from high memory and computational complexity.
Some studies target all spaces where edges can exist, so they require O ( | V | 2 ) space and computational complexity (Jiang et al., 2019; Franceschi et al., 2019).
Others using iterative training (or co-training) between the GNNs and the structure learning model are time-intensive in training (Franceschi et al., 2019; Stretcu et al., 2019).
We moderate this problem using graph attention, which consists of parallelizable operations, and our model is built on it without additional parameters.
Our model learns attention values that are predictive of edges, and this can be seen as a new paradigm of learning the graph structure.
M ODEL
In this section, we review the original GAT (Veliˇckovi´c et al., 2018) and then describe our selfsupervised GAT (SuperGAT) models. [Figure]
⃗
[Figure] ϕ ij [Figure] DP ϕ ij
[Figure]
SD e ij
[Figure]
ϕ ij
[Figure]
GO
[Figure]
σ σ
[Figure]
α ij softmax
[Figure]
= P (( j , i ) ∈ E ) e ij ϕ ij
Notation For a graph G =
( V, E ) , N is the number of nodes and F l is the number of features at layer l .
Graph attention layer takes a set of features H l = { h l 1 , . .
, h lN } , h li ∈ R F l as input and produces output features H l +
= { h l +11 , . . .
, h l +1 N } .
To compute h l +1 i , the model multiplies the weight matrix W l +1 ∈ R F l +1 × F l to H l , linearly combines the features of its ﬁrst-order neighbors (including itself) j ∈ N i ∪ { i } by attention coefﬁcients α l +1 ij , and ﬁnally applies a non-linear activation ρ .
That is h l +
i
[Equation] } α
[-+
-][Equation]
W l [-+
 h-][Equation]
(cid:17) .
We can compute
[Equation]
) by normalizing e l +1 ij = a e ( W l +1 h li , W l +1 h lj ) with softmax on N i ∪ { i } under leaky ReLU activation (Maas et al., 2013), where a e is a function of the form R F l +1 × R F l +1 → R .
Graph Attention Forms Among two widely used attention mechanisms, the original GAT (GO) computes the coefﬁcients by single-layer feed-forward network parameterized by a l +1 ∈ R 2 F l +1 .
The other is the dot-product (DP) attention, (Luong et al., 2015; Vaswani et al., 2017) motivated by prior work on node representation learning, and it adopts the same mathematical expression for link prediction score (Tang et al., 2015; Kipf & Welling, 2016),
[Equation]
From now on, we call GAT that uses GO and DP as GAT GO and GAT DP , respectively.
Self-supervised Graph Attention Network
We propose SuperGAT with the idea of guiding attention with the presence or absence of an edge between a node pair.
We exploit the link prediction task to self-supervise attention with labels from edges: for a pair i and j , 1 if an edge exists andotherwise.
We introduce a φ with sigmoid σ to infer the probability φ ij of an edge between i and j .
[Equation]
We employ four types (GO, DP, SD, and MX) of SuperGAT based on GO and DP attention.
For a φ , the form of which is the same as a e in GAT GO and GAT DP , we name them SuperGAT GO and
SuperGAT DP respectively.
For more advanced versions, we describe SuperGAT SD (Scaled Dotproduct) and SuperGAT MX (Mixed GO and DP) by unnormalized attention e ij and probability φ ij that an edge exist between i and j .
[Equation]
SuperGAT SD divides the dot-product of nodes by a square root of dimension as Transformer (Vaswani et al., 2017).
This prevents some large values to dominate the entire attention after softmax.
SuperGAT MX multiplies GO and DP attention with sigmoid.
The motivation of this form comes from the gating mechanism of Gated Recurrent Units (Cho et al., 2014).
Since DP attention with the sigmoid represents the probability of an edge, it can softly drop neighbors that are not likely linked while implicitly assigning importance to the remaining nodes.
Training samples are a set of edges E and the complementary set E c
= ( V × V ) \ E .
However, if the number of nodes is large, it is not efﬁcient to use all possible negative cases in E c .
So, we use negative sampling as in training word or graph embeddings (Mikolov et al., 2013; Tang et al., 2015;
Grover & Leskovec, 2016), arbitrarily choosing a total of p n · | E | negative samples E − from E c where the negative sampling ratio p n ∈ R + is a hyperparameter.
SuperGAT is capable of modeling graphs that are sparse with a sufﬁciently large number of negative samples (i.e., | V × V | (cid:29) | E | ), but this is generally not a problem because most real-world graphs are sparse (Chung, 2010).
We deﬁne the optimization objective of layer l as a binary cross-entropy loss L lE ,
[Equation]
where 1 · is an indicator function.
We use a subset of E ∪ E − sampled by probability p e ∈ (0 , 1] (also a hyperparameter) at each training iteration for a regularization effect from randomness.
Finally, we combine cross-entropy loss on node labels ( L V ), self-supervised graph attention losses for all L layers ( L lE ), and L2 regularization loss, with mixing coefﬁcients λ E and λ 2 .
[Equation]
We use the same form of multi-head attention in GAT and take the mean of each head’s attention value before the sigmoid to compute φ ij .
Note that SuperGAT has equivalent time and space complexity as GAT.
To compute L lE for one head, we need additional operations in terms of
O ( F l · | E ∪ E − | ) , and we do not need extra parameters.
E XPERIMENTS
Our primary research objective is to design graph attentions that are effective with edge selfsupervision.
To do this, we pose four speciﬁc research questions.
We ﬁrst analyze what basic graph attentions (GO and DP) learn (RQ1 and 2) and how that can be improved with edge self-supervision (RQ3 and 4).
We describe each research question and the corresponding experiment design below.
RQ1.
Does graph attention learn label-agreement?
First, we evaluate what the graph attentions of GAT GO and GAT DP learn without edge supervision.
For this, we present ground-truth of relational importance and a metric to assess graph attention with ground-truth.
Wang et al. (2019a) showed that node representations in the connected component converge to the same value in deep GATs. If there is an edge between nodes with different labels, then it will be hard to distinguish the two corresponding labels with GAT of sufﬁciently many layers; that is, ideal attention should give all weights to label-agreed neighbors.
In that sense, we choose label-agreement between nodes as ground-truth of importance.
We compare label-agreement and graph attention based on Kullback–Leibler divergence of the normalized attention α k
=
[ α kk , α k 1 , . . .
, α kJ ] with label agreement distribution for the center node k and its neighbors 1 to J .
The label agreement distribution, (cid:96) k
= [ (cid:96) kk , (cid:96) k 1 , . . .
, (cid:96) kJ ] is deﬁned by, (cid:96) kj = ˆ (cid:96) kj / (cid:80) s ˆ (cid:96) ks , ˆ (cid:96) kj = 1 ( if k and j have the same label ) or 0 ( otherwise ) .
We employ KL divergence in Eq. 8, whose value becomes small when attention captures well the label-agreement between a node and its neighbors.
[Equation]
[Equation]
RQ2.
Is graph attention predictive of edge presence?
To evaluate how well edge information is encoded in SuperGAT, we conduct link prediction experiments with SuperGAT GO and SuperGAT DP using φ ij of the last layer as a predictor.
We measure the performance by AUC over multiple runs.
Since link prediction performance depends on the mixing coefﬁcient λ E in Eq. 6, we adopt multiple λ E ∈ { 10 − 3 , 10 − 2 , . .
We train with an incomplete set of edges, and test with the missing edges and the same number of negative samples.
At the same time, node classiﬁcation performance is measured with the same settings to see how learning edge presence affects node classiﬁcation.
RQ3.
Which graph attention should we use for given graphs?
The above two research questions explore what different graph attention learns with or without supervision of edge presence.
Then, which graph attention is effective among them for given graphs?
We hypothesize that different graph attention will have different abilities to model graphs under various homophily and average degree. We choose these two properties among various graph statistics because they determine the quality and quantity of labels in our self-supervised task. From the perspective of supervised learning of graph attention with edge labels, the learning result depends on how noisy labels are (i.e., how low the homophily is) and how many labels exist (i.e., how high the average degree is).
So, we generate 144 synthetic graphs (Section 4.1) controlling 9 homophily (0.1 - 0.9) and 16 average degree (1 - 100) and perform the node classiﬁcation task in the transductive setting with GCN, GAT GO , SuperGAT SD , and SuperGAT MX .
RQ4.
Does design choice based on RQ3 generalize to real-world datasets?
Experiments on synthetic graphs provide an understanding of attention models’ performance, but they are oversimpliﬁed versions of real-world graphs.
Can design choice from synthetic datasets be generalized to real-world datasets, considering more complex structures and rich features in real-world graphs?
To answer this question, we conduct experiments on 17 real-world datasets with the various average degree (1.8 - 35.8) and homophily (0.16 - 0.91), and compare them with synthetic graph experiments.
4. D ATASETS
Real-world datasets We use a total of 17 real-world datasets (Cora, CiteSeer, PubMed, Cora-ML,
Cora-Full, DBLP, ogbn-arxiv, CS, Physics, Photo, Computers, Wiki-CS, Four-Univ, Chameleon,
Crocodile, Flickr, and PPI) in diverse domains (citation, co-authorship, co-purchase, web page, and biology) and scales (2k - 169k nodes).
We try to use their original settings as much as possible.
To verify research questions 1 and 2, we choose four classic benchmarks: Cora, CiteSeer, PubMed in the transductive setting, and PPI in the inductive setting.
See appendix A.1 for detailed description, splits, statistics (including degree and homophily), and references.
Synthetic datasets We generate random partition graphs of n nodes per class and c classes (Fortunato, 2010), using NetworkX library (Hagberg et al., 2008).
A random partition graph is a graph of communities controlled by two probabilities p in and p out .
If the nodes have the same class labels, they are connected with p in , and otherwise, they are connected with p out .
To generate a graph with an average degree of d avg = n · δ , we choose p in and p out by p in + ( c − 1) · p out = δ .
The input features of nodes are sampled from overlapping multi-Gaussian distributions (Abu-El-Haija et al., 2019).
We set n to 500, c to 10, and choose d avg between 1 and 100 , p in from { 0 . 1 δ, 0 .
2 δ, . .
9 δ } .
We use 20 samples per class for training, 500 for validation and 1000 for test.
4. E XPERIMENTAL S ET - UP
We follow the experimental set-up of GAT with minor adjustments.
All parameters are initialized by Glorot initialization (Glorot & Bengio, 2010) and optimized by Adam (Kingma & Ba, 2014).
We apply L2 regularization, dropout (Srivastava et al., 2014) to features and attention coefﬁcients, and early stopping on validation loss and accuracy.
We use ELU (Clevert et al., 2016) as a nonlinear activation ρ .
Unless speciﬁed, we employ a two-layer SuperGAT with F = 8 features and
K = 8 attention heads (total 64 features).
All models are implemented in PyTorch (Paszke et al., 2019) and PyTorch Geometric (Fey & Lenssen, 2019).
See appendix A.5 for detailed model and hyperparameter conﬁgurations.
Baselines We compare our model against representative graph neural architectures: graph convolutional network (GCN) (Kipf & Welling, 2017) and GraphSAGE (Hamilton et al., 2017).
In addition, for transductive datasets, we choose recent graph neural architectures that learn aggregation coefﬁcients (or discrete structures) over edges: graph attention network (GAT) (Veliˇckovi´c et al., 2018), constrained graph attention network (CGAT 1 ) (Wang et al., 2019a), graph learningconvolutional network (GLCN)
(Jiang et al., 2019), learning discrete structure (LDS) (Franceschi
[Figure]
R ESULTS
This section describes the experimental results which answer the research questions in Section 4.
We include a qualitative analysis of attention, quantitative comparisons on node classiﬁcation and link prediction, and recipes of graph attention design.
The results for sensitivity analysis of important hyper-parameters are in the appendix B.5.
Does graph attention learn label-agreement?
GO learns label-agreement better than DP.
We draw box plots of KL divergence between attention and label agreement distributions of twolayer and four-layer GAT with GO and DP attention for Cora dataset in Figure 2.
We see similar patterns in other datasets and place their plots in the appendix B.3.
At the rightmost of each subﬁgure, we draw the KLD distribution when uniform attention is given to all neighborhoods.
Note that the maximum value of KLD of each node is different since the degree of nodes is different.
Also, the KLD distribution shows a long-tail shape like a degree distribution of real-world graphs.
There are three observations regarding distributions of KLD.
First, we observe that the KLD distribution of GO attention shows a pattern similar to the uniform attention for all citation datasets.
This implies that trained GO attention is similar to the uniform distribution, which is in line with previously reported results in the case of entropy 2 (Wang et al., 2019b).
Second, KLD values of DP attention tend to be larger than those of GO attention for the last layer, resulting in bigger long-tails.
This mismatch between the learned distribution of DP attention and the label agreement distribution suggests that DP attention does not learn label-agreement in the neighborhood.
Third, the deeper the model (more than two), the larger the KLD value of DP attention in the last layer.
This is because the variance of DP attention increases as the layer gets deeper, as explained below in Proposition 1.
Proposition 1.
For l + 1 th GAT layer, if W and (cid:126) a are independent and identically drawn from zero-mean uniform distribution with variance σ 2 w and σ 2 a respectively, assuming that parameters are independent to input features h l and elements of h l are independent to each other,
[Equation]
The proof is given in the appendix B.1.
While the variance of GO depends on the norm of features only, the variance of DP depends on the expectation of the square of input’s dot-product and variance of input’s dot-product.
Stacking GAT layers, the more features of i and j correlate with each other, the larger the input’s dot-product will be.
After DP attention is normalized by softmax, which intensiﬁes the larger values among them, normalized DP attention attends to only a small portion of the neighbors and learns a biased representation.
Is graph attention predictive for edge presence?
DP predicts edge presence better than GO.
In Figure 3, we report the mean AUC over multiple runs (5 for PPI and 10 for others) for link prediction (red lines) and node classiﬁcation (gray lines).
As the mixing coefﬁcient λ E increases, the link prediction score increases in all datasets and attentions.
This is a natural result considering that λ E is the weight factor of self-supervised graph attention loss ( L E in Eq. 6).
For three out of four datasets,
DP attention outperforms GO for link prediction for all range of λ E in our experiment.
Surprisingly, [Figure] [Figure] even for small λ E , DP attention shows around 80 AUC, much higher than the performance of GO attention.
PPI is an exception where GO attention shows higher performance for small λ E than DP, but the difference is slight.
The results of this experiment demonstrate that DP attention is more suitable than GO attention in encoding edges.
This ﬁgure also includes node classiﬁcation performance.
For all datasets except PubMed, we observe a trade-off between node classiﬁcation and link prediction; that is, node classiﬁcation performance decreases in SuperGAT GO and SuperGAT DP as λ E increases and thus link prediction performance increases.
PubMed also shows a decrease in performance at the largest λ E we have tested.
This implies that it is hard to learn the relational importance from edges by simply optimizing graph attention for link prediction.
Which graph attention should we use for given graphs?
It depends on homophily and average degree of the graph.
In Figure 4, we draw the mean test accuracy gains (over 5 runs) against GAT GO as the average degree increases from 1 to 100, for different values of homophily, on 64 synthetic graphs with
GCN, SuperGAT SD , SuperGAT MX .
See full results from appendix B.2.
We deﬁne homophily h as the average ratio of neighbors with the same label as the center node (Pei et al., 2020).
That is h = 1 | V | (cid:80) i ∈
V (cid:16)(cid:80)
j ∈ N i 1 l ( i )
=
l ( j ) (cid:14) | N i | (cid:17) , where l ( i ) is the label of node i .
The expectation of homophily for random partition graphs is analytically p in /δ , and we just adopt this value to label the homophily of graphs in Figure 4.
We make the following observations from this ﬁgure.
First, if the homophily is low ( ≤ 0 . 2 ),
SuperGAT SD performs best among models because DP attention tends to focus on a small number of neighbors.
This result empirically conﬁrms what we analytically found in Proposition 1.
Second, even when homophily is low, the performance gain of SuperGAT against GAT increases as the average degree increases to a certain level (around 10), meaning relation modeling can beneﬁt from self-supervision if there are sufﬁciently many edges providing supervision.
This is in agreement with prior study of label noise for deep neural networks where they ﬁnd that the absolute amount of data with correct labels affects the learning quality more than the ratio between data with noisy and correct labels (Rolnick et al., 2017).
Third, if the average degree and homophily are high enough, there is no difference between all models, including GCNs.
If there are more correct edges beyond a certain amount, we can learn ﬁne representation without self-supervision.
Most importantly, if the average degree is not too low or high and homophily is above 0 . 2 , SuperGAT MX performs better than or similar to SuperGAT SD .
This implies that we can take advantage of both GO attention to learn label-agreement and DP attention to learn edge presence by mixing GO and DP.
Note that many of the real-world graphs belong to this range of graph characteristics.
[Table]
[Table] [Table] The results on synthetic graphs imply that understanding of graph domains should be preceded to design graph attention.
That is, by knowing the average degree and homophily of the graphs, we can choose the optimal graph attention in our design space.
Does design choice based on RQ3 generalize to real-world datasets?
It does for 15 of 17 realworld datasets.
In Figure 5, we plot the best-performed graph attention for synthetic graphs with square points in the plane of average degree and homophily.
The size is the performance gain of SuperGAT against GAT, and the color indicates the best model.
If the difference is not statistically signiﬁcant ( p -value ≥ .05) between GAT and SuperGAT, and between SuperGAT MX and SuperGAT SD , we mark as GAT-Any and SuperGAT-Any, respectively.
We call this plot a recipe since it introduces the optimal attention to a speciﬁc region’s graph.
Now we map the results of 17 real-world datasets in Figure 5 according to their average degree and homophily.
Average degree and homophily can be found in the appendix A.1, and experimental results of graph attention models are summarized in Tables 1, 2 and 3.
We report the mean and standard deviation of performance over multiple seeds (30 for graphs with more than 50k nodes and 100 for others).
We put unpaired t-test results of SuperGAT with GAT GO with asterisks.
We ﬁnd that the graph attention recipe based on synthetic experiments can generalize across realworld graphs.
PPI and Four-Univ ( (cid:78) ) are surrounded by squares of SuperGAT SD ( (cid:4) ) at the bottom of the plane.
Wiki-CS ( (cid:78) ), located in the SuperGAT-Any region ( (cid:4) ), also show no difference in performance between SuperGATs.
Nine datasets ( (cid:78) ), which SuperGAT MX shows the highest performance, are located in the MX regions ( (cid:4) ) or within the margin of two squares.
Note that there are two MX regions: lower-middle average degree (2.5 - 7.5) and high homophily (0.8 - 0.9), and uppermiddle average degree (7.5 - 50) and lower-middle homophily (0.3 - 0.5).
There are ﬁve datasets with no signiﬁcant performance change across graph attention ( (cid:78) ).
CiteSeer, Photo, and Computers are within a margin of one square from the GAT-Any region ( (cid:4) ); however, Flickr and Crocodile are in the SuperGAT-Any region.
To ﬁnd out the cause of this irregularity, we examine the distribution of degree and per-node homophily (appendix A.4).
We observe a more complex mixture distribution of homophily and average degree in Flickr and Crocodile, and this seems to be equivalent to mixing graphs of different characteristics, resulting in inconsistent results with our attention design recipe.
Comparison with baselines For a total of 17 datasets, SuperGAT outperforms GCN fordatasets, GAT for 12 datasets, GraphSAGE for 16 datasets.
Interestingly, for CS, Physics, Cora-
[Figure] ML, and Flickr, in which our model performs worse than GCN, GAT also cannot surpass GCN.
It is not yet known when the degree-normalized aggregation of GCN outperforms the attentionbased aggregation, and more research is needed to ﬁgure out how to embed the degree information into graph attention.
Tables 2 and 3 show performance comparisons between SuperGAT and recent GNNs for Cora, CiteSeer, PubMed, and PPI.
Our model performs better for CiteSeer (0.6%p) and PubMed (3.4%p) than GLCN, which gives regularization to all relations in a graph.
GCN +
NS (NeuralSparse) performs better than our model for CiteSeer (1.5%p) but not for Cora (0.6%p).
CGAT modiﬁed for semi-supervised learning shows lower performance than GAT.
Although LDS and GAM which use iterative training show better performance except for LDS on Cora, these models require signiﬁcantly more computation for the iterative training.
For example, GCN + GAM compared to our model needs more × 34 more training time for Cora, × 72 for CiteSeer, and ×for PubMed.
See appendix A.7 and B.4 for the experimental set-up and the result of wall-clock time analysis.
C ONCLUSION
We proposed novel graph neural architecture designs to self-supervise graph attention following the input graph’s characteristics.
We ﬁrst assessed what graph attention is learning and analyzed the effect of edge self-supervision to link prediction and node classiﬁcation performance.
This analysis showed two widely used attention mechanisms (original GAT and dot-product) have difﬁculty encoding label-agreement and edge presence simultaneously.
To address this problem, we suggested several graph attention forms that balance these two factors and argued that graph attention should be designed depending on the input graph’s average degree and homophily.
Our experiments demonstrated that our graph attention recipe generalizes across various real-world datasets such that the models designed according to the recipe outperform other baseline models.
A E XPERIMENTAL S ET - UP A. R EAL - WORLD D ATASET
In this section, we describe details (including nodes, edges, features, labels, and splits) of real-world datasets.
We report statistics of real-world datasets in Tables 4 and 5.
For multi-label graphs (PPI), we extend the homophily to the average of the ratio of shared labels on neighbors over nodes, i.e., h = 1 | V | (cid:80) i ∈
V (cid:16)(cid:80)
j ∈ N i | C
i ∩ C j | (cid:14) ( | N i | · | C | ) (cid:17) , where C i is a set of labels for node i and C is a set of all labels.
A.1. C ITATION N ETWORK
We use a total of 7 citation network datasets.
Nodes are documents, and edges are citations.
The task for all citation network datasets is to classify each paper’s topic.
Cora, CiteSeer, PubMed We use three benchmark datasets for semi-supervised node classiﬁcation tasks in the transductive setting (Sen et al., 2008; Yang et al., 2016).
The features of the nodes are bag-of-words representations of documents.
We follow the train/validation/test split of previous work (Kipf & Welling, 2017).
We use 20 samples per class for training, 500 samples for the validation, and 1000 samples for the test.
Cora-ML, Cora-Full, DBLP
These are other citation network datasets from Bojchevski &
G¨unnemann (2018).
Node features are bag-of-words representations of documents.
For CoraFull with features more than 5000, we reduce the dimension to 500 by performing PCA.
We use the split setting in Shchur et al.
(2018): 20 samples per class for training, 30 samples per class for validation, the rest for the test.
ogbn-arxiv
The ogbn-arxiv is a recently proposed large-scale dataset of citation networks (Hu et al., 2020a;
Wang et al., 2020).
Nodes represent arXiv papers, and edges indicate citations between papers, and node features are mean vectors of skip-gram word embeddings of their titles and abstracts.
We use the public split by publication dates provided by the original paper.
A.1. C O - AUTHOR N ETWORK
CS, Physics The CS and Physics are co-author networks in each domain (Shchur et al., 2018).
Nodes are authors, and edges mean whether two authors co-authored a paper.
Node features are paper keywords from the author’s papers, and we reduce the original dimension (6805 and 8415) to 500 using PCA.
The split is the 20-per-class/30-per-class/rest from Shchur et al.
The goal of this task is to classify each author’s respective ﬁeld of study.
A.1. A MAZON C O - PURCHASE
Photo, Computers The Photo and Computers are parts of the Amazon co-purchase graph (McAuley et al., 2015; Shchur et al., 2018).
Nodes are goods, and edges indicate whether two goods are frequently purchased together, and node features are a bag-of-words representation of product reviews.
The split is the 20-per-class/30-per-class/rest from Shchur et al.
The task is to classify the categories of goods.
A.1. W EB P AGE N ETWORK
Wiki-CS The Wiki-CS dataset is computer science related page networks in Wikipedia (Mernyei & Cangea, 2020).
Nodes represent articles about computer science, and edges represent hyperlinks between articles.
The features of nodes are mean vectors of GloVe word embeddings of articles.
There are 20 standard splits, and we experiment with ﬁve random seeds for each split (totalruns).
The task is to classify the main category of articles.
Chameleon, Crocodile These datasets are Wikipedia page networks about speciﬁc topics,
Chameleon and Crocodile (Rozemberczki et al., 2019).
Nodes are articles, and edges are mutual
[Table]
links between them.
Node features are a bag-of-words representation with informative nouns in the article.
The number of features is 13183, but we use a reduced dimension of 500 by PCA.
The split is 20-per-class/30-per-class/rest from Shchur et al.
The original dataset is for the regression task to predict monthly trafﬁc, but we group values into six bins and make it a classiﬁcation problem.
Four-Univ The Four-Univ dataset is a web page networks from computer science departments of diverse universities (Craven et al., 1998).
Nodes are web pages, edges are hyperlinks between them, and node features are TF-IDF vectors of web page’s contents.
There are ﬁve graphs consists of four universities (Cornell, Texas, Washington, and Wisconsin) and a miscellaneous graph from other universities.
As the original authors suggested 3 , we use three graphs of universities and a miscellaneous graph for training, another one graph for validation (Cornell), and the other one graph for the test (Texas).
Classiﬁcation labels are types of web pages (student, faculty, staff, department, course, and project).
A.1. F LICKR
The Flickr dataset is a graph of images from Flickr (McAuley & Leskovec, 2012; Zeng et al., 2020).
Nodes are images, and edges indicate whether two images share common properties such as geographic location, gallery, and users commented.
Node features are a bag-of-words representation of images.
We use labels and split in in Zeng et al.
For labels, they construct seven classes by manually merging 81 image tags.
A.1. P ROTEIN -P ROTEIN I NTERACTION
The protein-protein interaction (PPI) dataset (Zitnik & Leskovec, 2017; Hamilton et al., 2017; Subramanian et al., 2005) is a well-known benchmark in the inductive setting.
A graph is given for human tissue, the nodes are proteins, the node’s features are biological signatures like genes, and the edges illustrate proteins’ interactions.
The dataset consists of 20 training graphs, two validation graphs, and two test graphs.
This dataset has multi-labels of gene ontology sets.
A. L INK P REDICTION
For link prediction, we split 5% and 10% of edges for validation and test set, respectively.
We ﬁx the negative edges for the test set and sample negative edges for the training set at each iteration.
[Table]
A. S YNTHETIC D ATASET
To the best of our knowledge [Table] our synthetic datasets are not used in recent literature.
Therefore, we give some small examples of synthetic datasets to see qualitatively how the average degree and homophily vary according to δ and p in .
Speciﬁcally, we draw 2D TSNE plot of node features and edges in Figure 6. In this ﬁgure, we can observe that the average degree ( d avg = n · δ ) increases as δ increases, and homophily ( h = p in /δ ) increases as p in increases.
Note that these are raw input features sampled from the 2D Gaussian distribution, not learned node representation.
We use the code from the prior work 4 (Abu-El-Haija et al., 2019) and apply normalization by standard score.
We choose δ from { 0 . 025 , 0 . 2 } , p in from { 0 . 1 δ, 0 .
5 δ, 0 .
9 δ } , and ﬁx n = 100 and c = 5 .
A. D ISTRIBUTION OF D EGREE AND H OMOPHILY OF D ATASETS
In Figure 7, we draw kernel density estimation plots of per-node homophily and degree of nodes in real-world datasets.
We deﬁne per-node homophily as the ratio of neighbors with the same label as the center node, that is, h i = (cid:80) j ∈ N i 1 l
i )=
l ( j ) (cid:14) | N i | .
Note that we deﬁne homophily as h =| V | (cid:80) i ∈
V h i in Section 5.
py
[Figure] We focus more on the part outside of where the degree is 1 (0 with log scale), and the per-node homophily is 0.
These are leaf nodes incorrectly connected and does not signiﬁcantly affect the learning overall graph representation.
In most datasets, only the largest mode exists, or there are some small modes around it.
However, in Flickr and Crocodile, we can observe that the interval between modes is wide.
More speciﬁcally, Crocodile’s modes are in the area of (high degree, low per-node homophily) and (low degree, high per-node homophily), and Flickr’s modes cover most homophily at a speciﬁc degree.
Note that we can regard a mixture of distribution as a mixture of different sub-graphs.
We argue that this is the reason our recipe does not ﬁt for these two datasets.
A. M ODEL & H YPERPARAMETER C ONFIGURATIONS
Model Since we experiment with numbers of datasets, we maintain almost the same conﬁgurations across datasets.
We do not use other methods such as residual connections, deeper layers, batch normalization, edge augmentation, and more hidden features, although we have conﬁrmed from previous studies that these techniques contribute to performance improvement.
For example, prior [Table] [Table] work has shown f1-score close to 100 for PPI.
To clearly see the difference between the various graph attention designs, we intentionally keep a simple model conﬁguration.
Hyperparameter For real-world datasets, we tune two hyperparameters (mixing coefﬁcients λand λ E ) by Bayesian optimization for the mean performance of 3 random seeds.
We choose negative sampling ratio p n from { 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } , and edge sampling ratio p e from { 0 . 6 , 0 . 8 , 1 . 0 } .
We ﬁx dropout probability to 0.0 for PPI, 0.2 for ogbn-arxiv, 0.6 for others.
We set learning rate to 0.
(ogbn-arxiv), 0.01 (PubMed, PPI, Wiki-CS, Photo, Computers, CS, Physics, Crocodile, Cora-Full,
DBLP), 0.005 (Cora, CiteSeer, Cora-ML, Chameleon), 0.001 (Four-Univ).
For ogbn-arxiv, we set the number of features per head to 16 and the number of heads in the last layer to one; otherwise, we use eight features per head and eight heads in the last layer.
For synthetic datasets, we choose λ E from { 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 , 10 , 10 2 } and λ 2 from { 10 − 7 , 10 − 5 , 10 − 3 } .
We ﬁx learning rate to 0.01, dropout probability to 0.2, p n to 0.5, and, p e to 0.8 for all synthetic graphs.
Table 6 describes hyperparameters for SuperGAT on four real-world datasets.
For other datasets and experiments, please see the code ( ./SuperGAT/args.yaml ).
A. CGAT I MPLEMENTATION
CGAT (Wang et al., 2019a) has two auxiliary losses: graph structure based constraint L g and class boundary constraint L b .
We borrow their notation for this section: V as a set of nodes, N i as a set of one-hop neighbors, N + i as a set of neighbors that share labels, N − i as a set of neighbors that do not share labels, ζ · as a margin between attention values, and φ ( v i , v k ) as unnormalized attention value.
[Equation]
Since label information is included in these two losses, they are difﬁcult to use in semi-supervised settings that provide few labeled samples.
In fact, in the CGAT paper, they conduct experiments in full-supervised settings; that is, they use all nodes in training except validation and test nodes.
So, we only use L g modiﬁed for semi-supervised learning.
[Equation]
With L c , the multi-class cross-entropy on node labels, CGAT’s optimization objective is
[Equation]
) and our modiﬁed CGAT’s loss is
[Equation]
In addition to losses, CGAT proposes top-k softmax and node importance based negative sampling (NINS).
Top-k softmax picks up nodes with top-k attention values among neighbors.
NINS adopts importance sampling when choosing negative sample nodes.
Since the code for CGAT has not been released, we implement our own version.
In all experiments in our paper, we use only the modiﬁed losses (Equation 14) and top-k softmax due to the training and implementation complexity of NINS.
For PPI, even if we do not assume a semi-supervised setting, we use the same loss because we could not accurately implement multi-label cases for Equationand 11 with only CGAT’s description.
To verify the functionality of our implementation, we report the results of a full-supervised setting with the original loss (Equation 13) like CGAT paper, in Table 7.
Our implementation of CGAT shows almost the same performance reported in the original paper.
In addition, SuperGAT and
CGAT showed almost similar performance in a full-supervised setting.
Note that the original paper employs two hidden layers with hidden dimensions as 32 for Cora, 64 for CiteSeer, and three hidden layers with hidden dimensions 32 for PubMed, where models in our experiments are all two-layer with 64 features.
A. W ALL - CLOCK T IME E XPERIMENTAL S ET - UP
To demonstrate our model’s efﬁciency, we measure the mean wall-clock time of the entire training process of three runs using a single GPU (GeForce GTX 1080Ti).
We compare our model with
GAT (Veli ˇ ckovi ´ c et al., 2018) and GAM (Stretcu et al., 2019).
GAT is the basic model using a simpler attention mechanism than ours, and GAM is the state-of-the-art model using co-training with the auxiliary model.
For GAT and SuperGAT, we use our implementation (including hyperparameter settings) in PyTorch (Paszke et al., 2019).
For GAM, we adopt the code in TensorFlow (Abadi et al., 2015) from the authors 5 and choose GCN + GAM model, which showed the best performance.
We retain the default settings in the code but use the hyperparameters reported in the paper, if possible.
With this setting, GCN + GAM on PubMed is not ﬁnished after 24 hours; therefore, we manually early-stop the training at the best accuracy.
B R ESULTS B. P ROOF OF P ROPOSITION
Proposition 2.
For l + 1 th GAT layer, if W and a are independent and identically drawn from zero-mean uniform distribution with variance σ 2 w and σ 2 a respectively, assuming that parameters are independent to input features h l and elements of h l are independent to each other,
[Equation]
Proof.
Let h (cid:48)
=
W h l , then h (cid:48)
i,k = (cid:80) F l +1 r =1 W kr h li,r .
Note that,
[Equation] [Equation]
First, we compute E ( a 2 ) and E ( h (cid:48) 2 ) .
[Equation] [Equation]
[Equation]
For the variance of e l +1 ij, GO ,
[Equation]
Now we compute E ( h (cid:48)
i h (cid:48) j ) and E ( h (cid:48) 2 i h (cid:48) 2 j ) ,
E
[Equation]
i
[Equation]
k
[Equation]
j
[Equation]
k ) =
E
[Equation]
+
[Equation]
(cid:17)(cid:17) (
[Equation]
=
[Equation] [Equation]
[Equation]
Note that for the zero-mean uniform distribution U ( − u, u ) with variance σ 2 w ,
[Equation]
[Equation]
For the variance of e l +1 ij, DP ,
[Equation]
[Equation] [Equation]
[Table]
B
E XPERIMENTS
In Table 8, we report all results of synthetic graph experiments.
We experiment on a total ofsynthetic graphs controlling 9 homophily (0.1, 0.2, ..., 0.9) and 16 average degree (1, 1.5, 2.5, 3.5, 5, 7.5, 10, 12.5, 15, 20, 25, 32.5, 40, 50, 75, 100).
[Figure]
[Table]
B. L ABEL - AGREEMENT S TUDY FOR O THER D ATASETS AND D EEPER M ODELS
In Figure
[Table]
we draw box plots of KL divergence between attention distribution and label agreement distribution for all nodes and layers of two-layer GATs and four-layer GATs.
As shown in the paper
[Table]
we can see that DP attention does not capture label-agreement rather than GO attention.
Also, the degree of this phenomenon becomes stronger as the layer goes down.
B. W ALL - CLOCK T IME R ESULT
In Table
[Table]
we report the mean wall-clock time (over three runs) of the training of GAT, GAM, and SuperGAT MX .
In SuperGAT, we ﬁnd that negative sampling of edges is the bottleneck of training.
So
[Table]
we additionally implement SuperGAT MX + MPNS, which employs multi-processing when sampling negative edges.
There are three observations in this experiment.
GCN + GAM is highly time-intensive in the training stage ( × 53 . 9 – × 328 . 1 versus GAT) for all datasets.
Compared to
GAT, our model needs × 2 .
7 more training time for Cora and × 7 . 2 for PubMed
[Table]
and we reduce the time by applying multi-processing to negative sampling ( × 1 . 8 for Cora
[Figure] [Figure]
[Figure]
T e s t
P e r f .
B. S ENSITIVITY A NALYSIS OF H YPER - PARAMETERS
We analyze sensitivity of mixing coefﬁcient of losses λ E , negative sampling ratio p n , and edge sampling ratio p e .
We plot mean node classiﬁcation performance (over 5 runs) against each hyperparameter in Figure 9, 10, and 11 respectively.
We use the best model for each dataset: SuperGAT MX for citation networks and SuperGAT SD for PPI.
For λ E , there is a speciﬁc range that maximizes test performance in all datasets.
Performance on
PPI is the largest when λ E is 10 − 3 , but the difference is relatively small comparing to others.
We observe that there is an optimal level of the edge supervision for each dataset, and using too large λ E degrades node classiﬁcation performance.
For p n , using too many negative samples has been shown to decrease performance.
The optimal number of negative samples is different for each dataset, and all are less than the number of positive samples ( p n < 1 . 0 ).
Note that as p n increases, the required GPU memory also increases.
When p n = 5 . 0 , the model and data for PPI could not be accommodated by one single GPU (GeForce
GTX 1080Ti).
When p e changes, the performance also changes, but the pattern is different by datasets.
For Cora and PubMed, the performance against p e shows the convex curve.
Performance for CiteSeer generally decreases as p e increases, but there are intervals the performance change of