C HARACTERIZE AND T RANSFER A TTENTION IN G RAPH N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Does attention matter and, if so, when and how?
Our study on both inductive and transductive learning suggests that datasets have a strong inﬂuence on the effects of attention in graph neural networks.
Independent of learning setting, task and attention variant, attention mostly degenerate to simple averaging for all three citation networks, whereas they behave strikingly different in the protein-protein interaction networks and molecular graphs: nodes attend to different neighbors per head and get more focused in deeper layers.
Consequently, attention distributions become telltale features of the datasets themselves.
We further explore the possibility of transferring attention for graph sparsiﬁcation and show that, when applicable, attention-based sparsiﬁcation retains enough information to obtain good performance while reducing computational and storage costs.
Finally, we point out several possible directions for further study and transfer of attention.
I NTRODUCTION
The modeling of graphs has become an active research topic in deep learning (Bronstein et al., 2017).
Dozens of neural network models have been developed for exploiting the structural information of graphs (Scarselli et al., 2009; Bruna et al., 2014; Henaff et al., 2015; Duvenaud et al., 2015; Niepert et al., 2016; Defferrard et al., 2016), now collectively referred to as graph neural networks (GNNs).
Built upon the success of attention in NLP (Vaswani et al., 2017), Veliˇckovi´c et al.
(2018) proposed the graph attention networks (GATs) to integrate multi-head self-attention into node feature update for adaptive weighting, with several extensions (Thekumparampil et al., 2018; Zhang et al., 2018;
Monti et al., 2018; Svoboda et al., 2019; Trivedi et al., 2019).
While the use of attention in GNNs is an attractive direction, several works also report that attention contributes little to the performance of
GNNs (Zhang et al., 2018; Shchur et al., 2019).
Considering the high computational cost of attention, the question is then that does attention help and, if so, when and how?
In this paper, we take a ﬁrst step towards the question.
We ﬁrst identify the key questions for understanding attention and propose an analytical paradigm.
With extensive experiments, our ﬁndings suggest that, although attention is motivated by inductive learning, its functionality depends highly on the characteristics of the datasets.
The attention distributions across heads and layers are near uniform for all citation networks ( Cora , Citeseer and Pubmed ) while they get more concentrated over layers on the protein-protein interaction networks ( PPI ) and molecular graphs, with signiﬁcant diversity among heads.
That the attention distribution is a telltale sign of the nature of graph class is further veriﬁed with a meta graph classiﬁcation experiment.
With attention features as inputs, citation networks are indistinguishable whereas PPI and molecule graphs are.
Inspired by these ﬁndings, we hypothesize that attention carry semantic meanings when they are non-uniform and can be helpful for transfer learning.
This has been the case in the NLP community (Radford et al., 2019), and is motivating many research efforts on understanding multi-head attention (Jain & Wallace, 2019; Clark et al., 2019; Voita et al., 2019).
We attempt the idea of attention based sparsiﬁcation – sparsifying a graph by retaining edges where attention are higher, with the intuition being that the resulting graph preserves enough information.
We ﬁnd that not only such attention-based sparsiﬁcation is transferable (meaning, it can work on unseen graphs), it also affords us to train a cheaper model without using attention to ﬁt the downstream task.
Finally, we discuss several possible fruitful directions for further exploration, including theory, interpretability, and unsupervised learning.
R ELATED WORK
Visualize and understand attention Several works attempted to visualize the learned attention by coloring edges or nodes based on the attention magnitudes (Veliˇckovi´c et al., 2018; Qiu et al., 2018).
Thekumparampil et al.
(2018) studied the averaged attention values between nodes with different or the same class labels.
Shanthamallu et al.
(2018) studied the attention GAT learned on two citation networks Cora and Citeseer with interquartile range metric and showed that they are near uniform.
Knyazev et al.
(2019) investigated the effectiveness of attentional pooling and ﬁnd that attention is only effective when it is close to optimal.
Our work differs in that we propose a general paradigm for analyzing graph attention, including how to characterize the overall attention statistics and how to measure the layer-wise and head-wise differences of the learned attention.
Transfer attention In the computer vision community, transferring the attention maps using a teacher-student network to improve the downstream tasks is a well-studied technique (Zagoruyko &
Komodakis, 2017; Li et al., 2019).
Our approach for transferring attention is different from these works in that we use the trained graph attention network as a graph sparsiﬁer instead of a teaching signal.
Yang et al.
(2018) proposed to transfer the relational structure within the data, which is represented as a set of attention weights, to boost the performance of other tasks.
Our transferring strategy is different from their work because we reduce the sparsity of the afﬁnity matrix by removing the entries with smaller attention weights.
Thus, we can train a cheap graph sparsiﬁer to accelerate the training and testing speed.
B ACKGROUND 3. G RAPH NEURAL NETWORKS
Let G be an undirected graph with node set V and edge set E , where each node i ∈ V has a feature
h
i ∈ R n 0 .
In a wide class of GNNs (Kipf & Welling, 2017; Hamilton et al., 2017; Veli ˇ ckovi ´ c et al., 2018), the basic feature update function for node i ∈ V at the l + 1 -th
GNN layer takes the form of h
[Equation]
where σ is an activation function, N ( i ) is a set containing i and its neighbors, α l +1 i,j ∈ R is the attention weight of edge ( j, i ) in updating the feature of node i , W l +1 ∈ R n l +1 × n l is the projection matrix, and h li , h l +1 i are corresponding node features after the l -th and the l + 1 -th layer.
With a sparse implementation, it has a time complexity of O ( |V| n l +1 n l + |E| n l +1 ) .
Graph Convolutional Network (GCN) (Kipf & Welling, 2017) and the mean variant of GraphSAGE (Hamilton et al., 2017) use static attention 1 √
|N ( i ) | 1 √
|N ( j ) | and 1 |N ( i ) | , which we separately refer to as GCN and uniform attention.
GAT (Veliˇckovi´c et al., 2018) uses a parameterized subnetwork to output the attention weights α i,j s.
Rather than using a single attention head as in Eqn.
equation 3.1, GAT aggregates the outputs of multiple heads:
[Equation]
, where k is the index of the attention head and K l +1 is the number of attention heads in the l + 1 -th layer.
Aggregate l +1 aggregates all head results in the l + 1 -th layer
and we follow the approach of
Veliˇckovi´c et al.
(2018) to use concatenation for intermediate layers and average for the ﬁnal layer.
Attention variants As in Luong et al.
(2015), there are multiple ways to calculate the attention scores.
In this paper, we focus on the following three types of attention, namely concat , dot product ,
[Table]
and general :
[Equation]
GATs uses the concat attention.
Zhang et al.
and
Ryu et al.
(2018) separately explores dot product and general attention in GNNs.
Also, LReLU ( · ) means the leaky ReLU activation.
Graph-level prediction Based on the node representations generated by GNNs, we can also compute a graph representation (Li et al., 2018) for graph-level prediction problems like graph classiﬁcation and regression:
[Equation]
where L is the number of GNN layers, h Lv is the representation of node v output by the last layer of
GNN, g ( · ) calculates the impact of node v on the graph representation, and f ( · ) is a linear projection.
3. T ASKS AND DATASETS
We consider the tasks of node classiﬁcation and graph-level prediction.
For modeling, we treat all graphs as undirected with untyped nodes and edges.
Self loops are added to preserve information from previous node features.
As in Veli ˇ ckovi ´ c et al.
(2018), we consider four datasets – citation networks
Cora , Citeseer (Sen et al., 2008), Pubmed (Namata et al., 2012) and PPI (Zitnik & Leskovec, 2017).
Additionally, we include two more datasets of molecular graphs for graph-level prediction.
The Harvard Clean Energy Project (CEP) (Hachmann et al., 2011) estimates the photovoltaic efﬁciency of organic molecules.
We use a subset of it pre-processed by Duvenaud et al.
; Ryu et al.
The HIV dataset was initially introduced by the Drug Therapeutics Program (DTP)
AIDS Antiviral
Screen HIV for testing the ability of compounds to inhibit HIV replication.
It was later included in the MoleculeNet benchmark (Wu et al., 2018) as a binary classiﬁcation task.
For both datasets, the node features are extracted based on DeepChem (Ramsundar et al., 2019) and RDKit (Landrum), which includes atom type, degree, and many other chemical properties.
Since GAT was partially motivated to work on unseen data, we consider two learning settings: transductive learning and inductive learning.
In the transductive learning setting, the model can access the features of all nodes in the graph.
However, only a fraction of the nodes are labeled in the training phase and the model is asked to predict the missing labels.
In the inductive learning setting, we have two mutually exclusive sets of nodes separately for training and testing.
The model is trained only on the features and labels of the nodes in the training set and is asked to predict the labels of the nodes in the testing set.
A summary of the tasks and learning settings can be found in Table 1.
We leave more detailed information like dataset statistics, training/testing split and features in Appendix A.
M ETHODOLOGY
The introduction of multi-head attention into multi-layer GNNs poses many interesting questions, we investigate ﬁve in this paper.
Q1 :
In the GAT model, all nodes have different attention distributions on their incoming edges.
How should we characterize the overall statistics of these learned attention distributions?
Q2 :
How do attention distributions differ across different heads and layers?
Q3 :
How does the choice of dataset, attention variant, and learning setting affect the learned attention?
Q4 :
[Table]
Is the statistics of the learned attention related to the intrinsic properties of the graph?
Q5 : How to transfer attention for further usage?
To answer Q1 , we propose multiple metrics for characterizing attention distributions.
For Q2 , we examine the metrics at different layers and compare the change of them over layers.
To answer Q3 , we run experiments to see how varying the dataset, attention variant and the learning setting impacts the learned attention.
Previous works (Kipf & Welling, 2017; Hamilton et al., 2017; Veliˇckovi´c et al., 2018) only perform transductive learning on the citation networks and inductive learning on PPI .
To ﬁll in the gap, we perform transductive learning on PPI and inductive learning on the citation networks (see data processing strategy in Appendix B.1).
We show that learning tasks are largely irrelevant.
To answer Q4 , we propose a new task called Meta Graph Classiﬁcation which asks the model to distinguish the type of the graphs by the characteristics of the attention distributions.
For
Q5 , we transfer attention for graph sparsiﬁcation and examine whether we can preserve enough task-related information with a signiﬁcant number of edges removed.
C HARACTERIZING ATTENTION 5. A NALYZING ATTENTION METRICS
Experiment settings Our attention study is completely based on the GAT architecture except that we try different types of attention mentioned in Section 3.1.
We follow the experiment settings of the original authors whenever possible and perform a hyperparameter search otherwise.
To make a fair comparison between attention variants, we use the same hidden size for each layer output across attention variants.
The detailed settings can be found in Appendix B.2.
Unless explicitly mentioned, we perform 100 random runs for Cora , Citeseer and Pubmed and 10 random runs for PPI , CEP and
HIV .
The test performance is mostly consistent across attention variants and is comparable to the original reported numbers, which we report in Appendix B.4.
Learned attention v.s. static attention Attention-free GNNs employ static weights in updating node features.
The ﬁrst question is then how does learned attention differ from static weights?
If the learned attention are almost the same as the static ones, then there is no point in performing costly attention computation and we do not need to proceed with the analysis.
For any static attention ( GCN or uniform ), we quantify the discrepancy between it and learned attention with [Equation] − α static i,j | .
With a range of [0 , 1] , the larger the value, the larger the discrepancy is.
Table 2 shows the discrepancy between the static and the learned attention for the ﬁrst head with the ‘ concat ’ variant.
Surprisingly, the discrepancy against the uniform attention is very small for the citation networks, suggesting that each node attends similarly to different neighbors.
Head-wise and layer-wise differences Figure 2 visualizes the attention of a node over its incoming edges in Cora and PPI , based on three heads in the last layer.
We can ﬁnd that different heads behave distinctively in the PPI case and they are all uniform in the Cora case.
We summarize the change of attention over heads and layers with several metrics.
To quantify the variance between two head-wise distributions, we compute the averaged L 1 norm of the difference between the mean distribution over all heads and the learned distribution for each head.
[Equation]
To probe the concentration of attention, we compute the maximum pairwise difference of them within one-hop neighborhoods.
To verify whether attention are on self loops when they are concentrated and GNNs degenerate to MLPs, we monitor the self-loop attention values.
We average the metrics over all nodes and heads for a layer in each run, and compute the mean and standard deviation of the averaged metrics in all runs.
Varying settings and attention variants We leverage the deﬁned metrics to compare the learned attention by different attention variants.
Figure 1 visualizes the head-wise and layer-wise metrics for Cora , Pubmed , PPI and CEP .
Independent of the attention variant, the learned attention change little across layers for Cora and Pubmed while the increasing max pairwise difference indicates that they get more concentrated with deeper layers for PPI and CEP .
Besides, the attention does not get increasingly more concentrated on self loops while getting sharper over layers.
We also experiment with different training settings on these graphs.
Figure 3 shows the learned attention when training inductively on Cora while training transductively on PPI .
We observe the similar phenomenon of the learned attention, which eliminates the effect of training settings.
5. M ETA GRAPH CLASSIFICATION
Previous experiments suggest that the attention learned are highly graph-dependent and their characteristics can be predicted with proper knowledge of graph semantics.
To verify it, we attempt to infer the graph types based on the attention learned.
Speciﬁcally, we perform graph classiﬁcation with attention-based features.
[Table]
Synthetic dataset We construct a synthetic dataset for graph classiﬁcation with two steps.
First, we collect 480 samples/subgraphs of 20 to 30 nodes from each dataset.
For HIV and CEP, we choose 480 graphs and construct a balanced subset.
For the rest four datasets, we sample 480 graphs for each using random walk as in the case of inductive learning on citation networks.
Second, we separately train a 2-layer GAT for each collected dataset and compute the attention metrics for each layer.
The mean and standard deviation of the metrics are then used as the graph features.
We leave more details in Appendix B.5.
We train a logistic regression classiﬁer for graph classiﬁcation, where 20% of the graphs are used for training and the rest are used for testing.
We have separately experimented with attention metrics from all layers, the ﬁrst layer, and the second layer.
The classiﬁcation performance is reported in
Table 3, with all experiments repeated for 10 times.
In the experiments, we ﬁnd that more than 80% of the incorrect classiﬁcations happen within citation subgraphs.
This is also veriﬁed by our t-SNE (van der Maaten & Hinton, 2008; Pedregosa et al., 2011) visualization of the attention metrics in Figure 4.
We can see that the attention metrics of citation networks tend to be indistinguishable while the attention metrics of other datasets are better separated and clustered.
A TTENTION - BASED GRAPH SPARSIFICATION 6. PPI SPARSIFICATION FOR GAT PREDICTION
We consider two intuitive heuristics for attention based sparsiﬁcation: 1) local top-k sparsiﬁcation selects up to k incoming edges for each node with highest attention values; 2) global threshold sparsiﬁcation selects edges whose attention value exceeds a pre-speciﬁed threshold over the entire graph.
Across all GNN layers, we perform attention-based sparsiﬁcation for each attention head to get a subset of edges.
We then take the union of the edge subsets to get the edge set for the sparsiﬁed graph.
All self loops are selected to preserve the information of original node features.
Inspired by recent research on sampling based training of GNNs (Hamilton et al., 2017; Huang et al., 2018), we consider two baseline random sparsiﬁcation for comparison: 1) uniform neighbor sparsiﬁcation uniformly samples at most k incoming edges for each node without replacement and we compare it against local top-k sparsiﬁcation; 2) uniform graph-wise sparsiﬁcation uniformly
The result of training on unsparsiﬁed graphs is included for reference.
For local top-k sparsiﬁcation we consider 1 ≤ k ≤ 8 .
For global threshold sparsiﬁcation, we consider threshold in { 0 . 9 , 0 . 8 , 0 . 7 , 0 . 6 , 0 . 5 , 0 .
We ﬁrst experiment attention-based graph sparsiﬁcation with PPI , which tends to have sharp attentionbased on our previous study and is relatively dense ( 56944 nodes and 1644208 edges with self loops and bi-directional edges added).
Despite similar sharp attention, molecules are not good candidates for the experiment as they are already sparse and on average each atom only has less than three neighbors.
As illustrated in ﬁgure 6, after training a GAT on PPI, we perform attention-based sparsiﬁcation and re-train a GAT on the sparsiﬁed training and validation graphs.
Figure 5 compares the results of attention-based sparsiﬁcation with concat variant against random sparsiﬁcation.
With a similar degree of sparsiﬁcation, the attention-based sparsiﬁcation consistently performs better than the baseline in terms of test metric and its variance across runs; 2) We can reach a test accuracy comparable to the original result with only 40% ∼ 50% edges left in the training and validation graphs.
6. S PARSIFICATION WITH GENTLE ATTENTION
What if the attention are neither uniform nor sharp?
Our previous study shows that the attention in
Pubmed are not completely uniform in the last GAT layer with dot product and general attention variants
so we use it as a testbed for the study.
In cases where attention are not very sharp, threshold sparsiﬁcation is not very useful and we consider only top-k sparsiﬁcation.
Table 4 compares top-k sparsiﬁcation with dot product attention against uniform neighbor sparsiﬁcation.
With similar proportion of edges left in the training and validation graphs, the top-k sparsiﬁcation consistently outperforms uniform neighbor sparsiﬁcation and achieves a performance comparable to that of training on the raw graphs.
[Table]
[Table]
L IGHT GAT SPARSIFICATION WITH G RAPH SAGE PREDICTION
For practical usage of speeding up the computation [Table] we do not want to train two large GATs from scratch.
We propose to ﬁrst quickly train a light GAT with a lot fewer parameters for graph sparsiﬁcation and then train a large attention-free GNN for prediction on the sparsiﬁed graphs.
We explore this idea by varying the hidden sizes of attention heads in GAT and use a GraphSAGE for prediction on PPI .
The GraphSAGE has 3 layers and each layer has a hidden size of 512 .
We consider the mean aggregator for it with skip connection added.
The results are summarized in Table [Table]
Our experiments show that a GraphSAGE model can achieve a good performance with training on sparsiﬁed graphs as long as we sparsify test graphs.
Surprisingly, while a smaller hidden size in
GAT does harm its own prediction performance, it has little effect on capturing important edges for
GraphSAGE prediction even with top-1 sparsiﬁcation.
We note that the number of parameters in
GraphSage and the baseline GAT is 1.2M and 3.6M, respectively, i.e. a 3-fold reduction.
C ONCLUSIONS AND D ISCUSSIONS
In this work, we propose an analytical paradigm that can summarize the characteristics of multi-head attention learned over graphs and compare them against topology-based static attention.
This allows a deeper understanding of attention beyond simply comparing the performance of models and motivates further use of attention like transfer learning.
In addition to the attention-based sparsiﬁcation we explored, we believe below are several interesting directions that are underexplored:
• Theory.
Many efforts have performed to theoretically understand and explain GNNs, particularly their connection to kernel methods and Weisfeiler-Lehman tests (Morris et al., 2019; Xu et al., 2019; Maron et al., 2019), but few of them have considered attention.
• Interpretability.
The use of attention can add interpretability.
This is particularly valued in risk-sensitive scenarios like medicine.
Several efforts have been made in the chemistry community (Ryu et al., 2018; Preuer et al., 2019; Xiong et al., 2019).
• Unsupervised learning.
Unsupervised representation learning is an important approach when the training of a model is expensive and the labeled data is scarce.
This is particularly the case for biological networks and molecular graphs due to the need of wet-lab experiments.
The NLP community has witnessed the success of unsupervised learning with attention (Radford et al., 2019) and we might expect the same for GNNs.
Weihua Hu (2019) demonstrates the effectiveness of training GNNs with unsupervised learning for chemistry and biology, but did not employ attention.
A D ATASETS A. D ATASET SUMMARY
Table 6 and 7 summarize the statistics about the raw graph datasets.
When computing the number of edges and node degrees, we have not considered self loops.
Also when we model directed graphs as undirected graphs, the number of edges will get doubled.
For transductive learning, the number of edges is considered to be the same for training/validation/test as all edges may be involved in message passing.
[Table]
[Table]
(80.0 % ) # train nodes 44906 (78.9 % ) 497311 (60.0 % ) 846823 (79.1 % ) # train edges 613184 (77.3 % ) 600365 (60.0 % ) 906666 (78.7 % ) # val graphs 5995 (20.0 % ) 4191 (10.0 % )
# val nodes 6514 (11.4 % ) 165916 (20.0 % ) 117216 (11.0 % )
# val edges 99460 (12.5 % ) 200269 (20.0 % ) 128751 (11.2 % ) # test graphs 5997 (20.0 % ) 4192 (10.0 % ) # test nodes 5524 (9.7 % ) 165908 (20.0 % ) 105929 (9.9 % ) # test edges 80988 (10.2 % ) 200318 (20.0 % ) 116525 (10.1 % )
A. A DDITIONAL DETAILS
Graph Construction For citation networks, the nodes correspond to documents and the edges correspond to citations between pairs of documents.
For PPI , the nodes represent proteins and the edges represent physical interactions between them.
For molecules, the nodes correspond to atoms and the edges correspond to chemical bonds.
Node featurization and labeling For citation networks, nodes have bag-of-words features and labels for the topic of documents.
For PPI , the node features include positional gene sets, motif gene sets and immunological signatures and the node labels are gene ontology sets (Hamilton et al., 2017), collected from the Molecular Signatures Database (Subramanian et al., 2005).
For the CEP dataset, the node features consist of one hot encodings of atom type, node degree, the total number of hydrogens attached to it, the number of implicit hydrogens attached to it, and its aromaticity indicator.
For the HIV dataset, in addition to those features we also consider the formal charge of the atom, the number of radical electrons of the atom and the atom’s hybridization.
Dataset splits For Cora, Citeseer, Pubmed, and PPI, we consider a deterministic dataset split for training, validation and test.
For CEP, we randomly split the dataset in each run, where approximately the proportion of graphs used for training, validation and test is separately 60% , 20% and 20% .
The statistics of the CEP dataset in table 7 is obtained in one random run.
For the HIV dataset, we use the scaffold split (Wu et al., 2018; Li et al., 2017), which structurally separates molecules into training, validation and test subsets and poses a greater challenge for generalization.
Imbalanced
dataset The HIV dataset is highly imbalanced, with only 1487 compounds are positive, constitute approximately 3 .
5% of the dataset.
B E XPERIMENT SETTINGS B. V ARYING LEARNING SETTING FOR CITATION NETWORKS AND PPI Transductive Learning on PPI
To perform transductive learning on PPI , we sample two mutually exclusive subsets of the nodes as the training set and validation set for each graph, leaving the rest as the test set.
We experiment on two splitting settings.
In the ﬁrst setting, we sample about 5% nodes for training and 18% nodes for validation, similar to the splitting ratio of the transductive learning setting on Cora .
In the second setting, we sample 79% nodes for training and 11% nodes for validation, similar to the case of inductive learning on PPI .
Inductive Learning on Citation Networks To perform inductive learning on citation networks, we ﬁrst sample 120 graphs of 100 nodes for each dataset.
We use a random walk based sampling algorithm described in Algorithm 1, which by the study of (Leskovec & Faloutsos, 2006) performs best in preserving the properties of static graphs.
Separately, 60% , 20% , 20% of the graphs are used for training, validation and test.
B. H YPERPARAMETERS FOR A TTENTION S TUDY
For the attention study, we consider the hyperparameters below:
• Transductive learning on Cora and Citeseer: 2 -layer GAT with 8 heads in the ﬁrst layer and 1 head in the second layer, 8 hidden units for each head in the ﬁrst layer, a dropout of 0 .
6 , no residual connection, a learning rate of 0 .
005 , L2 regularization with coefﬁcient 0 . 0005 , cross entropy loss • Transductive learning on Pubmed: 2 -layer GAT with 8 heads in both layers, 8 hidden units for each head in the ﬁrst layer, a dropout of 0 .
6 , no residual connection, a learning rate of 0 .
01 , L2 regularization with coefﬁcient 0 .
001 , cross entropy loss • Inductive/Transductive learning on PPI:
no L2 regularization, no dropout, 3 -layer GAT with residual connections added for the last two layers; a learning rate of 0 .
005 for concat attention and a learning rate of 0 . 0001 for other attention variants; the number of attention
Require:
G =
( V , E ) the original graph, g size = 100 the target subgraph size 1: step =2: start ∼ Unif ( V ) (cid:46)
Uniformly choose a starting node.
3: V sub = { start } , E sub = { ( start, start ) } 4: src
= start 5: while |V sub | < g size and step < 100 ∗ g size do 6: step = step +7: back ∼ Bernoulli (0 . 15) , (cid:46) Return to the starting point with probability 0 .
8: if back then 9: src = start 10: else 11: dst ∼ Unif ( { j | ( src, j ) ∈ E} ) 12:
V sub =
V sub (cid:83) { dst } 13:
E sub =
E sub (cid:83) { ( dst, dst ) , ( src, dst ) , ( dst, src ) } 14: src = dst 15: Return ( V sub , E sub )
heads in the three layers is separately 8 , 8 , 6 for general attention and 4 , 4 , 6 for other attention variants; 128 hidden units for each head in the ﬁrst two layers for general attention and 256 hidden units for each head in other attention variants 1 ; we use a batch size 1 for general attention and a batch size 2 for other attention variants; binary cross entropy loss • Inductive learning on Cora, Citeseer, Pubmed: batch size 24 , 3 -layer GAT with separately 4 , 4 , 6 attention heads, residual connection is added for the last two layers, 8 hidden units per head for the ﬁrst two layers, a learning rate of 0 .
005 , a dropout of 0 . 6 ; L2 regularization with coefﬁcient 0 .
001 for pubmed and 0 .
0005 for the rest two citation networks; cross entropy loss • CEP: a batch size of 512 , 3-layer GAT where each layer has 4 heads and each head hashidden units, a dropout of 0 . 0 , residual connection is added for the last two layers, a learning rate of 0 .
001 , no L2 regularization; smooth L1 loss • HIV: a batch size of 64 , 2-layer GAT where each layer has 4 heads and each head hashidden units, a dropout of 0 . 0 , residual connection is added for the last two layers, no L regularization, an initial learning rate of 0 . 0005 , a decay of learning rate by 0 .
99 after each epoch; weighted focal loss with − ( wy (1 − p )
γ log p + (1 − y )
p γ log (1 − p )) ,
where γ =and w = # negative samples / # positive samples
An early stop is performed if the validation score hasn’t been improved for 100 epochs.
B. G RAPH - LEVEL PREDICTION
Based on node features updated with a GNN, we can also perform a graph-level prediction.
First, a graph representation can be obtained with:
[Equation]
where L is the number of GNN layers, g : R n L → R and f : R n L → R n G are two linear layers with bias added.
In all cases we consider n G = 128 .
A graph-level prediction is then computed with a 3 -layer MLP where all hidden sizes are equal to n G and a ReLU activation is applied after each of the ﬁrst two linear layers.
[Table]
T EST PERFORMANCE ACROSS ATTENTION VARIANTS
We evaluate test performance using different metrics for different datasets – accuracy for Cora ,
Citeseer , Pubmed , micro-averaged F1 score for PPI , mean absolute error for CEP and roc auc score for HIV .
See table 8 for a summary of the prediction performance, where different attention variants mostly have similar performance.
The reference numbers are from Veli
ˇ ckovi ´ c et al.
(2018) unless stated otherwise.
For Cora , Citeseer , Pubmed and PPI , we include the original results of GATs for reference.
For the rest datasets, we include the best performance of previous work for reference whenever applicable, but some models do not involve attention mechanism.
B. G RAPH CLASSIFICATION
Dataset construction 1)
As PPI has multiple graphs, we sample the same number of subgraphs from each original graph. 2) For CEP dataset, we ﬁrst sort the whole dataset based on the photovoltaic efﬁciency and split it into 96 buckets.
We sample 5 graphs from each bucket where 3 graphs are used for training, one graph is used for validation and one graph is used for test.
3) As the HIV dataset is intrinsically imbalanced, we construct a balanced subset by sampling a same number of positive samples and negative samples.
Also since we are considering a special dataset split, the training, validation and test subset is separately constructed from the training, validation and test set.
For
Cora, Citeseer, Pubmed and PPI, we construct a whole dataset of subgraphs ﬁrst and then perform a random split to get the training, validation and test set with a splitting ratio of 60% : 20% : 20% .
Graph feature extraction For graph feature extraction, we train a 2 -layer GAT with 8 heads for each layer and 64 hidden units for each head.
We use a batch size of 16 and perform an early stop if the validation score no longer improves for 10 epochs.
We use the same loss functions for each dataset as explained in B.2.
We also perform a hyperparameter search for learning rate, dropout, L2 regularization coefﬁcient λ and whether to perform a residual connection.
The selected hyperparameters are as follows:
• Cora: a dropout of 0 . 1 , residual connection is added for the second layer, no L2 regularization, a learning rate of 0 .
01 for concat attention and a learning rate of 0 .
005 for other attention variants • Citeseer: a dropout of 0 . 1 , residual connection is added for the second layer, a learning rate of 0.01, no L2 regularization • Pubmed: a dropout of 0 . 1 , residual connection is added for the second layer, no L2 regularization, a learning rate of 0 .
005 for general attention and a learning rate of 0 .
01 for other attention variants • PPI: no dropout, residual connection is added for the second layer, no L2 regularization, a learning rate of 0 .
005 for concat attention and a learning rate of 0 .
01 for other attention variants
• CEP: no dropout, a learning rate of 0 .
005 ; a residual connection is added for the second layer only with general attention; L2 regularization with coefﬁcient 0 .
001 is used except for concat attention • HIV: a residual connection is added for the second layer, no L2 regularization is used; a dropout of 0 .
6 is used only for concat attention; a learning rate of 0 .
005 is used for dot product attention and a learning rate of 0 .
01 is used for the rest attention variants
t-SNE visualization of attention metrics
In the text we included the t-SNE visualization of attention based features for concat attention only.
Here we include the results for all three variants for comparison in ﬁgure 7, 8 and 9.
Across all attention variants, we observe a similar pattern that the attention metrics for citation subgraphs get blurred while those for the rest datasets are better separated and clustered.