Learnable Graph Convolutional Attention Networks
Abstract
Existing Graph Neural Networks (GNNs) compute the message exchange between nodes by either aggregating uniformly ( convolving ) the features of all the neighboring nodes, or by applying a non-uniform score ( attending ) to the features.
Recent works have shown the strengths and weaknesses of the resulting GNN architectures, respectively, GCNs and GATs.
In this work, we aim at exploiting the strengths of both approaches to their full extent.
To this end, we first introduce the graph convolutional attention layer (CAT), which relies on convolutions to compute the attention scores.
Unfortunately, as in the case of GCNs and GATs, we show that there exists no clear winner between the three—neither theoretically nor in practice—as their performance directly depends on the nature of the data (i.e., of the graph and features).
This result brings us to the main contribution of our work, the learnable graph convolutional attention network (L-CAT): a GNN architecture that automatically interpolates between GCN, GAT and CAT in each layer, by adding two scalar parameters.
Our results demonstrate that L-CAT is able to efficiently combine different GNN layers along the network, outperforming competing methods in a wide range of datasets, and resulting in a more robust model that reduces the need of cross-validating.
Introduction
In recent years, Graph Neural Networks (GNNs) (Scarselli et al., 2008) have become ubiquitous in machine learning, emerging as the standard approach in many settings.
For example, they have been successfully applied for tasks such as topic prediction in citation networks (Sen et al., 2008); molecule prediction (Gilmer et al., 2017); and link prediction in recommender systems (Wu et al., 2020a).
These applications typically make use of message-passing
GNNs (Gilmer et al., 2017), whose idea is fairly simple: in each layer, nodes are updated by aggregating the information (messages) coming from their neighboring nodes.
Depending on how this aggregation is implemented, we can define different types of GNN layers.
Two important and widely adopted layers are graph convolutional networks (GCNs)
(Kipf & Welling, 2017), which uniformly average the neighboring information; and graph attention networks (GATs) (Velickovic et al., 2018), which instead perform a weighted average, based on an attention score between receiver and sender nodes.
More recently, a number of works have shown the strengths and limitations of both approaches from a theoretical (Fountoulakis et al., 2022; Baranwal et al., 2021; 2022), and empirical (Knyazev et al., 2019) point of view.
These results show that their performance depends on the nature of the data at hand (i.e., the graph and the features), thus the standard approach is to select between GCNs and
GATs via computationally demanding cross-validation.
In this work, we aim to exploit the benefits of both convolution and attention operations in the design of GNN architectures.
To this end, we first introduce a novel graph convolutional attention layer (CAT), which extends existing attention layers by taking the convolved features as inputs of the score function.
Following (Fountoulakis et al., 2022), we rely on a contextual stochastic block model to theoretically compare GCN, GAT, and CAT architectures.
Our analysis shows that, unfortunately, no free lunch exists among these three
GNN architectures since their performance, as expected, is fully data-dependent.
This motivates the main contribution of the paper, the learnable graph convolutional attention network (L-CAT): a novel GNN which, in each layer, automatically interpolates between the three operations by introducing only two scalar parameters.
As a result, L-CAT is able to learn the proper operation to apply at each layer, thus combining different layer types in the same GNN architecture while overcoming the need to cross-validate—a process that was prohibitively expensive prior to this work.
Our extensive empirical analysis demonstrates the capabilities of L-CAT on a wide range of datasets, outperforming existing baseline GNNs in terms of both performance, and robustness to input noise and network initialization.
Preliminaries
Assume as input an undirected graph G =
( V, E ),
where V =
[ n ] denotes the set of vertices of the graph, and E
⊆
V × V the set of edges.
Each node i ∈ [ n ] is represented by a d -dimensional feature vector X i ∈ R d , and the goal is to produce a set of predictions { ˆ y i } n i =1 .
To this end, a message-passing GNN layer yields a representation ˜ h i ∈ R d ′ for each node i , by collecting and aggregating the information from each of its neighbors into a single message; and using the aggregated message to update its representation from the previous layer, h i ∈ R d .
For the purposes of this work, we can define this operation as the following:
[Equation]
where N ∗ i is the set of neighbors of node i (including i ), W v ∈ R d ′ ×
d a learnable matrix, f an elementwise function, and γ ij ∈ [0 , 1] are coefficients such that (cid:80) j γ ij = 1 for each node i .
Let the input features be h
i
=
X i , and h Li
= ˆ y i the predictions, then we can readily define a message-passing GNN (Gilmer et al., 2017) as a sequence of L layers as defined above.
Depending on the way the coefficients γ ij are computed, we identify different GNN flavors.
Graph convolutional networks (GCNs) (Kipf & Welling, 2017) are simple yet effective.
In short, GCNs compute the average of the messages, i.e., they assign the same coefficient γ ij = 1 / | N ∗ i | to every neighbor:
[Equation]
Graph attention networks take a different approach.
Instead of assigning a fixed value to each coefficient γ ij , they compute it as a function of the sender and receiver nodes.
A general formulation for these models can be written as follows:
[Equation]
Here, Ψ( h i , h j ) def = α ( W q h i , W k h j ) is known as the score function (or attention architecture ), and provides a score value between the messages
h i and h j (or more generally, between a learnable mapping of the messages).
From these scores, the (attention) coefficients are obtained by normalizing them, such that (cid:80) j γ ij = 1.
We can find in the literature different attention layers and, throughout this work, we focus on the original GAT (Velickovic et al., 2018) and its extension GATv2 (Brody et al., 2022):
[Equation]
where the learnable parameters are now the attention vector a ; and the matrices W q , W k , and W v .
Following previous work (Velickovic et al., 2018; Brody et al., 2022), we assume that these matrices are coupled, i.e., W q = W k = W v .
Note that the difference between the two layers lies in the position of the vector a : by taking it out of the nonlinearity, Brody et al.
(2022) increased the expressiveness of GATv2.
Now, the product of a and a weight matrix does not collapse into another vector.
More importantly, the addition of two different attention layers will help us show the versatility of the proposed models later in § 6.
Previous work
In recent years, there has been a surge of research in GNNs.
Here, we discuss other GNN models, attention mechanisms, and the recent findings on the limitations of GCNs and GATs.
The literature on GNNs is extensive (Wu et al., 2020b; Hamilton et al., 2017a; Battaglia et al., 2018; Lee et al., 2019), and more abstract definitions of a message-passing GNN are possible, leading to other lines of work trying different ways to compute messages, aggregate them, or update the final message (Hamilton et al., 2017b; Xu et al., 2019; Corso et al., 2020).
Alternatively, another line of work fully abandons message-passing, working instead with higher-order interactions (Morris et al., 2019).
While some of this work is orthogonal—or directly applicable—to the proposed model, in the main paper we focus on convolutional and attention graph layers, as they are the most widely used (and cited) as of today.
While we consider the original GAT (Velickovic et al., 2018) and GATv2 (Brody et al., 2022), our work can be directly applied to any attention model that sticks to the formulation in
Eq. 3.
For example, some works propose different metrics for the score function, like the dot-product (Brody et al., 2022), cosine similarity (Thekumparampil et al., 2018), or a combination of various functions (Kim & Oh, 2021).
Other works introduce transformerbased mechanisms (Vaswani et al., 2017) based on positional encoding (Dwivedi & Bresson, 2020; Kreuzer et al., 2021) or on the set transformer (Wang et al., 2021a).
Finally, there also exist attention approaches designed for specific type of graphs, such as relational (Yun et al., 2019; Busbridge et al., 2019) or heterogeneous graphs (Wang et al., 2019b;
Hu et al., 2020b).
3. On the limitations of GCN and GAT networks
Baranwal et al.
(2021) studied classification on a simple stochastic block model, showing that, when the graph is neither too sparse nor noisy, applying one layer of graph convolution increases the regime in which the data is linearly separable.
However, this result is highly sensitive to the graph structure, as convolutions essentially collapse the data to the same value in the presence of enough noise.
More recently, Fountoulakis et al.
(2022) showed that
GAT is able to remedy the above issue, and provides perfect node separability regardless of the noise level in the graph.
However, a classical argument (see Anderson (2003)) states that in this particular setting a linear classifier already achieves perfect separability.
These works, in summary, showed scenarios for which GCNs can be beneficial in the absence of noise, and that GAT can outperform GCNs in other scenarios, leaving open the question of which architecture is preferable in terms of performance.
Convolved attention: benefits and hurdles
In this section, we propose to combine attention with convolution operations.
To motivate it, we complement the results of Fountoulakis et al.
(2022), providing a synthetic dataset for which any 1-layer GCN fails, but 1-layer GAT does not.
Thus, proving a clear distinction between GAT and GCN layers.
Besides, we show that convolution helps GAT as long as the graph noise is reasonable.
The proofs for the two statements in this section appear in
Appendix A and follow similar arguments as in Fountoulakis et al.
This dataset is based on the contextual stochastic block model ( CSBM )
(Deshpande et al., 2018).
Let ε 1 , . . .
, ε n be iid.
uniform samples from {− 1 , 0 , 1 } .
Let C k
= { j ∈ [ n ] | ε j = k } for k ∈ {− 1 , 0 , 1 } .
We set the feature vector X i ∼ N ( ε i µ , I · σ 2 ) where µ ∈ R d , σ ∈ R , and I ∈ { 0 , 1 } d × d is the identity matrix.
For a given pair p, q ∈ [0 , 1] we consider the stochastic adjacency matrix A ∈ { 0 , 1 } n × n defined as follows: for i, j ∈ [ n ] in the same class ( intra-edge ), we set a ij ∼ Ber( p ); 1 for i, j in different classes ( inter-edge ), we set a ij ∼ Ber( q ).
We denote
Ber the by ( X , A ) ∼ CSBM ( n, p, q, µ , σ 2 ) a sample obtained according to the above random process.
Our task is then to distinguish (or separate) nodes from C 0 vs. C − 1 ∪ C 1 .
Note that, in general, it is impossible to separate C 0 from C − 1 ∪ C 1 with a linear classifier and, using one convolutional layer is detrimental for node classification on the CSBM:although the convolution brings the means closer and shrinks the variance, the geometric structure of the problem does not change.
On the other hand, we prove that GAT is able to achieve perfect node separability when the graph is not too sparse:
Theorem 1.
Suppose that p, q = Ω(log 2 n/n ) and ∥ µ ∥ 2 = ω ( σ √ log n ).
Then, there exists a choice of attention architecture Ψ such that, with probability at least 1 − o n (1) over the data ( X , A ) ∼ CSBM ( n, p, q, µ , σ 2 ), GAT separates nodes C 0 from C 1 ∪ C − 1 .
Moreover, we show using methods from Baranwal et al.
(2021), that the above classification threshold ∥ µ ∥ can be improved when the graph noise is reasonable.
Specifically, by applying convolution prior to the attention score , the variance of the data is greatly reduced, and if the graph is not too noisy, the operation dramatically lowers the bound in Thm.
We exploit this insight by introducing the graph convolutional attention layer (CAT): Ψ( h i ,
[Equation]
j )
[Equation]
α
[Equation]
, W ˜
[Equation]
) where
[Equation]
where ˜ h i are the convolved features of the neighborhood of node i .
As we show now, CAT improves over GAT by combining convolutions with attention, when the graph noise is low.
Corollary 2.
Suppose p, q = Ω(log 2 n/n ) and ∥ µ ∥ ≥ ω (cid:16) σ (cid:113) ( p +2 q )log n n ( p − q ) 2 (cid:17) .
Then, there is a choice of attention architecture Ψ such that CAT separates nodes C 0 from C 1 ∪ C − 1 , with probability at least 1 − o (1) over the data ( X , A ) ∼ CSBM ( n, p, q, µ , σ 2 ).
The above proposition shows that under the CSBM data model, convolving prior to attention changes the regime for perfect node separability by a factor of | p − q | (cid:112) n/ ( p + 2 q ).
This is desirable when | p − q | (cid:112) n/ ( p + 2 q ) > 1, since the regime for perfect classification is increased.
Otherwise, applying convolution prior to attention reduces the regime for perfect separability.
Therefore, it is not always clear whether convolving prior to attention is beneficial.
L-CAT: Learning to interpolate
From the previous analysis, we can conclude that it is hard to know a priori whether attention, convolution, or convolved attention, will perform the best.
In this section, we argue that this issue can be easily overcome by learning to interpolate between the three.
First, note that GCN and GAT only differ in that GCN weighs all neighbors equally (Eq. 2) and, the more similar the attention scores are (Eq. 3), the more uniform the coefficients γ ij are.
Thus, we can interpolate between GCN and GAT by introducing a learnable parameter.
Similarly, the formulation of GAT (Eq. 3) and CAT (Eq. 6) differ in the convolution within the score, which can be interpolated with another learnable parameter.
Following this observation, we propose the learnable convolutional attention layer (L-CAT), which can be formulated as an attention layer with the following score:
[Equation]
where λ 1 , λ 2 ∈ [0 , 1].
As mentioned before, this formulation lets L-CAT learn to interpolate between GCN ( λ 1 = 0), GAT ( λ 1 = 1 and λ 2 = 0), and CAT ( λ 1 = 1 and λ 2 = 1).
L-CAT enables a number of non-trivial benefits.
Not only can it switch between existing layers, but it also learns the amount of attention necessary for each use-case.
Moreover, by
||
[Figure]
||
[Figure] [Figure]
GAT
CAT
[Figure] [Figure]
cc u r
[Figure]
cy ( % )
|| µ
|| = 0.
[Figure]
0.4 q   || µ ||
= 4.
10 −10|| µ ||   q = 0. 10 −10|| µ ||   q = 0.
comprising the three layers in a single learnable formulation, it removes the necessity of cross-validating the type of layer, as their performance is data-dependent (see §§ 3.1 and 4).
Remarkably, it allows to easily combine different layer types within the same architecture.
While we focus on GCN, L-CAT can be easily used with other GNN architectures such as
PNA (Corso et al., 2020) and GCNII (Chen et al., 2020), as L-CAT interpolates between two different adjacency matrices.
For further details and results, refer to App.
F.
Experiments
In this section, we first validate our theoretical findings on synthetic data ( § 6.1).
Then, we show through various node classification tasks that (L-)CAT is as competitive as the baseline models ( § 6.2).
Lastly, we move to more demanding scenarios from the Open Graph
Benchmark (Hu et al., 2020a), demonstrating that L-CAT is a more flexible and robust alternative to its baseline methods ( § 6.3) that reduces the need of cross-validating without giving up on performance.
Refer to Apps.
B to F for details and additional results.
The code to reproduce the experiments can be found at https://github . com/psanch21/LCAT .
6. Synthetic data
First, we empirically validate our theoretical results (Thm. 1 and Cor. 2).
We aim to better understand the behavior of each layer as the properties of the data change, i.e., the noise level q (proportion of inter-edges) and the distance between the means of consecutive classes ∥ µ ∥ .
We provide extra results and additional experiments in App.
B.
Experimental setup.
As data model, we use the proposed CSBM (see § 4) with n = 10000, p = 0 .
5, σ = 0 .
1, and d = n/ (cid:0) 5 log 2 ( n ) (cid:1) .
All results are averaged over 50 runs, and parameters are set as described in App.
A.
We conduct two experiments to assess the sensitivity to structural noise.
First, we vary the noise level q between 0 and 0 . 5, leaving the mean vector µ fixed.
We test two values of ∥ µ ∥ : the first corresponds to the easy regime ( ∥ µ ∥ = 10 σ √ 2 log n ) where classes are far apart; and the second correspond to the hard regime ( ∥ µ ∥ = σ ) where clusters are close.
In the second experiment we instead sweep ∥ µ ∥ in the range (cid:2) σ/ 20 , 20 σ √ 2 log n (cid:3) , covering the transition from hard (small ∥ µ ∥ ) to easy (large ∥ µ ∥ ) settings.
Here, we fix q to 0 .
1 (low noise) and 0 .
3 (high noise).
In both cases, we compare the behavior of 1-layer GAT and CAT, and include GCN as the baseline.
Results.
The two left-most plots of Fig.
1 show node classification performance for the hard and easy regimes, respectively, as we vary the noise level q .
In the hard regime, we observe that GAT is unable to achieve separation for any value of q , whereas CAT achieves perfect classification when q is small enough.
This exemplifies the advantage of CAT over GAT as stated in Cor. 2.
When the distance between the means is large enough, we see that GAT achieves perfect results independently of q , as stated in Thm. 1.
In contrast, when CAT fails to satisfy the condition in Cor. 2 (as we increase q ), it achieves inferior performance.
The right-most part of Fig.
1 shows the results when we fix q and sweep ∥ µ ∥ .
In these two plots, we can appreciate the transition in the accuracy of both GAT and CAT as a function of ∥ µ ∥ .
We observe that GAT achieves perfect accuracy when the distance between the means satisfies the condition in Thm.
1 (solid vertical line in Fig. 1).
Moreover, we can see
[Table]
the improvement CAT obtains over GAT.
Indeed, when ∥ µ ∥ satisfies the conditions of Cor.(dashed vertical line in Fig. 1)
[Table]
the classification threshold is improved.
As we increase q we see that the gap between the two vertical lines decreases, which means that the improvement of CAT over GAT decreases as q increments, exactly as stated in Cor. 2.
6. Real data
We study now the performance of the proposed models in a comprehensive set of real-world experiments, in order to gain further insights of the settings in which they excel.
Specifically, we found CAT and L-CAT to outperform their baselines as the average node degree increases.
For a detailed description of the datasets and additional results, refer to Apps.
C, D and F.
Models.
We consider as baselines a simple GCN layer (Kipf & Welling, 2017), the original
GAT layer (Velickovic et al., 2018) and its recent extension, GATv2 (Brody et al., 2022).
Based on the two attention models, we consider their CAT and L-CAT extensions.
To ensure fair comparisons, all layers use the same number of parameters and implementation.
Datasets.
We consider six node classification datasets.
The Facebook / GitHub / TwitchEN datasets involve social-network graphs (Rozemberczki et al., 2021), whose nodes represent verified pages/developers/streamers; and where the task is to predict the topic/expertise/explicitlanguage-use of the node.
The Coauthor Physics dataset (Shchur et al., 2018) represents a co-authorship network whose nodes represent authors, and the task is to infer their main research field.
The Amazon datasets represent two product-similarity graphs (Shchur et al., 2018), where each node is a product, and the task is to infer its category.
Experimental setup.
To ensure the best results, we cross-validate all optimization-related hyperparameters for each model using GraphGym (You et al., 2020).
All models use four
GNN layers with hidden size of 32, and thus have an equal number of parameters.
For evaluation, we take the best-validation configuration during training, and report test-set performance.
For further details, refer to App.
D.
Results are presented in Table 1.
In contrast with § 6.1, we here find GCN to be a strong contender, reinforcing its viability in real-world data despite its simplicity.
We observe both CAT and L-CAT not only holding up the performance with respect to their baselines models for all datasets, but in most cases also improving the test accuracy in a statistically significant manner.
These results validate the effectiveness of CAT as a GNN layer, and show the viability of L-CAT as a drop-in replacement , achieving good results on all datasets.
[Figure]
As explained in § 4, CAT differs from a usual GAT in that the score is computed with respect to the convolved features.
Intuitively, this means that CAT should excel in those settings where nodes are better connected, allowing CAT to extract more information from their neighborhoods.
Indeed, in the inset figure we can observe the improvement in accuracy of
CAT with respect to its baseline model, as a function of the average node degree of the dataset, and the linear regression fit of these results (dashed line).
This plot includes all datasets
[Table]
(from the manuscript and App. D)
[Table]
and shows a positive trend between node connectivity and improved performance achieved by CAT.
6. Open Graph Benchmark
In this section
[Table]
we assess the robustness of the proposed models, in order to fully understand their benefits.
For further details and additional results, refer to App.
E.
Datasets.
We consider four datasets from the OGB suite (Hu et al., 2020a): proteins , products , arxiv
[Table]
and mag .
Note that these datasets are significantly larger than those from § 6.2 and correspond to more difficult tasks, e.g., arxiv is a 40-class classification problem (see Table 5 in App.
C for details).
This makes them more suitable for the proposed analysis.
Experimental setup.
We adopt the same experimental setup as Brody et al.
(2022) for the proteins , products , and mag datasets.
For the arxiv dataset, we use instead the example code from OGB (Hu et al., 2020a), as it yields better performance than that of Brody et al.
Just as in § 6.2, we compare with GCN (Kipf & Welling, 2017), GAT (Velickovic et al., 2018), GATv2 (Brody et al., 2022), and their CAT and L-CAT counterparts.
We cross-validate the number of heads (1 and 8), and select the best-validation models during training.
All models are identical except for their λ values.
Results are summarized in Table 2.
Here we do not observe a clear preferred baseline: GCN performs really well in proteins and mag ; GAT excels in products ; and GATv2 does well in arxiv and mag .
While CAT obtains the best results on arxiv , its performance on proteins and products is significantly worse than the baseline model.
Presumably, an excessive amount of inter-edges could explain why convolving the features prior to computing the score is harmful, as seen in § 6.1.
As we explore in § 6.3.2, however, CAT improves over its baseline for most proteins scenarios, specially with a single head.
In stark contrast, L-CAT performs remarkably well, improving the baseline models in all datasets but products —even on those in which CAT fails—demonstrating the adaptability of L-CAT to different scenarios.
To better understand the training dynamics of the models, we plot in Fig.
2a the test accuracy of GCN and the GATv2 models during training on the arxiv dataset.
Interestingly, despite all models obtaining similar final results, CATv2 and L-CATv2 drastically improved their convergence speed and stability with respect to GATv2 , matching that of GCN.
To understand the behavior of L-CATv2, Fig.
2b shows the evolution of the λ parameters.
We observe that, to achieve these results, L-CATv2 converged to a GNN network that combines three types of layers: the first layer is a CATv2 layer, taking advantage of the neighboring information; the second layer is a quasi-GCN layer, in which scores are almost uniform and some neighboring information is still used in the score computation; and the third layer is a pure GCN layer, in which all scores are uniformly distributed.
It is important to remark that these dynamics are fairly consistent, as L-CATv2 reached the same λ values over all five runs.
[Figure]
6.3. Robustness to noise
One intrinsic aspect of real world data is the existence of noise.
In this section
[Figure]
we explore the robustness of the proposed models to different levels of noise
[Figure]
i e., we attempt to simulate scenarios where there exist measurement inaccuracies in the input features and edges
[Figure]
Experimental setup.
We consider the arxiv dataset, and the same experimental setup as in § 6.3.
We conduct two experiments.
First, we introduce to the node features additive noise of the form N ( 0 , 1 σ ), and consider different levels of noise, σ ∈ { 0 , 0 . 25 , 0 . 5 , 0 .
Then, as in (Brody et al., 2022), we simulate edge noise by adding fake edges with probability Bern ( p ) for p ∈ { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } .
Results are shown in the inset figures for feature (top) and edge noise (bottom), summarizing the performance of all models over five runs and two numbers of heads (1 and 8).
Baseline attention models are quite sensitive to feature noise, but are more robust to edge noise, as they can drop interclass edges (see § 3.1).
GCNs, as expected, are instead more robust to feature noise, but suffer more in the presence of edge noise.
In concordance with the synthetic experiments (see §§ 4 and 6.1), CAT is able to leverage convolutions as a variance-reduction technique, reducing the variance and improving its robustness to feature noise.
Remarkably, LCAT proves to be the most robust for both types of noise: by adapting the amount of attention used in each layer, it outperforms existing methods and reduces the variance.
6.3. Robustness to network initialization
Another important aspect for real-world applications is robustness to network initialization, i.e., the ability to obtain satisfying performance independently of the initial parameters.
Otherwise, a practitioner can waste lots of resources trying initilizations or, even worse, give up on a model just because they did not try the initial parameters that yield great results.
Experimental setup.
We follow once again the same setup for proteins as in § 6.3.
We consider two different network initializations.
The first one, uniform , uses uniform Glorot initilization (Glorot & Bengio, 2010) with a gain of 1, which is the standard initialization used throughout this work.
The second one, normal , uses instead normal Glorot initialization (Glorot & Bengio, 2010) with a gain of √ 2.
This is the initialization employed on the original GATv2 paper (Brody et al., 2022) exclusively for the proteins dataset.
Results —segregated by number of heads—are shown in Table 3, while the results for GCN appear in the inset table.
These results show that the baseline models perform poorly on the
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
uniform initialization.
However, this is somewhat alleviated when using 8 heads in the attention models.
Moreover, all baselines significantly improve with normal initialization, being
GCN the best model, and attention models obtaining 79 % accuracy on average with 8 heads.
[Figure]
[Table]
[Figure]
[Table]
Compared to the baselines, CAT does a good job and improves the performance in all cases except for normal with 8 heads.
Remarkably, L-CAT consistently obtains high accuracy in all scenarios and runs.
To emphasize consistency, bottom row shows the average accuracy across runs, showing that L-CAT is clearly more robust to parameter initialization than competing models.
To understand this performance, we inspect the distribution of λ 1 , λ 2 for L-CAT in Fig.
Here, we can spot a few interesting patterns.
Consistently, the first and last layers are always
GCNs, while the inner layers progressively admit less attention.
Second, the number of heads affects the amount of attention allowed in the network; the more heads, the more expressive the layer tends to be, and more attention is permitted.
Third, L-CAT adapts to the initialization used: in uniform , it allows more attention in the second layer; in normal , it allows more attention in the score inputs.
These results consolidate the flexibility of L-CAT.
Conclusions and future work
In this work, we studied how to combine the strengths of convolution and attention layers in
GNNs.
We proposed CAT, which computes attention with respect to the convolved features, and analyzed its benefits and limitations on a new synthetic dataset.
This analysis revealed different regimes where one model is preferred over the others, reinforcing the idea that selecting between GCNs, GATs, and now CATs, is a difficult task.
For this reason, we proposed L-CAT, a model which interpolates between the three via two learnable parameters.
Extensive experimental results demonstrated the effectiveness of L-CAT, yielding great results while being more robust than other methods.
As a result, L-CAT proved to be a viable drop-in replacement that removes the need to cross-validate the layer type.
We strongly believe learnable interpolation can get us a long way, and we hope L-CAT to motivate new and exciting work.
Specially, we are eager to see L-CAT in real applications, and thus finding out what combining different GNN layers across a model (without the hurdle of cross-validating all layer combinations) can lead to in the real-world.
Ethic statement
Given the nature of this work, we do not see any direct ethical concerns.
On the contrary,
L-CAT eases the application of GNNs to the practitioner, and removes the need of crossvalidating the layer type, which can potentially benefit other areas and applications, as
GNNs have already proven.
Reproducibility statement
For the theoretical results, we describe the data model used in § 4, and provide all the detailed proofs in App.
A.
For the experimental results, we include in the supplementary material the necessary code and scripts required to reproduce our experiments, and all required datasets are freely available.
Complete details about the experimental setup can be found in Apps.
B,
D and E, and we report in our results the mean and standard deviation computed using five trials or more.
In addition, we highlight in bold the results that are statistically significant.
Moreover, we include details of computational resources used for the all sets of experiments in App.
B, App.
D and App.
E.
Acknowledgements
We would like to thank Batuhan Koyuncu, Jonas Klesen, Miriam Rateike, and Maryam
Meghdadi for helpful feedback and discussions. Pablo S´anchez Mart´ın thanks the German
Research Foundation through the Cluster of Excellence “Machine Learning – New Perspectives for Science”, EXC 2064/1, project number 390727645 for generous funding support.
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRSIS) for supporting Pablo S´anchez Mart´ın.
Appendix Table of Contents A Theoretical results
A.1 A hard example for GCN . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 A solution for GAT and CAT . . . . . . . . . . . . . . . . . . . . . . . .
B
Synthetic experiments
B.1 Other experiments . .
C Dataset description
D Real data experiments
D.1 Experimental details . .
D.2 Additional results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Open Graph Benchmark experiments
E.1 Experimental details . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Additional results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F Extending L-CAT to other GNN models
F.1 PNA experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.2 GCNII experiments . .
A Theoretical results A. A hard example for GCN
In this subsection, we present a dataset and classification task for which GCN performs poorly.
Note that we follow the similar techniques and notation as (Fountoulakis et al., 2022), as described in the main paper.
We recall our data model.
Fix n, d ∈ N and let ε 1 , . .
. , ε n be i.i.d uniformly sampled from {− 1 , 0 , 1 } .
Let C k
= { j ∈ [ n ] | ε j = k } for k ∈ {− 1 , 0 , 1 } .
For each index i ∈ [ n ], we set the feature vector X i ∈ R d as X i ∼ N ( ε i · µ , I · σ 2 ), where µ ∈ R d , σ ∈ R and I ∈ { 0 , 1 } d × d is the identity matrix.
For a given pair p, q ∈ [0 , 1] we consider the stochastic adjacency matrix
A ∈ { 0 , 1 } n × n defined as follows.
For i, j ∈ [ n ] in the same class, we set a ij ∼ Ber( p ), and if i, j are in different classes, we set a ij ∼ Ber( q ).
We let D ∈ R n × n be a diagonal matrix containing the degrees of the vertices.
We denote by ( X , A ) ∼ CSBM ( n, p, q, µ , σ 2 ) a sample obtained according to the above random process.
The task we wish to solve is classifying C 0 vs C − 1 ∪ C 1 .
Namely, we want our model φ to satisfy φ ( X i ) <
if and only if i ∈ C 0 .
Moreover, note that the posed problem is not linearly classifiable .
To this end, we start by stating an assumption on the choice of parameters.
This assumption is necessary to achieve degree concentration in the graph.
Assumption 1.
p, q = Ω(log 2 n/n ) .
We now show the distribution of the convolved features.
The following lemma can be easily obtained using the techniques in Baranwal et al.
Lemma 3.
Fix p, q satisfying Assumption 1.
With probability at least 1 − o (1) over A and { ε i } i ,
[Equation]
To prove the above lemma
[Equation]
we need the following definition of our high probability event.
Definition 1.
We define the even E as the intersection of the following events over A and { ε i } i :
E 1 is the event that | C 0 | = n 3 ± O ( √ n log n ), | C 1 | = n 3 ± O ( √ n log n ) and | C − 1 | = n 3 ± O ( √ n log n ).
E 2 is the event that for each i ∈ [ n ], D ii = n ( p +2 q )
[Equation]
E 3 is the event that for each i ∈ [ n ] and k ∈ {− 1 , 0 , 1 } ,
[Equation]
The following lemma is a direct application of Chernoff bound and a union bound.
Lemma 4.
With probability at least 1 − 1 / poly( n ) the event E holds.
Proof of Lemma 3.
By applying Lemma 4, and conditioned on E , for any i ∈ [ n ]
[Equation]
Using the definition of E and properties of Gaussian distributions the lemma follows.
Lemma 3 shows that essentially, the convolution reduced the variance and moved the means closer, but the structure of the problem stayed exactly the same.
Therefore, one layer of
GCN cannot separate C 0 from C − 1 ∪ C 1 with high probability.
A. A solution for GAT and CAT
In what follows, we show that GAT is able to handle the above classification task easily when the distance between the means is large enough.
Then, we show how the additional convolution on the inputs to the score function improves the regime of perfect classification when the graph is not too noisy.
Our main technical lemma considers a specific attention architecture and characterize the attention scores for our data model
[Equation]
Lemma 5.
Suppose that p, q satisfy Assumption 1, ∥ µ ∥ ≥ ω (cid:0) σ √ log n (cid:1) , fix the LeakyRelu constant β ∈ (0 , 1) and R ∈ R .
Then, there exists a choice of attention architecture Ψ such that with probability at least 1 − o n (1) over the data ( X , A ) ∼ CSBM ( n, p, q, µ , σ 2 ) the following holds.
[Equation]
Proof.
We consider as an ansatz the following two layer architecture Ψ.
[Equation]
where R > 0 is an arbitrary scaling parameter.
The output of the attention model is defined as
[Equation]
Let ∆ ij def = S · (cid:20) ˜
w T X i ˜ w T X j (cid:21) + b ∈ R 8 , and note that for each element t ∈ [8] of ∆ ij , we have that ( ∆ ij ) t =
S t, 1 ˜ w T X i + S t, 2 ˜ w T X
j + b t .
Note that the random variable ( ∆ ij ) t is distributed as follows:
[Equation]
Therefore, for a fixed i, j ∈ [ n ] 2 we have that the entries of ∆ ij are distributed as follows (where we use N yx as abbreviation for the Gaussian N ( x, y ))
[Equation]
N
[Equation]
for i,
[Equation]
C 21 ,
[Equation]
for i
[Equation]
j ∈ C
[Equation]
N
[Equation]
N
[Equation]
N
[Equation]
N
[Equation]
for i, j ∈ C
[Equation]
× C
[Equation] [Equation]
N
[Equation]
N
[Equation]
N
[Equation]
for i, j
[Equation]
C
[Equation]
C
[Equation]
for
[Equation]
N
[Equation]
N
[Equation]
N σ− ∥ µ ∥
[Equation]
N σ
[Equation]
− ∥ µ ∥
[Equation]
N σ
[Equation]
− ∥ µ ∥
[Equation]
(cid:105) for i, j ∈ C 20 ,
Next, we will use the following lemma regarding LeakyRelu concentration.
Lemma 6 (Lemma A.6 in Fountoulakis et al.
Fix s ∈ N , and let z 1 , . .
, z s be jointly Gaussian random variables with marginals
z i ∼ N ( µ i , σ 2 i ).
There exists an absolute constant C > 0 such that with probability at least 1 − o s (1), we have
[Equation]
for all
i
[Equation] [Equation]
Using Lemma 6 with the assumption on ∥ µ ∥ and a union bound, we have that with probability at least 1 − o n (1), LeakyRelu( ∆ ij ) is (up to 1 ±
o (1))
[Equation]
for i, j ∈ C 21 ,
[Equation]
for i, j ∈ C
[Equation] [Equation]
−
[Equation]
−
[Equation]
−
[Equation]
−
[Equation]
(cid:105) for i, j ∈ C 1 × C − 1 ,
[Equation]
−
[Equation]
−
[Equation]
−
[Equation]
−
[Equation]
−
[Equation]
∥
[Equation]
2 (cid:105) for i, j ∈ C − 1 × C 1 ,
[Equation]
−
[Equation]
−
[Equation]
for i, j ∈ C
×
C 1 ,
[Equation]
for i, j ∈ C
×
C
[Equation] [Equation]
−
[Equation]
for i, j ∈ C 1 × C 0 ,
[Equation]
−
[Equation]
for i, j ∈ C −
[Equation]
× C 0 ,
[Equation]
−
[Equation]
for i, j ∈ C 20 .
Then,
[Equation]
 
[Equation]
if
[Equation] [Equation]
C
[Equation]
) if
[Equation] [Equation]
C 2 −−
[Equation]
if i ∈ C 1 , j ∈ C −10 Rβ ∥ µ ∥ (1 ± o (1)) if i ∈ C − 1 , j ∈ C−
R 2 ∥ µ ∥ (1 − 21 β )(1 ± o (1)) if i ∈ C 0 , j ∈ C−
R 2 ∥ µ ∥ (1 − 11 β )(1 ± o (1)) if i ∈ C 0 , j ∈ C −−
R 2 ∥ µ ∥ (1 − 5 β )(1 ± o (1)) if i ∈ C 1 , j ∈ C−
R 2 ∥ µ ∥ (1 − 5 β )(1 ± o (1)) if i ∈ C − 1 , j ∈ C2 Rβ ∥ µ ∥ (1 ± o (1)) if i, j ∈ C, and the proof is complete.
Next we will define our high probability event.
Definition 2.
E ′ def = E ∩ E ∗ , where E ∗ is the event that for a fixed w ∈ R d , all i ∈ [ n ] satisfy | w T X
[Equation]
≤ 10 σ
[Equation]
√ log n .
The following lemma is obtained by using Lemma 4 with standard Gaussian concentration and a union bound.
Lemma 7.
With probability at least 1 − 1 / poly( n ) event E ′ holds.
Corollary 8.
Suppose that p, q satisfy Assumption 1, ∥ µ ∥ = ω ( σ √ log n ) and fix R ∈ R .
Then, there exists a choice of attention architecture Ψ such that with probability 1 − o n (1) over ( A , X ) ∼ CSBM ( n, p, q, µ , σ 2 ) it holds that γ ij =
  3 np (1 ± o (1)) if i, j ∈ C 20 ∪ C 21nq (1 ± o (1)) if i, j ∈ C − 1 × C 1nq exp( − Θ( R ∥ µ ∥ )) if i, j ∈ C − 1 × C − 1 ∪ C 0np exp( − Θ( R ∥ µ ∥ )) otherwise , where R is a parameter of the architecture.
Proof.
The proof is immediate.
First applying the ansatz from Lemma 5 with β < 1 / 25,
Lemma 7 and a union bound.
Using the definition of γ ij concludes the proof.
Next, we prove Thm. 1 that the model distinguish nodes from C 0 for any choice of p, q satisfying Assumption 1.
We restate the theorem for convince.
Theorem 9 (Formal restatement of Thm. 1) .
Suppose that p, q satisfy Assumption 1 and ∥ µ ∥ 2 = ω ( σ √ log n ).
Then, there exists a choice of attention architecture Ψ such that with probability at least 1 − o n (1) over the data ( X , A ) ∼ CSBM ( n, p, q, µ , σ 2 ), the estimator ˆ x i def = (cid:88) j ∈ N i γ ij ˜ w T X j + b where ˜ w = µ / ∥ µ ∥ , b = −∥ µ ∥ /satisfies
ˆ x i <
if and only if i ∈ C 0 .
Proof.
Let Ψ be the architecture from Cor. 8 and let R satisfy R ∥ µ ∥ 2 = ω (1).
We will compute the mean and variance of the estimator ˆ x i conditioned on E ′ .
Suppose that i ∈ C 0 .
By using Cor. 8, Definition 2 and our assumption on ∥ µ ∥ and R , we have
[Equation]
and therefore
[Equation]
By similar reasoning we have that for i ∈ C − 1 ∪ C 1 , E (cid:2)
ˆ x i | E ′
[Equation]
∥
[Equation]
Next, we claim that for each i ∈ [ n ] the random variable ˆ x i given E ′ is sub-Gaussian with a small sub-Gaussian constant compared to the above expectation.
The following lemma is a straightforward adaptation of Lemma A.11 in Fountoulakis et al.
, and we provide its proof for completeness.
Lemma 10.
Conditioned on E ′ , the random variables { ˆ x i } i are sub-Gaussian with parameter ˜ σ 2 i
= O (cid:16)
σ 2 np (cid:17) if i ∈ C 0 ∪ C 1 and ˜ σ 2 i
= O (cid:16) σ 2 nq (cid:17) otherwise.
Proof.
Fix i ∈ [ n ], and write X i =
ε i µ + σ g
i where g i ∼ N (0 , I d ), and ε i denotes the class membership.
Consider ˆ x i as a function of g =
[ g 1 ◦ g 2 ◦ · · · ◦ g n ] ∈ R nd , where ◦ denotes vertical concatenation.
Namely, consider the function
[Equation]
Since g ∼ N (0 , I nd ), proving that ˆ x i given E ′ is sub-Gaussian for each i ∈ [ n ], reduces to showing that the function f i : R nd → R is Lipschitz over E ⊆ R nd defined by E ′ and the relation X
i =
ε i µ + σ g
i .
That is, E def = (cid:8) g ∈ R nd (cid:12)(cid:12) | ˜ w T g i | ≤ 10 √ log n, ∀ i ∈ [ n ] (cid:9) .
Specifically, we show that conditioning on the event E ′ (which restricts g ∈ E ), the Lipschitz constant
L f i of f i satisfies L f i
= O (cid:16) σ √ np (cid:17) for i ∈ C 0 ∪ C 1 and L f i
= O (cid:16) σnq (cid:17) otherwise, and hence proving the claim.
First note that event E ′ induce a transformation which transforms isotropic Gaussians to truncated Gaussians vectors.
Similarly to Fountoulakis et al.
(2022), we can show that this transformation can be obtained by a push-forward mapping whose Lipschitz constant is 1.
[Equation]
(8) where τ ( x ) def = Φ − 1 ((1 − 2 c )Φ( x ) + c ) for c = Φ( − 10 √ log n ).
To compute the Lipschitz constant of f i ( g ) for i ∈ [ n ], let us denote X
=
[ X 1 ◦ X 2 ◦ · · · ◦ X n ] and consider the function
[Equation]
Let us assume without loss of generality that i ∈ C 0 (the cases for i ∈ C 1 and i ∈ C − 1 are obtained identically).
Conditioning on the event E ′ , which imposes the restriction that
X ∈ ˜ E where
[Equation]
Conditioning on E ′ (which restricts X , X ′ ∈ ˜ E ), using Cor. 8 and recalling that R satisfies
R ∥ µ ∥ 2 = ω (1), we get
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
This shows the Lipschitz constant of ˜ f
i ( X ) over ˜ E satisfies L ˜ f
i = O (cid:16) 1 √ np (cid:17) .
On the other hand, by viewing X as a function of g , it is straightforward to see that the function h ( g ) : R nd → R nd defined by h ( g ) def =
X ( g ) has Lipschitz constant L h = σ , as
[Equation]
Therefore, since f i ( g ) = ˜ f
i ( h ( g )) and g ∈ E
if and only if X ∈ ˜
E , we have that, conditioning on E ′ , the function ˆ x i
= f i ( g ) is Lipschitz continuous with Lipschitz constant L f i =
L ˜ f
i L h =
O (cid:16) σ √ np (cid:17) .
Since g ∼ N (0 , I nd ), we know that ˆ x i is sub-Gaussian with sub-Gaussian constant ˜ σ 2 =
L 2 f i
=
O (cid:16) σ 2 np (cid:17) .
The following lemma will be used for bounding the misclassification probability.
Lemma 11 (Rigollet & H¨utter (2015)) .
Let x 1 , . . .
, x n be sub-Gaussian random variables with the same mean and sub-Gaussian parameter ˜ σ 2 .
Then,
[Equation]
Moreover, for any t >
[Equation]
We bound the probability of misclassification
[Equation]
for t < | E [ˆ x i ] | = ∥ µ ∥ 2 2 (1 ± o (1)).
By Lemma 10, picking t = Θ (cid:16) σ (cid:112) log | C 0 | (cid:17) and applying
Lemma 11 implies that the above probability is 1 / poly( n ).
Similarly for class C 1 ∪ C − 1 we have that the misclassification probability is
Pr
[Equation]
for t <
E [ˆ x i ].
Picking t =
Θ (cid:16) σ (cid:112) log | C 1 ∪ C − 1 | (cid:17) and applying Lemma 11 and a union bound over the misclassification probabilities of both classes conclude the proof of the corollary.
Combining Thm.
9 with Lemma 3, we immediately get Cor.
2 which we restate below.
Corollary 12.
Suppose p, q = Ω(log 2 n/n ) and ∥ µ ∥ ≥ ω (cid:16) σ (cid:113) (
[Equation]
n n ( p − q ) 2 (cid:17) .
Then, there is a choice of attention architecture Ψ such that, with probability at least 1 − o (1) over the data ( X , A ) ∼ CSBM ( n, p, q, µ , σ 2 ), CAT separates nodes C 0 from C 1 ∪ C − 1 .
B Synthetic experiments
In this section, we present the complete results for the synthetic data experiments of § 6.1.
First, we describe the parameterization we use for the 1-layer GCN, GAT, and CAT models; then, we verify the behavior of the normalized score function ( γ ij ) matches that of the theory presented in Cor. 8.
In particular, we visualize the average of the following three groups of gammas (Fig. 4):
• Gammas γ ij included in i, j ∈ C 20 ∪ C 21 .
Solid lines.
• Gammas γ ij included in i, j ∈ C − 1 × C 1 .
Dashed lines.
• The rest of gammas.
Dotted lines.
For completeness, we also include the empirical results that validate Thm. 1 and Cor. 2, which were discussed already in § 6.1.
Experimental setup.
We assume the following parametrization for the 1-layer GCN, GAT, and CAT:
[Equation]
where N i are the set of neighbors of node i , X j are the features of node j —obtained from the CSBM described in § 4, and h ′
i are the logits of the prediction of node i .
Note that for
GCN we have γ ij = 1 | N ∗ i | .
Otherwise, we consider the following parameterization of the score function Ψ:
[Equation]
where
[Equation]
For these experiments, we define the parameters ˜ w , S , b and r as in the proofs in App.
A:
[Equation]
∥ ,
[Equation] [Equation]
where R > 0 and C > 0 are arbitrary scaling parameters.
Both C and R and input to the score function are set different for each of the models, as indicated in Table 4.
In particular, we set R = 7 ∥ µ ∥ for both GAT and CAT such that: i) all γ ij are distinguishable as we decrease ∥ µ ∥ ; and ii) we avoid numerical instabilities in the implementation when computing the exponential of R × ∥ µ ∥ in order to obtain γ ij (see Cor.
8), as the exponential of small or large values leads to under/overflow issues.
As for C , we set C = 1 for GAT and
C = ( p − q ) / ( p + 2 q ) for CAT such that we counteract the fact that the distance between classes shrink as we increase q , see Lemma 3:
Regarding the data model, we set (as described in § 6.1) n = 10000, p = 0 .
5, σ = 0 .
1, and d = n/ (cid:0) 5 log 2 ( n ) (cid:1) .
We set the slope of the LeakyReLU activation to β = 1 / 5 for the GAT and β = 0 .
01 for CAT, such that the proof of Cor.
8 is valid.
As described in the main paper, to assess the sensitivity to structural noise, we present the complete results for two sets of experiments.
First, we vary the noise level q between 0 and 0 . 5, fixing the mean vector µ .
We test two values of ∥ µ ∥ : the first corresponds to the easy regime ( ∥ µ ∥ = 10 σ √ 2 log n ≈ 4 . 3) where classes are far apart; and the second correspond to the hard regime (
[Equation]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
where the distance between the clusters is small.
In the second experiment we modify instead the distance between the means, sweeping ∥ µ ∥ in the range (cid:2) σ/ 20 , 20 σ √ 2 log n (cid:3) which corresponds to [0 . 005 , 8 . 58], and thus covering the transition from the hard setting (small ∥ µ ∥ ) to the easy one (large ∥ µ ∥ ).
In these experiments, we fix q to 0 .
1 (low noise) and 0 .
3 (high noise).
Results are summarized in Fig.
The top row contains the node classification performance for each of the models (i.e., Fig. 1), the next two rows contain the γ ij values for GAT and
CAT respectively.
The two left-most columns of Fig.
4 show the results for the hard and easy regimes, respectively, as we vary the noise level q .
In the hard regime, we observe that GAT is unable to achieve separation for any value of q , whereas CAT achieves perfect classification when q is small enough.
The gamma plots help shed some light on this question.
For GAT, we observe that the gammas represented with the dotted and solid lines collapse for any value of q (see middle plot), while this does not happen for CAT when the noise level is low (see bottom plot).
This exemplifies the advantage of CAT over GAT as stated in
Cor. 2.
When the distance between the means is large enough, we see that GAT achieves perfect results independently of q , as stated in Thm. 1.
We also observe that, in this case,
[Figure]
the gammas represented with the dotted and solid lines do not collapse for any value of q .
In contrast, as we increase q , CAT fails to satisfy the condition in Cor. 2, and therefore achieves inferior performance.
We note that the low performance is due to the fact that all gammas collapse to the same value for large noise levels.
For the second set of experiments (two right-most columns of Fig. 1), where we fix q and sweep ∥ µ ∥ , we observe that, for both values of q , there exists a transition in the accuracy of both GAT and CAT as a function of ∥ µ ∥ .
As shown in the main manuscript, GAT achieves perfect accuracy when the distance between the means satisfies the condition in Thm.(solid vertical line in Fig. 1).
Moreover, we can see the improvement CAT obtains over
GAT.
Indeed, when ∥ µ ∥ satisfies the conditions of Cor. 2 (dashed vertical line in Fig. 1), the classification threshold is improved.
As we increase q , we see that the gap between the two vertical lines decreases, which means that the improvement decreases as q increments, exactly as stated in Cor. 2.
This transition from the hard regime to the easy regime is also observed in the gamma plots: we observe the largest difference in value between the different groups of lambdas for values of ∥ µ ∥ that satisfy the condition in Thm.
1 (that is to the right of the vertical lines).
B. Other experiments
In the following, we extend the results for the synthetic data presented above.
In particular, we aim to evaluate if L-CAT is able to achieve top performance regardless of the scenario.
That is, we want to evaluate if L-CAT consistently performs at least as good as the bestperforming model.
We change the fixed-parameter setting of the previous section and, instead, we evaluate the performance of GCN, GAT, CAT and L-CAT when we learn the model-dependent parameters.
Experimental setup.
We assume the same parametrization for the 1-layer GCN, GAT and CAT described in Eq.
9 and Eq.
For L-CAT, we add the parameters λ 1 and λ 2 , as indicated in Eq.
We fix the parameters shared among the models, that is, ˜ w , S , b , r , and
R , with the values indicated in Eq.
Different from previous experiments, we now learn C and, for L-CAT, we also learn λ 1 and λ 2 .
We choose to fix part of the parameters (instead of learning them all) to keep the problem as similar as possible to the theoretical analysis we provided in § 4 and App.
A.
If we instead learn all the parameters, it takes a single dimension of the features to be (close to) linearly separable to find a solution that achieves a similar performance regardless of the model, which hinders the analysis.
This is a consequence of the probabilistic nature of the features.
One way of solving this issue would be to make n big enough.
Instead, we opt to have a fixed n and reduce the degrees of freedom of the models by fixing the parameters shared across all models.
The rest of the experimental setup matches the one from App.
B. Additionally, we use the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0 . 05, and we train for 100.
Results are summarized in Fig.
The top row contains the node classification performance for every model, while the bottom row contains the learned values of λ 1 (solid line) and λ 2 (dashed line) with L-CAT.
The two left-most columns of Fig.
5 show the results for the hard and easy regimes, respectively, as we vary the noise level q .
In the hard regime, we see rather noisy results.
Still, the behaviour is similar to that of Fig.
4: the performance of CAT degrades as we increase q .
We also observe that, on average, CAT outperforms GAT.
In this case, we observe that L-CAT achieves similar performance as CAT, which can be explained by inspecting the learned values of lambda in the bottom row.
We observe that λ 1 = 1 and λ 2 ≥ 0 .
5 on average for all values of q .
This indicates that L-CAT is closer to CAT than to
GAT.
When the distance between the means is large enough (i.e., ∥ µ ∥ = 4 . 3), we see that
GAT achieves perfect results independently of q while the performance of CAT deteriorates with large values of q , the same trend as in Fig. 4.
Remarkably, we observe that L-CAT also achieves perfect results independently of q .
If we inspect the lambda values, we first see that λ
= 1 for all q , thus the interpolation happens between CAT and GAT.
Looking at the values of λ 2 , we observe that, for small values of q , λ 2 is pretty noisy, which is expected since any solution achieves perfect performance.
Interestingly, we have that λ
=
0 for large values of q , with negligible variance.
This indicates that L-CAT learns that it must behave like GAT in order to perform well.
For the second set of experiments (two right-most columns of Fig. 5), we fix q and sweep ∥ µ ∥ like we did in Fig.
Here, we observe a similar trend: for both values of q , there exists a transition in the accuracy of both GAT and CAT as a function of ∥ µ ∥ .
Yet once again, we observe that L-CAT consistently achieves a similar performance to the best-performing model in every situation.
C Dataset description
We present further details about the datasets used in our experiments, summarized in Table 5.
All datasets are undirected (or transformed to undirected otherwise) and transductive.
The upper rows of the table refer to datasets used in § 6.2 taken from the PyTorch Geometric framework.
The following paragraphs present a short description of such datasets.
Amazon Computers & Photos are datasets taken from Shchur et al.
(2018), in which nodes represent products, and edges indicate that the products are usually bought together.
The node features are a Bag of Words (BoW) representation of the product reviews.
The task is to predict the category of the products.
GitHub is a dataset introduced in Rozemberczki et al.
(2021), in which nodes correspond to developers, and edges indicate mutual follow relationship.
Node features are embeddings extracted from the developer’s starred repositories and profile information (e.g., location or employer).
The task is to infer whether a node relates to web or machine learning development.
FacebookPagePage is a dataset introduced in Rozemberczki et al.
(2021), where nodes are Facebook pages, and edges imply mutual likes between the pages.
Nodes features are text embeddings extracted from the pages’ description.
The task consist on identifying the page’s category.
TwitchEN is a dataset introduced in Rozemberczki et al.
Here, nodes correspond to Twitch gamers, and links reveal mutual friendship.
Node features are an embedding of games liked, location, and streaming habits.
The task is to infer if a gamer uses explicit content.
Coauthor Physics & CS are datasets introduced in Shchur et al.
In this case, nodes represent authors which are connected with an edge if they have co-authored a paper.
Node features are BoW representations of the keywords of the author’s papers.
The task consist on mapping each author to their corresponding field of study.
DBLP is a dataset introduced in Bojchevski & G¨unnemann (2018) that represents a citation network.
In this dataset, nodes represent papers and edges correspond to citations.
Node features are BoW representations of the keywords of the papers.
The task is to predict the research area of the papers.
PubMed, Cora & CiteSeer are citation networks introduced in Yang et al.
Nodes represent documents, and edges refer to citations between the documents.
Node features are
BoW representations of the documents.
The task is to infer the topic of the documents.
The bottom rows of Table 5 refer to the datasets from Open Graph Benchmark (OGB)
(Hu et al., 2020a) 5 used in § 6.3.
We include a short description of them in the paragraphs below.
ogbn-arxiv is a citation network of computer science papers in arXiv (Wang et al., 2020).
Nodes represent papers, and directed edges refer to citations among them.
Node features are embeddings of the title and abstract of the papers.
The task is to predict the research area of the nodes.
ogbn-products contains a co-purchasing network (Bhatia et al., 2016).
Nodes represent products, and links are present whenever two products are bought together.
Node features are embeddings of a BoW representation of the product description.
The task is to infer the category of the products.
ogbn-mag is a heterogeneous network formed from a subgraph of the Microsoft Academic
Graph (MAG) (Wang et al., 2020).
Nodes can belong to one of these four types: authors, papers, institutions and fields of study.
Moreover, directed edges belong to one of the following categories: “author is affiliated with an institution,” “author has written a paper,” “paper cites a paper,” and “paper belongs to a research area.”
Only nodes that are papers contain node features, which are a text embedding of the document content.
The task is to predict the venue of the nodes that are papers.
ogbn-proteins is a network whose nodes represent proteins and edges indicate different types of associations among them.
This dataset does not contain node features.
The tasks are to predict multiple protein functions, each of them being a binary classification problem.
D Real data experiments D. Experimental details
Computational resources.
We used CPU cores to run this set of experiments.
In particular, for each trial, we used 2 CPU cores and up to 16 GB of memory.
We ran the experiments in parallel using a shared cluster with 10000 CPU cores approximately.
General experimental setup.
As mentioned in § 6.2, we repeat all experiments 10 times, which correspond to 10 different random initialization of the parameters of the GNNs.
In all cases, we choose the model parameters with the best validation performance during training.
In order to run the experiments and collect the results, we used the GraphGym framework (You et al., 2020), which includes the data processing and loading of the datasets, as well as the evaluation and collection of the results.
We split the datasets in 70 % training, 15 % validation, and 15 % test.
We cross-validate the number of message-passing layers in the network (2 , 3 , 4), as well as the learning rate ([0 . 01 , 0 . 005]).
Then, we report the results of the best validation error among the 4 possible combinations.
However, in practice we found the best performance always to
[Table]
use 4 message-passing layers, and thus the only difference in configuration lies in the learning rate.
We use residual connections between the GNN layers, 4 heads in the attention models, and the Parametric ReLU (PReLU) (He et al., 2015) as the nonlinear activation function.
We do not use batch normalization (Ioffe & Szegedy, 2015), nor dropout (Srivastava et al., 2014).
We use the Adam optimizer (Kingma & Ba, 2015) with β = (0 . 9 , 0 . 999), and an exponential learning-rate scheduler with γ = 0 .
We train all the models for 2500 epochs.
Importantly, we do not use weight decay, since this will bias the solution towards λ 1 = 0 and λ 2 = 1.
We use the Pytorch Geometric (Fey & Lenssen, 2019) implementation of L-CAT for all experiments, switching between models by properly by setting λ 1 and λ 2 .
We parametrize λ 1 and λ 2 as free-parameters in log-space that pass through a sigmoid function—i.e., sigmoid (10 x )— such that they are constrained to the unit interval, and they are learned quickly.
D. Additional results
Table 6 shows the results presented in the main paper (with the addition of a dense feedforward network), while Table 7 presents the results for the remaining datasets, with smaller average degree.
If we focus on Table 7, we observe that all models perform equally well, yet in a few cases
CAT and L-CAT are significantly better than the baselines—e.g., L-CATv2 in CoauthorCS , or L-CAT in Cora .
Following a similar discussion as the one presented in the main paper, these results indicates that L-CAT achieves similar or better performance than baseline models and thus, should be the preferred architecture.
Competitive performance without the graph.
We also include in Tables 6 and 7 the performance of a feed-forward network, referred to as Dense (first row).
Note that the only data available to this model are the node features, and thus no graph information is provided.
Therefore, we should expect a significant drop in performance, which indeed happens for some datasets such as Amazon Computers ( ≈ 7% drop), FacebookPagePage ( ≈ 20% drop),
DBLP ( ≈ 9% drop) and Cora ( ≈ 14% drop).
Still, we found that for other commonly used datasets the performance is similar, e.g., Coauthor Physics and PubMed ; or it is even better CoauthorCS .
These results manifest the importance of a proper benchmarking, and of carefully considering the datasets used to evaluate GNN models.
[Table]
[Table]
E Open Graph Benchmark experiments E. Experimental details
Computational resources.
For this set of experiments, we had at our disposal a set of
Tesla V100-SXM
Statistical significance.
For each CAT and L-CAT model, we highlight significant improvements according to a two-sided paired t-test ( α = 0 . 05), with respect to its corresponding baseline model.
For example, for L-CATv2 with 8 heads we perform the test with respect to
GATv2 with 8 heads.
General experimental setup.
As mentioned in § 6.3, we repeat all experiments with OGB datasets 5 times.
In all cases, we choose the model parameters with the best validation performance during training.
Moreover, when we show the results without specifying the number of heads, we take the model with the best validation error among the two models with 1 and 8 heads.
We use the same implementation of L-CAT for all experiments, switching between models by properly setting λ 1 and λ 2 .
Experiments on arxiv , mag , products use a version of L-CAT implemented in Pytorch Geometric (Fey & Lenssen, 2019).
Experiments on proteins use a version of L-CAT implemented in DGL (Wang et al., 2019a).
We parametrize λ 1 and λ 2 as free-parameters in log-space that pass through a sigmoid function—i.e., sigmoid (10 x )—such that they are constrained to the unit interval, and they are learned quickly.
ArXiv.
As described in § 6.3, we use the example code from the OGB framework (Hu et al., 2020a).
The network is composed of 3 GNN layers with a hidden size of 128.
We use batch normalization (Ioffe & Szegedy, 2015) and a dropout (Srivastava et al., 2014) of 0 .between the GNN layers, and Adam (Kingma & Ba, 2015) with a learning rate of 0 .
We use the ReLU as activation function.
For the initial experiments, we train for 1500 epochs, while we train for 500 epochs for the noise experiments in § 6.3.1.
This is justified given the convergence plots in Fig.
MAG.
We adapted the official code from (Brody et al., 2022).
The network is composed of 2 layers with 128 hidden channels.
This time, we use layer normalization (Ba et al., 2016) and a dropout of 0 .
5 between the layers.
Again, we use ReLU as the activation function, and add residual connections to the network.
As with arxiv , we use Adam (Kingma & Ba, 2015) with learning rate 0 .
We set a batch size of 20000 and train for 100 epochs.
Products.
We use the same setup as (Brody et al., 2022), with a network of 3 GNN layers and 128 hidden dimensions.
We apply residual connections once again, with a dropout (Srivastava et al., 2014) of 0 .
5 between layers.
This time, we use ELU as the activation function.
The batch size is set to 256.
Adam (Kingma & Ba, 2015) is again the optimizer in use, this time with a learning rate of 0 .
We train for 100 epochs, although we apply early stopping whenever the validation accuracy stops increasing for more than 10 epochs.
Note the training split of this dataset only contains 8 % of the data.
Proteins.
We follow once more the setup of (Brody et al., 2022).
The network we use has 6 GNN layers of hidden size 64.
Dropout (Srivastava et al., 2014) is set to 0 .
25 between layers, with an input dropout of 0 . 1.
At the beginning of the network, we place a linear layer followed by a ReLU activation to encode the nodes, and a linear layer at the end of the network to predict the class.
Moreover, we use batch normalization (Ioffe & Szegedy, 2015) between layers and ReLU as the activation function.
We train the model for 1200 epochs at most, with early stopping after not improving for 10 epochs.
E. Additional results
We show in Tables 8 to 10 the results of the main paper for the arxiv , mag , products datasets, respectively, without selecting the best configuration for each type of model.
That is, we show the results for both number of heads.
Note that we already show the full table of results for the protein datasets in the main paper (Table 3).
All the trends discussed in the main paper hold.
[Table] [Table]
[Table]
Extrapolation ablation study.
Due to page constraints, these results were not added to the main paper.
Here, we study two questions.
First, how important are λ 1 and λ 2 in the formulation of L-CAT (Eq. 7)?
For the sake of completeness, the second question we attempt to answer here is whether we can obtain similar performance by just interpolating between GCN and GAT (fixing λ 2 = 0)?
Note that we theoretically showed in §§ 4 and 6.
that CAT fills up a gap between GCN and GAT, making it preferable in certain settings.
To this end, we repeat the experiments for network-initialization robustness in § 6.3.2, since they showed to be the best ones to tell apart the performance across models.
We include three additional models: GCN-GAT, which interpolates between GCN and GAT (or GATv2) by learning λ 1 and fixing λ 2 = 0; CAT- λ 1 which interpolates between GCN and CAT by learning λ 1 and fixing λ 2 = 1; and CAT- λ 2 , which interpolates between GAT and CAT by learning λ 2 and fixing λ 1 = 1.
Results using GAT and shown in Table 11, and using GATv2 in Table 12.
We can observe that GCN-GAT obtains results in between GCN and GAT for all settings, despite being able to interpolate between both layers in each of the six layers of the network.
Regarding learning λ 1 and λ 2 , we can observe that there is a clear difference between learning boths (L-CAT), and learning a single one.
For both attention models, CAT- λ 1 obtains better results than CAT- λ 2 in all settings, but uniform with 8 heads.
Still, the results of both variants are substantially worse than those of L-CAT in all cases, demonstrating the importance of learning to interpolate between the three layer types .
[Table] [Table]
[Table]
F Extending L-CAT to other GNN models
Due to their simplicity and popularity, in the manuscript we focus on the simplest form of
GCNs, as described in § 2.
However, we consider important to remark that L-CAT can be effortless extended to a large range of existing GNN models.
A more general formulation of a message-passing GNN layer than the one given in Eq.
1 is the following:
[Equation]
where (cid:98) γ ij is a scalar value, (cid:76) refers to any permutation invariant operation—e.g., sum, mean, maximum, or minimum operations—and M is the message operator, which can be parameterized, and produces a message based on the sender and receiver representations.
This formulation comprises most GNN architectures present in the current literature.
For example:
• GCN (Kipf & Welling, 2017): h ′ i
= (cid:80) j ∈ N ∗
i (cid:98) γ ij W v h j where (cid:98) γ ij = 1 | N ∗ i | as consider in the main paper, or (cid:98) γ ij = 1 √ d j d i , where d i is the number of neighbors of node i (including self-loops), if we consider the symmetric normalized adjacency matrix instead.
• GIN (Xu et al., 2019): h ′ i
=
(1 + ε ) (cid:98) γ ii h
i + (cid:80)
j ∈ N i (cid:98) γ ij h j with (cid:98) γ ij = 1.
• PNA (Corso et al., 2020):
h ′ i =
(cid:76) j ∈ N ∗ i (cid:98) γ ij M
( h i , h j ; θ M ) where (cid:98) γ ij = 1, M is an multi-layer perceptron, and (cid:76) is a set of permutation invariant operations, e.g., (cid:76) =
[ µ, σ, max , min].
• GCNII (Chen et al., 2020): h ′ i = (cid:16) α h
i
+ (1 − α ) (cid:80) j ∈ N ∗
i (cid:98) γ ij h j (cid:17) ((1 − β ) I + β W v ) where, just as in the GCN case, (cid:98) γ ij = 1 √ d j d i , and 0 ≤ α, β ≤ 1.
In all the models above, the values (cid:98) γ ij are taken from the adjacency matrix A (whose entries are 1 if there exists an edge between nodes i and j , and 0 otherwise), or a matrix derived from it, e.g., the symmetric normalized adjacency matrix.
Note that the attention coefficients γ ij defined in Eq.
3 can be understood as an attentionequivalent of the adjacency matrix.
Indeed, by defining A att as a matrix whose entries are | N ∗ i | γ ij , one can obtain a row-stochastic matrix that can substitute the adjacency matrix of any GNN model.
This technique to generalize attention models to GNN variants more complex than a GCN has been successfully applied in prior literature (Wang et al., 2021b).
With this new interpretation of attention-models, the interpolation performed by L-CAT (see Eq. 7) can similarly be re-interpreted.
Indeed, L-CAT learns to interpolate between the adjacency matrix A , and an attention-based adjacency matrix A att , which can be produced by either GAT (Eq. 3) or CAT (Eq. 6), depending on the value of λ 2 .
F. PNA experiments
In Tables 13 and 14, we show the results—for the datasets described in App.
C—using
L-CAT and CAT in conjunction with the PNA model (Corso et al., 2020).
First, we note that standard PNA works quite well in most cases.
Second, if we focus on Table 13, we observe that the standard attention models (i.e., PNAGAT and PNAGATv2) perform significantly worse than the other approaches, in particular on datasets with large average degree, e.g., on Amazon Computers .
Finally, we observe that the L-CAT models (i.e., L-PNACAT and
L-PNACATv2) drastically improve the performance of their attention counterparts and achieve similar performance as the PNA model, with lower performance on the GitHub and
Facebook datasets and higher performance on Cora and CiteSeer .
To keep the same number of parameters, we reuse the PNA weights to compute the attention scores ( W q = W k = W v ).
However, this could be detrimental, as the role of W v is completely different from that of W q and W v .
Tables 15 and 16 show the same experiments as before, but using different parameters to compute the keys and queries (i.e., W q = W k ̸ = W v ).
We observe that the increase of parameters generally helps both CAT and L-CAT models, now outperform the base PNA model in some settings
[Table] [Table]
[Table] [Table]
[Table] [Table]
F. GCNII experiments
Similarly, we have run the experiments from § 6.2, this time combining GCNII (Chen et al., 2020) with GAT, CAT, and L-CAT as explained above.
Results are shown in Tablesand 18, in which we can observe that the baseline model obtains the best results so far in the manuscript (in comparison with both GCN and PNA).
And just as before, we observe again that CAT and L-CAT always improve with respect to their base models, staying on par with the baseline GNCII model and, sometimes, even outperforming the baseline model on average (e.g., Coauthor Physics , TwitchEN ).
As with the experiments for PNA, Tablesand 20 shows the results when the attention matrices are different from the value matrix ( W q =
W k ̸ = W v ).
We can similarly observe that most of the results are improved with the additional parameters, beating the baseline model in different datasets.
[Table]
[Table] [Table]