An Analysis of Virtual Nodes in Graph Neural Networks for Link Prediction
Introduction
It is well known that the graph classification performance of graph neural networks (GNNs) often improves by adding an artificial virtual node to the graphs, which is connected to all graph nodes [1–4].
While virtual nodes were originally thought as aggregated representations of the entire graph, they also provide shortcuts for message passing between nodes along the graph edges.
Surprisingly, the advantages of virtual nodes have never been theoretically investigated, and their impact on other problems is still an open research question.
We adapt and study the virtual node concept for problems over networks, which are usually larger, often very sparse or dense, and overall more heterogeneous.
Many popular GNNs are based on message passing, which computes node embeddings by iteratively aggregating the features of (usually direct) neighbor nodes along the graph edges [1].
In this way, they are able to distinguish (non-)isomorphic nodes (to great extent)
[5], but this does not transfer to links [6]; for links, extra procedures may be needed (e.g., modeling enclosing subgraphs [7]).
Furthermore, on large graphs, GNNs may face the under-reaching problem if long-range dependencies beyond the model’s computing radius are important for the problem at hand (e.g., complex chains of proteinprotein interactions).
Over dense graphs, GNNs with many layers struggle with over-smoothing , node representations converging to similar values.
There have been several proposals to overcome these problems.
On the one hand, several works propose techniques that allow for larger numbers of GNN layers [8–14].
However, as shown in our later results, many of them do not perform well on link predictions tasks, especially on comparably dense graphs.
On the other hand, there are approaches that adapt message passing to consider neighbors beyond the one-hop neighborhood: based on graph diffusion [15–20] and other theories [21, 22].
Yet, most of these models are relatively complex and, in fact, in our experiments over the challenging graphs from the Open Graph Benchmark (OGB) [23], several ran out of memory.
In this paper, we show that virtual nodes may alleviate these typical issues of GNNs over larger graphs .
We focus on link prediction (but most results may be easily extended to node classification), which is important in view of incomplete graph data in practice in various different domains [24–27] .
Numerous models have been proposed to solve this problem in the past, ranging from knowledgegraph-specific predictors [27] to GNNs [7, 24].
We explore the application and effects of virtual nodes in link prediction both theoretically and empirically:
• We propose to use multiple virtual nodes in the network graph scenario and describe a graph-based technique to connect them to the graph nodes.
In a nutshell, we use a graph clustering algorithm to determine groups of nodes in the graph belonging together and then connect these nodes to a common virtual node (see Figure 1).
In this way, we add expressiveness, and under-reaching is decreased because clustered nodes can share information easily; meanwhile, the nodes are spared of unnecessary information from unrelated nodes (i
e., in contrast to the single virtual node model).
[Figure]
We also investigate alternative methods to determine the virtual node connections (i.e., randomization) and compare to the original model with a single virtual node.
• We provide the first theoretical analysis of the benefits of virtual nodes in terms of (I) expressiveness of the learned link representation and (II) potential impact on under-reaching and over-smoothing.
• We conducted experiments over challenging datasets, provide ablation studies and a detailed analysis of important factors.
Altogether, we show that, also for link prediction, virtual nodes are simple but powerful extensions that may yield rather stable performance increases for various standard GNNs.
Since the latter represent simple and proven models which are especially interesting for applications, our study provides practical guidance and explanations about where and how virtual nodes may provide benefits.
In this abstract, we give an overview of our main findings; for details and additional results, see the appendix.
Our code is available at https://github.com/eujhwang/vn-analysis .
Preliminaries
Link Prediction.
We consider a graph G =
( V, E ) with nodes V and undirected edges E ⊆
V × V .
This basic choice is only for ease of presentation; our techniques work for directed graphs and (with simple adaptation) for graphs with labelled nodes (edges).
We assume V to be ordered and may refer to a node by its index in V .
For a node v ∈ V , N v denotes the set of its neighbors.
Given two nodes, the link prediction task is to predict if there is a link between them.
Message-Passing Graph Neural Networks.
In this paper, we usually use the term graph neural networks (GNNs) to denote GNNs that use message passing as described in [1].
These networks compute for every v ∈ V a node representation h ℓv at layer ℓ , by aggregating its neighbor nodes based on a generic aggregation function and then combine the obtained vector with h ℓ − 1 v as below.
[Equation]
Link prediction with GNNs is usually done by combining the final representations of nodes u, v under consideration and passing them through several feed-forward layers with a final sigmoid function for scoring.
Our implementation follows this approach.
Virtual Nodes in Graph Neural Networks for Link Prediction
Multiple Virtual Nodes.
The intuition of using virtual nodes is to provide a shortcut for sharing information between the graph nodes.
However, the amount of information in a graph with possibly millions of nodes is enormous, and likely too much to be captured in a single virtual node embedding.
Further, not all information is equally relevant to all nodes.
Therefore we suggest to use multiple virtual nodes S = { s 1 , s 2 . . .
, s n } 2 each being connected to a subset of graph nodes, as determined by an assignment σ : V → [1 , n ] ; n is a hyperparameter.
We consider two options to obtain σ :
Randomness: GNN-RM.
Most simply, we can determine a fixed σ randomly once with initialization.
Clustering: GNN-CM.
Many types of graph data incorporate some cluster structure that reflects which nodes belong closely together (e.g., collaboration or social networks).
We propose to connect nodes in such a cluster to a common virtual node, such that the structure inherent to the given graph is reflected in our virtual node assignment σ .
More precisely, during initialization, we use a generic clustering algorithm (e.g., METIS [28]) which, given a number m , creates a set C
= { C 1 , C 2 . . .
C m } of clusters (i.e., sets of graph nodes) by computing an assignment ρ : V → [1 , m ] , assigning each graph node to a cluster.
We then set m = n and σ = ρ .
The Model.
We integrate the multiple virtual nodes into a generic message-passing GNN in a straightforward way extending the approach from [23] to include multiple virtual nodes, computing node representations h ℓv for a node v ∈ V at layer ℓ as below.
The highlighted adaptation of the standard GNN (Equation (1)) is only minor, but powerful.
In our implementation, COMB ℓ VN is addition combined with linear layers and layer normalization, AGG ℓ VN is a sum.
[Equation]
3.1 Analysis I: Virtual Nodes Increase Expressiveness
Additional structure-related features such as distance encodings [6, 29] are known to make graph representation learning more powerful.
Our multiple virtual notes have a similar effect.
Figure 1 gives an intuition of how they can increase the expressiveness of the regular 1-WL-GNN.
Consider the nodes v 2 , v 3 with the same local structure, which means they can get the same node representations by using 1-WL-GNN.
So we cannot discriminate the links ( v 1 , v 2 ) and ( v 1 , v 3 ) if we just use 1-WLGNN and concatenate the node representations for link prediction.
However, if we add 2 virtual nodes and add extra features to each node.
v 1 and v 2 get a new feature (1 , 0) , v 3 get new feature (0 , 1) . So it is easy to see ( v 1 , v 2 ) and ( v 1 , v 3 ) now have different representations. Furthermore, we have the following theorem to show that using multiple virtual nodes can increase the power of discriminating links. More details are in Appendix B.1.
Theorem 3.1 Given an arbitrary non-attributed graph with n nodes, a constant ϵ > 0 , and m virtual nodes that evenly divide the node set into m clusters, we have: If the degree of each node in the graph is between 1 and O (log 1 − ϵ 2 k ( n )) , then there are ω (cid:0) ( m − 1) 2 ( n ϵ m − 1) 3 (cid:1) many pairs of non-isomorphic links ( u, w ) , ( v, w ) such that a k -layer 1-WL-GNN gives u, v the same representation, while using m virtual nodes give ( u, w ) , ( v, w ) different representations.
3.2 Analysis II: Virtual Nodes Impact Node Influence
We assume we can learn useful embeddings for virtual nodes if the assignment is chosen appropriately.
Based on the above analysis, we expect virtual nodes to positively impact learning and prediction performance.
Following [8, 16], we measure the sensitivity of a node y on a node x by the influence score .
For a k -layer GCN, this score is known to be proportional in expectation to the k -step random walk distribution P rw from x to y .
3 We exploit this relationship and argument in terms of P rw .
Impact of Virtual Nodes.
For simplicity, we consider the influence score in an r -regular graph.
Consider the message passing between two nodes x and y .
For k = 1 , all the nodes can be classified into two cases: if y is not connected to x , the influence changes from 0 to 1 ( r +2) ; otherwise:
[Equation]
For k > = 2 , by adding a virtual node s in one GNN layer, the probability changes to:
[Equation]
[Table] [Figure] [Table] [Figure]
[Table] We continue along these lines and assume there is a shortest path of length ≤ k between x and y .
If x and y connect to the same virtual node s , then the above changes to:
[Equation]
Since the set C s of nodes connecting to s is much smaller than V , multiple virtual nodes can increase the impact of potentially important distant nodes more than a single virtual node .
Impact on Over-Smoothing The idea is to show that multiple virtual nodes help preserve more local information.
If we consider the influence of x onto itself, we can show that, with a single virtual node, a graph node can preserve less information for itself at each layer.
However, this changes in view of multiple virtual nodes; in particular, when | C s | ≤ r + 1 .
We encounter this scenario practically especially with dense graphs.
This fits nicely since dense graphs are particularly prone to over-smoothing and, as shown in [8, 16], additional capability to preserve local information in message passing steps helps to reduce over-smoothing .
More details are shown in Appendix B.2.
Evaluation
Datasets.
We use two datasets from OGB: ddi , a drug-drug interaction network; and collab , an author collaboration network [23].
Data statistics are in Appendix (Table 4).
ddi is dense with a low graph diameter; while collab is sparser with large diameter.
Both have high clustering coefficients.
Models.
Standard GNNs: GCN [30] and SAGE [31], which we extend with virtual nodes; deep
GNNs: SGC , APPNP , DeeperGCN , GCN- JKNet ; message passing beyond the direct neighborhood:
P-GNN [22], APPNP [16], GDC [20]; and an advanced GNN-based link predictor: SEAL [7].
Results, Table 1.
Overall Impact of Virtual Nodes.
The common approach of using a single virtual node (GNN-VN) yields good improvements over ddi and slight improvements over collab .
The numbers for GNN-RM reflect the randomness of their connections to the virtual nodes, there is no clear trend; but they clearly outperform the original models.
The virtual node assignment based on the graph structure (GNN-CM) yields consistently good improvements over ddi and collab .
We note that we obtained some ambiguous results on other smaller data that has less cluster structure, but overall can observe a positive impact.
Model Comparison.
The results of the best models from the OGB leaderboard vary strongly with the different datasets (e.g., SEAL), or have not been reported at all.
Most deep GNNs and models that use complex message-passing techniques perform disappointing and, overall, much worse than the standard GNNs.
We did thorough hyperparameter tuning for these models and it is hard to explain.
A possible reason may be most of their original evaluations focus on node or graph classification and consider very different types of data.
The model closest to our approach is the position-aware graph neural network (P-GNN) [22].
It assigns nodes to random subsets of nodes called “anchor-sets”, and then learns a non-linear aggregation scheme that combines node feature information from each anchor-set and weighs it by the distance between the node and the anchor-set.
So, it creates a message for each node for every anchor-set, instead of for each direct neighbor.
The fact that it ran out of memory on collab shows that practice may benefit from simpler or more efficient schemes.
Impact of Virtual Node Number, Figure 2.
The configurations of the best models provided in the appendix show that the chosen numbers of virtual nodes are indeed random for the “random” models, but GNN-CM consistently uses a high number of virtual nodes, which also suits it better according to our theoretical analysis.
In line with this, the more detailed analysis varying the numbers of virtual nodes, yields best results (also in terms of standard deviations) for SAGE-CM at rather high values.
For GCN, we do not see a clear trend, but (second) best performance with 64 virtual nodes.
[Table] Using Virtual Nodes Only at the Last GNN Layer [Table] Table 2.
[32] show that using a fully connected adjacency matrix at the last layer of a standard GNN helps to better capture information over long ranges.
We therefore investigated if it is a better architectural choice to use virtual nodes only at the last layer.
However, we observed that this can lead to extreme performance drops.
In Table 2, VN OL and CM OL indicate the ablation models which use virtual nodes only at the last layer.
It shows that these ablations models decrease the performance a lot.
That means using virtual nodes only at the last layer is not enough.
Impact of Clustering Algorithm, Table 3. Our architecture is generic in the clustering algorithm, and we investigated the effects of varying that. Note that CM metis in the table is our GNN-CM model used in Table 1. Graclus is similar in nature to METIS in that it also creates partitions based on the adjacency matrix, but it took much longer to run.
Diffpool considers the node features and yields improvements for GCN, but does not scale to larger datasets.
Over ddi , there is no clear winner and, given its efficiency, METIS turns out to be a good solution
[Table] [Table] Conclusions and Discussions. In a nutshell, our clustering-based virtual node assignment provides stable performance increases if the graph contains good cluster structure and is sufficiently large.
In smaller graphs, the GNNs alone were usually sufficient.
In line with our theoretical investigation, we expect virtual nodes to be especially beneficial over dense graphs.
A Appendix B Additional Theoretical Results B.1 Expressiveness of link representation
Given multiple virtual nodes S = { s 1 , ..., s m } , we obtain a node labeling that includes the node representations h ℓs i of the virtual nodes.
For every u ∈ V , we have additional features l ( u | S ) =
( h ℓs 1 , ..., h ℓs m ) T ( γ ( u | s 1 ) , ..., γ ( u | s m )) , where γ ( u | s i )
= 1 if u is connected to the virtual node s i , and γ ( u | v i ) =
otherwise.
We can initialize h
s i with the one-hot encoding of i to ensure the s i have different labels.
We can then show that this labeling increases the power of GNNs.
Theorem B.1 Given an arbitrary non-attributed graph with n nodes, if the degree of each node in the graph is between 1 and O (log 1 − ϵ 2 k ( n )) , for any constant ϵ > 0 , given m virtual nodes which evenly divide the node set into m clusters, there are ω (cid:0) ( m − 1) 2 ( n ϵ m − 1) 3 (cid:1) many pairs of non-isomorphic links ( u, w ) , ( v, w ) such that a k -layer 1-WL-GNN gives u, v the same representation, while using m virtual nodes give ( u, w ) , ( v, w ) different representations.
The theorem says that 1-WL-GNN with virtual nodes can discriminate many links that 1-WLGNN cannot discriminate.
On the other hand, it is intuitive adding virtual nodes can be at least as powerful as 1-WL-GNNs since it keeps all other components.
If there are links that 1-WL-GNN can discriminate we only need to assign the related nodes to the same virtual nodes, so that the virtual-nodes-based method can also discriminate them.
B.1.1 Proof of Theorem B.
The proof can be separated into two steps.
The first step is to prove that there exist n/o ( n 1 − ϵ )
= ω ( n ϵ )
many nodes that are locally h -isomorphic (which means their h -hop enclosing subgraphs are isomorphic).
This step is same as the proof of Theorem 2 in [6], so we omit the details here.
The basic idea is to expand the h -hop enclosing subgraph G h ( v ) of v to another subgraph ˜
G h ( v ) and then use the pigeon hole principle to count the possible isomorphic ˜
G h ( v ) .
After getting these locally isomorphic nodes, we denote the set of these nodes as V iso .
The second step is to find the non-isomorphic links.
Step 2.
We partition V iso = ∪ i =1 V i where V i is the subset of nodes connected to virtual node s i .
To be simple, we call each V i a cluster, and the sizes of different clusters are assumed to be the same |
V i | = | V iso | /m .
Consider two nodes u ∈ V
i
and v ∈ V j from different clusters.
Since both of them are in V iso , they have identical h -hop neighborhood structures, and h -layer 1-WL-GNN will give them the same representations.
Then let us select another node w in V i , h -layer 1-WL-GNN will also make ( u, w ) and ( v, w ) have the same representation.
However, if we use virtual nodes to label nodes and give them additional features, because u, w are in the same cluster while v, w belong to different clusters, ( u, w ) will have different representation from ( v, w ) .
Now we count the number of such non-isomorphic link pairs Y , we obtain:
[Equation]
Taking | V iso | = ω ( n ϵ ) into the above in-equation, we get
[Equation]
B.2 Node Influence
In the following, without loss of generality, we take a k -layer GCN [30] as the example, and hence consider layers described as follows:
[Equation]
Influence Score .
We measure the sensitivity of a node y on a node x by the influence score [8]
I ( x, y )
=
e T
∂h kx
∂h
y ; e is a vector of all ones and h kx is the embedding of x at the k th layer.
The influence score is known to be proportional in expectation to the k -step random walk distribution P rw from x to y :
[Equation]
( v 0 r , v 1 r , ..., v kr ) are the nodes in the path r from x := v 0 r to y := v kr , R k is the set of paths of length k .
In what follows, we exploit the relationship between the influence score and the probability P rw and argument in terms of the latter.
In particular, we will show how P rw changes in view of virtual nodes.
Note that we assume all the paths of message passing have the same probability.
We assume a self-loop at each regular graph node, this is standard and supported by Equation (1).
Hence, the denominator in the above equation changes slightly:
[Equation]
We neglect the self-loops with virtual nodes only for reasons of readability.
But it can be readily checked that the later equations hold similarly with an additional “ +1 ” in denominators.
For simplicity, we further consider the graph to be r -regular; in the standard case without virtual nodes,
Equation (4) then simplifies to:
[Equation]
[Equation]
We hypothesize that we can come to similar conclusions in a general graph with average degree r .
Impact of One Virtual Node.
We focus on the message passing between two nodes x and y , in layer k and calculate P srw ( x → y, k ) , the influence score in the setting with virtual nodes, here with one, s .
In particular we assume x, y ∈ V and hence x, y / ∈ { s } .
We argument inductively, based on k and, for each GNN layer, separate the impact of the messages coming from the virtual node.
For k = 1 , all the nodes can be classified into two cases: if y is not connected to x , the influence changes from 0 to 1 ( r +2) ; if y is connected to x , the influence score is:
[Equation]
Note that the probability for x → s is the same as from x to any other neighbor, 1 | V | for s → y follows from the | V | connected nodes at s .
For k ≥ 2 , we obtain:
[Equation]
Multiple Virtual Nodes.
In view of multiple virtual nodes, the above analysis gets more appealing.
We continue along the above lines and assume there is a shortest path of length ≤ k between x and y .
If x and y connect to the same virtual node s , then Equation (7) changes as follows:
[Equation]
Since the set C s of nodes connecting to s is much smaller than V , i.e., | C s | << | V | , the impact of multiple virtual nodes on the influence score is greater than that of a single virtual node .
In case that x and y do not connect to the same virtual node, the probability just slightly decreases.
The maximum possible decrease occurs when no nodes in the path between x and y are connected to a common virtual node, including x and y : δ wc = | R k | ( r +1)
k − | R k | ( r +2)
k ; here we subtract from the regular
P rw our P msrw , in which the second (virtual node) component is 0 .
Impact on Over-Smoothing The idea is to show that multiple virtual nodes help to preserve local information at the graph nodes.
To this end, we consider the influence of x onto itself.
For the setting with a single virtual node and k = 1 , the change in influence score is
[Equation]
This means the node will preserve less information for itself at layer considering the message coming from the single virtual node.
However, in view of multiple virtual nodes, we come to a different conclusion.
[Equation]
[Equation] Since | C s | << | V | , we obtain δ ms ( x, k = 1) >> 1 | V | ( r +1) − 1 ( r +1)
( r +2) , which means we can preserve much more local information than in the setting with a single virtual node .
Especially, when | C s | ≤ r + 1 , the self-transition probability is even higher than in the original setting without virtual nodes.
We encounter this scenario practically especially with dense graphs.
This fits nicely since these graphs are particularly prone to over-smoothing and, as shown in [8, 16], additional capability to preserve local information in message passing steps helps to reduce over-smoothing .
For k ≥ 2 , δ ms ( x, k ) =
| R k | ( r + 2) k + ( r + 2) | C s | P msrw ( x → x, k − 1) − | R k | ( r + 1) k
Assume P msrw ( x → x, k − 1) >
P srw ( x → x, k − 1) , since | C s | < | V | , then we get δ ms ( x, k ) <
| R k | ( r + 2) k + ( r + 2) | V | P srw ( x → x, k − 1) − | R k | ( r + 1) k
= δ s ( x, k ) .
Adding the condition that δ ms ( x, k = 1) > δ s ( x, k = 1) , we know that for any k , multiple virtual nodes can preserve more local information than single nodes. [Table]
B.3 Relationship with Labeling Tricks
Although the concept of link representation is from [6], we would like to clarify that our labeling strategy is not a valid labeling trick by the definition of [6].
Consider an undirected graph G as described in Section 2.
In addition, the tensor A ∈ R n × n × k contains all node and edge features (if available).
The diagonal components
A v,v, : denote the node features, while the off-diagonal components A u,v, : denote the edge features of edge ( u, v ) .
The labeling trick uses a target node set S ⊆ V and a labeling function to label all nodes in the node set V and stack the labels with A .
A valid labeling trick must meet two conditions: (1) the nodes in S have different labels from the rest of the nodes, (2) the labeling function must be permutation invariant.
Using virtual nodes is not a valid labeling trick in the following two aspects: First, the virtual node set S is not a subset of graph nodes V , and we use addition instead of concatenation.
Even if we extend V to V ∪ S , our labeling strategy still does not fit the permutation-invariant requirement.
Nevertheless, it can achieve similar effects in learning structural link representations.
C Additional Experimental Details and Results C.1 Data Statistics
Data Statistics of the ddi and collab are shown in Table 4.
From the table, we can see both of the datasets have good clustering structure.
ddi is extremely dense and collab is sparser.
That is interesting to note that on the denser ddi , our virtual nodes approaches achieved better performance gain.
C.2 Model Configurations and Training
We trained all models for 80 runs using the Bayesian optimization provided by wandb 5 to select hyperparameters from the following pool.
[Table]
Please note that we considered the wide ranges of values only in order to find a good general setting. For practical usage a hidden dimension of 256, learning rate of 0.0001, and dropout of 0.3 should work well; only on the small graphs a dropout of 0 might work better.
As usual, the number of layers depends on the type of data; however, note that the virtual nodes make it possible to use more that then the usual 2-3 layers.
Generally, higher numbers of virtual nodes work better, in line with our theoretical results.
Also note that we used less virtual nodes in the selection for the models -RM since preliminary results showed that larger numbers did not change the results greatly – probably due to the randomness.
We
# Layers
H it s @     0 .0 .0 .
# Layers
H it s @collab 1010 2 .
[Figure]
used maximally 64 virtual nodes due to memory issues with larger numbers (e.g., 128), especially on the larger datasets.
We used 500 epochs with a patience of 30.
Furthermore, for collab , we used the validation edges during testing (OGB contains both settings, with and without them).
We tuned all models for 80 runs, and thereafter ran the models with the best 3 configurations forruns and chose the best of these model as the final model (configuration).
We trained as suggested by the OGB (e.g., the splits, negative sampling) but used a batch size of 2 12 .
C.3 Additional Results
Impact of Virtual Nodes on Number of GNN Layers and Efficiency, Figure 3.
For the virtual nodes models, the scores increase with the number of layers for a longer time, GCN drops earlier.
On ddi , GCN-VN and -CM reach their best scores at 6 and 8 layers, respectively, which is remarkable for that very dense dataset, being prone to over-smoothing.
On collab it is the other way around.
The figure also gives an idea about the runtime increase with using virtual nodes.
It compares the 6-layer models, and shows the 4-layer GCN-CM which obtains performance similar to the 6-layer GCN-VN.