Original: R EVISITING V IRTUAL N ODES IN G RAPH N EURAL N ETWORKS FOR L INK P REDICTION
Revised: R EVISITING V IRTUAL N ODES IN G RAPH N EURAL N ETWORKS FOR L INK P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: It is well known that the graph classiﬁcation performance of graph neural networks often improves by adding an artiﬁcial virtual node to the graphs, which is connected to all nodes in the graph.
Revised: It is well known that the graph classiﬁcation performance of graph neural networks often improves by adding an artiﬁcial virtual node to the graphs, which is connected to all nodes in the graph.

Original: Intuitively, the virtual node provides a shortcut for message passing between nodes along the graph edges.
Revised: Intuitively, the virtual node provides a shortcut for message passing between nodes along the graph edges.

Original: Surprisingly, the impact of virtual nodes with other problems is still an open research question.
Revised: Surprisingly, the impact of virtual nodes with other problems is still an open research question.

Original: In this paper, we adapt the concept of virtual nodes to the link prediction scenario, where we usually have much larger, often dense, and more heterogeneous graphs.
Revised: In this paper, we adapt the concept of virtual nodes to the link prediction scenario, where we usually have much larger, often dense, and more heterogeneous graphs.

Original: In particular, we use multiple virtual nodes per graph and graph-based clustering to determine the connections to the graph nodes.
Revised: In particular, we use multiple virtual nodes per graph and graph-based clustering to determine the connections to the graph nodes.

Original: We also investigate alternative clustering approaches (e.g., random or more advanced) and compare to the original model with a single virtual node.
Revised: We also investigate alternative clustering approaches (e.g., random or more advanced) and compare to the original model with a single virtual node.

Original: We conducted extensive experiments over different datasets of the Open Graph Benchmark (OGB) and analyze the results in detail.
Revised: We conducted extensive experiments over different datasets of the Open Graph Benchmark (OGB) and analyze the results in detail.

Original: We show that our virtual node extensions yield rather stable performance increases and allow standard graph neural networks to compete with complex state-of-the-art models, as well as with the models leading the OGB leaderboards.
Revised: We show that our virtual node extensions yield rather stable performance increases and allow standard graph neural networks to compete with complex state-of-the-art models, as well as with the models leading the OGB leaderboards.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Link prediction is an important task to complete graphs that are missing edges in various domains: citation networks (Kipf & Welling, 2016), social networks (Adamic & Adar, 2003), medical drug interaction graphs (Abbas et al., 2021), or knowledge graphs (KGs) (Ji et al., 2021).
Revised: Link prediction is an important task to complete graphs that are missing edges in various domains: citation networks (Kipf & Welling, 2016), social networks (Adamic & Adar, 2003), medical drug interaction graphs (Abbas et al., 2021), or knowledge graphs (KGs) (Ji et al., 2021).

Original: Numerous kinds of models have been proposed to solve the link prediction problem, ranging from KG-speciﬁc predictors (Ji et al., 2021) to graph neural networks (GNNs) (Kipf & Welling, 2016; Zhang & Chen, 2018).
Revised: Numerous kinds of models have been proposed to solve the link prediction problem, ranging from KG-speciﬁc predictors (Ji et al., 2021) to graph neural networks (GNNs) (Kipf & Welling, 2016; Zhang & Chen, 2018).

Original: Over dense biomedical networks, GNNs turned out to work especially well (Hu et al., 2020).
Revised: Over dense biomedical networks, GNNs turned out to work especially well (Hu et al., 2020).

Original: In this work, we focus on graph neural networks for link prediction.
Revised: In this work, we focus on graph neural networks for link prediction.

Original: Many of the popular GNNs are based on the message-passing scheme, which computes node embeddings based on iteratively aggregating the features of (usually direct/one-hop) neighbor nodes along the graph edges (Gilmer et al., 2017).
Revised: Many of the popular GNNs are based on the message-passing scheme, which computes node embeddings based on iteratively aggregating the features of (usually direct/one-hop) neighbor nodes along the graph edges (Gilmer et al., 2017).

Original: Interestingly, best performance is usually obtained by only considering two to three hops of neighbors (i.e., 2-3 layers in the GNN).
Revised: Interestingly, best performance is usually obtained by only considering two to three hops of neighbors (i.e., 2-3 layers in the GNN).

Original: One main reason identiﬁed for this is over-smoothing , the problem that node representations become indistinguishable when the number of layers increases (Li et al., 2018).
Revised: One main reason identiﬁed for this is over-smoothing , the problem that node representations become indistinguishable when the number of layers increases (Li et al., 2018).

Original: The exponentially-growing amount of information has also been suggested as one issue connected to capturing long-range dependencies (Alon & Yahav, 2021).
Revised: The exponentially-growing amount of information has also been suggested as one issue connected to capturing long-range dependencies (Alon & Yahav, 2021).

Original: While it is likely that link prediction most often depends on the local node neighborhood, it is not beyond imagination that there are critical long-range dependencies (e.g., complex chains of drug-drug or drug-protein interactions).
Revised: While it is likely that link prediction most often depends on the local node neighborhood, it is not beyond imagination that there are critical long-range dependencies (e.g., complex chains of drug-drug or drug-protein interactions).

Original: Hence, using a small number of layers to overcome the above problems results in under-reaching .
Revised: Hence, using a small number of layers to overcome the above problems results in under-reaching .

Original: There have been several recent proposals to overcome under-reaching.
Revised: There have been several recent proposals to overcome under-reaching.

Original: On the one hand, several works propose techniques that allow for larger numbers of GNN layers (Xu et al., 2018;
Revised: On the one hand, several works propose techniques that allow for larger numbers of GNN layers (Xu et al., 2018;

Original: Wu et al., 2019; Liu et al., 2020; Chen et al., 2020; Sun et al., 2021;
Revised: Wu et al., 2019; Liu et al., 2020; Chen et al., 2020; Sun et al., 2021;

Original: Zhou et al., 2020; Li et al., 2020a).
Revised: Zhou et al., 2020; Li et al., 2020a).

Original: However, although (Chen et al., 2020) show that over-smoothing happens particularly in dense graphs, the link prediction experiments in these works consider citation or recommendation networks, but not the especially dense biomedical ones.
Revised: However, although (Chen et al., 2020) show that over-smoothing happens particularly in dense graphs, the link prediction experiments in these works consider citation or recommendation networks, but not the especially dense biomedical ones.

Original: And our experiments over the latter suggest that the reported results are not generalizable to the more challenging biomedical data.
Revised: And our experiments over the latter suggest that the reported results are not generalizable to the more challenging biomedical data.

Original: On the other hand, there are approaches that adapt the message-passing scheme to consider neighbors beyond the one-hop neighborhood: based on graph diffusion (Atwood & Towsley, 2016; Klicpera et al., 2019a; Abu-El-Haija et al., 2019; Xu et al., 2019a;
Revised: On the other hand, there are approaches that adapt the message-passing scheme to consider neighbors beyond the one-hop neighborhood: based on graph diffusion (Atwood & Towsley, 2016; Klicpera et al., 2019a; Abu-El-Haija et al., 2019; Xu et al., 2019a;

Original: Ma et al., 2020; Klicpera et al., 2019b) and other theories (Morris et al., 2019; You et al., 2019).
Revised: Ma et al., 2020; Klicpera et al., 2019b) and other theories (Morris et al., 2019; You et al., 2019).

Original: However, most of these models are relatively complex and, in fact, in our experiments over the challenging graphs from the Open Graph Benchmark (OGB) (Hu et al., 2020), several ran out of memory.
Revised: However, most of these models are relatively complex and, in fact, in our experiments over the challenging graphs from the Open Graph Benchmark (OGB) (Hu et al., 2020), several ran out of memory.

Original: Moreover, the majority has not considered link prediction, while this problem was recently shown to be more difﬁcult than node classiﬁcation (Zhang et al., 2020).
Revised: Moreover, the majority has not considered link prediction, while this problem was recently shown to be more difﬁcult than node classiﬁcation (Zhang et al., 2020).

Original: In this paper, we propose a simple but elegant solution to under-reaching based on the concept of virtual nodes (Gilmer et al., 2017;
Revised: In this paper, we propose a simple but elegant solution to under-reaching based on the concept of virtual nodes (Gilmer et al., 2017;

Original: Li et al., 2017; Pham et al., 2017; Ishiguro et al., 2019).
Revised: Li et al., 2017; Pham et al., 2017; Ishiguro et al., 2019).

Original: Virtual nodes are well known to often improve the graph classiﬁcation performance of graph neural networks, where an artiﬁcial virtual node is added to every graph and connected to all nodes in the graph.
Revised: Virtual nodes are well known to often improve the graph classiﬁcation performance of graph neural networks, where an artiﬁcial virtual node is added to every graph and connected to all nodes in the graph.

Original: While the virtual nodes were originally thought as representations of the entire graph, they also provide shortcuts for message passing between nodes along the graph edges.
Revised: While the virtual nodes were originally thought as representations of the entire graph, they also provide shortcuts for message passing between nodes along the graph edges.

Original: Surprisingly, the impact of virtual nodes for the link prediction problem has not been investigated yet.
Revised: Surprisingly, the impact of virtual nodes for the link prediction problem has not been investigated yet.

Original: The reason for this might be that the often very large and heterogeneous “network” graphs in link prediction are of very different nature and require novel/adapted solutions (e.g., a protein interaction network may easily contain millions of nodes, whereas a molecule to be classiﬁed contains usually less than ﬁfty).
Revised: The reason for this might be that the often very large and heterogeneous “network” graphs in link prediction are of very different nature and require novel/adapted solutions (e.g., a protein interaction network may easily contain millions of nodes, whereas a molecule to be classiﬁed contains usually less than ﬁfty).

Original: We explore application and effects of virtual nodes in link prediction theoretically and empirically:
Revised: We explore application and effects of virtual nodes in link prediction theoretically and empirically:

Original: • We propose to use multiple virtual nodes in the link prediction scenario and describe a graph-based technique to
Revised: • We propose to use multiple virtual nodes in the link prediction scenario and describe a graph-based technique to

Original: connect
Revised: connect

Original: them to the graph nodes.
Revised: them to the graph nodes.

Original: Consider
Revised: Consider

Original: Figure
Revised: Figure

Original: In a nutshell,
Revised: In a nutshell,

Original: we
Revised: we

Original: use a graph clustering
Revised: use a graph clustering

Original: algorithm
Revised: algorithm

Original: to
Revised: to

Original: determine groups of nodes in the
Revised: determine groups of nodes in the

Original: graph
Revised: graph

Original: that
Revised: that

Original: belong together
Revised: belong together

Original: and
Revised: and

Original: then
Revised: then

Original: connect these nodes to a common virtual node.
Revised: connect these nodes to a common virtual node.

Original: In this way
Revised: In this way

Original: , under-reaching is decreased because clustered
Revised: , under-reaching is decreased because clustered

Original: nodes can share information easily; at the same time, the nodes are spared of unnecessary information from unrelated nodes (i.e., in contrast to the single virtual node model).
Revised: nodes can share information easily; at the same time, the nodes are spared of unnecessary information from unrelated nodes (i.e., in contrast to the single virtual node model).

Original: • We also investigate alternative methods to determine the virtual node connections (e.g., randomization in clustering) and compare to the original model with a single virtual node.
Revised: • We also investigate alternative methods to determine the virtual node connections (e.g., randomization in clustering) and compare to the original model with a single virtual node.

Original: • We theoretically investigate the beneﬁt of using (multiple) virtual nodes in terms of two aspects: inﬂuence score and the expressiveness in learning a structural link representation.
Revised: • We theoretically investigate the beneﬁt of using (multiple) virtual nodes in terms of two aspects: inﬂuence score and the expressiveness in learning a structural link representation.

Original: • We conducted extensive experiments over challenging datasets of different type, provide ablation studies that conﬁrm the superiority of our proposed techniques, analyze the results in detail, and provide ﬁrst guidelines about how to use virtual nodes with different types of data and GNNs .
Revised: • We conducted extensive experiments over challenging datasets of different type, provide ablation studies that conﬁrm the superiority of our proposed techniques, analyze the results in detail, and provide ﬁrst guidelines about how to use virtual nodes with different types of data and GNNs .

Original: • Most importantly, we show that our virtual node extensions most often yield rather stable performance increases and allow standard GNNs to compete with complex state-of-the-art models that also try to improve message passing, as well as with the models leading the OGB leaderboards.
Revised: • Most importantly, we show that our virtual node extensions most often yield rather stable performance increases and allow standard GNNs to compete with complex state-of-the-art models that also try to improve message passing, as well as with the models leading the OGB leaderboards.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We give an overview on approaches that are similar from a technical perspective; for a more detailed summary, see Appendix A. For a more general overview of the large and diverse ﬁeld of link prediction, we refer to good summaries in recent works (Martínez et al., 2016; Zhang et al., 2020).
Revised: We give an overview on approaches that are similar from a technical perspective; for a more detailed summary, see Appendix A. For a more general overview of the large and diverse ﬁeld of link prediction, we refer to good summaries in recent works (Martínez et al., 2016; Zhang et al., 2020).

Original: Deeper GNNs.
Revised: Deeper GNNs.

Original: Several techniques address over-smoothing and hence allow for constructing deeper
Revised: Several techniques address over-smoothing and hence allow for constructing deeper

Original: GNNs to solve under-reaching.
Revised: GNNs to solve under-reaching.

Original: These models range from the simple but efﬁcient message propagation in SGC (Wu et al., 2019;
Revised: These models range from the simple but efﬁcient message propagation in SGC (Wu et al., 2019;

Original: Liu et al., 2020) and APPNP (Klicpera et al., 2019a) and connections in
Revised: Liu et al., 2020) and APPNP (Klicpera et al., 2019a) and connections in

Original: JKNet (Xu et al., 2018), to more advanced proposals (Chen et al., 2020; Sun et al., 2021;
Revised: JKNet (Xu et al., 2018), to more advanced proposals (Chen et al., 2020; Sun et al., 2021;

Original: Zhou et al., 2020; Li et al., 2020a) such as the differentiable aggregation functions in DeeperGCN (Li et al., 2020a).
Revised: Zhou et al., 2020; Li et al., 2020a) such as the differentiable aggregation functions in DeeperGCN (Li et al., 2020a).

Original: However, although (Chen et al., 2020) show that over-smoothing happens particularly in dense graphs, the experiments in most of these works consider citation or recommendation networks, but not the especially dense and important biomedical ones.
Revised: However, although (Chen et al., 2020) show that over-smoothing happens particularly in dense graphs, the experiments in most of these works consider citation or recommendation networks, but not the especially dense and important biomedical ones.

Original: And our experiments over the latter suggest that the reported results are not generalizable to the more challenging biomedical data.
Revised: And our experiments over the latter suggest that the reported results are not generalizable to the more challenging biomedical data.

Original: Beyond One-Hop Neighbors.
Revised: Beyond One-Hop Neighbors.

Original: Recently, graph diffusion methods are used in various ways to determine the message targets and thus extend standard message passing beyond the one-hop neighborhood.
Revised: Recently, graph diffusion methods are used in various ways to determine the message targets and thus extend standard message passing beyond the one-hop neighborhood.

Original: Atwood & Towsley (2016) use k-hop random walks to extend the node features.
Revised: Atwood & Towsley (2016) use k-hop random walks to extend the node features.

Original: APPNP (Klicpera et al., 2019a) applies personalized PageRank to propagate the node predictions generated by a neural network.
Revised: APPNP (Klicpera et al., 2019a) applies personalized PageRank to propagate the node predictions generated by a neural network.

Original: Other models concatenate (Abu-El-Haija et al., 2019) or aggregate (Xu et al., 2019a; Ma et al., 2020) node embeddings in every layer using a diffusion-based transition matrix.
Revised: Other models concatenate (Abu-El-Haija et al., 2019) or aggregate (Xu et al., 2019a; Ma et al., 2020) node embeddings in every layer using a diffusion-based transition matrix.

Original: The diffusion-based graph neural network (GDC) (Klicpera et al., 2019b) aggregates information from multiple neighborhood hops at each layer by sparsifying a generalized form of graph diffusion.
Revised: The diffusion-based graph neural network (GDC) (Klicpera et al., 2019b) aggregates information from multiple neighborhood hops at each layer by sparsifying a generalized form of graph diffusion.

Original: Subsequent works use diffusion methods on multiple scales (Liao et al., 2019; Luan et al., 2019;
Revised: Subsequent works use diffusion methods on multiple scales (Liao et al., 2019; Luan et al., 2019;

Original: Xhonneux et al., 2020) and attention (Wang et al., 2020).
Revised: Xhonneux et al., 2020) and attention (Wang et al., 2020).

Original: Morris et al.
Revised: Morris et al.

Original: (2019) take higher-order graph structures at multiple scales into account during message passing based on the k -dimensional
Revised: (2019) take higher-order graph structures at multiple scales into account during message passing based on the k -dimensional

Original: Weisfeiler and Leman graph algorithm.
Revised: Weisfeiler and Leman graph algorithm.

Original: All the above approaches are relatively complex, many terminated with memory errors in our experiments, and few have been evaluated for link prediction.
Revised: All the above approaches are relatively complex, many terminated with memory errors in our experiments, and few have been evaluated for link prediction.

Original: Virtual Nodes.
Revised: Virtual Nodes.

Original: To the best of our knowledge, virtual nodes have only been considered in the context of graph classiﬁcation so far, where a single virtual node (also called supernode ) is added to the graph to be classiﬁed and connected to all graph nodes (Gilmer et al., 2017; Li et al., 2017; Pham et al., 2017; Ishiguro et al., 2019).
Revised: To the best of our knowledge, virtual nodes have only been considered in the context of graph classiﬁcation so far, where a single virtual node (also called supernode ) is added to the graph to be classiﬁed and connected to all graph nodes (Gilmer et al., 2017; Li et al., 2017; Pham et al., 2017; Ishiguro et al., 2019).

Original: Note that the original idea was to compute a graph embedding in parallel with the node embeddings and even connected the virtual node only in one direction (i.e, via edges from the graph nodes) instead of bidirectionally (Li et al., 2017).
Revised: Note that the original idea was to compute a graph embedding in parallel with the node embeddings and even connected the virtual node only in one direction (i.e, via edges from the graph nodes) instead of bidirectionally (Li et al., 2017).

Original: There are some GNNs which point out special nodes that we could consider as “virtual”.
Revised: There are some GNNs which point out special nodes that we could consider as “virtual”.

Original: Fey et al.
Revised: Fey et al.

Original: (2020) propose a GNN for molecule graph classiﬁcation which clusters certain nodes within a molecule using a structure-based, molecule-speciﬁc algorithm and then applies message passing within and between these clusters.
Revised: (2020) propose a GNN for molecule graph classiﬁcation which clusters certain nodes within a molecule using a structure-based, molecule-speciﬁc algorithm and then applies message passing within and between these clusters.

Original: The graph-partition based message passing from Liao et al.
Revised: The graph-partition based message passing from Liao et al.

Original: (2018) also used clustering, but it just divides the original messages into inter- and intra-cluster.
Revised: (2018) also used clustering, but it just divides the original messages into inter- and intra-cluster.

Original: Our approach creates new “paths” in the graph and we theoretically demonstrate its expressiveness.
Revised: Our approach creates new “paths” in the graph and we theoretically demonstrate its expressiveness.

Original: P-GNN (You et al., 2019) assigns nodes to random clusters (“anchor-sets”) and then creates a message for each node for every anchor-set, while ignoring the message passing from original direct neighbors.
Revised: P-GNN (You et al., 2019) assigns nodes to random clusters (“anchor-sets”) and then creates a message for each node for every anchor-set, while ignoring the message passing from original direct neighbors.

Original: Our virtual nodes represent an alternative means to aggregate messages from multiple graph nodes which are not necessarily direct neighbors.
Revised: Our virtual nodes represent an alternative means to aggregate messages from multiple graph nodes which are not necessarily direct neighbors.

Original: We also explore the idea of similar random assignments in our context, but show that more elaborate techniques generally work better.
Revised: We also explore the idea of similar random assignments in our context, but show that more elaborate techniques generally work better.

Original: Most importantly, we do not propose a speciﬁc, new GNN but a new technique for augmenting existing graph neural networks .
Revised: Most importantly, we do not propose a speciﬁc, new GNN but a new technique for augmenting existing graph neural networks .

Original: Although it is a well-known trick, the advantage of using virtual nodes has never been theoretically investigated nor fully understood.
Revised: Although it is a well-known trick, the advantage of using virtual nodes has never been theoretically investigated nor fully understood.

Original: We focus on link prediction and considerably extend the virtual node technique.
Revised: We focus on link prediction and considerably extend the virtual node technique.

Original: There are commonalities in the advantages of using virtual nodes for graph classiﬁcation and link prediction, but their role in link prediction is to improve the representation of the link instead of the graph (nodes).
Revised: There are commonalities in the advantages of using virtual nodes for graph classiﬁcation and link prediction, but their role in link prediction is to improve the representation of the link instead of the graph (nodes).

Original: We analyze theoretically and empirically how they improve GNN performance.
Revised: We analyze theoretically and empirically how they improve GNN performance.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Link Prediction.
Revised: Link Prediction.

Original: We consider an undirected graph G =
Revised: We consider an undirected graph G =

Original: ( V, E ) with nodes V and edges E ⊆
Revised: ( V, E ) with nodes V and edges E ⊆

Original: V × V .
Revised: V × V .

Original: Note that this basic choice is only for ease of presentation.
Revised: Note that this basic choice is only for ease of presentation.

Original: All our techniques work for directed graphs and, with simple adaptation, also for graphs with labelled edges.
Revised: All our techniques work for directed graphs and, with simple adaptation, also for graphs with labelled edges.

Original: We assume V to be ordered and may refer to a node by its index in V .
Revised: We assume V to be ordered and may refer to a node by its index in V .

Original: For a node v ∈ V , N v denotes the set of its neighbors.
Revised: For a node v ∈ V , N v denotes the set of its neighbors.

Original: Given two nodes, the link prediction task is to predict whether there is a link between them.
Revised: Given two nodes, the link prediction task is to predict whether there is a link between them.

Original: Message-Passing Graph Neural Networks.
Revised: Message-Passing Graph Neural Networks.

Original: In this paper, we usually use the term graph neural networks (GNNs) to denote GNNs that use message passing as described by Gilmer et al.
Revised: In this paper, we usually use the term graph neural networks (GNNs) to denote GNNs that use message passing as described by Gilmer et al.

Original: These networks compute for every v ∈ V a node representation h (cid:96)v at layer (cid:96) ∈ [1 , 2 , . . .
Revised: These networks compute for every v ∈ V a node representation h (cid:96)v at layer (cid:96) ∈ [1 , 2 , . . .

Original: , k ] , by aggregating its neighbor nodes based on a generic aggregation function and then combine the obtained vector with h (cid:96) − 1 v as below; h 0 v are the initial node features.
Revised: , k ] , by aggregating its neighbor nodes based on a generic aggregation function and then combine the obtained vector with h (cid:96) − 1 v as below; h 0 v are the initial node features.

Original: [Equation]
Revised: [Equation]

Original: Link prediction with GNNs is usually done by combining (e.g., concatenating) the ﬁnal representations h Lu , h Lv , of the nodes u, v under consideration and passing them through several feed-forward layers with a ﬁnal sigmoid function for scoring.
Revised: Link prediction with GNNs is usually done by combining (e.g., concatenating) the ﬁnal representations h Lu , h Lv , of the nodes u, v under consideration and passing them through several feed-forward layers with a ﬁnal sigmoid function for scoring.

Original: We follow this approach.
Revised: We follow this approach.

Original: We further use [1 , n ] to denote an interval [1,2,..., n].
Revised: We further use [1 , n ] to denote an interval [1,2,..., n].

Original:  V IRTUAL N ODES IN G RAPH N EURAL N ETWORKS FOR L INK P REDICTION
Revised:  V IRTUAL N ODES IN G RAPH N EURAL N ETWORKS FOR L INK P REDICTION

Original: So far, virtual nodes have been only used for graph classiﬁcation.
Revised: So far, virtual nodes have been only used for graph classiﬁcation.

Original: Link prediction scenarios are different in that the graphs are usually very large, heterogeneous, sometimes dense, and the task is to predict a relationship that might strongly  inﬂuenced depend on surrounding relations.
Revised: Link prediction scenarios are different in that the graphs are usually very large, heterogeneous, sometimes dense, and the task is to predict a relationship that might strongly be inﬂuenced depending on surrounding relations.

Original: In the following, we propose approaches that ﬁt these scenarios.
Revised: In the following, we propose approaches that ﬁt these scenarios.

Original: 4. M ULTIPLE V IRTUAL N ODES
Revised: 4. M ULTIPLE V IRTUAL N ODES

Original: Our main goal of using virtual nodes is to provide a shortcut for sharing information between the graph nodes.
Revised: Our main goal of using virtual nodes is to provide a shortcut for sharing information between the graph nodes.

Original: However, the amount of information in a graph with possibly millions of nodes is enormous, and likely too much to be captured in a single virtual node embedding.
Revised: However, the amount of information in a graph with possibly millions of nodes is enormous, and likely too much to be captured in a single virtual node embedding.

Original: Further, not all information is equally relevant to all nodes.
Revised: Further, not all information is equally relevant to all nodes.

Original: Therefore we suggest to use multiple virtual nodes S = { s 1 , s 2 . . .
Revised: Therefore we suggest to use multiple virtual nodes S = { s 1 , s 2 . . .

Original: , s n }each being connected to a subset of graph nodes, as determined by an assignment σ : V → [1 , n ] ; n is considered as hyperparameter.
Revised: , s n }each being connected to a subset of graph nodes, as determined by an assignment σ : V → [1 , n ] ; n is considered as hyperparameter.

Original: We propose different methods to obtain σ :
Revised: We propose different methods to obtain σ :

Original: Random (GNN-RM).
Revised: Random (GNN-RM).

Original: Most simple, we can determine a ﬁxed σ randomly once with initialization.
Revised: Most simple, we can determine a ﬁxed σ randomly once with initialization.

Original: Increased Randomness (GNN-RM F ).
Revised: Increased Randomness (GNN-RM F ).

Original: Similarly a random assignment, but initialized with every forward pass.
Revised: Similarly a random assignment, but initialized with every forward pass.

Original: In this way, a single bad assignment does not determine the overall performance.
Revised: In this way, a single bad assignment does not determine the overall performance.

Original: Clustering (GNN-CM).
Revised: Clustering (GNN-CM).

Original: Many types of graph data incorporate a certain cluster structure (e.g., collaboration or social networks) that reﬂects which nodes belong closely together.
Revised: Many types of graph data incorporate a certain cluster structure (e.g., collaboration or social networks) that reﬂects which nodes belong closely together.

Original: We propose to connect such nodes in a cluster to a common virtual node, such that the structure inherent to the given graph is reﬂected in our virtual node assignment σ .
Revised: We propose to connect such nodes in a cluster to a common virtual node, such that the structure inherent to the given graph is reﬂected in our virtual node assignment σ .

Original: More precisely, during initialization, we use a generic clustering algorithm which, given a number m , creates a set C
Revised: More precisely, during initialization, we use a generic clustering algorithm which, given a number m , creates a set C

Original: = { C 1 , C 2 . . .
Revised: = { C 1 , C 2 . . .

Original: C m } of clusters (i.e., sets of graph nodes) by computing an assignment ρ : V → [1 , m ] , assigning each graph node to a cluster.
Revised: C m } of clusters (i.e., sets of graph nodes) by computing an assignment ρ : V → [1 , m ] , assigning each graph node to a cluster.

Original: We then obtain σ by choosing m = n and σ = ρ .
Revised: We then obtain σ by choosing m = n and σ = ρ .

Original: In this work, we decided for the METIS clustering (Karypis & Kumar, 1998) which turned out to provide a good trade off between quality and efﬁciency.
Revised: In this work, we decided for the METIS clustering (Karypis & Kumar, 1998) which turned out to provide a good trade off between quality and efﬁciency.

Original: Nevertheless, our idea is generic and can be applied with arbitrary algorithms.
Revised: Nevertheless, our idea is generic and can be applied with arbitrary algorithms.

Original: We will show ablation experiments for alternatives (e.g., Graclus (Dhillon et al., 2007) and Diffpool (Ying et al., 2018b)).
Revised: We will show ablation experiments for alternatives (e.g., Graclus (Dhillon et al., 2007) and Diffpool (Ying et al., 2018b)).

Original: Advanced Clustering (GNN-CM + ).
Revised: Advanced Clustering (GNN-CM + ).

Original: Not every type of graph data contains an inherent cluster structure or one that is sufﬁciently expressed.
Revised: Not every type of graph data contains an inherent cluster structure or one that is sufﬁciently expressed.

Original: Furthermore, using a ﬁxed clustering, we obtain a deterministic algorithm again taking the risk that we completely rely on a single, possibly not ideal, virtual node assignment – there may be critical long range dependencies that go beyond clusters.
Revised: Furthermore, using a ﬁxed clustering, we obtain a deterministic algorithm again taking the risk that we completely rely on a single, possibly not ideal, virtual node assignment – there may be critical long range dependencies that go beyond clusters.

Original: For these cases, we propose an alternative approach, which breaks up the determinism by extending the above clustering as follows.
Revised: For these cases, we propose an alternative approach, which breaks up the determinism by extending the above clustering as follows.

Original: We choose a relatively large m , with m (cid:29) n , and apply the above clustering algorithm, during initialization.
Revised: We choose a relatively large m , with m (cid:29) n , and apply the above clustering algorithm, during initialization.

Original: Then, in each epoch, we randomly guess an assignment σ (cid:48) :
Revised: Then, in each epoch, we randomly guess an assignment σ (cid:48) :

Original: [1 , m ] → [1 , n ] of clusters to virtual nodes and deﬁne σ ( n ) :
Revised: [1 , m ] → [1 , n ] of clusters to virtual nodes and deﬁne σ ( n ) :

Original: = σ (cid:48) ( ρ ( n )) .
Revised: = σ (cid:48) ( ρ ( n )) .

Original: Note that this approach is inspired by Chiang et al.
Revised: Note that this approach is inspired by Chiang et al.

Original: (2019), who apply a similar technique is to create batches based on clusters.
Revised: (2019), who apply a similar technique is to create batches based on clusters.

Original: Further note that we determine σ (cid:48) with every epoch instead of every forward pass since the the computation takes quite some time on large datasets and we observed that this yields good results.
Revised: Further note that we determine σ (cid:48) with every epoch instead of every forward pass since the the computation takes quite some time on large datasets and we observed that this yields good results.

Original: 4. T HE M ODEL
Revised: 4. T HE M ODEL

Original: We integrate the multiple virtual nodes into a generic message-passing graph neural network by extending the approach from Hu et al.
Revised: We integrate the multiple virtual nodes into a generic message-passing graph neural network by extending the approach from Hu et al.

Original: (2020) to the setting with multiple virtual nodes, by computing node representations h (cid:96)v for a node v ∈ V at layer (cid:96) as follows:
Revised: (2020) to the setting with multiple virtual nodes, by computing node representations h (cid:96)v for a node v ∈ V at layer (cid:96) as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that the highlighted adaptation of the standard GNN processing from Equation (1) is only minor – but powerful.
Revised: Note that the highlighted adaptation of the standard GNN processing from Equation (1) is only minor – but powerful.

Original: In our implementation, COMBINE (cid:96) VN is addition combined with linear layers and layer normalization, and we use sum for AGGREGATE (cid:96) VN .
Revised: In our implementation, COMBINE (cid:96) VN is addition combined with linear layers and layer normalization, and we use sum for AGGREGATE (cid:96) VN .

Original: 4. A NALYSIS : V IRTUAL N ODES C HANGE I NFLUENCE
Revised: 4. A NALYSIS : V IRTUAL N ODES C HANGE I NFLUENCE

Original: Inﬂuence Score.
Revised: Inﬂuence Score.

Original: Following (Xu et al., 2018; Klicpera et al., 2019a), we measure the sensitivity (also, inﬂuence ) of a node x to a node y by the inﬂuence score I ( x, y ) =
Revised: Following (Xu et al., 2018; Klicpera et al., 2019a), we measure the sensitivity (also, inﬂuence ) of a node x to a node y by the inﬂuence score I ( x, y ) =

Original: e T
Revised: e T

Original: ∂h kx
Revised: ∂h kx

Original: ∂h 

Revised: ∂h

Original: y ; e is a vector of all ones, h kx is the embedding of x at the k th layer, see Equations (1) and (3).
Revised: y ; e is a vector of all ones, h kx is the embedding of x at the k th layer, see Equations (1) and (3).

Original: For a k -layer GNN, the inﬂuence score is known to be proportional in expectation to the k -step random walk distribution from x to y :
Revised: For a k -layer GNN, the inﬂuence score is known to be proportional in expectation to the k -step random walk distribution from x to y :

Original: [Equation]
Revised: [Equation]

Original: r ∈ R
Revised: r ∈ R

Original: k
Revised: k

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: (cid:96) =  deg (
Revised: (cid:96) =  deg (

Original: [Equation]
Revised: [Equation]

Original: (cid:96)r
Revised: (cid:96)r

Original: [Equation]
Revised: [Equation]

Original: (4) ( v 0 r , v 1 r , ..., v kr ) are the nodes in the path r from x := v 0 r to y := v kr , R k is the set of paths of length k .
Revised: (4) ( v 0 r , v 1 r , ..., v kr ) are the nodes in the path r from x := v 0 r to y := v kr , R k is the set of paths of length k .

Original: In what follows, we will exploit this relationship and argument in terms of the probability P rw .
Revised: In what follows, we will exploit this relationship and argument in terms of the probability P rw .

Original: Virtual Nodes.
Revised: Virtual Nodes.

Original: For simplicity, consider the inﬂuence score in an m -regular graph; there we have
Revised: For simplicity, consider the inﬂuence score in an m -regular graph; there we have

Original: P rw ( x → y, k ) =
Revised: P rw ( x → y, k ) =

Original: | R k | m k .
Revised: | R k | m k .

Original: We hypothesize that we can come to similar conclusions in a general graph with average degree m .
Revised: We hypothesize that we can come to similar conclusions in a general graph with average degree m .

Original: Consider the message passing between two distant nodes x and y .
Revised: Consider the message passing between two distant nodes x and y .

Original: (I)
Revised: (I)

Original: In case the shortest path from x to y is of length >
Revised: In case the shortest path from x to y is of length >

Original: k , a k -layer GNN cannot capture it, and the probability
Revised: k , a k -layer GNN cannot capture it, and the probability

Original: P rw ( x → y, k ) is obviously zero.
Revised: P rw ( x → y, k ) is obviously zero.

Original: If we then consider virtual nodes in the GNN layer (even with only one), we can pass messages from x to y through the virtual nodes and obtain a nonzero probability.
Revised: If we then consider virtual nodes in the GNN layer (even with only one), we can pass messages from x to y through the virtual nodes and obtain a nonzero probability.

Original: (II) Consider the case where there is a shortest path of length ≤ k between x and y .
Revised: (II) Consider the case where there is a shortest path of length ≤ k between x and y .

Original: By adding a
Revised: By adding a

Original: virtual node s
Revised: virtual node s

Original: in one GNN layer, the probability changes to:
Revised: in one GNN layer, the probability changes to:

Original: [Equation]
Revised: [Equation]

Original: Compared to the original probability, we get the following impact ratio for using virtual nodes:
Revised: Compared to the original probability, we get the following impact ratio for using virtual nodes:

Original: [Equation]
Revised: [Equation]

Original: When m is large enough, ir can be approximated by ir (cid:39) (cid:16) 1 + m k
Revised: When m is large enough, ir can be approximated by ir (cid:39) (cid:16) 1 + m k

Original: [Equation]
Revised: [Equation]

Original: 1 | V
Revised: 1 | V

Original: || R k | (cid:17) .
Revised: || R k | (cid:17) .

Original: Here, we see that the impact of virtual nodes grows when m increases.
Revised: Here, we see that the impact of virtual nodes grows when m increases.

Original: Our experiments conﬁrm this theoretical observation.
Revised: Our experiments conﬁrm this theoretical observation.

Original: Multiple Virtual Nodes.
Revised: Multiple Virtual Nodes.

Original: In view of multiple virtual nodes, the above analysis gets even more appealing.
Revised: In view of multiple virtual nodes, the above analysis gets even more appealing.

Original: We continue along these lines and assume there is a shortest path of length ≤ k between x and y .
Revised: We continue along these lines and assume there is a shortest path of length ≤ k between x and y .

Original: If x and y connect to the same virtual node s , then Equation (5) changes as follows:
Revised: If x and y connect to the same virtual node s , then Equation (5) changes as follows:

Original: [Equation]
Revised: [Equation]

Original: Since the set C s of nodes connecting to s is much smaller than V , the impact of multiple virtual nodes is greater than that of a single virtual node.
Revised: Since the set C s of nodes connecting to s is much smaller than V , the impact of multiple virtual nodes is greater than that of a single virtual node.

Original: On the other hand, if x and y do not connect to the same virtual node, the probability just slightly decreases from | R k | m k to | R k | ( m +1) k .
Revised: On the other hand, if x and y do not connect to the same virtual node, the probability just slightly decreases from | R k | m k to | R k | ( m +1) k .

Original: In Appendix B, we further show that using multiple virtual nodes is related to (but not equal to) the labeling trick (Zhang et al., 2020) and distance encoding (Li et al., 2020b), and it can theoretically improve the expressiveness in learning structural link representations (see Theorem 1 and Figure 4(b)).
Revised: In Appendix B, we further show that using multiple virtual nodes is related to (but not equal to) the labeling trick (Zhang et al., 2020) and distance encoding (Li et al., 2020b), and it can theoretically improve the expressiveness in learning structural link representations (see Theorem 1 and Figure 4(b)).

Original:  E VALUATION
Revised:  E VALUATION

Original: We conducted extensive experiments and ablation studies to empirically investigate:
Revised: We conducted extensive experiments and ablation studies to empirically investigate:

Original: • How does the existing approach with one virtual node perform in link prediction ?
Revised: • How does the existing approach with one virtual node perform in link prediction ?

Original: • Do multiple virtual nodes improve performance , how do our proposed approaches compare?
Revised: • Do multiple virtual nodes improve performance , how do our proposed approaches compare?

Original: • In particular, are approaches based on the graph structure better ?
Revised: • In particular, are approaches based on the graph structure better ?

Original: • How exactly do virtual nodes support link prediction?
Revised: • How exactly do virtual nodes support link prediction?

Original: When do they help particularly?
Revised: When do they help particularly?

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: - CM +
Revised: - CM +

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Datasets.
Revised: Datasets.

Original: We focused on challenging data from the OGB: ddi , a drug-drug interaction network; ppa10 , a subset of the protein-protein association network ppa containing only 10% of the train edges (but full valid/test); and collab , an author collaboration network.
Revised: We focused on challenging data from the OGB: ddi , a drug-drug interaction network; ppa10 , a subset of the protein-protein association network ppa containing only 10% of the train edges (but full valid/test); and collab , an author collaboration network.

Original: To learn more about smaller data of similar type, we also tested on the citation networks pubmed (Yang et al., 2016).
Revised: To learn more about smaller data of similar type, we also tested on the citation networks pubmed (Yang et al., 2016).

Original: Since the datasets are not only very different in type but also in various other critical graph parameters and this is reﬂected in the performance of the models, we show relevant statistics in Table 1. 

Revised: Since the datasets are not only very different in type but also in various other critical graph parameters and this is reﬂected in the performance of the models, we show relevant statistics in Table 1.

Original: The datasets vary strongly in size with ddi being smallest among the biomedical; on the other hand, ddi is very dense.
Revised: The datasets vary strongly in size with ddi being smallest among the biomedical; on the other hand, ddi is very dense.

Original: The clustering coefﬁcient intuitively reﬂects the “cliquishness” of the graph’s subgraphs.
Revised: The clustering coefﬁcient intuitively reﬂects the “cliquishness” of the graph’s subgraphs.

Original: The large diameters suggest that the data suits testing under-reaching.
Revised: The large diameters suggest that the data suits testing under-reaching.

Original: Appendix C gives further details and describes datsets we consider in additional experiments in the appendix.
Revised: Appendix C gives further details and describes datsets we consider in additional experiments in the appendix.

Original: Baselines For a competitive comparison, we considered important baselines (described in Section 2):
Revised: Baselines For a competitive comparison, we considered important baselines (described in Section 2):

Original: • The deep GNNs SGC , APPNP , DeeperGCN , and two variants of JKNet .
Revised: • The deep GNNs SGC , APPNP , DeeperGCN , and two variants of JKNet .

Original: • Approaches extending message passing: P-GNN , APPNP , GCN-GDC , SAGE-GDC , GIN-GDC .
Revised: • Approaches extending message passing: P-GNN , APPNP , GCN-GDC , SAGE-GDC , GIN-GDC .

Original: • The popular GNNs GCN (Kipf & Welling, 2017), SAGE (Hamilton et al., 2017)
Revised: • The popular GNNs GCN (Kipf & Welling, 2017), SAGE (Hamilton et al., 2017)

Original: and GIN (Xu et al., 2019b), which we then extend with (multiple) virtual nodes.
Revised: and GIN (Xu et al., 2019b), which we then extend with (multiple) virtual nodes.

Original: [Table]
Revised: [Table]

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Overall Impact of Virtual Nodes, Tables 2, 7, 8 (Appendix).
Revised: Overall Impact of Virtual Nodes, Tables 2, 7, 8 (Appendix).

Original: We compare to GCN, SAGE, and
Revised: We compare to GCN, SAGE, and

Original: GIN.
Revised: GIN.

Original: The common approach of using a single virtual node (GNN-VN) yields good improvements over ddi , slight improvements over collab , but no deﬁnitive ones over ppa10 ; over pubmed , it works very well for GIN.
Revised: The common approach of using a single virtual node (GNN-VN) yields good improvements over ddi , slight improvements over collab , but no deﬁnitive ones over ppa10 ; over pubmed , it works very well for GIN.

Original: The numbers for GNN-RM and GNN-RM F reﬂect the randomness of their connections to the virtual nodes, there is no clear trend.
Revised: The numbers for GNN-RM and GNN-RM F reﬂect the randomness of their connections to the virtual nodes, there is no clear trend.

Original: Nevertheless, they clearly outperform the original models, with only few exceptions.
Revised: Nevertheless, they clearly outperform the original models, with only few exceptions.

Original: The increased randomness by re-assigning the virtual nodes with every forward pass (GNN-RM F ) seemingly suits SAGE but not the others.
Revised: The increased randomness by re-assigning the virtual nodes with every forward pass (GNN-RM F ) seemingly suits SAGE but not the others.

Original: As expected, over the small pubmed / cora , which also have no cluster structure, the results are not consistent or convincing overall; virtual nodes only yield improvement sometimes, and none for GCN.
Revised: As expected, over the small pubmed / cora , which also have no cluster structure, the results are not consistent or convincing overall; virtual nodes only yield improvement sometimes, and none for GCN.

Original: Yet, on the more challenging datasets, multiple virtual nodes turn out to be an efﬁcient means to boost the link prediction performance of GNNs if they are applied correctly.
Revised: Yet, on the more challenging datasets, multiple virtual nodes turn out to be an efﬁcient means to boost the link prediction performance of GNNs if they are applied correctly.

Original: Our virtual node connections based on the graph structure (GNN-CM) yield consistently good improvements over ddi and collab , and mostly help on the challenging ppa10 dataset.
Revised: Our virtual node connections based on the graph structure (GNN-CM) yield consistently good improvements over ddi and collab , and mostly help on the challenging ppa10 dataset.

Original: On collab , we did further experiments using GAT (Veli ˇ ckovi ´ c et al., 2017) and also observe a clear performance gain: 0.4745 vs. 0.5876 (GAT-CM).
Revised: On collab , we did further experiments using GAT (Veli ˇ ckovi ´ c et al., 2017) and also observe a clear performance gain: 0.4745 vs. 0.5876 (GAT-CM).

Original: GNN-CM and GNN-CM + are not always the best ones, but yield reliably good results, in contrast to the other models with virtual nodes (see variability of gray shades).
Revised: GNN-CM and GNN-CM + are not always the best ones, but yield reliably good results, in contrast to the other models with virtual nodes (see variability of gray shades).

Original: Interestingly, the advanced clustering yields especially good performance over ppa10 / ppa , while its results on the other datasets are not convincing.
Revised: Interestingly, the advanced clustering yields especially good performance over ppa10 / ppa , while its results on the other datasets are not convincing.

Original: Generally, the improvements of the virtual node models are strongest on ddi .
Revised: Generally, the improvements of the virtual node models are strongest on ddi .

Original: For an in-depth result analysis see Section 5.2.
Revised: For an in-depth result analysis see Section 5.2.

Original: Comparison to Related Works and SOTA, Table 3.
Revised: Comparison to Related Works and SOTA, Table 3.

Original: Most deep GNNs as well as the models that use complex message-passing techniques perform disappointing and, overall, much worse than the standard GNNs.
Revised: Most deep GNNs as well as the models that use complex message-passing techniques perform disappointing and, overall, much worse than the standard GNNs.

Original: We did thorough hyperparameter tuning for these models and it is hard to explain.
Revised: We did thorough hyperparameter tuning for these models and it is hard to explain.

Original: However, most of the original evaluations focus on node or graph classiﬁcation and consider very different types of data – often the standard citation networks (Lu & Getoor, 2003) – and, in fact, on collab we see the best numbers.
Revised: However, most of the original evaluations focus on node or graph classiﬁcation and consider very different types of data – often the standard citation networks (Lu & Getoor, 2003) – and, in fact, on collab we see the best numbers.

Original: For a more detailed discussion regarding P-GNN see Appendix H. Even if we assume that these numbers can be improved, the models do not seem apt for link prediction; in particular, the complex ones: many do not run at all on realistic link prediction data but yield memory errors.
Revised: For a more detailed discussion of P-GNN see Appendix H. Even if we assume that these numbers can be improved, the models do not seem apt for link prediction; in particular, the complex ones: many do not run at all on realistic link prediction data but yield memory errors.

Original: Further, our virtual node extensions make standard GNNs competitive to the models on the leaderboard .
Revised: Further, our virtual node extensions make standard GNNs competitive to the models on the leaderboard .

Original: In particular, their performance is much more stable.
Revised: In particular, their performance is much more stable.

Original: The results of the best models from the leaderboard vary strongly with the different datasets, or have not been reported at all.
Revised: The results of the best models from the leaderboard vary strongly with the different datasets, or have not been reported at all.

Original: None of these models can be called “good” overall, given the numbers in the rest of the table; in fact, SEAL and Adamic Adar perform rather bad on the very dense ddi .
Revised: None of these models can be called “good” overall, given the numbers in the - sometimes even missingrest of the table; in fact, SEAL and Adamic Adar perform rather bad on the very dense ddi .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Impact of Virtual Nodes on Number of GNN Layers and
Revised: Impact of Virtual Nodes on Number of GNN Layers and

Original: Efﬁciency, Figure 2.
Revised: Efﬁciency, Figure 2.

Original: For the virtual nodes models, the scores increase with the number of layers for a longer time, GCN drops earlier.
Revised: For the virtual nodes models, the scores increase with the number of layers for a longer time, GCN drops earlier.

Original: On ddi , GCN-VN and -CM reach their best scores at 6 and 8 layers, respectively, which is remarkable for that very dense dataset, which is prone to over-smoothing.
Revised: On ddi , GCN-VN and -CM reach their best scores at 6 and 8 layers, respectively, which is remarkable for that very dense dataset, which is prone to over-smoothing.

Original: On collab it is the other way around.
Revised: On collab it is the other way around.

Original: The ﬁgure also gives an idea about the runtime increase with using virtual nodes.
Revised: The ﬁgure also gives an idea about the runtime increase with using virtual nodes.

Original: It compares the 6-layer models, and shows the 4-layer GCN-CM which obtains performance similar to the 6-layer GCN-VN.
Revised: It compares the 6-layer models, and shows the 4-layer GCN-CM which obtains performance similar to the 6-layer GCN-VN.

Original: Impact of Virtual Node Number, Figure 3.
Revised: Impact of Virtual Node Number, Figure 3.

Original: First, consider the conﬁgurations of the best models for the overall results in
Revised: First, consider the conﬁgurations of the best models for the overall results in

Original: Table 2, which are provided in Table 6 in the appendix.
Revised: Table 2, which are provided in Table 6 in the appendix.

Original: Here, we see that the chosen numbers of virtual nodes are indeed random for the “random” models, but GNN-CM consistently uses a high number of virtual nodes, which also suits it better according to our theoretical analysis in Section 4.3.
Revised: Here, we see that the chosen numbers of virtual nodes are indeed random for the “random” models, but GNN-CM consistently uses a high number of virtual nodes, which also suits it better according to our theoretical analysis in Section 4.3.

Original: In line with this, the more detailed analysis varying the numbers of virtual nodes, yields best results (also in terms of standard deviations) for SAGE-CM at rather high values.
Revised: In line with this, the more detailed analysis varying the numbers of virtual nodes, yields best results (also in terms of standard deviations) for SAGE-CM at rather high values.

Original: In contrast, for GCN, wehere do not see a strong sensitivity or particular trend.
Revised: For GCN, we do not see a clear trend, but (second) best performance with 64 virtual nodes.

Original: Note that there is a trade off between number of virtual nodes and intra-cluster test edges, which we discuss in Section 5.2.
Revised: Note that there is a trade off between number of virtual nodes and intra-cluster test edges, discussed in Section 5.2.

Original: Using Virtual Nodes Only at the Last GNN Layer, Table 4.
Revised: Using Virtual Nodes Only at the Last GNN Layer, Table 4.

Original: Alon & Yahav (2021) show that using a fully connected adjacency matrix at the last layer of a standard GNN helps to better capture information over long ranges.
Revised: Alon & Yahav (2021) show that using a fully connected adjacency matrix at the last layer of a standard GNN helps to better capture information over long ranges.

Original: We therefore investigated if it is a better architectural choice to use virtual nodes only at the last layer.
Revised: We therefore investigated if it is a better architectural choice to use virtual nodes only at the last layer.

Original: However, we see that this can lead to extreme performance drops.
Revised: However, we see that this can lead to extreme performance drops.

Original: Impact of Clustering Algorithm, Table 9 (Appendix).
Revised: Impact of Clustering Algorithm, Table 9 (Appendix).

Original: Our architecture is generic in the clustering algorithm, and we investigated the effects of varying that.
Revised: Our architecture is generic in the clustering algorithm, and we investigated the effects of varying that.

Original: Graclus is similar in nature to METIS in that it also creates partitions based on the adjacency matrix, but it took much longer to run.
Revised: Graclus is similar in nature to METIS in that it also creates partitions based on the adjacency matrix, but it took much longer to run.

Original: Diffpool considers the node features and yields improvements for GCN
Revised: Diffpool considers the node features and yields improvements for GCN

Original: [Table]
Revised: [Table]

Original: but does not scale to larger datasets.
Revised: but does not scale to larger datasets.

Original: Over ddi , there is no clear winner and, given its efﬁciency, METIS turns out to be a good solution.
Revised: Over ddi , there is no clear winner and, given its efﬁciency, METIS turns out to be a good solution.

Original: [Table]
Revised: [Table]

Original: 5. D ISCUSSION AND C ONCLUSIONS
Revised: 5. D ISCUSSION AND C ONCLUSIONS

Original: The results show that our approach with multiple virtual nodes based on graph-based clustering yields performance increases for various GNNs and types of data, but there are clear differences.
Revised: The results show that our approach with multiple virtual nodes based on graph-based clustering yields performance increases for various GNNs and types of data, but there are clear differences.

Original: Dense Graphs with Medium/High Clustering Coefﬁcient.
Revised: Dense Graphs with Medium/High Clustering Coefﬁcient.

Original: Over ddi , we see strongest improvements for all virtual-node models.
Revised: Over ddi , we see strongest improvements for all virtual-node models.

Original: This can be explained by our proposed theory, showing that a very large node degree m increases the impact of the virtual node(s), and thus decreases the negative impact of the (too) many other neighbors (see Equation (6)).
Revised: This can be explained by our proposed theory, showing that a very large node degree m increases the impact of the virtual node(s), and thus decreases the negative impact of the (too) many other neighbors (see Equation (6)).

Original: Furthermore, the empirical results conﬁrm our proposed theory regarding multiple virtual nodes (see Equation (7)).
Revised: Furthermore, the empirical results conﬁrm our proposed theory regarding multiple virtual nodes (see Equation (7)).

Original: We see particularly good numbers for GNN-CM, which exploits the clustering inherent in the given graph.
Revised: We see particularly good numbers for GNN-CM, which exploits the clustering inherent in the given graph.

Original: GNN-CM + , which considers this given clustering only on a lower level, is shown to perform worse than GNN-CM overall.
Revised: GNN-CM + , which considers this given clustering only on a lower level, is shown to perform worse than GNN-CM overall.

Original: In fact, we computed the percentage of test edges that occur in the “virtual node cluster” (see
Revised: In fact, we computed the percentage of test edges that occur in the “virtual node cluster” (see

Original: Table 11 in the appendix) and it shows that the numbers for the advanced clustering are very similar to the random one, meaning the randomly merged smaller clusters break the data’s structure too much.
Revised: Table 11 in the appendix) and it shows that the numbers for the advanced clustering are very similar to the random one, meaning the randomly merged smaller clusters break the data’s structure too much.

Original: Interestingly, the experiments show that, even with the dense data that is prone to over-smoothing, virtual nodes make the GNNs score higher with more than the standard 2-3 layers; hence virtual nodes seem to alleviate over-smoothing to some extent, an interesting question for future work.
Revised: Interestingly, the experiments show that, even with the dense data that is prone to over-smoothing, virtual nodes make the GNNs score higher with more than the standard 2-3 layers; hence virtual nodes seem to alleviate over-smoothing to some extent, an interesting question for future work.

Original: Graphs with Large Problem Radius and Low Clustering Coefﬁcient.
Revised: Graphs with Large Problem Radius and Low Clustering Coefﬁcient.

Original: Over ppa10 , all GNNs use an unusually high number of layers, which hints at a large problem radius (e.g., GCN, which performs especially well, uses 7 layers).
Revised: Over ppa10 , all GNNs use an unusually high number of layers, which hints at a large problem radius (e.g., GCN, which performs especially well, uses 7 layers).

Original: Given the very low clustering of the data in addition, ppa represents a special challenge.
Revised: Given the very low clustering of the data in addition, ppa represents a special challenge.

Original: With the multiple virtual nodes, GNN-CM performs again better than
Revised: With the multiple virtual nodes, GNN-CM performs again better than

Original: GNN-VN.
Revised: GNN-VN.

Original: On the other hand, it does not perform much better than the random models on data without cluster structure.
Revised: On the other hand, it does not perform much better than the random models on data without cluster structure.

Original: This can be explained by its choice of number of virtual nodes, which is consistently high, but then there are less test edges within a virtual node cluster (see appendix Table 11).
Revised: This can be explained by its choice of number of virtual nodes, which is consistently high, but then there are less test edges within a virtual node cluster (see appendix Table 11).

Original: We hence see here that the positive effect of having many virtual nodes (recall Equation (7)) cancel out the beneﬁts of clustering.
Revised: We hence see here that the positive effect of having many virtual nodes (recall Equation (7)) cancel out the beneﬁts of clustering.

Original: Our advanced clustering, which merges some local clustering with randomness, is able to achieve best results with GCN and GIN (with SAGE, all models perform rather bad over ppa10 ).
Revised: Our advanced clustering, which merges some local clustering with randomness, is able to achieve best results with GCN and GIN (with SAGE, all models perform rather bad over ppa10 ).

Original: This can be explained by the fact that it randomly merges some local clusters – with each epoch anew – and hence allows more messages to pass across “virtual node clusters”.
Revised: This can be explained by the fact that it randomly merges some local clusters – with each epoch anew – and hence allows more messages to pass across “virtual node clusters”.

Original: We also did some experiments over the very large ppa , which is denser than ppa10 , and see a similar trend.
Revised: We also did some experiments over the very large ppa , which is denser than ppa10 , and see a similar trend.

Original: Sparse Graphs with Low to High Clustering Coefﬁcient.
Revised: Sparse Graphs with Low to High Clustering Coefﬁcient.

Original: We tested on three citation/collaboration networks of different sizes.
Revised: We tested on three citation/collaboration networks of different sizes.

Original: Note that, over this data, the problem radius is usually assumed to be rather small (Alon & Yahav, 2021), although the graph diameters are large.
Revised: Note that, over this data, the problem radius is usually assumed to be rather small (Alon & Yahav, 2021), although the graph diameters are large.

Revised: We investigated virtual nodes to augment link prediction in large and complex graphs; but we also want to provide insight into the behavior on smaller data.

Original: Overthe much smaller data pubmed (similarly on cora as we show in the appendix), virtual nodes do not provide any improvement for GCN.
Revised: Over pubmed (similarly on cora as shown in the appendix), virtual nodes do not provide any improvement for GCN.

Original: For GIN, a single virtual node yields good increases; overall, it usually outperforms the settings with multiple virtual nodes.
Revised: For GIN, a single virtual node yields good increases; overall, it usually outperforms the settings with multiple virtual nodes.

Original: We hypothesize that this is  due toa combination of the small graph size, the sparsity, and the missing cluster structure.
Revised: We hypothesize that this is mainly due to the small graph size and sparsity.

Original: In fact, collab is denser and we have clustering; accordingly, GNN-CM performs very good for all GNNs.
Revised: In fact, on the larger and denser collab , GNN-CM performs very good for all GNNs.

Original: The trends in the models’ performance and the corresponding explanations are similar to those for ddi but much less expressed, probably due to the much smaller node degrees.
Revised: The trends in the models’ performance and the corresponding explanations are similar to those for ddi but much less expressed, probably due to the much smaller node degrees.

Original: On the other hand, the performance is much more stable, possibly because it is larger and less dense.
Revised: Yet, the performance is much more stable, possibly because it is larger and not as dense.

Original: Conclusions.
Revised: Conclusions.

Original: We summarize our main ﬁndings to provide ﬁrst guidelines for applying virtual nodes:
Revised: We summarize our main ﬁndings to provide ﬁrst guidelines for applying virtual nodes:

Original: • Small + Sparse Graphs: A single virtual node is likely to boost performance of GIN, and virtual nodes should help with SAGE, but probably not with GCN.
Revised: • Small + Sparse Graphs: A single virtual node is likely to boost performance of GIN, and virtual nodes should help with SAGE, but probably not with GCN

Original: • Large + Sparse Graphs : If there is cluster structure, GNN-CM should yield stable performance increases.
Revised: • Large + Sparse Graphs : If there is cluster structure, GNN-CM should yield stable performance increases.

Original: If the problem radius is large or there is few cluster structure, GNN-CM + is worth a try.
Revised: If the problem radius is large or there is few cluster structure, GNN-CM + is worth a try.

Original: • Dense Graphs+ Clustering: Multiple virtual nodes ( i
Revised: • Dense Graphs

Original: .e. , GNN-CM) likely increase performance.
Revised: + Clustering: Multiple virtual nodes (i.e., GNN-CM) likely increase performance.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We propose a simple but elegant graph neural network extension using multiple virtual nodes that may considerably increase link prediction performance.
Revised: We propose a simple but elegant graph neural network extension using multiple virtual nodes that may considerably increase link prediction performance.

Original: We also advance research by providing theoretical justiﬁcations - the very ﬁrst about applying virtual nodes at all - and by showing their positive impact in various experiments.
Revised: We also advance research by providing theoretical justiﬁcations - the very ﬁrst about applying virtual nodes at all - and by showing their positive impact in various experiments.

Original: Future work includes the design of more advanced and scalable architectures, and it would be interesting to further investigate the huge performance increases on dense graphs.
Revised: Future work includes the design of more advanced and scalable architectures, and it would be interesting to further investigate the huge performance increases on dense graphs.

Original: A A DDITIONAL D ETAILS ON R ELATED W ORKS
Revised: A A DDITIONAL D ETAILS ON R ELATED W ORKS

Original: Deeper GNNs.
Revised: Deeper GNNs.

Original: We mention simpler approaches in the Section 2.
Revised: We mention simpler approaches in the Section 2.

Original: More advanced proposals are, for example, based on special features and connections (Chen et al., 2020), community-based normalization of node representations using random clustering (Zhou et al., 2020), boosting techniques (Sun et al., 2021), or differentiable aggregation functions in DeeperGCN (Li et al., 2020a).
Revised: More advanced proposals are, for example, based on special features and connections (Chen et al., 2020), community-based normalization of node representations using random clustering (Zhou et al., 2020), boosting techniques (Sun et al., 2021), or differentiable aggregation functions in DeeperGCN (Li et al., 2020a).

Original: Beyond One-Hop Neighbors.
Revised: Beyond One-Hop Neighbors.

Original: Graph diffusion methods (i.e., in graph theory, techniques for spreading information between nodes) are used in various ways to determine the message targets and thus extend standard message passing beyond the one-hop neighborhood.
Revised: Graph diffusion methods (i.e., in graph theory, techniques for spreading information between nodes) are used in various ways to determine the message targets and thus extend standard message passing beyond the one-hop neighborhood.

Original: (Atwood & Towsley, 2016) use k-hop random walks to aggregate node features and extend the latter by the aggregated ones.
Revised: (Atwood & Towsley, 2016) use k-hop random walks to aggregate node features and extend the latter by the aggregated ones.

Original: APPNP (Klicpera et al., 2019a) applies personalized PageRank to propagate the node predictions generated by a neural network.
Revised: APPNP (Klicpera et al., 2019a) applies personalized PageRank to propagate the node predictions generated by a neural network.

Original: Other models aggregate node embeddings in every layer, GraphHeat (Xu et al., 2019a) using the heat kernel, PAN (Ma et al., 2020)
Revised: Other models aggregate node embeddings in every layer, GraphHeat (Xu et al., 2019a) using the heat kernel, PAN (Ma et al., 2020)

Original: the transition matrix of maximal entropy random walks, and PinSage (Ying et al., 2018a) using random walks.
Revised: the transition matrix of maximal entropy random walks, and PinSage (Ying et al., 2018a) using random walks.

Original: (Abu-El-Haija et al., 2019) propose to concatenate embeddings aggregated using the transition matrices of k-hop random walks before applying one-hop neighbor aggregation.
Revised: (Abu-El-Haija et al., 2019) propose to concatenate embeddings aggregated using the transition matrices of k-hop random walks before applying one-hop neighbor aggregation.

Original: The diffusion-based graph neural network (GDC) (Klicpera et al., 2019b) aggregates information from multiple neighborhood hops at each layer by sparsifying a generalized form of graph diffusion.
Revised: The diffusion-based graph neural network (GDC) (Klicpera et al., 2019b) aggregates information from multiple neighborhood hops at each layer by sparsifying a generalized form of graph diffusion.

Original: Subsequent works use diffusion methods on multiple scales (Liao et al., 2019; Luan et al., 2019; Xhonneux et al., 2020).
Revised: Subsequent works use diffusion methods on multiple scales (Liao et al., 2019; Luan et al., 2019; Xhonneux et al., 2020).

Original: Recently, (Wang et al., 2020) integrated attention with diffusion-based message propagation.
Revised: Recently, (Wang et al., 2020) integrated attention with diffusion-based message propagation.

Original: Position Encodings.
Revised: Position Encodings.

Original: Our approach provides a kind of positional embedding (Srinivasan & Ribeiro, 2019) and hence has some commonalities with models extending nodes with positional encodings, e.g., (Li et al.).
Revised: Our approach provides a kind of positional embedding (Srinivasan & Ribeiro, 2019) and hence has some commonalities with models extending nodes with positional encodings, e.g., (Li et al.).

Original: B A DDITIONAL T HEORETICAL R ESULTS : S TRUCTURAL L INK R EPRESENTATION
Revised: B A DDITIONAL T HEORETICAL R ESULTS : S TRUCTURAL L INK R EPRESENTATION

Original: Adding structure-related features such as a distance encoding (Li et al., 2020b) has been demonstrated to make graph representation learning more powerful.
Revised: Adding structure-related features such as a distance encoding (Li et al., 2020b) has been demonstrated to make graph representation learning more powerful.

Original: For link prediction, (Zhang et al., 2020) propose the labeling trick extending distance encoding and making GNNs learn better link representations.
Revised: For link prediction, (Zhang et al., 2020) propose the labeling trick extending distance encoding and making GNNs learn better link representations.

Original: We ﬁrst recall the deﬁnitions from (Zhang et al., 2020) introducing the concept of labeling trick.
Revised: We ﬁrst recall the deﬁnitions from (Zhang et al., 2020) introducing the concept of labeling trick.

Original: Consider an undirected graph G as described in Section 3.
Revised: Consider an undirected graph G as described in Section 3.

Original: In addition, the tensor A ∈ R n × n × k contains all node and edge features (if available).
Revised: In addition, the tensor A ∈ R n × n × k contains all node and edge features (if available).

Original: The diagonal components
Revised: The diagonal components

Original: A v,v, : denote the node features, while the off-diagonal components A u,v, : denote the edge features of edge ( u, v ) .
Revised: A v,v, : denote the node features, while the off-diagonal components A u,v, : denote the edge features of edge ( u, v ) .

Original: The labeling trick uses a target node set S ⊆ V and a labeling function to label all nodes in the node set V and stack the labels with A .
Revised: The labeling trick uses a target node set S ⊆ V and a labeling function to label all nodes in the node set V and stack the labels with A .

Original: A valid labeling trick must meet two conditions: (1) the nodes in S have different labels from the rest of the nodes, (2) the labeling function must be permutation invariant.
Revised: A valid labeling trick must meet two conditions: (1) the nodes in S have different labels from the rest of the nodes, (2) the labeling function must be permutation invariant.

Original: Let us recall our method using multiple virtual nodes.
Revised: Let us recall our method using multiple virtual nodes.

Original: Assume we have multiple virtual nodes S
Revised: Assume we have multiple virtual nodes S

Original: = { s 1 , ..., s m } .
Revised: = { s 1 , ..., s m } .

Original: ∀ u ∈ V , we have the additional features for the node l ( u | S )
Revised: ∀ u ∈ V , we have the additional features for the node l ( u | S )

Original: = ( h ( s 1 ) , ..., h ( s m ))
Revised: = ( h ( s 1 ) , ..., h ( s m ))

Original: T ( γ ( u | s 1 ) , ..., γ ( u | s m )) , where γ ( u | s i )
Revised: T ( γ ( u | s 1 ) , ..., γ ( u | s m )) , where γ ( u | s i )

Original: = 1 if u is connected to the virtual node s i , and γ ( u | v i ) = 

Revised: = 1 if u is connected to the virtual node s i , and γ ( u | v i ) =

Original: otherwise.
Revised: otherwise.

Original: h ( s i ) is the node representation of virtual node s i , and is initialized by one-hot vectors so that each virtual node has different labels.
Revised: h ( s i ) is the node representation of virtual node s i , and is initialized by one-hot vectors so that each virtual node has different labels.

Original: Our labeling strategy is not a valid labeling trick by the deﬁnition of (Zhang et al., 2020).
Revised: Our labeling strategy is not a valid labeling trick by the deﬁnition of (Zhang et al., 2020).

Original: First, S is not a subset of V , and we use addition instead of concatenation.
Revised: First, S is not a subset of V , and we use addition instead of concatenation.

Original: Even if we extend V to V ∪ S , our labeling strategy still does not ﬁt the permutation-invariant requirement.
Revised: Even if we extend V to V ∪ S , our labeling strategy still does not ﬁt the permutation-invariant requirement.

Original: Nevertheless, it can achieve similar effects in learning structural link representations.
Revised: Nevertheless, it can achieve similar effects in learning structural link representations.

Original: Theorem 1.
Revised: Theorem 1.

Original: In any non-attributed graphs with n nodes, if the degree of each node in the graph is between 1 and O (log 1 − (cid:15) 2 h ( n )) for any constant (cid:15) > 0 , given m virtual nodes which evenly divide the node set into m clusters, then there exists ω (cid:0) ( m − 1) 2 ( n (cid:15) m − 1) 3 (cid:1) many pairs of non-isomorphic links ( u, w ) , ( v, w ) , such that an h -layer 1-WL-GNN (see deﬁnitions in (Li et al., 2020b) and (Zhang et al., 2020); one well-known example is GIN (Xu et al., 2019b)) gives u, v the same representation, while using m virtual nodes can give u, v different representations.
Revised: In any non-attributed graphs with n nodes, if the degree of each node in the graph is between 1 and O (log 1 − (cid:15) 2 h ( n )) for any constant (cid:15) > 0 , given m virtual nodes which evenly divide the node set into m clusters, then there exists ω (cid:0) ( m − 1) 2 ( n (cid:15) m − 1) 3 (cid:1) many pairs of non-isomorphic links ( u, w ) , ( v, w ) , such that an h -layer 1-WL-GNN (see deﬁnitions in (Li et al., 2020b) and (Zhang et al., 2020); one well-known example is GIN (Xu et al., 2019b)) gives u, v the same representation, while using m virtual nodes can give u, v different representations.

Original: Proof.
Revised: Proof.

Original: The proof can be separated into two steps.
Revised: The proof can be separated into two steps.

Original: The ﬁrst step is to prove that there exists n/o ( n 1 − (cid:15) )
Revised: The ﬁrst step is to prove that there exists n/o ( n 1 − (cid:15) )

Original: = ω ( n (cid:15) )
Revised: = ω ( n (cid:15) )

Original: many nodes that are locally h -isomorphic.
Revised: many nodes that are locally h -isomorphic.

Original: This step is same as the proof of
Revised: This step is same as the proof of

Original: Theorem 2 in (Zhang et al., 2020), so we omit the details here.
Revised: Theorem 2 in (Zhang et al., 2020), so we omit the details here.

Original: After getting these locally isomorphic nodes, we denote the set of these nodes as V iso .
Revised: After getting these locally isomorphic nodes, we denote the set of these nodes as V iso .

Original: The second step is to ﬁnd the non-isomorphic links.
Revised: The second step is to ﬁnd the non-isomorphic links.

Original: Step 2.
Revised: Step 2.

Original: Let us partition V iso = ∪ i =1 V i where V i is the subset of nodes connected to virtual node s i .
Revised: Let us partition V iso = ∪ i =1 V i where V i is the subset of nodes connected to virtual node s i .

Original: To be simple, we call each V i a cluster, and the sizes of different clusters are assumed to be the same |
Revised: To be simple, we call each V i a cluster, and the sizes of different clusters are assumed to be the same |

Original: V i | = | V iso | /m .
Revised: V i | = | V iso | /m .

Original: Consider two nodes u ∈ V
Revised: Consider two nodes u ∈ V

Original: i
Revised: i

Original: and v ∈ V j from different clusters.
Revised: and v ∈ V j from different clusters.

Original: Since both of them are in V iso , so they have identical h -hop neighborhood structures, and h -layer 1-WL-GNN will give them the same representations.
Revised: Since both of them are in V iso , so they have identical h -hop neighborhood structures, and h -layer 1-WL-GNN will give them the same representations.

Original: Then let us select another node w in V i , h -layer 1-WL-GNN will also make ( u, w ) and ( v, w ) have the same representation.
Revised: Then let us select another node w in V i , h -layer 1-WL-GNN will also make ( u, w ) and ( v, w ) have the same representation.

Original: However, if we use virtual nodes to label nodes and give them additional features, because u, w are in the same cluster while v, w belong to different clusters, ( u, w ) will have different representation from ( v, w ) .
Revised: However, if we use virtual nodes to label nodes and give them additional features, because u, w are in the same cluster while v, w belong to different clusters, ( u, w ) will have different representation from ( v, w ) .

Original: Now let us count the number of such non-isomorphic link pairs Y , we can have:
Revised: Now let us count the number of such non-isomorphic link pairs Y , we can have:

Original: [Equation]
Revised: [Equation]

Original: Taking | V iso | = ω ( n (cid:15) ) into the above in-equation, we get
Revised: Taking | V iso | = ω ( n (cid:15) ) into the above in-equation, we get

Original: [Equation]
Revised: [Equation]

Original: Example (Power of Using Multiple Virtual Nodes).
Revised: Example (Power of Using Multiple Virtual Nodes).

Original: In Figure 4, we show two cases with and without virtual nodes.
Revised: In Figure 4, we show two cases with and without virtual nodes.

Original: Consider the nodes v 2 , v 3 with the same local structure, which means they can get the same node representations by using 1-WL-GNN.
Revised: Consider the nodes v 2 , v 3 with the same local structure, which means they can get the same node representations by using 1-WL-GNN.

Original: So we cannot discriminate the links ( v 1 , v 2 ) and ( v 1 , v 3 ) if we just use 1-WL-GNN and concatenate the node representations for link prediction.
Revised: So we cannot discriminate the links ( v 1 , v 2 ) and ( v 1 , v 3 ) if we just use 1-WL-GNN and concatenate the node representations for link prediction.

Original: However, if we add 2 virtual nodes and add extra features to each node.
Revised: However, if we add 2 virtual nodes and add extra features to each node.

Original: v 1 and v 2 get a new feature (1 , 0) , v 3 get new feature (0 , 1) .
Revised: v 1 and v 2 get a new feature (1 , 0) , v 3 get new feature (0 , 1) .

Original: So it is easy to see ( v 1 , v 2 ) and ( v 1 , v 3 ) now have different representations.
Revised: So it is easy to see ( v 1 , v 2 ) and ( v 1 , v 3 ) now have different representations.

Original: C A DDITIONAL D ETAILS ON THE D ATA
Revised: C A DDITIONAL D ETAILS ON THE D ATA

Original: See Table 5 for the datasets we consider additionally in the appendix.
Revised: See Table 5 for the datasets we consider additionally in the appendix.

Original: [Table]
Revised: [Table]

Original: The datasets vary strongly in the number of nodes and edges with cora < pubmed < ddi < collab <
Revised: The datasets vary strongly in the number of nodes and edges with cora < pubmed < ddi < collab <

Original: ppa10 < ppa .
Revised: ppa10 < ppa .

Original: Yet, ddi shows a challenging average node degree and hence is very dense.
Revised: Yet, ddi shows a challenging average node degree and hence is very dense.

Original: The average clustering coefﬁcient intuitively reﬂects the extent to which direct neighbors of a node are themselves direct neighbors, and thus measures the “cliquishness” of the graph’s subgraphs.
Revised: The average clustering coefﬁcient intuitively reﬂects the extent to which direct neighbors of a node are themselves direct neighbors, and thus measures the “cliquishness” of the graph’s subgraphs.

Original: Here, we have ppa10 < pubmed < cora < ppa < ddi < collab .
Revised: Here, we have ppa10 < pubmed < cora < ppa < ddi < collab .

Original: Finally, the graph diameter varies similarly, and again shows that ppa represents a special challenge.
Revised: Finally, the graph diameter varies similarly, and again shows that ppa represents a special challenge.

Original: The diameter does not necessarily reﬂect the problem depth directly, but it shows which maximal problem depth can be expected.
Revised: The diameter does not necessarily reﬂect the problem depth directly, but it shows which maximal problem depth can be expected.

Original: In particular, the depths here hint at much larger numbers of GNN layers than the 2-3 layers we usually use, hence the data might suit testing under-reaching.
Revised: In particular, the depths here hint at much larger numbers of GNN layers than the 2-3 layers we usually use, hence the data might suit testing under-reaching.

Original: D M ODEL C ONFIGURATIONS AND T RAINING
Revised: D M ODEL C ONFIGURATIONS AND T RAINING

Original: We trained all models for 80 runs using the Bayesian optimization provided by wandb 4 and the following hyperparameters.
Revised: We trained all models for 80 runs using the Bayesian optimization provided by wandb 4 and the following hyperparameters.

Original: hidden dimension 32, 64, 128,learning rate 0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.
Revised: hidden dimension 32, 64, 128,learning rate 0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.

Original: dropout 
Revised: dropout 0, 0.3, 0.

Original: [Table] # of layers 1- # [Table] (random)
Revised: # of layers 1- # of virtual nodes (random)

Original: [Table]
Revised: 1- # of virtual nodes 1,2,4,8,16,32,

Original: [Table] [Table] APPNP - [Table]
Revised: SGC - K 2APPNP - α 0.05, 0.1, 0.2, 0.

Original: [Table] -GDC - [Table]
Revised: GNN-GDC - k 64,

Original: [Table]

Original: GNN
Revised: GNN

Original: [Table]
Revised: [Table]

Revised: Please note that we considered the wide ranges of values only in order to ﬁnd a good general setting.

Revised: For practical usage a hidden dimension of 256, learning rate of 0.0001, and dropout of 0.3 should work well; only on the small graphs a dropout of 0 might work better.

Revised: As usual, the number of layers depends on the type of data; however, note that the virtual nodes make it possible to use more that then the usual 2-3 layers.

Revised: Generally, higher numbers of virtual nodes work better, in line with our theoretical results.

Original: Note that we used less virtual nodes in the selection for the models (-RM, -RM F ) since especially -RM F was very slow and preliminary results showed that larger numbers did not change the results greatly – probably due to the randomness.
Revised: Also note that we used less virtual nodes in the selection for the models (-RM, -RM F ) since especially -RM F was very slow and preliminary results showed that larger numbers did not change the results greatly – probably due to the randomness.

Original: We used maximally 64 virtual nodes due to memory issues with larger numbers (e.g., 128), especially on the larger datasets.
Revised: We used maximally 64 virtual nodes due to memory issues with larger numbers (e.g., 128), especially on the larger datasets.

Original: We report the speciﬁc numbers of
Revised: We report the speciﬁc numbers of

Original: GNN layers and virtual nodes used by the trained models from Tables 2, 8, and 3 in Table 6.
Revised: GNN layers and virtual nodes used by the trained models from Tables 2, 8, and 3 in Table 6.

Original: For the ﬁrst clustering in GNN-CM + , we created 150 clusters on cora and pubmed , 200 clusters on ddi and collab , and 1000 on ppa10 .
Revised: For the ﬁrst clustering in GNN-CM + , we created 150 clusters on cora and pubmed , 200 clusters on ddi and collab , and 1000 on ppa10 .

Original: We tuned all models for 80 runs, and thereafter ran the models with the best 3 conﬁgurations forruns and chose the best of these model as the ﬁnal model (conﬁguration).
Revised: We tuned all models for 80 runs, and thereafter ran the models with the best 3 conﬁgurations forruns and chose the best of these model as the ﬁnal model (conﬁguration).

Original: We trained as suggested by the OGB (e.g., the splits, negative sampling) but used a batch size of 2and sometimes adapted the number of runs due to lack of resources; we used 3 for the experiments on collab and ppa10 in Table 2.
Revised: We trained as suggested by the OGB (e.g., the splits, negative sampling) but used a batch size of 2and sometimes adapted the number of runs due to lack of resources; we used 3 for the experiments on collab and ppa10 in Table 2.

Original: However, we ran several of our models for 10 runs as required for results on the OGB leaderboards and the numbers are comparable (see Table 10). 
Revised: However, we ran several of our models for 10 runs as required for results on the OGB leaderboards and the numbers are comparable (see Table 10). [Table] [Table]

Original: We used 500 epochs with a patience of 30.
Revised: We used 500 epochs with a patience of 30.

Original: Furthermore, for collab , we used the validation edges during testing (OGB contains both settings, with and without them).[Table] [Table]
Revised: Furthermore, for collab , we used the validation edges during testing (OGB contains both settings, with and without them).

Original: E A DDITIONAL E XPERIMENTAL R ESULTS E. R ESULTS ON ppa
Revised: E A DDITIONAL E XPERIMENTAL R ESULTS E. R ESULTS ON ppa

Original: The ppa dataset is challenging in both its size and density.
Revised: The ppa dataset is challenging in both its size and density.

Original: Since we were missing the resources to run experiments for all baselines on this dataset, we compare our best models (trained only on ppa10 , we did not do additional hyperparameter tuning) to the OGB leaderboard in Table 7.
Revised: Since we were missing the resources to run experiments for all baselines on this dataset, we compare our best models (trained only on ppa10 , we did not do additional hyperparameter tuning) to the OGB leaderboard in Table 7.

Original: For
Revised: For

Original: GCN, we see that our virtual node approach is able to improve the results considerably – even if only trained on 10% on the data.
Revised: GCN, we see that our virtual node approach is able to improve the results considerably – even if only trained on 10% on the data.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: E. R ESULTS ON cora
Revised: E. R ESULTS ON cora

Original: We ran the models also on the small cora data, yet the results conﬁrm our expectation, that virtual nodes for link prediction should be used in challenging graphs.
Revised: We ran the models also on the small cora data, yet the results conﬁrm our expectation, that virtual nodes for link prediction should be used in challenging graphs.

Original: In contrast, for cora , we get already good scores with a regular GCN.
Revised: In contrast, for cora , we get already good scores with a regular GCN.

Original: See Table 8.
Revised: See Table 8.

Original: E. R UNTIME
Revised: E. R UNTIME

Original: We show the runtimes on ddi in Figure 5.
Revised: We show the runtimes on ddi in Figure 5.

Original: Here we see that a single virtual node can have a positive impact at the same time on both prediction scores and efﬁciency, while the clustering takes more time.
Revised: Here we see that a single virtual node can have a positive impact at the same time on both prediction scores and efﬁciency, while the clustering takes more time.

Original: E. C OMPARISON OF C LUSTERING A LGORITHMS
Revised: E. C OMPARISON OF C LUSTERING A LGORITHMS

Original: See Table 9 and analysis in the main paper.
Revised: See Table 9 and analysis in the main paper.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E. A DDITIONAL R UNS FOR collab
Revised: E. A DDITIONAL R UNS FOR collab

Original: Table 10 compares several 10-run averages over collab to the 3-run averages.
Revised: Table 10 compares several 10-run averages over collab to the 3-run averages.

Original: The numbers are stable.
Revised: The numbers are stable.

Original: F C LUSTER A NALYSIS
Revised: F C LUSTER A NALYSIS

Original: We computed additional statistics about our “virtual node clusters” (i.e., a cluster represents a set of nodes connected to the same virtual node).
Revised: We computed additional statistics about our “virtual node clusters” (i.e., a cluster represents a set of nodes connected to the same virtual node).

Original: Our hypotheses was that our proposed clustering based on
Revised: Our hypotheses was that our proposed clustering based on

Original: [Table] [Table]
Revised: [Table] [Table]

Original: RM /
Revised: RM /

Original: [Table]
Revised: [Table]

Original: CM
Revised: CM

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: CM
Revised: CM

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the graph structure better reﬂects the distribution of test edges than, for example, random clustering.
Revised: the graph structure better reﬂects the distribution of test edges than, for example, random clustering.

Original: We report the results in Table 11.
Revised: We report the results in Table 11.

Original: For the -RM F and -CM + models we report two numbers.
Revised: For the -RM F and -CM + models we report two numbers.

Original: The upper one shows the average number of intra-cluster test edges over 10 runs.
Revised: The upper one shows the average number of intra-cluster test edges over 10 runs.

Original: The numbers in the lower part distinguish the actual edges and reﬂect how many different test edges occur in a cluster over the 10 runs.
Revised: The numbers in the lower part distinguish the actual edges and reﬂect how many different test edges occur in a cluster over the 10 runs.

Original: These numbers hence represent lower and upper bounds respectively.
Revised: These numbers hence represent lower and upper bounds respectively.

Original: As expected, the numbers for -CM are in between those bounds.
Revised: As expected, the numbers for -CM are in between those bounds.

Original: For ddi , we see that the -CM + and -RM F numbers are very similar, while the ones for -CM
Revised: For ddi , we see that the -CM + and -RM F numbers are very similar, while the ones for -CM

Original: + are much better over collab and ppa10 .
Revised: + are much better over collab and ppa10 .

Original: G I NVESTIGATION OF N ODE E MBEDDINGS
Revised: G I NVESTIGATION OF N ODE E MBEDDINGS

Original: We also investigated the embeddings of the virtual nodes and compared them to the ones of the regular grapoh nodes, but we could not derive many conclusions.
Revised: We also investigated the embeddings of the virtual nodes and compared them to the ones of the regular grapoh nodes, but we could not derive many conclusions.

Original: The main ﬁnding is that the virtual node embeddings are much more diverse than the mean of the embeddings in corresponding cluster – we would have expected them to be similar.
Revised: The main ﬁnding is that the virtual node embeddings are much more diverse than the mean of the embeddings in corresponding cluster – we would have expected them to be similar.

Original: H D ETAILS ABOUT P-GNN
Revised: H D ETAILS ABOUT P-GNN

Original: The model closest to our approach is the position-aware graph neural network (P-GNN)
Revised: The model closest to our approach is the position-aware graph neural network (P-GNN)

Original: (You et al., 2019).
Revised: (You et al., 2019).

Original: It assigns nodes to random subsets of nodes called “anchor-sets”, and then learns a non-linear aggregation scheme that combines node feature information from each anchor-set and weighs it by the distance between the node and the anchor-set.
Revised: It assigns nodes to random subsets of nodes called “anchor-sets”, and then learns a non-linear aggregation scheme that combines node feature information from each anchor-set and weighs it by the distance between the node and the anchor-set.

Original: That is, it creates a message for each node for every anchor-set, instead of for each direct neighbor.
Revised: That is, it creates a message for each node for every anchor-set, instead of for each direct neighbor.

Original: We ran experiments with P-GNN but did not obtain conclusive results.
Revised: We ran experiments with P-GNN but did not obtain conclusive results.

Original: It did not run on the larger datasets.
Revised: It did not run on the larger datasets.

Original: For ddi , we considered the number of anchor nodes as hyperparameter since the ﬁxed choice of 64 from the experiments of (You et al., 2019) did not yield good results.
Revised: For ddi , we considered the number of anchor nodes as hyperparameter since the ﬁxed choice of 64 from the experiments of (You et al., 2019) did not yield good results.

Original: However, larger numbers such as 128 or 512 resulted in very large runtimes (9 hrs / epoch).
Revised: However, larger numbers such as 128 or 512 resulted in very large runtimes (9 hrs / epoch).

Original: The result in Table 3 is an intermediate best value after 50 runs.
Revised: The result in Table 3 is an intermediate best value after 50 runs.

Original: We contacted the authors and they indeed mentioned that the model is not very scalable and suggested to use just the anchor-set distance as additional features, instead of overtaking the adapted message passing as well.
Revised: We contacted the authors and they indeed mentioned that the model is not very scalable and suggested to use just the anchor-set distance as additional features, instead of overtaking the adapted message passing as well.

Original: We did not do this extra experiment since the SAGE +dist model, whose numbers we report, follows a similar approach.
Revised: We did not do this extra experiment since the SAGE +dist model, whose numbers we report, follows a similar approach.