Geodesic Graph Neural Network for Efficient Graph Representation Learning
Abstract
Recently, Graph Neural Networks (GNNs) have been applied to graph learningtasks and achieved state-of-the-art results.
However, many competitive methodsemploy preprocessing on the target nodes, such as subgraph extraction and customized labeling, to capture some information which is hard to be learned bynormal GNNs.
Such operations are time-consuming and do not scale to largegraphs.
In this paper, we propose an efficient GNN framework called Geodesic
GNN (GD-GNN).
It injects conditional relationship between nodes into the modelwithout labeling.
Specifically, we view the shortest paths between two nodes asthe spatial graph context of the neighborhood around them.
The GNN embeddings of nodes on the shortest paths are used to generate geodesic representations.
Conditioned on the geodesic representations, GD-GNN is able to generate node,link, and graph representations that carry much richer structural information thanplain GNNs.
We theoretically prove that GD-GNN is more powerful than plain
GNNs, and present experimental results to show that GD-GNN achieves highlycompetitive performance with state-of-the-art GNN models on link prediction andgraph classification tasks while taking significantly less time.
Introduction
Graph Neural Network (GNN) is a type of neural networks that learns on relational data.
With theemergence of large-scale network data, it can be applied to solve many real-world problems, includingrecommender systems [43], protein structure modeling [28] and knowledge graph completion [2].
The growing and versatile nature of graph data poses great challenges to GNN algorithms both ontheir performance and on their efficiency.
GNNs use message passing to propagate features between connected nodes in the graph, and thenodes aggregate their received messages to generate representations that encode the graph structureand feature information around them.
These representations can be combined to form multi-nodestructural representations.
Because GNNs are efficient and have great generalizability, they are widelyemployed in node-level, edge-level and graph-level tasks.
A part of the power of GNNs comes fromthe fact that they resemble the process of 1-dimensional Weisfeiler-Lehman (1-WL) algorithm
The algorithm encodes subtrees rooted from each node through an iterative node coloring process,and if two nodes have the same color, they have the same rooted subtree and should have very similarsurrounding graph structures.
However, as pointed out by Xu et al.
[44], GNN’s expressive poweris also upper-bounded by the 1-WL test.
Specifically, GNN is not able to differentiate nodes thathave exactly the same subtrees but have different substructures.
For example, consider the graph in
Figure 1 with two connected components, because all nodes have the same number of degrees, theywill have exactly the same rooted subtrees.
Nevertheless, the nodes in the left component is clearlydifferent from nodes in the right component, because the left component is a 3-cycle and the right one is a 4-cycle.
Such cases can not be discriminated by GNNs or 1-WL test.
We refer to this type of
GNN as plain GNN or basic GNN.
On the other hand, even when a carefully-designed GNN can differentiate all node substructures in a graph, Zhang et al.
[53] showthat the learned node representation still cannot effectively performstructural learning tasks involving multiple nodes, including linkpredictions and subgraph predictions, because the nodes are not ableto learn the relative structural information.
We will discuss this withmore details in Section 3.1.
These limitations motivate recent research that pushes the limit of
GNN’s expressiveness.
One branch is to use higher-order GNNs that mimic higher-dimensional
Weisfeiler-Lehman tests [21, 24].
They extend node-wise message passing to node-tuple-wise andobtain more expressive representations.
Another branch is to use labeling methods.
By extractingsubgraphs around the nodes of interests, they design a set of subgraph-customized labels as additional node features.
The labels can be used to distinguish between the nodes of interests and other nodes.
The resulting node representations can encode subtructures which plain GNNs cannot, like cycles.
However, the major drawback of these methods is that compared to plain GNNs, their efficiency issignificantly compromised.
Higher-order GNNs update embeddings over tuples and incur at leastcubic complexity [51, 21].
Labeling tricks usually require subgraph extraction on a large graph andrunning GNNs on mini-batches of subgraphs.
Because the subgraphs overlap with each other, theaggregated size of the subgraphs is possibly hundreds times of the large graph.
Nowadays GNNs aremassive, the increased graph computation cost makes labeling methods inapplicable to real-worldproblems with millions of nodes.
We observe that a great portion of the labeling methods rely on the geodesics, namely the shortestpaths, between nodes.
For node representation, Distance-Encoding(DE) [23] labels the target nodes’surrounding nodes with their shortest path distance to the target nodes and achieves higher than 1-WLtest expressiveness.
Current state-of-the-art methods for link prediction, such as SEAL [49], leveragethe labeling tricks in order to capture the shortest path distance along with the number of shortestpaths between the two nodes of the link.
Our proposed Geodesic-GNN (GD-GNN) model is designedto also capture this characteristic, while addressing the efficiency issue of previous methods.
Specifically, our GD-GNN model learns a function that maps the learning target to a geodesicaugmented target representation (the target can be node, edge, graph, etc).
The model consists of twounits, a GNN unit and a geodesic pooling unit.
The two units are connected to perform end-to-endtraining.
We first apply the GNN to the graph to obtain embeddings of all nodes.
Then, for eachtarget, we find the task-specific geodesics and extract the corresponding GNN representations ofnodes on the geodesics.
The task-specific geodesics have three levels.
For node-level tasks, weextract the geodesics between the target node and all of its neighbors.
For edge-level tasks, weextract the geodesics between the two target nodes of the edge.
For graph-level tasks, we get thegeodesic-augmented node representation for each node as in node-level tasks and readout all noderepresentations as the graph representation.
Finally, using the geodesic pooling unit, we combinethe geodesic representation and target representation to form the final geodesic augmented targetrepresentation.
Figure 2 summarizes the GD-GNN framework.
All task-specific geodesics are formed by one or multiple pair-wise geodesics between two nodes.
We propose two methods to extract the pair-wise geodesic information, horizontal or vertical.
Thehorizontal geodesic is obtained by directly extracting one shortest path between the two nodes.
Thevertical geodesic is obtained by extracting all direct neighbors of the two nodes that are on any oftheir shortest paths.
They focus on the depth and breath respectively, and possess different properties.
Horizontal geodesic is good at capturing long distance information not covered by the GNN.
Verticalgeodesic is provably more powerful than plain GNN.
Moreover, by incorporating the informationof the subgraph induced by the vertical geodesic nodes, we can use GD-GNN to distinguish somedistance-regular graphs , a type of graph that many current most powerful GNNs cannot distinguish.
During inference, the GNN unit is applied only once to the graph, after which we feed nodes andgeodesic information to the efficient geodesic pooling unit.
This reduces the number of GNN runsfrom the query number (or number of nodes for graph-level tasks) to a constant of one, which largelyreduces the computation cost.
Compared to labeling methods, GD-GNN only requires one GNN run and is much more efficient.
Compared to plain GNN methods, GD-GNN uses geodesic informationand thus is more powerful.
We conducted experiment on link prediction and graph classificationacross datasets with different scales and show that GD-GNN is able to significantly improve theperformance of basic GNNs and achieves very competitive results compared to state-of-the-artmethods.
We also demonstrate GD-GNN’s runtime superiority over other more-expressive GNNs.
Preliminaries
A graph can be denoted by G =
( V , E , R ) , where V = { v 1 , ..., v n } is the node set, E ⊆{ ( v i , r, v j ) | v i , v j ∈ V , r ∈ R} is the edge set and R is the set of edge types in the graph.
When|R| = 1 , the graph can be referred to as homogeneous graph.
When |R| > 1 , the graph can bereferred to as heterogeneous graph.
Knowledge graphs are usually heterogeneous graphs.
The nodescan be associated with features X = { x v |∀ v ∈ V} .
For simplicity, we use homogeneous graphs toillustrate our ideas and methods.
We follow the definition of message passing GNNs in [44] with MESSAGE and AGGREGATEfunctions.
Such GNNs iteratively update the node embeddings by receiving and combining theneighbor nodes’ embeddings.
The k -th layer of a GNN can be expressed as:
[Equation]
where h ( k ) v is the node representation after k iterations, h (0) v = x v , N ( v ) is the set of directneighbors of v , m u is the message embedding.
Different GNNs vary mainly by the design choice ofthe MESSAGE and AGGREGATE functions.
In this paper, we use h v to represent the final noderepresentation of v from the plain GNN.
Throughout this paper, we use bold letters to representvectors, capital letters to represent sets, and N k ( v ) to refer to nodes within k-hop neighborhood of v .
Geodesic Graph Neural Network
Geodesic is conventionally defined as the shortest path between two nodes and indeed in homogeneousgraph it encodes only the distance.
However, we found that when combined with GNNs and noderepresentations, geodesics express information much richer than shortest path distance alone.
Inthis section, we first discuss the intuition behind GD-GNN and its effectiveness.
Then, we describethe GD-GNN framework and how its GNN unit and geodesic pooling unit are combined to makepredictions.
Finally, we provide theoretical results of the better expressiveness of GD-GNN.
3.1 Intuition and Overview
The key weakness of directly using plain GNNs is that the embeddings are generated withoutconditional information.
For node-level tasks, as pointed out in [48], a plain GNN is unaware ofthe position of the target node in the rooted subtree, and hence the target node is oblivious aboutthe conditional information about itself.
For edge-level tasks, Zhang et al.
[53] show that despite anode-most-expressive-GNN, without node labeling, the embedding of one node in the edge is notable to preserve the conditional information of the other node.
Concretely, take link prediction as
[Figure]
an example, in Figure 3a, we have two target links AB and AC.
Since AB is already connected by alink and AC is not, they should be represented differently.
On the other hand, All nodes in the graphare symmetric and should be assigned the same embeddings using any GNN.
Hence, if we simplyconcatenate the node representations of AB and AC, they will have the same link representations, andcannot be differentiated by any downstream learner.
To overcome this problem, previous methodsrely on labeling tricks to augment the computation graphs with conditional information.
When node
B and C are labelled respectively in two separate subgraphs, the resulting computation graphs of A isdifferent as shown in Figure 3b and 3c.
Notice that, for the labeling methods to take effect, subgraph extraction is necessary.
However,relabeling the graph and applying a GNN to every target to predict is extremely costly.
Hence, wewonder whether we can also incorporate the conditional information between nodes without thecustomized labeling?
Observe that in the example, the computation graph captures the number oflayers before node A first receives a message from a labelled node, which is essentially the shortestpath distance d between A and the other node.
This implies that even when we only have noderepresentations from a plain GNN, by adding the distance information (which is 1 and 3 for AB and
AC respectively) independent of the computation graph, we can still distinguish the links.
While the previous example demonstrates the effectiveness of a simple shortest path distance metric,we notice that distance alone is not sufficient in many times.
For example, in Figure 3d, node B andnode C are symmetric and hence should be assigned the same node embedding when the graph is notlabelled.
However, the links AB and AC are apparently different as v 1 connecting AB connects toanother group of nodes whereas v 2 connecting AC does not connect to any other node.
If we only useshortest path distance between AB and AC to assist the prediction, we still cannot differentiate them.
However, if we inject the entire geodesic expressed by the corresponding node embeddings alongthe shortest path to the model, we are able to tell AB and AC apart due to v 1 and v 2 ’s differentembeddings.
Consider a naive GNN that outputs all node embeddings as one. Taking the shortest path can beseen as extracting a one-valued vector with a length of the shortest distance between the two nodes. Whereas in the neural setting, the general GNN can model the naive GNN and the nodes on thesame vector can have different representations besides one by encoding their own neighborhood.
Theextension of shortest path to the neural setting grants stronger expressive power to the geodesics.
Compared to subgraphs in subgraph GNNs, the geodesic generation process and the GNN processis decoupled , because we select the geodesic nodes after all node representations are generated,while one subgraph GNN result is tied to a subgraph.
Dwelling on this decoupling and conditioningphilosophy, GD-GNN can run the GNN only once to generate generic node embeddings, and thenuse geodesic information to inject conditional information into the model in a much more efficientway.
3.2 Pair-wise Geodesic Representation
GD-GNN has different ways to leverage geodesic information for different levels of tasks, butthey all have the same basic component, pair-wise geodesic representation.
It consists of structuralinformation of nodes on the shortest paths of the pair.
Directly extracting all such nodes is timeconsuming and sometimes not effective, hence we present two primary ways to learn the geodesicrepresentation, horizontal and vertical .
3.2.1 Horizontal Geodesic Representation
Horizontal Geodesic Representation is designed to directly capture the distinctiveness as describedin the previous example shown in the Figure 3d.
It first finds a shortest path between two nodes asthe geodesic.
If there are multiple paths between them, we randomly choose one.
The nodes on theshortest path is denoted by P ( H ) ( u,v ) = { u, n 1 , n 2 , ..., n d − 1 , v } with a length of d .
Using P ( H ) ( u,v ) we canconstruct the geodesic representation g ( u,v ) by a geodesic pooling layer.
[Equation]
To prevent extremely long geodesics which may cause overfitting, we limit the maximum length to amanageable constant d max .
Any pair with distance larger than d max will have infinite distance and azero geodesic representation.
Because of the high computation cost of GNNs, the number of GNN layers is usually very small.
Using horizontal geodesics, we are able to connect nodes that are far away and do not coexist in a
GNN’s limited-height computation graph.
3.2.2 Vertical Geodesics Representation
Horizontal geodesics seem like an intuitive choice to directly connect the pair of nodes by a sequenceof adjacent nodes.
However, a natural question to ask is whether a single path is enough to representthe conditional information between the pair?
An example where horizontal geodesic falls short canbe found in Appendix H. Realizing this limitation, we propose vertical geodesic representation thatincludes all geodesic information in an efficient way.
For a pair of nodes u and v , vertical geodesic extracts the direct neighbors of them that are on anyof their shortest paths.
Specifically, let the distance between u and v be d ( u, v ) , we have the nodes ontheir shortest paths to be W ( u,v
[Equation]
The geodesics canbe represented as,
[Equation]
where P ( V ) ( u,v ) ,i := W ∩ N ( i ) , i ∈ { u, v } .
[Equation]
Here P ( V ) ( u,v ) ,i represents the nodes in the geodesic that are also the direct neighbors of i .
For sometasks, we only consider one P ( V ) ( u,v ) ,i as the geodesics instead of both.
We use a pooling layer togenerate vertical geodesic
[Equation]
While the vertical geodesic does not explicitly encode a long shortest path, it can still capture shortshortest path information due to pooling the embeddings of the nodes in it.
Compared with thehorizontal geodesic that takes one random shortest path, the vertical geodesic focuses more on breadthby incorporating all direct neighbors on any shortest paths.
Unlike nodes in horizontal geodesic, nodes in vertical geodesic induce a more complex subgraph.
Consider only P ( V ) ( u,v ) ,v (we will use P v here for simplicity) and the subgraph G v induced by nodes in
P v .
We can have a pooling function which encodes the graph structure of G v ,
[Equation]
The pooling layer can be a small plain GNN itself, and the initial embedding of G v can take thepooled node embeddings from the main GNN.
However, to keep our method efficient, we only addthe degree of nodes in the subgraph to the node embeddings.
And the vertical geodesic representationbecomes,
[Equation]
(6) where ⊕ means concatenation.
Note that a node’s degree in the subgraph is different from thatin the main graph.
This can be seen as a 1-layer GNN applied onto G v with identical initial nodeembeddings.
Even with this simple pooling design, the vertical geodesic is able to help distinguishsome distance regular graphs, which we cover in section 3.4.
Because vertical geodesics do not encode distance directly, we concatenate distance to the verticalgeodesics as an additional feature.
Like in horizontal geodesics, when d ( u, v ) is greater than d max ,we do not generate geodesic representation for the pair, and set their distance to infinity.
3.3 Task specific geodesic representation
Task specific geodesic representation is a collection of pair-wise geodesic representations.
Here, wefocus on three levels of tasks, node-level, edge-level, and graph-level.
For node-level tasks, the goal is to generate a representation z ( n ) v for a target node v .
We first findv ’s k-hop neighbor nodes S v = N k ( v ) .
Then for each node s i in S v , we find the pair-wise geodesic,g v,s i .
Note that in node level tasks, vertical geodesic only contains the geodesic nodes close to s i ,
P ( V ) ( v,s i ) ,s i .
Then the geodesic representations of the neighborhood is combined by a pooling layer,
[Equation]
While the GNN node embeddings of the neighborhood is not conditioned on v , by adding the geodesicinformation to every neighbor, we inject the conditional information of v to its neighbors.
Then thecollective information of the neighbors helps distinguish v .
For edge-level tasks, the goal is to generate an edge embedding z ( e ) ( u,v ) for nodes u and v .
Here, wedirectly use the pair-wise geodesic for the two nodes, z ( e ) ( u,v ) =
g ( u,v ) .
As pointed out by previousworks [56, 52], the advantage of this setup is that, when the downstream task is to rank one linkagainst N other links with one same node v , we only need to compute the distance vector from vonce and find the geodesic nodes efficiently.
For graph-level tasks, we need to generate a graph embedding z ( g ) G for G =( V , E ) .
We use anotherpooling layer to summarize the geodesic-augmented representation of every node in the graph,
[Equation]
GD-GNN can also be easily generalized to other levels of tasks such as subgraph-level.
3.4 Expressive Power of GD-GNN
Distinguishing power on graph-level tasks is often used to characterize the expressiveness of a GNNmodel.
For GNN models to be more expressive than 1-WL test, we can test whether they are able todistinguish regular graphs.
In the following theorem, we show that given an injective R ( g ) , the graphembedding z G generated based on vertical geodesics can distinguish most regular graphs.
Theorem 1.
Let the graph pooling layer R ( g ) be injective given input from a countable space.
Consider all pairs of n-sized r-regular graphs, where 3 ≤ r < √ 2 log n , n is the number ofnodes in the graph.
For any small constant ϵ > 0 , there exists a GD-GNN using vertical geodesicrepresentations with d max = ⌈ ( 12 + ϵ )
[Equation]
⌉ which distinguishes almost all (1 − o (1)) suchpairs of graphs.
We include the proof of Theorem 1 in Appendix A.
This theorem implies that GD-GNN with verticalgeodesics is more expressive than basic GNNs (and 1-WL) and can discriminate almost all r -regulargraphs.
Moreover, GD-GNN only needs to search a small neighborhood for geodesics to achieve thehigher expressiveness.
This guarantees the good efficiency and performance of GD-GNN.
Moreover, when the vertical geodesic pooling layer encodes the subgraph induced by the geodesicnodes (using Equation 6 for geodesic pooling), GD-GNN is able to distinguish some distance regulargraphs.
Figure 4 shows two distance regular graphs, Shrikhande graph and 4x4 rook’s graph.
Theyhave the same intersection array and number of nodes.
As proved by Li et al.
[23], distance encodingis not able to distinguish this pair of graphs, because nodes in the two graphs will be assigned thesame embedding.
However, vertical geodesic is able to discriminate them.
Suppose the black nodesare the target nodes, the brown nodes are one distance away from the black nodes and the light bluenodes are two distance away from the black nodes.
Consider the light blue nodes in both graphs,they each have a pair of brown common neighbors with a black node.
The brown pair is the verticalgeodesics of one light blue node.
If we count the number of edges in the subgraph formed by eachbrown pair, we will see that four out of nine pairs form a one-edge graph (in red) in the Shrikhandegraph, while zero out of nine pairs form a one-edge graph in the 4x4 rook’s graph.
Since Equation6 directly captures this and will give four light blue nodes in Shrikhande graph different pair-wisegeodesics with respect to the black node, an injective graph pooling layer will yield different noderepresentations for the two black nodes, and in turn discriminates the two graphs.
Complexity.
The main computation advantage of GD-GNN comes from the fact that, for GD-GNN,the number of GNN runs stays as a constant of 1, while subgraph-based methods incur GNN runswhich increases linearly with respect to the number of queries.
Hence, GD-GNN maintains the goodefficiency of basic GNN methods, while injecting the high-expressiveness of subgraph methods intothe model.
A detailed complexity analysis on different tasks can be found in Appendix B.
Limitation.
While a simple geodesic pooling layer satisfies the theoretical expressiveness, if weindeed use a GNN or other more complex layer for pooling as suggested in Section 3.2.2, thepooling will dominate the good amortized complexity of GDGNN.
We see this as a possible futuredirection where we can use a more complex pooling layer to further improve the expressiveness whilemaintaining the efficiency.
Related Works
GNNs beyond 1-WL test.
Designing GNNs with stronger representation power is a fundamentaltask in GNN research.
Xu et al.
[44] first proved that the expressiveness of basic GNNs, such as
GCN [18], GraphSAGE [15], and GAT [39], is upper-bounded by the 1-WL test.
Following worksconducted extensive research to break this limit.
They can be sorted into several directions: (1) Earlierworks proposed higher-order GNNs that are as expressive as k-WL algorithm, such as k-GNN [21],
RingGNN[6], and PPGN[24].
These methods require message passing on node tuples that growsexponentially with respect to k. The increased computation cost makes them not scalable to largegraphs.
(2) Another type of methods, like GSN [5] and MotifNet [25], incorporates isomorphismcounting on certain graph substructures like cycles, triangles, and cliques, similar to earlier kernelbased methods [45, 12].
They essentially augment the models with features undetectable by GNNs.
However, a properly-design substructure usually requires expert knowledge which is not available forall data.
(3) Some other works use identity or random node labeling within the plain GNN framework.
ID-GNN [48] extracts a subgraph around the target node and perform heterogeneous message passing.
DE-GNN [23] also requires subgraph extraction but labels nodes in the subgraph with distance tothe targets.
They both have stronger expressiveness than basic GNNs.
Abboud et al.
[1] shows that
GNNs possess universality with random node feature initialization (RNI).
Nevertheless, RNI alsoinduces a huge sample space, and GNNs adopting RNI is very hard to train.
Some recent workspropose to use additional subgraph structures to augment the node embeddings.
GraphSNN [42]incorporates properties of overlapping subgraphs between adjacent nodes to be more expressive than1-WL.
Nested-GNN [51] applies GNN to ego-subgraphs of every node and proves that it is able todiscriminate regular graphs.
Sharing the same idea of Nested-GNN, GNN-AK[54] uses a differentsubgraph pooling strategy and uses subgraph dropping to improve the model scalability.
Link prediction using GNNs.
Earlier works [19, 18, 7, 38] applied basic GNNs to link predictionusing an auto-encoder framework, where the GNN serves as the encoder of the nodes, and edges aredecoded by their nodes’ encoding vectors.
Another line of works, including SEAL [49], IGMC [50],and GraIL [35], use GNNs to encode a labelled subgraph to form the link representation.
Later work[33, 53] reveals that a good node-level/graph-level representation does not necessarily lead to a goodedge-level representation.
Zhang et al.
[53] systematically show that the labeling trick bridges the gapbetween node-level and edge-level GNN expressiveness.
Sharing the spirits of labeling trick, NBFNet[56] extends the generalized Bellman-Ford algorithm to the neural setting, and can encode manytraditional path-based methods.
NBFNet has a better scalability when there are multiple links sharingone same node. Some other works also propose to use path information between links to enhance theprediction like horizontal geodesic.
Jagvaral et al.
[17] uses CNN and BiLSTM to summarize the path information, however, their model requires fixed node embedding, while horizontal geodesicis based on graph structure generated from GNN and hence is inductive. Wang et al.
[41] learnsrelational path information by training a set of embeddings for all different types of relational path,while horizontal GD-GNN uses GNN to capture the graph structure and pool node embeddings alongthe geodesic without training relational-path-specific embeddings.
Experimental Results
5.1 Link Prediction
Datasets.
We use several types of datasets for link prediction.
(1) Knowledge Graph (KG) inductivelink prediction datasets.
Inductive link prediction means that the model is trained on a traininggraph while tested on a different graph.
This requires the model to generalize to unseen entities,which is a property preserved by a limited number of models.
We follow the standard inductive splitof WN18RR[10] and FB15K237[37] as in [35].
We rank each positive link against 50 randomlysampled negative links with the same head/tail as the positive link, and report Hit@10 ratio.
OGB large-scale link prediction dataset [16], including OGBL-COLLAB and OGBL-PPA.
We usethe official split provided by the OGB team.
We rank each positive link against a set of providednegative links in the dataset, and report Hit@N ratio.
(3) Results of citation datasets and comparisonto other distance-based methods can be found in Appendix D. (4) Full results on the (KG) datasets areincluded in Appendix C. These datasets vary by graph size and demonstrate GD-GNN’s capabilityacross scales.
Baselines.
We compare GD-GNN with Rules Mining methods, DRUM [29] and NeuralLP [46],and GNN-based methods, NBFNet [56] and GraIL [35] for the KG tasks.
Some other works like[40, 8] incorporate text data in the KG to achieve better performance. As we believe models thatemploy textual information is beyond the context of graph structural learning, we exclude them in ourcomparison.
For the OGB dataset, we compare GD-GNNs with other GNN-based methods achievingtop places on the OGB link prediction leaderboard, including Deep Walk [27], Resource Allocation(RA)
[55], SEAL, Deeper GCN [22].
Implementation Details.
For link prediction, we use GCN (RGCN for KG) as the basic GNN.
Forall the datasets, we search the number of GNN layers in { 2 , 3 , 4 , 5 } , and the max search distance forgeodesic, d max , is same as the GNN layers.
We use 32 hidden dimensions for all fully-connectedlayers in the model.
We train 50 epochs with a batch size of 64 for the KG datasets
[Table]
and we train50 epochs with a batch size of 4096 for the OGB datasets.
We record the test score using the modelwith the best set of hyperparameters chosen based on the performance on the validation set.
Theexperiment is repeated 5 times and we take the average of the scores.
[Table]
[Table]
Results and discussion.
We achieved SOTA performance on the inductive datasets.
GD-GNN isable to significantly improve over rule-mining methods and subgraph based method, GraIL.
We alsooutperform NBFNet on most of the datasets.
On the OGBL-COLLAB dataset (Table 2), vertical GD-GNN achieved SOTA performance.
On the OGBL-PPA dataset, GD-GNN is able to keep a verycompetitive result.
On both OGB datasets, GD-GNN improves significantly over its basic GNN,
GCN.
Moreover, as shown in Table 3, GD-GNN has running-time superiority over methods achievingcomparable performance, and has memory advantage to handle large graphs where NBFNet can not.
To make fair running time comparison, we run all models on 32 CPUs and 1 Nvidia GeForce 1080Ti GPU.
We also include ablation studies on all components of GD-GNN in Appendix F.
[Table] [Table]
5.2 Graph Classification
Datasets.
We use two types of datasets for graph classification.
(1) TU datasets contain D&D[11],
MUTAG[9], PROTEINS[11], PTC_MR[36], ENZYMES[31].
The evaluation metric is Accuracy(%).
We follow the evaluation protocol in [51], where we use 10-fold cross-validation to computethe score.
(2) OGB datasets[16], including OGBG-MOLHIV and OGBG-MOLPCBA.
The taskof OGBG-MOLHIV dataset is to predict whether a molecule inhibits HIV virus.
We use AUC toevaluate OGBG-MOLHIV dataset.
OGBG-MOLPCBA dataset has 128 classification tasks.
Wefollow the evaluation protocol provided by the OGB team and use AP averaged across all classes toevaluate the dataset.
(3) Synthetic datasets, including EXP[1] and CSL[26].
These datasets containsgraphs that can be only distinguished by model with > 1-WL expressiveness.
Baselines.
For the TU dataset, we compare with another plug-and-play GNN framework, NGNN[51].
Both GD-GNN and NGNN use GIN and GCN as their basic GNNs.
For the OGB dataset, we use GINas the basic GNN, we compare GD-GNN with methods achieving top places on the OGB leaderboard,including GIN[44], GSN[5], GIN-AK [54], Nested GNN [51], Directional GSN [3], PF-GNN [13]. [Table]
Implementation details.
We use vertical geodesic in the graph-leveltasks because of its theoretical advantage over horizontal GD-GNN.
For TU datasets, we search the GNN layers and d max in {2,3,4}.
We trainon the TU dataset for 100 epochs with a batch size of 64.
For OGBdatasets, we search the number of GNN layers in {2,3,4,5}, and train for100 epochs with a batch size of 256.
We adopt virtual node technique[14] to increase the connectivity of graphs for the OGBG-MOLPCBAdataset.
For synthetic datasets, we search the number of GNN layers in{2,3,4}, and train for 50 epochs with a batch size of 16.
[Table] [Table]
Results and discussions.
On the TU dataset (Table 6), GD-GNNs consistently outperform the basic
GNNs.
Compared to NGNN, GD-GNN is able to bring larger performance increase to the basic GNNs.
This shows GD-GNN’s enhancing power.
On the OGB dataset (Table 7), we can see that
GD-GNN’s performance is lower than PF-GNN and Directional GSN (so are GIN-AK and NGNN). This might indicate that pairwise conditional information is less effective for graph tasks than forlink tasks.
Nevertheless, GD-GNN still outperforms the basic GIN a lot, and as GD-GNN is ageneral framework that can be applied to any basic GNN, the performance might be able to be furtherimproved. On the synthetic data, we achieve comparable results to the universal PF-GNN, whichverifies our theory.
We include ablation studies of graph-level geodesics in Appendix F. We’ve also included experiments on node-level tasks in Appendix E and experiments on the impactof cutoff distance in Appendix G
Conclusions
In this paper, we propose a general GNN framework which incorporates geodesic informationto generate node, link, and graph representations.
Our framework is more powerful than basic
GNNs both practically and theoretically and is more efficient than current more-expressive GNNsas demonstrated by the running-time experiments.
Our framework has the potential to make GNNsencoding conditional information really applicable to large-scale real-world graphs.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We faithfully summarize the paper in the abstract.
(b) Did you describe the limitations of your work?
[Yes] As described in the Section 5 andAppendix C.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work
only involves existing datasets and solves general graph learning problem without
negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Assumption is described in the Theorem
(b) Did you include complete proofs of all theoretical results?
[Yes] In Appendix A3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] As described in the implementation details of Section
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] As shown in the experiment result tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
In the discussion of Section
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... [Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] The datasets and
[Table]
model source code repositories are all cited.
[Table] (b) Did you mention the license of the assets?
[N/A]
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[No] The datasets and source code are open source information allowed [Table]
for research use.
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[No] The data we are using is known to be de- [Table]
identified.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]
Appendix
A Proof of Theorem 1
The proof of Theorem 1 generally follows the proof of Nested-GNN’s expressiveness[51].
Let Q kv, Gbe the set of nodes in G that are exactly k distance away from node v .
We can have the followingdefinition,
Definition A.1.
The edge configuration bewteen Q kv, G and Q k +1 v, G is a list C kv, G =
( a 1 ,kv, G , a 2 ,kv, G , ... )
where a i,kv, G denotes the number of nodes in Q k +1 v, G of which each has exactly i edges from Q kv, G .
The proof leverages the fact that nodes in randomly sampled regular graphs are very likely to havedifferent edge configuration within their small neighborhood, and GD-GNN directly captures thisdistinction to discriminate the graphs.
Formally, we have the following excerpted Lemma from [51],
Lemma 2.
For two graphs G (1) =
( V (1) , E (1) ) and G (2) =
( V (2) , E (2) ) that are uniformly independently sampled from all n-node r-regular graphs, where 3 ≤ r < √ 2 log n , we pick anytwo nodes, each from one graph, denoted by v 1 and v 2 respectively.
Then there is at least onei ∈ (log n log( r − 1 − ϵ ) , ( 12 + ϵ ) log n log( r − 1 − ϵ ) ) with probability 1 − o ( n − 1 ) such that C iv 1 , G (1) ̸
= C iv 2 , G (2) .
Proof.
To prove Theorem 1, we first assume, without loss of generality, that the hidden dimension of
GD-GNN is one.
We know plain message passing GNN will generate the same embeddings for everynode in the regular graph, because all nodes have the same degree/subtree.
Hence, we can normalizeall the node representations to a constant of one.
For simplicity, we also reduce vertical geodesicpooling function R ( V ) gd in Equation 4 to a simple sum of the representations of the geodesic nodes.
Let d max = ⌈ ( 12 + ϵ ) log n log( r − 1 − ϵ )
⌉ , the maximum distance in which GD-GNN searches for a targetnode’s neighbors.
We then consider node v i
and it’s d max -hop rooted subgraph G d max v i , the verticalgeodesic pooling process is essentially counting the number of geodesic nodes of each node v j in
G v i .
And the pair-wise vertical geodesics between the v i and v j becomes,
[Equation]
[Equation] ) where d ( v i , v j ) is the distance between v i and v j , and hence v j ∈ Q d
( v i ,v j ) v i , G vi
and g ( V ) v i ,v j records thenumber of edges of v j that connects to nodes in Q d ( v i ,v j ) − 1 v i , G vi , and its distance to v i .
We also havethe node-level pooling function to be a distance sorted bin count function on the set of pair-wisevertical geodesic representations.
It first sorts the geodesic representations by their second dimension(distance), and perform bin count on their first dimension (number of geodesic nodes) separately foreach distance.
Then GD-GNN produces an mapping of G v i to its exact set of edge configurations,
C v i =
( C 1 v i , G vi , C 2 v i , G vi , ... ) .
Let G (1) =
( V (1) , E (1) ) and G (2) =
( V (2) , E (2) ) be two graphs uniformly independently sampledfrom all n-node r-regular graphs.
We consider two nodes v i ∈ V (1) and v j ∈ V (2) .
Becaused max = ⌈ ( 12 + ϵ ) log n log( r − 1 − ϵ ) ⌉ , the sets of edge configurations of the rooted subgraphs, C v i and C v j aredifferent with probability 1 − o ( n − 1 ) , according to Lemma 2.
In such case, GD-GNN also maps theirrooted subgraphs to different edge configurations, and hence the node-level geodesic representationz ( n ) v i and z ( n ) v j generated by GN-GNN will be different with the same probability.
Consider v i and all node v j ∈ V (2) , by union bound, z v i ̸∈ { z v j | v j ∈ V (2) } with a probability of1 − o (1) .
B Complexity Analysis
The complexity of GD-GNN differs by tasks.
On almost all tasks, GD-GNN demonstrates superiorityin efficiency.
The complexity of GD-GNN is formed by two parts, the geodesic extraction part andthe GNN part. We first note that the complexity of geodesic extraction and GNN cannot be directlycompared as their computation units are different, geodesic extraction operates on unit integers while GNN operates on d -dimensional vectors that are potentially very large, and running a geodesicextraction on a graph takes significantly less amount of time compared to running a GNN on thesame graph.
Meanwhile, we can always pre-compute geodesics which allows fast extraction duringinference. Hence, here we present complexity analysis of GD-GNN on the GNN part compared toother methods. B.1 Graph and node level tasks The analysis of graph and node level tasks is similar as graph representation is generated by takingall node representation.
Specifically, consider the problem where we need to infer k nodes in a graphwith | V | nodes and | E | edges.
In worst-case, the complexity for a T-layer subgraph-based GNN is O ( kT | E | ) , because it applies a T -layer GNN on k nodes’ subgraphs, where each subgraph contains| E | edges in the worst case (the full graph).
GD-GNN’s worst-case complexity is O
( T | E | + k | E | ) , O ( T | E | ) is for applying the GNN once to the full graph, and the number of nodes in a verticalgeodesic is bounded by the number of edges and hence the pooling takes O ( k | E | ) .
Hence, GDGNN is more efficient than subgraph GNNs. If we consider GNN’s layer complexity w.r.t. theembedding dimension d (mostly quadratic due to linear transformation/MLP over node embeddings,e.g. GCN and GIN), we will have O ( kT | E | d 2 ) for subgraph GNNs, but O ( T | E | d 2 + k | E | d ) for GD-GNN (the geodesic pooling is linear to d ).
Thus, the efficiency advantage of GD-GNN is moresignificant as shown in our experiments. For graphs, the comparison becomes O ( | V | T | E | ) versus O ( T | E | + | V || E | ) =
O ( | V || E | ) , this shows that GD-GNN helps amortize the expensive computationcost, T versus 1 , of applying a GNN. B.2 Edge level tasks For edge level tasks, we consider the worst-case scenario where we try to predict k links on agraph with | V | nodes and | E | edges.
The k links do not share any common nodes.
GDGNN takes O ( T | E | + k | V | ) for O ( T | E | ) GNN on the full graph and k geodesic pooling, where the number ofnodes in a link-level geodesic is bounded by the number of nodes O ( | V | ) in the graph.
Subgraph
GNN methods takes O ( kT | E | ) for applying T -layer GNN onto k links’ subgraphs (worst case O ( | E | )edges). NBFNet shares the same complexity O ( kT | E | ) because the links do not share commonnodes, and the GNN results can not be shared across links as suggested by the authors of NBFNet. Thus, k runs of GNN is necessary for NBFNet.
Then, comparing the two complexity, we can see thatwhen
O ( T | E | ) > O ( k | V | ) , subgraph method’s complexity O ( kT | E | ) grows linearly with respectto the number of queries, while GDGNN’s complexity O
( T | E | ) does not.
The practical runningtime of O ( T | E | ) and O ( k | V | ) are not on the same level as the GNN is actually quadratic to theembedding dimension d while the geodesic pooling is only linear to d . Therefore, in GD-GNN, themore expensive GNN layer complexity O ( Te ) does not increase linearly w.r.t. query number k ,which significantly saves the computational time compared to NBFNet or SEAL.
C Knowledge graph link prediction
For knowledge graphs, we use RGCN[30] as our base GNN.
We evaluate GD-GNN on FB15K237[37]and WN18RR[10] knowledge graphs for both inductive and transductive link prediction tasks.
We usethe standard split in [37] for transductive link prediction, and follow [35] for inductive link prediction.
We compare with Rules Mining methods, DRUM [29] and NeuralLP [46], and GNN-based methods,
NBFNet [56] and GraIL [35], for the inductive link prediction tasks.
For transductive tasks, weadditionally compared to general knowledge graph link prediction methods that are not inductive,including TransE [4], RotatE [34], and RGCN [30].
For inductive dataset, we randomly sample 50negative links that have either the same head or tail as the positive link, we report the H@10 againstthe 50 negative links.
For the transductive dataset, we follow the filtered ranking protocol [4], wherewe rank a positive link against all negative links that have the same head (or tail) as the positive links.
We report H@N and Mean Reciprocal Rank (MRR) for the transductive setting.
[-RotatE 33 .24 .
-]
[-RGCN 27 .18 .30 .
-]
[-GD-GNN-Vert 23 .15 .26 .45 .
-]
[-GD-GNN-Hor 
-]
We achieved SOTA performance on the inductive datasets.
GD-GNN is able to significantly improveover rule-mining methods and subgraph based method, GraIL.
We also outperform NBFNet on mostof the datasets.
We note that horizontal geodesic is able to exceed the performance of vertical geodesicon some datasets.
We suspect this is because horizontal geodesics implicitly capture the inter-nodetranslation process as in the embedding methods, and this piece of information is more important [Table] [Table] than graph structure in the knowledge graph setting.
We also acknowledge that GD-GNN does notperform as well in the transductive setting.
However, GD-GNN is still able to greatly improve theperformance of RGCN on WN18RR dataset, and RGCN requires trainable node embeddings while
GD-GNN does not.
We suspect the performance discrepancy between the homogeneous setting andthe knowledge graph setting is due to the increased number of edge types.
WN18RR dataset has 11types of relations, FB15K237 has 237.
Note that GNN methods like NBFNet[56] train a set of weightsfor every target relation type, meaning that their message passing is conditioned on the relation,while GD-GNN only uses one set of weights.
We notice that GD-GNN is able to outperform someembedding methods on WN18RR dataset, while not performing as good on FB15K237, possibly dueto the fact that FB15K237 dataset contains much more relations than WN18RR dataset.
This alignswith our hypothesis.
A potential solution is to also train multiple GD-GNNs that each handles onetarget relation type, note that we still only need to run each relation-specific GD-GNN once, and keepthe good amortized property of GD-GNN.
We leave this to future work.
D More link prediction results
Citation datasets[32] include Cora, Citeseer, and Pubmed datasets.
We follow the experimental settingin [56] and use 85:5:10 for the train/valid/test links split.
The evaluation metric is ROC-AUC (AUC)and Average Precision (AP) scores.
We compare GD-GNN with popular link prediction methods
VGAE [19], SEAL[49], NBFNet[56] for the citation datasets.
Note that for citation data, we do notuse the pre-trained node embeddings from the publications’ content, incorporating such informationonly increases the power of GD-GNN.
We use GCN as the basic GNN.
We search the number of GNN layers in { 2 , 3 , 4 , 5 } , and the maxsearch distance for geodesic, d max , is same as the GNN layers.
We use 32 hidden dimensions forall fully-connected layers in the model.
We train 100 epochs with a batch size of 64 for the citationdatasets.
On the citation dataset (Table 10), GD-GNN achieved very competitive results compared to the SOTAmethod NBFNet, and GD-GNN is more efficient than NBFNet as demonstrated in Table 3.
[Table]
We also includes C.elegans, NS, and PB dataset adopted by DE-GNN [23]. We follow its experimentsetup to split the edges into 8:1:1 train/test/valid splits, and repeat experiment 20 times and reportaverage AUC and 95% confidence range. For these datasets, we compare with other methods thatalso rely on distance encoding, including DE-GNN, PGNN[47], SEAL, and base GNN, GIN. [Table]
From the experiment, we can see that GD-GNN achieves very competitive results compared todistence-encoding GNN and SEAL, and outperforms PGNN [8] significantly on all datasets. Wesuspect the key reason is that PGNN relies on relative positions to the anchors, but the anchors that PGNN randomly chooses are not necessarily representative to all links. In contrast, the geodesicinformation is directly associated with each link.
E Node Classification
We use airport dataset, Brazil-Airport, Europe-Airport, and USA-Airport for our node classificationexperiments.
The task is to predict passenger flow level solely from the flight traffic network, wheregraph structure matters and more expressive GNNs are useful.
We compare to GCN, GIN and
DE-GNN that also relies on distance information.
We follow DE-GNN and split the dataset withtrain/test/valid ratio of 8:1:1.
We run experiment using verticala GDGNN with independent randominitialization 20 times and report average accuracy and 95% confidence range.
[Table]
DE-GNN has several variants and we take their best result here.
From Table 12, we can see that
GD-GNN achieves very competitive results compared to distance-encoded GNN, and significantlyimproves the performance of base GIN on two datasets, which experimentally verifies our claim thatnode-level geodesic is more expressive than plain GNNs.
GD-GNN has many components, and here we present ablation study of GD-GNN to demonstratethe effect of each component.
For link prediction dataset, we consider 4 settings, plain GCN (GCN),
GCN and distance between the two target nodes of the link (GCN+Dist), GD-GNN-Vertical withoutgeodesic degree (GD-GNN-Vert), and GD-GNN-Vertical with geodesics node degree (GD-GNNVert-Deg).
Note that the results we report in Section 5 include geodesic node degree.
[Table] From the experiment results (Table 13), we can see that distance is already able to assist link prediction,especially on sparser datasets like Cora and OGBL-COLLAB.
However, on OGBL-PPA, where theconnectivity of the graph is much larger, distance does not help, whereas geodesic representation isable to significantly improve the expressive power of basic GNNs.
Also, when the graph is sparse, itis rather unlikely for the geodesic nodes to form subgraphs, meaning that most of nodes on the verticalgeodesic have a degree of zero, and hence geodesic degree does not improve much.
Meanwhile, in
OBGL-PPA, geodesic degree is able to increase the performance of GD-GNN, which aligns with theintuition and example we present.
For horizontal geodesics, we present results where part of the horizontal geodesic is used.
[Table] Partial represents geodesics where only the head/tail, head/tail’s direct neighbors on the shortestpath are used to form the geodesic. N-GDGNN means the cutoff distance is N . We do not comparewith 2-3 distance because Partial and horizontal geodesic will be exactly the same in that case.
Weadd distance to both Partial and hortizontal geodesics to make fair comparison.
We can see thatwithout the full horizontal geodesic, the performance on Cora dataset indeed dropped by 1.5%.
While the difference is not as significant, we still see that horizontal geodesics outperforms Partial on
OGBL-COLLAB dataset.
This shows that while being connected within some distance is already agood indicator of the likelihood of the link, incorporating the node structure information can stillimprove the representation of the link.
For graph classification tasks, we study two simpler version of GD-GNN.
We reduce the verticalpair-wise geodesic function (Equation 4) to be a pooling function only on the embeddings of the pairof nodes.
Then, the node-level geodesic is essentially combining the node embedding and its k-hopneighbor embedding, we refer to this as Nei.
Another version is we append the pair-wise distance tothe neighbor embeddings, we refer to this as Dist.
Note that vertical geodesic takes part of the directneighbors of a neighbor node, we study the variant where all neighbors are used as the geodesic, wecall this FullNei.
From Table 15, we can see that the neighbor embedding alone is not very effective, and we needto assist it with pair-wise distance, a explicit form of geodesic to improve the performance.
Byincorporating the geodesic representation, we are able to consistently improve over the ’Nei+Dist’version, which aligns with Theorem 1.
Geodesic degree does not give significant improvement on
TU dataset possibly due to its high testing variance, but we can still see notable improvement on
OGBG-MOLHIV dataset.
For FullNei, we see that it does not bring much performance increase onto
[Table]
GD
[Table]
GIN+Nei+Dist compared to GD-GNN-Vert.
This shows that FullNei essentially weighs in as an extralayer of GNN which can be covered by hyperparameter search.
G Impact of different cutoff distances
A non-negligible hyperparameter in our model is the cutoff distance.
In this section, we presentresults on the impact of different cutoff distances.
[Table]
(L) represents link prediction task, and (G) represents graph classification task.
N-GDGNN means
GNN with different max-cutoff distance, we use vertical geodesics with 3-layers of base GNN.
From
Table 16, we see that for link prediction task (cora in particular), as the cutoff distance increases,the performance increases, this is because more links can be connected by geodesic extraction, anddistance itself is already a good indicator of the likelihood of the link.
However, disconnectednodes are not always predicted negative.
When disconnected, the link will have zero geodesicrepresentation, but still have meaningful node representations from the base GNN .
In such case,
GD-GNN degenerates embeddings similar to models like VGAE, which is still able to statisticallylearn the probability of a link based on the node structure around the two nodes of the link.
Thechoice of cutoff distance is really data-dependent, as we can see in the results of OGBL-PPA and
OGBG-MOLHIV, 4-GDGNN is worse than 3-GDGNN, and the actual number can be determined byhyperparameter tuning.
In general, cutoff distance resembles the max number of hops in subgraphextraction process.
H Limitation of Horizontal Geodesics
Figure
Figure 5 shows an example where vertical geodesic is more expressivethan horizontal geodesic.
The gray nodes are symmetric and hence willbe assigned the same embedding by a 1-WL GNN, the yellow nodes,because they all connect to two gray nodes, will also be assigned sameembedding by a 1-WL GNN.
In this case, both horizontal geodesicsbetween AB and BC are gray-yellow-gray nodes, hence we can notdifferentiate the two links.
Meanwhile, the vertical geodesic of ABis one yellow node, while the vertical geodesic of BC is three yellownodes, and hence the two links will have different representation aftersummarizing the geodesics information.