Geodesic Graph Neural Network for Efficient Graph Representation Learning
Abstract
Recently, Graph Neural Networks (GNNs) have been applied to graph learningtasks and achieved state-of-the-art results.
However, many competitive methodsemploy preprocessing on the target nodes, such as subgraph extraction and customized labeling, to capture some information which is hard to be learned bynormal GNNs.
Such operations are time-consuming and do not scale to largegraphs.
In this paper, we propose an efficient GNN framework called Geodesic
GNN (GD-GNN).
It injects conditional relationship between nodes into the modelwithout labeling.
Specifically, we view the shortest paths between two nodes asthe spatial graph context of the neighborhood around them.
The GNN embeddings of nodes on the shortest paths are used to generate geodesic representations.
Conditioned on the geodesic representations, GD-GNN is able to generate node,link, and graph representations that carry much richer structural information thanplain GNNs.
We theoretically prove that GD-GNN is more powerful than plain
GNNs, and present experimental results to show that GD-GNN achieves highlycompetitive performance with state-of-the-art GNN models on link prediction andgraph classification tasks while taking significantly less time.
Introduction
Graph Neural Network (GNN) is a type of neural networks that learns on relational data.
With theemergence of large-scale network data, it can be applied to solve many real-world problems, includingrecommender systems [38], protein structure modeling [25] and knowledge graph completion [2].
The growing and versatile nature of graph data poses great challenges to GNN algorithms both ontheir performance and on their efficiency.
GNNs use message passing to propagate features between connected nodes in the graph, and thenodes aggregate their received messages to generate representations that encode the graph structureand feature information around them.
These representations can be combined to form multi-nodestructural representations.
Because GNNs are efficient and have great generalizability, they are widelyemployed in node-level, edge-level and graph-level tasks.
A part of the power of GNNs comes fromthe fact that they resemble the process of 1-dimensional Weisfeiler-Lehman (1-WL) algorithm
The algorithm encodes subtrees rooted from each node through an iterative node coloring process,and if two nodes have the same color, they have the same rooted subtree and should have very similarsurrounding graph structures.
However, as pointed out by Xu et al.
[39], GNN’s expressive poweris also upper-bounded by the 1-WL test.
Specifically, GNN is not able to differentiate nodes thathave exactly the same subtrees but have different substructures.
For example, consider the graph in
Figure 1 with two connected components, because all nodes have the same number of degrees, theywill have exactly the same rooted subtrees.
Nevertheless, the nodes in the left component is clearlydifferent from nodes in the right component, because the left component is a 3-cycle and the right one is a 4-cycle.
Such cases can not be discriminated by GNNs or 1-WL test.
We refer to this type of
GNN as plain GNN or basic GNN.
On the other hand, even when a carefully-designed GNN can differentiate all node substructures in a graph, Zhang et al.
[46] showthat the learned node representation still cannot effectively performstructural learning tasks involving multiple nodes, including linkpredictions and subgraph predictions, because the nodes are not ableto learn the relative structural information.
We will discuss this withmore details in Section 3.1.
These limitations motivate recent research that pushes the limit of
GNN’s expressiveness.
One branch is to use higher-order GNNs that mimic higher-dimensional
Weisfeiler-Lehman tests [18, 21].
They extend node-wise message passing to node-tuple-wise andobtain more expressive representations.
Another branch is to use labeling methods.
By extractingsubgraphs around the nodes of interests, they design a set of subgraph-customized labels as additionalnode features.
The labels can be used to distinguish between the nodes of interests and other nodes.
The resulting node representations can encode subtructures which plain GNNs cannot, like cycles.
However, the major drawback of these methods is that compared to plain GNNs, their efficiency issignificantly compromised.
Higher-order GNNs update embeddings over tuples and incur at leastcubic complexity.
Labeling tricks usually require subgraph extraction on a large graph and running GNNs on mini-batches of subgraphs.
Because the subgraphs overlap with each other, the aggregatedsize of the subgraphs is possibly hundreds times of the large graph.
Nowadays GNNs are massive,the increased graph computation cost makes labeling methods inapplicable to real-world problemswith millions of nodes.
We observe that a great portion of the labeling methods rely on the geodesics, namely the shortestpaths, between nodes.
For node representation, Distance-Encoding(DE) [20] labels the target nodes’surrounding nodes with their shortest path distance to the target nodes and achieves higher than 1-WLtest expressiveness.
Current state-of-the-art methods for link prediction, such as SEAL [43], leveragethe labeling tricks in order to capture the shortest path distance along with the number of shortestpaths between the two nodes of the link.
Our proposed Geodesic-GNN (GD-GNN) model is designedto also capture this characteristic, while addressing the efficiency issue of previous methods.
Specifically, our GD-GNN model learns a function that maps the learning target to a geodesicaugmented target representation (the target can be node, edge, graph, etc).
The model consists of twounits, a GNN unit and a geodesic pooling unit.
The two units are connected to perform end-to-endtraining.
We first apply the GNN to the graph to obtain embeddings of all nodes.
Then, for eachtarget, we find the task-specific geodesics and extract the corresponding GNN representations ofnodes on the geodesics.
The task-specific geodesics have three levels.
For node-level tasks, weextract the geodesics between the target node and all of its neighbors.
For edge-level tasks, weextract the geodesics between the two target nodes of the edge.
For graph-level tasks, we get thegeodesic-augmented node representation for each node as in node-level tasks and readout all noderepresentations as the graph representation.
Finally, using the geodesic pooling unit, we combinethe geodesic representation and target representation to form the final geodesic augmented targetrepresentation.
Figure 2 summarizes the GD-GNN framework.
All task-specific geodesics are formed by one or multiple pair-wise geodesics between two nodes.
We propose two methods to extract the pair-wise geodesic information, horizontal or vertical.
Thehorizontal geodesic is obtained by directly extracting one shortest path between the two nodes.
Thevertical geodesic is obtained by extracting all direct neighbors of the two nodes that are on any oftheir shortest paths.
They focus on the depth and breath respectively, and possess different properties.
Horizontal geodesic is good at capturing long distance information not covered by the GNN.
Verticalgeodesic is provably more powerful than plain GNN.
Moreover, by incorporating the informationof the subgraph induced by the vertical geodesic nodes, we can use GD-GNN to distinguish somedistance-regular graphs , a type of graph that many current most powerful GNNs cannot distinguish.
During inference, the GNN unit is applied only once to the graph, after which we feed nodes andgeodesic information to the efficient geodesic pooling unit.
This reduces the number of GNN runsfrom the query number (or number of nodes for graph-level tasks) to a constant of one, which largelyreduces the computation cost.
Compared to labeling methods, GD-GNN only requires one GNN run and is much more efficient.
Compared to plain GNN methods, GD-GNN uses geodesic informationand thus is more powerful.
We conducted experiment on link prediction and graph classificationacross datasets with different scales and show that GD-GNN is able to significantly improve theperformance of basic GNNs and achieves very competitive results compared to state-of-the-artmethods.
We also demonstrate GD-GNN’s runtime superiority over other more-expressive GNNs.
Preliminaries
A graph can be denoted by G =
( V , E , R ) , where V = { v 1 , ..., v n } is the node set, E ⊆{ ( v i , r, v j ) | v i , v j ∈ V , r ∈ R} is the edge set and R is the set of edge types in the graph.
When|R| = 1 , the graph can be referred to as homogeneous graph.
When |R| > 1 , the graph can bereferred to as heterogeneous graph.
Knowledge graphs are usually heterogeneous graphs.
The nodescan be associated with features X = { x v |∀ v ∈ V} .
For simplicity, we use homogeneous graphs toillustrate our ideas and methods.
We follow the definition of message passing GNNs in [39] with MESSAGE and AGGREGATEfunctions.
Such GNNs iteratively update the node embeddings by receiving and combining theneighbor nodes’ embeddings.
The k -th layer of a GNN can be expressed as:
[Equation]
where h ( k ) v is the node representation after k iterations, h (0) v = x v , N ( v ) is the set of directneighbors of v , m u is the message embedding.
Different GNNs vary mainly by the design choice ofthe MESSAGE and AGGREGATE functions.
In this paper, we use h v to represent the final noderepresentation of v from the plain GNN.
Throughout this paper, we use bold letters to representvectors, capital letters to represent sets, and N k ( v ) to refer to nodes within k-hop neighborhood of v .
Geodesic Graph Neural Network
Geodesic is conventionally defined as the shortest path between two nodes and indeed in homogeneousgraph it encodes only the distance.
However, we found that when combined with GNNs and noderepresentations, geodesics express information much richer than shortest path distance alone.
Inthis section, we first discuss the intuition behind GD-GNN and its effectiveness.
Then, we describethe GD-GNN framework and how its GNN unit and geodesic pooling unit are combined to makepredictions.
Finally, we provide theoretical results of the better expressiveness of GD-GNN.
3.1 Intuition and Overview
The key weakness of directly using plain GNNs is that the embeddings are generated withoutconditional information.
For node-level tasks, as pointed out in [42], a plain GNN is unaware ofthe position of the target node in the rooted subtree, and hence the target node is oblivious aboutthe conditional information about itself.
For edge-level tasks, Zhang et al.
[46] show that despite anode-most-expressive-GNN, without node labeling, the embedding of one node in the edge is notable to preserve the conditional information of the other node.
Concretely, take link prediction as
[Figure]
an example, in Figure 3a, we have two target links AB and AC.
Since AB is already connected by alink and AC is not, they should be represented differently.
On the other hand, All nodes in the graphare symmetric and should be assigned the same embeddings using any GNN.
Hence, if we simplyconcatenate the node representations of AB and AC, they will have the same link representations, andcannot be differentiated by any downstream learner.
To overcome this problem, previous methodsrely on labeling tricks to augment the computation graphs with conditional information.
When node
B and C are labelled respectively in two separate subgraphs, the resulting computation graphs of A isdifferent as shown in Figure 3b and 3c.
Notice that, for the labeling methods to take effect, subgraph extraction is necessary.
However,relabeling the graph and applying a GNN to every target to predict is extremely costly.
Hence, wewonder whether we can also incorporate the conditional information between nodes without thecustomized labeling?
Observe that in the example, the computation graph captures the number oflayers before node A first receives a message from a labelled node, which is essentially the shortestpath distance d between A and the other node.
This implies that even when we only have noderepresentations from a plain GNN, by adding the distance information (which is 1 and 3 for AB and
AC respectively) independent of the computation graph, we can still distinguish the links.
While the previous example demonstrates the effectiveness of a simple shortest path distance metric,we notice that distance alone is not sufficient in many times.
For example, in Figure 3d, node B andnode C are symmetric and hence should be assigned the same node embedding when the graph is notlabelled.
However, the links AB and AC are apparently different as v 1 connecting AB connects toanother group of nodes whereas v 2 connecting AC does not connect to any other node.
If we only useshortest path distance between AB and AC to assist the prediction, we still cannot differentiate them.
However, if we inject the entire geodesic expressed by the corresponding node embeddings alongthe shortest path to the model, we are able to tell AB and AC apart due to v 1 and v 2 ’s differentembeddings.
Finding the shortest path can be seen as extracting a chain of nodes whose embeddings are allone, whereas in the neural setting, the nodes on the same chain can have different representationsby encoding their own neighborhood.
The extension of shortest path to the neural setting grantsstronger expressive power to the geodesics.
Note that the geodesic generation process and the GNNprocess is still decoupled , because we select the geodesic nodes after all node representations aregenerated.
Dwelling on this decoupling and conditioning philosophy, GD-GNN can run the GNN onlyonce to generate generic node embeddings, and then use geodesic information to inject conditionalinformation into the model in a much more efficient way.
3.2 Pair-wise Geodesic Representation
GD-GNN has different ways to leverage geodesic information for different levels of tasks, butthey all have the same basic component, pair-wise geodesic representation.
It consists of structuralinformation of nodes on the shortest paths of the pair.
Directly extracting all such nodes is timeconsuming and sometimes not effective, hence we present two primary ways to learn the geodesicrepresentation, horizontal and vertical .
3.2.1 Horizontal Geodesic Representation
Horizontal Geodesic Representation is designed to directly capture the distinctiveness as describedin the previous example shown in the Figure 3d.
It first finds a shortest path between two nodes asthe geodesic.
If there are multiple paths between them, we randomly choose one.
The nodes on theshortest path is denoted by P ( H ) ( u,v ) = { u, n 1 , n 2 , ..., n d − 1 , v } with a length of d .
Using P ( H ) ( u,v ) we canconstruct the geodesic representation g ( u,v ) by a geodesic pooling layer.
[Equation]
To prevent extremely long geodesics which may cause overfitting, we limit the maximum length to amanageable constant d max .
Any pair with distance larger than d max will have infinite distance and azero geodesic representation.
Because of the high computation cost of GNNs, the number of GNN layers is usually very small.
Using horizontal geodesics, we are able to connect nodes that are far away and do not coexist in a
GNN’s limited-height computation graph.
3.2.2 Vertical Geodesics Representation
Horizontal geodesics seem like an intuitive choice to directly connect the pair of nodes by a sequenceof adjacent nodes.
However, a natural question to ask is whether a single path is enough to representthe conditional information between the pair?
Realizing this limitation, we propose vertical geodesicrepresentation that includes all geodesic information in an efficient way.
For a pair of nodes u and v , vertical geodesic extracts the direct neighbors of them that are on anyof their shortest paths.
Specifically, let the distance between u and v be d ( u, v ) , we have the nodes ontheir shortest paths to be W ( u,v ) =
{ w | d ( u, w ) + d ( w, v ) = d ( u, v ) , ∀ w ∈ V} .
The geodesics canbe represented as,
[Equation]
where P ( V ) ( u,v ) ,i := W ∩ N ( i ) , i ∈ { u, v } .
[Equation]
Here P ( V ) ( u,v ) ,i represents the nodes in the geodesic that are also the direct neighbors of i .
For sometasks, we only consider one P ( V ) ( u,v ) ,i as the geodesics instead of both.
We use a pooling layer togenerate vertical geodesic
[Equation]
While the vertical geodesic does not explicitly encode a long shortest path, it can still capture shortshortest path information due to pooling the embeddings of the nodes in it.
Compared with thehorizontal geodesic that takes one random shortest path, the vertical geodesic focuses more on breadthby incorporating all direct neighbors on any shortest paths.
Furthermore, when combined with aninjective graph pooling function, the vertical geodesic can help distinguish a great portion of regulargraphs that plain GNNs cannot.
We will demonstrate this in Section 3.4.
Unlike nodes in horizontal geodesic, nodes in vertical geodesic induce a more complex subgraph.
Consider only P ( V ) ( u,v ) ,v (we will use P v here for simplicity) and the subgraph G v induced by nodes in
P v .
We can have a pooling function which encodes the graph structure of G v ,
[Equation]
The pooling layer can be a small plain GNN itself, and the initial embedding of G v can take thepooled node embeddings from the main GNN.
However, to keep our method efficient, we only addthe degree of nodes in the subgraph to the node embeddings.
And the vertical geodesic representationbecomes,
[Equation]
6) where ⊕ means concatenation.
Note that a node’s degree in the subgraph is different from thatin the main graph.
This can be seen as a 1-layer GNN applied onto G v with identical initial nodeembeddings.
Even with this simple pooling design, the vertical geodesic is able to help distinguishsome distance regular graphs, which we cover in section 3.4.
Because vertical geodesics do not encode distance directly, we concatenate distance to the verticalgeodesics as an additional feature.
Like in horizontal geodesics, when d ( u, v ) is greater than d max ,we do not generate geodesic representation for the pair, and set their distance to infinity.
3.3 Task specific geodesic representation
Task specific geodesic representation is a collection of pair-wise geodesic representations.
Here, wefocus on three levels of tasks, node-level, edge-level, and graph-level.
For node-level tasks, the goal is to generate a representation z ( n ) v for a target node v .
We first findv ’s k-hop neighbor nodes S v = N k ( v ) .
Then for each node s i in S v , we find the pair-wise geodesic,g v,s i .
Note that in node level tasks, vertical geodesic only contains the geodesic nodes close to s i ,
P ( V ) ( v,s i ) ,s i .
Then the geodesic representations of the neighborhood is combined by a pooling layer,
[Equation]
While the GNN node embeddings of the neighborhood is not conditioned on v , by adding the geodesicinformation to every neighbor, we inject the conditional information of v to its neighbors.
Then thecollective information of the neighbors helps distinguish v .
For edge-level tasks, the goal is to generate an edge embedding z ( e ) ( u,v ) for nodes u and v .
Here, wedirectly use the pair-wise geodesic for the two nodes, z ( e ) ( u,v ) =
g ( u,v ) .
The advantage of this setup isthat, when the downstream task is to rank one link against N other links with one same node v , weonly need to compute the distance vector from v once and find the geodesic nodes efficiently.
Thisallows us to perform 1-vs-N link prediction on large scale knowledge graphs, where subgraph-basedmethods may not be applicable.
For graph-level tasks, we need to generate a graph embedding z ( g ) G for G =( V , E ) .
We use anotherpooling layer to summarize the geodesic-augmented representation of every node in the graph,
[Equation]
GD-GNN can also be easily generalized to other levels of tasks such as subgraph-level.
3.4 Expressive Power of GD-GNN
Distinguishing power on graph-level tasks is often used to characterize the expressiveness of a GNNmodel.
For GNN models to be more expressive than 1-WL test, we can test whether they are able todistinguish regular graphs.
In the following theorem, we show that given an injective R ( g ) , the graphembedding z G generated based on vertical geodesics can distinguish most regular graphs.
Theorem 1.
Let the graph pooling layer R ( g ) be injective given input from a countable space.
Consider all pairs of n-sized r-regular graphs, where 3 ≤ r < √ 2 log n , n is the number ofnodes in the graph.
For any small constant ϵ > 0 , there exists a GD-GNN using vertical geodesicrepresentations with d max = ⌈ ( 12 + ϵ ) log n log [Equation] r
[Equation]
ϵ )
⌉ which distinguishes almost all (1 − o (1)) suchpairs of graphs.
We include the proof of Theorem 1 in Appendix A.
This theorem implies that GD-GNN with verticalgeodesics is more expressive than basic GNNs (and 1-WL) and can discriminate almost all r -regulargraphs.
Moreover, GD-GNN only needs to search a small neighborhood for geodesics to achieve thehigher expressiveness.
This guarantees the good efficiency and performance of GD-GNN.
Moreover, when the vertical geodesic pooling layer encodes the subgraph induced by the geodesicnodes (using Equation 6 for geodesic pooling), GD-GNN is able to distinguish some distance regulargraphs.
Figure 4 shows two distance regular graphs, Shrikhande graph and 4x4 rook’s graph.
Theyhave the same intersection array and number of nodes.
As proved by Li et al.
[20], distance encodingis not able to distinguish this pair of graphs, because nodes in the two graphs will be assigned the sameembedding.
However, vertical geodesic is able to discriminate them.
Suppose the black nodes are thetarget nodes, the brown nodes are one distance away from the black nodes and the light blue nodesare two distance away from the black nodes.
Consider the light blue nodes in both graphs, they eachhave a pair of brown common neighbors with a black node.
The brown pair is the vertical geodesicsof one light blue node.
If we count the number of edges in the subgraph formed by each brown pair,we will see that four out of nine pairs form a one-edge graph in the Shrikhande graph, while zero outof nine pairs form a one-edge graph in the 4x4 rook’s graph.
Since Equation 6 directly captures thisand will give four light blue nodes in Shrikhande graph different pair-wise geodesics with respect to
Complexity.
The main computation advantage of GD-GNN comes from the fact that, for GD-GNN,the number of GNN runs stays as a constant of one, while subgraph-based methods incur a number of GNN runs which increases linearly with respect to the number of queries.
Note that the overhead ofgeodesic finding has the same complexity as that for subgraph extraction.
Hence, GD-GNN maintainsthe good efficiency of basic GNN methods, while injecting the high-expressiveness of subgraphmethods into the model.
A detailed complexity analysis can be found in Appendix B.
Related Works
GNNs beyond 1-WL test.
Designing GNNs with stronger representation power is a fundamentaltask in GNN research.
Xu et al.
[39] first proved that the expressiveness of basic GNNs, such as
GCN [15], GraphSAGE [13], and GAT [36], is upper-bounded by the 1-WL test.
Following worksconducted extensive research to break this limit.
They can be sorted into several directions: (1) Earlierworks proposed higher-order GNNs that are as expressive as k-WL algorithm, such as k-GNN [18],
RingGNN[6], and PPGN[21].
These methods require message passing on node tuples that growsexponentially with respect to k. The increased computation cost makes them not scalable to largegraphs.
(2) Another type of methods, like GSN [4] and MotifNet [22], incorporates isomorphismcounting on certain graph substructures like cycles, triangles, and cliques, similar to earlier kernelbased methods [40, 11].
They essentially augment the models with features undetectable by GNNs.
However, a properly-design substructure usually requires expert knowledge which is not available forall data.
(3) Some other works use identity or random node labeling within the plain GNN framework.
ID-GNN [42] extracts a subgraph around the target node and perform heterogeneous message passing.
DE-GNN [20] also requires subgraph extraction but labels nodes in the subgraph with distance tothe targets.
They both have stronger expressiveness than basic GNNs.
Abboud et al.
[1] shows that
GNNs possess universality with random node feature initialization (RNI).
Nevertheless, RNI alsoinduces a huge sample space, and GNNs adopting RNI is very hard to train.
Some recent workspropose to use additional subgraph structures to augment the node embeddings.
GraphSNN [37]incorporates properties of overlapping subgraphs between adjacent nodes to be more expressive than1-WL.
Nested-GNN [45] applies GNN to ego-subgraphs of every node and proves that it is able todiscriminate regular graphs.
Sharing the same idea of Nested-GNN, GNN-AK[47] uses a differentsubgraph pooling strategy and uses subgraph dropping to improve the model scalability.
Our work isfundamentally different from existing models.
It does not require subgraph extraction and inject theconditional information between nodes via geodesic selection.
This keeps our model both efficientand more expressive than basic GNNs.
Link prediction using GNNs.
Earlier works [16, 15, 7, 35] applied basic GNNs to link predictionusing an auto-encoder framework, where the GNN serves as the encoder of the nodes, and edges aredecoded by their nodes’ encoding vectors.
Another line of works, including SEAL [43], IGMC [44],and GraIL [32], use GNNs to encode a labelled subgraph to form the link representation.
Later work[30, 46] reveals that a good node-level/graph-level representation does not necessarily lead to a goodedge-level representation.
Zhang et al.
[46] systematically show that the labeling trick bridges thegap between node-level and edge-level GNN expressiveness.
Sharing the spirits of labeling trick, NBFNet [49] extends the generalized Bellman-Ford algorithm to the neural setting, and can encodemany traditional path-based methods.
NBFNet has a better scalability when there are multiple linkssharing one same node, but has the same complexity as other labeling methods when all links in the query set have different nodes.
On the contrary, our GD-GNN maintains the same good efficiency inany scenario. Some other works also propose to use path information between links to enhance theprediction.
WalkPooling [23] computes the random-walk probability to form a random walk profilefor link prediction. However, this method is still subgraph-based, whereas GD-GNN is also able toencode path information and remains computationally friendly in large graph context.
Experimental Results
5.1 Link Prediction
Datasets.
We use several types of datasets for link prediction.
(1) Knowledge Graph (KG) inductivelink prediction datasets.
Inductive link prediction means that the model is trained on a traininggraph while tested on a different graph.
This requires the model to generalize to unseen entities,which is a property preserved by a limited number of models.
We follow the standard inductivesplit of WN18RR[9] and FB15K237[34] as in [32].
We rank each positive link against 50 randomlysampled negative links with the same head/tail as the positive link, and report Hit@10 ratio.
(2) OGBlarge-scale link prediction dataset [14], including OGBL-COLLAB and OGBL-PPA.
We use theofficial split provided by the OGB team.
We rank each positive link against a set of provided negativelinks in the dataset, and report Hit@N ratio.
(3) Results of citation datasets can be found in Appendix D. (4) Full results on the knowledge graph datasets are included in Appendix C. These datasets varyby graph size and demonstrate GD-GNN’s capability across scales.
Baselines.
We compare GD-GNN with Rules Mining methods, DRUM [26] and NeuralLP [41],and GNN-based methods, NBFNet [49] and GraIL [32] for the knowledge graph tasks.
For the
OGB dataset, we compare GD-GNNs with other GNN-based methods achieving top places on the OGB link prediction leaderboard, including Deep Walk [24], Resource Allocation (RA) [48], SEAL,
Deeper GCN [19].
Implementation Details.
For link prediction, we use GCN (RGCN for knowledge graph) as thebasic GNN.
For all the datasets, we search the number of GNN layers in { 2 , 3 , 4 , 5 } [Table] and the maxsearch distance for geodesic, d max , is same as the GNN layers.
We use 32 hidden dimensions for allfully-connected layers in the model.
We train 50 epochs with a batch size of 64 for the knowledgegraph datasets
[Table]
and we train 50 epochs with a batch size of 4096 for the OGB datasets.
We record thetest score using the model with the best set of hyperparameters chosen based on the performance onthe validation set.
The experiment is repeated 5 times and we take the average of the scores
[Table]
[Table]
[Table]
Results and discussion.
We achieved SOTA performance on the inductive datasets.
GD-GNN isable to significantly improve over rule-mining methods and subgraph based method, GraIL.
We alsooutperform NBFNet on most of the datasets.
Note that NBFNet is only efficient when the query linkscontain many nodes in common, whereas GD-GNN’s efficiency is not bounded by such limitation.
On the OGBL-COLLAB dataset (Table 2), vertical GD-GNN achieved SOTA performance.
On the OGBL-PPA dataset, despite being outperformed by top performance methods, GD-GNN is able tokeep a very competitive result.
On both OGB datasets, GD-GNN improves significantly over its basic
GNN, GCN (a relative increase of 22% for OGBL-COLLAB and 144% for OGBL-PPA).
Moreover,as shown in Table 3, GD-GNN has running-time superiority over methods achieving comparableperformance, and has memory advantage to handle large graphs where NBFNet can not.
To makefair running time comparison, we run all models on 32 CPUs and 1 Nvidia GeForce 1080Ti GPU.
We also include ablation studies on all components of GD-GNN in Appendix E.
[Table] [Table]
5.2 Graph Classification
Datasets.
We use two types of datasets for graph classification.
(1) TU datasets contain D&D[10],
MUTAG[8], PROTEINS[10], PTC_MR[33], ENZYMES[28].
The evaluation metric is Accuracy(%).
We follow the evaluation protocol in [45], where we use 10-fold cross-validation to computethe score.
(2) OGB datasets[14], including OGBG-MOLHIV and OGBG-MOLPCBA.
The taskof OGBG-MOLHIV dataset is to predict whether a molecule inhibits HIV virus.
We use AUC to [Table] evaluate OGBG-MOLHIV dataset.
OGBG-MOLPCBA dataset has 128 classification tasks.
Wefollow the evaluation protocol provided by the OGB team and use AP averaged across all classes toevaluate the dataset.
Baselines.
For the TU dataset, we compare with another plug-and-play GNN framework, NGNN[45].
Both GD-GNN and NGNN use GIN and GCN as their basic GNNs.
For the OGB dataset, we use GINas the basic GNN, we compare GD-GNN with methods achieving top places on the OGB leaderboard,including GCN[15], GIN[39], GSN[4], GINE [5], GIN-AK [47], Nested GNN [45].
Implementation details.
We use vertical geodesic in the graph-level tasks because of its theoreticaladvantage over horizontal GD-GNN.
For TU datasets, we search the number of GNN layers in{2,3,4}, and keep the geodesics searching distance the same as the GNN layers.
We train on the TU dataset for 100 epochs with a batch size of 64.
For OGB datasets, we search the number of GNN layers in {2,3,4,5} [Table] and train for 100 epochs with a batch size of 256.
Following previoussuccessful practice
[Table] we adopt virtual node technique [12] to increase the connectivity of graphs forthe OGBG-MOLPCBA dataset
[Table]
[Table] [Table]
Results and discussions.
On the TU dataset (Table 5), GD-GNNs consistently outperform the basic
GNNs with an average relative performance increase of 11.8%.
Compared to NGNN (7.9% relativeperformance increase), GD-GNN is able to bring larger performance increase to the basic GNN.
Thisshows GD-GNN’s enhancing power.
On the OGB dataset (Table 6), note that the GNN we use for
GD-GNN, GIN, is not the most powerful GNN, but GD-GNN is still able to achieve very competitiveresults compared to usually heavily-tuned methods on the OGB leaderboard.
Similarly, GD-GNNis able to increase the performance of GIN by a significant amount. We also note the efficiencyadvantage of GD-GNN in Table 4. GD-GNN is able to maintain a running time very close to thebase GNN and significantly improve over other similar performance methods.
To study the effect ofdifferent components of GD-GNN, we include ablation studies of graph-level geodesics in Appendix E.
Conclusions
In this paper, we propose a general GNN framework which incorporates geodesic information togenerate node, link, and graph representations.
Our framework is more powerful than basic GNNs and is more efficient than current more-expressive GNNs.
Theoretical results show that our models can distinguish almost all regular graphs (which 1-WL always fails to) and even some distance regular graphs (which distance encoding fails to).
Experimental results show that our framework achieved very competitive performance to state-of-the-art models with significantly less time.
Our frameworkhas the potential to make GNNs encoding conditional information really applicable to large-scalereal-world graphs
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] We faithfully summarize the paper in the abstract.
(b) Did you describe the limitations of your work?
[Yes] As described in the Section 5 and Appendix C
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work
only involves existing datasets and solves general graph learning problem without
negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Assumption isdescribed in the Theorem
(b) Did you include complete proofs of all theoretical results?
[Yes] In Appendix A3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] As described in the implementation details of Section
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] As shown in the experiment result tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
In the discussion of Section
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] The datasets and
[Table]
model source code repositories are all cited.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] The datasets and source code are open source information allowed
for research use.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] The data we are using is known to be de-
identified.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and
[Table]
total amountspent on participant compensation?
[N/A]
Appendix
A Proof of Theorem 1
The proof of Theorem 1 generally follows the proof of Nested-GNN’s expressiveness[45].
Let Q kv, Gbe the set of nodes in G that are exactly k distance away from node v .
We can have the followingdefinition,
Definition A.1.
The edge configuration bewteen Q kv, G and Q k +1 v, G is a list C kv, G =
( a 1 ,kv, G , a 2 ,kv, G , ... )
where a i,kv, G denotes the number of nodes in Q k +1 v, G of which each has exactly i edges from Q kv, G .
The proof leverages the fact that nodes in randomly sampled regular graphs are very likely to havedifferent edge configuration within their small neighborhood, and GD-GNN directly captures thisdistinction to discriminate the graphs.
Formally, we have the following excerpted Lemma from [45],
Lemma 2.
For two graphs G (1) =
( V (1) , E (1) ) and G (2) =
( V (2) , E (2) ) that are uniformly independently sampled from all n-node r-regular graphs, where 3 ≤ r < √ 2 log n , we pick anytwo nodes, each from one graph, denoted by v 1 and v 2 respectively.
Then there is at least onei ∈ (log n log( r − 1 − ϵ ) , ( 12 + ϵ ) log n log( r − 1 − ϵ ) ) with probability 1 − o ( n − 1 ) such that C iv 1 , G (1) ̸
= C iv 2 , G (2) .
Proof.
To prove Theorem 1, we first assume, without loss of generality, that the hidden dimension of
GD-GNN is one.
We know plain message passing GNN will generate the same embeddings for everynode in the regular graph, because all nodes have the same degree/subtree.
Hence, we can normalizeall the node representations to a constant of one.
For simplicity, we also reduce vertical geodesicpooling function R ( V ) gd in Equation 4 to a simple sum of the representations of the geodesic nodes.
Let d max = ⌈ ( 12 + ϵ ) log n log( r − 1 − ϵ )
⌉ , the maximum distance in which GD-GNN searches for a targetnode’s neighbors.
We then consider node v i
and it’s d max -hop rooted subgraph G d max v i , the verticalgeodesic pooling process is essentially counting the number of geodesic nodes of each node v j in
G v i .
And the pair-wise vertical geodesics between the v i and v j becomes,
[Equation]
(9) where d ( v i , v j ) is the distance between v i and v j , and hence v j ∈ Q d
( v i ,v j ) v i , G vi
and g ( V ) v i ,v j records thenumber of edges of v j that connects to nodes in Q d ( v i ,v j ) − 1 v i , G vi , and its distance to v i .
We also havethe node-level pooling function to be a distance sorted bin count function on the set of pair-wisevertical geodesic representations.
It first sorts the geodesic representations by their second dimension(distance), and perform bin count on their first dimension (number of geodesic nodes) separately foreach distance.
Then GD-GNN produces an mapping of G v i to its exact set of edge configurations,
C v i =
( C 1 v i , G vi , C 2 v i , G vi , ... ) .
Let G (1) =
( V (1) , E (1) ) and G (2) =
( V (2) , E (2) ) be two graphs uniformly independently sampledfrom all n-node r-regular graphs.
We consider two nodes v i ∈ V (1) and v j ∈ V (2) .
Becaused max = ⌈ ( 12 + ϵ ) log n log( r − 1 − ϵ ) ⌉ , the sets of edge configurations of the rooted subgraphs, C v i and C v j aredifferent with probability 1 − o ( n − 1 ) , according to Lemma 2.
In such case, GD-GNN also maps theirrooted subgraphs to different edge configurations, and hence the node-level geodesic representationz ( n ) v i and z ( n ) v j generated by GN-GNN will be different with the same probability.
Consider v i and all node v j ∈ V (2) , by union bound, z v i ̸∈ { z v j | v j ∈ V (2) } with a probability of1 − o (1) .
B Complexity Analysis
The complexity of GD-GNN differs by tasks.
On almost all tasks, GD-GNN demonstrates superiorityin efficiency.
Take graph-level task as an example, to evaluate a graph with N nodes, GD-GNN needsto find node-level geodesics N times.
For each node-level geodesic, GD-GNN computes the shortestdistances between the node and its local subgraph, this takes O ( ae ) where a is the average number of nodes in the subgraph and e is the average degree of nodes.
In the local subgraph.
The total numberof nodes included in the geodesic is also bounded by the number of edges O ( ae ) .
For each node-levelgeodesic, we apply a pooling layer with complexity linear to the number of geodesic nodes, and thecomplexity of geodesic pooling unit is O ( aec ) , where c is the GNN hidden dimension.
The overallcomplexity of evaluating a N nodes graph is O
( T |E| c + N · aec ) , where O ( T |E| c ) is the complexityof one run of the GCN.
Since |E| = O ( Ne ) , we can rewrite the complexity as O ( NTec + Naec ) .
Tis almost always smaller than a , we have the overall complexity to be O
( Naec ) .
Consider a labelingbased method, its complexity is O ( NTaec ) , for applying GNN N times to N extracted subgraphs.
Comparing the two complexity, GD-GNN amortizes the expensive computation cost of GNN.
Inreality, the architecture of GNN layer is more complex with hidden dimension larger than the outputdimension and even has internal MLP layers.
This will exacerbates the already heavy computationload of the labeling method, whereas GD-GNN can apply the same complex GNN once, and enjoythe efficient inference complexity using a control-sized embedding.
The same argument applies toedge-level and node-level tasks in a large graph, the more prediction targets we have, the better theamortized complexity of GD-GNN. Hence, GD-GNN has great scalability.
C Knowledge graph link prediction
For knowledge graphs, we use RGCN[27] as our base GNN.
We evaluate GD-GNN on FB15K237[34]and WN18RR[9] knowledge graphs for both inductive and transductive link prediction tasks.
We usethe standard split in [34] for transductive link prediction, and follow [32] for inductive link prediction.
We compare with Rules Mining methods, DRUM [26] and NeuralLP [41], and GNN-based methods,
NBFNet [49] and GraIL [32], for the inductive link prediction tasks.
For transductive tasks, weadditionally compared to general knowledge graph link prediction methods that are not inductive,including TransE [3], RotatE [31], and RGCN [27].
For inductive dataset, we randomly sample 50negative links that have either the same head or tail as the positive link, we report the H@10 againstthe 50 negative links.
For the transductive dataset, we follow the filtered ranking protocol [3], wherewe rank a positive link against all negative links that have the same head (or tail) as the positive links.
We report H@N and Mean Reciprocal Rank (MRR) for the transductive setting. [Table] [Table] H@
H@ MRR H@ H@
H@
DRUM 34 .25 .37 .51 .48 .42 .51 .58 .
NeuralLP 24 .- - 36 .43 .37 .43 .56 .
TransE 29 .- - 46 .22 .- - 50 .
{+RotatE 33 .24 .
+}
{+RGCN 27 .18 .30 .
+}
NBFNet 41.
{+GD-GNN-Vert 23 .15 .26 .45 .
+}
{+GD-GNN-Hor 
+}
We achieved SOTA performance on the inductive datasets.
GD-GNN is able to significantly improveover rule-mining methods and subgraph based method, GraIL.
We also outperform NBFNet on mostof the datasets.
We note that horizontal geodesic is able to exceed the performance of vertical geodesicon some datasets.
We suspect this is because horizontal geodesics implicitly capture the inter-nodetranslation process as in the embedding methods, and this piece of information is more importantthan graph structure in the knowledge graph setting.
We also acknowledge that GD-GNN does notperform as well in the transductive setting.
However, GD-GNN is still able to greatly improve theperformance of RGCN on WN18RR dataset, and RGCN requires trainable node embeddings while
GD-GNN does not.
We suspect the performance discrepancy between the homogeneous setting andthe knowledge graph setting is due to the increased number of edge types.
WN18RR dataset has 11types of relations, FB15K237 has 237.
Note that GNN methods like NBFNet[49] train a set of weightsfor every target relation type, meaning that their message passing is conditioned on the relation,while GD-GNN only uses one set of weights.
We notice that GD-GNN is able to outperform someembedding methods on WN18RR dataset, while not performing as good on FB15K237, possibly dueto the fact that FB15K237 dataset contains much more relations than WN18RR dataset.
This alignswith our hypothesis.
A potential solution is to also train multiple GD-GNNs that each handles onetarget relation type, note that we still only need to run each relation-specific GD-GNN once, and keepthe good amortized property of GD-GNN.
We leave this to future work.
D Citation dataset results
Citation datasets[29] include Cora, Citeseer, and Pubmed datasets.
We follow the experimental settingin [49] and use 85:5:10 for the train/valid/test links split.
The evaluation metric is ROC-AUC (AUC)and Average Precision (AP) scores.
We compare GD-GNN with popular link prediction methods
VGAE [16], SEAL[43], NBFNet[49] for the citation datasets.
Note that for citation data, we do notuse the pre-trained node embeddings from the publications’ content, incorporating such informationonly increases the power of GD-GNN.
We use GCN as the basic GNN.
We search the number of GNN layers in { 2 , 3 , 4 , 5 } , and the maxsearch distance for geodesic, d max , is same as the GNN layers.
We use 32 hidden dimensions forall fully-connected layers in the model.
We train 100 epochs with a batch size of 64 for the citationdatasets.
[Table]
On the citation dataset (Table 9), GD-GNN achieved very competitive results compared to the SOTAmethod NBFNet, and GD-GNN is more efficient than NBFNet as demonstrated in Table 3.
E Ablation study
GD-GNN has many components, and here we present ablation study of GD-GNN to demonstratethe effect of each component.
For link prediction dataset, we consider 4 settings, plain GCN (GCN),
GCN and distance between the two target nodes of the link (GCN+Dist), GD-GNN-Vertical withoutgeodesic degree (GD-GNN-Vert), and GD-GNN-Vertical with geodesics node degree (GD-GNNVert-Deg).
Note that the results we report in Section 5 include geodesic node degree.
From the experiment results (Table 10), we can see that distance is already able to assist link prediction,especially on sparser datasets like Cora and OGBL-COLLAB.
However, on OGBL-PPA, where theconnectivity of the graph is much larger, distance does not help, whereas geodesic representation isable to significantly improve the expressive power of basic GNNs.
Also, when the graph is sparse, it [Table] is rather unlikely for the geodesic nodes to form subgraphs, meaning that most of nodes on the verticalgeodesic have a degree of zero, and hence geodesic degree does not improve much.
Meanwhile, in
OBGL-PPA, geodesic degree is able to increase the performance of GD-GNN, which aligns with theintuition and example we present.
For graph classification tasks, we study two simpler version of GD-GNN.
We reduce the verticalpair-wise geodesic function (Equation 4) to be a pooling function only on the embeddings of the pairof nodes.
Then, the node-level geodesic is essentially combining the node embedding and its k-hopneighbor embedding, we refer to this as Nei.
Another version is we append the pair-wise distance tothe neighbor embeddings, we refer to this as Dist.
[Table]
From Table 11, we can see that the neighbor embedding alone is not very effective, and we needto assist it with pair-wise distance, a explicit form of geodesic to improve the performance.
Byincorporating the geodesic representation, we are able to consistently improve over the ’Nei+Dist’version, which aligns with Theorem 1.
Geodesic degree does not give significant improvement on
TU dataset possibly due to its high testing variance, but we can still see notable improvement on
OGBG-MOLHIV dataset.