OOD Link Prediction Generalization Capabilities of Message-Passing GNNs in Larger Test Graphs
Anonymous Author(s)
AfﬁliationAddress email
Abstract
This work provides the ﬁrst theoretical study on the ability of graph Message Passing Neural Networks (gMPNNs) —such as Graph Neural Networks (GNNs)—to perform inductive out-of-distribution (OOD) link prediction tasks, where deployment (test) graph sizes are larger than training graphs.
We ﬁrst prove nonasymptotic bounds showing that link predictors based on permutation-equivariant(structural) node embeddings obtained by gMPNNs can converge to a randomguess as test graphs get larger.
We then propose a theoretically-sound gMPNN thatoutputs structural pairwise (2-node) embeddings and prove non-asymptotic boundsshowing that, as test graphs grow, these embeddings converge to embeddings of acontinuous function that retains its ability to predict links OOD.
Empirical resultson random graphs show agreement with our theoretical results.
Introduction
Link prediction is the task of predicting whether two nodes likely have a missing link [1, 10, 27, 34,58].
Link prediction tasks arise in many settings, ranging from predicting edges on bipartite graphsbetween users and products or content in recommender systems [5, 9, 28, 29, 35, 54], to knowledgegraph reconstruction [4, 12, 18, 47, 58, 59], to predicting protein-protein interactions [49].
In recent years, there has been growing interest in applying neural network models to inductive linkprediction tasks.
Inductive link prediction considers methods trained on a graph G tr and deployedat test time on another graph G te .
It also encompasses the task of training the method on a smallerinduced subgraph G tr of a larger graph G te , then deploying it on the entire graph.
In particular, ourwork focuses on graph message-passing Neural Networks (gMPNNs) [19, 52] or, more precisely, thewidely used Graph Neural Network (GNN) framework [7, 8, 11, 20, 22, 25, 53, 56, 61].
Our work asks the following questions: Are link prediction methods able to cope with the task ofinductive out-of-distribution (OOD) link prediction, where (unseen) test graphs are signiﬁcantlylarger than training graphs?
How can these OOD link prediction tasks be theoretically deﬁned?
Canwe obtain non-asymptotic bounds on the generalization capabilities of these methods?
The majority of today’s link prediction methods are based on a similar principle.
Consider anattributed graph G =
( V, E ) , with node set V = { 1 , ..., N } , edge set E ✓ V ⇥ V , and node features
F 2 R N ⇥ F 0 , F 0  1 .
Then, given a pair of nodes i, j 2 V , after T  1 iterations over G , thesemethods produce associated node embeddings (representation vectors) ⇥
• i , ⇥ • j 2 R F T , F T  1 ,which are then used in a link function ⌘ • : R F T ⇥ R F T !
[0 , 1] such that ⌘ • ( ⇥ • i , ⇥ • j ) predicts theprobability that i and j have a missing link in G .
In our notation we will denote all node embeddingsand associated functions with the superscript “ • ”.
Henceforth we denote gMPNNs that outputstructural node embeddings as gMPNNs • .
Node embeddings.
The ﬁrst part of our work considers a subset of these methods, where the outputnode embeddings are permutation equivariant (a.k.a.
structural node embeddings [57]).
Informally,a sequence of node embeddings ⇥ • 2 R N ⇥ F T given by an embedding method is permutationequivariant if for any arbitrary graph G and any permutation ⇡ 2 S N of the node indices, where S Nis the symmetric group, the resulting isomorphic graph G 0 = ( ⇡  V, ⇡  E, ⇡  F ) gets permutednode embeddings ⇥ •
= ⇡  ⇥ • , where ⇡  M deﬁnes the action of ⇡ on M (we will provide a formaldeﬁnition in Section 2).
We leave the study of OOD link prediction with positional node embeddings(a.k.a.
permutation-sensitive node embeddings [57]) to future work, since each such method has adifferent procedure to induce permutation sensitivity:
Adding features with one-hot encodings [25],the node’s own eigenvectors as positional encoding [14, 30], or random node features [33, 45, 68] thatare used to break the permutation equivariance of gMPNNs and force positional nodes embeddings.
The application of GNNs to link prediction tasks is made difﬁcult by the fact that, by construction,permutation-equivariant GNNs give the same embeddings ⇥ • i , ⇥ • j to any isomorphic nodes i, j in G , as noted by You et al.
[68] and Srinivasan and Ribeiro [57].
Isomorphic nodes are nodes that arestructurally indistinguishable in G (even when considering node features) except by their (assumedarbitrary) node indices i, j 2 V .
Thankfully, both large real-world and large random graphs (see the
Appendix for a discussion) tend to have only very few isomorphic nodes.
Therefore, in practice, nodeisomorphism in large graphs G is a non-issue for link prediction (as we see in our experiments).
Pairwise embeddings.
Taking a different route, Srinivasan and Ribeiro [57] shows that the linkprediction task between i and j can always be performed by pairwise embeddings ⇥ •• ij ( G ) , i.e., forany pair of nodes i, j in a graph G , there exists a pairwise embedding ⇥ •• ij ( G ) and a link function⌘ •• : R F T !
[0 , 1] such that ⌘ •• ( ⇥ •• ij ) approximates the probability that i and j have a hidden link.
In our notation we will denote all pairwise (joint 2-node) embeddings and associated functions withthe superscript “ •• ”.
And while one could still apply standard GNNs to obtain pairwise embeddingsby deﬁning a dual graph where all edges in G become nodes in G dual [42], there are new purposefullybuilt gMPNNs that generate pairwise embeddings, such as [64, 70, 72, 73].
Unfortunately, as thetest graph grows, we were unable to prove these methods converge to a procedure that is able toperform OOD link prediction tasks.
Hence, we propose a different family of gMPNNs for pairwiseembeddings, denoted gMPNNs •• henceforth.
The second part of our work considers the OODgeneralization capability of these gMPNNs •• .
Contributions.
In this work we study inductive OOD link prediction tasks for larger test graphsusing permutation-equivariant node and pairwise embeddings, ⇥ • and ⇥ •• , respectively.
Our workmakes the following contributions:
We provide a theoretical framework deﬁning OOD inductive link prediction tasks, where testgraphs are signiﬁcantly larger than training graphs.
We show that structural node embeddings from message-passing GNNs can fail in OOD linkprediction tasks if the test graph (from the same graph family) is signiﬁcantly larger than thetraining graph.
Our work ﬁlls an important gap in the literature , where Bevilacqua et al.
[6]studied the OOD capabilities of GNNs for graph classiﬁcation using random graph models.
Ourwork studies the OOD capabilities of GNNs for inductive link prediction in a similar setting.3.
We provide non-asymptotic bounds on the convergence of node and pairwise gMPNNs embeddingsto an easy-to-analyze continuous model.
Extensive empirical experiments using stochastic blockmodels (SBMs [55]) validate our theoretical results.
Our work focuses on providing a theoreticalunderstanding of the challenges of OOD link prediction tasks rather than propose real-world linkprediction tasks and compare baselines.
Our work, however, lays the theoretical foundation (andchallenges) for future application-focused works.
Preliminaries
Given an attributed graph G =
( V, E ) , with node set V = { 1 , ..., N } , edge set E ✓ V ⇥ V , adjacencymatrix A 2 { 0 , 1 } N ⇥ N , where A ij = 1 { ( i,j ) 2 E } , and node features F 2 R N ⇥ F 0 , F 0 > 0 .
Let 83 P ⇡ 2 B N be a permutation matrix associated with permutation ⇡ 2 S N (where S N is the symmetric 84 group), where
B N denotes the Birkhoff polytope of N ⇥ N doubly-stochastic matrices.
Doublysctochastic matrices are non-negative square matrices whose rows and columns sum to one.
Thematrix P ⇡ deﬁnes the action of permutation ⇡ on these matrices, e.g., ⇡  A = P ⇡ AP T ⇡ .
We denotea pair of nodes i, j 2 V as isomorphic in G if exists ⇡ 2 S N such that ⇡ i = j , ⇡ j = i , A = P ⇡ AP T ⇡ ,and F = P ⇡ F .
Node features can be deﬁned by the graph signal f : V !
R F 0 as a function that maps a node to an F 0 -dimensional feature in R F 0 .
Then the signal of the graph F can be representedby a matrix F =
[ f 1 , ..., f N ] T 2 R N ⇥ F 0 , where f i 2 R F 0 are the features of node i 2 V .
Random graph model for G .
Let ( X , d, µ ) be a metric-measure space, where X is a set, d is a metric,and µ is a probability Borel measure.
A graphon is deﬁned as a mapping W : X ⇥ X !
In what follows we deﬁne how the graph G is sampled from the graphon models.
The signal deﬁnitionfollows Maskey et al.
[39] and the edge samples follow Airoldi et al.
[3], Lawrence and Hyvärinen [31].
[Figure]
[Figure]
Deﬁnition 1 (Random graph model) .
A random graph model for
G on ( X , d, µ ) is deﬁned as a pair ( W, f ) of a graphon W : X ⇥
X !
[0 , 1] and a metric-space signal f : X !
R F 0 , where f 2 99 L 1 ( X ) is an essentially bounded measurable function with theessential supremum norm.
We obtain ( G, F ) by ﬁrst sampling Ni.i.d.
random points
X 1 , ..., X N from X with probability densityµ , as the nodes of G .
Then the edge ( i, j ) between nodes i and j issampled with probability W
( X i , X j ) , i.e., the adjacency matrix
A =
( A i,j ) i,j of G is deﬁned as A i,j = ( Z i,j < W
( X i , W j ))for i, j = 1 , ..., N , where { Z i,j } Ni,j =1 are sampled i.i.d.
from
Uniform (0 , 1) .
The graph signal F =
[ f 1 , ..., f N ] T 2 R N ⇥ F 0is deﬁned as f i = f
( X i ) .
We say ( G, F ) ⇠ ( W, f ) .
Further,we restrict our attention to graphons W such that there exists aconstant d min satisfying the graphon degree
d W ( x ) :=
R X
W ( x, y ) dµ ( y )  d min > 0 , 8 x 2 X .
In an abuse of notation we identify node i 2 V with the sampled value X i ⇠ µ, 8 i 2 { 1 , ..., N } , 111 since generally µ is such that P ( X i = X j ) =
almost everywhere for i 6 =
j (e.g., µ is uniform).
The causal DAG of the data generation process of G is given in Figure 1.
Furthermore, we note thatall proofs are relegated to the Appendix due to space constraints.
In what follows we use the termsnode embeddings and node representations interchangeably. 2.1 Inductive structural node representations with graph message-passing neural networks
Graph message-passing Neural Network (gMPNN • ) is deﬁned by realizing a message-passing Neural Network (MPNN) on a graph.
Deﬁnition 2 (MPNN) .
Let T 2 N denote the number of layers.
For t = 1 , ..., T , let  ( t ) :
R 2 F t  1 !
R H t  1 and ( t ) :
R F t  1 + H t  1 !
R F t be functions, where F t 2 N is called the feature dimension oflayer t .
The corresponding MPNN ⇥ is deﬁne by the sequence of message functions (  ( t ) )
Tt =1 andupdate functions ( ( t ) )
Tt =1 , i.e. ⇥ =
((  ( t ) )
Tt =1 , ( ( t ) )
Tt =1 ) .
The message and update functions in Deﬁnition 2 are usually given by MLPs.
We now introducethe gMPNN • with T message-passing layers.
For each node i 2 V , f • ( t ) i at layer t 2 { 1 , . . .
, T } isdeﬁned recursively using (a) its own representation at layer t  1 ( f • ( t  1) i ) and (b) an aggregatedrepresentation of its neighbors m ( t ) i (commonly) deﬁned via two distinct permutation-invariantaggregation functions as follows:
Deﬁnition 3 (gMPNN • ) .
Let ( G, F ) be a graph with graph signals as in Deﬁnition 1 and ⇥ be a
MPNN as in Deﬁnition 2.
For layer t = 1 , ..., T , deﬁne ⇥ • ( t ) A and ⇥ • ( t ) A as maps from the input graph
G and graph signals F (0) =
F 2 R N ⇥ F 0 to the features in the t -th neural layer by
[Equation]
where F ( t ) is deﬁned by either the neighbor-average aggregation procedure (a),
8 i 2 V , for ⇥ • ( t )
A
[-Nj =
 A i,j d i  ( t ) ( f • ( t  1) i , f • ( t  1) j ) , f • ( t ) i := ( t ) ( f • ( t  1) i , m ( t ) i ) ,-][Equation] where the N -normalized degree of node i is d i = 1 N
P
Nj =
A i,j , or the ( N -normalized) sumaggregation procedure (b), 8 i 2 V , for ⇥ • ( t ) A , m ( t ) i [-: = 1 N P Nj =
-]:=
[Equation]
N
P
Nj
[Equation]
i
[Equation]
j  ( t ) ( f • ( t  1) i , f • ( t  1) j ) , andf • ( t ) i := ( t ) ( f • ( t  1) i , m ( t ) i ) .
Given a gMPNN • , ⇥ • ( T ) A or ⇥ • ( T ) A , with T  1 layers as in Deﬁnition 3, their outputs are the graphsignals ⇥ • ( T ) A ( F ) 2 R N ⇥ F T for the neighbor-average aggregation and ⇥ • ( T ) A ( F ) 2 R N ⇥ F T for the( N -normalized) sum aggregation, and are henceforth denoted as node embedding outputs of thegMPNN • .
We denote ⇥ • ( T )
A ( F ) i as the node embedding for node
i 2 V .
2.2 Node embeddings with continuous message passing neural networks
Here we deﬁne continuous message passing neural networks for structural node embeddings.
Deﬁnition 4 (Continuous message-passing) .
Given a MPNN ⇥ as in Deﬁnition 2, the node continuousmessage passing neural network (cMPNN • ) on graphons and metric-space signals f : X !
R F 0can be deﬁned by replacing the graph node features and the aggregation scheme in Deﬁnition 3 bythe following continuous counterparts.
Using a message signal U : X ⇥ X !
R H , the continuousmean aggregation of U is deﬁned by M • W ( U )( x ) =
R X
W ( x,y )
d W ( x ) U ( x, y ) dµ ( y ) , while the integralaggregation is deﬁned as M • W ( U )( x ) =
R X
W ( x, y ) U ( x, y ) dµ ( y ) , where W is a graphon andd W ( x ) =
R X
W ( x, y ) dµ ( y ) is the graphon degree as in Deﬁnition 1.
By replacing the discrete aggregations in Deﬁnition 3 by the above continuous aggregations, the same
MPNN ⇥ can process metric-space signals instead of graph signals in the following deﬁnition.
Deﬁnition 5 (cMPNN • ) .
Let ( W, f ) be a random graph model as in Deﬁnition 1 and ⇥ be a MPNN as in Deﬁnition 2.
For t = 1 , ..., T , deﬁne ⇥ • ( t ) W and ⇥ • ( t ) W as maps from input metric-space signal f • (0) =
f
: X !
R F 0 to the features in the t -th layer by
[Equation]
where f • ( t ) are deﬁned sequentially either through the degree-average aggregation (a) for ⇥ • ( t ) W :g • ( t ) ( x ) :
= M • W (  ( t ) ( f • ( t  1) , f • ( t  1) ))
( x [Equation] =
[Equation]
f • ( t ) ( x ) : =
( t ) ( f • ( t  1) ( x ) , g • ( t ) ( x )) , or through the integral aggregation (b) for ⇥ • ( t ) W : g • ( t ) ( x ) :
= M • W (  ( t ) ( f • ( t  1) , f • ( t  1) ))
( x ) =
R X
W ( x, y )  ( t ) ( f • ( t  1) ( x ) , f • ( t  1) ( y )) dµ ( y ) , with f • ( t ) ( x ) :
= ( t ) ( f • ( t  1) ( x ) , g • ( t ) ( x )) .
Size-stability of node representation and its drawbacks
We now present our main theorems about convergence of gMPNN • to cMPNN • for test graphs G tesampled from the graphon random graph model (see Deﬁnition 1), and how it leads to size-stability ofgMPNN • for nodes that have the same representation under cMPNNs • .
In what follows we will focuson the neighbor-average aggregation procedure (a) of Deﬁnition 3, since this is the more difﬁcultcase to prove.
Similar results for the ( N -normalized) sum aggregation procedure (Deﬁnition 3(b))are shown in the Appendix due to space constraints.
Moreover, common deﬁnitions (e.g., Lipschitzcontinuous functions) are also relegated to the Appendix to save space.
3.1 Convergence of gMPNNs towards cMPNNs as test graph size increase
We now prove that, with high probability, the maximum inﬁnity difference between the gMPNN •and cMPNN • node representations decreases with N te , the size of G te .
Theorem 1.
Let ⇥ =
((  ( l ) )
Tl =1 , ( ( l ) )
Tl =1 ) be a MPNN as in Deﬁnition 2 with T layers such that ( l ) : R 2 F l  1 !
R
H
l  1 and ( l ) :
R F l  1 + H l  1 !
R F l are Lipschitz continuous with Lipschitzconstants L ( l )  and L ( l ) .
Let gMPNN • ⇥ • ( T ) A and cMPNN • ⇥ • ( T ) W be as in Deﬁnitions 3 and 5.
For arandom graph model ( W, f ) satisfying Deﬁnition 1, consider training ( G tr , F tr ) ⇠ ( W, f ) and test( G te , F te )
⇠ ( W, f ) graphs, where N tr and N te are their respective number of nodes, N te > N tr .
Let X te 1 , ..., X te N te and A te be as in Deﬁnition 1.
Let p 2 (0 ,
P
Tl =1 2( H l +1) ) .
Then, if
[Equation]
N te
[Equation]
[Equation]
we have with probability at least 1
[Equation]
[Equation]
where the constants C 1 and C 2 are deﬁned in the Appendix and depend on { L ( l )  , L ( l ) } Tl =1 and( G tr , F tr ) , N tr .
Theorem 1 above shows that as the test graph size N te grows, the node representations from thediscrete gMPNNs • converge to the continuous cMPNNs • .
This has profound consequences when itcomes to predicting links using the node representations obtained by a gMPNN • .
Next, Corollary 1shows that for any two nodes i, j 2 V te that are indistinguishable in the degree-aggregated cMPNN •(deﬁned as ⇥ • ( T ) W ( f ) ( X te i ) = ⇥
• ( T ) W ( f )( X te j ) ), they will get increasingly similar representations inthe discrete gMPNN • as N te grows.
Corollary 1.
Let ⇥ =
((  ( l ) )
Tl =1 , ( ( l ) )
Tl =1 ) , ⇥ • ( T ) A , ⇥ • ( T ) W , p, ( W, f ) , ( G tr , F tr ) , ( G te , F te ) , N tr ,
N te , A te , and X te 1 , ..., X te N te be as in Theorem 1.
If there exists i, j 2 V te , i 6 = j, s.t. ⇥ • ( T ) W ( X i ) =⇥
• ( T ) W ( X j ) and Equation (1) is satisﬁed, then, with C 1 and C 2 as in Theorem 1, we have that withprobability at least 1
[Equation]
[Equation]
Implications of Corollary 1 on Stochastic Block Models (SBMs).
In what follows, we willdiscuss circumstances where two nodes i, j 2 V get the same cMPNN • representations (i.e., both⇥ • ( T ) W ( f )( X i ) = ⇥
• ( T ) W ( f )( X j ) and ⇥ • ( T ) W ( f )
( X i ) = ⇥
• ( T ) W ( f )( X j ) ).
In what follows we restrictour results to an important family of graphon models: Stochastic Block Models (SBMs) [55], wherewe also model node attributes.
SBMs were chosen because they can consistently model large graphsgenerated by any piecewise Lipschitz graphon model [3].
SBMs are also intuitive models, whichmakes them useful to illustrate our results.
Deﬁnition 6 (Stochastic Block Model (SBM)) .
An SBM ( W, f ) is a random graph model (Definition 1) with cluster structures in W and f .
Partition the node set into r  2 disjoint subsets
S 1 , S 2 , ..., S r ✓ V (known as blocks or communities) with an associated r ⇥ r symmetric matrix S , where the probability of an edge ( i, j ) , i 2 S A and j 2 S b is S ab , for a, b 2 { 1 , . . .
, r } .
Let X =
[0 , 1] , and µ be the uniform distribution on [0 , 1] .
By dividing X =
[0 , 1] into disjoint convex sets [ t 0 , t 1 ) , [ t 1 , t 2 ) , . . .
, [ t r  1 , t r ] , where t 0 = 0 and t r = 1 , node i belongs toblock S a if X i ⇠ Uniform (0 , 1) satisﬁes
X i 2 [ t a  1 , t a ) .
The graphon function W is deﬁned 193 as W ( X i , X j ) =
[Equation]
We take the liberty toalso deﬁne node signals in our SBM model, where for B =
[ B 1 , ..., B r ] T 2 R r ⇥ F
the metric-spacesignal f : X !
R F 0 is deﬁned as f ( x ) =
[Equation]
We deﬁne the action of permutation ⇡ on B of Deﬁnition 6 as ⇡  B , where ( ⇡  B ) ⇡ a = B a .
Deﬁnition 7 (Isomorphic SBM blocks) .
For the SBM model ( W, f ) in Deﬁnition 6, we say twoblocks a, b 2 { 1 , . . .
, r } are isomorphic if the SBM satisﬁes the following two conditions: (a)t a  t a  1 = t b  t b  1 , and (b) exists ⇡ 2 S r , such that ⇡ a = b , ⇡ b = a
, S =
⇡  S , and B = ⇡  B .
A similar deﬁnition can be obtained for the general graphons in Deﬁnition 1 using the isomorphicgraphon deﬁnition of Lovász and Szegedy [36].
Now that we have the deﬁnition for isomorphicblocks in SBM models, we can prove that all nodes in these isomorphic blocks will obtain the samerepresentations under degree-average aggregation and integral aggregation cMPNNs • .
Lemma 1.
Let ⇥ =
((  ( l ) )
Tl =1 , ( ( l ) )
Tl =1 ) be a MPNN as in Deﬁnition 2, and ⇥ • ( T ) W , ⇥ • ( T ) W as in
Deﬁnition 5.
For the SBM model ( W, f ) in Deﬁnition 6 with N te nodes X 1 , . . .
, X N te .
If there existsi, j 2 V te such that X
te
i , X te j are nodes that belong to isomorphic SBM blocks (Deﬁnition 7), then⇥ • ( T ) W ( f )
( X te i ) = ⇥
• ( T ) W ( f )( X te j ) and ⇥ • ( T ) W ( f )
( X te i ) = ⇥
• ( T ) W ( f )( X te j ) .
Note that even though any two nodes in isomorphic SBM blocks get the same cMPNN • representations per Lemma 1, these nodes are likely not isomorphic in G te (as shown in Proposition 1 in
Appendix) and, hence, they get different gMPNN • representations.
However, Corollary 1 shows thatthese representations become increasingly similar as the test graph size grows ( N te  1) .
We usethis observation to understand the ability of gMPNNs • to perform link prediction tasks next.
3.2 The hardness of OOD inductive link prediction using structural node embeddings
The convergence of gMPNNs • to cMPNNs • as the test graph size N te grows (Theorem 1) impliesthrough Corollary 1 and Lemma 1 that node representations of distinct SBM blocks can becomeincreasingly similar as the test graph size grows ( N te  1) , even though these nodes are notisomorphic in G te with high probability (see Proposition 1 in the Appendix).
Deﬁnition 8 (Link prediction function from structural node embeddings) .
An inductive link predictionfunction ⌘ • : R F T ⇥ R F T !
[0 , 1] takes the gMPNN • node representations of two nodes i, j 2 V teand predicts the edge probability P ( A te ij = 1) .
We assume ⌘ • is Lipschitz continuous with Lipschitzconstant L ⌘ .
In the context of graphon random graph models (Deﬁnition 1), we aim to learn⌘ • ( ⇥ • ( T )
A te ( F te ) i , ⇥ • ( T )
A te ( F te ) j ) ⇡ W
( i, j ) .
We further assume we predict a link if ⌘ • ( · , · ) > ⌧ , whileno link if ⌘ • ( · , · ) < ⌧ , for some (arbitrary) threshold ⌧ 2 [0 , 1] chosen by the user of such system.
The next corollary showcases the difﬁculty in OOD predicting links using structural node representations as N te grows.
Corollary 2.
Let ⇥ =
((  ( l ) )
Tl =1 , ( ( l ) )
Tl =1 ) be the MPNN with T layers and ⇥ • ( T ) A , ⇥ • ( T ) W asin Theorem 1.
Let ⌘ • : R F T ⇥ R F T !
[0 , 1] be as in Deﬁnition 8.
Consider the SBM ( W, f )in Deﬁnition 6 with isomorphic blocks (Deﬁnition 7).
Let ( G tr , F tr ) ⇠ ( W, f ) and ( G te , F te )
⇠( W, f ) be the training and test graphs with N tr and N te nodes, respectively.
Consider any two testnodes i, j 2 { 1 , ..., N te } , i 6 = j , for which we can make a link prediction decision with ⌘ • (i.e.,⌘ • ( ⇥ • ( T )
A te ( F te ) i , ⇥ • ( T ) A ( F te ) j ) 6 = ⌧ ).
Let G te be large enough to satisfy both Equation (1) and
[Equation]
where p , C 1 , and C 2 are as given in Corollary 1.
Then, if i and j belong to isomorphic blocks (i.e.,⇥ • ( T ) W ( f )( X te i )
= ⇥
• ( T ) W ( f )( X te j ) ), with probability at least 1
P
Tl =1 2( H l +1) p the link predictionmethod in Deﬁnition 8 will make the same link prediction regardless of the SBM probability matrix
S (Deﬁnition 6) and whether i and j are in the same block or distinct isomorphic blocks .
Corollary 2 proves that link prediction with structural node embeddings form gMPNNs • is unreliable.
That is, for any link prediction method satisfying Deﬁnition 8, as the test graph grows N te  1 , themethod will increasingly struggle to give different predictions within and across isomorphic SBMblocks, even when these probabilities are arbitrarily different in the underlying graph model.
In whatfollows we show that pairwise embeddings can address this challenge.
Size-stability of structural pairwise embeddings and its advantages
We have discussed the limitation of gMPNNs • on node representation for link prediction.
Now weclaim that a joint continuous message passing graph neural network is capable of link predictionin graphon random graph models (Deﬁnition 1).
We deﬁne the joint continuous message passinggraph neural network inspired by the cMPNNs • for node representations (Deﬁnition 5).
First, weneed to deﬁne the graphon fraction of common neighbors for graphon nodes x and y , c W ( x, y ) :=
R X
W ( x, z ) W ( y, z ) dµ ( z ) .
We only consider graphons W such that there exists d cmin satisfyingc W ( x, y )  d cmin > 0 , 8 x, y 2 X in this section.
Since we do not have edge feature as in Deﬁnition 1,we deﬁne the metric-space pair-wise signal as f •• ( x, y )
= 1 , 8 x, y 2 X .
Deﬁnition 9 (cMPNN •• ) .
Let ( W, f ) be a random graph model as in Deﬁnition 1 and ⇥ be a MPNNas in Deﬁnition 2.
For t = 1 , ..., T , deﬁne the continuous (pairwise) cMPNN •• ⇥ •• ( t ) W as the mappingthat maps input pairwise metric-space signals
f •• (0) =
f •• to the features in the t -th layer by
[Equation]
where f •• ( t ) are deﬁned recursively by
[Equation]
M
[Equation] The intuition of the aggregation function is that two edges with one same node is considered neighborsin a higher-order graph [43], and to go from ( x, y ) to ( x, z ) , we need to transition from y to z , whichhas probability W ( y, z ) .
The same holds for going from ( x, y ) to ( y, z ) .
Lemma 2.
If  ( x, y ) =
y and ( x, y ) =
x/y , then f •• ( t ) ( x, y )
=
W ( x, y ) , 8 x, y 2 X is a stationary point in the cMPNN •• , i.e. if f •• ( t  1) ( x, y )
=
W ( x, y ) , then f •• ( [Equation] ) ( x, y ) =
W ( x, y ) , 8 x, y
X .
We deﬁne the corresponding gMPNN •• as follows.
First we deﬁne the fraction of common neighborsbetween nodes i and j as c Ai,j = 1 N
P
[Equation] z .
If two nodes do not have common neighbors,then we set c Ai,j = 1 N to avoid computation error.
Further, we deﬁne f •• i,j = 1 8 i, j 2 V for anygraph G , and F •• = ( f •• i,j ) i,j 2 V as the pair-wise graph signals.
Deﬁnition 10 (gMPNN •• ) .
Let ( G, F ) be a graph with graph signals as in Deﬁnition 1 and ⇥ be a
MPNN as in Deﬁnition 2.
For t = 1 , ..., T layers we deﬁne the gMPNN •• ⇥ •• ( t ) A as the mapping that maps input pairwise graph signals F •• (0) =
F •• to the features in the t  th layer by
[Equation]
where f •• ( t ) are deﬁned recursively by the following function,
[Equation]
Next, Theorem 2 proves a similar convergence results for cMPNN •• as Theorem 1 for cMPNN • .
Theorem 2.
Let ⇥ =
((  ( l ) )
Tl =1 , ( ( l ) )
Tl =1 ) be a MPNN as in Deﬁnition 2 with T layers suchthat  ( l ) and ( l ) are Lipschitz continuous with Lipschitz constants L ( l )  and L ( l ) .
Let gMPNN ••⇥ •• ( T ) A and cMPNN •• ⇥ •• ( T ) W be as in Deﬁnitions 9 and 10. For a random graph model ( W, f ) as in Deﬁnition 1 with d cmin , consider training ( G tr , F tr ) ⇠ ( W, f ) and test ( G te , F te )
⇠ ( W, f ) graphs,where N te > N tr .
Let X te 1 , ..., X te N te
and A te be as in Deﬁnition 1.
Let p 2 (0 ,
P
Tl =1 2( H l +1) ) .
Then,if p N te p log(2( N te ) 2 /p )  4 p 2 d cmin , we have with probability at least 1
[Equation]
[Equation]
where the constants C 3 and C 4 are deﬁned in the Appendix and depend on { L ( l )  , L ( l ) } Tl =1 and( G tr , F tr ) , N tr .
Hence, as N te gets larger, the link predictor using gMPNN •• will converge to a continuous method(cMPNN •• ) that can predict links in OOD tasks (i.e., W ( X te i , X te j ) is a stationary solution ofcMPNN •• per Lemma 2).
This convergence is observed in our empirical results.
Further Related Work
In what follows we describe works related to learning transferability in GNNs.
The concept oftransferability of GNN is introduced by Levie et al.
[32], Ruiz et al.
[50], which state that if twographs represent same phenomena (e.g., are sampled from the same distribution), then a transferable
1.5 2.0 2.5 3.0 3.5 4.0 log 10 ( N te ) 2.
l o g 10 (
A W ) (a) slope=-1/ 1.5 2.0 2.5 3.0 3.5 4.0 log 10 ( N te ) 0.
l o g 10 (
A W ) (b) slope=-1/ 1.5 2.0 2.5 3.0 3.5 4.0 log 10 ( N te ) 2.0 1.
l o g 10 ( A W ) (c) slope=-1/ GNN has approximately the same predictive performance on both graphs.
This is closely relatedto in-distribution generalization capabilities of GNNs to unseen test data, i.e., generalization errorwhen train and test data come from the same distribution.
Existing works [24, 38, 50, 51] provethe transferability for spectral-based GCNs under graphon models, and Maskey et al.
[39] extendsthese results to more general message passing GNNs.
Our results are also based on general messagepassing GNNs.
Our goal (OOD link prediction) is, however, signiﬁcantly different than these priorworks, which focus on in-distribution graph and node classiﬁcation.
Related works relating to therepresentation power, higher order structural and positional link prediction methods (not alreadycovered in our introduction) can be found in Appendix E due to space constraints.
Empirical Evaluation
In what follows we empirically validate our theoretical results in two parts.
We implement all ourmodels in Pytorch Geometric [17] and make it available 1 .
Due to space constraints we relegate adetailed description of our experiments to the Appendix.
Convergence and stability.
First we will empirically validate Theorems 1 and 2 and Corollary 1.
Consider an SBM (Deﬁnition 6) with three blocks ( r = 3 ) and S a,a = 0 . 55 , a = 1 , 2 , 3 , S 1 , 2 = S 2 , 1 = 0 . 05 , S 1 , 3 = S 3 , 1 = 0 .
The probability a node belongs to block one or three is 0 .
45 , whilefor block two it is 0 . 1 .
Note that one and three are isomorphic blocks (see Deﬁnition 7).
Since ourresults are valid for any gMPNN functions ⇥ , for our ﬁrst experiment with node embeddings we usea randomly initialized GraphSAGE [22] GNN model, where following standard GNN procedures weinitialize node features as size-normalized degrees (see d i in Deﬁnition 3).
For the experiment withpairwise embeddings, we test both the  and of Lemma 2, and a scenario where is a randomlyinitialized feedforward neural network.
In the Appendix we show how to efﬁciently compute theexact cMPNN • and cMPNN •• embeddings of our GraphSAGE and gMPNN •• models.
Figures 2(a-c) show log-log plots of the convergence of gMPNNs to their continuous cMPNNcounterparts as the test graph size N te increases.
The empirical approximation errors  • A-W (Theorem 1) (Figure 2(a)) and  •• A-W (Theorem 2) are shown as a function of the test graph size N te = 2 n ,n = 5 , ..., 13 .
The empirical results show agreement with the theory since  • A-W and
•• A-W are boundedabove by O ( p log N te / p N te ) , which is approximated by the slope  1 / 2 in a log-log plot.
Figures 2(d-e) show histograms of the difference between gMPNN • embeddings of different nodesin G te .
Let  • i,j := ⇥ • ( T )
A ( F ) i  ⇥ • ( T ) A ( F ) j for i, j 2 V te ,  • i,j 2 R F T and further deﬁne • iso (resp.
non-iso) i,j :=
(  • i,j ) argmax k | (  • i,j ) k | , where k 2 { 1 , . . .
, F T } is the dimension of the embedding.
We use subscript iso (resp.
non-iso ) when i, j 2 V te are in isomorphic (resp. non-isomorphic) SBM blocks (Deﬁnition 7).
As N te increases, Figure 2(d) shows that embeddings between isomorphicblocks converge, validating Corollary 1, while Figure 2(e) shows that non-isomorphic blocks do not.
Link prediction performance evaluation with SBMs (in-distribution and OOD).
In what followswe introduce empirical results using the SBM described earlier.
We start by sampling the traininggraph ( G tr , F tr ) with N tr = 10 3 nodes.
We randomly split E tr into positive train (80%) and validation(10%) edges (we reserve 10% of E tr for the transductive test scenario), and uniformly sample the samenumber of across-block non-edges as negative train and validation edges.
The embedding methodgMPNN • (resp. gMPNN •• ) along with link predictor ⌘ • (resp. ⌘ •• ) are trained in an end-to-end [Table] manner for predicting positive and negative edges in training using cross-entropy loss.
The modelachieving the best validation accuracy is then chosen to be applied in test.
Our experiments considerthree scenarios (in all scenarios we use the same number of negative test edges as positive test edges,sampled from non-edges in G te with endpoints in different isomorphic blocks): (i) (In-distribution)transductive scenario where G te =
G tr , where positive test edges are the 10% reserved in E tr not usedin training or validation; (ii) In-distribution inductive scenario where G te is sampled from the same
SBM with N te = N tr , where we sample 0 .
1 | E tr | positive test edges from E te (for fair comparisonacross all scenarios); (c) OOD inductive scenario where G te is sampled from the same SBM with
N te = 10 ⇥ N tr , where we sample 0 .
1 | E tr | positive test edges from E te .
For structural node embeddings we consider GraphSAGE [22], GCN [25] (without positionalfeatures), GAT [62] and GIN [67] as the representatives of gMPNN • models.
The link predictor⌘ • is as feedfoward network (with 3 hidden layers and 10 neurons each) that receives the two nodeembeddings as input, and has link prediction threshold ⌧ = 0 .
5 (see Deﬁnition 8 for details).
For structural pairwise embeddings we choose our proposed gMPNN •• method of Deﬁnition 10,since
we can prove that our approach is theoretically sound in Lemma 2.
We test gMPNN •• in twoversions: The  and functions in Lemma 2 (denoted ﬁxed ) and a feedforward neural networkfor with 2 hidden layers and 5 neurons each (denoted learn ).
The link predictor ⌘ •• is the sameas ⌘ • except it just takes one pairwise embedding as input, rather than two node embeddings.
Table 1 presents our empirical results.
The oracle predictor knows the graphon values W ( X te i , X te j ) .
Our evaluation metrics include the Matthews correlation coefﬁcient (mcc) [40], balanced accuracy,and Hits@
K for K = 10 , 50 , 100 that counts the ratio of positive edges ranked at the k -th placeor above against all negative edges.
Note that gMPNN • structural node representations can veryaccurately predict links in the transductive tasks, and still performs reasonably well in inductivein-distribution tasks.
However, as expected from Corollary 2, this performance suffers signiﬁcantly as N te becomes 10 ⇥ larger than N tr .
Now all gMPNN • methods produce predictors that are no betterthan a random guess over all metrics (e.g., see OOD mcc and accuracy (in red)).
In contrast, thegMPNN •• is able to consistently offer good performance on both in-distribution and OOD tasks.
Conclusions
This work studied and provided the ﬁrst theoretical framework for the task of out-of-distribution(OOD) link prediction, where test graphs are larger than training graphs.
Using non-asymptoticbounds, this work showed that OOD link prediction methods using structural node embeddingsgiven by message-passing GNNs converge to link predictors that may perform no better than randomguesses.
The work also proposed a theoretically-sound structural pairwise embedding with a messagepassing algorithm which converges to its continuous version that is proven to be able to perform OOD link prediction tasks.
Extensive empirical evaluation showed agreement with these theoreticalresults.
We do not foresee adverse social impacts for this theoretical work.