E QUIVARIANT AND S TABLE P OSITIONAL E NCODING FOR M ORE P OWERFUL G RAPH N EURAL N ETWORKS
A BSTRACT
Graph neural networks (GNN) have shown great advantages in many graph-based learning tasks but often fail to predict accurately for a task based on sets of nodes such as link/motif prediction and so on.
Many works have recently proposed to address this problem by using random node features or node distance features.
However, they suffer from either slow convergence, inaccurate prediction or high complexity.
In this work, we revisit GNNs that allow using positional features of nodes given by positional encoding (PE) techniques such as Laplacian Eigenmap,
Deepwalk, etc.. GNNs with PE often get criticized because they are not generalizable to unseen graphs (inductive) or stable.
Here, we study these issues in a principled way and propose a provable solution, a class of GNN layers termed PEG with rigorous mathematical analysis.
PEG uses separate channels to update the original node features and positional features.
PEG imposes permutation equivariance w.r.t.
the original node features and rotation equivariance w.r.t.
the positional features simultaneously.
Extensive link prediction experiments over 8 real-world networks demonstrate the advantages of PEG in generalization and scalability.
I NTRODUCTION
Graph neural networks (GNN), inheriting from the power of neural networks (Hornik et al., 1989), have recently become the de facto standard for machine learning with graph-structured data (Scarselli et al., 2008).
While GNNs can easily outperform traditional algorithms for single-node tasks (such as node classiﬁcation) and whole graph tasks (such as graph classiﬁcation), GNNs predicting over a set of nodes often achieve subpar performance.
For example, for link prediction, GNN models, such as GCN (Kipf & Welling, 2017), GAE (Kipf & Welling, 2016) may perform even worse than some simple heuristics such as common neighbors and Adamic Adar (Liben-Nowell & Kleinberg, 2007)
(see the performance comparison over the networks Collab and PPA in Open Graph Benchmark (OGB) (Hu et al., 2020)).
Similar issues widely appear in node-set-based tasks such as network motif prediction (Liu et al., 2022; Besta et al., 2021), motif counting (Chen et al., 2020), relation prediction (Wang et al., 2021a;
Teru et al., 2020) and temporal interaction prediction (Wang et al., 2021b), which posts a big concern for applying GNNs to these relevant real-world applications.
The above failure is essentially induced by the loss of node identities during the intrinsic computation of GNNs.
Nodes that get matched under graph automorphism will be associated with the same representation by GNNs and thus are indistinguishable (see Fig. 1(i)).
A naive way to solve this problem is to pair GNNs with one-hot encoding as the extra node feature.
However, it violates the fundamental inductive bias, i.e., permutation equivariance which GNNs are designed for, and thus may lead to poor generalization capability: The obtained GNNs are not transferable (inductive) across different node sets and different graphs or stable to network perturbation.
Many works have been recently proposed to address such an issue.
The key idea is to use augmented node features, where either random features (RF) or deterministic distance encoding (DE) can be adopted.
Interested readers may refer to the book chapter (Li & Leskovec, 2021) for detailed discussion.
Here we give a brief review.
RF by nature distinguishes nodes and guarantees permutation equivariance if the distribution to generate RF keep invariant across the nodes.
Although GNNs paired with RF have been proved to be more expressive (Murphy et al., 2019;
Sato et al., 2021), the training procedure is often hard to converge and the prediction is noisy and inaccurate due to the
[Figure]
Encoding original features ? ?
a b u v d c ⊕ a b u v d c ⊕ ⊕ ⊕ ⊕ ⊕ (i) (ii) b (−0.39,0.5) a (−0.39,−0.5) u (−0.45,0.0) c (0.39,0.5)
d (0.39,−0.5) v (0.45,0.0) 𝑢 (cid:2870)
𝑢 (cid:2871)
Rotation equivariance (iii) injected randomness (Abboud et al., 2020).
On the other hand, DE deﬁnes extra features by using the distance from a node to the node set where the prediction is to be made (Li et al., 2020).
This technique is theoretically sound and empirically performs well (Zhang & Chen, 2018; Li et al., 2020;
Zhang et al., 2020).
But it introduces huge memory and time consumption.
This is because DE is speciﬁc to each node set sample and no intermediate computational results, e.g., node representations in the canonical GNN pipeline can be shared across different samples.
To alleviate the computational cost of DE, absolute positions of nodes in the graph may be used as the extra features.
We call this technique as positional encoding (PE).
PE may approximate
DE by measuring the distance between positional features and can be shared across different node set samples.
However, the fundamental challenge is how to guarantee the GNNs trained with PE keep permutation equivariant and stable.
Using the idea of RF, previous works randomize PE to guarantee permutation equivariance.
Speciﬁcally, You et al.
(2019) designs PE as the distances from a node to some randomly selected anchor nodes.
However, the approach suffers from slow convergence and achieves merely subpar performance.
Srinivasan & Ribeiro (2020) states that
PE using the eigenvectors of the randomly permuted graph Laplacian matrix keeps permutation equivariant.
Dwivedi & Bresson (2020); Kreuzer et al.
(2021) argue that such eigenvectors are unique up to their signs and thus propose PE that randomly perturbs the signs of those eigenvectors.
Unfortunately, these methods are problematic.
They fail to provide permutation equivariant GNNs when the matrix has multiple eigenvalues, which thus are dangerous when applying to many practical networks.
For example, large social networks, when not connected, have multiple 0 eigenvalues; small molecule networks often have non-trivial automorphism that may give multiple eigenvalues.
Even if the eigenvalues are distinct, these methods are unstable.
We prove that the sensitivity of node representations to the graph perturbation depends on the inverse of the smallest gap between two consecutive eigenvalues, which could be actually large when two eigenvalues are close (Lemma 3.4).
In this work, we propose a principled way of using PE to build more powerful GNNs.
The key idea is to use separate channels to update the original node features and positional features.
The GNN architecture keeps not only permutation equivariant w.r.t.
node features but also rotation equivariant w.r.t.
positional features.
This idea applies to a broad range of PE techniques that can be formulated as matrix factorization (Qiu et al., 2018) such as Laplacian Eigenmap (LE) (Belkin & Niyogi, 2003) and Deepwalk (Perozzi et al., 2014).
We design a GNN layer PEG that satisﬁes such requirements.
PEG is provably stable:
In particular, we prove that the sensitivity of node representations learnt by
PEG only depends on the gap between the p th and ( p + 1) th eigenvalues of the graph Laplacian if p -dim LE is adopted as PE, instead of the smallest gap between any two consecutive eigenvalues that previous works have achieved.
PEG gets evaluated in the most important node-set-based task, link prediction, over 8 real-world networks.
PEG achieves comparable performance with strong baselines based on DE while having much lower training and inference complexity.
PEG achieves signiﬁcantly better performance than other baselines without using DE.
Such performance gaps get enlarged when we conduct domain-shift link prediction, where the networks used for training and testing are from different domains, which effectively demonstrates the strong generalization and transferability of PEG.
1. O THER RELATED WORKS
As long as GNNs can be explained by a node-feature-reﬁnement procedure (Hamilton et al., 2017;
Gilmer et al., 2017; Morris et al., 2019; Veliˇckovi´c et al., 2018; Klicpera et al., 2019; Chien et al., 2021), they suffer from the aforementioned node ambiguity issue.
Some GNNs cannot be explained as node-feature reﬁnement as they directly track node-set representations (Maron et al., 2019b; Morris et al., 2019; Chen et al., 2019; Maron et al., 2019a).
However, their complexity is high, as they need to compute the representation of each node set of certain size.
Moreover, they were only studied for graph-level tasks.
EGNN (Satorras et al., 2021) seems to be a relevant work as it studies when the nodes have physical coordinates given in prior.
However, no analysis of PE has been provided.
A few works studied the stability of GNNs by using tools like graph scattering transform (for graph-level representation)
(Gama et al., 2019a;b) and graph signal ﬁltering (for node-level representations) (Levie et al., 2021;
Ruiz et al., 2020; 2021; Gama et al., 2020; Nilsson & Bresson, 2020).
They all focus on justifying the stability of the canonical GNN pipeline, graph convolution layers in particular.
None of them consider positional features let alone the stability of GNNs using PE.
N OTATIONS AND P RELIMINARIES
In this section, we prepare the notations and preliminaries that are useful later.
First, we deﬁne graphs.
Deﬁnition 2.1 (Graph) .
Unless speciﬁed, we always consider undirected graphs of N nodes and let [ N ] = { 1 , 2 , ..., N } .
One such graph can be denoted as G =
( A, X ) , where A is the adjacency matrix.
X ∈ R N × F denotes the node features, where the v th row, X v , is the feature vector of node v .
A graph may have self loops, i.e., A has nonzero diagonals.
Denote D as the diagonal degree matrix where D vv = (cid:80) u ∈
[ N ] A vu for v ∈
[ N ] .
Let d max = max v ∈
N ] D vv .
Denote the normalized adjacency matrix as ˆ A
= D − 12 AD − 12 and the normalized Laplacian matrix as L =
I − ˆ A .
Deﬁnition 2.2 (Permutation) .
An N -dim permutation matrix P is a matrix in { 0 , 1 } N × N where each row and each column has only one single 1.
All such matrices are collected in Π( N ) , simpliﬁed as Π .
We denote the vector (cid:96) 2 -norm as (cid:107) · (cid:107) , the Frobenius norm as (cid:107) · (cid:107) F and the operator norm as (cid:107) · (cid:107) op .
Deﬁnition 2.3 (Graph-matching) .
Given two graphs G ( i ) =
( A ( i ) , X ( i ) ) and their normalized
Laplacian matrices L ( i ) for i ∈ { 1 , 2 } , their matching can be denoted by a permutation matrix P ∈ Π that best aligns the graph structures and the node features.
[Equation]
Using L instead of A to represent graph structures is for notational simplicity.
Actually for an unweighted graph, there is a bijective mapping between L and A .
One can rewrite the ﬁrst term with
A .
Later, we use P ∗ by not specifying G 1 and G 2 if there is no confusion.
The distance between the two graphs can be deﬁned as d ( G 1 , G 2 ) =
(cid:107) L (1) − P ∗ L (2) P ∗ T (cid:107) F + (cid:107) X
(1) − P ∗ X (2) (cid:107) F .
Next, we review eigenvalue decomposition and summarize arguments on its uniqueness in Lemma 2.6.
Deﬁnition 2.4 (Eigenvalue Decomposition (EVD)) .
For a positive semideﬁnite (PSD) matrix B ∈ R N × N , it has eigenvalue decomposition B = U Λ U T where Λ is a real diagonal matrix with the eigenvalues of B , 0 ≤ λ 1 ≤ λ 2 ≤ ... ≤ λ N as its diagonal components.
U =
[ u 1 , u 2 , ..., u N ] is an orthogonal matrix
where u i ∈ R N is the i th eigenvector, i.e. Bu i
= λ
i u i .
Deﬁnition 2.5 (Orthogonal Group in the Euclidean space) .
SO ( k ) =
{ Q ∈ R k × k | QQ T
= Q T Q =
I } includes all k -by- k orthogonal matrices.
A subgroup of SO ( k ) includes all diagonal matrices with ± 1 as the diagonal components, SN ( k ) =
{ Q ∈ R k × k | Q ii ∈ {− 1 , 1 } , Q ij = 0 , for i (cid:54) =
j } .
Lemma 2.6.
EVD is not unique.
If all the eigenvalues are distinct, i.e., λ i (cid:54) =
λ j , U is unique up to the signs of its columns, i.e., replacing u i by − u i also gives EVD.
If there are multiple eigenvalues, say ( λ i − 1 < )
λ i
=
λ
i +
= ...
=
λ
i + k − 1 ( < λ i + k ) , then [ u i , u
i +1 , ..., u i + k − 1 ] lie in an orbit induced by the orthogonal group
SO ( k ) , i.e., replacing [ u i , u i +1 , ..., u i + k − 1 ] by [ u i ,
u i +1 , ...,
u i + k − 1 ] Q for any Q ∈ SO ( k ) while keeping eigenvalues and other eigenvectors unchanged also gives EVD.
Next, we deﬁne Positional Encoding (PE), which associates each node with a vector in a metric space where the distance between two vectors can represent the distance between the nodes in the graph.
Deﬁnition 2.7 (Positional Encoding) .
Given a graph G =
( A, X ) , PE works on A and gives
Z =
PE ( A )
∈ R N × p where each row
Z v gives the positional feature of node v .
The absolute values given by PE may not be useful while the distances between the positional features are more relevant.
So, we deﬁne PE-matching that allows rotation to best match positional features, which further deﬁnes the distance between two collections of positional features.
Deﬁnition 2.8 (PE-matching) .
Consider two groups of positional features Z (1) , Z (2) ∈ R N × p .
Their matching is given by Q ∗ ( Z (1) , Z (2) )
(cid:44) arg min Q ∈ SO ( p ) (cid:107) Z (1) − Z (2) Q (cid:107) F .
Later, Q ∗ is used if it causes no confusion.
Deﬁne the distance between them as η ( Z (1) , Z (2) )
= (cid:107) Z (1) − Z (2) Q ∗ (cid:107) F .
[Figure] [Figure]
[Figure]
E QUIVARIANT AND S TABLE P OSITIONAL E NCODING FOR GNN
In this section, we will study the equivariance and stability of a GNN layer using PE .
3. O UR G OAL : B UILDING P ERMUTATION E QUIVARIANT AND S TABLE GNN LAYERS
A GNN layer is expected to be permutation equivariant and stable.
These two properties, if satisﬁed, guarantee that the GNN layer is transferable and has better generalization performance.
Permutation equivariance implies that model predictions should be irrelevant to how one indexes the nodes, which captures the fundamental inductive bias of many graph learning problems: A permutation equivariant layer can make the same prediction over a new testing graph as that over a training graph if the two graphs match each other perfectly, i.e. the distance between them is 0.
Stability is actually an even stronger condition than permutation equivariance because it characterizes how much gap between the predictions of the two graphs is expected when they do not perfectly match each other.
Speciﬁcally, given a graph A with node features X , we consider a GNN layer ˜
g updating the node features, denoted as ˆ X = ˜
g
( A, X ) .
We deﬁne permutation equivariance and stability as follows.
Deﬁnition 3.1 (Permutation Equivariance) .
A GNN layer ˜ g is permutation equivariant, if for any
P ∈ Π and any graph G =
( A, X ) , P ˜ g
( A, X ) =
˜ g ( PAP T , PX ) .
Deﬁnition 3.2 (Stability) .
A GNN layer ˜ g is claimed to be stable, if there exists a constant C > 0 , for any two graphs G (1) =
( A (1) , X (1) ) and G (2) =
( A (2) , X (2) ) , letting P ∗ =
P ∗ ( G (1) , G (2) ) denote their matching,
˜ g satisﬁes (cid:107) ˜ g ( A (1) , X (1) )
− P ∗ ˜
g
( A (2) , X (2) )
(cid:107) F ≤ Cd ( G (1) , G (2) ) .
By setting
A (1) = PA (2) P T and X (1) =
PX (2) for some P ∈ Π , the RHS becomes zero, so a stable ˜ g makes the LHS zero too.
So, stability is a stronger condition than permutation equivariance.
Our goal is to guarantee that the GNN layer that utilizes PE is permutation equivariant and stable.
To distinguish from the GNN layer ˜
g that does not use PE, we use g to denote a GNN layer that uses PE, which takes the positional features Z as one input and may update both node features X and Z , i.e., ( ˆ X, ˆ Z ) =
g ( A, X, Z ) .
Now, we may deﬁne PE-equivariance and PE-stability for the GNN layer g .
Deﬁnition 3.3 (PE-stability & PE-equivariance) .
Consider a GNN layer g that uses PE.
When it works on any two graphs G ( i ) =
( A ( i ) , X ( i ) ) , i ∈ { 1 , 2 } and gives ( ˆ X ( i ) , ˆ Z ( i ) )
=
g ( A ( i ) , X
( i ) , Z ( i ) ) , let
P ∗ be the matching between the two graphs.
g is PE-stable, if for some constant C > 0 we have
[Equation]
Recall η ( · , · ) measures the distance between two sets of positional features as deﬁned in Def. 2.8.
Similar as above, a weaker condition of PE-stability is PE-equivariance:
If A (1) = PA (2) P T and
X (1) =
PX (2) for some P ∈ Π , we expect a perfect match between the updated node features and positional features, ˆ X (1) =
P ˆ X (2) and η ( ˆ Z (1) , P ˆ Z (2) ) = 0 .
Note that previous works also consider g that updates only node features, i.e., ˆ X
=
g
( A, X, Z ) .
In this case, PE-stability can be measured by removing the second term on ˆ Z from Eq.1.
3. PE- STABLE GNN LAYERS BASED ON L APLACIAN E IGENMAP AS P OSITIONAL E NCODING
To study the requirement of the GNN layer that achieves PE-stability, let us start from a particular
PE technique, i.e., Laplacian eigenmap (LE) (Belkin & Niyogi, 2003) to show the key insight behind.
Later, we generalize the concept to some other PE.
Let Z LE denote LE, which includes the eigenvectors that correspond to the p smallest eigenvalues of the normalized Laplacian matrix L .
Previous works failed to design PE-equivariant or PE-stable GNN layers.
Srinivasan & Ribeiro (2020) claims that if a GNN layer ˜ g ( A, X ) that does not rely on PE is permutation equivariant,
PE-equivariance may be kept by adding Z LE to node features, i.e., g ( A, X, Z LE ) =
˜ g
( A, X
+
MLP ( Z LE )) , where MLP ( · ) is a multilayer perceptron that adjusts the dimension properly.
This statement is problematic when Z LE is not unique given the graph structure A .
Speciﬁcally, though sharing graph structure A (1) =
A (2) , if different implementations lead to different LEs Z (1) LE (cid:54) =
Z (2) LE , then ˜ g
( A (1) , X (1) + MLP ( Z (1) LE ))
(cid:54) =
˜ g
( A (2) , X (2) + MLP ( Z (2) LE )) , which violates PE-equivariance.
Srinivasan & Ribeiro (2020) suggests using graph permutation augmentation to address the issue, which makes assumptions on an invariant distribution of Z LE and empirically does not work well.
Dwivedi & Bresson (2020); Kreuzer et al.
(2021) claim the uniqueness of Z LE up to the signs and suggest building a GNN layer that uses random Z LE as g ( A, X, Z LE )
=
˜ g
( A, X + MLP ( Z LE S ))
where S is uniformly at random sampled from SN ( p ) .
They expect PE-equivariance in the sense of expectation.
However, this statement is generally wrong because it depends on the condition that all eigenvalues have to be distinct as stated in Lemma 2.6.
Actually, for multiple eigenvalues, there are inﬁnitely many eigenvectors that lie in the orbit induced by the orthogonal group.
Although many real graphs have multiple eigenvalues such as disconnected graphs or graphs with some non-trivial automophism, one may argue that the methods work when the eigenvalues are all distinct.
However, the above failure may further yield PE-instability even when the eigenvalues are distinct but have small gaps due to the following lemma.
Lemma 3.4.
For any PSD matrix B ∈ R N × N without multiple eigenvalues, set positional encoding
PE ( B ) as the eigenvectors given by the smallest p eigenvalues sorted as 0 = λ 1 < λ 2 < ... <
λ p ( < λ p +1 ) of B .
For any sufﬁciently small (cid:15) > 0 , there exists a perturbation (cid:52) B , (cid:107)(cid:52) B (cid:107) F ≤ (cid:15) such that
[Equation]
Lemma 3.4 implies that small perturbation of graph structures may yield a big change of eigenvectors if there is a small eigengap.
Consider two graphs G (1) =
( A (1) , X (1) ) and G (2) =
( A (2) , X (2) ) , where X (1) =
X (2) and A (2) is A (1) with a small perturbation (cid:52) A (1) .
The perturbation is small enough so that the matching P ∗ ( G (1) , G (2) )
=
I .
However, the change of PE even after removing the effect of changing signs min S ∈ SN ( p ) (cid:107)
PE
( A (1) )
− PE ( A (2) )
S (cid:107) could be dominated by the largest inverse eigengap among the ﬁrst p + 1 eigenvalues max 1 ≤ i ≤ p | λ i +1 − λ i | − 1 .
In practice, it is hard to guarantee all these p eigenpairs have large gaps, especially when a large p is used to locate each node more accurately.
Plugging this PE into a GNN layer gives the updated node features ˆ X ( i ) a large change (cid:107) ˆ X (1) − ˆ X (2) (cid:107) F and thus violates PE-stability.
PE-stable GNN layers.
Although a particular eigenvector may be not stable, the eigenspace, i.e., the space spanned by the columns of PE ( A ) could be much more stable.
This motivates our following design of the PE-stable GNN layers .
Formally, we use the following lemma that can characterize the distance between the eigenspaces spanned by LEs of two graph Laplacians.
The error is controlled by the inverse eigengap between the p th and ( p + 1) th eigenvalues min i
=1 , 2 {| λ ( i ) p − λ ( i ) p +1 | − 1 } , which by properly setting p is typically much smaller than max 1 ≤ k ≤ p | λ k − λ k +1 | − 1 in Lemma 3.4.
We compute the ratio between these two values over some real-world graphs as shown in Fig.
Lemma 3.5.
For two PSD matrices B (1) , B (2) ∈ R N × N , set PE ( B ) as the eigenvectors given by the p smallest eigenvalues of B .
Suppose B ( i ) has eigenvalues 0 = λ ( i ) 1 ≤ λ ( i ) 2 ≤ ... ≤ λ ( i ) p ≤ λ ( i ) p + and δ = min i =1 , 2 {| λ ( i ) p − λ
( i ) p +
| − 1 } < ∞ .
Then, for any permutation matrix P ∈ Π ,
[Equation]
Inspired by the stability of the eigenspace, the idea to achieve PE-stability is to make the GNN layer invariant to the selection of bases of the eigenspace for the positional features.
So, our proposed PEstable GNN layer g that uses PE should satisfy two necessary conditions: 1) Permutation equivariance w.r.t.
all features; 2) Rotation equivariance w.r.t.
positional features, i.e.,
PE-stable layer cond.
[Equation]
PE-stable layer cond.
[Equation]
Rotation equivariance reﬂects the eigenspace instead of a particular selection of eigenvectors and thus achieves much better stability.
Interestingly, these requirements can be satisﬁed by EGNN recently proposed (Satorras et al., 2021) as one can view the physical coordinates of objects considered by
EGNN as the positional features.
EGNN gets brieﬂy reviewed in Appendix F. Thm.
3.6 proves
PE-equivariance under the conditions Eqs.
4 and 5.
Theorem 3.6.
A GNN layer g ( A, X, Z ) that uses Z = Z LE and satisﬁes Eqs.
4,5 is PE-equivariant if the p th and ( p + 1)
th eigenvalues of the normalized Laplacian matrix L are different, i.e., λ p (cid:54)
= λ p +1 .
Note that satisfying Eqs.
4,5 is insufﬁcient to guarantee PE-stability that depends on the form of g .
We implement g in our model PEGN with further simpliﬁcation which has already achieved good empirical performance:
Use a GCN layer with edge weights according to the distance between the end nodes of the edge and keep the positional features unchanged.
This gives the PEG layer ,
PEG
[Equation]
(cid:105) , Z (cid:17) , where Ξ uv = φ ( (cid:107) Z u − Z
[Equation]
(cid:107) ) , ∀ u, v ∈ [ N ] .
Here ψ is an element-wise activation function, φ is an MLP mapping from R → R and (cid:12) is the
Hadamard product.
Note that if ˆ A is sparse, only Ξ uv for an edge uv needs to be computed.
We may also prove that the PEG layer g PEG satisﬁes the even stronger condition PE-stability.
Theorem 3.7.
Consider two graphs G ( i ) =
( A ( i ) , X ( i ) ) , i = 1 , 2 .
Denote their normalized Laplacian matrices’ p th eigenvalue as λ ( i ) p and ( p +1)
th eigenvalue as λ ( i ) p +1 .
Assume that δ = min i
=1 , 2 ( λ ( i ) p
+1 − λ
( i ) p ) − 1 < ∞ .
Also, assume that ψ and φ in the PEG layer in Eq.
6 are (cid:96) ψ , (cid:96)
φ -Lipschitz continuous respectively.
Then, the PEG layer that uses Z = Z LE satisﬁes PE-stability with the constant C in
Eq.
1 as C =
[(7 δ (cid:107) X (1) (cid:107) op + 2 d (2)max ) (cid:96) ψ (cid:96) φ (cid:107) W (cid:107) op + 3 δ ] .
Note that to achieve PE-stability, we need to normalize the node initial features to keep (cid:107) X (1) (cid:107) op bounded, and control (cid:107) W (cid:107) op and (cid:96) φ .
In practice (cid:96) ψ ≤ 1 is typically satisﬁed, e.g. setting ψ as
ReLU.
Here, the most important term is δ .
PE-stability may only be achieved when there is an eigengap between λ p and λ p +1 and the larger eigengap, the more stable.
This observation may be also instructive to select the p in practice.
As previous works may encounter a smaller eigengap (Lemma 3.4), their models will be generally more unstable.
Also, the simpliﬁed form of g PEG is not necessary to achieve PE-stability.
However, as it has already given consistently better empirical performance than baselines, we choose to keep using g PEG .
3. G ENERALIZED TO O THER PE TECHNIQUES : D EEP W ALK AND LINE
It is well known that LE, as to compute the smallest eigenvalues, can be written as a low-rank matrix optimization form Z LE ∈ arg min Z ∈ R N × p tr ( L N ZZ T ) , s.t.
Z T Z
= I .
Other PE techniques, such as
Deepwalk (Perozzi et al., 2014), Node2vec (Grover & Leskovec, 2016) and LINE (Tang et al., 2015) can be uniﬁed into a similar optimization form, where the positional features Z are given by matrix factorization M ∗ =
Z (cid:48)
Z T ( M ∗ may be asymmetric so Z (cid:
) ∈ R N × p may not be Z ) and M ∗ satisﬁes
M
[Equation]
=
[Equation]
min
[Equation]
Here f + ( · ) : R N × N → R N × N ≥typically revises A by combining degree normalization and a power series of A , f − ( · ) : R N × N → R N × N ≥corresponds to edge negative sampling that may be related to node degrees, and g ( · ) is a component-wise log-sigmoid function x − log(1+exp( x )) .
E.g., in LINE, f + ( A ) =
A , f − ( D ) =
c 11 T D 34 , for some positive constant c , where 1 is the all-one vector.
More discussion on Eq.7 and the forms of f + and f − for other PE techniques are given in Appendix G.
According to the above optimization formulation of PE, all the PE techniques generate positional features Z based on matrix factorization, thus are not unique.
Z always lies in the orbit induced by
SO ( p ) , i.e., if Z solves Eq.7, ZQ for Q ∈ SO
( p ) solves it too.
A GNN layer g still needs to satisfy
Eqs.
4,5 to guarantee PE-equivariance.
PE-stability actually asks even more.
Theorem 3.8.
Consider a general PE ( A ) technique that has an optimization objective as Eq.7, computes its optimal solution M ∗ and decomposes M ∗ =
Z (cid:48)
Z T , s.t.
Z T Z = I ∈
R p × p to get positional features Z .
Essentially, Z consists of the right-singular vectors of M ∗ .
If Eq.7 has a unique solution M ∗ , rank ( M ∗ ) = p and f + , f − therein satisfy f + ( PAP T ) =
Pf + ( A ) P T and f − ( PDP T ) =
Pf − ( D ) P T , then a GNN layer g ( A, X, Z ) that satisﬁes Eqs.
4,5 is PE-equivariant.
Note that the conditions on f + and f − are generally satisﬁed by Deepwalk, Node2vec and LINE.
However, solving Eq.7 to get the optimal solution may not be guaranteed in practice because the low-rank constraint is non-convex.
One may consider relaxing the low-rank constraint into the nuclear norm (cid:107) M (cid:107) ∗ ≤ τ for some threshold τ (Recht et al., 2010), which reduces the optimization Eq.7 into a convex optimization and thus satisﬁes the conditions in Thm. 3.8.
Empirically, this step and the step of computing SVD seem unnecessary according to our experiments.
The PE-stability is related to the value of the smallest non-zero singular value of M ∗ .
We leave the full characterization of
PE-equivariance and PE-stability for the general PE techniques for the future study.
E XPERIMENTS
In this work, we use the most important node-set-based task link prediction to evaluate PEG, though it may apply to more general tasks.
Two types of link prediction tasks are investigated: traditional link prediction (Task 1) and domain-shift link prediction (Task 2).
In Task 1, the model gets trained, validated and tested over the same graphs while using different link sets.
In Task 2, the graph used for training/validation is different from the one used for testing.
Both tasks may reﬂect the effectiveness of a model while Task 2 may better demonstrate the model’s generalization capability that strongly depends on permutation equivariance and stability.
All the results are based on 10 times random tests.
4. T HE E XPERIMENTAL P IPELINE OF PEG FOR L INK P REDICTION
We use PEG to build GNNs.
The pipeline contains three steps.
First, we adopt certain PE techniques to compute positional features Z .
Second, we stack PEG layers according to Eq. 6.
Suppose the ﬁnal node representations are denoted by ( ˆ X, Z ) .
Third, for link prediction over ( u, v ) , we concatenate ( ˆ X u ˆ X Tv , Z u Z Tv ) denoted as H uv and adopt MLP ( H uv ) to make ﬁnal predictions.
In the experiments, we test LE and Deepwalk as PE, and name the models PEG-LE and PEG-DW respectively.
To verify the wide applicability of our theory, we also apply PEG to GraphSAGE Hamilton et al.
(2017) by similarly using the distance between PEs to control the neighbor aggregation according to Eq.
6 and term the corresponding models PE-SAGE-LE and PE-SAGE-DW respectively.
For some graphs, especially those small ones where the union of link sets for training, validation and testing cover the entire graph, the model may overﬁt positional features and hold a large generalization gap.
This is because the links used as labels to supervise the model training are also used to generate positional features.
To avoid this issue, we consider a more elegant way to use the training links.
We adopt a 10-fold partition of the training set.
For each epoch, we periodically pick one fold of the links to supervise the model while using the rest links to compute positional features.
Note that PEs for different folds can be pre-computed by removing every fold of links, which reduces computational overhead.
In practice, the testing stage often corresponds to online service and has stricter time constraints.
Such partition is not needed so there is no computational overhead for testing.
We term the models trained in this way as PEG-LE+ and PEG-DW+ respectively.
4. T ASK 1 — T RADITIONAL LINK PREDICTION
Datasets.
We use eight real graphs, including Cora, CiteSeer and Pubmed (Sen et al., 2008), Twitch (RU), Twitch (PT) and Chameleon (Rozemberczki et al., 2021), DDI and COLLAB (Hu et al., 2021).
Over the ﬁrst 6 graphs, we utilize 85%, 5%, 10% to partition the link set that gives positive examples for training, validation and testing and pair them with the same numbers of negative examples (missing edges).
For the two OGB graphs, we adopt the dataset splits in (Hu et al., 2020).
The links for validation and test are removed during the training stage and the links for validation are also not used in the test stage.
All the models are trained till the loss converges and the models with the best validation performance is used to test.
Baselines.
We choose 6 baselines: VGAE (Kipf & Welling, 2016),
P-GNN (You et al., 2019), SEAL (Zhang & Chen, 2018), GNN Trans.
(Dwivedi & Bresson, 2020), LE (Belkin & Niyogi, 2003) and
Deepwalk (DW) (Perozzi et al., 2014).
VGAE is a variational version of GAE (Kipf & Welling, 2016) that utilizes GCN to encode both the structural information and node feature information, and then decode the node embeddings to reconstruct the graph adjacency matrix.
SEAL is particularly designed for link prediction by using enclosing subgraph representations of target node-pairs.
P-GNN randomly chooses some anchor nodes and aggregates only from these anchor nodes.
GNN Trans. adopts LE as PE and merges LE into node features and utilizes attention mechanism to aggregate information from neighbor nodes.
For VGAE, P-GNN and GNN Trans., the inner product of two node embeddings is adopted to represent links.
LE and DW are two network embedding methods, where the obtained node embeddings are already positional features and directly used to predict links.
[Table] [Table]
Implementation details.
For VGAE
[Table]
we consider six types of features: (1) node feature (N.): original feature of each node.
(2) constant feature (C.): node degree.
(3) positional feature (P.): PE extracted by Deepwalk.
(4) one-hot feature (O.): one-hot encoding of
Results are shown in Table 1 and Table 2.
Over the small datasets in Table 1, VGAE with node features outperforms other features in Cora, Citeseer and Pubmed because the nodes features therein are mostly informative, while this is not true over the other three datasets.
One-hot features and positional features almost achieve the same performance, which implies that that GNNs naively using
PE makes positional features behave like one-hot features and may have instability issues.
Constant features are not good because of the node ambiguity issues.
Random features may introduce heavy noise that causes trouble in model convergence.
Concatenating node features and positional features gives better performance than only using positional features but is sometimes worse than only using node features, which is again due to the instability issue by using positional features.
Although PGNN and GNN Trans. utilize positional features, they achieve subpar performance.
SEAL outperforms all of the state-of-art methods, which again demonstrates that the effectiveness of distance features (Li et al., 2020; Zhang et al., 2020).
PEG signiﬁcantly outperforms all the baselines except SEAL.
PEG+ by better using training sets achieves comparable or even better performance than SEAL, which demonstrates the contributions of the stable usage of PE.
Moreover, PEG can achieve comparable performance in most cases even only paired with constant features, which beneﬁts from the more expressive power given by PE (avoids node ambiguity).
Note that PE without GNNs (LE or DW
solo) does not perform well, which justiﬁes the beneﬁt by joining GNN with PE
[Table]
Regarding the OGB datasets
[Table]
PGNN and GNN Trans.
do not perform well either.
Besides
[Table]
PGNN cannot scale to large graphs such as collab .
The results of DW and LE demonstrate that the original positional features may only provide crude information
[Table]
so pairing them with GNNs is helpful.
PEG achieves the best results on ddi
[Table]
and performs competitively with SEAL on collab .
The complexity of PEG is comparable to canonical GNNs.
Note that we do not count the time of PE as it relates to the particular implementation.
If PEG is used for online serving when the time complexity is more important, PE can be computed in prior.
For a fair comparison, we also do not count the pre-processing time of SEAL, PGNN or GNN Trans..
Most importantly, PEG performs signiﬁcantly faster than SEAL on test because SEAL needs to compute distance features for every link while PE in
PEG is shared by links.
Interestingly, DW seems better than LE as a PE technique for large networks.
4. T ASK 2 —D OMAIN - SHIFT LINK PREDICTION
Datasets & Baselines.
Task 2 better evaluates the generalization capability of models.
We consider 3 groups, including citation networks (cora → citeseer and cora → pubmed) (Sen et al., 2008), userinteraction networks (Twitch (EN) → Twitch (RU) and Twitch (ES) → Twitch (PT))
(Rozemberczki et al., 2021) and biological networks (PPI) (Hamilton et al., 2017).
For citation networks and userinteraction networks, we utilize 95%, 5% dataset splitting for training and validation over the training graph, and we use 10% existing links as test positive links in the test graph.
For PPI dataset, we randomly select 3 graphs as training, validation and testing datasets and we sample 10% existing links in the validation/testing graphs as validation/test positive links.
Baselines & Implementation details.
We use the same baselines as Task 1, while we do not use one-hot features for VGAE, since the training and test graphs have different sizes.
As for node features, we randomly project them into the same dimension and then perform row normalization on them.
Other settings are the same as Task 1.
PE is applied to training and testing graphs separately while PE over testing graphs is computed after removing the testing links.
Results are shown in Table 3.
Compared with Table 1, for VGAE, we notice that node features perform much worse (except PPI) than Task 1, which demonstrates the risks of using node features when the domain shifts.
Positional features, which is not speciﬁed for the same graph, is possibly more generalizable over different graphs.
Random features are generalizable while still hard to converge.
PGNN and GNN Trans.
do not utilize positional features appropriately and perform far from ideal.
Both SEAL and PEG outperform other baselines signiﬁcantly, which implies their good stability and generalization.
PEG and SEAL again achieve comparable performance while PEG has much lower training and testing complexity.
Our results successfully demonstrate the signiﬁcance of using permutation equivariant and stable PE.
C ONCLUSION
In this work, we studied how GNNs should work with PE in principle, and proposed the conditions that keep GNNs permutation equivariant and stable when PE is used to avoid the node ambiguity issue.
We follow those conditions and propose the PEG layer.
Extensive experiments on link prediction demonstrate the effectiveness of PEG.
In the future, we plan to generalize the theory to more general
PE techniques and test PEG over other graph learning tasks.
A CKNOWLEDGMENTS
We greatly thank the actionable suggestions given by reviewers.
H. Yin and P. L. are supported by the 2021 JPMorgan Faculty Award and the National Science Foundation (NSF) award HDR-2117997.
A P ROOF OF L EMMA 3.
Recall the eigenvalues of the PSD matrix B are 0 = λ
1 < λ 2 < ... <
λ p < λ p +1 ≤ λ p +2 ≤ ... ≤ λ N .
Suppose one EVD of B = U Λ U T .
In U =
[ u 1 , u 2 , ..., u N ] , u i is the eigenvector of λ
i so
Bu i = λ i u i .
Without loss of generality, we set PE ( B ) =
[ u 1 , u 2 , ..., u p ] .
Suppose k = arg min 1 ≤ i ≤ p | λ i +
− λ i | .
Now, we perturb B by slightly perturbing u k and u k +1 .
We set
[Equation]
Set u (cid:48)
i =
u i for i ∈
[ N ] , i (cid:54) =
k, k + 1 .
Note that (cid:107) u (cid:48)
k
(cid:107) =
(cid:107) u (cid:48)
k +1 (cid:107) =
1 and u (cid:48)
Tk u (cid:48)
k +1 = 0 .
Then, the columns of U (cid:48) =
[ u (cid:48) 1 , u (cid:48) 2 , ..., u (cid:48) N ] still give a group of orthogonal bases.
Now we denote the above perturbation of B as B + (cid:52) B
= (cid:80) Ni =
1 λ i u (cid:48)
i u (cid:48)
Ti .
Then, PE ( B + (cid:52) B ) could be [ u (cid:48) 1 , u (cid:48) 2 , ..., u (cid:48) p ] S (cid:48) for any S (cid:48) ∈ SN ( p ) .
Therefore, for sufﬁciently small (cid:15) > 0 , min S (cid:48) ∈ SN ( p ) (cid:107) PE ( B + (cid:52) B ) S (cid:48)
− PE ( B ) (cid:107) 2 F = (cid:107) [(
[Equation]
(cid:15)
[Equation] [Equation]
[Equation] [Equation]
[Equation]
k
[Equation]
2 F = (cid:107) (
[Equation] [Equation]
(cid:15)
[Equation]
1) u k +
[Equation]
k +1 (cid:107)
[Equation]
(cid:107) ( (cid:112) 1 − (cid:15)
[Equation]
[Equation]
k +1 − (cid:15)u k (cid:107)
[Equation]
=4(1 − (cid:112)
[Equation]
(cid:15) 2 )
=2 (cid:15)
+ o ( (cid:15) 2 ) .
Next, we characterize (cid:107)(cid:52) B (cid:107) F .
(cid:107)(cid:52) B (cid:107) 2 F = (cid:107) B
+ (cid:52) B − B (cid:107) 2 F = (cid:107)
N (cid:88) i
= λ
i u (cid:48) i u (cid:48)
Ti −
N (cid:88) i
= λ
i u i u Ti (cid:107) 2 F = (cid:107)
λ k ( u (cid:48)
k u (cid:48)
[Equation]
Combining Eqs.
8,9, we have, for sufﬁciently small (cid:15) > 0 , min S
[Equation]
PE
[Equation]
which concludes the proof.
B P ROOF OF L EMMA 3.
The result of the Lemma 3.5 can be derived from the Davis-Kahan theorem (Davis & Kahan, 1970) and its variant (Yu et al., 2015) that characterizes the eigenspace perturbation.
We apply the Theorem 2 Eq.3 of (Yu et al., 2015) to two PSD matrices.
Theorem B.1 (Theorem 2 (Yu et al., 2015)) .
Let two PSD matrices B (1) , B (2) ∈ R N × N , with eigenvalues
= λ ( i )≤ λ ( i )≤ ...
≤ λ ( i ) p ≤ ... ≤ λ ( N ) p such that λ (1) p +1 − λ (1) p > 0 .
For i = 1 , 2 , let U ( i ) =
( u ( i ) 1 , u ( i ) 2 , ..., u ( i ) p ) have orthonormal columns satisfying B ( i ) u ( i )
k
= λ k u ( i ) k for k = 1 , 2 , ..., p .
Then, there exists an orthogonal matrix Q ∈ SO ( p ) such that
[Equation]
By symmetry, use the above theorem again and we know there exists Q (cid:
) ∈
SO ( p )
[Equation]
Because (cid:107) U (1) Q (cid:48) − U (2) (cid:107) F = (cid:107) U (2) Q (cid:48)
T − U (1) (cid:107) F , then there exists Q ∈ SO
( p ) ,
[Equation] [Equation]
) (10) where δ = min { ( λ (1) p +1 − λ (1) p ) − 1 , ( λ (2) p +1 − λ (2) p ) − 1 } .
When we apply a permutation matrix P ∈ Π to permute the rows and columns of B (2) , then
[Equation]
=
[Equation]
for any k .
Moreover, permuting the rows and columns of a PSD matrix will not change its eigenvalues.
This means that B (2) can be replaced by
PB (2) P T in Eq.10 as long as U (2) is replaced by PU (2) .
Therefore,
[Equation]
C P ROOF OF T HEOREM 3.
To prove PE-equivariance, consider two graphs G (1) =
( A (1) , X (1) ) and G (2) =
( A (2) , X (2) ) that have perfect matching P ∗ .
So, L (1) =
P ∗ L (2) P ∗ T , X (1) =
P ∗ X (2) .
Let Z ( i ) denote the Laplacian eigenmaps of L ( i ) , i = 1 , 2 .
Set B ( i ) =
L ( i ) , i = 1 , 2 and P = P ∗ and use Lemma 3.5.
Because L (1) =
P ∗ L (2) P ∗ T and λ p (cid:54) =
λ p
+1 , then there exists Q ∈ SO
( p ) .
[Equation]
Now, we consider a GNN layer g that satisﬁes Eqs.
Also denote the output as ( ˆ X (1) , ˆ Z (1) )
=
g ( A (1) , X (1) , Z (1) ) and ( ˆ X (2) , ˆ Z (2) )
=
g ( A (2) , X (2) , Z (2) ) .
[Equation]
Here (a) is because the perfect matching between G (1) and G (2) , and Eq.
(b) is due to Eq.
4 and (c) is due to Eq.
Therefore, ˆ X (1) =
P ∗ ˆ X (2) and η ( ˆ Z (1) , ˆ Z (2) ) = 0 , which implies that g satisﬁes PE-equivariance.
D P ROOF OF T HEOREM 3.
To prove PE-stability, consider two graphs G (1) =
( A (1) , X (1) ) and G (2) =
( A (2) , X (2) ) .
Let P ∗ denote their matching.
We study the PEG layer in Eq.
6 and denote ( ˆ X ( i ) , ˆ Z ( i ) )
=
g PEG ( A ( i ) , X ( i ) , Z ( i ) ) for i = 1 , 2 .
Let us ﬁrst bound the easier term regarding the positional features η ( ˆ Z (1) , P ∗ ˆ Z (2) ) .
Because ˆ Z ( i ) =
Z ( i ) , η ( ˆ Z (1) , P ∗ ˆ Z (2) )
=
η ( Z (1) , P ∗ Z (2) ) , while the bound of the later is given by Lemma 3.5.
Set
B ( i ) =
L ( i ) , i = 1 , 2 and P = P ∗ .
Then, there exists a Q ∈ SO ( p ) ,
[Equation]
Next, we bound the harder part (cid:107) ˆ X (1) − P ∗ ˆ X (2) (cid:107) F .
First,
[Equation]
Here, without loss of generality
[Equation]
we set Ξ ( i ) uv
= φ ( (cid:107) Z ( i ) u
− Z ( i ) v (cid:107) ) for u, v ∈ [ N ] such that ˆ A (1) uv (cid:54) =
and otherwise 0.
Moreover, Ξ ( i ) uv is bounded because Ξ ( i ) uv =
[Equation]
[Table]
of the
[Table]
Lipschitz continuity of
[Table]
Compute the difference
[Table]
Here, (a) is due to the (cid:96) ψ -Lipschitz continuity of the component-wise function ψ .
(b) and (d) use that for any two matrices A, B , (cid:107) AB (cid:107) F ≤ min {(cid:107) A (cid:107) F (cid:107) B (cid:107) op , (cid:107) A (cid:107) op (cid:107) B (cid:107) F } and (d) also uses the triangle inequality of (cid:107) · (cid:107) F .
(e) uses (cid:107) PA (cid:107) op
= (cid:107) A (cid:107) op and (cid:107) PA (cid:107) F = (cid:107) A (cid:107) F for any P ∈ Π .
Next, we only need to bound (cid:107) ˆ A (1) (cid:12) Ξ (1) − P ∗ (cid:16) ˆ A (2) (cid:12) Ξ (2) (cid:17) P ∗ T (cid:107) F and (cid:107) ˆ A (2) (cid:12) Ξ (2) (cid:107) op .
(cid:107) ˆ A (1) (cid:12) Ξ (1) − P ∗ (cid:16)
[Equation]
A (2) (cid:12)
[Equation] [Equation]
) (cid:17)
[Equation]
max
[Equation]
(16) where (a) is because P ( A (cid:12) B ) P T =
( PAP T ) (cid:12) ( PBP T ) for any P ∈ Π .
(b) is because for any two matrices A, B , (cid:107) A (cid:12) B (cid:107) F
[Equation]
max u,v | A uv |(cid:107) B (cid:107) F .
(c) is because max u,v ∈ [ N ] | ˆ A (1) uv | ≤ 1 , max u,v ∈ [ N ] | Ξ (2) uv | ≤ (cid:96) φ .
[Equation] [Equation]
where (a) is because for any two matrices A, B , (cid:107) A (cid:12) B (cid:107) op ≤ (cid:107) A (cid:107) op (cid:107) B (cid:107) op .
(b) is because graphs are undirected and both ˆ A (2) and Ξ (2) are symmetric matrices.
Hence, ˆ A (2) and Ξ (2) can be diagonalized.
For diagonalizable matrices, their operator norms equal their spectral radius ρ .
(c) is because the following facts: It is known that a degree normalized adjacency matrix ˆ A (2) has eigenvalues between -1 and 1;
And, ρ (Ξ (2) ) ≤ max v ∈
[ N ] (cid:80) Nu =1 | Ξ (2) vu | ≤ 2 max v ∈
[ N ] d (2) v (cid:96) φ = 2 d (2)max (cid:96) φ .
Here, we use that Ξ (2) vu is not zero iff vu is an edge in the graph.
Lastly, we need to bound (cid:107) Ξ (1) − P ∗ Ξ (2) P ∗ T (cid:107) F in Eq.16.
Let π : [ N ] →
[ N ] denote the permutation mapping deﬁned by P ∗ , i.e., P ∗ uv = 1 when v =
π ( u ) and P ∗ uv =
otherwise.
Pick the Q ∈ SO ( p ) that matches the two groups of positional features Z (1) , Z (2) .
[Equation]
=
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where (a) is due to the (cid:96) φ -Lipschitz continuity of the component-wise function φ and (b) is because of triangle inequalities.
Plugging Eq.18 into Eq.16 and plugging Eqs.16,17 into Eq.
[Equation]
further using Eqs.
[Equation] [Equation]
we achieve
[Equation] [Equation]
Combining Eq.19 with the bound on positional features in Eq.12, we conclude the proof by
[Equation]
[Equation]
E P ROOF OF T HEOREM 3.
To prove PE-equivariance, consider two graphs G (1) =
( A (1) , X (1) ) and G (2) =
( A (2) , X (2) ) that have perfect matching P ∗ , L (1) =
P ∗ L (2) P ∗ T , X (1) =
P ∗ X (2) .
Let Z ( i ) denote the positional features obtained by decomposing the optimal solution M ( i ) ∗ to the optimization problem Eq.7.
Because L (1) =
P ∗ L (2) P ∗ T , we have A (1) =
P ∗ A (2) P ∗ T and
D (1) =
P ∗ D (2) P ∗ T .
Then,
[Equation]
Here (a) is because A (1) =
P ∗ A (2) P ∗ T and the assumptions on f + and f − .
(b) is because for two squared matrices A, B , tr ( AB ) =
tr ( BA ) .
(c) is because g is component-wise function.
(d) is because M ∗ (2) is the unique solution of arg min M : rank ( M ) ≤ p
tr ( f + ( A (2) )
g ( M ) + f − ( D (2) ) g ( − M )) .
Recall M (1) ∗ =
Z (cid:48) (1) Z (1) T , Z (1) T Z (1)
=
I and M (2) ∗ =
Z (cid:48) (2) Z (2) T , Z (2) T Z (2)
=
I .
Note that
Z (1) , Z (2) that satisfy such decompositions are not unique.
As rank ( M (1) ∗ ) = rank ( M (2) ∗ ) = p , so
Z (1) , Z (cid:48) (1) and P ∗ Z (2) , P ∗ Z (cid:48) (2) have full-rank columns.
Since M (1) ∗
=
P ∗ M (2) ∗ P ∗ T , Z (cid:48) (1) Z (1) T =
P ∗ Z (cid:48) (2) Z (2) T P ∗ T .
Because Z (cid:48) (1) has full-rank columns, Z (cid:48) (1) T Z (cid:48) (1) is non-singular.
Let Q = Z
(cid:48) (2) T P ∗ T Z (cid:48) (1) ( Z (cid:48) T (1) Z (cid:48) (1) ) − 1 .
Then, Z (1) =
P ∗ Z (2) Q .
Since, Z (1) T Z (1) =
Z (2) T Z (2)
=
I , we have Q T Q
= I .
Q is a squared matrix so
Q ∈ SO ( p ) .
That means Z (1) =
P ∗ Z (2) Q for some Q ∈ SO ( p ) .
Now, we consider a GNN layer g that satisﬁes Eqs.
Also denote the output as ( ˆ X (1) , ˆ Z (1) )
=
g ( A (1) , X (1) , Z (1) ) and ( ˆ X (2) , ˆ Z (2) )
=
g ( A (2) , X (2) , Z (2) ) .
( ˆ X (1) , ˆ Z (1) )
=
g ( A (
[Equation]
, X
[Equation] [Equation]
Here (a) is because the perfect matching between G (1) and G (2) , and Z (1) =
P ∗ Z (2) Q .
(b) is due to
Eq.
4 and (c) is due to Eq.
Therefore, ˆ X (1) =
P ∗ ˆ X (2) and η ( ˆ Z (1) , ˆ Z (2) ) = 0 , which implies that g satisﬁes PE-equivariance.
F R EVIEW OF THE E-GNN LAYER
Satorras et al.
(2021) studies the problem when the nodes of a graph have physical coordinates as features and proposes E-GNN to deal with this kind of graph data.
E-GNN aims to keep permutation equivariant with respect to the node order, and translation equivariant, rotation equivariant with respect to the physical coordinate features.
As E-GNN asks even more (translation equivariance) than
Eqs.
4,5, E-GNN can be adopted to leverage PE techniques to keep PE-equivariance.
The speciﬁc form of E-GNN is as follows.
Given a graph G =
( V , E ) with nodes v i ∈ V and edges e ij ∈ E .
E-GNN layer takes the node embeddings h l = { h l 0 , ..., h lM − 1 } , coordinate embeddings x l = { x l 0 , ..., x lM − 1 } and edge information
E = ( e ij ) as input and outputs h l +1 and x l +1 , respectively.
Thus, the E-GNN layer can be denoted as: h l +1 , x
l +1 =
EGCL ( h l , x l , E ) .
The layer is deﬁned as following:
[Equation]
Where a ij is the edge attributes, φ e represents edge operation, φ x represents edge embedding operation and φ h represents node operation.
G T HE OPTIMIZATION FORM FOR GENERAL PE TECHNIQUES
Given an undirected and weighted network G =
( V, E, A ) with N nodes, LINE (Tang et al., 2015) with the second order proximity (aka LINE (2nd)) aims to extract two latent representation matrices
Z, Z (cid:48) ∈ R N × p .
Let Z i , Z (cid:48)
i denote the i th row of Z and the i th row of Z (cid:48) , respectively.
The objective function of LINE (2nd) follows
[Equation]
where g ( x ) is the log sigmoid function g ( x ) = x − log(1 + exp( x )) and b is a positive constant.
The ﬁrst term corresponds to the positive examples, i.e., links in the graph while the second term is based on network negative sampling.
Also, P V is some distribution deﬁned over the node set.
LINE adopts
P V ( j ) ∝ d 34 j where d j is the degree of node j .
By ﬁlling the expectation and using matrix form to rewrite the objective, we have
[Equation]
where c = b (cid:80) Nj =1 d 3 / 4 j .
So for LINE, f
+ ( A ) =
A and f − ( D ) =
c 11 T D 34 .
It is easy to validate that for all P ∈ Π , f
+ ( PAP T ) =
Pf + ( A ) P T and f − ( PDP T ) =
Pf − ( D ) P T , which satisﬁes the condition in Theorem 3.8.
Deepwalk (Perozzi et al., 2014) can be rewritten by following the similar idea.
Deepwalk ﬁrstly performs random walks with certain length for many times starting from each node and then treat each walk as a sequence of node-id strings.
Deepwalk trains a skip-gram model on these node-id strings (Mikolov et al., 2013).
Now we consider the skip-gram model with negative sampling (SGNS).
Denote the collection of observed words and their context pairs as D .
#( w, c ) denotes the number of times that the pair ( w, c ) appears in D .
#( w ) and #( c ) indicates the number of times w and c occurred in D .
Let Z w denote the vector representation of w and Z (cid:48)
c denote the vector representation of c .
According to (Levy & Goldberg, 2014), the objective function of SGNS follows
[Equation]
Deepwalk adopts this objective by viewing each node v in V as a word and viewing any node that gets sampled simultaneously with v within T hops of the random walk as the context c .
Let Φ
= D − 1 A denote the random walk matrix of the graph.
In this case, for a node v as the word and for another node u as the context, the expected number #( v, u ) ∝ Φ
[Equation]
vu =
[Equation]
d
[Equation]
Φ k )
[Equation]
d u
[Equation]
Φ k ) uv ) .
The expected number #( v ) ∝ d v which uses the stationary distribution of random walk over a connected graph is proportional to the node degrees.
We also set the negative context sampling probability P c is proportional to the node degrees, i,e., P c ( u ) ∝ d u .
Then, in Deepwalk, the SGNS objective reduces to
[Equation]
for some positive constant c .
Similar to the derivation for LINE, we can rewrite it into the matrix form
[Equation]
[Table]
Therefore, for Deepwalk, f + ( A ) =
Φ (cid:48)
= (cid:80) Tk =
( D Φ k + Φ Tk D ) and f − ( D ) =
cD 11 T D , where Φ =
D − 1 A and c is a positive constant.
It is also easy to verify that for all P ∈ Π , f + ( PAP T ) =
Pf + ( A ) P T and f − ( PDP T ) =
Pf − ( D ) P T , which satisﬁes the conditions in Theorem 3.8.
As for Node2vec (Grover & Leskovec, 2016), it performs a 2nd-order random walk to collect node-id strings and then train an SGNS model.
The optimization form is more involved, interested readers could check (Qiu et al., 2018; Grover & Leskovec, 2016) for more details.
H S UPPLEMENT FOR EXPERIMENTS
We put more speciﬁcs of datasets and baselines adopted in Sec.
H.1 and Sec.
H.2, respectively.
We describe how we tune our model in Sec.
H.3.
We list further experimental results in Sec.
H.4.
We further conduct three supplementary experiments t demonstrate the generalization capability, stability and wide applicability of our theory and proposed PEG layer in Secs.
H.6-H.8.
H.
D ATASETS
The citation networks-
Cora , Citeseer and Pubmed are collected by Sen et al.
(2008), where nodes represent documents and edges (undirected) represent citations.
Node features are the bag-of-words representation of documents.
The Cora dataset contains 2708 nodes, 5429 edges and 1433 features per node.
The Citeseer dataset contains 3327 nodes, 4732 edges and 3703 features per node.
The
Pubmed dataset contains 19717 nodes, 44338 edges and 500 features per node.
Twitch is obtained from Rozemberczki et al.
Twitch is a user-user networks of gamers, where nodes correspond to users and edges correspond to mutual friendship between them.
Node features correspond to the games they played and liked, users’ location and streaming habits.
Twitch contains 7 user networks over different countries, including Germany (DE), England (EN), Spain (ES), France (FR), Portugal (PT) and Russia (RU).
The details of these datasets are shown in Table 4.
Chameleon is used in Chien et al.
Chameleon is a page-page network on topic ‘Chameleon’ in Wikipedia (December 2018), where nodes correspond to articles and edges represent mutual links between the articles.
Node features indicate the presence of several informative nouns in the articles and the average monthly trafﬁc (October 2017 - November 2018).
Chameleon dataset containsnodes, 36101 edges and 2325 features per node.
Protein-protein interaction (PPI) dataset contains 24 graphs corresponding to different human tissues (Zitnik & Leskovec, 2017).
We adopt the preprocessed data provided by Hamilton et al.
(2017) to construct graphs.
The average number of nodes per graph is 2372 and each node has 50 features, which correspond to positional gene sets, motif gene sets and immunological signatures.
Ogbl-ddi and ogbl-collab are chose from the open graph benchmark (OGB) (Hu et al., 2021), which adopt more realistic train/validation/test splitting, such as by time (ogbl-collab) and by by drug target in the body (ogbl-ddi).
ogbl-ddi is a drug-drug interaction network, where nodes represent drugs and edges represent interaction between drugs, where the joint effect of taking the two drugs together is considerably different from the effect that taking either of them independently.
Ogbl-collab is an author collaboration graph, where nodes represent authors and edges represent the collaboration between authors.
The 128 dimension node features of ogbl-collab is extracted by by averaging
[Figure] [Figure]
[Figure]
H. B ASELINE DETAILS
We have 4 baselines based on GNNs and 2 baselines based on network embedding techniques, namely,
LE (Belkin & Niyogi, 2003) and Deepwalk (DW) (Perozzi et al., 2014).
We will ﬁrst introduce the implementation of LE and DW, then discuss other baselines.
Both LE and DW embed the networks in R 128 in an unsupervised way.
For LE, we factorize of the graph Laplacian matrix:
∆ =
I − D − 12 AD − 12 =
U Λ U T , where A is the adjacency matrix, D is the degree matrix, and Λ , U correspond respectively to the eigenvalues and eigenvectors.
We use the 128 smallest eigenvectors as LE.
For DW, we use the code provide by OpenNE 2 .
Both methods use the inner product between pairwise node representations as the link representations.
Then, the link representations are fed to an MLP for ﬁnal predictions.
Regarding to GNN-based baselines, VGAE is implemented according to the code 3 (Kipf & Welling, 2016) given by the original paper, with 2 message passing layers with 32 hidden dimensions.
P-GNN is implemented by adopting the code 4 provided by the original paper (You et al., 2019), withmessage passing layers with 32 dimensions, with a tuned dropout ratio in {0, 0.5}.
GNN transformer layer is implemented by adopting the code 5 provided by the original paper (Dwivedi & Bresson, 2020) with 2 message passing layers with 128 hidden dimensions.
SEAL is implemented by adapting the code 6 provided by the original paper.
Notice the ogbl-ddi graph contains no node features, so the we use free-parameter node embeddings as the input node features and train them together with the
GNN parameters.
We slightly tune the hidden dimensions and layers of these baselines and present the best results.
Moreover, we aim to understand whether PEG is sensitive to the dimension of positional features.
We conduct experiments on Cora, Pubmed and Twitch(RU) to test the sensitivity of the dimension positional features.
In Fig.
4, we compare the performance of PEG-DW, PEG-LE, PEG-DW+ and
PEG-LE+ with PE in different dimensions.
All the experiment settings follow the setting to get
Table 1 except for the dimension of positional features.
H. H YPERPARAMETERS T UNING FOR PE-GNN
Table 5 lists the most important hyperparameters, which applies to our proposed model PE-GNN.
Grid research is used to ﬁnd the best hyperparameter combination.
Note that our model actually got very slightly tuned.
We believe more extensively hyperparameter tuning may yield even better results.
The models are trained until the loss converges and we report the best model by running each fortimes.
We provide our code for reproducing the experimental results to the reviewers and area chairs in the discussion forums.
[Table]
F URTHER ANALYSIS
As a supplement to the Lemma 3.4 and Lemma 3.5, we compute the ratio between the inverse eigengap between the p th and ( p + 1) th eigenvalues {| λ p −
λ
p
+
| − 1 } and max 1 ≤ k ≤ p | λ k − λ k +1 | −over more graphs as shown in Fig.
Fig.
4 shows that different PE techniques have different dimensional sensitivity.
In general, DW seems to be more stable than LE over the three datasets and PEG-DW+ achieves stable performance.
When the dimension increases, PEG are more likely to achieve higher performance, but will be more time consuming.
H. E DGE WEIGHT VISUALIZATION
In the PEG layer, we calculate an edge weight according to the distance between the representations of the end nodes of the edge.
To further understand the learnt relationship between edge weights and the distance between node representations, we visualize the edge weight transformation curves.
We run the experiment traditional link prediction (task 1) and use DW as the node positional embedding, i.e., the model PEG-DW.
For each dataset, we draw the edge weight transformation curves in Fig.
Note that each curve ranges in x-axis from the smallest distance to the largest distance observed from the corresponding graph, we also randomly select 500 distances from each graph and scatter them on the curve as shown in Fig.
Fig.
5 shows that the edge weight increases monotonically with respect to the input distance, but the relationship between them is non-linear.
Most of the weights are close to 1 but a few weights are much smaller.
H. L INK PREDICTION OVER R ANDOM GRAPHS
To further demonstrate the generalization of our model, we conduct inductive link prediction over random graph models.
Speciﬁcally, we use stochastic block models (SBM) with two blocks Holland et al.
(1983) to generate random graphs.
Each block contains 500 nodes and the probability to form a link between two nodes within each block is 0.3, while the probability to form a link between two nodes across different blocks is 0.1.
Here, we randomly select links inside the blocks as positive samples, and randomly select the missing links (unconnected node pairs) from the generated graph as negative samples.
The model is trained by using various numbers of graphs (e.g. 1, 10, 30, 50), validated and tested on 10 random graphs respectively.
For each graph that is used for training or testing, we utilize 10% positive links and pair them with the same number of negative missing links for training or testing respectively.
The rest settings are remained the same as in Task 2.
We choose three variants of GAE that use constant features (node degree), random features and positional features as baselines.
All the models are trained until they converge and the models with the best validation performance are used to report the results (averaged by 10 independent runs), which are shown in Fig.
For various sizes of input graphs, PEG achieves AUC around 0.98 by just using one training graph and slightly improves AUC by using more training graphs, which consistently outperforms all three GAE variants that even use the entire 50 training graphs with large margins.
This directly demonstrates the stability and better generalization of PEG models.
H. T HE STUDY OF PE STABILITY BY PERTURBING THE INPUT GRAPHS
To further investigate PE stability, we consider two versions of models utilizing PE – one is PEG, which is PE equivariant and stable, and the others are GAE/VGAE (N./C. + P.), which fail to satisfy the two equivariant properties and thus may not be stable.
These two versions of models are evaluated on the Cora dataset.
All the models are trained over the original graph.
Then, we perturb the input graph by randomly adding or dropping a certain percentage of links during the inference.
The rest of the settings remain the same for link prediction.
Speciﬁcally, after training the model, we inject a perturbation on top of the input graph by adding extra links (negative samples) or dropping positive links with the ratio of 10%, 20% and 30%, respectively.
We recompute the positional encodings to reﬂect the structural perturbation of the given graph and plug in the perturbed positional encodings into different models.
Other experiment settings remain the same as in Task 1.
The results are summarized and plotted in Fig.
By comparing the models with different levels of input perturbation, we can observe from Fig.
7 that
PEG achieves consistently and signiﬁcantly better performance with either node features or constant features in all four scenarios, which is far more robust than GAE/VGAE-based models that adopt
PE under the same level of perturbations.
Compared with adding edges, removing edges is more destructive for small-size graphs such as Cora, and thereby does more harm to the model performance.
Particularly, all the models using C. + P. achieve subpar performance when considerable number of edges are removed.
However, PEG can still show strong resilience against this type of perturbation (especially with node features), while GAE/VGAE generally collapses.
These results further support that the performance boost indeed beneﬁts from the equivariance and stability with PEs instead of simply using PE variants.
H. P RELIMINARY RESULTS FOR NODE CLASSIFICATION BASED ON PE
In order to evaluate PEG on a wider category of tasks, we consider node classiﬁcation on citation networks – Cora , Citeseer and Pubmed (Sen et al., 2008) and closely follow the transductive experimental setup of Yang et al.
Suppose the ﬁnal readout of PEG is denoted as ( ˆ X, Z ) .
To classify the node u , we only utilize its node representation ˆ X u to make the ﬁnal predictions.
The rest parts and the hyperparameters of the model are kept the same as the one that is good for our task 1 (traditional link prediction).
We did not speciﬁcally tune the model for node classiﬁcation.
The results of evaluation on node-level are summarized in Table 6.
We report the mean accuracy with standard deviation of classiﬁcation on the test set of our method after 10 runs.
Both PEG-LE and PEG-DW signiﬁcantly outperform GCN and provide comparable results against
GAT, which illustrates the effectiveness of the proposed PEG on node level tasks.
Recall that we almost did not tune our model to achieve such results.
Even better results may be expected by further ﬁner tuning the model on each dataset.
Moreover, GAT uses multi-head attention to aggregate information from neighbours where the attention weights are based on node features, while PEG only adopts GCN layers with some edge weights derived from positional encodings that merely depend on the network structures.
PEG and GAT essentially utilize orthogonal information source to re-weight
[Table]
the edges.
Hence, PEG should be easily further combined with the attention mechanism based on node features to get even better predictions, which this is out of the scope of this work, so we leave it for future study.