G RAPH N EURAL N ETWORKS FOR L INK P REDICTION WITH S UBGRAPH S KETCHING
Anonymous authors
Paper under double-blind review
A BSTRACT
Many Graph Neural Networks (GNNs) perform poorly compared to simple heuristics on Link Prediction (LP) tasks.
This is due to limitations in expressive power such as the inability to count triangles (the backbone of most LP heuristics) and because they can not distinguish automorphic nodes (those having identical structural roles).
Both expressiveness issues can be alleviated by learning link (rather than node) representations and incorporating structural features such as triangle counts.
Since explicit link representations are often prohibitively expensive, recent works resorted to subgraph-based methods, which have achieved state-of-the-art performance for LP, but suffer from poor efficiency due to high levels of redundancy between subgraphs.
We analyze the components of subgraph GNN (SGNN) methods for link prediction.
Based on our analysis, we propose a novel full-graph
GNN called ELPH (Efficient Link Prediction with Hashing) that passes subgraph sketches as messages to approximate the key components of SGNNs without explicit subgraph construction.
ELPH is provably more expressive than Message
Passing GNNs (MPNNs).
It outperforms existing SGNN models on many standard
LP benchmarks while being orders of magnitude faster.
However, it shares the common GNN limitation that it is only efficient when the dataset fits in GPU memory.
Accordingly, we develop a highly scalable model, called BUDDY, which uses feature precomputation to circumvent this limitation without sacrificing predictive performance.
Our experiments show that BUDDY also outperforms SGNNs on standard LP benchmarks while being highly scalable and faster than ELPH.
I NTRODUCTION
Link Prediction (LP) is an important problem in graph ML with many industrial applications.
For example, recommender systems can be formulated as LP; link prediction is also a key process in drug discovery and knowledge graph construction.
There are three main classes of LP methods: (i) heuristics (See Appendix C.1)
that estimate the distance between two nodes (e.g. personalized page rank (PPR) (Page et al., 1999) or graph distance
(Zhou et al., 2009)) or the similarity of their neighborhoods (e.g Common Neighbors (CN), Adamic-Adar (AA) (Adamic & Adar, 2003), or
Resource Allocation (RA) (Zhou et al., 2009)); (ii) unsupervised node embeddings or factorization methods, which encompass the majority of production recommendation systems (Koren et al., 2009;
Chamberlain et al., 2020); and, recently, (iii) Graph Neural Networks , in particular of the MessagePassing type (MPNNs) (Gilmer et al., 2017; Kipf & Welling, 2017; Hamilton et al., 2017).
1 GNNs excel in graph- and node-level tasks, but often fail to outperform node embeddings or heuristics on common LP benchmarks such as the Open Graph Benchmark (OGB)
(Hu et al., 2020).
There are two related reasons why MPNNs tend to be poor link predictors.
Firstly, due to the equivalence of message passing to the Weisfeiler-Leman (WL) graph isomorphism test (Xu et al., 2019; Morris et al., 2019), standard MPNNs are provably incapable of counting triangles (Chen et al., 2020) and consequently of counting Common Neighbors or computing onehop or two-hop LP heuristics such as AA or RA.
Secondly, GNN-based LP approaches combine permutation-equivariant structural node representations (obtained by message passing on the graph) and a readout function that maps from two node representations to a link probability.
However, generating link representations as a function of equivariant node representations encounters the problem that all nodes u in the same orbit induced by the graph automorphism group have equal representations.
Therefore, the link probability p ( u, v ) is the same for all u in the orbit independent of e.g. the graph distance d ( u, v ) (Figure 1).
(Srinivasan & Ribeiro, 2019).
[Figure]
As a result, a conventional GNN will assign the same probability to links (1,2) and (1,4).
This is a result of GNN’s built-in permutation equivariance, which produces equal representation for any nodes whose enclosing subgraphs (corresponding to the receptive field of the GNN) are isomorphic.
2 We refer to this phenomenon as the automorphic node problem and define automorphic nodes (denoted u ∼ = v ) to be those nodes that are indistinguishable by means of a given k -layer GNN.
On the other hand, transductive node embedding methods such as
TransE (Bordes et al., 2013) and DeepWalk (Perozzi et al., 2014), or matrix factorization (Koren et al., 2009) do not suffer from this problem as the embeddings are not permutation equivariant.
Several methods have been proposed to improve GNN expressivity for LP.
Most simply, adding unique node IDs immediately makes all structural node representations distinguishable, but at the expense of generalization (Abboud et al., 2021) and training convergence (Sato et al., 2021).
Substructure counts may act as permutation-equivariant approximately unique identifiers (Bouritsas et al., 2022), but they require a precomputation step which may be computationally intractable in the worst case.
More successfully, a family of structural features, sometimes referred to as labeling tricks , have recently been proposed that solve the automorphic node problem while still being equivariant and having good generalization (Li et al., 2020; Zhang et al., 2021;
You et al., 2021).
However, adding structural features amounts to computing structural node representations that are conditioned on an edge and so can no longer be efficiently computed in parallel.
For the purpose of tractability, state-of-the-art methods for LP restrict computation to subgraphs enclosing a link, transforming link prediction into binary subgraph classification (Zhang et al., 2021; Zhang & Chen, 2018; Yin et al., 2022).
Subgraph GNNs (SGNN) are inspired by the strong performance of LP heuristics compared to more sophisticated techniques and are motivated as an attempt to learn data-driven LP heuristics.
Despite impressive performance on benchmark datasets, SGNNs suffer from some serious limitations: (i)
Constructing the subgraphs is expensive; (ii) Subgraphs are irregular and so batching them is inefficient on GPUs (iii); Each step of inference is almost as expensive as each training step because subgraphs must be constructed for every test link.
These drawbacks preclude many applications, where scalability or efficient inference are required.
Main contributions.
(i)
We analyze the relative contributions of SGNN components and reveal which properties of the subgraphs are salient to the LP problem.
(ii)
Based on our analysis, we develop an MPNN (ELPH) that passes subgraph sketches as messages.
The sketches allow the most important qualities of the subgraphs to be summarized in the nodes.
The resulting model removes the need for explicit subgraph construction and is a full-graph MPNN with the similar complexity to
GCN.
(iii)
We prove that ELPH is strictly more expressive than MPNNs for LP and that it solves the automorphic node problem.
(iv) As full-graph GNNs suffer from scalability issues when the data exceeds GPU memory, we develop BUDDY, a highly scalable model that precomputes sketches and node features.
(v) We provide an open source Pytorch library for (sub)graph sketching that generates data sketches via message passing on the GPU.
Experimental evaluation shows that our methods compares favorably to state-of-the-art both in terms of accuracy and speed.
P RELIMINARIES
Notation.
Let G =
( V , E ) be an undirected graph comprising the set of n nodes (vertices) V and e links (edges) E .
We denote by d ( u, v ) the geodesic distance (shortest walk length) between nodes u and v .
Let S =
( V S ⊆ V , E S ⊆ E ) be a node-induced subgraph of G satisfying ( u, v ) ∈
E S iff ( u, v ) ∈
E for any u, v ∈ V S .
We denote by S kuv =
( V uv , E uv ) a k -hop subgraph enclosing the link ( u, v ) , where V uv is the union of the k -hop neighbors of u and v and E uv is the union of the links that can be reached by a k -hop walk originating at u and v (for simplicity, where possible, we omit k ).
Similarly, S ku is the k -hop subgraph enclosing node u .
The given features of nodes V uv are denoted by X uv and the derived structure features by Z uv .
The probability of a link ( u, v ) is denoted by p ( u, v ) .
When nodes u and v have isomorphic enclosing subgraphs (i.e., S u ∼ = S v ), we write u ∼ = v .
Sketches for Intersection Estimation.
We use two sketching techniques, HyperLogLog (Flajolet et al., 2007; Heule et al., 2013a) and MinHashing (Broder, 1997).
Given sets A and B , HyperLogLog efficiently estimates the cardinality of the union |A
∪ B| and MinHashing estimates the Jaccard index
J ( A , B ) = |A
∩ B| / |A ∪ B| .
We combine these approaches to estimate the intersection of node sets produced by graph traversals (Pascoe, 2013).
These techniques represent sets as sketches, where the sketches are much smaller than the sets they represent.
Each technique has a parameter p controlling the trade-off between the accuracy and computational cost.
Running times for merging two sets, adding an element to a set, and extracting estimates only depend on p , but they are constant with respect to the size of the set.
Importantly, the sketches of the union of sets are given by permutationinvariant operations (element-wise min for minhash and element-wise max for hyperloglog).
More details are provided in Appendix C.3.
Graph Neural Networks for Link Prediction.
Message-passing GNNs (MPNNs) are parametric functions of the form Y =
GNN( X ) , where X and Y are matrix representations (of size n × d and n × d ′ , where n is the number of nodes and d, d ′ are the input and output dimensions, respectively) of input and output node features.
Permutation equivariance implies that Π GNN( X ) =
GNN( ΠX ) for any n × n node permutation matrix Π .
This is achieved in GNNs by applying a local permutationinvariant aggregation function □ (typically sum, mean, or max) to the neighbor features of every node (‘message passing’), resulting in a node-wise update of the form
[Equation]
where ϕ, γ are learnable functions.
MPNNs are upperbounded in their discriminative power by the Weisfeiler-Leman isomorphism test (WL) (Weisfeiler & Leman, 1968), a procedure iteratively refining the representation of a node by hashing its star-shaped neighborhood.
As a consequence, since WL always identically represents automorphic nodes ( u ∼ = v ), any MPNN would do the same: y u
= y v .
Given the node representations Y computed by a GNN, link probabilities can then be computed as p ( u, v ) =
R ( y u , y v ) , where R is a learnable readout function with the property that
R ( y u , y v ) =
R ( y u , y w ) for any v ∼ = w .
This node automorphism problem is detrimental for LP as p ( u, v ) =
p ( u, w ) if v ∼ = w even when d ( u, v ) ≫ d ( u, w ) .
As an example, in Figure1 2 ∼ = 4 , therefore p (1 , 2) = p (1 , 4) while d (1 , 2) = 2 < d (1 , 4) = 3 .
As a result, a GNN may suggest to link totally unrelated nodes ( v may even be in a separate connected component to u and w , but still have equal probability of connecting to u (Srinivasan & Ribeiro, 2019)).
Subgraph GNNs (SGNN).
(Zhang et al., 2021; Zhang & Chen, 2018; Yin et al., 2022) convert LP into binary graph classification.
For a pair of nodes u, v and the enclosing subgraph S uv , SGNNs produce node representations Y uv and one desires
R
( Y uv )
= 1 if ( u, v ) ∈ E and zero otherwise.
In order to resolve the automorphic node problem, node features are augmented with structural features (Bouritsas et al., 2020) that improve the ability of networks to count substructures (Chen et al., 2020).
SGNNs were originally motivated by the strong performance of heuristics on benchmark datasets and attempted to learn generalized heuristics.
When the graph is large it is not tractable to learn heuristics over the full graph, but global heuristics can be well approximated from subgraphs that are augmented with structural features with an approximation error that decays exponentially with the number of hops taken to construct the subgraph (Zhang & Chen, 2018).
A NALYZING S UBGRAPH M ETHODS FOR L INK P REDICTION
SGNNs can be decomposed into the following steps: (i) subgraph extraction around every pair of nodes for which one desires to perform LP; (ii) augmentation of the subgraph nodes with structure features; (iii) feature propagation over the subgraphs using a GNN, and (iv) learning a graph-level readout function to predict the link.
Steps (ii)–(iv) rely on the existence of a set of subgraphs (i), which is either constructed on the fly or as a preprocessing step.
In the remainder of this section we discuss the inherent complexity of each of these steps and perform ablation studies with the goal of understanding the relative importance of each.
None
ZO
DE
DRNL
[Figure]
H R @ 66.
HR@100 by structure feature type
Cora Citeseer
Pubmed (a) mean sum edge
H R @ 76.
HR@100 by readout function
[Figure]
[Figure]
Structure Features Structure features address limitations in GNN expressivity stemming from the inherent inability of message passing to distinguish automorphic nodes.
In SGNNs, permutation-equivariant distances d ( u, i ) and d ( v, i ) ∀ i ∈ V uv are used.
The three most well known are Zero-One (ZO) encoding (You et al., 2021), Double Radius Node Labeling (DRNL) (Zhang & Chen, 2018) and Distance Encoding (DE) (Li et al., 2020).
To solve the automorphic node problem, all that is required is to distinguish u and v from V uv \ { u, v } , which ZO achieves with binary node labels.
DRNL has z u
=
z v = 1 and z j = f ( d ( u, j ) , d ( v, j )) > 1 , where f : N 2 → N is a bijective map.
Distance Encoding (DE) generalizes DRNL; each node is encoded with a tuple z j = ( d ( u, j ) , d ( v, j ))
(See Figure 5).
Both of these schemes therefore include a unique label for triangles / common neighbors (See Appendix C.4 for more details on labeling schemes).
The relative performance of ZO, DRNL, DE and no structure features is shown in Figure 2a.
The use of structure features greatly improves performance and DRNL and DE slightly outperform ZO, with a more pronounced outperformance for the larger Pubmed dataset.
It is important to note that structure features are conditioned on edges and so can not be easily paralellized in the same way that node features usually are in GNNs and must be calculated for each link at both training and inference time.
In particular DRNL requires two full traversals of each subgraph, which for regular graphs is
O ( deg k ) , but for complex networks becomes O ( |E| ) .
In Figure 3 we investigate the relative importance of DRNL structure features.
Feature importance is measured using the weights in a logistic regression link prediction model with only structure feature counts as inputs.
We then sum up feature importances corresponding to different max distance r and normalize the total sum to one.
The Figure indicates that most of the predictive performance is concentrated in low distances.
Propagation / GNN In SGNNs, structure features are usually embedded into a continuous space, concatenated to any node features and propagated over subgraphs.
While this procedure is necessary for ZO encodings, it is less clear why labels that precisely encode distances and are directly comparable as distances should be embedded in this way.
Instead of passing embedded DE or DRNL structure features through a GNN, we fixed the number of DE features by setting the max distance to three and trained an MLP directly on the counts of these nine features (e.g. (1,1): 3, (1,2): 1, etc.).
Figure 4a shows that doing so, while leaving ceteris paribus (node features still propagated over the subgraph) actually improves performance in two out of three datasets.
We also investigated if any features require SGNN propagation by passing both raw node features and structure feature counts through an MLP.
The results in the right columns of Figure 4b indicate that this reduces performance severely, but by pre-propagating the features as x ′ u = 1 / | N ( u ) | (cid:80) i ∈ N ( u ) x
i (middle columns) it is possible to almost recover the performance of propagation with the SGNN (left columns).
GNN
MLP
H R @ 100 81.
HR@100 by structure feature propagation (a) Propagating structure features
GNN preprop no prop
H R @ 85.
HR@100 by node feature propagation
Cora Citeseer
Pubmed
[Figure]
u v 1, 1, 1, 1, 2,
Y uv on the subgraph, a readout function R ( S uv , Y uv ) maps a representations to link probabilities.
For graph classification problems, this is most commonly done by pooling node representations (graph pooling) typically with a mean or sum operation plus an MLP.
For
LP, an alternative is edge pooling with R ( y u , y v ) , usually with the
Hadamard product.
A major advantage of edge pooling is that it can be formulated subgraph free.
Figure 2b indicates that edge pooling produces better predictive performance than either mean or sum pooling across all nodes in V uv .
Analysis Summary
The main results of Section 3 are that (i)
The inclusion of structure features leads to very large improvements across all datasets (Figure 2a); (ii) The processing of these features, by embedding them and propagating them with an SGNN is sub-optimal both in terms of efficiency and performance (Figure 4a); (iii)
Most of the importance of the structure features is located in the lowest distances (Figure 3); and (iv) edge level readout functions greatly outperform mean or sum pooling over subgraphs (Figure 2b).
If, on one hand, subgraphs are employed as a tractable alternative to the full graph for each training edge, on the other, generating them remains an expensive operation ( O ( deg k ) time complexity for regular graphs and O ( |E| ) for complex networks with power law degree distributions 3 , see Appendix C.2).
Within this context, our analysis shows that if the information necessary to compute structure features for an edge can be encoded in the nodes, then it is possible to recover the predictive performance of SGNNs without the cost of generating a different subgraph for each edge.
We build upon this observation to design an efficient yet expressive model in
Section 4.
L INK P REDICTION WITH S UBGRAPH S KETCHING
We now develop a full-graph GNN model that uses node-wise subgraph sketches to approximate structure features such as the counts of DE and DRNL labels, which our analysis indicated are sufficient to encompass the salient patterns governing the existence of a link (Figure 4a).
4. S TRUCTURE F EATURES C OUNTS
Let A uv [ d u , d v ] be the number of ( d u , d v ) labels for the link ( u , v ), which is equivalent to the number of nodes at distances
exactly d u and d v from u and v respectively (See Figure 5).
We compute
A uv [ d u , d v ] for all d u , d v less than the receptive field k , which guarantees a number of counts that do not depend on the graph size and mitigates overfitting.
To alleviate the loss of information coming from a fixed k , we also compute B uv [ d ] = (cid:80) ∞
d v = k
+1 A uv
[ d, d v ] , counting the number of nodes at distance d from u and at distance > k from v .
We compute B uv [ d ] for all 1 ≤ d ≤ k .
Figure 6 shows
Structure features A and B measure the cardinalities of intersections of these neighborhoods.
how A and B relate to the neighborhoods of the two nodes.
Overall, this results in k 2 counts for A and 2 k counts for B ( k for the source and k for the destination node), for a total of k ( k + 2) count features.
These counts can be computed efficiently without constructing the whole subgraph for each edge.
Defining N d u ,d v ( u, v ) ≜ N d u ( u ) ∩ N d v ( v ) , we have
[Equation]
where N d ( u ) are the d -hop neighbors of u (i.e. nodes at distance ≤ d from u ).
Estimating Intersections and Cardinalities | N d u ,d v ( u, v ) | and | N d ( u ) | can be efficiently approximated with the sketching techniques introduced in section 2.
Let h ( d ) u and m ( d ) u be the HyperLogLog and MinHash sketches of node u ’s d -hop neighborhood, obtained recursively from the initial node sketches h (0) u
and
m (0) u using the relations m ( d ) u
[Equation]
and
[Equation]
We can approximate the intersection of neighborhood sets as
[Equation]
where H ( x, y ) = 1 /n (cid:80) ni
δ x i ,y i is the Hamming similarity, card is the HyperLogLog cardinality estimation function (see Appendix C.3.1) and max is taken elementwise.
MinHashing approximates the Jaccard similarity ( J ) between two sets and HyperLogLog approximates set cardinality.
| N d ( u ) | can be approximated as | N d ( u ) | ≈ card ( h ( d ) u ) .
The cost of this operation only depends on additional parameters of HyperLogLog and MinHashing (outlined in more detail in Section 2) which allow for a tradeoff between speed and accuracy, but is nevertheless independent of the graph size.
Using this approximations we obtain estimates ˆ A and ˆ B of the structure features counts.
4. E FFICIENT L INK P REDICTION WITH H ASHES (ELPH)
We present ELPH, a novel MPNN for link prediction.
In common with other full-graph GNNs, it employs a feature propagation component with a link level readout function.
However, by augmenting the messages with subgraph sketches, it achieves higher expressiveness for the same asymptotic complexity.
ELPH’s feature propagation fits the standard MPNN formalism (Gilmer et al., 2017):
[Equation]
where ϕ, γ are learnable functions, □ is a local permutation-invariant aggregation function (typically sum, mean, or max), x (0) u are the original node features, and m (0) u and h (0)
u are the MinHashing and
HyperLogLog single node sketches respectively.
Minhash and hyperloglog sketches of N l ( u ) are computed by aggregating with min and max operators respectively the sketches of N l − 1 ( u ) (Eq. 7).
These sketches can be used to compute the intersection estimations ˆ B uv
and ˆ A uv up to the l -hop neighborhood, which are then used as edge features (Eq.
Intuitively, the role of the edge features is to modulate message transmission based on local graph structures, similarly to how attention is used to modulate message transmission based on feature couplings.
A link predictor is then applied to the final node representations of the form
[Equation]
where [ k ] = { 1 , . .
. , k } , k is a receptive field, ψ is an MLP and ⊙ is the element-wise product.
p ( u, v ) decouples the readout function from subgraph generation as suggested in Figure 2b.
This approach moves computation from edgewise operations e.g. generating subgraphs, to nodewise operations (generating sketches) that encapsulate the most relevant subgraph information for LP.
We have shown that (i) GNN propagation is either not needed (structure features) or can be preprocessed globally (given features) (ii) structure features can be generated from sketches instead of subgraphs (iii) an edge pooling readout function performs as well as a graph pooling readout.
In combination these approaches circumvent the need to explicitly construct subgraphs.
The resulting model efficiently combines node features and the graph structure without explicitly constructing subgraphs and is as efficient as GCN (Kipf & Welling, 2017).
Expressive power of ELPH ELPH combines the advantageous traits of both GNNs and subgraphbased methods, i.e. tractable computational complexity and the access to pair-wise structural features.
Furthermore, as the following results show, it is more expressive than MPNNs:
Proposition 4.1.
Let M ELPH be the family of ELPH models as per Equations 7 - 9 and 10 where estimates are exact ( ˆ A ≡ A , ˆ B ≡ B ) 4 .
M ELPH does not suffer from the automorphic node problem.
While we rigorously define the automorphic node problem and prove the above in Section A.2, this result states that there exists non-automorphic links with automorphic nodes that an ELPH model is able to discriminate (thanks to structural features).
Contrary to ELPHs, MPNNs are not able to distinguish any of these links; we build upon this consideration, as well as the observation that ELPH models subsume MPNNs, to obtain this additional result:
Theorem 4.2.
Let M MPNN be the family of Message Passing Neural Networks (Equation 1).
M ELPH is strictly more powerful than M MPNN ( M ELPH ⊏ M MPNN ).
We prove this in Section A.2.
Intuitively, the Theorem states that while all links separated by MPNNs are also separated by ELPHs, there also exist links separated by the latter family but not by the former.
S CALING ELPH WITH P REPROCESSING (BUDDY)
Similarly to other full-graph GNNs (e.g. GCN), ELPH is efficient when the dataset fits into GPU memory.
When it does not, the graph must be batched into subgraphs.
Batching is a major challenge associated with scalable GNNs and invariably introduces high levels of redundancy across batches.
Here we introduce a large scale version of ELPH, called BUDDY, which uses preprocessing to side-step the need to have the full dataset in GPU memory.
Preprocessing Figure 4b indicated that a fixed propagation of the node features almost recovers the performance of learnable SGNN propagation.
This propagation can be achieved by efficient sparse scatter operations and done only once in preprocessing.
Sketches can also be precomputed in a similar way:
[Equation]
where scatter_min
[Equation]
, G
[Equation]
( u ) m
( l − 1) v and scatter_max and scatter_mean are defined similarly, X (0) are the original node features, and M (0) and H (0) are the MinHashing and HyperLogLog single node sketches respectively.
Similarly to (Rossi et al., 2020), we concatenate features diffused at different hops to obtain the input node features: Z = (cid:2) X (0)
∥ X (1) ∥ ...
∥ X ( k ) (cid:3) .
Link Predictor We now have
[Equation]
, where ψ is an MLP and ˆ A uv
and ˆ B uv are computed using M and H as explained in Section 4.1.
Doing so effectively converts a GNN into an MLP, removing the need to sample batches of subgraphs when the dataset overflows GPU memory (Rossi et al., 2020).
Further improvements in efficiency are possible when multiple computations on the same edge sets are required.
In this case, the edge features { ˆ B uv [ d ] , ˆ A uv [ d u , d v ] } can be precomputed and cached.
Time Complexity Denoting the complexity of hash operations as h , the node representation dimension d
and the number of edges E = |E| , the preprocessing complexity of BUDDY is O ( kEd + kEh ) .
The first term being node feature propagation and the second sketch propagation.
Preprocessing for SEAL and NBFNet are O (1) .
A link probability is computed by (i) extracting k ( k + 2) k -hop structural features, which costs O ( k 2 h ) and (ii) An MLP on structural and node features, which results in O ( k 2 h + kd 2 ) operations.
Both are independent of the size of the graph .
Since SGNNs construct subgraphs for each link they are O ( Ed 2 ) for complex networks (See Section C.2).
Finally, the total complexity of NBFNet is O ( Ed + Nd 2 ) with an amortized time for a single link prediction of O ( Ed/N + d 2 ) (Zhu et al., 2021).
However, this is only realized in situations where the link probability p ( u, j ) is required ∀ j ∈ V .
Time complexities are summarized in Table 1 and space complexities in Appendix B.4.
Expressiveness of BUDDY Similarly to ELPH, BUDDY does not suffer from the automorphic node problem (see Proposition and its Proof in Appendix A.2).
However, due to its non-parametric feature propagation, this class of models does not subsume MPNNs and we cannot exclude the presence of links separated by MPNNs, but not by BUDDY.
Nevertheless, BUDDY empirically outperforms common MPNNs by large margins (see Section 7), while also being extremely scalable.
R ELATED W ORK
[Table]
Subgraph methods for link prediction were introduced as the Weisfeiler
Leman Neural Machine (WLNM) in (Zhang & Chen, 2017).
As an MLP is used to learn from subgraphs instead of a GNN their model is not permutation-equivariant and uses a hashing-based WL algorithm (Kersting et al., 2014) to generate node indices for the MLP.
Note that ‘hashing’ here refers to injective neighbor aggregation functions, distinct from the data sketching methods employed in the current work.
The MLP in WLNM was replaced by a GNN in the seminal SEAL (Zhang & Chen, 2018), thus removing the need for a hashing scheme.
Additional methodological improvements were made in (Zhang et al., 2021; Zhang & Li, 2021) and subgraph generation was improved in (Yin et al., 2022).
Applications to recommender systems were addressed in (Zhang & Chen, 2019) and random walks are used to represent subgraph topology in (Yin et al., 2022; Pan et al., 2022).
The DRNL labeling scheme was introduced in SEAL and further labeling schemes are developed in (You et al., 2021;
Li et al., 2020) and generalized with equivariant positional encodings (Wang et al., 2022).
Neural Bellman-Ford Networks (Zhu et al., 2021) takes a different approach.
It is concerned with single source distances where each layer of the neural networks is equivalent to an iteration of the
Bellman-Ford algorithm and can be regarded as using a partial labeling scheme.
The same graph and augmentation can be shared among multiple destinations and so it is faster (amortized) than
SGNNs.
However, it suffers from poor space complexity and thus requires impractically high memory consumption.
Neo-GNN (Yun et al., 2021) directly learns a heuristic function using an edgewise and a nodewise neural network.
#GNN (Wu et al., 2021) directly hashes node features, instead of graph structure, and is limited to node features that can be interpreted as set memberships.
CFLP Zhao et al.
(2022) applies causal inference to LP.
Another paradigm for LP uses self-supervised node embeddings combined with an (often) approximate distance metric.
Amongst the best known models are TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), or complEx (Trouillon et al., 2016), which have been shown to scale to large LP systems in e.g. (Lerer et al., 2019; El-Kishky et al., 2022).
Decoupling feature propagation and learning in GNNs was first implemented in SGC (Wu et al., 2019) and then scalably in SIGN (Rossi et al., 2020).
[Table]
E XPERIMENTS
Datasets, Baselines and Experimental Setup We report results for the most widely used Planetoid citation networks Cora (McCallum et al., 2000), Citeseer (Sen et al., 2008) and Pubmed (Namata et al., 2012) and the OGB link prediction datasets (Hu et al., 2020).
We report results for: Three heuristics that have been successfully used for LP, Adamic-Adar (AA) (Adamic & Adar, 2003),
Resource Allocation (RA) (Zhou et al., 2009) and Common Neighbors (CN); two of the most popular GNN architectures: Graph Convolutional Network (GCN) (Kipf & Welling, 2017) and
GraphSAGE (Hamilton et al., 2017); the state-of-the-art link prediction GNNs Neo-GNN (Yun et al., 2021), SEAL (Zhang & Chen, 2018) and NBFNet (Zhu et al., 2021).
Additional details on datasets and the experimental setup are included in Appendix B.
Results Results are presented in Table 2 with metrics given in the first row.
Either ELPH or BUDDY achieve the best performance in five of the seven datasets, with SEAL being the closest competitor.
Being a full-graph method, ELPH runs out of memory on the two largest datasets, while its scalable counterpart, BUDDY, performs extremely well on both.
Despite ELPH being more general than
BUDDY, there is no clear winner between the two in terms of performance, with ELPH outperforming
BUDDY in three of the five datasets where we have results for both.
Ablation studies for number of hops, sketching parameters and the importance of node and structure features are in Appendix D.2.
[Table]
Runtimes Wall times are shown in
Table 3.
We report numbers for both the static mode of SEAL (all subgraphs are generated and labeled as a preprocessing step), and the dynamic mode (subgraphs are generated on the fly).
BUDDY is orders of magnitude faster both in training and inference.
In particular, Buddy is 200–1000 × faster than SEAL in dynamic mode for training and inference on the Citation dataset.
We also show GCN as a baseline.
Further runtimes and a breakdown of preprocessing costs are in Appendix D.1 with learning curves in Appendix F.
C ONCLUSION
We have presented a new model for LP that is based on an analysis of existing state of the art models, but which achieves better time and space complexity and superior predictive performance on a range of standard benchmarks.
The current work is limited to undirected graphs or directed graphs that are first preprocessed to make them undirected as is common in GNN research.
We leave as future work extensions to directed graphs and temporal / dynamically evolving graphs and investigations into the links with graph curvature (See Appendix C.5).
A T HEORETICAL A NALYSES A. P RELIMINARIES
We introduce some preliminary concepts that will be useful in our analysis.
Let us start with the definition of graph isomorphism and automorphism .
Definition A.1 (Graph isomorphism and automorphism) .
Let G 1 =
( V 1 , E 1 ) , G 2 =
( V 2 , E 2 ) be two simple graphs.
An isomorphism between G 1 , G 2 is a bijective map φ : V 1 → V 2 which preserves adjacencies, that is: ∀ u, v ∈ V 1 : ( u, v )
∈
E 1 ⇐⇒ ( φ ( u ) , φ ( v ))
∈
E 2 .
If G 1 =
G 2 , φ is called an automorphism .
In view of the definition above, two graphs are also called isomorphic whenever there exists an isomorphism between the two.
Intuitively, if two graphs are isomorphic they encode the same exact relational structure, up to relabeling of their nodes.
As we shall see next, automorphisms convey a slightly different meaning: they allow to define graph symmetries by formalizing the concept of node structural roles.
The above definitions can be extended to attributed graphs, more conveniently represented through third-order tensors A ∈ R n 2 × d , with n = | V | , d > 0 the number of features (Maron et al., 2019;
Zhang et al., 2021).
Indexes in the first two dimensions of tensor A univocally correspond to vertexes in V through a bijection ι : V → { 1 , . . . , n } .
Here, elements (cid:0) A (cid:1)
i,i, : correspond to node features, while elements (cid:0) A (cid:1) i,j, : , i ̸
=
j to edge features, which include the connectivity information in E .
Within this context, a graph is defined as a tuple G =
( V, E, ι, A ) , and isomorphisms and automorphisms are interpreted as permutations acting on A as:
[Equation]
where S n is the set of all bijections (permutations) over n symbols.
We then define the concepts of isomorphism and automorphism as:
Definition A.2 (Graph isomorphism and automorphism on attributed graphs) .
Let G 1 , G 2 be two attributed graphs represented by tensors A 1 , A 2 ∈ R n 2 × d .
An isomorphism between G 1 , G 2 is a bijective map (permutation) σ ∈ S n such that σ · A 1 =
A 2 .
For a graph G = A , σ ∈ S n is an automorphism if σ · A = A .
Let us get back to the information conveyed by automorphisms by introducing the following definition:
Definition A.3 (Graph automorphism group) .
Let G be a simple graph and A G be the set of graph automorphisms defined on its vertex set.
Aut G =
( A G , ◦ ) is the group having A G as base set and function composition ( ◦ ) as its operation.
Group Aut G induces an equivalence relation ∼ G ⊆
V × V : ∀ u, v ∈ V, u ∼ G v ⇔ ∃ φ ∈
A G : u = φ ( v ) (it is easily proved that ∼ G is indeed reflexive, symmetric and transitive, thus qualifying as an equivalence relation).
The equivalence classes of ∼ G partition the vertex set into orbits :
Orb G ( v ) =
(cid:8) u ∈ V (cid:12)(cid:12)
u ∼ G v (cid:9) =
(cid:8) u ∈ V (cid:12)(cid:12)
∃
φ ∈
A G : u = φ ( v ) (cid:9) .
The set of orbits identifies all structural roles in the graph.
Two vertexes belonging to the same orbit have the same structural role in the graph and are called ‘symmetric’ or ‘automorphic’.
In fact, it is possible to define automorphisms between node-pairs as well (Srinivasan & Ribeiro, 2019; Zhang et al., 2021):
Definition A.
(Automorphic node pairs) .
Let G
=
A be an attributed graph and ℓ 1 = ( u 1 , v 1 ) , ℓ 2 = ( u 2 , v 2 ) ∈
V × V be two pairs of nodes.
We say ℓ 1 , ℓ 2 are automorphic if there exists an automorphism σ ∈ A G such that σ · ℓ 1 = ( σ ( u 1 ) , σ ( v 1 ))
= ( u 2 , v 2 ) =
ℓ 2 .
We write ℓ 1 ∼ (2) G ℓ 2 .
With the above it is possible to extend the definition of orbits to node-pairs, by considering those node-pair automorphisms which are naturally induced by node ones, i.e.: ∀ φ ∈
A G , φ ∗ : ( u, v ) (cid:55)→ ( φ ( u ) , φ ( v )) .
Notably, for two node pairs
[Equation]
ℓ 1 ∼ (
[Equation]
) G ℓ 2 , i.e., automorphism between nodes does not imply automorphism between node pairs.
For instance, one counter-example of automorphic nodes involved in non-automorphic pairs is depicted in
Figure 1, while another one is constituted by pair ( v 0 , v 2 ) , ( v 0 , v 3 ) in Figure 7, whose automorphisms are reported in Tables 4 and 5.
This ‘phenomenon’ is the cause of the so-called “automorphic node problem”.
Definition A.5 (Automorphic node problem) .
Let M be a family of models.
We say M suffers from the automorphic node problem if for any model M ∈ M , and any simple (attributed) graph G =
( V, E, ι, A ) we have that ∀ ( u 1 , v 1 ) , ( u 2 , v 2 )
∈ V × V , (cid:8) u 1 ∼ G u 2 , v 1 ∼ G v 2 (cid:9) =
⇒ M (cid:0) ( u 1 , v 1 ) (cid:1) =
M (cid:0) ( u 2 , v 2 ) (cid:1) .
The above property is identified as a ‘problem’ because there exist examples of non-automorphic node pairs composed by automorphic nodes.
A model with this property would inevitably compute the same representations for these non-automorphic pairs, despite they feature significantly different characteristics, e.g. shortest-path distance or number of common neighbors.
Importantly, as proved by Srinivasan & Ribeiro (2019) and restated by Zhang et al.
(2021), the model family of Message Passing Neural Networks suffer from the aforementioned problem:
Proposition A.6.
Let M MPNN be the family of Message Passing Neural Networks (Equation 1) representing node-pairs as a function of their computed (equivariant) node representations.
M MPNN suffers from the automorphic node problem.
We conclude these preliminaries by introducing the concept of link discrimination , which gives a (more) fine-grained measure of the link representational power of model families.
Definition A.7 (Link discrimination) .
Let G =
( V, E, ι, A ) be any simple (attributed) graph and M a model belonging to some family M .
Let ℓ 1 = ( u 1 , v 1 ) , ℓ 2 = ( u 2 , v 2 )
∈
V × V be two node pairs.
We say M discriminates pairs ℓ 1 ,
ℓ 2 iff M ( ℓ 1 ) ̸
= M ( ℓ 2 ) .
We write ℓ 1 ̸ = M ℓ 2 .
If there exists such a model M ∈ M , then family M distinguishes between the two pairs and we write ℓ 1 ̸
= M ℓ 2 .
Accordingly, M does not discriminate pairs ℓ 1 , ℓ 2 when it contains no model instances which assign distinct representations to the pairs.
We write ℓ 1 = M ℓ 2 .
We can compare model families based on their expressiveness , that is, their ability to discriminate node pairs:
Definition A.8 (More expressive) .
Let M 1 , M 2 be two model families.
We say M 1 is more expressive than M 2 iff ∀ G
=
( V, E, ι, A ) , ℓ 1 , ̸∼ (2) G ℓ 2 ∈ V × V, ℓ 1 ̸ =
M 2 ℓ 2 = ⇒ ℓ 1 ̸ = M 1 ℓ 2 .
We write M 1 ⊑ M 2 .
Put differently, M 1 is more expressive than M 2 when for any two node pairs, if there exists a model in M 2 which disambiguates between the two pairs, then there exists a model in M 1 which does so as well.
When the opposite is not verified, we say M 1 is strictly more expressive than M 2 :
Definition A.9 (Strictly more expressive) .
Let M 1 , M 2 be two model families.
We say M 1 is strictly more expressive than M 2 iff M 1 ⊑ M 2 ∧ M 2 ̸⊑ M 1 .
Equivalently, M 1 ⊑ M 2 ∧ ∃ G =
( V, E, ι, A ) , ℓ 1 , ̸∼ (2) G ℓ 2 ∈ V × V, s .
t .
ℓ 1 ̸ = M 1 ℓ 2 ∧ ℓ 1 =
M 2 ℓ 2 .
In other words, M 1 is strictly more expressive than M 2 when there exists no model in the latter family disambiguating between two non-automorphic pairs while there exist some models in the former which do so.
A. D EFERRED THEORETICAL RESULTS AND PROOFS
Let us start by reporting the Proof for Proposition 4.1, stating that ELPH models do not suffer from the automorphic node problem.
Proof of Proposition 4.1.
In order to prove the Proposition it is sufficient to exhibit an ELPH model which distinguishes between two node-pairs whose nodes are automorphic.
Consider G = C 6 , the chordless cycle graph with 6 nodes, which we depict in Figure 7.
Due to the symmetry of this graph, all nodes are in the same orbit, and are therefore automorphic: we report the set of all graph automorphisms A G in Table 4.
Let us then consider pairs ℓ 1 = ( v 0 , v 2 ) , ℓ 2 = ( v 0 , v 3 ) .
The two pairs satisfy the premise in the definition of the automorphic node problem, as v 0 ∼ G v 0 , v 2 ∼ G v 3 .
One
[Figure]
[Table] [Table]
[Table] [Table]
single ELPH message-passing layer ( k = 1 ) produces the following (exact) structural features.
Pair ℓ 1 : A v 0 ,v 2 [1 , 1] = 1 , B v 0 ,v 2 [1] = 1 ; pair ℓ 2 : A v 0 ,v 3 [1 , 1] = 0 , B v 0 ,v 3 [1] = 2 .
Thus, a one-layer ELPH model M is such that ℓ 1 ̸
= M ℓ 2 if the readout layer p ( u, v ) =
ψ (cid:16) x 1 u ⊙ x 1 v , ( ˆ B uv [1] , ˆ A uv [1 , 1]) (cid:17)
=
ˆ A uv [1 , 1] .
An MLP implementing the described ψ is only required to nullify parts of its input and apply an identity mapping on the remaining ones.
This MLP trivially exists (it can be even manually constructed).
We now move to the counterpart of the above result for BUDDY models.
Proposition A.10.
Let M BUDDY be the family of BUDDY models as described per Equations 10 and pre-processed features Z = (cid:2) X (0)
∥ X (1) ∥ ...
∥ X ( k ) (cid:3) , where estimates are exact ( ˆ A ≡ A , ˆ B ≡ B ).
M BUDDY does not suffer from the automorphic node problem.
Proof of Proposition A.10.
In order to prove the Proposition it is sufficient to notice that the readout in the above Proof completely neglects node features, while only replicating in output the structural features A .
This is also a valid BUDDY readout function, and allows BUDDY to output the same exact node-pair representations of the ELPH model described above.
As we have observed, these are enough to disambiguate ( v 0 , v 2 ) , ( v 0 , v 3 ) from graph C 6 depicted in Figure 7.
This concludes the proof.
In these results we have leveraged the assumption that A estimates are exact.
We observe that, if the
MinHash and HyperLogLog estimators are unbiased, it would be possible to choose a sample size large enough to provide distinct estimates for the two counts of interest, so that the above results continue to hold.
Unbiased cardinality estimators are, for instance, described in (Ertl, 2017).
[Table] [Table]
A more precise assessment of the representational power of ELPH can be obtained by considering the node-pairs this model family is able to discriminate w.r.t.
others.
Lemma A.11.
Let M ELPH be the family of ELPH models (Equations 7, 8, 9, and 10), M MPNN that of Message Passing Neural Networks (Equation 1).
M ELPH is more powerful than M MPNN ( M ELPH ⊑ M MPNN ).
Proof of Lemma A.11.
We prove this Lemma by noticing that the ELPH architecture generalizes that of an MPNN, so that an ELPH model can learn to simulate a standard MPNN by ignoring the structural features.
Specifically, ELPH defaults to an MPNN with (1) ϕ
( l ) (cid:16) x ( l − 1) u , x ( l − 1) v , e ( l ) u,v (cid:17)
=
ϕ ( l ) (cid:16) x ( l − 1) u , x ( l − 1) v (cid:17) , and (2) ψ (cid:16) x ku ⊙ x kv , { ˆ B uv [ d ] , ˆ A uv [ d u , d v ] : ∀ d, d u , d v = 1 , . . .
, k } (cid:17) =
ψ (cid:0) x ku ⊙ x kv (cid:1) .
This entails that, anytime a specific MPNN instance distinguishes between two non-automorphic node-pairs, there exists an ELPH model which does so as well: the one which exactly simulates such MPNN.
Lemma A.12.
There exist node-pairs distinguished by an ELPH or BUDDY model (with exact cardinality estimates) which are not distinguished by any MPNN model.
Proof of Lemma A.12.
The Lemma is proved simply by considering non-automorphic pairs ( v 0 , v 2 ) , ( v 0 , v 3 ) from graph C 6 depicted in Figure 7.
We have already shown how there exists both an ELPH and BUDDY model (computing exact estimates of A , B ) which separate the two.
On the other hand, we have already observed how the nodes in the pairs are automorphic as all belonging to the same orbit.
Thus, the two pairs cannot possibly be distinguished by any MPNN, since they suffer from the automorphic node problem (Srinivasan & Ribeiro, 2019; Zhang et al., 2021), as remarked in Proposition A.6.
Proof of Theorem 4.2.
The Theorem follows directly from Lemmas A.11 and A.12.
Does this expressiveness result also extend to BUDDY?
While we have proved BUDDY does not suffer from the automorphic node problem, its non-parametric message-passing scheme is such that this family of models do not generally subsume MPNNs.
On one hand, in view of Lemma A.12, we know there exist node-pairs separated by BUDDY but not by any MPNN; on the other, this does not exclude the presence of node-pairs for which the vice-versa is true.
[Table]
B A DDITIONAL E XPERIMENTAL D ETAILS B. D ATASETS AND T HEIR P ROPERTIES
B. E XPERIMENTAL S ETUP
In all cases, we use the largest connected component of the graph.
LP tasks require links to play dual roles as both supervision labels and message passing links.
For all datasets, at training time the message passing links are equal to the supervision links, while at test and validation time, disjoint sets of links are held out for supervision that are never seen at training time.
The test supervision links are also never seen at validation time, but for the Planetoid and ogbl-collab 5 datasets, the message passing edges at test time are the union of the training message passing edges and the validation supervision edges.
OGB datasets have fixed splits whereas for Planetoid, random 70-10-20 percent train-val-test splits were generated.
On DDI, NBFnet was trained without learned node embeddings and with a batch size of 5.
B. H YPERPARAMETERS
The p parameter used by HyperLogLog was 8 and the number of permutations used by MinHashing was 128.
All hyperparameters were tuned using Weights and Biases random search.
The search space was over hidden dimension (64–512), learning rate (0.0001–0.01) and dropout (0–1), layers (1–3) and weight decay (0–0.001).
Hyperparameters with the highest validation accuracy were chosen and results are reported on a test set that is used only once.
B. S PACE C OMPLEXITY
The space complexity for ELPH is almost the same as a GCN based link predictor.
We define the number of feature as F , the sketch size H (sum of hll and minhash sketch sizes), number of nodes as
N , the number of layers L , the batch size B and the number of edges as E .
We split link prediction into i) learning node representations and ii) predicting link probabilities from node representations.
Learning node representations with GCN has complexity
[Equation]
where the second and third terms represent the weight matrices and the node representation respectively.
ELPH has complexity,
[Equation]
And so the only difference is the additional space required to store the node sketches.
The GCN link predictor has space complexity
[Equation]
Where the first term is a batch of edges and the second term are the link predictor weight matrices.
ELPH also uses structure features, which assuming the number of GNN layers and the number of hops are the same, take up L(L+2) space per node.
The link prediction complexity of ELPH is then
[Equation]
[Equation]
Typically the number of layers L is 1,2,3 and so the additional overhead is small.
The space complexity of the BUDDY link predictor is the same as ELPH and node representations are built as a preprocessing step.
The preprocessing has space complexity
[Equation]
which correspond to the adjacency matrix, the propagated node features, the hll cardinality estimates and the minhash and hll sketches respectively.
The only difference between this and ELPH is that cardinality estimates are cached (LN) and no weight matrices are used.
B. I MPLEMENTATION D ETAILS
Our code is implemented in PyTorch (Paszke et al., 2019), using PyTorch geometric (Fey & Lenssen, 2019).
Code and instructions to reproduce the experiments will be made available following publication.
We utilized either AWS p2 or p3 machines with 8 Tesla K80 and 8 Tesla V100 respectively to perform all the experiments in the paper.
C M ORE ON S TRUCTURAL F EATURES C. H EURISTIC M ETHODS FOR L INK P REDICTION
Heuristic methods are classified by the receptive field and whether they measure neighborhood similarity or path length.
The simplest neighborhood similarity method is the 1-hop Common
Neighbor (CN) count.
Many other heuristics such as cosine similarity, the Jaccard index and the
Probabilistic Mutual Information (PMI) differ from CN only by the choice of normalization.
AdamicAdar and Resource Allocation are two closely related second order heuristics that penalize neighbors by a function of the degree
[Equation]
where N ( u ) are the neighbors of u .
Shortest path based heuristics are generally more expensive to calculate than neighborhood similarity as they require global knowledge of the graph to compute exactly.
The Katz index takes into account multiple paths between two nodes.
Each path is given a weight of α d , where α is a hyperparameter attenuation factor and d is the path length.
Similarly
Personalized PageRank (PPR) estimates landing probabilities of a random walker from a single source node.
For a survey comparing 20 different LP heuristics see e.g. (Lü & Zhou, 2011).
C. S UBGRAPH G ENERATION C OMPLEXITY
The complexity for generating regular k
-hop
subgraph is O
( deg k ) , where deg is the degree of every node in the subgraph.
For graphs that are approximately regular, with a well defined mean degree, the situation is slightly worse.
However, in general we are interested in complex networks (such as social networks, recommendation systems or citation graphs) that are formed as a result of preferential attachment.
Complex networks are typified by power law degree distributions of the form p ( deg ) =
deg − γ .
As a result the mean is only well defined if γ > 2 and the variance is finite only for γ > 3 .
As most real world complex networks fall into the range 2 < γ < 3 , we have that the maximum degree is only bounded by the number of edges in the graph and thus, so is the complexity of subgraph generation.
Table 6 includes the average size of one thousand 1-hop and 2-hop randomly generated graphs for each dataset used in our experiments.
In all cases, the size of subgraphs greatly exceeds the average degree baseline with very large variances.
C. S UBGRAPH S KETCHES
We make use of both the HyperLogLog and MinHash sketching schemes.
The former is used to estimate the size of the union, while the latter estimates the Jaccard similarity between two sets.
Together they can be used to estimate the size of set intersections.
C.3. H YPERLOGLOG
HyperLogLog efficiently estimates the cardinality of large sets.
It accomplishes this by representing sets using a constant size data sketch.
These sketches can be combined in time that is constant w.r.t the data size and linear in the sketch size using elementwise maximum to estimate the size of a set union.
The algorithm takes the precision p as a parameter.
From p , it determines the number of registers to use.
The sketch for a set S is comprised of m registers M 1 . . .
M m where m = 2 p .
A hash function h ( s ) maps elements from S into an array of 64-bits.
The algorithm uses the first p bits of the hash to associate elements with a register.
From the remaining bits, it computes the number of leading zeros, tracking the maximum number of leading zeros per register.
Intuitively a large number of leading zero bits is less likely and indicates a higher cardinality and for a single register the expected cardinality for a set where the maximum number of leading zeros is n is 2 n .
This estimate is highly noisy and there are several methods to combine estimates from different registers.
We use hyperloglog++ (Heule et al., 2013b), for which the standard error is numerically close to 1 . 04 /sqrt
( m ) for large enough m .
Hyperloglog can be expressed in three functions Initialize, Union, and Card.
Sketches are easily merged by populating a new set of registers with the element-wise max values for each register.
To extract the estimate, the algorithm finds the harmonic mean of 2 M [ m ] for each of the the m registers.
This mean estimates the cardinality of the set divided by m .
To find the estimated cardinality we multiply by m and α m .
α m is used to correct multiplicative bias.
Additional information about the computation of α m along with techniques to improve the estimate can be found in (Flajolet et al., 2007;
Heule et al., 2013a).
The full algorithm is presented in Algorithm 1.
Complexity The Initialize operation has a O ( m + | S | ) running time.
Union and Card are both
O ( m ) operations.
The size of the sketch for a 64-bit hash is 6 ∗ 2 p bits.
C.3. M INHASHING
The MinHash algorithm estimates the Jaccard index.
It can similarly be expressed in three functions
Initialize, Union, and J .
The p parameter is the number of permutations on each element of the set.
P i ( x ) computes a permutation of input x where each i specifies a different permutation.
The algorithm stores the minimum value for each of the p permutations of all hashed elements.
The
Jaccard estimate of the similarity of two sets is given by the Hamming similarity of their sketches.
The full algorithm is presented as Algorithm 2.
Complexity The Initialize operation has a O ( np | S | ) running time.
Union and J are both O ( np ) operations.
The size of the sketch is np longs.
Minhashing gives an unbiased estimate of the Jaccard with a variance given by the Cramer-Rao lower bound that scales as O (1 /np )
(Chamberlain et al., 2018).
C. L ABELING S CHEMES
Empirically, we found the best performing labeling scheme to be DRNL, which scores each node in
S uv based on it’s distance to u and v with the caveat that when scoring the distance to u , node v and all of its edges are masked and vice versa.
This improves expressiveness as otherwise for positive
Algorithm
Parameter p is used to control precision.
m = 2 p procedure HLLI NITIALIZE ( S ) for i ∈ range ( m ) do
M [ i ] =end for for v ∈ S do x = h ( v ) idx = ⟨ x 31 , . . .
, x 32 − p ⟩w = ⟨ x 31 − p , . . .
, x 0 ⟩
M [ idx ] = max ( M [ idx ] , ϱ ( w ))) end for return M end procedure procedure HLLU NION
( M 1 , M 2 ) for i ∈ range ( m ) do
M [ i ] = max ( M 1[ i ] , M 2[ i ])) end for return M end procedure
C ARD ( M ) return α end procedure edges every neighbor of u would always be a 2-hop neighbor of v and vice-versa.
The result of masking is that distances can be very large even for 2-hop graphs (for instance one node may have an egonet that is a star graph with a ring around the outside.
If the edges of the star are masked then the ring must be traversed).
The first few DRNL values are
disconnected nodes: ( ∞ , 0) , (0 , ∞ ) →2.
link nodes: (0 , 1) , (1 , 0) →3.
common neighbors: (1 , 1) →4.
12-common-neighbors (1 , 2) , (2 , 1) →5.
2-hop common neighbors: (2 , 2) →
and the pattern has a hash function given by
[Equation]
(18) where d = d ui + d vi .
It is slightly suboptimal to assign infinite distances to 0, which is at least part of the reason that they are one-hot encode them as labels.
Indeed, DE (Li et al., 2020) uses a max distance label, which they claim reduces overfitting.
C. S UBSTRUCTURE C OUNTING AND C URVATURE
The eight features depicted in Figure 6 can be used to learn local substructures.
A uv [1 , 1] counts the number of triangles that ( u, v ) participates in.
A uv [2 , 1] and A uv [1 , 2] will double count a four-cycle and single count a four-cycle with a single diagonal.
The model can not distinguish a four-clique from two triangles or a five-clique from three triangles.
A uv [2 , 2] counts five cycles.
LP has also recently been shown to be closely related to notions of discrete Ricci curvature on graphs (Topping et al., 2022), a connection that we plan exploring in future work.
Algorithm
Parameter np controls the number of permutations.
procedure M IN H ASH I NITIALIZE ( S ) for i ∈ range ( np ) do
M [ i ] =
maxvalue end for for v ∈ S do x = h ( v ) for i ∈ range ( np ) do
M [ i ] =
min ( M [ i ] , P i ( x )) end for end for return M end procedure procedure M IN H ASH U
NION
( M 1 , M 2 ) for i ∈ range ( np ) do
M [ i ] =
min ( M 1[ i ] , M 2[ i ])) end for return M end procedure procedure J( M 1 , M 2 ) num _ equal =for i ∈ range ( np ) do if M 1[ i ] =
M 2[ i ] then num _ equal + + end if end for return num _ equal/np end procedure
C. E XAMPLE S TRUCTURE F EATURES
Figure 8 provides an example of how structure features are calculated from a subgraph.
The figure provides the z values for each node in an eight node subgraph.
From these the structure features are calculated.
As an example
[Equation]
The value of A 67 [2 , 1] = 1 indicates that the is one node in common between the two-hop neighbors of node 6 and the 1-hop neighbors of node 7.
The common node is node 2.
Similarly B 67 [2] =means there is one element in the two-hop neighbors of 6 that is not in any of the k-hop neighbors of 7: node 4.
D A DDITIONAL E XPERIMENTS D. R UNTIMES AND D ISCUSSION
The results in Tables 3 and 7 are obtained by running methods on a single Tesla K80 GPU on an
AWS p2 machine.
In all cases SEAL used GCN (fastest GNN) and parameters were taken from the
SEAL OGB repo.
For the OGB datasets, SEAL runtimes are estimated based on samples due to the high runtimes.
Table 8 breaks BUDDY preprocessing times down into (i) generating hashes and (ii) propagating features for each node, where the same values are used for training and inference and (iii) constructing structure features from hashes for each query edge, which has a separate cost for inference.
We stress that both forms of preprocessing depend only on the dataset and so must only be done once ever for a fixed dataset and not e.g. after every epoch.
This is akin to the static behavior of SEAL (Zhang & Chen, 2018), where subgraphs are constructed for each edge (both training and inference) as a preprocessing step.
D. A BLATION S TUDIES
This section contains ablation studies for (i) the affect of varying the number of Minhash permutations and the HyperLogLog p parameter and (ii) removing either node features or structure features from
BUDDY.
D.2. H ASHING P ARAMETER A BLATION
Figures 9 and 10 are ablation studies of the hashing parameters that trade-off between accuracy and time/space complexity of the intersection estimates.
The method is relatively insensitive to both parameters allowing smaller values to be chosen when space / time complexity is a constraint.
Figure 9a shows that good performance is achieved providing more than 16 minhash permutations are used, while Figure 9b shows that p can be as low as 4 in the hyperloglog procedure.
For Figurevalues were calculated with no node features to emphasize the affect of only the hashing parameters.
This was not required for Figure 9 because relatively speaking the node features are less important for Collab (See Table 9).
Data sketching typically introduces a tradeoff between estimation accuracy and time and space complexity.
However, in our model, the time complexity for generating structure features from hashes is negligible at both training and inference time compared to the cost of a forward pass of the MLP.
The only place where these parameters have an appreciable impact on runtimes is in preprocessing the hashes.
When preprocessing the hashes, the cost of generating hyperloglog sketches is also negligible compared to the cost of the minhash sketches.
The relationship between the number of minhash permutations and the runtime to generate the hashes is shown in Figure 12.
[Figure]
[Figure] [Figure]
[Table] [Table]
DDI hashing node 0.
feature propagation node 0.
NA structure features edge 0.
D.2.
F EATURE A BLATION
Table 9 shows the degradation in performance of BUDDY with either structure features or node features removed with all hyperparameters held fixed.
DDI has no node features and BUDDY did
[Figure]
[Table]
not use the node features from PPA, which are one-hot species labels.
For datasets Collab and PPA, the structure features dominate performance.
For the Citation dataset, the contribution of structure features and node features is almost equal, with a relatively small incremental benefit of adding a second feature class.
For the Planetoid datasets adding structure features gives a significant, but relatively small incremental benefit beyond the node features.
It also be that combining node and structure features dramatically reduces the variance over runs for the Planetoid and Collab datasets.
E F ULL BUDDY A LGORITHM
A sketch of the full algorithm is given in Algorithm
Algorithm 3 Complete Procedure
Preprocess structure features and cache propagated node features with Graph G and features X procedure P REPROCESSING (G, X)
H 1 = M IN H ASH I NITIALIZE ( G )
H 2 = HLLI NITIALIZE ( G )
X ′ =
Propagate ( X ) end procedure
Generate edge probability predictions y using an MLP procedure P REDICT (H1,H2,X’) for edge ( u, v ) ∈ epoch do
SF u,v =
GetStructureFeatures ( u, v, H 1 , H 2) x u = GetNodeFeatures ( u, X ′ )
x v = GetNodeFeatures ( v, X ′ )
y
=
MLP ( SF u,v , x u , x v ) end for end procedure
F L EARNING CURVES
We provide learning curves in terms of number of epochs in Figures 13 - 19.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
G S OCIETAL I MPACT
We study LP in graph-structured datasets focusing primarily on methods rather than applications.
Our method, in principle, may be employed in industrial recommendation systems.
We have no evidence that our method enhances biases, but were it to be deployed, checks would need to be put in place that existing biases were not amplified
[Figure]