I N S TA N T E M B E D D
I N G :
E F F I C
I
E N T L O C A L N O D E R E P
R E
S
E
N
TAT
I O N S
Anonymous authors
Paper under double-blind review
A B S T R AC T
In this paper, we introduce InstantEmbedding, an efﬁcient method for generating single-node representations using local PageRank computations.
We theoretically prove that our approach produces globally consistent representations in sublinear time.
We demonstrate this empirically by conducting extensive experiments on real-world datasets with over a billion edges.
Our experiments conﬁrm that InstantEmbedding requires drastically less computation time (over 9 , 000 times faster) and less memory (by over 8 , 000 times) to produce a single node’s embedding than traditional methods including DeepWalk, node2vec, VERSE and FastRP.
We also show that our method produces high quality representations, demonstrating results that meet or exceed the state of the art for unsupervised representation learning on tasks like node classiﬁcation and link prediction.
I N T RO D U C T I O N
Graphs are widely used to represent data when are objects connected to each other, such as social networks, chemical molecules, and knowledge graphs.
A widely used approach in dealing with graphs is learning compact representations of graphs (Perozzi et al., 2014; Grover & Leskovec, 2016; Abu-El-Haija et al., 2018), which learns a d -dimensional embedding vector for each node in a given graph.
Unsupervised embeddings in particular have shown improvements in many downstream machine learning tasks, such as visualization (Maaten & Hinton, 2008), node classiﬁcation (Perozzi et al., 2014) and link prediction (Abu-El-Haija et al., 2018).
Importantly, since such embeddings are learned solely from the structure of the graph, they can be used across multiple tasks and applications.
Typically, graph embedding models often assume that graph data ﬁts in memory (Perozzi et al., 2014) and require representations for all nodes to be generated.
However, in many real-world applications, it is often the case that graph data is large but also scarcely annotated.
For example, the Friendster social graph (Yang & Leskovec, 2015) has only 30% nodes assigned to a community, from its total 65M entries.
At the same time, many applications of graph embeddings such as classifying a data item only require one current representation for the item itself, and eventually representations of labeled nodes.
Therefore, computing a full graph embedding is at worst infeasible and at best inefﬁcient.
These observations motivate the problem which we study in this paper – the Local Node Embedding problem.
In this setting, the embedding for a node is restricted to using only local structural information, and can not access the representations of other nodes in the graph or rely on trained global model state.
In addition, we require that a local method needs to produce embeddings which are consistent with all other node’s representations, so that the ﬁnal representations can be used in the same downstream tasks that graph embeddings have proved adapt at in the past.
In this work, we introduce InstantEmbedding, an efﬁcient method to generate local node embeddings on the ﬂy in sublinear time which are globally consistent.
Considering previous work that links embedding learning methods to matrix factorization (Tsitsulin et al., 2018; Qiu et al., 2018), our method leverages a high-order similarity matrix based on Personalized PageRank (PPR) as foundations on which local node embeddings are computed via hashing.
We offer theoretical guarantees on the locality of the computation, as well as the proof of the global consistency of the generated embeddings.
We show empirically that our method is able to produce high-quality representations on par with state of the art methods, with efﬁciency several orders of magnitude better in clock time and memory consumption: running 9 , 000 times faster and using 8 , 000 times less memory on the largest graphs that contenders can process.
[Table]
P R E L I M I NA R I E S & R E L AT E D W O R K 2. G RAPH E MBEDDING
Let G =
( V, E ) represent an unweighted graph, which contains a set of nodes V , | V | = n , and edges
E ⊆ ( V × V ) , | E | = m .
A graph can also be represented as an adjacency matrix A ∈ { 0 , 1 } n × n where A u,v = 1 iff ( u, v ) ∈
E .
The task of graph embedding then, is to learn a d -dimensional node embedding matrix X ∈ R n × d where X v serves as the embedding for any node v ∈ V .
We note that d (cid:28) n , i.e. the learned representations are low-dimensional, and the challenge of graph embedding is to best preserve graph properties (such as node similarities) in this space.
Following the formalization in Abu-El-Haija et al.
(2018), many graph embedding can be thought of minimizing an objective in the general form: min X L ( f ( X ) , g ( A )) , where f : R n × d → R n × n is a pairwise distance function on the embedding space, g : R n × n → R n × n is a distance function on the (possibly transformed) adjacency matrix, and L is a loss function over all ( u, v ) ∈ ( V × V ) pairs.
A number of graph embedding methods have been proposed.
One family of these methods simply learn X as a lookup dictionary of embeddings and calculate the loss via distance (Kruskal, 1964), or matrix factorization (either implicit (Perozzi et al., 2014; Grover & Leskovec, 2016) or explicit (Ou et al., 2016)).
Another line of work focuses on leveraging the graph structure using neighborhood aggregation (Battaglia et al., 2016; Scarselli et al., 2008), or the Laplacian matrix of the graph (Kipf & Welling, 2016).
On attributed structured data, Graph Convolutional Networks (Kipf & Welling, 2016) have been successfully applied to both supervised and unsupervised tasks (Veli ˇ ckovi ´
c et al., 2018).
However, in the absence of node-level features, Duong et al.
(2019) demonstrated that these methods do not produce meaningful representations.
Graph Embedding via Random Projection
The computational efﬁciency brought by advances in random projection (Achlioptas, 2003; Dasgupta et al., 2010) paved the way for its adaptation in graph embedding to allow direct construction of the embedding matrix X .
Two recent works,
RandNE (Zhang et al., 2018) and FastRP (Chen et al., 2019) iteratively project the adjacency matrix to simulate the higher-order interactions between nodes.
As we show in the experiments, these methods suffer from high memory requirements and are not always competitive with other methods.
2. L OCAL A LGORITHMS ON G RAPHS
Local algorithms on graphs (Suomela, 2013) solve graph without using the full graph.
A wellstudied problem in this space is personalized recommendation (Jeh & Widom, 2003) where users are represented as nodes in a graph and the goal is to recommend items to speciﬁc users leveraging the graph structure.
Classic solutions to this problem are Personalized PageRank (Gupta et al., 2013) and
Collaborative Filtering (Schafer et al., 2007;
He et al., 2017).
Interestingly, these methods have been recently applied to graph neural networks (Klicpera et al., 2019;
He et al., 2020).
We now recall the deﬁnition of Personalized PageRank that is one of the main ingredients in our embedding algorithm.
Deﬁnition (Personalized PageRank (PPR, variation)) .
Given s ∈ R n ( s i ≥ 0 , (cid:80)
i s i = 1 ), a distribution of the starting node of random walks, and α ∈ (0 , 1) , a decay factor, the Personalized PageRank vector π ( s ) ∈ R n is deﬁned recursively as:
[Equation]
where 12 ( I + D − 1 A ) is the lazy random-walk matrix.
PPR takes as input a distribution of starting nodes s , which is typically a n dimensional one-hot vector e i with 1 in the i -th coordinate, enforcing a local random walks starting from node i .
Following this practice, we denote π i ∈ R n , the PPR vector starting from a single node i , and PPR ∈ R n × n , the full PPR matrix for all nodes in the graph, where PPR i, : =
π ( e i ) .
VERSE (Tsitsulin et al., 2018) proposes to learn node embeddings by implicitly factorizing PPR .
Its stochastic approach can perform well, but lacks guarantees of stability and convergence.
The idea of learning embeddings based on local random walks has also been used in the property testing framework, a direction in graph algorithm aiming at analyzing the clustering structure of a graph (Kale & Seshadhri, 2008;
Czumaj & Sohler, 2010; Czumaj et al., 2015; Chiplunkar et al., 2018).
2. P ROBLEM S TATEMENT
In this work, we consider the problem of embedding a single node in a graph quickly.
More formally, we consider what we term the Local Node
Embedding problem: given a graph G and any node v , return a globally consistent structural representation for v using only local information around v , in time sublinear to the size of the graph.
A solution to the local node embedding problem should possess the following properties:
Locality .
The embeddings for a node are computed locally, i.e. the embedding for a node can be produced using only local information and in time independent of the total graph size.
Global Consistency .
A local method must produce embeddings that are globally consistent (i.e. able to relate each embedding to each other, s.t. distances in the space preserve proximity).
While many node embedding approaches have been proposed (Chen et al., 2018), to the best of our knowledge we are the ﬁrst to examine the local embedding problem.
Furthermore, no existing methods for positional representations of nodes meet these requirements.
We brieﬂy discuss these requirements in detail below, and put the related work in terms of these properties in Table 1.
Locality.
While classic node embedding methods, such as DeepWalk (Perozzi et al., 2014), node2vec (Grover & Leskovec, 2016), or VERSE (Tsitsulin et al., 2018) rely on information aggregated from local subgraphs (e.g. sampled by a random walk), they do not meet our locality requirement.
Speciﬁcally, they also require the representations of all the nodes around them, resulting in a dependency on information from all nodes in the graph (in addition to space complexity O ( nd ) where d is the embedding dimension) to compute a single representation.
Classical random-projection based methods also require access to the full adjacency matrix in order to compute the higher-order ranking matrix.
We brieﬂy remark that even methods capable of local attributed subgraph embedding (such as GCN or DGI) also do not meet this deﬁnition of locality, as they require a global training phase to calibrate their graph pooling functions.
Global Consistency .
This property allows embeddings produced by local node embedding to be used together, perhaps as features in a model.
While existing methods for node embeddings are global ones which implicitly have global consistency, this property is not trivial for a local method to achieve.
Speciﬁcally, a local method must produce a node representation that resides in a space, preserving proximities to all other node embeddings that may be generated, without relying on a global state.
One exciting implication of a local method which is globally consistent is that it can wait to compute a representation until it is actually required for a task.
For example, in a production system, one might only produce representations for immediate classiﬁcation when they are requested.
We propose our approach satisfying these properties in Section 3, and experimentally validate it in Section 4, followed by conclusions in Section 5.
M E T H O D
Here we outline our proposed approach for local node embedding.
We begin by discussing the connection between a recent embedding approach and matrix factorization.
Then using this analysis, we propose an embedding method based on randomly hashing the PPR matrix.
We note that this approach has a tantalizing property – it can be decomposed into entirely local operations per node.
With this observation in hand, we present our solution, InstantEmbedding.
Finally, we analyze the algorithmic complexity of our approach, showing that it is both a local algorithm (which runs in time sublinear to the size of G ) and that the local representations are globally consistent.
3. G LOBAL E MBEDDING USING PPR
A recently proposed method for node embedding, VERSE (Tsitsulin et al., 2018), learns node embeddings using a neural network which encodes Personalized PageRank similarities.
Their objective function, in the form of Noise Contrastive Estimation (NCE) (Gutmann & Hyv¨arinen, 2010), is:
[Equation]
where PPR is the Personalized PageRank matrix, σ is the sigmoid function, b is the number of negative samples, and U is a uniform noise distribution from which negative samples are drawn.
Like many SkipGram-style methods (Mikolov et al., 2013), this learning process can be linked to matrix factorization by the following lemma:
Lemma 3.1 (Tsitsulin et al.
VERSE implicitly factorizes the matrix log( PPR )+log n − log b into XX (cid:62) , where n is the number of nodes in the graph and b is the number of negative samples.
H ASHING FOR G RAPH E MBEDDING
Lemma 3.1 provides an incentive to ﬁnd an efﬁcient alternative to factorizing the dense similarity matrix M = log( PPR )+log n − log b .
Our choice of the algorithm requires two important properties: a) providing an unbiased estimator for the inner product, and b) requiring less than O ( n ) memory.
The ﬁrst property is essential to ensure we have a good sketch of M for the embedding, while the second one keeps our complexity per node sublinear.
In order to meet both requirements we propose to use hashing (Weinberger et al., 2009) to preserve the essential similarities of PPR in expectation.
We leverage two global hash functions h d : N → { 0 , ..., d − 1 } and h sgn : N → {− 1 , 1 } sampled from universal hash families U d
and U − 1 ,respectively, to deﬁne the hashing kernel H h d ,h sgn : R n → R d .
Applied to an input vector x , it yields h
= H h d ,h sgn ( x ) , where h
i = (cid:80) k ∈ h − 1 d ( i ) x k h sgn ( k ) .
We note that although H h d ,h sgn is proposed for vectors, it can be trivially extended to matrix M when applied to each row vector of that matrix, e.g. by deﬁning H h d ,h sgn ( M ) i, : ≡ H h d ,h sgn ( M i, : ) .
In the appendix we prove the next lemma that follows from (Weinberger et al., 2009) and highlights both the aforementioned properties:
Lemma 3.2.
The space complexity of H h d ,h sgn is O (1) and:
[Equation]
This matrix sketching technique is strongly related to the factorization proposed in Lemma 3.1.
To better understand this, we consider the approximation M ≈ U Σ U T .
If the (asymptotic) solution of VERSE is U √ Σ , then our method aims to directly approximate U Σ .
We show that this rescaled solution is more computationally tractable, while still preserving critical information for high-quality node representations.
Our algorithm for global node embedding is presented in Algorithm 1.
First, we compute the PPR matrix PPR (Line 2) with a generic approach ( CreatePPRMatrix ), which takes a graph and (cid:15) , the desired precision of the approximation.
We remark that any of the many proposed approaches for computing such a matrix (e.g. from Jeh & Widom (2003); Andersen et al.
; Lofgren et al.
(2014)) can be used for this calculation.
As the PPR could be dense, the same could be said about the implicit matrix M .
Thus, we ﬁlter the signal from non-signiﬁcant PPR values by applying the max operator.
We remove the constant log b from the implicit target matrix.
In lines (4-6), the provided hash function accumulates each value in the corresponding embedding dimension.
Interestingly, the projection operation only uses information from each node’s individual PPR vector π i to compute its representation.
In the following section, we will show that local calculation of the
PPR can be utilized to develop an entirely local algorithm for node embedding.
3. L OCAL N ODE E MBEDDING VIA I NSTANT E MBEDDING
Having a local projection method, all that we require is a procedure that can calculate the PPR vector for a node in time sublinear to size of the graph.
Speciﬁcally, for InstantEmbedding we propose that the CreatePPRMatrix operation consists of invoking the SparsePPR routine from Andersen et al.
(Andersen et al., 2007) once for each node i .
This routine is an entirely local algorithm for efﬁciently constructing π i , the PPR vector for node i , which offers strong guarantees.
The next lemma follows from (Andersen et al., 2007) and formalizes the result (proof in Appendix A.4).
Lemma 3.3.
The I NSTANT E MBEDDING ( v, G, d, (cid:15) )
algorithm computes the local embedding of a node v by exploring at most the O
( 1 / (1 − α ) (cid:15) ) nodes in the neighborhood of v .
We present InstantEmbedding, our algorithm for local node embedding, in Algorithm 2.
As we will show, it is a self-contained solution for the local node embedding problem that can generate embeddings for individual nodes extremely efﬁciently.
Notably, per Lemma 3.3, the local area around v explored by InstantEmbedding is independent of n .
Therefore the algorithm is strictly local.
Algorithm 2 InstantEmbedding
Input: node v , graph G , embedding dimension d , PPR precision (cid:15) , hash functions h d , h sgn Output: embedding vector w 1: function I NSTANT E MBEDDING ( v, G, d, (cid:15), h d , h sgn ) 2: π v ← SparsePPR ( v G ) 3: w ← 0 d 4: for do 5: 6: return w
3.2. A NALYSIS
We now prove some basic properties of our proposed approach.
First, we show that the runtime of our algorithm is local and independent of n , the number of nodes in the graph.
Then, we show that our local computations are globally consistent, i.e., the embedding of a node v is the same independently if we compute it locally or if we recompute the embeddings for all nodes in the graph at the same time.
Note that we focus on bounding the running time to compute the embedding for a single node in the graph.
Nonetheless, the global complexity to compute all the embeddings can be obtained by multiplying our bound by n , although it is not the focus of this work.
We state the following theorem and prove it in Appendix A.5.
Theorem 3.4.
The InstantEmbedding ( v, G, d, (cid:15) ) algorithm has running time
O ( d + 1 / α (1 − α ) (cid:15) ) .
Besides the embedding size d , both the time and and space complexity of our algorithm depend only on the approximation factor (cid:15) and the decay factor α .
Both are independent of n , the size of the graph, and m , the size of the edge set.
Notably, if O ( 1 / α (1 − α ) (cid:15) ) ∈
o ( n ) , as commonly happen in real world applications, our algorithm has sublinear time w.r.t.
the graph size.
Lastly, we note that the space complexity is also sublinear (due to Lemma 3.3), which we show in the appendix.
Now we turn our attention to the consistency of our algorithm, by showing that for a node v the embeddings computed by InstantEmbedding and GraphEmbedding are identical.
In the following we denote the graph embedding computed by GraphEmbedding ( G, d, (cid:15) ) for node v by
GraphEmbedding ( G, d, (cid:15) ) v , and we prove the following theorem (Appendix A.6).
Theorem 3.5 (Global Consistency) .
InstantEmbedding ( v, G, d, (cid:15) ) output equals one of
GraphEmbedding ( G, d, (cid:15) ) at position v .
Complexity Comparison.
Table 1 compares the complexity of InstantEmbedding with that of previous works: d , n , m stands for embedding dimension, size of graph and number of edges respectively.
Speciﬁcally, b ≥ 1 stands for the number of samples used in node2vec and VERSE.
It is noteworthy that all the previous works have time complexity depending on n , and perform at least linear w.r.t.
size of the graph.
In contrast, our algorithm depends only on (cid:15) and α , and has sublinear time w.r.t.
n , the graph size.
In Section 4, we experimentally verify the advantages of our principled method.
E X P E R I M E N T S
In the light of the theoretical guarantees about the proposed method, we perform extended experiments in order to verify our two main hypotheses:
1. H1.
Computing local node-embedding is more efﬁcient than generating a global embedding.
H2.
The local representations are consistent and of high-quality, being competitive with and even surpassing state-of-the-art methods on several tasks.
We assess H1 in Section 4.2, in which we measure the efﬁciency of generating a single node embedding for each method.
Then in Section 4.3 we validate H2 by comparing our method against the baselines on multiple datasets using tasks of node classiﬁcation, link prediction and visualization.
4. D ATASETS AND EXPERIMENTAL SETTINGS
[Table]
To ensure a relevant and fair evaluation, we compare our method against multiple strong baselines, including DeepWalk (Perozzi et al., 2014), node2vec (Grover & Leskovec, 2016), VERSE (Tsitsulin et al., 2018), and FastRP (Chen et al., 2019).
Each method was run on a virtual machine hosted on the Google Cloud Platform, with a 2.3GHz 16core CPU and 128GB of RAM.
All reported results use dimensionality d = 512 for every method.
We provide extended results for 4 additional baselines: RandNE (Zhang et al., 2018), NodeSketch (Yang et al., 2019), LouvainNE (Bhowmick et al., 2020) and FREDE (Tsitsulin et al., 2020) on a subset of tasks, along with full details regarding each method and its parameterization in the Appendix B.1.
For reproducibility, we release an implementation of our method.
InstantEmbedding Instantiation .
As presented in Section 3, our implementation of the presented method relies on the choice of PPR approximation used.
For instant single-node embeddings, we use the highly efﬁcient PushFlow (Andersen et al., 2007) approximation that enables us to dynamically load into memory at most 2 / (1 − α ) (cid:15) nodes from the full graph to compute a single PPR vector π .
This is achieved by storing graphs in binarized compressed sparse row format that allows selective reads for nodes of interest.
In the special case when a full graph embedding is requested, we have the freedom to approximate the PPR in a distributed manner (we omit this from runtime analysis, as [Figure] we had no distributed implementations for the baselines, but we note our local method is trivially parallelizable).
We refer to Appendix B.5 for the study of the inﬂuence of (cid:15) on runtime and quality.
Datasets .
We perform our evaluations on 10 datasets as presented in Table 2.
Detailed descriptions, scale-free and small-world measurements for these datasets are available in the supplementary material.
Note that on YouTube and Orkut the number of labeled nodes is much smaller than the total.
We observe this behavior in several real-world application scenarios, where our method shines the most.
4. P ERFORMANCE C HARACTERISTICS
We report the mean wall time and total memory consumption (Wolff) required to generate an embedding ( d =512) for a single node in the given dataset.
Note that due to the nature of all considered baselines, they implicitly have to generate a full graph embedding in order to get a node representation.
We repeat the experiment 1,000 times for InstantEmbedding due to its locality property; for the baselines, we average the running time from 5 experiments, and measure the memory usage once.
For reference, we also provide the performance of InstantEmbedding producing a full graph embedding in Appendix B.3.
Running Time .
As Figure 1(a) shows, InstantEmbedding is the most scalable method, drastically outperforming all the other methods, at the task of producing a single node embedding.
We are over 9,000 times faster than the next fastest baseline in the largest graph both methods can process, and can scale to graphs of any size.
Memory Consumption .As
Figure 1(b) shows, InstantEmbedding is the most efﬁcient method having been able to run in all datasets using negligible memory compared to the other methods.
Compared to the next most memory-efﬁcient baseline (VERSE) we are over 8,000 times more efﬁcient in the largest graph both methods can process.
The results of running time and memory analysis conﬁrm hypothesis H1 and show that InstantEmbedding has a signiﬁcant speed and space advantage versus the baselines.
The relative speedup continues to grow as the size of the datasets increase.
On a dataset with over 1 billion edges (Friendster), we can compute a node embedding in 80ms – fast enough for a real-time application!
4. E MBEDDING Q UALITY
Node Classiﬁcation.
This task measures the semantic information preserved by the embeddings by training a simple classiﬁer on a small fraction of labeled representations.
For each method, we perform three different random splits of the data.
More details are available in the Appendix B.4.1.
In Table 3 we report the mean Micro F1 scores with their respective conﬁdence intervals (corresponding Macro-F1 scores in the supplementary material).
For each dataset, we perform Welch’s [Table] [Table] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
t-test between our method and the best performing contender.
We observe that InstantEmbedding is remarkably good on these node classiﬁcation, despite its several approximations and locality restriction.
Speciﬁcally, on four out of ﬁve datasets, no other method is statistically signiﬁcant above ours, and three of these (PPI, CoCit and YouTube) we achieve the best classiﬁcation results.
In Figure 2, we study how our hyperparameter, the PPR approximation error (cid:15) , inﬂuences both the classiﬁcation performance, running time, and memory consumption.
There is a general sweet spot (around (cid:15) = 10 − 5 ) across datasets where InstantEmbedding outperforms competing methods while being orders of magnitude faster.
Data on the other datasets is available in Section B.5.
[Figure]
Link prediction.
We conduct link prediction experiments to assess the capability of the produced representations to model hidden connections in the graph.
For the dataset which has temporal information (CoAuthor), we select data until 2014 as training data, and split co-authorship links between 2015-2016 in two balanced partitions that we use as validation and test.
For the other datasets, we uniformly sample 80% of the available edges as training (to learn embeddings on), and use the rest for validation (10%) and testing (10%).
Over repeated runs, we vary the splits.
More details about the experimental design are available in the supplementary material.
We report results for each method in in Table 4, which shows average ROC-AUC and conﬁdence intervals for each method.
Across the datasets, our proposed method beats all baselines except VERSE, however we do achieve the best performance on YouTube by a statistically signiﬁcant margin.
Visualization.
Figure 3 presents UMAP (McInnes et al., 2018) projections on the CoCit dataset, where we grouped together similar conferences.
We note that our sublinear approach is especially well suited to visualizing graph data, as visualization algorithms only require a small subset of points (typically downsampling to only thousands) to generate a visualization for datasets.
The experimental analysis of node classiﬁcation, link prediction, and visualization show that despite relying on two different approximations (PPR & random projection), InstantEmbedding is able to very quickly produce representations which meet or exceed the state of the art in unsupervised representation learning for graph structure, conﬁrming hypothesis H2 .
We remark that interestingly
InstantEmbedding seems slightly better at node classiﬁcations than link prediction.
We suspect that the randomization may effectively act as a regularization which is more useful on classiﬁcation.
[Figure]
DB
[Figure]
IR)
[Figure]
A
[Equation]
E N D
[Equation]
A P RO O F S A. V ERSE AS M ATRIX F ACTORIZATION
Lemma A.1.
(restated from Tsitsulin et al.
(2020), (ref.
Lemma 3.1))
VERSE implicitly factorizes the matrix log( PPR ) + log n − log b into XX (cid:62) , where n is the number of nodes in the graph and b is the number of negative samples.
Proof.
(from Tsitsulin et al.
(2020), following Levy & Goldberg (2014);
Qiu et al.
Since
PPR is right-stochastic and the noise distribution does not depend on j we can decompose the positive and negative terms from the objective of VERSE:
[Equation]
Isolating the loss for a pair of vertices i, j :
[Equation]
We substitute z ij = x (cid:62) i x j , use our independence assumption, and solve for ∂ L ij ∂z ij =
PPR ij σ ( − z ij ) − bn σ ( z ij )
=
to get z ij =log n · PPR ij b , hence log( PPR )+log n − log b
=
XX (cid:62) .
A. H ASH K ERNEL
Lemma A.2.
(restated from Weinberger et al.
The hash kernel is unbiased:
[Equation]
Corollary A.2.1.
(ref.
Lemma 3.2)
The space complexity of H h d ,h sgn is O (1) and:
[Equation]
Proof.
We note that the space complexity required to store a hash function from an universal family is
O (1) .
Indeed, one can choose an universal hash family such that its elements are uniquely determined by a ﬁxed choice of keys.
As an example, the multiplication hash function (Cormen et al.
h A ( x ) = (cid:100) n ( xA mod 1)) (cid:101) requires constant memory to store the key A ∈ (0 , 1) .
In order to prove the projection provides unbiased dot-products, considering the expectation per each entry, we have:
[Equation]
From Lemma A.
[Equation]
which holds for all i, j pairs.
A. S PARSE P ERSONALIZED P AGE R ANK P ROPERTIES
Theorem
A.3.
(restated from
Andersen et al.
Properties of the
S PARSE P ERSONALIZED P AGE R ANK ( v, G, (cid:15) )
(3) algorithm are as follows.
For any starting vector v , and any constant (cid:15) ∈ (0 , 1] , the algorithm computes an (cid:15) -approximate
PersonalizedPageRank vector p .
Furthermore the support of p satisﬁes vol ( Supp ( p )) ≤ O ( 1 / (1 − α ) (cid:15) ) , and the running time of the algorithm is O ( 1 / α(cid:15) ) .
We note here that Andersen et al.
(2007) prove their results for the lazy transition matrix and not the standard transition matrix that we consider here.
Nevertheless as discussed in Appendix A. switching between the two deﬁnitions does not change the asymptotic of their results.
Input: node v , graph G , precision (cid:15) , return probability α Output: PPR vector π 1: function S PARSE P ERSONALIZED P AGE R ANK ( v , G , (cid:15) , α ) 2: r ← 0 n ( sparse ) 3: π ← 0 n ( sparse ) 4: r
[ v ] =5: while ∃ w ∈ G, r [ w ] > (cid:15) × deg ( w ) do 6: ˆ r ← r [ w ] 7: π [ w ] ← π [ w ] + α ˆ r 8: r [ w ] ← (1 − α )
ˆ r9: r [ u ] ← r [ u ] + (1 − α )
ˆ r 2 deg ( w ) , ∀ ( w, u ) ∈ G 10: return π
A. I NSTANT E MBEDDING L OCALITY
Lemma A.4.
(ref.
Lemma 3.3)
The I NSTANT E MBEDDING ( v, G, d, (cid:15) )
algorithm computes the local embedding of a node v by exploring at most the O
( 1 / (1 − α ) (cid:15) ) nodes in the neighborhood of v .
Proof.
First recall that the only operation that explores the graph in I NSTANT E MBEDDING is
S PARSE P ERSONALIZED P AGE R ANK , which explores a node w in the graph if and only if a neighbor of w has a positive score (i.e. r [ w (cid:48) ] > (cid:15) × deg ( w (cid:48) ) was true and thus π [ w (cid:48) ] > 0 ), and so it is part of the support of π .
Furthermore at the beginning of the algorithm only v is active.
So it follows that every node explored by the algorithm is connected to v via a path composed only by the nodes with π score strictly larger than 0 .
So its distance from v is bounded by the support of the π vector that is O ( 1 / (1 − α ) (cid:15) ) cf.
Theorem A.3.
A. I NSTANT E MBEDDING C OMPLEXITY
Theorem A.5.
(ref.
Theorem 3.4)
The I NSTANT E MBEDDING ( v, G, d, (cid:15) ) algorithm has running time
O ( d + 1 / α (1 − α ) (cid:15) ) .
Proof.
The ﬁrst step of the I NSTANT E MBEDDING is computing the approximate Personalized
PageRank vector.
As noted in Theorem A.3, this can be done in time
O ( 1 / α(cid:15) ) .
We now focus our attention to the second part of our algorithm, projecting the sparse PPR vector into the embedding space.
For each non-zero entry r j of the PPR vector π , we compute hash functions h d ( j ) , h sgn ( j ) and max(log( r j ∗ n ) , 0) in O (1) time.
The total number of iterations is equal to the support size of π , i.e. O ( 1 / (1 − α ) (cid:15) ) .
Finally, we note that our algorithm always generates a dense embedding, handling this variable in
O ( d ) time complexity.
However, in practice this term is negligible as 1 / e >> d .
Summing up the aforementioned bounds we get the total running time of our algorithm:
[Equation]
A. G LOBAL C ONSISTENCY
Theorem A.6.
(ref.
Theorem 3.5)
I NSTANT E MBEDDING ( v, G, d, (cid:15) ) output equals G RAPH E MBEDDING ( G, d, (cid:15) ) v .
Proof.
We begin by noting that for a ﬁxed parameterization, the
S PARSE P ERSONALIZED P AGE R ANK routine will compute an unique vector for a given node.
Analyzing now the W v,j entry of the embedding matrix generated by
G RAPH E MBEDDING ( G, d, (cid:15) ) , we have:
[Equation]
The entire computation is deterministic and directly dependent only on the hash functions of choice and the indexing of the graph.
By ﬁxing the two hash functions h d and h sgn , we also have that
W v,j = w vj where w v =
I NSTANT E MBEDDING ( v, G, d, (cid:15) ) , ∀ v ∈ [0 ..n − 1] , j ∈ [0 ..
d − 1] .
A. R EPARAMETERIZATION
We note that Andersen et al.
(2007) in their paper use a lazy random walk transition matrix.
Furthermore in their analysis they also consider a lazy random walk.
Nevertheless, this does not affect the asymptotic of their results, in fact in Proposition 8.1 in Andersen et al.
(2007) they show that the two deﬁnition are equivalent up to a small change in α .
More precisely, a standard personalized PageRank with decay factor α is equivalent to a lazy random walk with decay factor α 2 − α .
So all the asymptotic of the bounds in Andersen et al.
(2007) apply also to the classic random walk setting that we study in this paper.
B E X P E R I M E N T S B. M ETHODS D ESCRIPTIONS
We ran all baselines on 128 and 512 embedding dimensions.
As we expect our method to perform better as we increase the projection size, we performed an auxiliary test with embedding sizefor InstantEmbedding.
We also make the observation that learning-based methods generally do not scale well with an increase of the embedding space.
The following are the description and individual parameterization for each method.
• DeepWalk (Perozzi et al., 2014):
Constructs node-contexts from random-walks and learns representations by increasing the nodes co-occurrence likelihood by modeling the posterior distribution with hierarchical softmax.
We set the number of walks per node and their length to 80 , and context windows size to 10 .
• Node2Vec
(Grover & Leskovec, 2016): Samples random paths in the graph similar to DeepWalk, while adding two parameters, p and q , controlling the behaviour of the walk.
Estimates the likelihood through negative sampling.
We set again the number of walks per node and their length to 80 and windows size 10 , number of negative samples to 5 per node and p = q = 1 .
• Verse (Tsitsulin et al., 2018):
Minimizes objective through gradient descent, by sampling nodes from PPR random walks and negatives samples from a noise distribution.
We train it over 10 5 epochs and set the stopping probability to 0 .
• FastRP (Chen et al., 2019):
Computes a high-order similarity matrix as a linear combination of multiple-steps transitions matrices and projects it into an embedding space through a sparse random matrix.
We ﬁx the linear coefﬁcients to [0 , 0 , 1 , 6] and the normalization parameter − 0 .
• InstantEmbedding (this work): Approximate per-node PPR vectors with return probability α and precision (cid:15) , which are projected into the embedding space using two ﬁxed hash functions.
In all our experiments, we set α = 0 .
15 and (cid:15) > 1 n , where n is the number of nodes in the graph.
Four additional baselines were considered for extending a subset of our experiments, as follows:
• RandNE (Zhang et al., 2018): Linearnly combine transition matrices of multiple orders, randomly projected through an orthogonal Gaussian matrix. We used transitions up to order 3, with coefﬁcients [1 , 100 , 10 4 , 10 5 ] . • NodeSketch
(Yang et al., 2019):
Employs a recursive sketching process using a hash function that preserves hamming distances.
Where provided, we use the recommended default parameters, and on all other datasets we choose the best performing (‘order‘, α ) parameters from { (2 , 0 . 0001) , (5 , 0 . 001) } .
• LouvainNE
(Bhowmick et al., 2020):
Aggregates node representations from a successive sub-graph hierarchy.
We use the recommended defaults across all datasets, with Louvain partition strategy and a damping parameter a = 0 .
[Table]
• FREDE (Tsitsulin et al., 2020):
Sketches matrix 2 by iteratively computing per-node PPR vectors and using frequent directions.
We use the provided default parameters for all datasets in order to measure running times.
B. D ATASET D ESCRIPTIONS
The graph datasets we used in our experiments are as follows:
• PPI (Stark et al., 2006): Subgraph of the protein-protein interaction for Homo Sapiens species and ground-truth labels represented by biological states.
Data originally processed by Grover & Leskovec (2016).
• Blogcatalog (Tang & Liu, 2010): Network of social interactions between bloggers.
Authors specify categories for their blog, which we use as labels.
• Microsoft Academic Graph (MAG) (mag, 2016): Collection of scientiﬁc papers, authors, journals and conferences.
Two distinct subgraphs were originally processed by Tsitsulin et al.
(2018), based on co-authorship (CoAuthor) and co-citations (CoCit) relations.
For the latter one, labels are represented by the unique conference where the paper was published.
• Flickr (Tang & Liu, 2010): Contact network of users within 195 randomly sampled interest groups.
YouTube (Tang & Liu, 2010)
Social network of users on the video-sharing platform.
Labels are represented by group of interests with at least 500 subscribers.
• Amazon2M (Chiang et al., 2019): Network of products where edges are represented by co-purchased relations.
• Orkut (Yang & Leskovec, 2015): Social network where users can create and join groups, used at ground-truth labels.
We followed the approach of Tsitsulin et al.
and selected only the top 50 largest groups.
• Friendster (Yang & Leskovec, 2015): Social network where users can form friendship edge each other.
It also allows users form a group which other members can then join.
To better understand the variety of our chosen datasets, we report the scale-free and small-world characteristics of the networks in Table 5.
ﬁtting a power-law distribution to node degrees, and comparing it to other 2 distributions (exponential and log-normal) through the Kolmogorov-Smirnov test (R = mean log-likelihood ratio, p = degree of conﬁdence).
We note that on the CoAuthorship,
Flickr, YouTube, Amazon2M and Orkut networks, the h0 hypothesis can be rejected ( p < 0 . 05 ) and thus conclude that the log-normal distribution is a better ﬁt (graphs are not scale-free). Additionally, we report two small-world related measures: the pseudo-diameter of the graphs and their global clustering coefﬁcient (transitivity). We observe that that graphs we use in the study are diverse, covering the spectrum of small-world and large-diameter networks. B. R UNTIME A NALYSIS B.3. G ENERAL S ETUP
For the runtime analysis we use the same parameterization as described in B.1 for all methods.
In the special case of InstantEmbedding, we dynamically load into memory just the required subgraph in order to approximate the PPR vector for a single node.
We individually ran each method on a virtual machine hosted on the Google Cloud Platform, with a 2.3GHz 16-core CPU and 128GB of RAM.
B.3. R UNTIME : S PEED
All methods, except the single-threaded FastRP, leveraged the 16 cores of our machines.
Some methods did not complete all the tasks: none ran on Friendster; node2vec was unable to run on
Amazon2M and Orkut; FastRP did not run on Orkut speciﬁcally for a 512-dimension embedding.
We note that all reported numbers are real execution times, taking into account also loading the data in memory.
The detailed results are shown in Table 7.
For reference, we also provide the total running time for producing a full graph embedding. We note that when computing the PPR matrix (of an entire graph) a local measure may be suboptimal, and we leave optimizing global run time as future work.
Nevertheless, here we report the total running time of our local method successively applied to all nodes in a graph in Table 6 , with an additionalrecent baselines. All methods ran on the same machine and produced a 512-dimensional embedding for the node classiﬁcation task. From the additional baselines, only RandNE and LouvainNE could scale to Orkut, while FREDE could only produce an embedding on half of the datasets.
B.3. T ASK : M EMORY U SAGE
The methods that failed to complete in the Running Times section are also marked here accordingly.
We note that due to the local nature of our method, we can consistently keep the average memory usage under 1MB for all datasets.
This observation reinforces the sublinear guarantees of our algorithm when being within a good (cid:15) -regime, as stated in Lemma A.4.
The detailed results are shown in
[Table]
[Table]
[Table]
B. E MBEDDING Q UALITY B.4. Q UALITY : N ODE C LASSIFICATION
These tasks aim to measure the semantic information preserved by the embeddings, through the means of the generalization capacity of a simple classiﬁer, trained on a small fraction of labeled representations.
All methods use 512 embedding dimensions.
For each methods, we perform three different splits of the data, and for our method we generate ﬁve embeddings, each time sampling a different projection matrix.
We use a logistic regression (LR) classiﬁer from using Scikit-Learn (Pedregosa et al., 2011) to train the classiﬁers.
In the case of multi-class classiﬁcation (we follow
Perozzi et al.
and use a one-vs-rest LR ensemble) – we assume the number of correct labels K is known and select the top K probabilities from the ensemble.
To simulate the sparsity of labels in the real-wold, we train on 10% of the available labels for PPI and Blogcatalog and only 1% for the rest of them, while testing on the rest.
We treat CoCit as a multi-class problem as each paper is associated an unique conference were it was published.
Also, for Orkut we follow the approach from Tsitsulin et al.
and select only the top 50 largest communities, while further ﬁltering nodes belonging to more than one community.
In these two cases, are ﬁtting a simply logistic regression model on the available labeled nodes.
The other datasets have multiple labels per node, and we are using a One-vs-The-Rest ensemble.
When evaluating, we assume the number of correct labels, K , is known and select the top K probabilities from the ensemble.
For each methods, we are performing three different splits of the data, and for our method we generate ﬁve embeddings, sampling different projection matrices.
We report the average and 90% conﬁdence interval for micro and macro F1-scores at different fractions of known labels.
The following datasets are detailed for node classiﬁcation: PPI (Table 9),
BlogCatalog (Table 10), CoCit (Table 11), Flickr (Table 12), and YouTube (Table 13). We also report experiments with 4 additional baselines in Table 14. The classiﬁcation task is the same, however for NodeSketch we used the recommended SVC classiﬁer with hamming kernel
[Table]
as the Logistic Regression could not infer a separation boundary in this particular case.
Additionally, for FREDE we are referencing the scores from the original paper, having a comparable evaluation framework. We note that although FREDE produces better scores, it does not scale past a mediumsized graph [Table] and its extremely high running times (Table 6) takes this approach out of the original scope of our paper.
[Table]
For this task we create edge embeddings by combining node representations
[Table]
and treat the problem as a binary classiﬁcation. We observed that different strategies for aggregating embeddings could maximize the performance of different methods under evaluation
[Table]
so we conducted an in-depth
[Table]
[Table]
[Table] [Table] investigation in order for the fairest possible evaluation.
Speciﬁcally, for two node embeddings w and ˆ w we adopt the following strategies for creating edge representations:
dot-product: w (cid:62) ˆ w 2.
cosine distance: w (cid:62) ˆ w (cid:107) w (cid:107)(cid:107) ˆ w (cid:107) 3.
hadamard product: w (cid:12) ˆ w 4.
element-wise average: 12 ( w + ˆ w ) 5.
L1 element-wise distance: | w − ˆ w | 6.
L2 element-wise distance ( w − ˆ w ) (cid:12) ( w − ˆ w )
While the ﬁrst two strategies directly create a ranking from two embeddings, for the other ones we train a logistic regression on examples from the validation set.
In all cases, a likelihood scalar value will be attributed to all edges, and we report their ROC-AUC score on the test set.
Taking into account that different embedding methods may determine a speciﬁc topology of the embedding space, that may in turn favour a speciﬁc edge aggregation method, for each method we consider only the strategy that consistently provides good results on all datasets.
This ensures that all methods can be objectively compared to one another, independent of the particularities of induced embedding space geometry.
The following tables show detailed analysis of link prediction results for BlogCatalog (Table 15) and
CoAuthor (Table 16). [Table]
node2vec 15.
DNC VERSE 12.
81.52 RandNE 15.
59.01 FastRP 15.
DNC NodeSketch 12. DNC LouvainNE 14. 45.08 FREDE 19. DNC
DNC
DNC
InstantEmbedding 17.
[Figure]
[Figure]
B. E PSILON I NFLUENCE
In order to gain insight into the effect of (cid:15) on the behaviour of our method [Figure] we test 6 values in the range of [10 − 1 , ..., 10 − 6 ] .
We note that the decrease of (cid:15) is strongly correlated with a better classiﬁcation performance, but also to a larger computational overhead.
The only apparent exception seems to be the Micro-F1 score on the Blogcatalog dataset, which drops suddenly when (cid:15) =
10 − 6 .
We argue that this is due to the fact that more probability mass is dispersed further away from the central node, but the max operator cuts that information away (as the number of nodes is small), and thus the resulting embedding is actually less accurate.
B.5. T ASK : V ISUALIZATION
Figure 3 presents multiple UMAP (McInnes et al., 2018) projections on the CoCit dataset, where we grouped together similar conferences. We note that our sublinear approach is especially well suited to
[Table] [Figure]
[Figure] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
visualizing graph data, as visualization algorithms (such as t-SNE or UMAP) only require a small subset of points (typically downsampling to only thousands) to generate a visualization for datasets.
[Figure]