F AST N ODE E MBEDDINGS : L EARNING E GO -C ENTRIC R EPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Representation learning is one of the foundations of Deep Learning and allowed important improvements on several Machine Learning tasks, such as Neural Machine Translation, Question Answering and Speech Recognition.
Recent works have proposed new methods for learning representations for nodes and edges in graphs.
Several of these methods are based on the SkipGram algorithm, and they usually process a large number of multi-hop neighbors in order to produce the context from which node representations are learned.
In this paper, we propose an effective and also efﬁcient method for generating node embeddings in graphs that employs a restricted number of permutations over the immediate neighborhood of a node as context to generate its representation, thus ego-centric representations.
We present a thorough evaluation showing that our method outperforms state-of-the-art methods in six different datasets related to the problems of link prediction and node classiﬁcation, being one to three orders of magnitude faster than baselines when generating node embeddings for very large graphs.
I NTRODUCTION
Many important problems involving graphs require the use of learning algorithms to make predictions about nodes and edges.
These predictions and inferences on nodes and edges from a graph are typically done using classiﬁers with carefully engineered features (Grover & Leskovec, 2016).
These features, besides taking time and manual labor to be developed and acquired, usually do not generalize well to other problems or contexts.
The ﬁeld of Natural Language Processing (NLP) has had many advances due to the use of algorithms that learn word representations, instead of manually extracted features.
Originally proposed by
Bengio et al.
and commonly used with Word2Vec algorithms like CBOW and SkipGram (Mikolov et al., 2013a), word embeddings are used in many state-of-the-art solutions for neural machine translation (Luong & Manning, 2016; Firat et al., 2016), question answering (Andreas et al., 2016) and natural language generation (Wen et al., 2015).
Recent works have proposed new methods for learning representations for nodes and edges in graphs, based on random walks (Perozzi et al., 2014; Grover & Leskovec, 2016) or auto-encoding adjacency vectors (Wang et al., 2016).
In this paper, we propose a new general purpose method for generating node embeddings in very large graphs, which we call Neighborhood Based Node Embeddings (or simply NBNE).
NBNE is based on the SkipGram algorithm and uses nodes neighborhoods as contexts.
NBNE outperforms state-of-the-art DeepWalk (Perozzi et al., 2014) and Node2Vec (Grover & Leskovec, 2016) for the tasks of link prediction and node classiﬁcation on six collections, being one to three orders of magnitude faster.
In this work, we considered DeepWalk and Node2Vec as baselines.
The main reason for this improvement on effectiveness and efﬁciency is that we concentrate learning on the “predictable” parts of the graph.
A study by Facebook research (Edunov et al., 2016) found that each person in the world (at least among the people active on Facebook) is connected to every other person by an average 3 . 57 other people.
In a graph of this magnitude and connectedness, learning node embeddings by maximizing the log-probability of predicting nearby nodes in a random walk (with a window size of 5) can be highly inefﬁcient and make it ‘harder’ for the embeddings to be constructed, even if these random walks are biased like in Node2Vec.
We suspect this can also make them more unstable, which would explain why they need more iterations before embedding convergence.
The main contributions of this work are:
• We present a new general purpose method that is more effective and more efﬁcient than state-of-the-art methods for generating node embeddings in graphs.
• Experimental results in solving the link prediction and node classiﬁcation problems for six graph sets show that our method outperforms, in terms of effectiveness and efﬁciency , the methods DeepWalk and Node2Vec.
• In spite of the fact that our method has the same time complexity of the two baselines DeepWalk and Node2Vec, we were able to improve the training time by one to three orders of magnitude , which is important when dealing with very large graphs.
For instance, to learn node embeddings for a graph containing 317 , 080 nodes and 1 , 049 ,edges, collected from the DBLP 1 repository (Yang & Leskovec, 2012), NBNE took approximately 14m30s minutes, DeepWalk approximately 164m34s and Node2Vec approximately 3 , 285 m59s
(more than 54 hours).
• We provide a thorough evaluation of our method in real and synthetic graphs, motivating our choice for a semi supervised algorithm .
Our method has a single tunable parameter (number of permutations, which will be explained later) that can be tuned at once on the training set to avoid overﬁtting the representations.
R ELATED W ORK
The deﬁnition of node similarity and ﬁnding general purpose node and/or edge representations are non-trivial challenges (L ¨ u & Zhou, 2011).
Many deﬁnitions of similarity in graphs use the notion of ﬁrst and second order proximity.
First-order proximity is the concept that connected nodes in a graph should have similar properties, while the second-order proximity indicates that nodes with similar neighborhoods should have common characteristics.
Some earlier works on ﬁnding these embeddings use various matrix representations of the graph, together with dimensionality reduction techniques, to obtain the nodes’ representations (Roweis &
Saul, 2000; Tenenbaum et al., 2000).
A problem with these approaches is that they usually depend on obtaining the matrix’ eigenvectors, which is infeasible for large graphs ( O ( n 2 . 376 )) with the
Coppersmith-Winograd algorithm (Coppersmith & Winograd, 1987)).
Recent techniques attempt to solve this problem by dynamically learning representations for nodes in a graph using non-linear techniques based either on ﬁrst and second order proximities (Tang et al., 2015; Wang et al., 2016) or random walks (Perozzi et al., 2014; Grover & Leskovec, 2016).
Other recent works focus on ﬁnding representations for speciﬁc types of graphs.
TriDNR (Pan et al., 2016) uses a graph structure together with node content and labels to learn node representations in two citation networks.
Their work can be directly applied to any graph where nodes have labels and/or text contents.
TEKE (Wang & Li, 2016) and KR-EAR (Lin et al., 2016)
ﬁnd representations for entities in knowledge graphs and metapath2vec (Dong et al., 2017) ﬁnds node representations in heterogeneous networks.
The method LINE (Tang et al., 2015) ﬁnds a d dimensional representation for each node based on ﬁrst and second-order graph proximities, not being feasible for large graphs, because its cost function depends on the whole adjacency matrix ( O ( | V | 2 ) ).
Another method, Structural Deep Network Embedding (SDNE) (Wang et al., 2016), is also based on ﬁrst and second order proximities.
It uses autoencoders to learn a compact representation for nodes based on their adjacency matrix (second-order proximity), while forcing representations of connected nodes to be similar (ﬁrst-order proximity) by using an hybrid cost function.
SDNE is also not feasible for large graphs, since the autoenconders are trained on the complete adjacency vectors.
Each vector has size O ( | V | ) and is created at least once, creating a lower bound on time complexity
O ( | V | 2 ) .
The method DeepWalk (Perozzi et al., 2014) generates k random walks starting on each vertex in the graph to create sentences where each “word” is a node.
These sentences are then trained using the SkipGram algorithm to generate node embeddings.
This method has a time complexity bounded by O ( | V | log | V | ) .
Node2Vec (Grover & Leskovec, 2016) also uses random walks with SkipGram and can be seen as a generalization of DeepWalk.
The difference between the two methods is that Node2Vec’s walks are random, but biased by two pre-assigned parameters p and q .
During the creation of the walks, these parameters are used to increase the chance of the walk returning to a parent node or going farther from it.
This method uses a semi-supervised approach which requires several models to be generated and a small sample of labeled nodes to be used so that the best parameters p and q can be chosen.
Node2Vec is not efﬁcient for densely connected graphs, since its time and memory dependencies on the graph’s branching factor b are O ( b 2 ) .
In this work, we considered DeepWalk (Perozzi et al., 2014) and Node2Vec (Grover & Leskovec, 2016) as baselines, since they are scalable, having a time complexity ( O ( | V | log | V | ) ).
The main differences between NBNE and the two baselines are: (i) we use a different sentence sampling strategy which is based in a node’s neighborhood instead of random walks, (ii) NBNE is more effective than both Node2Vec and DeepWalk, as supported by our experiments in six different datasets, and (iii) NBNE is efﬁcient for both dense and sparse graphs and scalable for very large applications, having a faster training time than both Node2Vec and DeepWalk.
N EIGHBORHOOD B ASED N ODE E MBEDDINGS
The context of a word is not a straightforward concept, but it is usually approximated by the words surrounding it.
In graphs, a node’s context is an even more complex concept.
As explained above,
DeepWalk and Node2Vec use random walks as sentences and consequently as contexts in which nodes appear.
In this work, the contexts are based solely on the neighborhoods of nodes, deﬁned here as the nodes directly connected to it, focusing mainly on the second-order proximities.
Consequently, nodes’ representations will be mainly deﬁned by their neighborhoods and nodes with similar neighborhoods (contexts) will be associated with similar representations.
3. S ENTENCE G ENERATION
In our Neighborhood Based Node Embedding’s (NBNE) method, as the name implies, sentences are created based on the neighborhoods of nodes.
There are two main challenges in forming sentences from neighborhoods, as follows:
• A sentence containing all the neighbors from a speciﬁc highly connected root node might be of little use.
Most neighbors would be distant from each other in the sentence, not inﬂuencing each other’s representations, and not directly inﬂuencing the root node.
• There is no explicit order in the nodes in a neighborhood.
So there is no clear way to choose the order in which they would appear in a sentence.
In this work, the solution is to form small sentences, with only k neighbors in each, using random permutations of these neighborhoods.
Algorithm 1 presents the code for generating sentences.
As a trade-off between training time and increasing the training dataset the user can select the number of permutations n .
Selecting a higher value for n creates a more uniform distribution on possible neighborhood sentences, but also increases training time.
1: procedure GET S ENTENCES (graph, n ) 2: sentences ← [ ∅ ] 3: for j in 0 : n do 4: for node in graph .nodes() do 5: neighbors ← random permutation ( node .neighbors() ) 6: for i in 0 : len ( neighbors ) /k
do 7: sentence ← [ node ] + neighbors [ i · k : i · ( k + 1)] 8: sentences
3. L EARNING R EPRESENTATIONS
As described in Section 3.1, Algorithm 1 forms a set of sentences S , where each word is actually a node from the graph.
We train the vector representations of nodes by maximizing the log probability of predicting a node given another node in a sentence and given a set of representations r .
We use a window of size k which is equal to the size of the generated sentences, so that each node in a sentence predicts all the others.
The log probability maximized by NBNE is given by:
[Equation]
where p ( s | r ) is the probability of each sentence, which is given by:
[Equation]
where v i is a vertex in the graph and v j are the other vertices in the same sentence.
The probabilities in this model are learned using the feature vectors r v i , which are then used as the vertex representations.
The probability p ( v j | v i , r ) is given by:
[Equation]
where r (cid:48)
Tv j is the transposed output feature vector of vertex j , used to make predictions.
The representations r (cid:48) v and r v are learned simultaneously by optimizing Equation (1).
This is done using stochastic gradient ascent with negative sampling (Mikolov et al., 2013b).
By optimizing this log probability, the algorithm maximizes the predictability of a neighbor given a node, creating node embeddings where nodes with similar neighborhoods have similar representations.
Since there is more than one neighbor in each sentence, this model also makes connected nodes have similar representations, because they will both predict each others neighbors, resulting in representations also having some ﬁrst order similarities.
A trade-off between ﬁrst and second order proximity can be achieved by changing the parameter k , which simultaneously controls both the size of sentences generated and the size of the window used in the SkipGram algorithm.
A further discussion on this effect can be seen in Appendix B.3.
3. A VOIDING O VERFITTING R EPRESENTATIONS
When using large values of n (i.e., number of permutations) on graphs with few edges per node, some overﬁtting can be seen on the representations, as shown in details in Section 5.1 and in Appendix B.2.
This overﬁtting can be avoided by sequentially training on increasing values of n and testing the embeddings on a validation set every few iterations, stopping when performance stops improving, as shown in Algorithm 2.
1: procedure TRAIN NBNE(graph, max n) 2: sentences ← get sentences ( graph, max n ) 3: model ← [ initialize model() ] 4: for j in 0 : log 2 ( max n ) do 5: model ← train ( model, sentences [2 j : 2 j +1 ]) 6: error ← test ( new model, validation set ) 7: if error > old error then 8: break
E XPERIMENTS
NBNE was evaluated on two different tasks: link prediction, and node classiﬁcation.
2 We used a total of six graph datasets to evaluate NBNE and Table 1 presents details about these datasets.
A
[Table]
brief description of each dataset can be found in Appendix A and an analysis of their assortativity properties in Appendix B.1.
We present results for the link prediction problem in Section 4.1 and for the node classiﬁcation problem in Section 4.2.
For all experiments we used sentences of size k =and embeddings of size d = 128 , while the number of permutations was run for n ∈ { 1 , 5 , 10 } .
The best value of n was chosen according to the precision on the validation set and we used early stopping, as described in Section 3.3.
On both these tasks, DeepWalk and Node2Vec were used as baselines, having been trained and tested under the same conditions as NBNE and using the parameters as proposed in (Grover & Leskovec, 2016).
More speciﬁcally, we trained them with the same training, validation and test sets as NBNE and used a window size of 10 ( k ), walk length ( l ) of 80 and 10 runs per node ( r ).
For Node2Vec, which is a semi-supervised algorithm, we tuned p and q on the validation set, doing a grid search on values p, q ∈ { 0 . 25; 0 . 5; 1; 2; 4 } .
We also evaluated NBNE on two synthetic graphs with different sizes and sparseness, which can be seen on Appendix C, and an author name disambiguation task, on Appendix F. A comparison between NBNE and SDNE can be seen on Appendix D.
4. L INK P REDICTION
Setup.
Link prediction attempts to estimate the likelihood of the existence of a link between two nodes, based on observed links and the nodes’ attributes (L¨u & Zhou, 2011).
Typical approaches to this task use similarity metrics, such as Common Neighbors or Adamic-Adar (Adamic & Adar, 2003).
Instead of these hand made similarity metrics, we propose to train a logistic classiﬁer based on the concatenation of the embeddings from both nodes that possibly form an edge and predict the existence or not of the edge.
To train NBNE on this task, we ﬁrst obtained a sub-graph with 90% randomly select edges from each dataset, and obtained the node embeddings by training NBNE on this sub-graph.
We, then, separated a small part of these sub-graph edges as a validation set, using the rest to train a logistic regression with the learned embeddings as features.
After the training was completed, the unused 10% of the edges were used as a test set to predict new links.
10-fold cross-validation was used on the entire training process to access the statistical signiﬁcance of the results, analyzing statistical difference between the baselines and NBNE.
To evaluate the results on this task, we used as metrics: AUC (area under the ROC curve)
(Baeza-Yates & Ribeiro-Neto, 2011), and training time.
The logistic regressions were all trained and tested using all available edges (respectively in the training or test set), and an equal sized sample of negative samples, which, during training, included part of the 10% removed edges.
Results.
Table 2 presents results for this task.
Considering AUC scores on the Link Prediction task,
NBNE was statistically better 4 than both DeepWalk and Node2Vec on the Astro and PPI datasets, with more than 7% improvement, also showing a 4.67% performance gain in Wikipedia and a small, but statistically signiﬁcant, gain on Blog.
Only losing by a small percentage on Facebook, with a difference that was not statistically signiﬁcant
In all experiments we performed t-tests with p = 0 .
The symbol ∗ marks results which are statistically different from NBNE.
[Table]
In DBLP
[Table]
NBNE again presents the best AUC score, although this difference was small and its statistical signiﬁcance could not be veriﬁed due to the large training times of the baselines.
This dataset contains the largest graph analyzed in this work ( 317 , 080 nodes and 1 , 049 , 866 edges) and in it, to train a single fold, Node2Vec took 3 , 285 m59s
(more than 54 hours)
and DeepWalk took 164m34s (approximately 2 hours and 44 minutes), while NBNE took only 14m30s, which represents a 226/11 times improvement over Node2Vec and DeepWalk, respectively.
Considering training time for this task, NBNE presents the biggest improvements on sparser networks of medium size, like Astro, PPI and Wikipedia datasets.
On these graphs, the best results are for n = 1 , resulting in more than 50x faster training than DeepWalk and more than 1 , 500 times faster than Node2Vec, achieving a 6 , 049 times faster training than Node2Vec on Wikipedia.
For the
Blog and Facebook datasets the best results are for n = 5 , resulting in larger training times, but still more than one order of magnitude faster than DeepWalk and more than 350 times faster than
Node2Vec.
For the DBLP dataset, the best results were achieved with n = 10 , still much faster than the baselines.
4. N ODE C LASSIFICATION
Setup.
Given a partially labeled graph, node classiﬁcation is the task of inferring the classiﬁcation of the unknown nodes, using the structure of the graph and/or the properties of the nodes.
In this task, the node embeddings were trained using NBNE on the complete graph.
After obtaining the node embeddings, 80% of the labeled nodes in the graph were used to train a logistic classiﬁer that predicted the class of each node, while 5% of the nodes were used for validation and the remaining 15% nodes were used as a test set.
This test was repeated for 10 different random seed initializations to access the statistical relevance of the results.
Results.
Results on the Blog, PPI and Wikipedia datasets are shown in Table 3 and are presented in terms of Macro F1 scores and training times.
NBNE produces statistically similar results to its baselines, in terms of Macro F1, on both PPI and Wikipedia, while showing a statistically signiﬁcant 22.45% gain in the Blog dataset, indicating that NBNE’s embeddings did not only get a better accuracy on Blog, but also that correct answers were better distributed across the 39 possible classes.
Considering training times, NBNE is more than 10 times faster than DeepWalk on these three datasets and is [300 ∼ 600] times faster than Node2Vec.
NBNE didn’t show statistically worse result in any dataset analyzed here 5 , while having an order of magnitude faster training time than
DeepWalk and more than two orders of magnitude faster training time than Node2Vec.
[Table]
F URTHER A NALYSIS 5. N UMBER OF P ERMUTATIONS ( n )
The quality of NBNE’s embeddings depends on both the size of the embeddings ( d ) and the number of permutations ( n ).
For highly connected graphs, larger numbers of permutations should be chosen ( n ∈ [10 , 1000] ) to better represent distributions, while for sparser graphs, smaller values can be used ( n ∈ [1 , 10] ).
[Figure]
Figure 1 shows test accuracies versus embedding sizes for several values of n on the Facebook link prediction task.
Quadratic functions approximating log( test accuracy ) were plotted to allow for a better understanding of the results.
It can be seen that for large embeddings ( d > 1000 ) a larger number of permutations result in better accuracies.
The plot also shows that n = 10 gives fairly robust values for all tested embedding sizes.
A further analysis can be seen in Table 4, which shows that graphs with more edges per node tend to get better results with larger values of n , while graphs with a smaller branching factor have better results with only one permutation ( n = 1 ).
Other factors also enter into account when choosing n , like graph size.
For example, link prediction on the DBLP graph had its best results for n = 10 , although its branching size was only 3.31.
Further experiments on this parameter can be seen in Appendices B.2 and
C.1.
[Table] [Table]
5. T IME C OMPLEXITY
SkipGram’s time complexity is linear on the number of sentences, embedding size ( d ) and logarithmic on the size of the vocabulary (Mikolov et al., 2013a).
Since the number of sentences is linear on the number of permutations ( n ), branching factor of the graph ( b ) and on the number of nodes,
[Figure]
which is the size of the vocabulary ( | V | )
[Figure]
the algorithm will take a time bounded by:
[Equation]
Figure 2 (Top-Left and Top-Right) show training time is indeed linear on both embedding size and number of permutations.
It also shows that Node2Vec is considerably slower than DeepWalk
[Figure]
and only has a similar training time to running NBNE with at least n = 1000 .
NBNE with n < 10 was by far the fastest algorithm
[Figure]
NBNE, Node2Vec and DeepWalk run in a time bounded by O ( | V | log | V | ) , as can be seen in Figure 2 (Bottom-Left).
Figure 2 (Bottom-Right) shows that NBNE’s time complexity is linear in the branching factor b , while Node2Vec’s is quadratic.
DeepWalk’s running time is constant in this parameter, but for a graph with a larger branching factor, a higher number of walks per node should be used to train this algorithm, which would make it indirectly dependent on this factor.
C ONCLUSIONS
The proposed node embedding method NBNE shows results similar or better than the state-of-theart algorithms Node2Vec and DeepWalk on several different datasets.
It shows promising results in two application scenarios: link prediction and node classiﬁcation, while being efﬁcient and easy to compute for large graphs, differently from other node embedding algorithms, such as LINE (Tang et al., 2015) or SDNE (Wang et al., 2016).
NBNE focuses learning on node’s immediate neighbors, creating more ego-centric representations, which we suspect makes them more stable and faster to learn.
Empirical results show that, although it has a similar time complexity, NBNE can be trained in a fraction of the time taken by DeepWalk (10 to 190 times faster) or Node2Vec (200 to 6 , 000 times faster), giving fairly robust results.
Since embeddings are learned using only a node’s immediate neighbors, we suspect it should also be easier to implement more stable asynchronous distributed algorithms to train them, and we leave this as future work.
A D ATASETS
We used a total of six graph datasets to evaluate NBNE, with DeepWalk and Node2Vec being used as baselines.
Next we brieﬂy describe these datasets:
Facebook (McAuley & Leskovec, 2012):
A snapshot of a subgraph of Facebook, where nodes represent users and edges represent friendships.
Astro (Leskovec et al., 2007):
A network that covers scientiﬁc collaborations between authors whose papers were submitted to the Astrophysics category in Arxiv.
Protein-Protein Interactions (PPI) (Breitkreutz et al., 2008):
We use the same subgraph of the PPI network for Homo Sapiens as in (Grover & Leskovec, 2016).
This subgraph contains nodes with labels from the hallmark gene sets (Liberzon et al., 2011) and represent biological states.
Nodes represent proteins, and edges indicate biological interactions between pairs of proteins.
Wikipedia (Mahoney, 2011): A co-occurrence network of words appearing in the ﬁrst million bytes of the Wikipedia dump.
Labels represent Part-of-Speech (POS) tags.
Blog (Zafarani & Liu, 2009): A friendship network, where nodes are bloggers and edges are friendships between them.
Each node in this dataset has one class which is referent to the blogger’s group.
DBLP (Yang & Leskovec, 2012):
A co-authorship network where two authors are connected if they published at least one paper together.
Figure 3 shows the distribution of classes in the three datasets used for node classiﬁcation.
While
Wikipedia has a long tailed distribution, with one class being present in almost 50% of its nodes,
PPI’s probabilities are well distributed along the 49 different possible classes.
[Figure]
datasets with classes sorted by their frequency.
B A SSORTATIVITY
Assortativity, also referred to as homophily in social network analysis, is a preference of nodes to attach themselves to others which are similar in some sense.
In this section, we further investigate assortativity properties related to both the representations generated by our algorithm, as of the graphs themselves.
In Section B.1, we do a quantitative analysis on the homophily inherent to the datasets considered in this work.
In Section B.2, we make a qualitative analysis of how assortativity varies depending on the number of permutations n .
In Section B.3, we make a qualitative analysis on the trade-off of ﬁrst and second order proximities based on the choice of k .
B. D ATASETS ’ H OMOPHILY
There are several ways to quantitatively capture the homophily present in a graph.
Jensen & Neville describe relational auto-correlation, which is Pearson’s contingency coefﬁcient on the characteristics of nodes which share edges (Jensen & Neville, 2002; Neville & Jensen, 2007).
Park & Barab´asi (2007) deﬁne dyadicity and heterophilicity, which respectively measure how a graph’s nodes share common/different characteristics in edges, compared to a random model.
Table 5 presents both degree and label assortativity properties of the six graphs analysed here, calculated using the deﬁnition of Newman (2003).
We can see in this table that the datasets analyzed in this work cover a broad specter of assortativity properties.
PPI, Wikipedia and Blog graphs present negative degree assortativity, which means nodes in these graphs are more likely to connect with nodes of different connectivity degrees.
At the same time, Facebook, Astro and DBLP present positive degree assortativity, which indicates that their nodes tend to connect to others with similar degrees.
[Table]
We also analyze graphs with both positive and negative label assortativity in our label classiﬁcation task.
While PPI and Blog datasets present positive label assortativity, with connected nodes more frequently sharing classes, Wikipedia has a negative assortativity, with its connected nodes being more likely to have different classes.
B. M ORE ON THE N UMBER OF P ERMUTATIONS ( n )
Here, we further analyze how the number of permutations ( n ) inﬂuences both homophily and overﬁtting in our learned representations.
We qualitatively measure homophily by comparing either cosine or euclidean distances between nodes on edges to the distances in non-edges.
The cosine distances for the PPI dataset, shown by the box plots in Figure 4 (top-left), clearly show for larger values of n how the embeddings overﬁt to the speciﬁc graph structure, with the learned similarity on edges not generalizing to the links which were previously removed.
In this graph, for larger numbers of permutation the removed edges have a distribution more similar to the non edges than to the edges used during training, which is a tendency that can be observed in the other graphs, although in a smaller scale.
The box plots in Figure 4 (top-right) show the cosine distance for Facebook nodes.
We can see that for n = 5 there is a larger separation between removed edges and non edges, which justiﬁes the algorithm’s choice of this value.
For larger values of n we can again see an overlap between the distributions, caused by the embeddings overﬁtting.
On the other hand, the cosine distances for the
DBLP in Figure 4 (bottom-left) show the largest separation for n = 10 .
Finally, the box plots in Figure 4 (bottom-right) show cosine distances for the Blog dataset.
We can see that for n = 1 and n = 5 there is actually a larger cosine distance between nodes in removed edges than in non edges, with this situation only inverting for n ≥ 10 .
This happens due to this graph’s negative degree homophily.
This is also observed for similar graphs in the PPI and
Wikipedia datasets, though with a smaller intensity in the PPI graph, which has a smaller absolute value of degree assortativity and where only embeddings for n = 1 present this property.
The box plots from Figure 4 further support our intuition that graphs with larger branching factors should have larger values of n .
At the same time, this choice also depends on the graph size and structure, as shown by the algorithms choice of n = 10 for the DBLP dataset, which contains the largest degree assortativity.
The best choice of n depends on the analyzed task, but we believe that, at least for link prediction, this choice is both directly proportional to a graph’s branching size and degree assortativity.
Nonetheless, the difﬁculty in analyzing these graphs supports our choice for a semi-supervised approach, automatically choosing n on a per graph instance
[Figure]
Considering again the experiment on the PPI dataset with the number of permutations n = 1 in
Figure 4 (top-left), in Figure 5 we present in detail the euclidean distances between nodes that share or not an edge for this number of permutations.
We can see that the distribution of removed edges is a lot closer to the edges used for training than to the non edges.
[Figure]
B. T RADE - OFF BETWEEN FIRST AND SECOND ORDER PROXIMITY
The window size and the number of neighbors in a sentence are both adjusted by a single variable, k , and this variable also controls a trade-off between ﬁrst and second order proximities in the node embeddings.
This can be explained intuitively by analyzing both the sentence sampling method in
Algorithm 1 and Equations 1, 2 and 3, in Section 3.2.
When a smaller k is chosen, each node’s embedding r v i will be mainly used to predict its own neighbors.
This causes nodes with shared neighbors to have closer representations (second order proximity).
When larger values of k are chosen, nodes will appear more often in its neighbors sentences, and will predict not only its own neighbors, but his neighbors’ neighbors.
This will result in connected nodes to have more similar embeddings, increasing ﬁrst order similarity.
We further analyze this by examining the distribution of cosine distances between nodes at different graph distances.
For this analysis, we use three different synthetic graphs: Barab´asi-Albert (Barab´asi & Albert, 1999); Erd˜os-R´enyi (Erdos & R´enyi, 1960); Watts-Strogatz (Watts & Strogatz, 1998).
We choose these graphs because of their structural differences, believing they cover an ample specter of different graphs’ properties.
These graphs were created with | V | = 2000 and b = 20 , and
Watts-Strogatz graphs had a probability β = 0 .
2 of generating non-lattice edges.
To train our representations we used n = 10 and d = 128 .
Figure 6 shows box plots of these cosine distances of nodes’ representations versus their graph distance on these different artiﬁcial random graphs.
In this ﬁgure, we can see that, for both Barab ` asiAlbert and Erd˜os-R´enyi graphs, when using a sentence size ( k ) equal to 1 , the cosine similarity is larger for nodes which are two steps away than for nodes which share an edge (second order proximity), while for larger values of k , nodes which share an edge have larger similarity (ﬁrst order proximity).
[Figure]
The box plots in Figure 6 also show that the difference in similarity increases with the value of k .
The larger the value of k , the larger the difference between similarities of nodes which share an edge and nodes with larger distances, as can be seen in detail in Figure 7 for the Barab´asi-Albert graph.
C G RAPH S IZE AND S PARSENESS A NALYSIS
In this section, we analyze how a graph’s sparseness (represented here by its branching factor) and size (represented here by its number of vertices) affect the choice of the number of permutations ( n ) and of the window/sentence size ( k ).
With this purpose we ran several link prediction experiments on two different synthetic graphs: Watts-Stogratz and Barab´asi-Albert.
These graphs were generated for different sizes ( | V | ) and sparseness ( b ), and we ran experiments with the same setup as in Section 4.1, with Watts-Stogratz graphs having again β = 0 .
7 Section C.1 presents this analysis for the they struc its removed presented in this section are ten a single instance of graph size.
[Figure]
Barabasi
[Figure]
Albert (|V|=2000, b=20)
[Figure]
=
[Figure]
=
[Figure]
k=
[Figure]
of
C. N UMBER OF P ERMUTATIONS ( n )
We analyze, in this section, how a graph’s size and sparseness affect the choice of the number of permutations ( n ), for both Watts-Stogratz and Barab ´ asi-Albert graphs.
Analyzing the graphs in
Figure 8, we see a correlation between the best choice of n and a graph’s number of vertices ( | V | ) and branching factor ( b ).
In Figure 8a
[Figure]
which contains the experiments in the most sparse graphs, results for n = 1 are better for all graph sizes.
A random algorithm would return an AUC score of 0.5, so results bellow this value clearly expose a problem in the learning algorithm.
This is the case for both n = 10 and n = 5 in these graphs, which overﬁt its representations
[Figure]
In Figure 8b we can see that, when considering a graph with a branching size of b = 4 , for smaller graphs a smaller value of n is preferable, while for larger graphs a larger number of permutations gives better results ( n = 10 ).
In Figure8c we can see that, for a branching size of b = 8 , results for larger values of n are always better than for n = 1 .
Notice also that, while results for b = 2 and b = 4 were around 0 .
55 ∼ 0 .
7 , results for b = 8 are closer to 0 .
9 , showing that this algorithm is better at learning with more information.
Our experiments in link prediction using synthetic Barab´asi-Albert graphs present slightly more complex results.
Figure 9 shows that for smaller branching factors ( b ≤ 8 ), n = 1 indeed generate better results for small graphs, but for larger graphs, a larger number of permutations is necessary.
For intermediary branch sizes the best value of n is harder to determine, and only for b = 64 we start to see a tendency of larger number of permutations consistently giving better results.
[Figure]
We can also see from Figure 9 that edges in Barab ´ asi-Albert graphs are considerably more difﬁcult to predict, specially for smaller branching sizes.
Most of our results are around 60% and our best
AUC scores in these graphs are around 70% .
Again, n ’s dependency on these graph properties ( | V | and b ) depends highly on the graph’s structure, further supporting our choice of a semi-supervised approach, choosing n on a per graph instance by validating results on a small validation set.
This can be considered as a form of early stopping when training these node embeddings.
C. W INDOW AND S ENTENCE S IZES ( k )
In this section, we again use Watts-Stogratz and Barab ´ asi-Albert graphs, this time to analyze how a graph’s size and sparseness affect results for different window and sentence sizes ( k ) in our model.
For these experiments we keep n = 5 ﬁxed.
Figure 10a shows that, for a small branching factor ( b = 2 ), all choices of k clearly overﬁt for WattsStrogatz graphs, but k = 5 overﬁts less than larger choices of k .
For b = 8 , k = 5 produces slightly better results in these graphs, while larger values of k produce better results for a larger branching size ( b = 32 ).
[Figure]
Barab´asi-Albert graphs’ edges are considerably harder for our algorithm to predict, as shown in the previous section, so we only report results for larger values of b (the algorithm, with our choice of hyper-parameters, overﬁts for smaller values).
We can see from Figure 11 that larger values of k usually produce better results for this graph, but are more propense to overﬁt, specially when being applied to larger sparse graphs ( | V | ≥ 800 and b = 16 ).
[Figure]
Further analysis on the representations’ properties for different values of k could provide better motivation on its choice, but we leave this to future studies, keeping our choice of k = 5 constant in this work.
Studying if geometric operations between representations have comprehensible meanings would also be interesting, such as was done for Word2Vec algorithms, but this is also left as future work.
D C OMPARISON WITH SDNE
Structural Deep Network Embedding (Wang et al., 2016) is another algorithm used for learning node embeddings in graphs.
As described in Section 2, SDNE is based on ﬁrst and second order proximities, using autoencoders to learn compact representations based on a node’s adjacency matrix (second-order proximity), while forcing representations of connected nodes to be similar (ﬁrst-order proximity) by using an hybrid cost function.
This algorithm has a time complexity of O ( | V | 2 ) , but its main computation, which is calculating the gradients of its cost function and updating model parameters, can be highly parallelized by using modern GPUs and Deep Learning frameworks.
In this section, we compare NBNE and SDNE in terms of both efﬁciency and efﬁcacy, analysing both AUC/Macro F1 scores and training time.
With this objective, we trained SDNE embeddings using both a dedicated K40 GPU with CUDA 8.0 and a dedicated 16 core linux server.
In their original work, SDNE was run in a semi-supervised setting, ﬁnding the best value of α , β and ν by tuning them on a small validation set.
In this work we ﬁx α = 0 .
2 and β = 10 , since in their work they state that these values commonly give the best results, while only choosing ν in a semi-supervised manner.
We use SDNE’s architecture with [10 , 300; 1 , 000; 128] nodes on each layer and test it on both Link Prediction and Node Classiﬁcation tasks, using the same steps described in
Sections 4.1 and 4.2.
We train these embeddings using ν ∈ { 0 . 1 , 0 . 01 , 0 . 001 } and choose the best value on the same validation sets used to tune n for NBNE and p and q for Node2vec.
Table 6 shows results using both NBNE and SDNE embeddings on Link Prediction tasks.
In this table we can see that both algorithms produce similar results in terms of AUC scores, with each having a statistically signiﬁcant better result on two datasets, and NBNE having a non statistically signiﬁcant, but slightly better result on the ﬁfth.
It is clear that even when training SDNE using a
K40 GPU, NBNE still has more than an order of magnitude faster training time on all datasets, being more than two orders of magnitude faster on most.
When comparing to SDNE trained on a CPU,
NBNE has more than three orders of magnitude faster training time.
On Astro, the dataset with the largest number of nodes analyzed here, NBNE had a 2 , 009 times faster training time compared to
SDNE on a GPU and 44 , 896 times faster compared to SDNE on CPU.
[Table]
Table 7 shows the results of running NBNE and SDNE on the Node Classiﬁcation task.
On this task
NBNE gave statistically better results on two datasets, with an impressive gain of 29 .
27% on PPI and 45 .
39% on Blog, only losing on Wikipedia with an also large gain of − 20 .
We can again see that NBNE has a more than an order of magnitude faster training time than SDNE on a GPU in this dataset, being more than two orders of magnitude faster when SDNE is trained on a CPU.
[Table]
Analyzing both these tables we can also see that the largest gains in training time occur when using
NBNE on a large but sparse network, such as Astro.
This agrees with our theoretical expectations, since SDNE’s time complexity grows quadratically with the number of nodes
O ( | V 2 | ) and NBNE’s grows with O ( | V | · log ( | V | ) · b ) , which is close to linear on the number of nodes for large graphs.
E F ULL R ESULTS
In this section, we extend the results presented in Section 4, considering now the precision on the training and test sets.
We present results for the link prediction problem in Table 8 and for the node classiﬁcation problem in Table 9.
NBNE produces statistically similar results to its baselines, in terms of Macro F1, on the node classiﬁcation task using both PPI and Wikipedia datasets, while showing a statistically signiﬁcant [Table] 22.45% gain in the Blog dataset.
Node classiﬁcation results in the PPI dataset had the smallest precision among all datasets, with only approximately 14%, but there are 49 classes in it and a classiﬁer which always guessed the most common class would only get 2.95% precision.
NBNE only shows a statistically worse result in test precision for node classiﬁcation on the
Wikipedia dataset, losing to DeepWalk, but having an order of magnitude faster training time than
DeepWalk and more than two orders of magnitude faster training time than Node2Vec.
On all other experiments in either node classiﬁcation or link prediction it presented either statistically better or similar results to its baselines, while showing much faster training times. [Table]
F A UTHOR N AME D ISAMBIGUATION
One of the hardest problems faced by current scholarly digital libraries is author name ambiguity (Ferreira et al., 2012).
This problem occurs when an author publishes works under distinct names or distinct authors publish works under similar names (Ferreira et al., 2015).
Automatic solutions, which are effective, efﬁcient and practical in most situations, are still in need (Santana et al., 2014).
In this section [Table] we test our algorithm against the case where distinct authors publish works under similar names.
F. E XPERIMENTAL S ETUP
For this problem [Table] the DBLP repository was crawled and the proﬁles of fourteen of the most proliﬁc ambiguous authors were obtained, together with their direct co-authors’ complete proﬁles.
With this
[Table]
we created a new dataset, called here DBLP-ambiguous, consisting of fourteen separate coauthorship networks (14 separate graphs), each with all the connections of one of these homonymous authors and their co-authors’ connections.
Details on these graphs can be seen in Table
[Table]
[Table]
Using these co-authorship networks, embeddings were obtained by training on the graphs with 20% of the papers from each ambiguous author removed.
After the embeddings had already been learned for each author, the probability of each possible author-coauthors “sentence” was calculated as:
[Equation]
possible [Equation] = [ v possible
[Equation]
, v coauthor [Equation] [Equation] v coauthor j ]
This probability is given by:
[Equation]
where v 1 = author , which comes from the NBNE model itself.
As a baseline, we used the typical solution that classiﬁes the closest of the possible ambiguous authors as co-author for each of the test papers.
If no path on the graph existed to any of the possible ambiguous authors, or if there was a tie between the distances to two or more of them, a random one was chosen between the possible ones.
DeepWalk and Node2Vec were not used as baselines for this task due to the size of the 14 graphs analyzed here, most with more than 100 , 000 nodes and 500 ,edges, which would result in a prohibitive training time.
F. E XPERIMENTAL R ESULTS
Table 11 presents the results for the author name disambiguation task for each chosen author.
This experiment was run using NBNE as an unsupervised algorithm with a ﬁxed number of permutations n = 10 , having no validation set.
We also used sentences of size k = 5 and node embeddings of size d = 128 . [Table]
After the embeddings had already been learned for each author [Table] which can be done off-line [Table]
the NBNE algorithm was faster in assigning the authors than its baseline.
This occurred because it only required computing the probability of each possible author-coauthors “sentence” ( p ( s ) ), while the baseline had to dynamically get the distance between the papers’ co-authors and the possible authors.
It can be seen in Table 11 that for all but two authors the precision was higher when using the NBNE embeddings instead of the graph baseline, while for the other two precision score remained the same.