BCDR:
B ETWEENNESS
C
ENTRALITY - BASED D
ISTANCE
R ESAMPLING
FOR
G RAPH
S HORTEST
D ISTANCE E MBEDDING
Anonymous authors
Paper under double-blind review
A BSTRACT
Along with unprecedented development in network analysis such as biomedical structure prediction and social relationship analysis, Shortest Distance Queries (SDQs) in graphs receive an increasing attention.
Approximate algorithms of
SDQs with reduced complexity are of vital importance to complex graph applications.
Among different approaches, embedding-based distance prediction has made a breakthrough in both efﬁciency and accuracy, ascribing to the signiﬁcant performance of Graph Representation Learning (GRL).
Embedding-based distance prediction usually leverages truncated random walk followed by Pointwise
Mutual Information (PMI)-based optimization to embed local structural features into a dense vector on each node and integrates with a subsequent predictor for global extraction of nodes’ mutual shortest distance.
It has several shortcomings.
Random walk as an unstrained node sequence possesses a limited distance exploration, failing to take into account remote nodes under graph’s shortest distance metric, while the PMI-based maximum likelihood optimization of node embeddings reﬂects excessively versatile local similarity, which incurs an adverse impact on the preservation of the exact shortest distance relation during the mapping from the original graph space to the embedded vector space.
To address these shortcomings, we propose in this paper a novel graph shortest distance embedding method called Betweenness Centrality-based Distance Resampling (BCDR).
First, we prove in a statistical perspective that Betweenness
Centrality(BC)-based random walk can occupy a wider distance range measured by the intrinsic metric in the graph domain due to its awareness of the path structure.
Second, we perform Distance Resampling (DR) from original walk paths before maximum likelihood optimization instead of the PMI-based optimization and prove that this strategy preserves distance relation with respect to any calibrated node via steering optimization objective to reconstruct a global distance matrix.
Our proposed method possesses a strong theoretical background and shows much better performance than existing methods when evaluated on a broad class of realworld graph datasets with large diameters in SDQ problems.
I NTRODUCTION
Shortest Distance Queries (SDQs) in graphs focus on determining the minimum distance between arbitrary node pairs, deﬁned as sum of related edge weights.
Despite the fact that Shortest Path
Queries (SPQs) (Wei, 2012; Sommer, 2014; Wang et al., 2021) are already renowned in graph structure exploration, SDQs play an essential role in increasing applications, such as social relationship analysis (Carlton, 2020; Melkonian et al., 2021;
Chen et al., 2021; Zaki et al., 2021; Parveen &
Varma, 2021), biomedical structure prediction (Yue et al., 2019; Galoviˇcov´a et al., 2021; Sokolowski & Wasserman, 2021), learning theory (Yang et al., 2021; Yuan et al., 2021), optimization (Melkonian et al., 2021; Rahmad Syah et al., 2021; Jiang et al., 2021), etc.
The key challenge in the SDQ problem is the prohibitive complexity in very large graphs.
e.g., for an undirected dense graph with N nodes and k queries, the time complexity of A*(Hart et al., 1968) and Dijkstra Algorithm(Thorup &
Zwick, 2004) are up to O ( kN 2 ) and O ( kN log N ) for unweighted and weighted graph, respectively.
Accuracy loss of approximate methods are evaluated on facebook(A.7.2) by mRE(2).
N -number of nodes, T -walk length, K -window size, l -number of landmarks, d -dimension of embedding space, α, β, c -constant independent of N .
[Table]
[Figure]
[Table]
[Figure]
to
: Weight decay on walks causes instability of shortest distance since walks rooted at v a have large probability to steer clear of v b for starters and back to v b as the end, which results in a extremely weak correlation between v a and v b despite the fact that they have an immediate edge.
: A sufﬁcient number of 2-hop links between v c and v a induce a shorter distance in embedding space than that of v b and v a .
: v b and v c sharing substantial connection deserve to be mapped closed to each other even if they have a large shortest distance gap, while the divergence of distance between v b , v c and v a is also plagued with extraction.
To address this issue, a surging number of approximate algorithms in a wide range of ﬁelds have been proposed in the past a few years.
They can be categorized into oracle-based (Thorup & Zwick, 2004;
Baswana & Kavitha, 2006), landmark-based (Sarma et al., 2010; Gubichev et al., 2010), embeddingbased distance prediction methods (Xiaohan et al., 2010; 2011; Rizi et al., 2018).
Among these categories, embedding-based methods are of high accuracy with signiﬁcantly reduced complexity (see Table1), owing much to rapid advances of representation learning in graphs (Perozzi et al., 2014; Grover & Leskovec, 2016).
Embedding-based methods tackle the SDQ problem with two stages.
First, they embed local structural features into a dense vector on each node, which preserves the essential information for indicating where the node is.
Then, for arbitrary node pairs in the training set, subsequent predictor extracts mutual shortest distance globally via their embedding tuples and minimizes the mean of square loss between a predicted value and the real one.
As predictors serve as a non-linear metric in the embedding space, the performance of embeddingbased models depends highly on the ﬁrst stage.
These methods usually embed graph structures by leveraging truncated random walk to serialize node’s neighborhood in a statistical perspective and maximize co-occurrence likelihood of nodes in one walk path to reﬂect their correlation, which is also proved to implicitly factorize a Pointwise Mutual Information (PMI) matrix (Levy & Goldberg, 2014; Shaosheng et al., 2015).
Although existing embedding-based methods have achieved a great success, they have several shortcomings.
On one side, random walk is an unstrained node sequence from the root, possessing a limited distance exploration.
This is because each transition on nodes is not implied for a speciﬁc direction to move towards or beyond the root, especially after several walk steps, which is backbreaking to accommodate correlation with remote nodes under graph’s shortest distance metric (see
Figure 1a).
On the other side, the PMI-based optimization reﬂects excessively versatile local sim- ilarity, which is not guaranteed for shortest distance-preserved mapping from the original graph space to the embedded vector space.
As a matter of fact, it exerts a too-general metric over nodes’ correlation, wherein the more links or paths exist between two nodes, the stronger correlation they share.
That means we have many ways to claim a shorter distance for two nodes (e.g., by adding mutual edges, deleting links to other nodes) even if some of the operations do not change their actual shortest path distance (see Figures 1c and 1d).
In this paper, we address the above shortcomings by proposing a novel graph shortest distance embedding method called Betweenness Centrality-based Distance Resampling (BCDR).
Here, random walk paths are simulated by considering nodes’ betweenness centrality on each transition for covering a wider distance range of nodes.
Then, a sampling process of nodes based on their mutual shortest distances is performed before optimization of co-occurrence likelihood to preserve pairwise distance relation during the mapping from graph to the embedding space.
We summarize our major contributions as follows
• We propose a Betweenness Centrality (BC)-based random walk for accommodating correlation with nodes of wider distance range under the intrinsic graph metric
(see Section 3.1).
To the best of our knowledge, there is no existing method that combines betweenness centrality and random walk for the graph representation.
We prove in a statistical perspective that the transition driven by nodes’ BC value tends to explore high order proximity of the root due to the awareness of local path structure.
• We propose a Distance Resampling (DR) strategy to preserve nodes‘ mutual shortest distance during the mapping from graph to the embedding space (see Section 3.2).
We prove that by sampling node sequences from original walk paths before maximum likelihood optimization instead of PMI-based optimization, the objective can be steered to reconstruct a global distance matrix, and for any calibrated node, it strictly preserves the shortest distance relation to other nodes on the graph.
• We evaluate BCDR and compare it with existing embedding-based methods with a broad class of real-world graph
datasets with divergent diameters .
BCDR shows a much better empirical performance in solving the SDQ problem than existing embedding-based methods (see Section 4).
N OTATION & P ROBLEM D EFINITION 2. N OTATION
G =
( V, E ) denotes an undirected graph, with V
= { v 1 , v 2 , ..., v | V | } being the set of nodes and E being the set of edges.
An edge e ij =
( v i , v j ) represents an undirected edge between nodes v i and v j .
A node v
i ’s neighborhood
N i is a set of nodes with an edge with v i , i.e., N i
= { v j | ( v i , v j ) ∈
E } .
We use Z | V |× d to represent an embedding matrix, where d is the embedding size.
For any matrix B , the symbol B i represents the i -th row of B , and B ij represents the element at i -th row and j -th column.
A truncated random walk W i rooted at node v i of length l is a random vector of (cid:104)W 1 i , W 2 i , · · · , W li (cid:105) , where W ki is a node chosen from the neighborhood of node W k − 1 i for k = 1 , ..., l , with W
i := v i .
P i is a ﬁnite set of walk paths sampled from W i , and W i is the multiset of nodes on the walk paths in P i .
2. P ROBLEM D EFINITION
An arbitrary path p of length l ∈ N on graph G is an ordered sequence of nodes ( v 1 , v 2 , ...v l +1 ) , where each node except the last one has an edge with the subsequent node, i.e., ( v i , v i +1 )
∈ E for 1 ≤ i ≤ l .
The shortest path ˚ p ij is one of the paths with the minimum length ˚ l between two nodes v i and v j , with the shortest distance D ij deﬁned as the length of ˚ p ij .
The global distance matrix D comprises { D ij } .
An embedding-based model learns two mappings F and G for structure embedding and mutual distance extraction, respectively, as follows.
[Equation]
A shortest distance query Q can be simply deﬁned as a set of node pairs, i.e., Q = { ( v 11 , v 12 ) , ( v 21 , v 22 ) , ..., ( v | Q | , 1 , v | Q | , 2 ) } .
For each node pair ( v i , v j ) , the model intends to ﬁnd out approximation ˆ D ij =
G ( F ( v i ) , F ( v j )) nearest to real D ij .
The commonly used metrics of approximation quality are mean of Relative Error (mRE) and mean of Absolute Error (mAE).
mRE is deﬁned as the relative loss of the prediction value with respect to the real value, while mAE measures the absolute gap between the prediction value and the real value, i.e.,
[Equation]
M ETHOD
Random walk as a serialization strategy of similarity measurement has been widely used in many graph representation and learning methods, aiming to model long-range dependency of nodes (Grover & Leskovec, 2016; Zhuang & Ma, 2018).
The optimization of maximizing cooccurrence likelihood of nodes on walk paths also yields an impressive performance in networkstructure extraction via approximating PMI matrix (Levy & Goldberg, 2014).
But in terms of the shortest distance representation of a graph, we contend that this intuitive approach has several limitations on the performance.
Consider a walk path p
= ( v a , v a 1 , v a 2 , · · · , v a l ) ∈ P a sampled from a general random walk W a from root node v a .
As an unstrained sequence of nodes, the distance measured along the walk path p is not consistent with that on the graph (see Figure 1b), i.e., for v a i , v a j ∈ p ,
[Equation]
where i and j are indexes of node v a i and v a j on p and 1 ≤ i, j ≤ l .
Using the aforementioned unstrained walk paths for maximizing co-occurrence likelihood incurs two problems.
Problem 1: Limited exploration range of walks .
Each transition on walks only considers a local structure of the current node, causing agnostic tendency to move towards or beyond the root node under the graph’s shortest-distance metric after a few steps (see Figure 1a).
Problem 2: Intractability of shortest distance on paths .
Distance measured on walk paths may not actually reﬂect the graph’s shortest distance because of the unbalanced number of links between different nodes (see Figure 1c and 1d).
In this section, we describe in detail our proposed method, which is a decent way of encoding shortest distances using BC-based random walk plus a distance resampling strategy, and present a theoretical analysis for its interpretability and efﬁciency.
More speciﬁcally, we ﬁrst present BCbased random walks and distance resampling, and then an algorithm to integrate them.
We provide an intuitive complexity explanation.
Additionally, we discuss the connection to Multidimensional Scaling (MDS) and graph structure decomposition in Appendix A.1 and A.2, respectively.
3. BC- BASED R ANDOM W ALK
Deﬁnition 1.
(Betweenness Centrality)
Deﬁne G =
( V, E ) as undirected graph.
v i , v s , v t are arbitrary nodes in the node set V .
σ st ( v i ) represents the number of shortest paths between v s and v t that pass v i , and σ st is the total number of shortest paths between v s and v t .
Then we say that
BC of v i is
[Equation]
To address Problem 1, we propose a BC-based random walk.
As deﬁned in Deﬁnition 1, BC( v i ) determines the probability of v i located on shortest paths of arbitrary node pairs.
Thus, we consider a node with large BC value vitally signiﬁcant to drive the walk to move away from root node, since it reveals a easy way of traveling to some nodes with minimal cost.
And to leverage this property, in
BC-based random walk W a =
(cid:104)W 1 a , W 2 a , ...
W ja , ... (cid:105) on node v a , we prefer choosing nodes with the largest BC values among their neighborhoods when simulating walk paths, i.e.,
[Equation]
where N i is the neighborhood of v i .
The following theorem, proved in Appendix A.3, indicates that a BC-based random walk tends to transit from N ( h ) a to N ( h +1) a , leading to a deeper exploration measured by the intrinsic graph’s shortest distance.
It also reveals that our method performs better when the number of ﬁnal nodes f h ( v a ) is larger or there are more links between ﬁnal nodes f h ( v a ) to connective nodes e h ( v a ) at any h -order neighborhood of v a .
Theorem 1.
Deﬁne N ( h ) a = { v j | D aj
= h } as a set of nodes that are h -hops away from v a and
N h ( v a )
= |N ( h ) a | as the number of nodes in the set.
Let the number of the nodes that have connection with nodes in N ( h +1) a (called connective nodes) be e h ( v a ) and the number of other nodes (called ﬁnal nodes) be f h ( v a ) .
The BC we use is an approximate value by considering only the shortest path of nodes within a range of k -hops locally.
Let p R
( N h ( v a ) → N h +1 ( v a )) represent the probability to transit from nodes of N h ( v a ) to N h +1 ( v a ) by a general random walk, and p B ( N h ( v a ) → N h +1 ( v a )) represent that by a BC-based random walk.
Let P R
( f h ( v a ) →
e h ( v a )) be a transition from f h ( v a ) nodes to e h ( v a ) .
Then, for any node v a in graph G and any h > 1 ,
[Equation]
where C ≥ 0 , and E ( v a ) is the eccentricity of v a , E ( v a )
= max v b ∈ G D ab .
3. D ISTANCE R ESAMPLING
To address Problem 2, we propose a Distance Resampling (DR) strategy, which is inspired by
Sampling-Importance-Resampling (SIR) method (Smith & Gelfand, 1992).
Let v x ( v 1 , v 2 ) be a random vector of the node tuple.
Deﬁne p ( v x ) as the joint distribution reﬂecting the real shortest distance of v 1 , v 2 on the graph.
We have
[Equation]
as a probabilistic objective for representing the shortest distance D 12 , where f is a normalized learnable density function determined by nodes’ mutual distance in the embedding space.
According to the rearrangement inequality, the above equation arrives at the maximum when f ( v 1 , v 2 ) varies consistent with p ( v 1 , v 2 ) , which means f indicates the shortest distance metric on the graph with respect to any v 1 , v 2 as p does.
Therefore, the critical issue is to extract accurate p ( v 1 , v 2 ) in a graph.
Let q ( v a ) be the joint distribution reﬂecting distance relation on sampled walks.
We try to leverage q on walk paths to approximate p .
Here, we adopt a simple but effective resampling by considering both their mutual distance D 12 as well as BC values.
Without loss of generality, let v i represent the root node of walks and v j be the j -th node on a walk path (i.e., sampled from W ji ).
Our objective is thus to sample from W i to generate a new random vector ˚ W i = (cid:104) ˚ W 1 i , ˚ W 2 i , ..., ˚ W ˚ li (cid:105) of length ˚ l .
Then, each node v j sampled from ˚ W ji can be described as a weighted resampling in original walk paths W i based on D ij and BC( v j ) , i.e.,
[Equation]
where α is a hyper-parameter of the weight decay coefﬁcient on paths.
Reminiscent of previous discussions on relating maximizing co-occurrence likelihood with matrix factorization, we demonstrate the connection between the above intuitive design and graph shortest distance metric.
As we perform distance resampling for walk paths, the objective to learn an optimal node embedding is therefore interpreted as
[Equation]
Thereinto, f ( Z i , Z j ) = log σ ( Z i Z Tj ) , p N ( v k | v i ) is the distribution of negative sampling.
In our algorithm, p N can be speciﬁed as a weighted random sampling over all occurred nodes in W i (simulated based on Equation 5) by occurrence frequency, i.e.,
[Equation]
where #( · ) is a counting function indicating the number of occurrence times of speciﬁed nodes, i.e., the cardinality of a sampled set.
Proposition 1.
Deﬁne G =
( V, E ) as an undirected graph.
Z | V |× d is the embedding matrix of V corresponding to maximizing likelihood objective E v x ∼ p
[ f ( Z i , Z j )] deﬁned in Equation 10.
p N ( v b | v a ) is the negative sampling distribution of v b from W a simulated by Equation 11.
Let the weight decay coefﬁcient in distance resampling be α , 0 < α < 1 .
Then, the inner product of embedding matrix corresponds to a global distance matrix, i.e. ZZ T
= ˆ D .
For any v a and any v b that is n -hops away from v a , the distance between them in the embedding space varies linearly with respect to distance n , namely,
[Equation]
where A is a constant independent of v b .
Proposition 1 is proved in Appendix A.4.
Proposition 1 indicates that optimization of Equation 10 conforms to reconstruct a global distance matrix where nodes far away from each other in the graph under shortest distance metric(i.e.
, large
D ab ) should be mapped with large distance in the embedding space(i.e.
, large ˆ D ab ).
We can also conclude that ˆ D ab varies linearly with respect to the distance n between two nodes, while A is a constant independent of v b but related to v a , which means when we ﬁx the source (or destination) node as v a , any destination (or source) node v b ’s distance with v a could be compared with each other (we call that distance is measurable with respect to calibrated node v a ).
Consider the preservation of shortest distance relation.
Some studies on metric learning (Hermans et al., 2017; Zeng et al., 2020) have revealed that a tuple of samples ( v a , v b , v c ) being easy to learn means if v b shares strong correlation with v a , the distance between v b and v a in the embedding space should be shorter than that of v c and v a .
With this property, we have the following theorem, which indicates that our method is consistent with distance relation under intrinsic graph metric.
This will be used in the prediction task described next.
Theorem 2.
Each symbol here follows the deﬁnition in Proposition 1.
Let D be a global distance matrix deﬁned on graph G and D ab be graph’s shortest distance between node v a and v b .
Then for any nodes v a , v b , v c ∈ G ,
[Equation]
The proof is presented in Appendix A.5.
3. A LGORITHM
Our BCDR algorithm is presented in Algorithm 1.
It contains three steps.
First, pre-computation of the BC value (line 21) is required for each node on the graph.
Second, for each node v i , we sample a batch of walk paths P i guided by the BC value according to Section 3.1 (line 1 to 20).
Meanwhile,
[Table]
distance relation of each node with v i is also recorded for subsequent resampling.
Third, for each node v i
[Table]
we resample nodes in W i by their distances with v i and BC values to formulate ˚ P i (line 25 to 34), which is optimized by maximizing pair-wise co-occurrence likelihood (line 37).
This embedding algorithm takes O ( w out l 2 out | V | log | V | +( w in l in + w out l out ) | V | ) of time complexity and
O (( w in l in + w out l out ) | V | ) of space complexity for any sparse graph (detailed analysis is presented in Appendix A.6).
The above result reveals a feasible way of reducing time complexity by just taking larger w in l in and smaller w out l out in BCDR.
This practice is applied in our experiments
[Table]
[Figure]
[Table]
E XPERIMENTAL E VALUATION
In this section, we evaluate our model’s performance with several graph datasets and compare it with conventional embedding-based methods for SDQ tasks.
4. D ATASETS
We test our model using three real-world graph datasets as well as some simulated datasets.
As this paper focuses on the shortest distance prediction of connected and undirected graphs, we make a trivial change for some directed graphs by simply eliminating the direction of each edge.
For graphs with multiple connected components, we repeatedly add one edge between two unconnected components until the whole graph is connected.
The graphs we use are of complex inner connections and divergent diameters.
The basic information of these datasets is outlined in Table 2 (columnsto 7).
The detailed information is presented in Appendix A.7.
We deﬁne T , V , and E as datasets for train, validation, and test, respectively.
Each of them is composed of ( v i , v j , D ij ) .
As simulations of the above sets in huge graphs have to be of O ( | V | ) or less complexity, in our experiments, we initially select a group S of nodes as sources, and for each source v i ∈ S , a destination set D with their shortest distances to the source is randomly collected.
Then, |S| × |D i | input node pairs are simulated.
The detailed setup for each dataset is presented in Table 2 (columns 8 to 11).
4. P ARAMETER S ETUP
We compare our method with conventional graph embedding models based on random walk as well as matrix factorization (see Appendix A.8).
We simulate 40 walks on each node for random walkbased methods, and each walk is truncated at a length of 40 .
Sliding window size and negative sampling size are ﬁxed at 20 and 5 , respectively, for each dataset.
Node2Vec takes hyper-parameter
[Table]
p = q = 1 .
Regularization r of GF is set to 1 . 0 .
DADL takes the Hadamard operator for embedding aggregation.
Like previous research (Zhuang & Ma, 2018), linear regression is utilized as a predictor for general graph embedding models.
For our model, we take the same conﬁguration as previous random walk-based models using input walk length l in = 40 and input sample size w in = 40 .
In order to keep complexity of parameters equal to or even less than that of baselines, the multiplication of output length l out and output sample size w out is ﬁxed at 40 , with l out = 10 , w out = 160 for Cora and GrQc and l out = 40 , w out =for Facebook.
Meanwhile, the weight decay coefﬁcient α is set to 0 .
1 for Cora and GrQc datasets and 0 .
98 for Facebook.
To avoid dramatically unbalanced values of BC, we also take smoothed normalization ( τ = True ) in all datasets except Facebook.
Finally, all of the above models use the
Adam optimizer with a learning rate (cid:15) r = 1 e − 4 for training, and embedding dimension d is ﬁxed at 16 for every dataset.
4. P ERFORMANCE OF P REDICTION A CCURACY
To compare embedding-based methods, we train each model on each dataset up to 500 iterations and save the results every 10 epochs.
The best models are selected by their mAE and mRE scores on validation set V .
Each model is evaluated 5 times independently on each dataset, and their average performance is recorded.
The resulting mAE and mRE are reported in Table 3.
Some comparisons with the latest GRL methods and further discussion are presented in Appendix A.9, and run-time test compared with general random walk is reported in Appendix A.10.
We can see from the table that our model outperforms previous models signiﬁcantly for all three datasets.
4. R ESULT OF E XPLORATION D ISTANCE
As stated in Section 3.1, exploration distance under intrinsic graph metric plays an indispensable role in modeling a wider range of node correlation.
Here, we compare our BC-based random walk with existing renowned walk strategies, including DeepWalk (Perozzi et al., 2014; Zhuang & Ma, 2018), Node2Vec (Grover & Leskovec, 2016), and
Random Surﬁng (Cao et al., 2016).
We use
TG (20 , 1 , 3 , 10) as a test graph of large diameters.
We randomly sample 20 root nodes and, for each root, simulate 10 walks with length 10 to show how many nodes in different order proximity are visited.
The ideal situation for a batch of walk paths with length l is to cover up to nodes l − hop away from the current root.
The results are shown in Figure 2.
From the ﬁgure, we can see that
our
BC-based walk is much more competitive regarding exploration distance.
A further illustration of diverse graph structures is presented in Appendix A.11.
4. P RESERVATION OF D ISTANCE R ELATION
We have also evaluated the property of shortest distance preservation during the mapping from the original graph space into embedded vector space.
First, we test distance variation in the embedding space when adjusting the shortest distance between nodes on the graph, taking the same conﬁguration of simulation dataset as TG(20 , 1 , 3 , 10) .
Distance in the embedding space is measured by inner product Z i Z Tj for any node v i and v j .
We initially train embedding vectors using walks simulated by each model and randomly sample 20 source nodes with 100 destinations for each source.
The results are shown in Figure 3, which indicates that our model has a better tendency to maintain a linear distance relation for the mapping.      :DONOHQJWK
      3UREDELOLW\RI([SORUHG'LVWDQFH (a)      :DONOHQJWK
      3UREDELOLW\RI([SORUHG'LVWDQFH (b)      :DONOHQJWK
      3UREDELOLW\RI([SORUHG'LVWDQFH (c)      :DONOHQJWK
      3UREDELOLW\RI([SORUHG'LVWDQFH ( [Figure]      VKRUWHVWGLVWDQFHEHWZHHQQRGHVRQWKHJUDSK         GLVWDQFHPHDVXUHGRQWKHHPEHGGLQJVSDFH (a)      VKRUWHVWGLVWDQFHEHWZHHQQRGHVRQWKHJUDSK         GLVWDQFHPHDVXUHGRQWKHHPEHGGLQJVSDFH (b)      VKRUWHVWGLVWDQFHEHWZHHQQRGHVRQWKHJUDSK         GLVWDQFHPHDVXUHGRQWKHHPEHGGLQJVSDFH (c)      VKRUWHVWGLVWDQFHEHWZHHQQRGHVRQWKHJUDSK         GLVWDQFHPHDVXUHGRQWKHHPEHGGLQJVSDFH (d) Second, we try to ﬁnd out how much the probability distance relation is violated in the embedding space.
We randomly take 10000 node triple ( v a , v b , v c ) , and record if they violate Equation 13.
The results are shown in Figure 4.
The ﬁgure conﬁrms that our model is much more satisfactory in preserving distance relation during mappings.
      VDPSOHGQRGHWULSOHV          YDOXHRIGLVWDQFHH[SUHVVLRQ
YLRODWHGSUHVHUYHG (a) 
     VDPSOHGQRGHWULSOHV        YDOXHRIGLVWDQFHH[SUHVVLRQ YLRODWHGSUHVHUYHG (b) 
     VDPSOHGQRGHWULSOHV          YDOXHRIGLVWDQFHH[SUHVVLRQ YLRODWHGSUHVHUYHG (c) 
     VDPSOHGQRGHWULSOHV         YDOXHRIGLVWDQFHH[SUHVVLRQ YLRODWHGSUHVHUYHG (d)  C ONCLUSION
In this paper, we propose a novel graph shortest distance embedding method called Betweenness
Centrality-based Distance Resampling (BCDR).
It improves the graph embedding for the shortest distance representation with two components we propose in this paper.
The ﬁrst is Betweenness
Centrality-based random walk to accommodate long-distance correlation on graphs by covering a wider range of nodes under the intrinsic graph metric.
The second is a distance resampling strategy to preserve shortest distances during the mapping from graph to the embedding space via reconstructing a global distance matrix.
The experimental evaluation indicates that BCDR possesses a better capacity than existing graph embedding methods to extract distance structure from original graphs.
BCDR can be integrated into graph-based learning models (especially in graph neural networks), which should improve their performance on graph structure recognition.
This will be our future work.
A A PPENDIX A. C ONNECTIONS TO MDS
Our algorithm also shares some connection with conventional MDS methods in global distance metric.
Take a view of original MDS.
Input data X distributes in agnostic high-dimensional euclidean space and mutual distance between samples is extractable as a global distance matrix D X .
A approximate distance matrix D Y calculated by embedding matrix Y endeavor to be optimized closest to D X by Frobenius norm, i.e.,
[Equation]
Here, we suppose D X and Y have been double-centered for stability, and ﬁnally, Y could be deduced as the group of top-d leading eigenvectors in D X .
Then, as a non-linear version of MDS,
Isomap (Tenenbaum et al., 2000) generalizes D X as distribution on a manifold and utilizes short hops to measure the mutual distance between samples.
Deriving from the above idea, we regard the global distance matrix D X as a canonical metric on graphs and exert embedding matrix Z | V |× d to reconstruct it.
Different from previous work, we encounter prohibitive complexity to acquire all elements in D X and leverage an iterative process to approximately minimize the accuracy loss between
D Z and D X .
         LWHUDWLRQ       ORJRIORVV &RUD)DFHERRN*U4F (a)   
      LWHUDWLRQ 
     IUREQRUPHUURU &RUD)DFHERRN*U4F (b)
Cora
Facebook
[Table] Time [Table] A. C ONNECTIONS TO G RAPH S TRUCTURE D ECOMPOSITION
A newly published work (Chanpuriya et al., 2020) provides an exciting perspective to embed complex sparse graphs perfectly into low-rank representations, which is helpful for downstream ML tasks.
Nevertheless, we need to address the limitation of LPCA in shortest distance prediction as follows.
First and most importantly, it should be clariﬁed that a perfect representation of graph structure is not equal to that of graph shortest path structure, since they have a large calculation gap.
According to Floyd-Warshall algorithm (Floyd, 1962), even if each node is aware of all related path structures, inference of the exact shortest path structure also needs up to O ( N 3 ) complexity.
Second, LPCA has poor embedding performance on some relatively dense graphs, despite converging fast on sparse graphs.
We test this method on the three real-world datasets used in this paper, and illustrate the converge curve in Figure 5.
The converge time is reported in Table 4.
The results show LPCA meets a bottleneck on the relative-dense graph (Facebook) and consumes a long time to converge.
Compared to the above method, the motivation of this paper is to directly embed graph shortest distance matrix with sub-linear time complexity for fast and accurate online queries of shortest distance.
To the best of our knowledge, there is no existing embedding method that could directly and perfectly represent shortest path structures in linear time.
Moreover, our method takes a more ﬂexible objective to represent shortest distances, since only high-level restrictions are implicitly exerted on the embedding space by Equation 12 and 13.
A. P ROOF OF T HEOREM
Proof.
We simplify symbols N h ( v a ) , e h ( v a ) , f h ( v a ) as N h , e h , f h for short.
[Equation]
According to deﬁnition in the theorem, we have N h
=
e h + f h .
Since only e h nodes could travel from
N h to N h
+1 , We ﬁrstly consider P R ( e h → e h +1 |
e h → N h +1 ) and P B ( e h → e h +1 | e h → N h +1 ) .
For general random walk, the choice of destination is based on uniform sampling, thus causing
[Equation]
For BC-based random walk, we need calculate BC value of e h +1 and f h +1 nodes for starters.
Let
BC( e h +1 ) and BC( f h +1 ) represent the BC value of nodes in e h +1 and f h +1 respectively, and the correspond legal shortest path counts comes from 4 sources as { ← N h
→ → N h +2 } , { ← N h → N h +1 } , { N h +1 → → N h +2 } and { N h +
→ N h +1 } .
And we use BC( {· → ·} ) as the BC gain from the speciﬁed source, then [Equation]
[Equation]
where β (1) e means average BC gain between nodes in e h +1 and → N h +2 , and β (2) e means average BC gain between nodes both in e h +1 , which are constantly related with G .
Therefore, we have
[Equation]
Likewise, we calculate
[Equation]
To compare the above BC( e h +1 ) and BC( f h +1 ) ,
[Equation]
note that for any N j and N ( x,y ) =
→ N 0 − ← N x − → N y where j, x, y ∈ { 0 , [Equation] [Equation]
[Equation]
Equation 19 is reduced to
[Equation]
Then [Equation] we perform weighted random sampling based on BC and get
[Equation]
Now, we consider the relation between P R ( N h → e h +1 |
N h
→ N h +1 )
and P B ( N h → e h +1 | N h →
N h +1 ) .
[Equation]
Let C =
[Equation]
β
[Equation]
1) f [Equation]
( k )][1 − p R ( f h → e h )] (1+ fheh )
[ e h +
f h β (1) f (cid:15) ( k )] p R ( f h → e h ) .
Since
C is a non-negative value independent of k , ﬁnally we get
[Equation]
25) where
[Equation]
A. P ROOF OF P ROPOSITION
Proof.
Initially, we rewrite the negative sampling item of Equation 10 as
[Equation]
Then, for each pair of v i ∈ V and v j ∈ W i , we get independent objective by combing similar items in overall likelihood expression E v x ∼ p
[ f ( Z i , Z j )] , and get
E
[Equation]
∼
[Equation]
[Equation]
[Equation] i ∈ [Equation] Let ˆ D = ZZ T , for each node pair ( v a , v b ) with mutual shortest distance ˆ D ab , consider
[Equation]
Suppose v b is n -hops away from v a , and denote BC ( v b ) by γ b , according to Equation 5,
[Equation]
Note that p N ( v b | v a ) is a negative sampling in walk set W a (Equation 11) which is selected by BCbased random walk locally restricted to v a according to Equation 5, we have
[Equation]
where κ ( v a ) is related with the neighbor structure of v a .
Then, Equation 29 could be described as
[Equation]
Solve the above problem by just let ∂ L ( v a ,v b ) ∂ ˆ D ab be equal to zero, i.e.,
[Equation]
After some simpliﬁcation, we get
[Equation]
Let A = − λκ ( v a ) and there holds
[Equation]
A. P ROOF OF T HEOREM
Proof.
Let v b and v c in graph be n and m -hops away from v a respectively.
In terms of node pair ( v a , v b ) , as proved in Proposition 1, their mutual distance D ab in the embedding space varies linear with respect to the graph shortest distance n , i.e.,
[Equation]
Likewise, we have for ( v a , v c )
[Equation]
37) where 0 < α < 1 and A is independent of v b and v c .
Then, consider the distance relation of v a , v b and v c , there holds
[Equation]
A. C OMPLEXITY A NALYSIS OF BCDR E MBEDDING A LGORITHM
We analyze the complexity of Algorithm 1 as follows.
The ﬁrst step depends on the algorithm used for BC calculation, wherein some approximation methods could reduce complexity to O ( | V | ) or O ( | V | log | V | ) (Brandes & Pich, 2007; Ercsey-Ravasz & Toroczkai, 2010).
Then, in the second step, for each walk rooted at each node v i (line 23), we use a loop(line 2 to 20) up to w in times for generating nodes on the walk.
Normalization of p i
→ j (line 5 to 18) is calculated for l in times for each node, and summation of γ is up to O ( | E | ) during the whole routine.
Therefore, it takes totally O ( w in l in | V | + | E | ) time complexity.
Finally, as for the third step, line 26 to 33 re-weights explored area of every node v i which need loops up to w in l in | V | times.
For the sake of resampling w out l out nodes, line 34 also requires normalization by O ( w out l out | V | ) .
Line 37 use a maximum likelihood optimization which occupies
O ( w out l 2 out | V | log | V | + w out l out | V | + | E | ) time
complexity(Mikolov et al., 2013a).
As for space complexity, distance map D i and p ( v j | v i ) are stored temporarily up to O ( w in l in ) , while visit sign set S i is up to O ( w in ) .
For dumped and resampled walk paths, O (( w in l in + w out l in ) | V | ) space is required.
Finally, our embedding algorithm takes O ( w out l 2 out | V | log | V | +( w in l in + w out l out ) | V | + | E | ) time complexity and O (( w in l in + w out l out ) | V | ) space complexity.
Especially, for sparse graph, time complexity is reduced to O ( w out l 2 out | V | log | V | + ( w in l in + w out l out ) | V | ) .
A. C OMPLEMENTARY I NFORMATION OF D ATASETS
This part provides detailed information about graph datasets we used, and presents some visualization result in Figure 6.
[Figure]
A.7. C ORA
Cora graph dataset describes the citation relationship of papers, which contains 2708 nodes and 10556 directed edges among them.
Each node also has a predeﬁned feature with 1433 dimensions.
A.7. F ACEBOOK
Facebook dataset(Leskovec & Krevl, 2014) describes the relationship among Facebook users by their social circles(or friend lists), which is collected from a group of test users.
Facebook has also encoded each user with a reindexed user ID to protect their privacy.
A.7. G R Q C
Arxiv GR-QC (General Relativity and Quantum Cosmology) collaboration network(Leskovec &
Krevl, 2014) is recorded from the e-print arXiv in the period from January 1993 to April 2003, which used to represent co-author relationship based on their submission.
We suppose an undirected edge ( v i , v j ) if an author v i co-authored a paper with another author v j .
If one paper is owned by k authors, a complete graph of k nodes is generated correspondingly.
A.7. T EST G RAPH ( TG )
We deﬁne TG( cls, c, r, n ) as a parameterized graph simulation for SDQ tasks.
cls is the number of sub-graphs to represent communities, and n is the number of nodes in each community.
c and r determine the inner and outer connectivity of each community, respectively.
TG is guaranteed to be an undirected and connected graph in our experiment, which is also expected to be sparse and of large diameter.
Some simulation results of TG(20 , 1 , 3 , 10) that we used in Section 4.4 and 4.5 are presented in Figure 6 (d), (e), (f). A.7. O THER G RAPHS
We also use some other graphs with diverse structures in this paper.
The visualization of these graphs is shown in Figure 7.
We describe each graph as follows.
[Figure]
• Circle Graph: a graph that contains several circles of different sizes. The simulation of circle graphs takes an iterative process where for each newly introduced circle, there are a limited number of nodes (called exit nodes) connected to the previous node set. • Triangle Graph: a graph possessing several cliques which are linearly connected mutually. • Tri-circle Graph: a graph that combines the properties of circle graphs and triangle graphs.
Here, each circle is simulated by connecting triangle sub-graphs end to end.
• Tree Graph: a graph that is generated from one root to several leaves recursively.
There is no cycle in tree graphs.
To control the tree structure, we deﬁne a splitting probability that is decayed exponentially with current depth.
• Spiral Graph: a graph shaped like a spiral line.
We ﬁrst simulate a line graph and add edges between nodes with exponentially increased distances by their indices on the line.
• Net Graph: a graph containing grid-like connections between nodes.
We deﬁne a small probability to drop those edges stochastically.
A. B ASELINE A LGORITHM
We outline all of the baseline models used in performance comparison as follows.
A.8. LLE
Locally Linear Embedding(LLE)(Roweis & Saul, 2000) is an effective method of dimensional reduction that preserves the local linear combination property of nodes.
In contrast with previous
PCA(Jol, 2002) and LDA(Blei et al., 2003) which are guaranteed to discover an optimal reduction in euclidean space, LLE that addresses local distance relation gives a more compact representation for non-linear manifolds.
A.8. LE
Analog to Laplace-Beltrami operator on manifolds, Laplacian Eigenmap(LE)(Roweis & Saul, 2000) leverages graph Laplacian matrix to generate embedding vectors.
The objective of LE is to minimize [Table] mutual distance on edges by L 2 loss, which maps nodes with more ﬁrst-order links closer to each other.
A.8. GF
Graph Factorization(GF)(Ahmed et al., 2013) utilize matrix factorization methods(Gemulla et al., 2011) to deal with graph structure exploration.
By approximating the graph adjacency matrix on a bundle of random nodes, GF is of high efﬁciency for large graphs.
A.8. D EEP W ALK
DeepWalk(Perozzi et al., 2014) is a random walk-based embedding method of sub-linear complexity to represent graph structure.
This method derives from skip-gram(Mikolov et al., 2013a) and negative sampling(Mikolov et al., 2013b) to learn each node representation by predicting its context
. A.8. N ODE 2V EC
Node2Vec(Grover & Leskovec, 2016) improves DeepWalk by performing a parametrized random walk.
For each transition, Node2Vec considers second-order proximity to decide the next node be near or far from the previous node, which allows exploration both in local and global scopes.
A.8. DADL
DADL(Rizi et al., 2018) leverages conventional GRL methods to embed local structure on each node, which implies some similarities between the current node and its neighbors.
Then, by considering different binary operations over node embeddings in deep learning techniques, this method outperforms conventional ones in the shortest distance prediction task. A. F URTHER C OMPARISON IN A CCURACY
We also evaluate our embedding method with the latest GRL techniques.
Parameter setup takes the same conﬁguration as stated in Section 4.2 except for some trivial modiﬁcation to match the embedding dimension d = 16 .
All compared models take default parameters.
We list out the comparison among these methods in Table 5.
Moreover, we further illustrate the relation between prediction accuracy and path length for each method on both sparse graph (GrQc) and relatively-dense graph (Facebook).
The sampling frequency of different path lengths in each dataset is presented in Figure 8.
Length-level prediction accuracy is presented in Figure 9.
The experimental results show our method outperforms others mainly beneﬁted by decent representations of node pair’s similarity in a large range of shortest distance.
On the one hand, BC-based walk occupies a wider range of shortest paths on each node, making remote nodes available before embedding.
On the other hand, the distance resampling strategy discerns nodes by different order proximity to preserve their mutual shortest distance.
A. R UN T IME OF BCDR AND G ENERAL R ANDOM W ALK
As is stated in Section 4.2, the time complexity of BCDR could be adjusted by sampling parameters w in , w out , l in , l out .
To keep the capacity of accomodating large shortest distance, w in and l in are ﬁxed as the previous.
Deﬁne β as the compressing coefﬁcient of path length, and use l out = 40 × β .
w out is thus adapted for inputs, i.e., w out = l in w in l out = 40 β .
The comparasion between BCDR
   
     SDWKOHQJWK  
   VDPSOHIUHTXHQF\ (a)          SDWKOHQJWK      VDPSOHIUHTXHQF\ (b)
  
    SDWKOHQJWK          P5(VFRUH OOHP5(OHP5(JIP5(GZP5(QYP5(GDGOP5(EFGUP5(OSFDP5(QHWPIP5(JUSP5(YVP5( (a)         SDWKOHQJWK          P5(VFRUH OOHP5(OHP5(JIP5(GZP5(QYP5(GDGOP5(EFGUP5(OSFDP5(QHWPIP5(JUSP5(YVP5( (b) and general random walk method is provided in Figure 10 and 11 for run time and corresponding accuracy, respectively.
The experimental results reveal that our method could reduce time complexity with high accuracy to keep pace with general random walk in sparse graphs, but for relatively-dense graphs(like Facebook), there is some room for further optimization.
A. F URTHER D ISCUSSION ON W ALKING P ATTERNS
We test the performance of BC-based walk and other walking patterns on six synthetic graphs.
The statistics of these graphs we simulated are listed in Table 6.
To address the aspect of getting out of local cliques or circles, we provide another BFS-like searching pattern where each transition tends to choose the edges that could get out of the local clique by considering up to second-order proximity (with probability p out ≈ 0 . 901 ).
Properties and visualization of these graphs are presented in
Appendix A.7.5.
And traversal results are illustrated in Figure 12 to Figure 17.
The ﬂuctuation of
BC value on each graph is illustrated in Figure
The results are analyzed as follows
• For circle graphs and tri-circle graphs,
BC-based walk tends to choose the exit nodes of each circle since they share a large BC gain by locating on the shortest path between inner nodes and outer nodes.
     EHWD   
  PLOOLVHFRQGRIUXQWLPH %&'5*HQHUDO5: (a) 
    EHWD        PLOOLVHFRQGRIUXQWLPH %&'5*HQHUDO5: (b)      EHWD         PLOOLVHFRQGRIUXQWLPH %&'5*HQHUDO5: (c)
     EHWD      P5( %&'5*HQHUDO5: (a) 
    EHWD         P5( %&'5*HQHUDO5: (b) 
    EHWD       P5( %&'5*HQHUDO5: (c)
• For triangle graphs, transitions on every triangle clique tend to move forward since the number of nodes beyond the current clique is often larger than that of inner nodes, contributing to more shortest paths.
• For tree graphs, each transition appears to move forward from the root to the leaves, and all walking patterns show a similar exploration.
• For spiral graphs, although some exit nodes are located on the circle, BC gains on exit nodes and inner nodes on circles are usually on par, which misleads the direction choice of the next transition.
Thus, BC-based walk only performs slightly better than others.
• For net graphs, all walking patterns are limited on exploration distance, since BC values on different nodes do not change considerably.
To conclude, the above results show that BC-based walk possesses a better all-around performance to capture remote nodes with large shortest distances.
Circle
Triangle
Tri-circle
Tree
Spiral
Net | V |      | E |      | E | / | V | 1 .2 .2 .0 .1 .1 .avg.
{+Degree 2 .4 .4 .1 .
+}
.3 .diameter            :DONOHQJWK       3UREDELOLW\RI([SORUHG'LVWDQFH (a)       :DONOHQJWK       3UREDELOLW\RI([SORUHG'LVWDQFH (b)       :DONOHQJWK       3UREDELOLW\RI([SORUHG'LVWDQFH (c)       :DONOHQJWK       3UREDELOLW\RI([SORUHG'LVWDQFH (d)       :DONOHQJWK       3UREDELOLW\RI([SORUHG'LVWDQFH (e)
      
  :DONOHQJWK 
     
  3UREDELOLW\RI([SORUHG'LVWDQFH (a) 
      :DONOHQJWK

     
  3UREDELOLW\RI([SORUHG'LVWDQFH (b)       
 :DONOHQJWK       
  3UREDELOLW\RI([SORUHG'LVWDQFH (c) 
     
  :DONOHQJWK 
     
  3UREDELOLW\RI([SORUHG'LVWDQFH (d)       
  :DONOHQJWK 
     
  3UREDELOLW\RI([SORUHG'LVWDQFH (e)
        :DONOHQJWK 
       
3UREDELOLW\RI([SORUHG'LVWDQFH (a)         :DONOHQJWK 
       
3UREDELOLW\RI([SORUHG'LVWDQFH
(b)         :DONOHQJWK 
       
3UREDELOLW\RI([SORUHG'LVWDQFH
(c)         :DONOHQJWK 
        3UREDELOLW\RI([SORUHG'LVWDQFH
(d)         :DONOHQJWK 
        3UREDELOLW\RI([SORUHG'LVWDQFH
(e)
     :DONOHQJWK
      3UREDELOLW\RI([SORUHG'LVWDQFH (a)      :DONOHQJWK
      3UREDELOLW\RI([SORUHG'LVWDQFH (b)      :DONOHQJWK
      3UREDELOLW\RI([SORUHG'LVWDQFH (c)      :DONOHQJWK
      3UREDELOLW\RI([SORUHG'LVWDQFH (d)      :DONOHQJWK
      3UREDELOLW\RI([SORUHG'LVWDQFH (e)
    
 :DONOHQJWK 
     3UREDELOLW\RI([SORUHG'LVWDQFH
(a) 
   
 :DONOHQJWK 
     3UREDELOLW\RI([SORUHG'LVWDQFH (b)     
 :DONOHQJWK 
     3UREDELOLW\RI([SORUHG'LVWDQFH (c) 
   
 :DONOHQJWK 
   
 3UREDELOLW\RI([SORUHG'LVWDQFH
(d) 
   
 :DONOHQJWK 
   
 3UREDELOLW\RI([SORUHG'LVWDQFH
(e)
        :DONOHQJWK         3UREDELOLW\RI([SORUHG'LVWDQFH (a)         :DONOHQJWK         3UREDELOLW\RI([SORUHG'LVWDQFH (b)         :DONOHQJWK         3UREDELOLW\RI([SORUHG'LVWDQFH (c)         :DONOHQJWK         3UREDELOLW\RI([SORUHG'LVWDQFH (
[Figure]
)         :DONOHQJWK         3UREDELOLW\RI([SORUHG'LVWDQFH (e)


    QRGHV  
      %&YDOXHRIQRGHV *B&LUFOH (a) 

   
  QRGHV  
    %&YDOXHRIQRGHV *B7ULDQJOH (b) 

   QRGHV  
    %&YDOXHRIQRGHV *B7ULFLUFOH (c) 

   QRGHV  
      %&YDOXHRIQRGHV *B7UHH (d) 
     QRGHV  
   %&YDOXHRIQRGHV *B6SLUDO (e) 
     QRGHV        %&YDOXHRIQRGHV *B1HW (f)