A
D EEP
L ATENT
S PACE
M ODEL
FOR
D IRECTED
G RAPH R EPRESENTATION
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph representation learning is a fundamental problem for modeling relational data and beneﬁts a number of downstream applications.
Traditional Bayesian-based random graph models and recent deep learning based methods are complementary to each other in interpretability and scalability.
To take the advantages of both models, some combined methods have been proposed.
However, existing models are mainly designed for undirected graphs , while a large portion of real-world graphs are directed.
The focus of this paper is on the more challenging directed graphs where both the existences and directions of edges need to be learned.
We propose a Deep Latent Space Model (DLSM) for directed graphs to incorporate the traditional latent space random graph model into deep learning frameworks via a hierarchical variational auto-encoder architecture.
To adapt to directed graphs, our model generates multiple highly interpretable latent variables as node representations and the interpretability of representing node inﬂuences is theoretically proved.
The experimental results on real-world graphs demonstrate that our proposed model achieves the state-of-the-art performances on link prediction and community detection tasks while generating interpretable node representations of directed graphs.
I NTRODUCTION
Learning representations on graphs is a fundamental problem for graph analysis and beneﬁts a number of downstream applications, typically including link prediction, community detection and inﬂuential node identiﬁcation.
Traditionally, a plethora of Bayesian-based random graph models have been proposed for learning graph representations (Holland et al., 1983; Hoff et al., 2002; Karrer & Newman, 2011; Sewell & Chen, 2015).
Despite the ideal interpretability of these methods, they are unscalable to model large-scale graphs due to the expensive iterative inference procedures.
Boosted by the powerful representation learning ability, graph neural networks (GNNs) have been proposed to learn the topology of graph-structured data (Kipf & Welling, 2017; Hamilton et al., 2017;
Veli ˇ ckovi ´ c et al., 2018; Zeng et al., 2020).
However, these GNN-based methods usually bring about the interpretability issues, which restrict the practical applications of these methods.
As an effective deep learning method based on the Bayesian theory, the variational auto-encoders (VAEs) (Kingma & Welling, 2014b) have been proposed for representation learning on graphstructured data (Kipf & Welling, 2016; Grover et al., 2019; Salha et al., 2019).
To capture the mutual facilitation of random graph models and deep learning based methods, some recent research attempts to combine them using VAEs, and build the deep generative random graph models (Mehta et al., 2019; Sarkar et al., 2020).
However, most of these VAE-based methods are designed only for undirected graphs .
In contrast, a large portion of real-world graphs, such as social and citation networks, are directed graphs , of which undirected graphs are special cases (i.e. an undirected edge can be regarded as a bidirectional edge).
The representation learning on directed graphs is particularly challenging since not only the existences but also the directions of edges need to be learned.
Although directed graphs have been extensively studied by traditional research on random graph models (Hoff et al., 2002; Krivitsky et al., 2009; Sewell & Chen, 2015), there are still few deep learning based methods and no combined models existed to address them.
In this paper, we present the ﬁrst VAE-based representation learning model for directed graphs and propose the Deep Latent Space Model (DLSM), which combines graph convolutional networks
(GCNs) with the Bayesian random graph model using a hierarchical VAE architecture.
Within our model, a deep GCN is leveraged to encode the asymmetric adjacency matrix and node features as hidden states, which are layer-wise transformed to variational parameters to generate latent random variables as interpretable node representations.
Then, the hierarchical decoder network reconstructs the adjacency matrix using the learned representations.
To better capture the characteristics of graphs for deep random graph models, our method generates three types of interpretable representations, including the latent positions, node random factors and community memberships.
The latent positions measure the distances between nodes and enable our method to model the link reciprocity of directed graphs, which means that both bidirectional and unidirectional edges exist.
To characterize the degree heterogeneity , we introduce a pair of node random factors, namely the social activity and popularity, thus the asymmetric structure of directed edges can be modeled.
Besides, we use a binary latent variable, named the community membership, to accommodate an overlapping community structure .
We further prove that these node-speciﬁc variables, including the latent positions and random factors, are interpretable for representing the existence and strength of node inﬂuences.
For fast inference, we leveraged the stochastic gradient variational Bayes (SGVB) algorithm (Kingma & Welling, 2014a), which is far more efﬁcient and scalable compared to the traditional iterative inference methods.
We conduct experiments on link prediction task to verify the effectiveness of directed graph modeling with several real-world graphs, and further demonstrate the interpretability of latent variables on community detection task.
The main contributions of this work are summarized as follows:
We propose the DLSM model for directed graphs to incorporate the latent space model into deep learning frameworks using a hierarchical VAE architecture, which can generate highly interpretable representations of graph nodes for multiple downstream tasks.
To adapt our method to directed graphs, we introduce the social activity and popularity factors of each node, with theoretical proof of their capabilities in representing node inﬂuences.
We conduct experiments on ﬁve real-world graph datasets
, and the results show that our proposed method can well support downstream tasks and achieve the state-of-the-art performances in link prediction and community detection.
R ELATED W ORK
In this section, we brieﬂy review the traditional Bayesian-based random graph models and deep learning based generative graph representation methods.
2. B AYESIAN - BASED RANDOM GRAPH MODELS
Classic Bayesian-based random graph models have developed for decades and are still valued in modeling relational data.
This methods regard graph nodes as random variables following a prior distribution, and learns low-dimensional embeddings via the posterior distributions of nodes.
One of the most well-known methods is the stochastic blockmodel (SBM) (Holland et al., 1983), which generates a latent variable indicating the community membership of each node.
Following SBM, a large number of variants have been proposed, such as allowing an overlapping community structure (Miller et al., 2009) and involving degree heterogeneity (Karrer & Newman, 2011).
Another stream of research is the latent space model (LSM)
(Hoff et al., 2002), which assumes each node as a position in an unobservable latent space and use distances to measure the relationships between nodes.
Such method has soon been extended to model directed graphs by involving the degree heterogeneity of nodes (Krivitsky et al., 2009; Sewell & Chen, 2015).
One of the most important advantages for LSM is that the symmetry of distances can naturally endow node embeddings with the link reciprocity of directed graphs.
Although these above methods provide good theoretical properties and learn interpretable node embeddings, they rely on either the MCMC posterior sampling or mean-ﬁeld variational inference with dramatically high computational complexity, and thus are usually unfeasible when modeling large-scale graphs.
2. D EEP LEARNING BASED GRAPH MODELS
The intriguing achievements of deep learning models for learning representations of Euclidean data have encouraged efforts to employ them on graph-structured data.
The earliest attempts include
DeepWalk (Perozzi et al., 2014) and node2vec (Grover & Leskovec, 2016), both of which encode local relations as node representations by conducting random walks on a graph.
More recently, many GNNbased deep generative models have been proposed.
Kipf & Welling (2017)
ﬁrst leverages the spectral
GCN to generate node embeddings using the global topology of a graph.
GraphSAGE (Hamilton et al., 2017) samples a ﬁxed-size neighborhood of each node for the inductive representation learning, which allows unseen nodes to be excluded during training.
Later, GraphGAN (Wang et al., 2018) uniﬁes a generator and a discriminator to learn node embeddings by playing a minimax game.
The graph attention networks (Veliˇckovi´c et al., 2018) introduce the attention mechanism to aggregate the information from neighbors.
Enlightened by the powerful ability of VAEs for representation learning
Kingma & Welling (2014b);
Sønderby et al.
(2016), the variational graph auto-encoders (VGAE)
(Kipf & Welling, 2016) is proposed to generate Normal node embeddings using the VAE (Kingma & Welling, 2014b), which is then modiﬁed by Graphite (Grover et al., 2019) with a perception component based on GCN.
Salha et al.
(2019) proposes an asymmetric decoding scheme which handles unidirectional graph proximity.
To take advantages of the nice theoretical properties of random graph models and the powerful representation ability of deep learning methods, some research work tries to combine these two approaches for graph representation learning.
Mehta et al.
ﬁrst combines the classic SBM with GCN to learn a sparse real-valued node embedding interpreted as the membership and strengths of each node belonging to different communities.
Furthermore, Sarkar et al.
(2020) builds a sparse ladder VAE (Sønderby et al., 2016) architecture to discover the communities at multiple levels of granularities.
All of these combined models, however, are only designed for undirected graphs without considering graph characteristics, such as degree heterogeneity.
In this paper, we design a hierarchical VAE architecture which combines the LSM and GCN to learn representations on directed graphs involving the community structure and degree heterogeneity.
D EEP L ATENT S PACE M ODEL
Consider a directed graph containing n nodes.
The data to be modeled include an asymmetric adjacency matrix A
=
( A ij ) ∈ { 0 , 1 } n × n , where each binary element A ij denotes the presence ( 1 ) or not ( 0 ) of the directed edge from node i to j , and a node attribute matrix X ∈ R n × p .
Throughout this paper we assume all edges to be conditionally independent and satisfy A ij | Θ ∼ Bernoulli ( p ij ) , where Θ is the collection of latent representations and p ij ∈ (0 , 1) is the posterior probability to form an edge.
The objective is to learn the node representations which can best reconstruct the adjacency matrix.
We propose a deep VAE architecture composed of a GCN encoder and a hierarchical latent space model (HLSM) decoder, as shown in Fig.
The encoder block takes the adjacency A and attribute matrices X , if available, as inputs and learns a hidden state for each node using a directed GCN.
Then, the decoder block recursively generates interpretable latent variables from variational posteriors via Monte Carlo sampling and reconstructs the adjacency matrix as outputs.
For inference, the hierarchical architecture enables the model to learn variables using not only the priors passed from previous layers of the decoder, but also the approximate likelihood learned by the encoder.
3. HLSM DECODER
The HLSM decoder learns interpretable node representations based on the classic LSM approach and generates an adjacency matrix.
Assume each node to correspond to a latent position in an unobservable D -dimensional space, denoted as z i ∈ R D .
The distances between latent positions indicate the relationships of nodes.
The closer two latent positions are, the more possible an edge will exist.
Formally, an directed edge is generated by following
[Equation]
where σ ( · ) is the sigmoid function.
The latent positions z
i are involved as Euclidean distances, indicating the relationships between nodes.
γ i and δ i ∈
R D are the activity and popularity factors,
( 1 ) H z ( 2 ) H z ...
( - 1 ) H L z ( 1 ) Z ... ( 2 ) Z ( -1)
Z L ( 1 ) H s ( 2 )
H s ...
( - 1 ) H s L ( 1 ) S ... ( 2 ) S ( -1)
S L
Z ( 1 ) H   ( 2 ) H   ...
( - 1 ) H L   ...
( 0 ) H
[Figure]
X ˆ A
GCN Encoder
HLSM Decoder ,   (1) (1) ,   (2) (2) ,   ( 1 ) ( 1 ) , L L   the reverse of which represent the tendencies for a node to send and receive an edge, respectively.
The global weights β out and β in are to measure the importance of activity and popularity, respectively, and β 0 is a bias.
To be mentioned, though Eq.
(1) is designed for directed graphs, it can be easily degraded to an undirected variant by setting γ i = δ i and β out
= β in .
We build an HLSM decoder to generate three types of latent variables, i.e. the latent positions z ( l )
i ∈
R G l , the community membership s ( l ) i ∈
R G l and the node random factors γ
i , δ ( l ) i ∈
R G l , where G l is the size of the l − th decoder layer.
All of these variables are randomly sampled from the variational distributions.
At the ﬁrst layer ( l = 1 ), the parameters of variational distributions are deﬁned by priors solely, while for other layers ( l = 2 , . . .
, L − 1 ), the parameters are obtained by the feedforward representations generated from previous layers as well as priors.
Latent position
The latent positions z ( l ) i represent the location of each node in a latent space, generated as
[Equation]
where f ( · ) is a nonlinear activation function (e.g. leaky ReLU), (cid:12) denotes the element-wise multiplication, diag ( · ) denotes a diagonal matrix, σ ( l ) i
2 ∈ R G l is the prior variance to be speciﬁed, and
W ( l − 1) z ∈ R G l × G l − 1 is a weight matrix to transform the variable from dimension G l − 1 to G l .
Community membership The latent positions in Eq.
(2) are separated into different communities by the sparse binary variables s ( l ) i =
( s i 1 , . . .
, s iG l ) (cid:48) , thus an overlapping community structure can be adapted.
Referring to Miller et al.
(2009), we employ the Indian buffet process (IBP) prior on s ( l ) i to learn the effective number of communities given a sufﬁciently large truncation parameter G l , i.e.
[Equation]
[Equation]
Typically, the log odds π ( l ) ig is generated using a stick-breaking construction, where v ( l ) ih is drawn from a Beta distribution (Teh et al., 2007).
In our model we simplify such hierarchical prior structure by specifying a global v for all nodes.
At each layer, the community membership s ( l ) ig denotes whether node i belongs to community g , hence the size of each decoder layer G l can be explained as the number of communities.
Additionally, the proposed HVAE architecture enables our model to detect community structures at multiple levels of granularities.
Letting the layer sizes G l be downward increasing, the top layers indicate the coarse-grained communities and the bottom layers indicate the ﬁne-grained communities.
Node random factors To model the prevalent power-law of node degrees, we propose the Dirichlet random factors γ ( l ) i , δ ( l )
i ∈
R G l , i.e.
[Equation]
where ξ ( l ) i and ψ ( l ) i are prior parameters to be speciﬁed, and W ( l − 1) γ ,
[Equation]
are weight matrices.
Note that the node random factors are dependent on the community membership s ( l ) i as well, which can be explained as only the random effects within the community are involved in the representations.
At the output layer ( l = L ), the latent positions z
i and node random factors γ i , δ i are obtained by linear transformation from dimension G L − 1 to D .
Each column of the weight matrices belongs to a ( D − 1) -simplex.
Such transformation also changes the interpretation of the layer size, from the number of communities to the dimension of the latent space.
Last, the adjacency matrix is reconstructed with the latent variables by Eq.
3. GCN ENCODER
The proposed DLSM employs a deep encoder as a non-iterative recognition model to infer the parameters of posterior distributions.
Assuming the mean-ﬁeld approximation of the variational distributions, the true joint posterior of the latent variables p θ (Θ | A , X ) can be approximated by a variational posterior q φ (Θ) , where θ and φ denote the generative (decoder) and inference (encoder) parameters to be trained, respectively.
Then, the variational posterior is given as
[Equation]
where h ( l ) i ∈
R K l is the output of the encoder and K l denotes the size of the l -th layer.
GCN has been proved effective in learning the topology of non-Euclidean data, and thus is an ideal choice for the encoder of our model.
Referring to Kipf & Welling (2017), we propose the directed
GCN operator as
[Equation]
Here ˜
A =
A + I n
and I n is the n -dimensional identity matrix.
˜ D out and ˜ D in are diagonal matrices with elements as the out- and in-degrees of ˜ A , respectively.
H (0) =
X if X is available and H (0)
=
I n if not.
During inference, the hidden states are passed to the corresponding layer of the decoder and then combined with the prior information from previous layers to generate the parameters of variational distributions.
Note that the vanilla GCN used here can be substituted by any other GNN for directed graphs.
I NFERENCE
We now introduce our fast inference method using the SGVB algorithm (Kingma & Welling, 2014a).
Compared with the iterative methods such as MCMC adopted by traditional Bayesian random graph approaches, SGVB is much more efﬁcient and scalable.
Such method requires differential Monte
Carlo expectations to perform backpropagation, thus the reparameterization trick for each of the latent variables is leveraged.
Denoting z ∗ ∈ R G l a vector with standard Normal elements, the latent positions are reparametrized as
[Equation]
ˆ µ ( l ) i ( z ( l − 1) i , s ( l − 1)
i , h ( l ) i )
+ ˆ σ ( l )
i
( z ( l − 1)
i , s ( l − 1)
i , h ( l ) i ) (cid:12) z ∗ , where ˆ µ ( l ) i
( z ( l − 1)
i , s ( l − 1)
i , h ( l ) i ) , ˆ σ ( l ) i ( z ( l − 1) i , s ( l − 1)
i , h ( l ) i ) are variational posterior parameters.
Following Mehta et al.
, the Bernoulli posterior p θ ( s ( l ) ig ) is approximated by the Binary Concrete distribution (Maddison et al., 2017), i.e.
[Equation]
where u ∼ U (0 , 1) , ˆ π ( l ) ig ( s ( l − 1) i , h ( l ) i ) is a variational posterior parameter, λ ∈ R + is the temperature to be speciﬁed, and then s ( l ) ig = σ (˜ s ( l ) ig ) .
Referring to the recent literature (Joo et al., 2020), we use multiple normalized Gamma variables with uniﬁed rate parameters to compose the Dirichlet distributions p θ ( γ ( l ) i ) and p θ ( δ ( l ) i ) , i.e.
[Equation]
where ˆ ξ i ( l ) ( γ ( l − 1) i , s ( l − 1)
i , h ( l ) i ) and ˆ ψ i ( l ) ( δ ( l − 1) i , s ( l − 1)
i , h ( l ) i ) are variational posterior parameters.
For the convenience of notations, here the parameters of Gamma distribution are symbolized as vectors, meaning the element-wise operations.
The node random factors are then derived by γ ( l ) i =
˜ γ ( l ) i / (cid:80) nj ˜ γ ( l ) j , δ ( l ) i = ˜
δ ( l ) i / (cid:80) nj ˜ δ ( l ) j .
In practice, the Dirichlet variables are magniﬁed by n times to avoid too small values of γ ( l ) i and δ ( l ) i when n is large.
In particular, the initial variational posterior parameters ( l = 0 ) are set as nonlinear combinations of h (1) i and h
( L − 1)
i , as illustrated in Fig.
The loss function is deﬁned by minimizing the negative evidence lower bound (ELBO), i.e.
L =
[Equation]
where KL [ q ( · ) || p ( · )] denotes the Kullback-Leibler (KL) divergence between q ( · ) and p ( · ) (of which the full expression is given in Appendix A).
The second term is the cross entropy of adjacency matrix reconstruction.
Here all of the true posteriors p θ (Θ ( l ) ) and variational posteriors q φ (Θ ( l ) ) , except for the input layer ( l = 1 ), are conditioned on Θ ( l − 1) , which is omitted for simpliﬁcation.
M ODEL I NTERPRETABILITY
In this section, we show that the proposed latent positions and node random factors are interpretable for representing the existence and strength of node inﬂuences, by hypothesizing that the true posterior of z i can be approximated by the variational Normal distribution.
Imagine that there is a new node i with n i ≥ 2 neighbors entering a graph and we want to calculate z i using other existed nodes.
It is rational to suppose that the new node should be posited around its neighbors and thus can be denoted as a residual to the mean of its neighbors, i.e.
[Equation]
where ¯ z i = 1 n i (cid:80) j ∈N i z j is the mean position of node i ’s neighbors, and (cid:15) i ∈
R D is the residual for z i to the mean position.
Intuitively, the residual should depend on all of the neighbors to various extent, of which the inﬂuential ones are likely to contribute more.
Formally, we decompose the residual as
[Equation]
where r ij ≥ 0 is a scalar and e ij ∈ R D is a unit vector at the direction of z j −
¯
z
i .
The non-negative scalar denotes the strength of inﬂuences for neighbor j exerting on node
i (see Fig.
2 in Appendix for an example in a two-dimensional latent space).
An extreme case is r ij = 0 , which means the latent position of i is not inﬂuenced by neighbor j at all.
To further analyze the relation between the latent variables and node inﬂuences, we deﬁne the prior of r ij as a mixture of a point mass on 0 and an exponential distribution, i.e.
[Equation]
[Equation]
where 0 < p 0 < 1 and f Exp λ ( · ) denotes the density function of the exponential distribution with parameter λ .
What is of our interest is the posterior odds ratio (OR) for r ij > 0 , deﬁned as
OR ( r ij > 0 | A , X
[Equation]
= (cid:82)
[Equation]
( r ij = v | A , X ) /f
r ( r ij =
| A , X ) dv , which measures the ratio for node inﬂuences to exist.
Theorem 1 Let Θ denote a collection of all parameters except z i .
The posterior OR for r ij >satisﬁes
[Equation]
(15) where
[Equation]
and Φ( · ) and φ ( · ) denote the cumulative distribution function and probability density function of the standard Normal distribution, respectively.
The full proof is provided in Appendix B.1.
It is interesting that in Theorem 1 ( z i − ¯
z
i − (cid:80) k ∈N i ,k (cid:54) =
j r ik e ik ) (cid:48)
e ij is a scalar projection of z
i ’s residual component onto the unit vector whose direction is determined by z j .
In a word, it shows that the posterior latent positions are able to model the existence of node inﬂuences via the residual component projections.
Now we consider the interpretation of node random factors, which explains the strengths of node inﬂuences.
Formally, we have the following conclusion, i.e.
Theorem 2 Assume the vectors from ¯ z i to all inﬂuential neighbors ( r ij > 0 ) are orthogonal to each other, and the random factors are the same at all dimensions of the latent space, denoted as γ i and δ i , respectively.
The strength of inﬂuences for neighbor j exerting on node i can be denoted as
[Equation]
) where θ is the angle between z i − ¯ z
i and z j − ¯ z
i , o is the angle between ¯ z i − z
j and z i − z j .
Here h ( γ i , δ j , θ, o ) is a function of the random factors as well as θ
and o (more details are given in
Appendix B.2).
Theorem 2 reﬂects that given the direction of e ij and z j − z i , the node random factors can be regarded as coefﬁcients for measuring the ratio of r ij to (cid:107)
z
j − z i (cid:107) , which means that the strength of inﬂuences for a neighbor delivering to the node can be strengthened or undermined via shrinking or enlarging the distance between them by the random factors.
E XPERIMENTS
To evaluate the effectiveness and interpretability of the proposed model, we conduct a series of experiments, including link prediction and community detection, on several real-world graphs.
The experimental results show that the learned node representations can better ﬁt directed graphs and our model signiﬁcantly outperforms the baseline methods.
The interpretability of node representations are also illustrated through clustering via projecting the latent positions onto a 2D space.
6. D ATASETS
The experiments of link prediction are conducted on ﬁve real-world datasets, namely Emails, Political blogs, Cora, WikiVote and Google.
Speciﬁcally, the Emails network consists of the members from
[Table]
departments of a European research institution (Leskovec et al., 2007).
An edge exists if a person sends at least one email to another.
Political blogs is a well-studied social network composed of U.S. political blog nodes (Adamic & Glance, 2005), which are labeled as “liberal” and “conservative” clusters.
Two blog pages are connected if one is referenced by another.
Cora is a citation network of scientiﬁc publications which are classiﬁed into seven classes (Sen et al., 2008).
Each node and directed edge represent a paper and a citation, respectively.
WikiVote is a network of Wikipedia users (Leskovec et al., 2010), where each edge represents a user voting on another to become the administrator.
Google is a network of web pages which are connected by hyperlinks (Palla et al., 2007).
All of the graphs have been preprocessed by omitting the isolated nodes and loops.
In our experiments, the edges are randomly splitted as 85% for training, 10% for testing, and 5% for validation.
The descriptive statistics of the datasets are given in Appendix C.
6. B ASELINES
We compare our proposed DLSM 1 with four recent deep generative methods for graph representation learning, i.e. the variational graph auto-encoder (VGAE)
(Kipf & Welling, 2016), the deep generative latent feature relational model (DGLFRM) (Mehta et al., 2019), the gravity graph variational autoencoder (GGVAE) (Salha et al., 2019), the ladder Gamma variational auto-encoder for graphs (LGVG) (Sarkar et al., 2020), and the directed graph generative adversarial network (DGGAN) (Zhu et al., 2021).
For link prediction, we also consider a non-generative GNN-based model, i.e. SEAL (Zhang et al., 2021), which is the state-of-the-art method for this task.
The methods that are not particularly designed for directed graphs, including VGAE, DGLFRM and LGVG, are slightly modiﬁed by altering the original encoder with our proposed directed GCN given in Eq (7).
6. E XPERIMENTAL R ESULTS
Link Prediction For link prediction, we employ the widely used area under the ROC curve (AUC) and average precision (AP) as evaluation metrics.
Hyperparameter settings are provided in
Appendix D. The experimental results of DLSM and the baselines are presented in Table 1, where the reported results are the means and standard deviations of 10 independent random splits.
On all datasets, our model achieves better or comparable performances to the non-generative method SEAL.
However, since it requires to materialize a subgraph for each link, SEAL is much more unscalable than our DLSM and takes about 20 multiples longer for running one epoch.
The traditional LSM model is unpractical to ﬁt large graphs such as WikiVote and Google because of the MCMC-based inference methods, while the SGVB method adopted by our model is much more computationally efﬁcient.
In addition, it seems that GGVAE performs well on approximately unidirectional graphs (i.e. the reciprocal rate is close to 0 as shown in Table 4) such as Cora, whereas on other more generalized directed graphs where more bidirectional edges exist, our method is signiﬁcantly superior to GGVAE.
This is because our proposed DLSM considers the link reciprocity of directed graphs, which is neglected by GGVAE due to its absolutely asymmetric decoding scheme.
We also conduct experiments of link prediction on three real-world undirected graphs, given in Appendix E, which show that our proposed model achieves comparable performances with the baselines and verify the capability for modeling undirected graphs as special cases. [Table]
[Table] Community Detection We further conduct community detection on three real-world datasets with ground-truth community labels,
i
[Table]
e. Emails, political blogs and Cora.
In particular, for the Emails dataset
[Table]
communities (departments) with less than 30 nodes are excluded and ﬁnally 10 communities are retained.
We utilize the node embeddings learned by each model (the latent positions
z i for
DLSM)
to conduct K-means clustering with ground-truth community numbers.
The commonly used macro and micro F1 scores are leveraged as evaluation metrics, which are computed using the most likely mappings between true and predicted clusters.
Table 2 shows that our proposed model signiﬁcantly outperforms all other baselines for graph representation learning.
Such results verify the advantages of leveraging the Euclidean distance rather than inner product to compute the divergences between nodes for community detection.
(See Appendix G for visualizations of node embeddings using a 2D t-SNE projectionthe distributions of the reverse node random factors, which shows a consistent power-law with node degrees and offers more interpretability.)
Ablation study To justify the effectiveness of each component in our framework, including the community membership s i , node random factors, δ i , γ i , and the HLSM decoder architecture, we compare the proposed DLSM with several variants of our method, which are named by the corresponding ablated components.
For example, - ∆ - Γ indicates the variant removed node random factors (see Appendix F for more details).
The link prediction results are presented in Table 3 and the community detection results are in Table 8 of Appendix.
Our DLSM performs best on all datasets for both tasks, verifying that all components are useful under the proposed framework.
Interestingly, from the ﬁrst two lines of Table 3, we ﬁnd that the community membership seems ineffective when the node random factors are not involved.
This reﬂects that the binary s i mainly beneﬁts our model by shrinking the redundant latent variables to 0 when there are too many parameters to optimize.
C ONCLUSION
We establish a deep generative model to learn multiple highly interpretable node representations for directed graphs.
Our proposed DLSM, comprised of a deep GCN encoder and a HLSM decoder, combines the traditional random graphs models with deep learning based methods to take the complementary advantages of interpretability and scalability.
Series of experiments have shown that the model is effective for ﬁtting directed graphs and achieves the state-of-the-art performance on link prediction and community detection.
In addition, the interpretable node representations learned by the model can naturally represent both the community structure and degree heterogeneity of complex directed graphs.
In the future, we shall extend our model for the more complicated scenes such as weighted or dynamic graphs.
While the former with multi-valued edges can be simply achieved, the latter demands for a more efﬁcient method to learn the evolutionary topology of graphs.
A L OSS F UNCTIONS
The KL divergence between Normal distributions Kingma & Welling (2014b) for z ( l ) i is
[Equation]
The KL divergence between Binary Concrete distributions Maddison et al.
(2017) for s ( l ) i is
[Equation]
where ˜ s ( l ) ig is the log odds of s ( l ) ig .
The KL divergence between Gamma distributions Joo et al.
(2020) for γ ( l ) i (the same for δ ( l ) i ) is
[Equation]
where Γ( · ) and ψ ( · ) denote the Gamma and digamma function, respectively.
B D ETAILS OF T HEOREMS B. P ROOF OF T HEOREM
For any positive value v > 0 , the posterior probability density of r ij at v is
[Equation]
[Equation]
According to Eq.
(12) and (13), we have f (cid:0)
z i (cid:12)(cid:12) r ij = v, Θ (cid:1)
f ( r ij = v ) f
(cid:0) z i (cid:12)(cid:12) r ij = 0 , Θ (cid:1) f
( r ij = 0)
= 1 − ppexp   −
D (cid:89)
d σ 2 id     z i − ¯
z
i − v e ij − (cid:88) k ∈N i ,k (cid:54) =
j r ik e ik   (cid:48)   z
i − ¯
z
i − v e ij − (cid:88) k ∈N i ,k (cid:54) =
j r ik e ik   −
  z
i − ¯
z
i − (cid:88) k ∈N i ,k (cid:54) =
j r ik e ik   (cid:48)   z
i − ¯
z
i − (cid:88) k ∈N i ,k (cid:54) =
j r ik e ik 
 
 − λv   =1 − ppexp   −
D (cid:89)
d σ 2 id   v 2 − 2 v   z
i − ¯
z
i − (cid:88) k ∈N i ,k (cid:54) =
j r ik e ik   (cid:48)
e ij 
 − λv  
= 1 − ppexp 
 −
D (cid:89)
d σ 2 id   v −     z i − ¯
z
i − (cid:88) k ∈N i ,k (cid:54) =
j r ik e ik   (cid:48)
e ij − λ
D (cid:89)
d σ 2 id   
 +
D (cid:89)
d σ 2 id     z i − ¯
z
i − (cid:88) k ∈N i ,k (cid:54) =
j r ik e ik   (cid:48)
e ij − λ
D (cid:89)
d σ 2 id   2   .
By dividing f ( r ij = 0 | A , X ) on both sides of Eq.
(21) and integrating over v on (0 , + ∞ ) , we can obtain
OR
[Equation]
r
[Equation]
>
[Equation] [Equation]
[Equation]
where g ( z i , Θ
[Equation]
=
[Equation]
The conclusion is proved.
B. P ROOF OF T HEOREM
We start our proof by giving two deﬁnitions, the standard distance (SD) and actual distance (AD) between node i and neighbor j .
Consider a simple case, where the inﬂuence discrepancies of all neighbors are neglected and the latent position
z i is only dependent on the distances between the node and neighbors (see Fig. 2(b) for an illustration).
In this case, the residual (cid:15) i =
and z i will coincide with ¯ z i , thus the distance from neighbor j to node i is equal to (cid:107)
z
j − ¯ z i (cid:107) , which we refer as the standard distance.
Now we reconsider the inﬂuence discrepancies of neighbors upon the node.
According to Eq.
(1), the actual distance between node i and neighbor j when involving node inﬂuences is node inﬂuences in our DLSM are involved by rescaling the standard distance.
Assume the vectors from the ¯ z i to all inﬂuential neighbors ( r ij > 0 ) are orthogonal to each other, and the random factors are the same at all dimensions of the latent space, denoted as γ i and δ i , respectively.
Then, the actual distance between z i and z j is
[Equation]
The coefﬁcients β out γ i + β in δ j enlarge or shrink SD ij = (cid:107) z
j − ¯ z i (cid:107) to AD ij using the node activity and popularity factors.
Let θ ∈ (0 , π/ 2) be the angle between z i − ¯
z
i and z j − ¯ z i , i.e.
[Equation]
[Equation]
Since e ij is orthogonal to other residual components, the modulus of (cid:15) i can be denoted as (cid:107) (cid:15)
i (cid:107) =
r ij / cos θ .
According to the law of cosines
[Equation]
we have
[Equation]
By combining Eq.
(22) and Eq.
[Equation]
we obtain
[Equation]
[Figure]
z
[Figure]
i (
[Figure]
i z j z k z
i
z i standard distance actual distance l z h z ij r 
(b) Standard and actual distances
Given θ ∈ (0 , π/ 2) (when θ ∈ ( π/ 2 , π ) , r ij = 0 ), the sign for the right side depends on the positive or negative correlation for (cid:107) (cid:15)
i (cid:107) and AD given θ , which is determined by the angle between ¯ z i − z j
and z i − z j , denoted as
[Equation]
The sign is positive when o ∈ (0 , π/ 2 − θ ) and negative when o ∈ ( π/ 2 − θ, π ) .
In summary, we have the following conclusion, i.e.
[Equation]
, where
[Equation]
=    cos 2 θ − cos θ (cid:104) ( β out γ i + β in δ j ) 2 − sin 2 θ (cid:105) 12 , if o ∈ (cid:0) 0 , π 2 − θ (cid:1) cos 2 θ + cos θ (cid:104) ( β out γ i + β in δ j ) 2 − sin 2 θ (cid:105) 12 , if o ∈ (cid:0) π 2 − θ, π (cid:1) , and θ , o is deﬁned by Eq.
(23) and Eq.
(25), respectively.
C D ATASETS FOR D IRECTED L INK P REDICTION
The descriptive statistics of the directed graph datasets for link prediction and community detection are summarized in Table 4.
D H YPERPARAMETER S ETTING AND S ENSITIVITY A NALYSIS
The proposed DLSM employs a hierarchical VAE architecture composed of a GCN encoder and
HLSM decoder.
In practice, both of the two parts have three layers, and the layer sizes are ﬁxed to be 32/64/128 and 50/100/50 for the encoder and decoder, respectively.
Note that the last layer of the
[Table]
decoder ( l = L ) is a linear full connection layer which transforms the sparse latent variables to dense for adjacency reconstruction.
Fig.
3(a) shows the link prediction results for increasing numbers of layers
[Table]
and the performance of our method is continuously rising until more than 4 layers.
The HLSM stochastic decoder requires prior distributions to generate latent variables.
Speciﬁcally, we employ a standard Normal prior for the latent positions z ( l ) i , i.e. the variance σ ( l ) i = 1 .
For the community membership s ( l ) i , we use the IBP prior to infer community numbers given the truncation
G l (i.e. layer sizes of the decoder) and a stick-breaking parameters v ∈ (0 , 1) .
Typically, a smaller v results in fewer active communities (i.e. non-zero elements of s ( l ) i ).
Fig.
3(b) presents the link prediction results of our method with different values of v on the Political blogs dataset, and our method performs best when v ∈ (0 . 9 , 1) .
Therefore, we set v = 0 .
9 to ensure the model fully exploring the potential community structure of graphs.
Last, we adopt ﬂat priors for the Dirichlet node random factors γ
i
and δ i , where all parameters are set as ξ ( l ) ig = ψ ( l ) ig = 1 /G l .
For inference, the temperature parameter λ of the Binary Concrete distribution is ﬁxed to 1.
All models are trained by 1,000-2,000 iterations with a learning rate of 0.01 on an RTX 2080 Ti GPU.
[Figure]
E L INK P REDICTION ON U NDIRECTED G RAPHS
Undirected graphs can be regarded as special cases in our framework, as an undirectional edge is treated as a bidirectional one.
In this section we provide some experimental results for link prediction
[Table] [Table]
on three real-world undirected graph datasets.
The baselines are the same as those in the experiments of directed graphs.
E. D ATASETS
We consider three real-world undirected graph datasets, namely NIPS12, Cora and Pubmed.
NIPS is a coauthor network of the authors in NIPS papers from volumes 1-12 Zhou (2015).
Cora is the same dataset used in the experiments for directed link prediction, except that the directions of edges are neglected here.
Pubmed is a citation network of scientiﬁc publications Sen et al.
Both
Cora and Pubmed datasets contain sparse bag-of-words feature matrices, which are used as node attributes.
The descriptive statistics of the datasets are summarized in Table 5.
E. L INK PREDICTION RESULTS
The results of link prediction on undirected graphs are presented in Table 6.
On all datasets, our
DLSM achieves the best or comparative performances with the state-of-the-art baselines for undirected graphs.
F M ORE D ETAILS FOR A BLATION S TUDY
We consider ﬁve variants of our proposed model.
Here we provide more details about these variants.
-HLSM indicates to replace the HLSM decoder of our model using a plain VAE with the same numbers of layers.
Others are simpliﬁed variants by removing the corresponding components, as presented in Table 7.
In particular, - ∆ refers to set γ i
= δ i (thus the model has degenerated to an undirected form yet still involves degree heterogeneity of graphs), and -S- ∆ - Γ can be viewed as a variant of VGAE by changing the plain decoder with our proposed HLSM architecture.
The community detection results of ablation study is given in Table 8.
Generally, models including the community membership perform better than those without this component, and the full version
DLSM reaches the best performances on all datasets.
[Table]
[Table]
[Figure]
G V ISUALIZATIONS OF N ODE R EPRESENTATIONS
Latent position We leverage a 2D t-SNE projection (Van der Maaten & Hinton, 2008) to visualize the learned latent positions for the Emails dataset.
As comparisons, we also illustrate the node representations learned by DGLFRM and LGVG, both of which also consider the community structure but overlook the degree heterogeneity of graphs.
The transformed latent variables learned by these three models are plotted in Fig.
It is clearly seen that our model performs best in ﬁtting such directed graphs
[Figure]
Reverse node random factors
The pairwise node random factors γ
i and δ i are supposed to measure the heterogeneity of out-degrees and in-degrees, respectively, which typically follow the power-law (Barab´asi & Albert, 1999).
Fig. 5(a) and (b) present the probability density distributions (PDD) of the node degrees and reverse random factors learned by DLSM on the political blogs network.
It seems that the degree distributions are well ﬁtted by the reverse node random factors, indicating that our DLSM can well represent the degree heterogeneity via these latent variables.
Furthermore, Fig. 5(c) illustrates the complementary cumulative distributions (CCD) of the random factors.
As can be seen, the logarithm CCD of both γ i and δ i are approximately linear, with different slopes though.
This shows that the proposed Dirichlet latent variables are ﬂexible enough to accommodate the power-law distribution of degrees.
[Figure]