Variational Graph Auto-Encoders for Heterogeneous Information Network
Abstract
Heterogeneous Information Network (HIN), where nodes and their attributes denote real-world entities and links encode relationships between entities, are ubiquitous in many applications.
The presence of multiple types of nodes and links pose significant challenges to the state-of-the-art methods for learning node embeddings from heterogeneous graphs.
To address these challenges, we consider three variants of graph variational autoencoder models for heterogeneous networks that avoid the computationally expensive sampling of meta-paths. The proposed methods also maintain uncertainty estimates of node embeddings that help improve generalization performance.
We report the results of experiments on link prediction using three different real-world heterogeneous network benchmark data sets that show that the proposed methods significantly outperform state-of-the-art baselines.
Introduction
Graph-structured data, which naturally encodes the relations between individuals, are ubiquitous in many real world applications, including e-commerce [1–4], social networks [5–7] and molecular structures [8–10], among others.
Such data present a range of machine learning problems, including node classification [11] and link prediction [12].
Graph embedding techniques [13] aim to construct low-dimensional information-preserving representations of graph-structured data.
The past several decades have seen an explosion of interest in algorithms for mining graph-structured data [14–16].
Of particular interest are Graph Neural Networks (GNNs)
[17, 18], a type of deep neural networks [19] that offer state-of-the-art performance on representation learning from graph-structured data. Such techniques have found applications across many domains including recommender systems [20], bioinformatics [21], drug discovery [22], social network analysis [23] and material science [24], among others.
The earliest GNN methods [25–29] focused on node and link classification problems in homogeneous networks, i.e., networks with only one type of nodes and links.
However, many real-world applications present heterogeneous graphs, i.e., graphs with multiple types of nodes and links.
For example, atoms in a chemical structure can be of different types, and atoms of different types may be linked by different types of chemical bonds (heterogeneous edges).
Similarly, a bibliographic graph consisting of multiple types of nodes, namely, authors, papers, topics, and venues contain not only the edges between nodes of the same type, e.g., papers (paper citation) but also edges that link nodes of different types e.g., authors and papers (authorship) or papers and venues (publication).
Such applications call for effective GNN methods that work with heterogeneous graphs.
Existing approaches for learning embeddings of heterogeneous graphs rely on sampling metapaths [30], i.e., sequences of node types encoding relations between node types; for learning node embeddings—different meta-paths express different semantic meanings.
Most existing work, e.g., [30], assumes that domain experts specify the meta-paths.
While recent work, e.g., Wan et al.
[31], offers approaches to discovering meta-paths, their flexibility comes at a high computational cost.
Extracting meta-paths from heterogeneous graphs often involves dealing with with the attendant loss of information.
Meta-paths fail to account for possible differences in the importance of heterogeneous links based on the types of nodes involved and their influence on nodes for which embedding is being generated.
Although some meta-path-free methods have been developed recently, e.g., Hussein et al.
[32], they generally limit that the edges to be between nodes of the same type. Furthermore, heterogeneous graphs are often constructed from real-world data, e.g., collections of articles in the case of citation networks.
Both the data from data and the algorithms used to extract heterogeneous graphs can introduce noise, and methods used to learn node embeddings from such graphs should be sufficiently robust to cope with noise.
Against this background, this paper explores three node embedding techniques for heterogeneous graphs.
The key contributions of this paper are as follows:
We precisely define the problem of learning node embeddings from heterogeneous graphs and highlight some of the deficiencies of the existing methods.
We introduce three novel heterogeneous graph embedding methods that directly utilize the information encoded by a heterogeneous graph.
These methods utilize a node and link type-aware transformations to aggregate information from local neighborhoods around nodes, taking into account, the heterogeneous nodes and links in the graphs. We maintain uncertainty associated with the learned embeddings of nodes to improve the noise-tolerance of the methods.
We evaluate the performance of the proposed models on the link-prediction task on three heterogeneous graphs derived from three real-world data sets (IMDB, DBLP, and AMiner).
We present the results of experiments that demonstrate the superiority of the proposed methods over the state-of-the-art baselines.
Additionally, we visually inspect the learned low-dimensional representations and analyze the effects of the models’ settings on the predicted neighborhood of the graph.
The rest of the paper is organized as follows.
Section 2 summarizes related work.
Section 3 provides a precise description of the problem of learning node embeddings from heterogeneous graphs. Section 4 elaborates on our assumptions and describes the learning algorithms .
Section 5 compares the the proposed methods with that of state-of-the-art baselines using several real-world data sets.
Sectionconcludes the paper with a summary and an outline of some directions for further research.
Related Work 2.1 Heterogeneous Graph Neural Networks
Most existing work on heterogeneous graph neural networks (H-GNNs) tend to utilize the metapaths [30] to adapt current homogeneous graph embedding methods, such as Kipf and Welling [25], Perozzi et al.
[33], Grover and Leskovec [34], Wang et al.
[35], to work with heterogeneous graphs.
Metapath2vec [36] designs meta-paths to guide the random walks in a heterogeneous graph and then follows the skip-gram model to learn the latent representations of the vertices.
Inspired by Metapath2vec [36], Shi et al.
[37] propose HERec to fuse different representations learned in the view of different meta-path schemes.
Both Metapath2vec and HERec are optimized for link prediction.
To optimize the embeddings for a specific task, Chen and Sun [38] unify task-guided and linkageguided objectives to learn the heterogeneous graph embeddings.
Wang et al.
[39] introduce twolevel hierarchical attention to GNN, in which node-level attention captures the relations between neighboring nodes encoded by the meta paths, and semantic-level attention aggregates multiple meta-paths for each node in the graph.
Unlike previous meta path-based models that perform selective aggregation of information from node neighborhoods, HetSANN [40] directly utilizes a HIN for more informative representations without using meta-paths to improve both efficiency and performance of the resulting methods relative to state-of-the-art baselines.
2.2 Variational Methods
Inspired by Kingma and Welling [41], VGAE [26] takes a Bayesian approach to link prediction problem by extending GCNs using a Variational Auto-Encoder [41].
Unlike previous works [33, 34, 42] on unsupervised learning with structured graph data, VGAE treats the latent representations of the nodes as random variables.
The posterior distribution associated with such random variables is approximated using factorized variational distribution, whose mean and variance are modelled by multi-layered GNNs.
BAM [43] is a stochastic version of the basic the attention mechanism wherein the non-negative elements of the attention matrix are used to represent their uncertainty.
Problem Statement
[Table]
We define a heterogeneous graph G H =
( V , E , R ) , where V represents the set of nodes, E represents the set of edges and R denotes the types of relations that exist in such a graph.
We consider a graph with N = |V| number of nodes.
Each node v ∈ V is assumed to have a type ϕ
( v ) =
t ∈ T where ϕ ( · ) :
V → T is the mapping from V to T .
An edge e ∈ E in such a graph is defined as a triplet e =
( i, j, r ) , where r ∈ R is the type of relationship between nodes i ∈ V and j ∈ V and is dictated by the types of the nodes linked by the edge, i.e., < ϕ
( i ) , ϕ ( j ) > .
Each node in the graph is associated with set of node features.
For nodes of type t ∈ T , the set of features is compactly represented by the feature matrix X t ∈ R N t × F t , where N t is the number of nodes of type t and F t is the number of the features for nodes of type t .
Further, A denotes the adjacency matrix representation of G H with undirected edges, such that A ij =
A ji and A ii = 1 (self loop).
The degree matrix D is given by:
[Equation]
A normalized adjacency matrix is given by: ˆA =
[Equation]
Our objective is to impute the unobserved edges in the graph.
This is achieved by learning the latent representations { z i } of all the nodes, utilizing the structural information and the attributes of the nodes.
The likelihood of an edge between node i and j is predicted using a transformation of the latent representations z i and z j , as explained in the next section.
The key notations used in the paper are presented in Table 1.
Node Embedding Methods for Heterogeneous Graphs
This section introduces three new representation learning algorithms for heterogeneous graphs.
We observe that nodes of different types may be described by different sets of features with disparate semantic interpretations.
Furthermore, the roles and the importance of nodes may vary according to their type.
Hence, we introduce an attention mechanism that can weigh the neighbor’s contributions differently.
Furthermore, it is helpful to model the uncertainty associated with the latent representation.
The models introduced below is designed to achieve all these objectives.
[Equation]
[Table]
[Equation]
× × ✓
[Table]
A
[Table]
[Equation]
× ✓
[Table]
A
[Table]
[Equation]
[Table] [Equation] ✓ ✓ [Equation]
[Table]
[Equation]
× ✓
4.1 Heterogeneous Graph Variational Auto Encoder
The Heterogeneous Graph Variational Autoencoder (H-GVAE) model incorporates uncertainty in the latent representation of the nodes and uses a node type specific transformation to accommodate the differences in features associated with different types of nodes.
More precisely, we seek to compute the posterior distribution of the latent representation of the nodes p ( Z = { z i }|X , A ) , where z i is the latent vector corresponding to node i .
The prior distribution of the latent variables is assumed to be standard normal, i.e., p ( Z ) =
(cid:81)
Ni =1 p ( z i ) =
(cid:81)
Ni =1 N ( z i | 0 , I ) .
Because computing the true posterior is difficult and intractable, we approximate it using a variational distribution of the following form:
[Equation]
The variational parameters µ = { µ i } and σ = { σ i } are computed by stacking the latent representa- tion matrices for each node type as given below:
[Equation]
Here, ζ ( · ) is the stacking operator and HGCN (Heterogeneous Graph Convolution Network) is a special kind of GNN layer that we elaborate next.
Using the reparametrization trick [41], one can sample from the variational posterior z i ∼ q ( z i |X , A ) as
[Equation]
While designing the l th layer of the Heterogeneous Graph Convolution Network (HGCN), we assume that the representation of node i is given by h ( ℓ ) i , which is updated as follows:
[Equation]
Here h (0) i is set to X i , the observed attributes of the i th node.
We assume the existence of a weight matrix W ( ℓ ) ϕ
( i ) ,ϕ ( j ) for transforming the latent representation of the source node j to match the semantic interpretation of the target node i .
The learned representations of all nodes of type ϕ
( i ) are compactly preserved in the matrix H ( ℓ ) ϕ
( i ) .
Further H ( ℓ ) contains the collection of all the matrices { H ( ℓ ) t } t ∈T .
A layer that employs Eq.
(3) is conveniently addressed as Heterogeneous Graph Convolution Network (HGCN).
The transformation W ( ℓ ) ϕ ( i ) ,ϕ ( j ) makes HGCN fundamentally different from the standard update equation of other GNN algorithms [29].
Since the different set of nodes may live in disparate semantic space, it is necessary to map the latent representation to a shared semantic space before aggregating information from the neighbors.
For example, in an author-paper-venue network, the author’s attributes may not have any semantic correlation with those of the venues or the papers.
Hence, while aggregating information from the papers a given author has written, it is necessary to transform the latent representation of the papers in the process of learning the latent representation of the author.
Note that such transformation is superfluous for a pair of nodes of a similar type and hence is not relevant for learning embedding in a homogeneous graph.
The decoder in HGVAE is an inner product model followed by a Sigmoid activation function.
It is derived from the loss function of Mikolov et al.
[44] and unsupervised loss of Hamilton et al.
[45], which encourages that nodes with high topological similarity must have similar representation:
[Equation]
The weight parameters W ( ℓ ) ϕ
( i ) ,ϕ ( j ) are learned by optimizing the Evidence Lower Bound (ELBO) as given below:
[Equation]
4.2 HetSANN-GVAE
The H-GVAE model successfully resolves the problem of utilizing information from HIN for learning node embeddings.
Further, motivated by Hong et al.
[40] (HetSANN), we introduce a type-aware attention mechanism to implicitly weigh the heterogeneous neighbours of a node in the propagation rule.
The HetSANN Graph Variational Autoencoder (HetSANN-GVAE) model is similar to H-GVAE in that it maintains uncertainty in the latent representation, the only difference being that HetSANN layers are used to calculate the parameters of the variational distribution instead of HGCN layers.
Thus, the preceding Eq.
(2) now assumes the following form:
[Equation]
Corresponding to each relation r , we compute the attention coefficients η ( ℓ ) ij , which are parameterized by attention weights a ( ℓ ) r , and is very similar to the attention mechanism used in NLP applications.
The key and query vectors in an attention head are the node features and their neighbourhood features in a relation.
[Equation]
where
[Equation]
Here η ( ℓ ) ij indicates the importance of node j to node i corresponding to the relation r ; calculated using the bi-linear scoring function given in Eq.
For simplicity, we have assumed in Eq.
(6) that there is only one type of edge/relation between nodes of type ϕ ( j ) and ϕ ( i ) .
However, it is straightforward to generalize our framework to accommodate multiple types of edges between nodes of type ϕ ( j ) and ϕ ( i ) .
Eq.
(6) yields attention coefficients between each pair of nodes in relation r , regardless of whether an edge exists between them.
We aim to maintain attention coefficients only between nodes and their neighbors in the graph; hence we use the adjacency matrix as a mask to impose structural information about the graph into the attention coefficients.
Therefore, { η ( ℓ ) i,j } j is computed for all nodes j ∈ N i , where N i are the neighbour nodes of i and η ( ℓ ) ij =
if j ̸∈ N i . { η ( ℓ ) ij } j values are normalized using softmax function across all neighbors of i yielding the following definition of the attention coefficients:
[Equation]
The implicit weighting between nodes obtained using type-aware attention is used to aggregate the information from each node’s neighbors in the graph in the node’s embedding.
Unlike Kipf and Welling [25], Wu et al.
[27] and the aforementioned H-GVAE, where information from neighbours is aggregated using a pre-computed normalized adjacency matrix, HetSANN-GVAE uses the trainable attention coefficients between nodes aiding the model’s performance.
Hence, the latent representation are aggregated using θ ( ℓ ) ij instead of ˆA and (3) changes to:
[Equation]
,ϕ ( j ) h ( ℓ − 1) j ) .
Here, h ( ℓ ) i represents the output from only one attention head.
The model can accommodate multiple attention heads resulting in multiple h ( ℓ ) i .
These representations can be concatenated or added as suggested by Vaswani et al.
[46] for improving the stability of the attention mechanism.
The inference model, generator model and the ELBO remain the same as Eq.
(1), Eq.
(4) and Eq.
4.3 HetSANN-BAM-GVAE
Adding uncertainty to the attention weights allows the model to accommodate the node type specific differences in the semantics of nodes and links in a heterogeneous graph.
Hence, we further improve the HetSANN aggregation by integrating it with probabilistic attention modules Fan et al.
In HetSANN-BAM-GVAE, we consider a Bayesian model with a prior distribution p ( S ) and likelihood p ( A | Z ) ; where S is the set of attention coefficients in the model.
Hence, θ ( ℓ ) ij coefficients of the HetSANN layer are replaced by their stochastic counterparts, i.e., HetSANN BAM layer with S ( ℓ ) ij coefficients.
The mechanism for calculating S ( ℓ ) ij remains the same as in Eq.
(6), but a prior belief is incorporated in the weights, which helps in regularization.
In an attention mechanism, importance weights are calculated between the key-query vector pairs.
These weights are non-negative and follow simplex constraints.
Hence, the prior distribution over such weights needs to enforce such constraints on the samples.
A natural choice for such a prior is the Dirichlet distribution.
Unfortunately, Dirichlet distribution is non-reparameterizable and hence cannot be used in a model that uses gradient descent-based optimization.
Fan et al.
[43] suggest a reparameterizable, non-negative distribution, such as Weibull or Log-Normal, to generate the samples and normalize subsequently to satisfy the simplex constraint.
We use the Weibull distribution, with parameters k and λ , as the variational distribution.
The Gamma distribution, with parameters α and β , is used as a prior.
The parameter k in the Weibull variational posterior is the global hyper-parameter for the attention coefficients.
Since we do not use a contextual prior [43], α and β are kept fixed hyper parameters for p ( S ) .
Using an amortized variational inference, λ is computed using:
[Equation]
where Γ is the Gamma function.
Note that we use a Weibull distribution because it is reparametrizable and produces non- negative samples.
[Equation]
The sampled values { ˆ S ( ℓ ) ij } j are normalized using the softmax activation to satisfy the simplex constraint of the attention coefficients i.e. (cid:80) j ˆ S ( ℓ ) ij = 1 .
These samples are then used in the propagation rule for the HetSANN BAM layer:
[Equation]
Eq.
(8) is the operation which is computed in the HetSANN BAM layer.
After stacking multiple layers of representations, we obtain the final representation Z :
[Equation]
Since the Weibull distribution closely resembles the Gamma distribution and the KL divergence of the
Weibull distribution from a Gamma distribution has an analytical form [47], the Gamma distribution is used as a prior:
[Equation]
where γ is the Euler’s constant.
Putting it all together, the ELBO for the model is:
[Equation]
Here, D KL [ q ( S ) ∥
p ( S )] has a non-analytical form because of the dependencies between different stochastic layers.
Hence, the joint distributions p ( S ) and q ( S ) are decomposed into a product of conditionals which have an analytic solution.
These conditionals are derived using the dependencies between the stochastic layers and has the form q ( S ( ℓ ) | S (1: ℓ − 1) ) and p ( S ( ℓ ) | S (1: ℓ − 1) ) .
This alternative tractable KL-Divergence is represented using Λ and results in a semi-analytical form of the ELBO as given below:
[Equation]
The generator model remains the same as that specified by Eq.
The ELBO is optimized with respect to all of the parameters of the model.
Experiments
In this section, we empirically compare the performance of each of the models presented in Sectionwith state-of-the-art baselines on the link prediction task using three heterogeneous graph benchmark data sets.
5.1 Benchmark Data
We perform experiments with three different data sets:IMDB, DBLP, and AMiner.
These data sets also come with labels of the nodes.
We ignore such labels since we aim to impute the edges and learn the embeddings based on the adjacency graph (It would be interesting to incorporate information supplied by the node labels to construct a supervised loss function for link prediction).
To reduce memory footprint and computational overhead, we further prune the DBLP and the AMiner data sets by randomly removing a fraction of the nodes.
The details of the IMDB datasets and the pruned
DBLP and AMiner datasets are as follows:
IMDB is an online database about movies that contains three types of nodes; 4278 Movies (M), 2081 Directors (D), and 5257 Actors (A).
The features of the nodes in this data set are represented as bag-of-words, containing the key-words related to a specific type of node.
The IMDB data set contains only two types of edges; 4278 Movie-Director edges andMovie-Author edges.
For meta-path based schemes we have used MAM , MDM , MDMAM , MAMDM , AMA and DMD .
The DBLP dataset used for the experiments is a subset of the original dataset, a heterogeneous academic graph data set extracted by Gao et al.
[48], Ji et al.
The original dataset contains 4057 Authors (A), 14328 Papers (P),
Terms (T), and 20 Venues (V) with 19645 Author-Paper edges, 85810 Paper-Term edges, and 14328 Paper-Venue edges.
The reduced data set used in this paper contains 2303 Authors (A), 5328 Papers (P), 4007 Terms (T), and 9 Venues (V) with 7127 Author-Paper edges, 31740 Paper-Term edges, andPaper-Venue edges.
The features of the authors, papers, and terms all have bag-of-words representation, whose dimensions are reduced further using PCA [50].
For meta-path based schemes we have used APA , TPT , VPV , PAP , PTP , PVP , APTPA , PTPAPVP , TPAPVPT and APVPTPA .
AMiner is a dataset extracted by Hong et al.
[40], where the authors remove the venue nodes from the AMiner academic graph, leaving only 8052 Authors (A) and 20201 Papers (P).
This data set contains 31224 publishing relations (Author-Paper), 44551 citation relations (Paper-Paper), and 32029 collaboration relations (Author-Author).
Features of the papers are a bag of words representing the keywords.
Each author in this graph has five features – the number of citations, H-Index, P-index with equal A-index, and P-index with unequal A-index.
We further prune this graph, resulting in 3588 Authors (A) and 8129 Papers (P) with 7519 publishing relations (Author-Paper), 7100 citation relations (Paper-Paper), and 11852 collaboration relations (Author-Author).
Note that not all papers are connected to a legitimate author, making link prediction in such a noisy graph even more challenging.
Furthermore, out of the 3 data sets, this one is the most densely connected with only 2 types of nodes.
In such a graph, random walks based method usually perform really well; thus providing a challenge to non-random walks methods.
For meta-path based schemes we have used PP , PAP , PAPP , PPAP , PAPAP , AA , APA , APAA , AAPA and APAPA .
5.2 Baseline Methods
We considered the following baseline methods:
Node2vec [34] is an algorithmic framework for representational learning on homogeneous graphs, which is a generalized version of DeepWalk [33].
We apply it to our datasets by ignoring the heterogeneity of graph structure and dropping all node features.
After obtaining the representations for the nodes, edge features are generated using binary operators.
These edge features are then used in a logistic regression classifier to predict binary values indicating whether an edge is present.
Average, Hadamard, L-1, and L-2 operators are used as the binary operators, and the best results out of the four operators are presented in Table2. HAN [39] is a heterogeneous GNN model that computes node representations from multiple meta-path-based homogeneous graphs and uses the attention mechanism to combine them.
To generate edges from node representations, we use the inner product decoder from Eq.
Meta-path adjacency matrices are obtained by multiplying sub-type adjacency matrices.
E.g., in the IMDB dataset, the MAM meta-path matrix is obtained by multiplying the adjacency matrix MvA and AvM (Transpose of MvA).
GVAE [26] We use a homogeneous GVAE as a baseline model. That is, we treat the heterogeneous graph as if it were a homogeneous graph.
Note that this model assumes that the features of all the nodes lie in the same semantic space and are of the same dimensionality.
However, as discussed in the previous sections, the nodes in HIN may have attributes of different sizes based on the nodes’ type.
To resolve such discordance, we equalize the dimensionality of features across node types by padding them with zeros as needed.
5.3 Experimental Setup
The edges of the datasets are randomly split into training, test, and validation with 85%, 10%, and 5% split percentages .
Node2vec models parameters are the same for all the datasets.
For Node2vec we set the walklength to 50 and number of walks to 10.
The window size is set to 10 and the embedding dimension size is 128 for both Node2vec.
For GNN based models, we observe overfitting in the IMDB dataset if we use more than one layer.
For both DBLP and AMiner, however, we use two layers.
For attention based models, We assign three different attention heads for each semantic relation for the models that use attention.
For DBLP and AMiner, we use two layers with five attention heads in the first layer and three attention heads in the second layer.
Each attention head’s values are concatenated in each set if the layer is not the last; otherwise, the values get summed up.
For GNN based models, we set the output dimension size to 32 for all the experiments with the IMDB data set.
We set the hidden layer’s size to 32 and the output dimension to 16 for the experiments with DBLP and AMiner.
For the HetSANN-BAM-GVAE model, the parameters of the Gamma prior distribution are α = 1 e − 17 and β = 1 e − 15 .
For the Weibull variational posterior, k is set to 10 .
The learning rates for GVAE and H-GVAE are 0 .
01 and for HetSANN-GVAE and HetSANN-BAM-GVAE are 0 .
All the models are trained using the Adam optimizer [51].
We use the same parameters that produce the best results on the validation set to predict links on the test set.
We report the mean performance on the test set obtained across 15 runs with random initialization on fixed data set splits.
5.4 Link Prediction Results
The results of our experiments are shown in table 3.
Table 3 shows that that on IMDB and DBLP data, random walk based methods which do not use node features perform substantially worse than the graph neural network based methods.
Among the graph based methods, GVAE, which treats a heterogeneous graph as a homogeneous graph, and therefore fails to make use of semantic and structural information provided by the heterogeneous graph, produces unsatisfactory results compared to the rest of the graph based models.
Even though
H-GVAE does not useattention whereas HAN does, the performance of H-GVAE is comparable to that of HAN on IMDB and DBLP data, and substantially superior to that of HAN on AMiner data.
HetSANN-GVAE substantially outperforms H-GVAE.
This suggests the effectiveness of utilizing the structural and semantic information in heterogeneous graphs in HetSANN-GVAE. HetSANN-BAMGVAE outperforms all other methods on all data sets. [Table]
[Figure] We further note that AMiner data presents challenges for most of the models due to the fact that it is partially homogeneous and partially heterogeneous graph. HAN underperforms all the other models on AMiner data.
It could be the case that HAN needs better and perhaps longer meta-path [Table] but specifying such a meta-path is itself a notoriously difficult task.
Interestingly, Node2vec performs reasonably well compared to the best performing model, i
[Table] e., the HetSANN-BAM-GVAE. The main observation deduced from the T-SNE projections is about the discriminative power of the models—the GVAE model clusters the majority of the nodes whose edges have been inaccurately predicted towards the center.
The introduction of transformations for the heterogeneous features and variational attention modules improves the discriminative power as the distance between the clusters increases, an indicator of the ability to distinguish between the different neighborhood structures.
Conclusion
We considered three variants of graph variational autoencoders for learning node embeddings from heterogeneous graphs.
Two of the three proposed variants outperform several state-of-the-art baselines on three real-world benchmark data sets on the link prediction task. Their superior performance can be attributed to their ability to incorporate the structural and semantic information supplied by heterogeneous graphs, to model the uncertainty of node embeddings, and to to exploit node type specific attention variables.
Further improvements are possible, e.g., by using better decoders [52, 53], or utilizing a better prior [54–58].
These possibilities offer some exciting directions for future work.
Acknowledgements
This work was supported in part by a grant to Suhang Wang and Vasant Honavar from the George
Mason University’s Criminal Investigations and Network Analysis Center funded by the Department of Homeland Security, and in part by a National Science Foundation Grant (IIS 2041759) to Vasant
Honavar.
This work has benefited from discussions with Suhang Wang.