Variational Graph Auto-Encoders for Heterogeneous Information Network
Abstract
Heterogeneous networks, where nodes and their attributes denote real-world entities and links encode relationships between entities, are ubiquitous in many applications, including social media, recommender systems, biological networks, molecular structures, and brain networks.
Such applications typically involve node classification or link prediction based on the attributes of the nodes and their connections in the graph.
Graph neural networks naturally model the topological structures of homogeneous graphs by learning low-dimensional embeddings of nodes.
However, heterogeneous graphs with various types of nodes and links pose significant challenges in learning efficient low-dimensional embeddings.
Existing methods for learning such representation rely on computationally expensive sampling of meta-paths and heuristic adaptation of models developed for homogeneous networks.
Therefore, the computational inefficiency and the generalization performance of such methods are frequently criticized and scrutinized, preventing their adoption in solving real-world problems.
To mitigate such issues, we present three variants of graph variational autoencoder models for heterogeneous networks that avoid the computationally expensive sampling of meta-paths and maintain uncertainty estimates of node embeddings that help with better generalization.
We report the results of experiments on link prediction using three different real-world benchmark heterogeneous network data sets that show that the proposed methods significantly outperform state-of-the-art baselines.
Introduction
Graph-structured data, which naturally encodes the relations between individuals, are ubiquitous in many real world applications, including e-commerce [1–4], social networks [5–7] and molecular structures [8–10], among others.
Such data lend to a range of machine learning problems, including node classification [11] and link prediction [12].
Graph embedding techniques [13] aim to construct low-dimensional information-preserving representations of graph-structured data.
The past several decades have seen an explosion of interest in algorithms for mining graph-structured data [14–16].
Of particular interest are Graph Neural Networks (GNNs)
[17, 18], a kind of deep neural network [19] that offer state-of-the-art performance on representation learning from graph-structured data across many domains including recommender systems [20], bioinformatics [21], drug discovery [22], social network analysis [23] and material science [24], among others.
The first GNN methods to be introduced [25–29] focused on node and link classification problems in homogeneous networks, i.e., networks with only one type of node and links.
However, many realworld applications present heterogeneous graphs, i.e., graphs with multiple types of nodes and links.
For example, atoms in a chemical structure can be of different types, and atoms of different types may be linked by different types of chemical bonds (heterogeneous edges).
Similarly, a bibliographic graph with the author, paper, topic, and venue domains contains not only the homogeneous edges between papers (paper citation) but also heterogeneous edges that link authors and papers (authorship) or papers and venues (publication).
Such applications call for effective GNN methods that work with heterogeneous graphs.
Existing approaches for learning embeddings of heterogeneous graphs rely on sampling metapaths [30], i.e., sequences of node types encoding relations between node types; for learning node embeddings—different meta-paths express different semantic meanings.
Most existing work, e.g., [30], assumes that domain experts specify the meta-paths.
While recent work, e.g., Wan et al.
[31], offers approaches to discovering meta-paths, their flexibility comes at a high computational cost.
Extracting meta-paths from heterogeneous graphs often involves dealing with with the attendant loss of information.
Meta-paths fail to account for possible differences in the importance of heterogeneous links based on the types of nodes involved and their influence on nodes for which embedding is being generated.
Although to overcome this issue, there have been meta-path-free methods such as Hussein et al.
[32], particular conditions constrain them, i.e., it requires edges between the same types of nodes, and in the absence of such edges, it degenerates to homogeneous graph-based algorithm.
In what we propose below, we address all of the existing issues that deter the adoption of GNNs for complex heterogeneous graphs.
Put formally, the main contributions of our work are the following:
• We define the representation learning problem in heterogeneous graphs and highlight the deficiencies of the existing techniques, setting the stage for the proposed methods.
• We derive three novel heterogeneous graph embedding methods that directly utilize the information encoded by a heterogeneous graph.
These methods utilize a type-specific transformation to aggregate information from a heterogeneous neighborhood, attending to neighbors differently based on their type and influence and maintaining uncertainty in representing the nodes and attention coefficients.
• We evaluate the performance of the proposed models on the link-prediction tasks on three data sets with heterogeneous graphs – IMDB, DBLP, and AMiner.
We present the results of experiments that demonstrate the superiority of the proposed methods over the state-of-the-art baselines.
Additionally, we visually inspect the learned low-dimensional representations and analyze the effects of the models’ settings on the predicted neighborhood of the graph.
The remainder of the paper is organized as follows.
We outline the related works in Section 2.
Section 3 provides a formal description of the problem we intend to solve.
Section 4 elaborates on the assumptions and learning algorithms for the models proposed for heterogeneous graphs.
Sectionreports the empirical results with real-world data.
Finally, the conclusion and future works are listed in Section 6.
Related Work 2.1 Heterogeneous Graph Neural Networks (H-GNNs)
Most existing work on H-GNNs tend to utilize the meta-paths [30] to adapt current homogeneous graph embedding methods, such as Kipf and Welling [25], Perozzi et al.
[33], Grover and Leskovec [34], Wang et al.
[35], to work with heterogeneous graphs.
Metapath2vec [36] designs meta-paths to guide the random walks in a heterogeneous graph and then follows the skip-gram model to learn the latent representations of the vertices.
Inspired by Metapath2vec [36], Shi et al.
[37] propose
HERec to fuse different representations learned in the view of different meta-path schemes.
Both
Metapath2vec and HERec are optimized for link prediction.
To obtain optimum embeddings for a specific task, Chen and Sun [38] unify task-guided and linkageguided objectives to learn the heterogeneous graph embeddings.
Wang et al.
[39] introduce twolevel hierarchical attention to GNN, in which node-level attention captures the relations between neighboring nodes generated by one meta paths scheme, and semantic-level attention aggregates multiple meta-path schemes for each node in a graph.
Unlike previous meta path-based models that worked on the principle of guided-selective neighborhood aggregation, HetSANN [40] directly utilizes a HIN for more informative representations without using meta-paths and achieves better performance.
2.2 Variational Methods
Inspired by Kingma and Welling [41], VGAE [26] incorporates the advantages of a Bayesian approach for solving a link prediction problem by extending GCNs using a Variational Auto-Encoder [41].
Unlike previous works [33, 34, 42] on unsupervised learning with structured graph data, VGAE treats the latent representations of the nodes as random variables.
The posterior distribution associated with such random variables is approximated using factorized variational distribution, whose mean and variance are modelled by multi-layered GNNs.
The vanilla attention module uses key and query vectors to obtain attention coefficients such that each row of the attention coefficient matrix is normalized to yield a unity value.
BAM [43] is the stochastic version of the attention mechanism where the non-negative attention coefficient matrix values are treated as random variables.
Adding such uncertainty in the attention coefficients enables the underlying model to capture complicated dependencies in the target data distribution.
One can seamlessly incorporate any existing knowledge about the domain in the prior distribution of the attention weights.
Finally, the posterior of the attention coefficients lends a better basis for model analysis and uncertainty estimation.
Problem Statement
We define a heterogeneous graph G H =
( V , E , R ) , where V represents the set of nodes, E represents the set of edges and R denotes the types of relations that exist in such a graph.
We consider a graph with N = |V| number of nodes.
[Table]
layer l z i
Embedding of Node i h ( ℓ )
i
Hidden representation of node i at layer l
W ( ℓ ) ϕ ( i ) ,ϕ ( j )
Weight matrix for transforming
Nodes of type ϕ ( j ) to ϕ ( i ) at layer l a ( ℓ ) r
Attention weights for relation r θ ( ℓ ) ij
Attention Coefficient between node i and j f ()
Activation function ζ ()
Matrix Stacking/Concatenating Operator
Each node v ∈ V is assumed to belong to a type ϕ ( v ) =
t ∈ T where ϕ ( · ) :
V → T is the mapping from V to T .
An edge e ∈ E in such a graph is defined as a triplet e =
( i, j, r ) , where r ∈ R is the type of relationship between nodes i ∈ V and j ∈ V and is dictated by the tuple < ϕ
( i ) , ϕ ( j ) > .
Therefore, in general, we assume the existence of as many as |T |×|T | number of relationships in such a heterogeneous graph.
Each node in the graph is also endowed with a set of features.
For nodes of type t ∈ T , the set of features is compactly represented by the feature matrix X t ∈ R N t × F t , where
N t is the number of nodes of type t and F t is the number of the features for nodes of type t .
Further, A is the adjacency matrix representation of G H with undirected edges, such that A ij =
A ji and A ii = 1 (self loop).
The degree matrix D is defined as:
[Equation]
if otherwise
A normalized adjacency matrix is calculated using the following: ˆA
=
D − 1 / 2 AD − 1 / 2 .
Our objective is to impute the unobserved edges in the graph.
This is achieved by learning the latent representations { z i } of all the nodes, utilizing the structural information and the attributes of the nodes available.
The likelihood of an edge between node i and j is predicted using a transformation of the latent representations z i and z j , as we elaborate in the next section.
The key notations used in the paper are presented in Table 1 in the Appendix.
Models
This section introduces three new representation learning algorithms for heterogeneous graphs.
We recognize that nodes of various types may have sets of features with completely disparate semantic interpretations.
Arguably, nodes of different types may have unequal implicit importance.
Hence, we introduce an attention mechanism that can weigh the neighbor’s contributions differently to achieve prediction.
Furthermore, it is helpful to model the uncertainty associated with the latent representation.
The models introduced below accomplish all these objectives.
[Table]
× × [Table] ✓ [Table]
[Equation]
BAM
[Table]
[Equation]
✓ × ✓
4.1 Heterogeneous Graph Variational Auto Encoder (H-GVAE)
The Heterogeneous Graph Variational Autoencoder (H-GVAE) model introduces uncertainty in the latent representation of the nodes and resolves the problem associated with the disparity in the features associated with different sets of nodes adopting a type-specific transformation.
More formally, we aspire to compute the posterior distribution of the latent representation of the nodes p ( Z = { z i }|X , A ) , where z i is the latent vector corresponding to node i .
The prior distribution of the latent variables is assumed to be standard normal, i.e., p ( Z ) =
(cid:81)
Ni =1 p ( z i ) =
(cid:81)
Ni =1 N ( z i | 0 , I ) .
Since finding the true posterior is often difficult and intractable, we approximate it using a variational distribution of the following form:
[Equation]
The variational parameters µ = { µ i } and σ = { σ i } are computed by stacking the latent representa- tion matrices for each node type as given below:
[Equation]
Here, ζ ( · ) is the stacking operator and HGCN (Heterogeneous Graph Convolution Network) is a special kind of GNN layer that we elaborate next.
Using the reparametrization trick [41], one can sample from the variational posterior z i ∼ q ( z i |X , A ) as
[Equation]
While designing the l th layer of the Heterogeneous Graph Convolution Network (HGCN), we assume that the representation of node i is given by h ( ℓ ) i , which is updated according to the following convolution operation:
[Equation]
Here h (0) i is set to X i , the observed attributes of the i th node.
In what we propose above, we assume the existence of a weight matrix W ( ℓ ) ϕ
( i ) ,ϕ ( j ) whose objective is to transform the latent representation of the source node j to match the semantic interpretation of the target node i .
The learned representations of all nodes of type ϕ
( i ) are compactly preserved in the matrix H ( ℓ ) ϕ
( i ) .
Further H ( ℓ ) contains the [Equation] collection of all the matrices { H ( ℓ ) t } t ∈T .
A layer that employs Eq.
(3) is conveniently addressed as
Heterogeneous Graph Convolution Network (HGCN).
The transformation W ( ℓ ) ϕ ( i ) ,ϕ ( j ) makes HGCN fundamentally different from the standard update equation of other GNN algorithms [29].
Since the different set of nodes may live in disparate semantic space, it is necessary to bring the latent representation to the same semantic space before aggregating information from the neighbours.
For example, in an author-paper-venue network, the author’s attributes may not have any semantic correlation with those of the venues or the papers.
Hence, while aggregating information from the papers a given author has written, one needs to transform the latent representation sagaciously to resolve any incongruity for learning the representation of the author.
Note that such transformation is superfluous for a pair of nodes of a similar type and hence is not relevant for learning embedding in a homogeneous graph.
The decoder in HGVAE is an inner product model followed by a Sigmoid activation function.
It is derived from the loss function of Mikolov et al.
[44] and unsupervised loss of Hamilton et al.
[45], which encourages that nodes with high topological similarity must have similar representation:
[Equation]
The weight parameters W ( ℓ ) ϕ
( i ) ,ϕ ( j ) are learned by optimizing the Evidence Lower Bound (ELBO) as given below:
[Equation]
The H-GVAE model successfully resolves the problem of utilizing information from HIN for learning lower-dimensional representations.
Further, for learning more expressive representations, motivated by Hong et al.
[40] (HetSANN), we introduce a type-aware attention mechanism to implicitly weigh the heterogeneous neighbours of a node in the propagation rule.
The HetSANN Graph Variational
Autoencoder (HetSANN-GVAE) model is similar to H-GVAE in the way it maintains uncertainty in the latent representation.
The only difference in this model is that HetSANN layers are used to calculate the parameters of the variational distribution instead of HGCN layers.
Thus, Eq.
(2) changes and assumes the following form:
[Equation]
Corresponding to each relation r , we compute the attention coefficients η ( ℓ ) ij , which are parameterized by attention weights a ( ℓ ) r , and is very similar to the attention mechanism used in NLP applications.
The key and query vectors in an attention head are the node features and their neighbourhood features in a relation.
[Equation]
where
[Equation]
h [Equation] Here η ( ℓ ) ij indicates the importance of node j to node i corresponding to the relation r ; calculated using the bi-linear scoring function given in Eq.
For simplicity and brevity of the notations,
Eq.
(6) assumes that there is only one type of edge/relation between ϕ ( j ) and ϕ ( i ) .
It is possible to accommodate multiple types of edges in this framework.
Using Eq.
(6) we obtain attention coefficients between every pair of nodes in relation r , irrespective of the pair having an edge between them.
We aim to maintain attention coefficients only between nodes and their neighbors; hence we use the adjacency matrix as a mask to inject the structural information into the attention coefficients.
Therefore, { η ( ℓ ) i,j } j is computed for all nodes j ∈ N i , where N i are the neighbour nodes of i and η ( ℓ ) ij =
if j ̸∈ N i . { η ( ℓ ) ij } j values are normalized using softmax function across all neighbors of i for easy comparability to yield the following attention coefficients:
[Equation]
The implicit weighting between nodes obtained using type-aware attention is used for aggregating the information.
Unlike Kipf and Welling [25], Wu et al.
[27] and the aforementioned H-GVAE, where information from neighbours is aggregated using a pre-computed normalized adjacency matrix, HetSANN-GVAE uses the trainable attention coefficients between nodes aiding the model’s performance.
Ergo, the latent representation are aggregated using θ ( ℓ ) ij instead of ˆA and (3) changes to:
[Equation]
Here, h ( ℓ ) i represents the output from only one attention head.
The model can accommodate multiple attention heads resulting in multiple h ( ℓ ) i .
These representations can be concatenated or added as suggested by Vaswani et al.
[46] for improving the stability of the attention mechanism.
The inference model, generator model and the ELBO remain the same as Eq.
(1), Eq.
(4) and Eq.
4.3 HetSANN-BAM-GVAE
Adding uncertainty to the attention weights helps the model incorporate complex dependencies in the data distribution to obtain better predictive performance.
In a heterogeneous graph, adding uncertainty in attention modules is propitious to model intricate dependencies stemming from different semantic relations present in the graph.
Hence, we further improve the HetSANN aggregation by integrating it with probabilistic attention modules as in Fan et al.
In HetSANN-BAM-GVAE, we consider a Bayesian model with a prior distribution p ( S ) and likelihood p ( A | Z ) ; where S is the set of attention coefficients in the model.
Hence, θ ( ℓ ) ij coefficients of the HetSANN layer change to their stochastic counterparts, i.e., HetSANN BAM layer with S ( ℓ ) ij coefficients.
The mechanism for calculating S ( ℓ ) ij remains the same as in Eq.
(6), but a prior belief is incorporated in the weights, which helps in regularization.
In an attention mechanism, importance weights are calculated between the key-query vector pairs.
These weights are non-negative and follow simplex constraints.
Hence, the prior distribution over such weights needs to enforce such constraints on the samples.
A natural choice for such a prior is the Dirichlet distribution.
Unfortunately, Dirichlet distribution is non-reparameterizable and hence cannot be used in a model that uses gradient descent-based optimization.
Fan et al.
[43] suggest a reparameterizable, non-negative distribution, such as Weibull or Log-Normal, to generate the samples and normalize subsequently to satisfy the simplex constraint.
We use the Weibull distribution, with parameters k and λ , as the variational distribution.
The Gamma distribution, with parameters α and β , is used as a prior.
The parameter k in the Weibull variational posterior is the global hyper-parameter for the attention coefficients.
Since we do not use a contextual prior [43], α and β are kept fixed hyper parameters for p ( S ) .
Using an amortized variational inference, λ is computed using:
[Equation]
where Γ is the Gamma function.
Note that we use a Weibull distribution because it is reparametrizable and produces non-negative samples.
[Equation]
The sampled values { ˆ S ( ℓ ) ij } j are normalized using the softmax activation to satisfy the simplex constraint of the attention coefficients i.e. (cid:80) j ˆ S ( ℓ ) ij = 1 .
These samples are then used in the propagation rule for the HetSANN BAM layer:
[Equation]
Eq.
(8) is the operation which is computed in the HetSANN BAM layer.
After stacking multiple layers of representations, we obtain the final representation Z :
[Equation] BAM ( X , A )) .
Since the Weibull distribution closely resembles the Gamma distribution and the KL divergence of the
Weibull distribution from a Gamma distribution has an analytical form [47], the Gamma distribution is used as a prior:
[Equation]
where γ is the Euler’s constant.
Putting it all together, the ELBO for the model is:
[Equation]
Here, D KL [ q ( S ) ∥
p ( S )] has a non-analytical form because of the dependencies between different stochastic layers.
Hence, the joint distributions p ( S ) and q ( S ) are decomposed into a sequential product of conditionals which have an analytic solution.
These conditionals are derived using the dependencies between the stochastic layers and has the form q ( S ( ℓ ) | S (1: ℓ − 1) ) and p ( S ( ℓ ) | S (1: ℓ − 1) ) .
This alternative tractable KL-Divergence is represented using Λ and results in a semi-analytical form of the ELBO as given below:
[Equation]
The generator model remains the same as Eq.
The ELBO is optimized with respect to all the parameters of the model.
Experiments
In this section, we present the experimental results for link prediction using each of the models presented in Section 4 on three heterogeneous graph data sets.
The utility of such models can only be grooved on when compared to a strong baseline.
5.1 Datasets
We perform experiments with three different datasets – IMDB, DBLP, and AMiner.
These datasets also come with labels of the nodes.
We ignore such labels since we aim to impute the edges and learn the embeddings based on the adjacency graph.
In a future extension of our work, we may incorporate such labels to construct a supervised loss function for better link prediction performance.
To reduce memory footprint and computational overhead, we further prune the DBLP and the AMiner datasets by randomly removing a fraction of the nodes.
• IMDB is an online database about movies that contains three types of nodes; 4278 Movies (M), 2081 Directors (D), and 5257 Actors (A).
The features of the nodes in this data set are represented as bag-of-words, containing the key-words related to a specific type of node.
The IMDB dataset contains only two types of edges; 4278 Movie-Director edges and 12828 Movie-Author edges.
For meta-path based schemes we have used MAM , MDM , MDMAM , MAMDM , AMA and DMD .
• The DBLP dataset used for the experiments is a subset of the original dataset, a heterogeneous academic graph dataset extracted by Gao et al.
[48], Ji et al.
The original dataset contains 4057 Authors (A), 14328 Papers (P),
Terms (T), and 20 Venues (V) with 19645 Author-Paper edges, 85810 Paper-Term edges, and 14328 Paper-Venue edges.
The reduced dataset used in this paper contains 2303 Authors (A), 5328 Papers (P), 4007 Terms (T), and 9 Venues (V) withAuthor-Paper edges, 31740 Paper-Term edges, and 5328 Paper-Venue edges.
The features of the authors, papers, and terms all have bag-of-words representation, whose dimensions are reduced further using PCA [50].
For meta-path based schemes we have used APA , TPT , VPV , PAP , PTP , PVP , APTPA , PTPAPVP , TPAPVPT and APVPTPA .
• AMiner is a dataset extracted by Hong et al.
[40], where the authors remove the venue nodes from the AMiner academic graph, leaving only 8052 Authors (A) and 20201 Papers (P).
This dataset contains 31224 publishing relations (Author-Paper), 44551 citation relations (Paper-Paper), and 32029 collaboration relations (Author-Author).
Features of the papers are a bag of words representing the keywords.
Each author in this graph has five features – the number of citations, H-Index, P-index with equal A-index, and P-index with unequal A-index.
We further prune this graph, resulting in 3588 Authors (A) and 8129 Papers (P) with 7519 publishing relations (Author-Paper),citation relations (Paper-Paper) [Table] and 11852 collaboration relations (Author-Author).
Note that not all papers are connected to a legitimate author, making link prediction in such a noisy graph even more challenging.
Furthermore, out of the 3 datasets, this one is the most densely connected with only 2 types of nodes.
In such a graph, random walks based method usually perform really well; thus providing a challenge to non-random walks methods.
For meta-path based schemes we have used PP ,
PAP , PAPP , PPAP , PAPAP , AA , APA , APAA , AAPA and APAPA .
5.2 Baseline Methods
[Table]
• Node2vec [ is an algorithmic framework for representational learning on homogeneous graphs, which is a generalized version of
DeepWalk [
We apply it to our
datasets
by ignoring the heterogeneity of graph structure and dropping all node features.
After obtaining the representations for the nodes, edge features are generated using binary operators.
These edge features are then used in a logistic regression classifier to predict binary values indicating whether an edge is present.
Average, Hadamard, L-1, and L-2 operators are used as the binary operators, and the best results out of the four operators are presented in Table• HAN [39] is a heterogeneous GNN model that computes node representations from multiple metapath-based homogeneous graphs and uses the attention mechanism to combine them.
To generate edges from node representations, we use the inner product decoder from Eq.
Meta-path adjacency matrices are obtained by multiplying sub-type adjacency matrices.
E.g., in the IMDB dataset, the MAM meta-path matrix is obtained by multiplying the adjacency matrix MvA and AvM (Transpose of MvA). • GVAE [26] Another way to achieve a strong baseline is to treat the heterogeneous graph as a homogeneous graph and employ the existing homogeneous GNN models.
To that end, we use a homogeneous GVAE as a baseline model.
Note that this model assumes that the features of all the nodes lie in the same semantic space and are of the same dimensionality.
However, as discussed in the previous sections, the nodes in HIN may have attributes of different sizes based on the nodes’ type.
To resolve such discordance, we apply zero-padding to features to equal dimensionality, which is also a prevalent practice with CNN filters in computer vision.
5.3 Experiment Setup
The edges of the datasets are randomly split into training, test, and validation with 85%, 10%, and 5% split percentages .
Node2vec models parameters are the same for all the datasets.
For Node2vec we set the walklength to 50 and number of walks to 10.
The window size is set to 10 and the embedding dimension size is 128 for both Node2vec.
For GNN based models, we observe overfitting in the IMDB dataset if we use more than one layer.
For both DBLP and AMiner, however, we use two layers.
For attention based models, We assign three different attention heads for each semantic relation for the models that use attention.
For DBLP and AMiner, we use two layers with five attention heads in the first layer and three attention heads in the second layer.
Each attention head’s values are concatenated in each set if the layer is not the last; otherwise, the values get summed up.
[Figure]
For GNN based models, we set the output dimension size tofor all the experiments with the IMDB dataset.
We set the hidden layer’s size to 32 and the output dimension to 16 for the experiments with DBLP and AMiner.
For the HetSANN-BAM-GVAE model, the parameters of the Gamma prior distribution are α = 1 e −and β
= 1 e − 15 .
For the Weibull variational posterior, k is set to 10 .
The learning rates for GVAE and H-GVAE are 0 .
01 and for
HetSANN-GVAE and HetSANN-BAM-GVAE are 0 .
All the models are trained using the Adam optimizer [51].
Numbers from Table 3 show mean results for 15 runs with random initialization on fixed dataset splits.
5.4 Link-Prediction Results
The results successfully demonstrate the state-of-the-art performance of the models for all the datasets.
We use the same parameters that produce the best results on the validation set to predict links on the test set.
For IMDB and DBLP datasets, random walk-based methods prove to be inferior to graph neural network-based methods as they do not use node features.
GVAE treats a heterogeneous graph as a homogeneous graph, and due to such loss of semantic and structural information, GVAE produces unsatisfactory results compared to the rest of the models.
Moreover, accumulating information from the neighbors with zero-padded feature vectors is suboptimal, as the identity of the source node is lost in such transformation.
Even though HAN utilizes attention, its performance is almost on-par with the H-GVAE model, which does not use attention.
Thus, transformations that directly utilize raw heterogeneous information of the graph and map features into neighboring feature spaces significantly differ in performance across the datasets.
HetSANN-GVAE proves to be an improvement over
H-GVAE.
It utilizes the basic heterogeneous information by handling individual semantic relations in one dedicated attention head, and Bayesian attention modules act as an additional regularizer that further improves the model and proves to be the best performing model.
The AMiner dataset confuses most models as it is a partially homogeneous and heterogeneous graph.
We can see this by looking at the performance of HAN, which underperforms all the other models.
It could be the case that HAN needs a better and perhaps longer meta-path, but finding such a meta-path is itself a notoriously difficult task.
Node2vec performs reasonably well compared to the best performing model, i.e., the HetSANN-BAM-GVAE model.
The main observation deduced from the T-SNE projections is about the discriminative power of the models—the GVAE model clusters majority of the nodes whose edges have been unsuccessfully predicted towards the center.
After introducing transformations for the heterogeneous features with variational attention modules improves the discriminative power as the distance between the clusters increases, an indicator of the ability to distinguish between the different neighborhood structures.
Conclusion
To the best of our knowledge, ours is the first work that presents benchmarking results for link prediction on heterogeneous graphs using variational graph autoencoders.
We notice substantial improvements in performance than the current state-of-the-art models used for link prediction by introducing uncertainty in node representation, attention coefficients and by adopting type-specific attention variables.
In general, the performance of the proposed models is limited by choice of the decoder, the prior distribution and the capacity of the variational distribution.
One can propose better decoders [52, 53], use a more complicated form of the prior [54] or endow the variational distribution with an implicit one [55–57].
We leave these as exciting directions for our future work.