C ORRELATED V ARIATIONAL A UTO -E NCODERS
netflix
A BSTRACT
Variational Audo-Encoders ( VAE s) are capable of learning latent representations for high dimensional data.
However, due to the i.i.d.
assumption, VAE s only optimize the singleton variational distributions and fail to account for the correlations between data points, which might be crucial for learning latent representations from dataset where a priori we know correlations exist.
We propose Correlated
Variational Auto-Encoders ( CVAE s) that can take the correlation structure into consideration when learning latent representations with VAE s. CVAE s apply a prior based on the correlation structure.
To address the intractability introduced by the correlated prior, we develop an approximation by the average of a set of tractable lower bounds over all maximal acyclic subgraphs of the undirected correlation graph.
Experimental results on matching and link prediction on public benchmark rating datasets and spectral clustering on a synthetic dataset show the effectiveness of the proposed method over baseline algorithms
I NTRODUCTION
Variational Auto-Encoders ( VAE s) (Kingma & Welling, 2014; Rezende et al., 2014) are a family of powerful deep generative models that learns stochastic latent embeddings for input data.
By applying variational inference on deep generative models, VAE s are able to successfully identify the latent structures of the data and learn latent distributions that can potentially extract more compact information that is not easily and directly obtained from the original data.
VAE s assume each data point is i.i.d.
generated, which means we do not consider any correlations between the data points.
This is a reasonable assumption under many settings.
However, sometimes we know a priori that data points are correlated, e.g., in graph-structured datasets (Bruna et al., 2015;
Shi et al., 2014; Kipf & Welling, 2016;
Hamilton et al., 2017) .
In these cases, it is more reasonable to assume the latent representation for each data point also respects the correlation graph structure.
In this paper, we extend the standard VAE s by encouraging the latent representations to take the correlation structure into account, which we term CVAE s. In CVAE s, rather than a commonly used i.i.d.
standard Gaussian prior, we apply a correlated prior on the latent variables following the structure known a priori .
We develop two variations, CVAE ind and CVAE corr :
In CVAE ind , we still use a fully-factorized singleton variational density via amortized inference; while in CVAE corr , instead of only learning singleton latent variational density functions, we also incorporate pairwise latent variational density functions to achieve a better variational approximation.
With a correlated prior, the standard variational inference objective becomes intractable to compute.
We sidestep the challenging objective by considering the average of a set of tractable lower bounds over all maximal acyclic subgraphs of the given undirected correlation graph.
The experimental results show that both CVAE ind and CVAE corr can outperform the baseline methods on three tasks: matching dual user pairs using the rating records on a movie recommendation dataset, clustering the vertices with latent embeddings drawn from a tree-structured undirected graphical model on a synthetic dataset, and predicting links using the rating records on a product rating dataset.
CVAE s ON ACYCLIC GRAPHS
In this section, we extends VAE s to ﬁt with a set of latent variables with an acyclic correlation graph.
We start with acyclic graphs since the prior and posterior approximation of the latent variables can be expressed exactly for such correlation structures.
2. V ARIATIONAL A UTO -E NCODINGS
Assume that we have input data x = { x 1 , . . .
, x n } ⊆ R D .
Standard VAE s assume that the data point x i is generated i.i.d.
from the following process: First, generate the latent variables
z
= { z 1 , . . .
, z n
} ⊆ R d (usually d (cid:28) D ) by drawing z i i.i.d.
∼ p 0 ( z i ) from the prior distribution p(parameter-free, usually a standard Gaussian distribution) for each i ∈ { 1 , . . . , n } .
Then generate the data points x i ∼ p θ ( x i
| z i ) from the model conditional distribution p θ , for i ∈ { 1 , . . . , n } independently.
We are interested in optimizing θ to maximize the likelihood p θ ( x ) , which requires computing the posterior distribution p θ ( z | x ) = n (cid:81)
i =1 p θ ( z i | x i ) .
For most models, this is usually intractable.
VAE s sidestep the intractability and resort to variational inference by approximating this posterior distribution as q λ ( z | x ) = n (cid:81) i =
1 q λ
( z i | x i ) via amortized inference and maximize the evidence lower bound (ELBO):
[Equation]
The ELBO is a lower bounds to the log-likelihood log p θ ( x ) , and maximizing this lower bound is equivalent to minimizing the KL-divergence between the variational distribution q λ ( z | x ) and the true posterior p θ ( z | x ) .
The KL-divergence term in the ELBO can be viewed as regularization that pulls the variational distribution q λ ( z | x ) towards the prior p
( z ) .
Since the approximation family q λ ( z | x ) factorizes over data points and the prior is i.i.d.
Gaussian, the KL-divergence in the ELBO is simply a sum over the per-data-point KL-divergence terms, which means that we do not consider any correlations of latent representation between data points.
2. C ORRELATED PRIORS ON ACYCLIC GRAPHS
As motivated earlier, sometimes we know a priori that there exist correlations between data points.
If we have access to such information, we can incorporate it into the generative process of VAE s, giving us correlated VAE s.
Formally, assume that we have n data points x 1 , . . .
, x n .
In addition, we assume that the correlation structure of these data points is given by an undirected graph G =
( V, E ) , where V = { v 1 , ..., v n } is the set of vertices corresponding to all data points (i.e., v i corresponds to x i ) and ( v i , v
j ) ∈
E if x i and x j are correlated.
For now we assume G is acyclic, and later in Section 3 we will extend the results to general graphs.
Making use of the correlation information, we change the prior distribution p corrof the latent variables z 1 , . . .
, z n to take the form of a distribution over ( z 1 , . . .
, z n ) ∈ R d × . . .
×
R d whose singleton and pairwise marginal distributions satisfy
[Equation]
Here p 0 ( · ) is a parameter-free density function that captures the singleton prior distribution and p 0 ( · , · ) is a parameter-free density function that captures the pairwise correlation between each pair of variables.
Furthermore
[Equation]
they should satisfy the following symmetry and marginalization consistency properties:
[Equation]
The symmetry property (the ﬁrst part of Eq. 3) preserves the validity of the pairwise marginal distributions since p
( z i , z j ) and p 0 ( z j , z i ) are representing exactly the same distribution.
The marginalization consistency property (the second part of Eq. 3) maintains the consistency between the singleton and pairwise density functions.
This prior will help the model take the correlation information into consideration since we have a
KL-divergence regularization term in the ELBO that will push the variational distribution towards the prior distribution.
With the correlated prior deﬁned above, the generative process of a CVAE is straightforward: First we sample z from this new prior p corr 0 , then we sample each data point x i conditionally independently from z i , similar to a standard VAE.
In general, the prior distributions deﬁned in Eqs.
2 and 3 do not necessarily form a valid joint distribution on z , that is, there exists a graph G and choice of prior distributions as above such that no joint distribution on z has the priors deﬁned by p 0 as its marginals.
Nonetheless, if G is an undirected acyclic graph, such a joint distribution does exist, independent of which distributions we choose for p 0 ( · ) and p 0 ( · , · ) (Wainwright & Jordan, 2008):
[Equation]
In other words, the joint correlated prior distribution on z can be expressed explicitly with only the singleton and pairwise marginal distributions, without having an intractable normalization constant.
2. CVAE s ON ACYCLIC GRAPHS
To perform variational inference, we need to specify which variational family we will use.
In this paper, we explore two different variational families: one where the prior distribution only involves singleton marginals, which we denote CVAE ind , and one in which the prior distribution has the form in Eq. 4, which we denote CVAE corr .
Singleton variational family.
In CVAE ind , we approximate the posterior distribution p ( z | x ) as q λ ( z | x ) = n (cid:81) i =1 q λ
( z i | x i ) which consists of fully-factorized distributions.
The corresponding ELBO for CVAE ind on acyclic graphs can be found in Appendix A. Even though this variational family does not consider pairwise correlations, the prior p corris still more expressive than the standard Gaussian prior used in standard VAE s.
Correlated variational family.
In CVAE corr , where the prior p corrcan be written as in Eq.
4, it makes sense to use a variational distribution q λ ( z | x ) expressed in the same form, i.e., with only singleton and pairwise marginal distributions:
[Equation]
As in Eq. 3, the marginals need to satisfy the following properties:
[Equation]
The corresponding ELBO for CVAE corr on acyclic graphs can be found in Appendix A.
This ELBO yields a tighter lower bound on the log-likelihood log p θ ( x ) under the prior in Eq.
4 as compared to the ELBO of CVAE ind , since optimizing this ELBO involves with optimizing over a larger set of distributions than the factorized distributions as in CVAE ind .
CVAE corr not only takes the correlation structure into consideration as CVAE ind , but also learns correlated variational distributions that can potentially yield better approximations to the model posterior.
CVAE s ON GENERAL GRAPHS
We have introduced the CVAE s for acyclic correlation structures.
In this section, we extend these models to general undirected graphs.
3. W HY THE TRIVIAL GENERALIZATION FAILS
A simple generalization is just to directly apply the ELBOs (shown in Appendix A) for acyclic graphs that we developed in Section 2.
However, this simple approach can easily fail.
First, as described earlier, Eq. 4 is not guaranteed to be a valid distribution (i.e., it does not integrate to 1 over z ) for a general graph G (Wainwright & Jordan, 2008).
Moreover, optimizing these acyclic ELBOs may lead to pathological cases where the objectives go to inﬁnity due to these objectives not guaranteed to be a lower bound of the log-likelihood for general graphs.
We construct a simple example to demonstrate this in Appendix B. Therefore, we can not directly apply the ELBOs on acyclic graphs.
We need to deﬁne a new prior distribution p corr g
( z ) and derive a new lower bound for the log-likelihood log p θ ( x ) under this prior, for general graphs.
3. I NFERENCE WITH A WEIGHTED OBJECTIVE
Though a general graph G may contain cycles, its acyclic subgraphs may contain correlation structures that well-approximate it, especially when G is sparse.
For acyclic graphs, we have already derived
ELBOs for CVAE ind and CVAE corr in Section 2.
We extend these two methods for general graph G by considering an average of the loss over its “maximal acyclic subgraphs”, which is deﬁned as follows:
Deﬁnition 1 (Maximal acyclic subgraph) .
For an undirected graph G =
( V, E ) , say an acyclic subgraph G (cid:48)
=
( V (cid:48) , E (cid:48) ) is a maximal acyclic subgraph of G if:
• V (cid:48)
=
V , i.e., G (cid:48) contains all vertices of G .
Adding any edge from E/E (cid:48) to E (cid:48) will create a cycle in G (cid:48) .
A maximal acyclic subgraph of G may be in similar structure to G , especially when G is sparse.
When
G is acyclic, it only contains one maximal acyclic subgraph (i.e., G itself).
When G is connected, any subgraph G (cid:48) is a maximal acyclic subgraph of G if and only if G (cid:48) is a spanning tree of G .
In general, any subgraph G (cid:48) is a maximal acyclic subgraph of G , if and only if, for any of G ’s connected components, G (cid:48) contains a spanning tree over this connected component, see Figure 1.
We will use
A G to denote the set of all maximal acyclic subgraphs of G .
[Equation]
While Eq. 4 is not guaranteed to be a valid prior distribution on z if G contains cycles, we can use it to deﬁne a new prior over G ’s set of maximal acyclic subgraphs.
More speciﬁcally, we deﬁne the prior distribution of z as a uniform mixture over all subgraphs in A G :
[Equation]
6) where p G (cid:48)
( z ) =
n (cid:81)
[Equation]
) p
[Equation]
z i
[Equation]
p
[Equation]
z j ) .
Eq.
6 deﬁnes a uniform mixture model over the set of prior distributions p G (cid:48) 0 on the maximal acyclic subgraphs of G , so it is also a valid density.
Note that this reduces to Section 2 when G is acyclic as |A G | = 1 .
Under this deﬁnition, the log-likelihood log p θ ( x ) becomes
[Equation]
The inequality is a direct application of Jensen’s inequality, which is also applied when deriving the normal ELBO in Eq.
Here q G (cid:48) λ
( z | x ) is a variational distribution related to the maximal acyclic subgraph G (cid:48)
=
( V, E (cid:48) ) .
Similar to Section 2, we can specify either a singleton or a correlated variational family.
We describe the construction for correlated variational families as singleton variational families are simply a special case of this.
We deﬁne q G (cid:48) the same way as in Eq. 5:
[Equation]
Under this deﬁnition, we construct |A
G | different variational distributions q G (cid:48) λ .
These distributions may be different from each other due to the difference in graph structure, but they share the same singleton and pairwise marginal density functions q λ ( ·|· ) and q λ ( · , ·|· , · ) as well as the same set of variational parameters λ .
Though these singleton and pairwise marginal density functions are not guaranteed to form a real joint density of z for the whole graph G , we can still guarantee that each of the q G (cid:48) λ is a valid density on z .
Moreover, these locally-consistent singleton and pairwise density functions will approximate the singleton and pairwise marginal posterior distributions.
With this deﬁnition of q G (cid:48) λ , the lower bound in Eq.
7 becomes the sum of a set of singleton terms over vertices in V and a set of pairwise terms over edges in E .
The singleton terms have the same weights on all vertices, while the pairwise terms may have different weights: for each edge e ∈ E , the weight is the fraction of times e appears among all subgraphs in A G .
We deﬁne this weight as follows:
Deﬁnition 2 (Maximum acyclic subgraph edge weight) .
For an undirected graph G =
( V, E ) and an edge e ∈ E , deﬁne w MAS G,e to be the fraction of G ’s maximal acyclic subgraphs that contain e , i.e., w
[Equation]
Since each maximal acyclic subgraph of G is a disjoint union of spanning trees, one per connected component of G , we have:
Proposition 1 (Maximum acyclic subgraph edge weight sum) .
For an undirected graph G =
( V, E ) , denote CC ( G ) as the set of connected components of G , we have (cid:80)
[Equation]
Using the edge weights, we can show (details in Appendix A) that the lower bound in Eq.
7 equals n (cid:88) i =
[Equation]
(cid:88)
[Equation]
w MAS G, ( v i
[Equation]
j ) · (cid:16) KL ( q λ ( z i , z
j | x i , x j ) || p
( z i , z j ))
− KL ( q λ ( z i | x i ) || p
[Equation]
z i ))
− KL (
[Equation]
λ
[Equation]
j | x j ) || p
[Equation]
j ))
[Equation]
:=
L CVAE corr ( λ , θ ) .
Eq.
9 deﬁnes a valid lower bound of the log-likelihood log p θ ( x ) under the mixture model prior in
Eq.
We deﬁne this as the loss function for CVAE corr on general graphs.
As long as the weights w MAS are tractable, optimizing this lower bound is tractable.
We will show how to compute these weights efﬁciently in Section 3.3.
For CVAE ind , the ELBO L CVAE ind is just Eq.
9 except that we change q λ
( z i , z j | x i , x j ) to be the product of the two singleton density functions q λ ( z i | x i ) and q λ ( z j | x j ) .
3. C OMPUTING THE SUBGRAPH WEIGHTS
To optimize L CVAE corr ( λ , θ ) , we need to efﬁciently compute the weights w MAS G,e for each edge e ∈ E .
These weights are only related to the graph G , but not the model distribution p θ , the variational density functions q λ ’s or the data x .
Recall that w MAS G,e is just the fraction of G ’s maximal acyclic subgraphs which contain the edge e .
For simplicity, we ﬁrst consider a special case where G is a connected graph.
Then w MAS G,e is just the fraction of G ’s spanning trees which contain the edge e .
To compute this quantity, we need to know the total number of spanning trees of G , as well as the number of spanning trees of G which contain the edge e .
The Matrix Tree Theorem (Chaiken & Kleitman, 1978) gives a formula for the total number of spanning trees of a given graph G :
Theorem 1 (Matrix Tree Theorem (Chaiken & Kleitman, 1978)) .
For an undirected graph G =
( V, E ) , the number of spanning trees of G is the determinant of the sub-matrix of the Laplacian matrix L of G after deleting the i th row and the i th column (i.e., the ( i, i ) -cofactor of L ), for any i
= 1 , ..., n .
The Laplacian matrix L of a undirected graph G =
( V = { v 1 , . . .
, v n } , E ) is deﬁned as follows.
L ∈ R n × n is a symmetric matrix where:
[Equation]
Similarly, we compute the number of spanning trees of G that contain edge e
=
( v i , v j ) :
Theorem 2 (Number of spanning trees containing a speciﬁc edge) .
For an undirected graph G =
( V, E ) and an edge ( v i , v j ) ∈ E , the number of spanning trees of G containing this edge is the determinant of the sub-matrix of the Laplacian matrix L of G after deleting the i th , j th rows and the i th , j th columns of it (i.e., the complement of the minor M ij,ij of L ).
Proof.
See Appendix C.1.
Directly using these two theorems to compute w MAS G,e has several issues.
First, we need to compute | E | different determinants for matrices of size ( | V | − 2) ×
( | V | − 2) (by applying Theorem 2), whose time complexity is O ( | E || V | 3 ) , which is very inefﬁcient.
Second, the results we get from these two theorems can be exponentially large compared to | V | and | E | (e.g., a complete graph K n with n vertices contains n n − 2 spanning trees), which can result in signiﬁcant numerical issues under division.
Since we only care about the ratios, notice that the numbers we compute from Theoremare cofactors of the matrices (sub-matrices of L ) on which we compute determinants from Theorem 1, we derive the following formula for computing the weights w MAS G,e ’s, using the relationship between cofactors, determinants, and inverse matrices.
Theorem 3 (Computing the spanning tree edge weights) .
For an undirected connected graph G =
( V, E ) and an edge e =
( v i ,
v j ) ∈ E , the weight w MAS G,e =
L + i,i − L
+ i,j − L + j,i + L + j,j .
Here L + is the
Moore-Penrose inverse of the Laplacian matrix L of G .
Proof.
See Appendix C.2.
With this theorem, given an undirected connected graph G , we can ﬁrst compute the Moore-Penrose inverse L + of G ’s Laplacian matrix and then compute the weight w MAS G,e for every edge e ∈ E .
The time complexity is O
( | V | 3 ) , which is not unreasonable, even for relatively large graphs.
Furthermore, there should be no numerical issues in the computations since computing the Moore-Penrose inverse is numerically stable.
We have just illustrated how to compute the weights w MAS G,e when G is connected.
When G is not connected, from the deﬁnition of maximal acyclic graphs, we know that, w MAS G,e is equal to w MASCC ( G,e ) ,e , where CC ( G, e ) is the connected component of G that contains the edge e .
Therefore, we just need to apply Theorem 3 for all connected components of G .
The details of computing the weights w MAS G,e for all e ∈ E are shown in Algorithm 1 in appendix.
The time complexity of this algorithm is O
( | V | 3 ) in the worst case, but potentially much smaller if each connected component of G has a small number of vertices.
We can in fact relax the premise of Theorem 3 that G is connected: since the Moore-Penrose inverse of block diagonal matrices is equivalent to computing the Moore-Penrose inverse for each of these sub-matrices, Theorem 3 is also correct for general graphs.
Hence, we can compute the weights without identifying the connected components.
However, performing Algorithm 1 is at least as efﬁcient as directly computing the Moore-Penrose inverse of the whole matrix.
3. R EGULARIZATION WITH NON - EDGES
With Algorithm 1, we can efﬁciently compute all the weights w MAS G,e and optimize the ELBO L CVAE corr in Eq. 9.
This ELBO may be a good objective function to optimize if our goal is only to use the trained generative model p θ ( x | z ) or to get a good approximation to the singleton and pairwise marginal posterior.
However, if we want to use the learned pairwise variational density functions q λ ( · , ·|· , · ) for predictive tasks that may take inputs ( x u , x v ) where ( u, v ) (cid:54)∈ E (e.g., perform link predictions using these density functions), then purely optimizing Eq. 9 is not sufﬁcient since this loss function may consider all pairs of vertices as correlated, due to only “positive examples” (correlated pairs) are present in the data.
As a result, the learned pairwise density functions are not capable of identifying the correlations of new inputs.
To address this issue, we add a regularization term to the ELBO L CVAE corr , which is akin to negative sampling in language modeling.
Recall in Eq.
9, we compute the average KL terms over all maximal acyclic subgraphs of G .
These KL terms are the “positive” examples.
For the “negative” examples, we apply the average KL terms over all maximal acyclic subgraphs of the complete graph K n as the graph, and treat the prior on z to be i.i.d.
on each z i to regularize the density functions q ’s towards independent for the negative samples.
Since K n is a complete graph, the edge weight w MAS should be the same among all edges.
By Proposition 1, since K n is connected and has n ( n − 1)edges, we get
Proposition 2 (Maximal acyclic subgraph weights for complete graphs) .
For a complete graph K n , the weight w MAS K n ,e for any edge e of K n is 2 n .
Therefore, we can deﬁne the loss function for CVAE corr with the negative sampling regularization as
[Equation]
Here γ > 0 is a parameter that controls the regularization.
In this loss function, the edges in E appear in both the positive and the negative samples.
However, by Proposition 1, the average weight of the edges in E in the positive samples is | V |−| CC ( G ) | | E | .
Therefore, as long as γ ≤ O (cid:16) | V | ( | V |−| CC ( G ) | ) | E |
[Equation]
, the regularization term will not dominate the effect of the positive samples.
This negative sampling regularization can help CVAE corr learn better latent embeddings for many predictive tasks as shown in Section 4.
CVAE ind does not need such regularization since it does not learn correlated variational density functions, but only fully-factorized ones.
E XPERIMENTS
We test the performance of CVAE ind and CVAE corr on different tasks on three datasets, and compare their performances to the baseline methods.
4. E XPERIMENT SETTINGS
Tasks.
We test our methods on 3 tasks: user matching on a public movie rating dataset, spectral clustering on a synthetic tree-structured dataset and link prediction on a public product rating dataset.
For each dataset, we have a high-dimensional feature for each user (or data point) and a undirected correlation graph users
Baselines.
We have two baseline methods:
• The standard Variational Auto-Encoders.
[Table]
• The GraphSAGE algorithm (Hamilton et al., 2017): the state-of-the-art method on learning latent embeddings with graph convolutional networks.
It is capable of learning latent embeddings that take the correlation structures into consideration.
Evaluations.
Different tasks may have different evaluation metrics.
However, all of our results are computed based on the (expected) quadratic pairwise distance of the latent distributions on the evaluation dataset.
For VAE , CVAE ind and CVAE corr , denote the evaluation dataset as x 1 , . . .
, x n and the marginal variational distribution on ( z i , z j ) as q ( z i , z j ) , then the distance between the data points i and j ( i (cid:54) = j ) is deﬁned as dis i,j = E q
( z i , z j ) (cid:2) ||
z i − z
j || 22 (cid:3) .
Notice that, for VAE and CVAE ind , the distribution q ( z i , z j ) =
q ( z i ) q ( z j ) is always factorized.
This does not hold for the CVAE corr .
For
GraphSAGE, since the learned embeddings z i ’s are not stochastic, we use dis i,j = ||
z i − z
j || 22 as the distance between the data points i and j .
Additional information related to the models and inference settings are in Appendix E.
4. R ESULTS 4.2. U SER MATCHING
We evaluate CVAE with a bipartite correlation graph.
We use the MovieLens 20M dataset (Harper &
Konstan, 2016).
This is a public movie rating dataset that contains ≈ 138 K users and ≈ 27 K movies.
We binarize the rating data and only consider whether a user has watched a movie or not, i.e., the feature vector for each user is a binary bag-of-word vector, and we only keep ratings for movies that have been rated at least 1000 times.
For all the experiments, we did a stochastic train/test split over users with a 90 / 10 ratio.
For each user u i , we randomly split the movies that this user has watched into two halves and construct two synthetic users u Ai and u Bi .
This creates a bipartite graph where we know the synthetic users which were generated from the same real user should be more related than two random synthetic users.
The goal of the evaluation is that, when given the watch history of a synthetic user u Ai from a held-out set, we try to identify its dual user u Bi .
This can be potentially helpful with identifying close neighbors when using matching to estimate causal effect, which is generally a difﬁcult task especially in high-dimensional feature spaces (Imbens & Rubin, 2015).
We train all the methods on all the synthetic user pairs from the training set.
To evaluate, we select a ﬁxed number of N eval = 1000 pairs of synthetic user from the test sets.
For each synthetic user u Ai (or u Bi ), we ﬁnd the ranking of u Bi (or u Ai ) among all candidates in the set of all other 2 N eval −synthetic users in terms of the latent embedding distance to u Ai (or u Bi ).
The latent embedding distance metrics dis i,j ’s for all methods are deﬁned in Section 4.1.
For CVAE corr , we set the negative sampling regularization parameter γ = 1 .
We report the average Reciprocal Rank (RR) of the rankings for all methods in Table 1.
CVAE ind and
CVAE corr strongly outperform the standard VAE , which means that the correlation structure helps in learning useful latent embeddings.
Here CVAE corr improves over CVAE ind by learning a correlated variational approximation.
We do not compare our methods to the GraphSAGE algorithm for this task since the graph for this task is just a bipartite matching graph with many connected components while GraphSAGE works well for graphs where the local neighborhoods can provide substantial information for the vertices.
4.2. S PECTRAL CLUSTERING
We perform spectral clustering (Von Luxburg, 2007) on a synthetic dataset with a tree-structured latent variable graphical model.
The dataset contains N = 10000 data points x 1 , . . .
, x N ∈ R D with D = 1000 .
Each x i is generated independently from the distribution p ( x i
| z i ) given the ground truth latent embeddings z 1 , . . .
, z N ∈ R d , where the likelihood p ( ·|· ) is element-wise Bernoulli distribution with the logits come from a two-layer feedforward neural network that takes the latent embeddings as inputs.
The latent embeddings z 1 , . . .
, z n are drawn from an tree-structured undirected graphical model.
The probability distribution of this graphical model is of the same form as Eq. 4, where the singleton prior density p 0 ( · ) is the standard normal and the pairwise prior density is p 0 ( · , · ) =
N (cid:18) µ = 0 2 d , Σ = (cid:18) I d τ · I d τ · I d
I d (cid:19)(cid:19) .
With the latent embeddings z 1 , . . .
, z N , we generate a binary cluster label c i ∈ { 0 , 1 } for each data point x i by performing a principle components analysis on the latent embeddings and set c
i = 1 if and only if z i has a coefﬁcient rank at least N 2 among all the N data points on the ﬁrst component.
We perform spectral clustering based on the latent embedding distance metrics dis i,j ’s discussed in
Section 4.1 for all algorithms.
Since spectral clustering requires a non-negative symmetric similarity matrix S ∈ R N × N , we set S ij = exp ( − dis i,j / 2) .
We apply a normal spectral clustering procedure by computing the eigenvector v ∈ R N corresponding to the second smallest eigenvalue of the normalized Laplacian matrix of S .
Then we cluster the data points x 1 , . . .
, x n by clustering the set of coordinates of v with value larger than the median of all these coordinates as one cluster, and the rest as the other cluster.
To evaluate clustering, we apply the normalized mutual information score (Vinh et al., 2010), which is in the range of [0 , 1] (larger the better).
The scores for all methods are shown in Table 2.
Here we apply negative sampling regularization parameter γ = 0 .
1 for CVAE corr .
From
Table 2, we can see that CVAE ind and CVAE corr strongly outperform both baseline methods.
CVAE corr does not signiﬁcantly improve over CVAE ind potentially due to that we do not have sufﬁciently many edges to learn a good correlation function, but it still outperforms the baseline methods.
[Table] [Table]
L INK PREDICTION
We perform link prediction on a general undirected graph G =
( V, E ) .
In this experiment, we use the Epinions dataset (Massa & Avesani, 2007), which is a public product rating dataset that contains ≈ 49 K users and
≈ 140 K products.
We again binarize the rating data and create a bag-of-words binary feature vector for each user.
We only keep products that have been rated at least 100 times and only consider users who have rated these products at least once.
We construct a correlation graph G =
( V, E ) .
The Epinions dataset has a set of single-directional “trust” statements between the users, i.e., a directed graph G (cid:48)
=
( V (cid:48) , E (cid:48) ) among all users.
Since we need undirected correlation structures, we take all vertices from G (cid:48)
(i.e., set V = V (cid:48) ) and set an edge ( v i , v
j ) ∈
E if both ( v (cid:48)
i , v (cid:48) j ) and ( v (cid:48)
j , v (cid:48) i ) are in E (cid:48) .
To split the train/test dataset for the link prediction task, for each vertex v i ∈ V , we hold out max (cid:0) 1 , 120 · degree ( v i ) (cid:1) edges on v i as the testing edge set E test , and put all edges that are not selected into the training edge set E train .
We train all methods on the product ratings and the correlation graph G train =
( V, E train ) .
For evaluation, we ﬁrst compute the latent embedding distance dis i,j ’s that was deﬁned in Section 4.
for 1 ≤ i (cid:54)
=
j ≤ N .
Then for each user u i , we compute the Cumulative Reciprocal Rank NCRR i of the ratings of u i ’s testing edges, among all possible connections except for the edges in the training edge set, in terms of the latent embedding distance metrics.
Formally, this value equals
CRR i =
(cid:80) ( v i ,v
j ) ∈
E test
[Equation]
|
[Equation] [Equation]
(cid:54)∈
[Equation]
}| .
Larger CRR i indicates better ability to predict held-out links.
We further normalize the CRR values to within the range of [0 , 1] and show the average metrics among all users in Table 3.
Evidently, CVAE ind and CVAE corr again strongly outperform the baseline methods.
Here we apply a large regularization value of γ = 1000 for the CVAE corr , which may help when the input graph has a complex structure (unlike the previous experiments) yet does not have a dense connection.
This choice of γ provides
CVAE corr enough regularization for learning the correlation and helps it improve over CVAE ind .
R ELATED W ORK
Shaw et al.
(2011) incorporated graph structure to metric learning.
The major difference with CVAE s is that the metric learned from Shaw et al.
(2011) is inherently linear while CVAE s are capable of capturing more complex non-linear relations in the feature space.
There has been some previous work on handling optimizations with intractability or inconsistency over general graphs by leveraging the problems on the spanning trees.
In graphical model inference, the tree-reweighed sum-product algorithm (Wainwright, 2003) and the tree-reweighed Bethe variational principle (Wainwright et al., 2005) extend the ordinary Bethe variational principle (Wainwright &
Jordan, 2005) over a convex combination of tree-structured entropies.
In combinatorial optimization,
Tang & Jebara (2017) used the maximal spanning trees of the pairwise matching scores to provide consistent initializations for multi-way matching.
Moreover, there has been some recent work on incorporating structures in variational inference and
VAE s. Hoffman & Blei (2015) proposed structured stochastic variational inference to improve over the naive mean-ﬁeld family.
Johnson et al.
(2016) proposed structured VAE s which enable the prior to take a more complex form (e.g., a Gaussian mixture model, or a hidden Markov model).
Lin et al.
(2018) extend the work of Johnson et al.
(2016) to variational message passing, which is applied to analyze time-seres data in Pearce et al.
Burda et al.
(2016) and Kingma et al.
(2016) proposed variational approximation families that can learn better posterior approximations compared to the commonly applied mean-ﬁeld family, with normalizing ﬂows and inverse autoregressive ﬂows, respectively.
Ainsworth et al.
(2018) proposed output-interpretable VAE s which combine a structured VAE comprised of group-speciﬁc generators with a sparsity-inducing prior.
The resent work StructVAE
(Yin et al., 2018) extends the standard VAE to handle tree-structured latent variables.
However, most of these works are designed to model the structures between dimensions within each data point, while CVAE s considers general graph structures between data points.
Another related line of work are the recent advances in convolutional networks for graphs (Bruna et al., 2015; Duvenaud et al., 2015; Defferrard et al., 2016;
Niepert et al., 2016; Kipf & Welling, 2016) and the extensions, e.g., our baseline method GraphSAGE (Hamilton et al., 2017).
C ONCLUSION
We introduce CVAE ind and CVAE corr to account for correlations between data points that are known a priori .
They extend the standard VAE s by applying a correlated prior on the latent variables.
Furthermore, CVAE corr adopts a correlated variational density function to achieve a better variational approximation.
These methods successfully outperform the baseline methods on several machine learning tasks using the latent variable distance metrics.
For future work, we will study the correlated variational auto-encoders that consider higher-order correlations.
A CKNOWLEDGMENTS
This work was supported in part by National Science Foundation grant III: Small: Collaborative
Research: Approximate Learning and Inference in Graphical Models III-1526914.
A PPENDIX
In the appendix, we ﬁrst show the ELBOs for the CVAE s under acyclic correlation graphs, as well as the derivation for the ELBO of CVAE corr (Eq. 9) for general correlation graphs.
Also we show the counter example mentioned in Section 3.1.
Then, we show the proofs of Theorems 2 and 3 and the details for Algorithm 1.
At the end, we provide more information related to the models and inference settings.
A ELBO S AND D ERIVATIONS
ELBO for CVAE ind with acyclic graphs.
CVAE ind applies a correlated prior p corr 0 ( z ) as in Eq. 4.
Therefore,
[Equation] [Equation]
This ELBO is a lower bound of the log probability p ( x ; θ ) under the correlated prior (as in Eq. 4).
Optimizing this ELBO helps the variational distributions to regularize according to the correlation graph G .
ELBO for CVAE corr with acyclic graphs.
By changing q to be the correlated distribution as in
Eq. 5, the ELBO in Eq.
11 becomes
[Equation]
Derivation for Eq.
We derive the ELBO of CVAE corr (Eq. 9) for general correlation graphs here.
Recall that Eq.
7 shows that
[Equation]
By the deﬁnition of A G
[Equation]
the right hand side of the above inequality equals the following sum n (cid:88)
[Equation]
The pairwise sum part of the above equation is an average over a sum over all edges of all maximal acyclic subgraphs of G .
Therefore, for each edge e = ( v i , v j ) ∈ E , the number of times it appears in this pairwise sum part of the above sum is just the number of maximal acyclic subgraphs containing this edge.
Therefore, this part can be viewed as a weighed sum over all edges in E , where the weights come from the fraction ratios in Deﬁnition 2.
With this deﬁnition, we can further write the above sum as
[Equation]
which is exactly Eq. 9.
B C OUNTER EXAMPLE FOR S ECTION 3.
In Section 3.1, we mentioned that directly applying the ELBOs we derived (equations shown in
Appendix A) will fail partly due to that optimizing over these ELBOs may lead the algorithm to learn useless parameters that lead the loss goes to inﬁnity due to that these two equations are not guaranteed to be a lower bound of some log-likelihood function for general graphs.
Here we provide a simple example to illustrate this.
Let us consider G =
( V, E ) is a K 4 complete graph with | V | = 4 vertices and (cid:0) | V | 2 (cid:1)
= 6 edges.
For simplicity, we consider the latent variables z 1 , z 2 , z 3 , z 4 ∈ R and the prior distribution p
( z i , z j ) =
N (cid:18)(cid:18)
z i z j (cid:19) ; µ = 0 2 , Σ
= (cid:18)τ τ (cid:19)(cid:19) for some τ ∈ (0 , 1) .
If we extend the CVAE ind and for the variational distribution, we set q ( z i | x i ) to be the normal distribution N ( µ i , σ 2 i ) , by simply apply
Eq.
11, then the loss function becomes
[Equation]
If we maintain σ i ’s unchanged, set the model parameter θ in a way that makes p θ ( x i
| z i ) unrelated to z i (e.g. set the parameter that multiply with z i to be 0) and set µ 1 = µ 2 = µ 3 = µ 4 = µ and let µ → ∞ , then L will go to + ∞
if τ > 12 .
Therefore, directly applying Eq.
11 does not work.
Directly applying Eq.
12 (i.e. extending the CVAE corr ) will make the result even worse since it always has an optimal value at least as high as Eq.
In general, any general graphs with K 4 subgraph may suffer from the issue we just mentioned.
We can not obtain useful latent embeddings by directly applying the Eqs.
11 and 12.
C P ROOFS
We prove the Theorems 2 and 3 here.
C. P ROOF OF T HEOREM
Proof.
Given the graph G =
( V, E ) and the edge ( v i , v j ) ∈ E .
Denote G ’s Laplacian matrix as L .
Also denote L − a, − b as the sub-matrix of L after deleting the a th row and the b th column.
Denote
L − ab, − cd as the sub-matrix of L after deleting the a th , b th rows and the c th , the d th columns.
By Matrix Tree Theorem (Theorem 1 (Chaiken & Kleitman, 1978)), we know that the ( i, i ) -cofactor
C i,i = | L − i, − i | is the number of spanning trees of G .
Construct a graph G (cid:48)
=
( V, E/ { v i , v j } ) , i.e. the graph G after removing the edge ( v i , v j ) .
Denote the Laplacian matrix of G (cid:48) as L (cid:48) .
Then we will ﬁnd that the matrix L (cid:48) − i, − i is the same with L − i, − i except that they L − i, − i is 1 larger than L (cid:48)− i, − i on the entry at ( j, j ) .
By Matrix Tree Theorem, | L (cid:48)− i, − i | is the number of spanning trees of G (cid:48) .
Since G differs from G (cid:48) by only having one more edge ( v i , v j ) , we know that | L − i, − i | − | L (cid:48)− i, −
i | represents the number of spanning trees in G that contains the edge ( v i , v j ) .
Since we know that
[Equation]
Subtract the second equation from the we get
[Equation]
which is just the complement of the Minor M ij,ij of L .
Hence, the number of spanning trees of G that contains the edge ( v i , v j ) is M ij,ij , the determinant of the sub-matrix of the Laplacian matrix L of G , after deleting the the i th , j th rows and the i th , the j th columns.
C. P ROOF OF T HEOREM
Proof.
We borrow the notations from Appendix C.1.
Given the undirected connected graph G =
( V, E ) and an edge ( v i , v j ) ∈ E , we want to compute the ratio w MAS G, ( v i ,v j ) .
Since G is connected, this ratio is just the fraction of G ’s spanning trees containing the edge ( v i , v j ) .
By Theorem 1 and 2, this ratio is just | L − ij, −
[Equation]
Since G is connected, it contains at least one spanning tree.
Hence | L − i, − i | > 0 , which means
L − i, − i is invertible.
Denote the entry at ( j, j ) of L − i, − i as L − i, − i ; j,j .
Since the L − i, − i is invertible, we know that
[Equation]
Consider the original Laplacian matrix L before deleting any row and column.
Denote | V | = n .
Since
L is always symmetric and always have an eigenvector v n = 1 √ n 1 n with corresponding eigenvalue λ n = 0 , we perform eigenvalue decomposition on L and write L as:
[Equation]
Where λ 1 , . .
, λ n − 1 , λ n are L ’s eigenvalues and v 1 , . .
. , v n − 1 , v n are the corresponding orthogonal unit eigenvectors (i.e. Q v = ( v· · · v n −v n ) is an orthogonal matrix).
Denote v a,b as the b -the coordinate of v a and v a, − b as the sub-vector v a after deleting the b th coordinate.
Then
[Equation]
L − i, − i is invertible, which is of rank n − 1 .
While each of the matrix v k, − i v (cid:62) k, − i is of rank 1.
Hence, we must have λ
i (cid:54) = 0 for all i ∈ { 1 , . . . , n − 1 } .
Construct vectors u 1 , . .
. , u n ∈ R n such that u
[Equation]
Also, construct the matrix
[Equation]
Since we know that ( v· · · v n −v n ) forms an orthogonal basis and v n = 1 √ n 1 n , it is easy to see (after simple calculations)
that
[Equation]
Therefore, we will have
U − i, − i L − i, − i = I n −which indicates that U − i, − i
=
L − 1 − i, − i .
Hence, the ratio we want to ﬁnd is just U − i, − i,j,j .
Recall the deﬁnition of u 1 , . .
. , u n in Eq. 13, denote Q u
= ( u· · · u n −u n ) , we get Q u =
P i Q v , where
P i
[Equation]
where the − 1 ’s appear on the i th column.
Hence, denote D = diag ( λ 1 , . . . , λ n ) , we have
[Equation]
Therefore, the ratio w MAS G, ( v i ,v j ) we want to ﬁnd, which is equal to U − i, − i ; j,j , is just the ( j, j ) -entry of
P i L + P (cid:62) i , which is
[Equation]
D A LGORITHM FOR COMPUTING THE WEIGHTS w MAS G,e
Given a correlation graph G =
( V, E ) , we show the details of computing the weights w MAS G,e for all e ∈ E as in Algorithm 1.
Input: undirected graph G =
( V = { v 1 , . . .
, v n } , E ) .
Compute all the connected components CC 1 , . .
, CC K of G using depth-ﬁrst search or breadthﬁrst search.
for k = 1 to K do
Compute the Moore-Penrose inverse L + k of the Laplacian matrix L k of the component CC k .
Apply Theorem 3 to compute w MAS G,e for each edge e in the component CC k .
end for
Return The all
E A DDITIONAL EXPERIMENT SETTINGS
We provide some additional information related to the models and inference settings here.
For all methods, we set the latent embeddings to have the dimensionality d = 100 .
For the standard VAE s, the CVAE ind and the CVAE corr , we apply a two-layer feed-forward neural network for the generative model p θ ( x i
| z i ) and a two-layer feed-forward neural network for the singleton variational approximations q λ
( z i | x i ) .
The model likelihood functions p θ ( x | z ) ’s are
Multinomial distribution (for the user matching and link prediction experiments) and Bernoulli distribution (for the spectral clustering experiments).
The singleton variational approximations q λ ( z i | x i ) are all diagonal normal distributions.
The singleton prior density function p 0 ( · ) is the standard normal distribution.
For the
CVAE s, we set the pairwise prior density function p 0 ( · ) =
N (cid:18) µ = 0 2 d , Σ = (cid:18) I d τ · I d τ · I d
I d (cid:19)(cid:19) for τ = 0 . 99 .
It can be seen that, the singleton prior density function p 0 ( · ) and the pairwise prior density function p 0 ( · , · ) satisfy the constraints in Eq. 2.
For the CVAE corr , we treat q λ
( z i , z
j | x i , x j ) as a multivariate normal distributions such that the covariance matrices
Cov ( z i , z i ) , Cov ( z j , z j ) and Cov ( z i , z j ) are all diagonal matrices.
Instead of only learning the singleton density function q λ ( ·|· ) , the CVAE corr also learn a two-layer feedforward neural network that takes the concatenation ( x i , x j ) as input and output the covariance between z i and z j on each of these d dimensions.
As a result, q λ ( z i , z j | x i , x j ) can be factorized as a product of d bi-variate normal distributions, whose marginal distributions on z i and z j are consistent with the singleton variational approximations q λ ( z i | x i ) and q λ ( z j | x j ) , respectively.
For GraphSAGE, we choose to use K = 2 aggregation steps and use the mean aggregator function.
We use Q = 20 negative samples to optimize the loss function.
For all methods, we apply stochastic gradient optimizations with a step size of 10 − 3 .
We use the Adam algorithm (Kingma & Ba, 2015) to adjust the learning rates.
All methods involve with stochastic batches with singleton terms.
For these terms, we use a batch size B 1 = 64 .
For the CVAE s, there are some pairwise terms of the sum involve sampling edges from the graph G =
( V, E ) (i.e. the “positive sampling”) or sampling edges from the complete graph K n (i.e. the “negative sampling”).
We use a batch size B 2 = 256 for sampling these pairwise terms.