Nonparametric Exponential Family Graph Embeddings for Multiple Representation Learning
Abstract
In graph data, each node often serves multiple functionalities.
However, most graph embedding models assume that each node can only possess one representation.
We address this issue by proposing a nonparametric graph embedding model.
The model allows each node to learn multiple representations where they are needed to represent the complexity of random walks in the graph.
It extends the
Exponential family graph embedding model with two nonparametric prior settings, the Dirichlet process and the uniform process.
The model combines the ability of Exponential family graph embedding to take the number of occurrences of context nodes into account with nonparametric priors giving it the flexibility to learn more than one latent representation for each node.
The learned embeddings outperforms other state of the art approaches in link prediction and node classification tasks.
INTRODUCTION
Data in the form of graphs is drastically growing across disciplines to represent complex observations and their relationships in the graph topology.
One common challenge for such data is unsupervised representation learning (embedding) which discovers underlying functions or characterizations of nodes solely from the graph structure without requiring availability of node attributes.
Such research has shown encouragingly that the learned latent representations can be used as features for different predictive tasks with promising performance.
Despite the success of such models most of the proposed methods consider only the co-appearance pattern of nodes in walks across a graph.
The prominence of nodes in their surroundings, for example as hubs or bridges, is an important trait of the network structure but is often ignored.
Moreover, it is a common phenomenon that each graph node can serve different functions or roles: a node can, for example, act both as a local hub for its nearby nodes and also as a crucial bridge along a path between far-off connected areas of a graph. However, most methods are unable to properly represent this: they are restricted to single representation learning where each node is only assigned one latent vector representation.
A model that only supports one embedding per node tries to collapse all underlying roles of the node into one vector representation could omit necessary information: this can yield poor representations that are ’inbetween’ the roles of the node and do not represent any of them well or represent only some roles while ignoring others.
In this paper we introduce a novel embedding model, which extends exponential family embedding [Rudolph et al., 2016] with nonparametric priors and allows a node to have more than one latent representation.
We allocate such latent representations following two nonparametric priors, the
Dirichlet process and the uniform process.
While Dirichlet processes are popular in nonparametric modeling, the uniform process has been neglected in such models; our results show the uniform process is a promising prior for the proposed model.
A tailored truncation-free inference algorithm is developed.
Different from the traditional approaches, the algorithm introduces new latent embedding vectors over iterations which provides more efficient inference.
We evaluate the proposed model with two tasks, link prediction and node classification.
Results over several datasets show the proposed multiple representation learning method improves performance compared to state of the art baselines. The contributions of this work are:
• We introduce the notion of multiple representation to graph embeddings: each node can have more than one latent vector representation.
• We propose a graph embedding model leveraging Bayesian nonparametrics, which is unprecedented and challenging to do well.
The number of latent represen-
tations are thus decided by the observed data. • In addition to the Dirichlet process, we explore the uniform process, and show it is an important option for achieving best results. • We develop an adaptive inference algorithm for efficient computation. The paper is organized as follows.
Section 2 describes background concepts.
Section 3 introduces the proposed model.
Section 4 develops the inference algorithm.
Experiments are conducted in Section 5 and Section 6 draws the conclusions.
FUNDAMENTAL CONCEPTS
This section provides a brief overview of some basic concepts that are related to our approach.
2.1 EXPONENTIAL FAMILY EMBEDDING
Exponential family embedding (EFE)
[Rudolph et al., 2016] is a probabilistic extension of the CBOW embedding model [Mikolov et al., 2013a,b].
Observations are made of objects v that occur at locations n surrounded by a context which is a set of other objects.
In a traditional word embedding scenario an object would be a word and the context would be the surrounding words in a sentence; in the graph embedding scenario that we address, objects are instead nodes of a graph and contexts are other nodes on a random walk in the graph.
Let x n,v denote the observed value for object v at location n .
Denote the context by a set c n = { v ′ } of other objects v ′ and a vector ˜ x c n = { ˜ x n,v ′ } of their values in the context.
In our graph embedding case, the values represent whether the object (graph node) occurs at the location and how many times the context objects (nodes) occur in the context.
In EFE, conditioning on the context set c n and context values ˜ x c n , the observed value x n,v for object v is assumed to be exponential family distributed:
[Equation]
where ExpFam is an exponential family distribution, η v ( c n , ˜ x c n ) is the natural parameter, and T (cid:0) x n,v (cid:1) denotes the sufficient statistics.
In EFE, each object v is represented in two ways, with an embedding vector ρ v ∈ R D and a context vector α v ∈ R D where D is the embedding dimensionality.
The EFE captures the co-occurrence pattern by constructing the natural parameter based on interaction between the embedding vector of the center object and the context vectors of its context objects weighted by their context values. The model can be seen as a special generalized linear model since the natural parameter is modeled as a link function of an inner product, so that
[Equation]
Since ExpFam can be any exponential distribution,
CBOW can be seen as the special case of employing a
Bernoulli distribution where the observed value x n,v can be either 1 or 0 .
One principal merit of the generalization to other probability distributions is the capability of capturing latent patterns by incorporating the observed values.
For example, in a shopping cart scenario, quantity of an observed item is modeled by the quantities of its context items (i.e., other products in the shopping cart) which are not binary but positive integers.
Similarly, in a graph embedding scenario counts of graph nodes in a context will be positive integers.
2.2 RANDOM WALK BASED NODE EMBEDDING
Let G =
( V , E ) be a graph where V denotes the set of vertices, and
E ⊆ V × V denotes the edge set.
A random walk w = { w 1 , . . .
, w L } of length L is a simulated sequence of nodes over the graph where each node is chosen at random from the neighbors of the previous node.
Extraction of such random walks is a way to describe a graph by extracting sequence data representing graph connectivity.
Such sequences can then be modeled by a generative model. Random walk based embedding approaches [Perozzi et al., 2014, Grover and Leskovec, 2016] model co-occurrence of nodes in a set of random walks W .
The generative process models the sequence content, and thus the graph connectivity, through embeddings of nodes: the model is conditional on the nodes and generates the sequences.
Given a walk w ∈ W , the occurrence of node w n at position n in the walk is conditional on the set c n of its surrounding (context) nodes in the walk.
The occurrence probability is modeled as depending on embedding vectors of the node and embedding vectors of the context nodes.
The representation learning aims to optimize the probability of occurrence of the nodes w n given their contexts, i.e., [Equation]
2.3 BAYESIAN NONPARAMETICS
In Bayesian nonparametric models, the number of parameters is not fixed in advance but learned during model fitting up to a potentially infinite number of parameters.
The models are typically described as mixtures: each observation is modeled by a parameter drawn from a distribution G over the space of parameters (e.g. R D ) where only a finite number of parameter values have nonzero probability, but G itself is drawn as
[Equation]
from a stochastic process prior NP with base distribution
G 0 and concentration parameter γ .
The process NP yields distributions over the parameter space, with different numbers of possible values up to a potentially infinite number, but each draw from NP has a finite number.
Thus fitting the model to data with the prior NP will infer how many parameters are needed to describe the data.
2.4 RELATED WORK
Among random walk based unsupervised node embeddings, Deepwalk [Perozzi et al., 2014] has been the classical method.
Grover and Leskovec [2016], Ribeiro et al.
[2017] simulate variant random walks emphasizing different structural features of the graph.
Celikkanat and Malliaros [2020] extend the models with different likelihoods with
EFE framework; in their work, the context vectors are taken to represents the vertices.
A group of models have been proposed to learn multiple representations.
Among those, Sun et al.
[2019] decide the number of embedding with a community detection task; Liu et al.
[2019], Park et al.
[2020], Chen et al.
[2020] impose a fixed number of embedding vectors for all nodes with a predefined value.
The most similar method to ours is Epasto and Perozzi [2019] which uses local neighborhood clustering to generate multiple representations for nodes where different nodes can have different number of embedding vectors.
Those methods often depend on extra simulations of the graph data in addition to the random walks data, whereas our method only requires the generated random walks.
Besides random walk based methods, there are other proposed approches include, for example, methods based on matrix factorization [Ou et al., 2016, Wang et al., 2017, Qiu et al., 2018] and neural network based approaches [Li et al., 2018, Velickovic et al., 2019, Wu et al., 2020].
PROPOSED MODEL
The proposed model is a Bayesian nonparametric extension of exponential family node embedding.
We next describe the two notions and how they are used to learn multiple node representations.
Figure 1 shows an overall illustration.
In the figure, random walks are first extracted from a graph, yielding sequences whose sliding windows each contain a center node and counts of other nodes in the context. The occurrence of the center node will be modeled based on the context, where dependency is characterized using vectorial embeddings: each node has one embedding as a context node and can have multiple embeddings as a center node.
The generation of the observed sequence content can be written as a graphical plate representation where nonparametric priors are used to generate the embedding vectors of center nodes, and the center and context embedding vectors together are used to generate observed values, that is, the observed center nodes in each window of a random walk.
3.1 EXPONENTIAL FAMILY NODE EMBEDDINGS
Given a simulated random walk node sequence w = { w 1 , . . .
, w L } of length L , we slide windows of length K along it.
In each window the center node w n is surrounded by context nodes { w n − K , . . .
, w n − 1 , w n +1 , . . .
, w n + K } .
For each possible vertex v we denote x n,v = 1 if it was the center node so that w n = v , otherwise x n,v = 0 .
The context is denoted by the set c n of unique vertices in the context nodes and the counts ˜ x
c n = { ˜ x n,v ′ } how many times each vertex v ′ ∈ c n occurred in them, ˜ x n,v ′ ≤ K − 1 .
We will model dependency of node occurrences along a sequence, based on distributions whose natural parameter compares observed values to their context.
In more detail, the natural parameter is based on comparison of node embedding vectors that characterize what kind of surroundings each node tends to appear in.
We first describe the distribution and then describe the construction of the natural parameter for different exponential families (different likelihoods).
We model the co-occurrence pattern between w n and the context ( c n , ˜ x c n ) with an exponential family
[Equation]
where η v ( c n , ˜ x c n ) is the natural parameter and T (cid:0) x n,v (cid:1) the sufficient statistics.
In this work occurrence of a node is represented as a one-hot choice vector and it is modeled as a draw from an exponential family distribution whose parameters depend on the surrounding nodes.
Concretely, if the vertex appears at the location n , the positive likelihood is then defined as
[Equation]
where f is the corresponding probability density function of the exponential family distribution.
For a vertex that does not appear at location n , the likelihood of the non-appearance (also called a ‘negative likelihood’) is
[Equation]
Since random walks only yield positive samples of vertices that occurred in the center of their windows, learning from them alone would bias the model; thus we use a popular negative sampling approach, and randomly generate several negative samples (5 in experiments) for each location n .
A negative sample has the same context ( c n , ˜ x c n ) as the positive sample at n , but x n,v is instead set to 1 for a random vertex among those that did not appear in the location.
In this work, we explore three different exponential family distributions: Bernoulli, Poisson, and Gaussian. [Figure] 1  6 ρρρ 1 ρ 1 ρρ 6 ρ 5 ρρ 7 ααα 6 αααα [Figure]
Bernoulli Likelihood .
We employ Bernoulli distribution to model the co-occurrence patterns of nodes.
Let ρ n,v ∈ R D denote the embedding vector of the node v at the location n , α v ∈ R D denote the embedding vector for the vertex v , the natural parameter is then defined as
[Equation]
[Equation] where S denotes the sigmoid function S = [Equation] x , and | c n | is the number of distinct nodes in the context.
The appearance of the node v at the location n , i.e. whether x n,v = 1 or x n,v = 0 , is thus sampled from a Bernoulli distribution with parameter p n so that
[Equation]
Note that we use the Bernoulli likelihood to model only the co-appearance of the nodes, which can be seen as an extension of Skip-gram based models.
The number of occurrences of nodes in the context is not taken into the account.
To incorporate the number of occurrences of nodes, we employ the Poisson and Gaussian distributions.
Poisson Likelihood .
For a Poisson distribution, the parameter λ n is defined as
[Equation]
where | c n | is again the number of distinct nodes in context and x n,v ′ denotes the number of occurrences of node v ′ in the context.
The appearance of the node v is generated as
[Equation]
The pivotal difference between the Bernoulli and Poisson cases is that the latter takes the number of occurrences of nodes in the context into account when constructing the natural parameter.
The Gaussian case takes the same setting.
Gaussian Likelihood .
Similar to the settings for Poisson
Likelihood, the natural parameter here is defined as
[Equation]
without a specific link function, and the appearance of the node v at the location n is generated as
[Equation]
where we set σ as a fixed hyper-parameter; in the experiments we arbitrarily choose the σ from {1, 5, 10}.
When several different likelihoods are feasible, The model choice can depend on domain expertise, or cross-validation can be used as a model selection process.
3.2 NONPARAMETRIC EMBEDDING
Instead of restricting each vertex v to have a single role represented, to better capture the complexity of vertex roles in a graph as observed in random walks, we present a multiple representation learning model which enables each vertex to have multiple latent vector representations, so that the ocurrence of the the vertex at each location in a walk can arise from a different role of the vertex.
To do so, we set a nonparametric prior on the embedding vectors ρ .
That is, we assume that at each location n , an embedding vector ρ n,v is generated from a stochastic process G v specific to the vertex, so
that
[Equation]
where G v is a stochastic process with a base distribution Gand a concentration parameter γ .
The base distribution Ghas an infinite number of possible embedding vectors and
G v is a draw from it allocating nonzero probability to a finite number of possibilities { ρ (1) v , . .
, ρ ( s ) v , . .
, ρ ( S ) v , . .
. , } where S is the number of observed embedding vectors.
We set the base distribution to be a d -dimensional Normal distribution N (0 , σ 0 ) .
In experiments we set σ 0 = 5 for Bernoulli likelihood and σ 0 = 10 for both Poisson and Gaussian likelihood.
For simplicity, similar to the settings of Rudolph et al.
[2017], Rudolph and Blei [2018], although we allow multiple embedding vectors ρ n,v for a vertex we will use only one context vector
α v per vertex; this setting can already generate good results in the experiments, and generalization to allow multiple context vectors is a future work.
In the following, let n v = n + v ∪ n − v denote locations related to vertex v , so that n + v denotes locations where the v appears and n − v locations where v is the negative sample.
Moreover, denote by n v,<n the subset of n v where the location is before n , and denote by superscript ( s ) those locations where the embedding vector was the s :th embedding vector of v .
Dirichlet Process .
One of the most common nonparametric process priors is a Dirichlet process.
The predictive probability of ρ n,v is defined based on numbers of occurrences of embedding vectors of v at earlier locations n ′ < n in positive or negative samples, so that
[Equation]
where | n ( s ) v,<n | is the number of locations before n where ρ ( s ) v has been selected, and γ governs the generation of a new embedding vector.
Uniform process .
An alternative to Dirichlet process is a uniform process [Wallach et al., 2010] with the predictive probability
[Equation]
where S v denotes the number of different embedding vectors used for v before location n , and the embedding vector ρ n,v is generated independently from the occurrence frequencies of previous generated values.
The generation is only controlled by the concentration parameter γ .
Despite the popularity of Dirichlet process, it suffers from the “rich get richer” issue, as it tends to repeat previous values and tends to model the first (or first few) embedding vectors as highly dominant, which can limit model flexibility.
The uniform process was proposed to address this issue.
Figure 2 show an example where the Dirichlet process concentrates on the first embedding vector and uniform process delivers smoother weights.
The uniform process has been neglected by the research community, with most applications employing Dirichlet processes as priors.
Overall generative process.
The proposed model can be summarized with the generative process shown below (corresponding plate model shown in Figure 1, Right):
For each vertex v ∈ V : [Equation]
For each walk w = { w 1 , . . . , w L } ∈ W - For location n :
[Equation]
INFERENCE
We adapt a truncation-free variational inference algorithm proposed by [Huynh et al., 2016].
Using a stick-breaking construction [Sethuraman, 1994]
[Equation]
for vertex v we have
[Equation]
The posterior distribution for the stick breaking parameters β
[Equation]
) is then
[Equation]
where parameter θ v governs the general prevalence over all potential embedding vectors.
For each location, the embedding vector ρ n,v is decided by a label z n,v sampled from a
Multinomial distribution
[Equation]
The variational distribution q ( z v,n ) is updated as
[Equation]
where the first term is the fitness of the selected embedding ρ ( s ) v [Equation] and the second term is related to the prior.
If the prior is a Dirichlet process [Equation] the second term in Equation (20) is
[Equation]
where n v denotes the locations of vertex v and | n v | denotes its size.
We then have
[Equation]
On the other hand, if the prior is a uniform process, the second term in Equation (20) has a simpler form:
[Equation]
The truncation-free algorithm starts with setting S = 1 , where q ( z ( S +1) v,n ) = 0 .
When E [ θ ( S +1) v ] > 1 , the algorithm sets S = S + 1 , increasing the dimension of vector z v,n , and sets q ( z ( S +1) v,n ) = 0 .
We can then use the θ v to calculate the expected weighting of the vector ρ ( s ) v .
[Equation]
Algorithm input : Random walks W , negative samples ˜ W , initial learning rate ξ , number of epochs, number of mini-batches M output : embedding vectors Φ = { ρ , α } , embedding weights { ˆ β } foreach v ∈ V do
Set S v = 1 , initialize embedding vectors ρ (1) v , α v endforeach epoch do
Divide input data into M random partitions.
for m ← 1 to M do
Use the subset W ( m ) and ˜ W ( m ) foreach v do foreach n ∈ n ( m ) v do update z n,v with Equation (20) end updata θ v with Equation (22) - (25)
Calculate ˆ β v with Equation (27) if E [ θ ( S +1) v ] > 1 then
S v = S v +foreach n ∈ n v do increase the dimension of z n,v and set z ( S +1) n,v =end end end update embedding vectors Φ = { ρ , α } Φ = Φ
[Equation] ξ ∗ ∂ L ∂ Φ ξ is set with Adam[Kingma and Ba, 2015] end end
Inference of embedding vectors. After updating the E q (cid:2) z n,v (cid:3) , the inference is conducted by optimizing the objective function L = L prior + L likelihood .
The term L prior = log p ( ρ ) + log p ( α ) is derived from the Gaussian prior N (0 , σ 2 0 ) for the embedding vectors: log p [Equation] [Equation]
Data
[Table] Data [Table]
For Bernoulli likelihood we have
L likelihood = (cid:88) [Equation] (cid:88) [Equation]
(cid:88)
[Equation]
For Poisson likelihood we have
L likelihood = (cid:88)
[Equation]
For Gaussian likelihood, we have
L likelihood = (cid:88) v
[Equation]
(cid:88)
[Equation] (cid:33) [Equation] We then use gradient descent to update the embedding vectors over iterations.
4.1 STOCHASTIC INFERENCE
We employ stochastic inference.
For each epoch, the input data is randomly partitioned into M mini-batches and only one mini-batch is used for each iteration.
When mini-batch m is used, the sum over locations n v can be approximated by a sum over a subsampled set n ( m ) v , so the right-hand side of (22) is approximated by | n v | | n ( m ) v | (cid:80) n ∈ n ( m ) v q ( z n,v = s ) and similarly in the other sums.
The inference procedure is summarized in Algorithm 1.
For all the experiments conducted in this work, we run two epochs with 1000 mini-batches and initial learning rate ξ = 0 .
For the negative samples, we generate ˜ W with 5 negative samples for each location following the procedure of Mikolov et al.
[2013b], Celikkanat and Malliaros [2020].
EXPERIMENTS
For generality, we run experiments with two standard tasks commonly adopted in graph embedding works, link prediction and node classification, with 3 data sets [Csardi and Nepusz, 2006, Rossi and Ahmed, 2015, Rozemberczki et al., 2020] for each task (Tables 1 and 2).
The data sets cover varied domains and aim to represent typical use scenarios of the proposed method.
We denote our method variants by prior (dp: Dirichlet process, up: uniform process) and ExpFam distribution (Bern, Pois, Norm), e.g. ‘up-emb (Bern)’.
We compare to random walk based methods DeepWalk [Perozzi et al., 2014], node2Vec [Grover and Leskovec, 2016], struc2vec
[Ribeiro et al., 2017], and EFGE [Celikkanat and Malliaros, 2020], and Splitter [Epasto and Perozzi, 2019].
To evaluate effect of embedding dimensionality, for each method we run three dimension settings: D = 50 , 100 , and 150 .
The concentration parameter for our model is chosen from γ = { 0 . 01 , 0 . 05 , 0 .
1 } for Dirichlet process and γ = { 0 . 0000001 , 0 . 0000005 , 0 . 000001 } for uniform process.
The input random walks are generated with the R package igraph [Csardi and Nepusz, 2006] with 80 walks per node with length L = 10 , the random walks are also fed to EFGE.
For other methods, parameters are all set to default values.
5.1 TASK: LINK PREDICTION
In link prediction, for each graph we first randomly move 50% of the edges into a held-out test set while keeping the remaining training graph connected.
In both training and test sets, randomly sampled negative edges are added in equal amount to the positive edges.
A classifier is trained based on the reduced training graph and the training negative edges; the classifier is used to classify the held-out test-set edges.
As in the previous single-representation learning works including Deepwalk, node2vec, struc2vec, and EFGE, logistic regression is selected as the classifier.
In our approach, to incorporate multiple representations when training the classifier, we employ logistic regression with sample weights, embedding ρ ( s ) v is weighted by ˆ β ( s ) v .
The Splitter used maximum dot-product similarity, we transform the similarity into a class probability using logistic regression.
Note that when logistic regression is trained with sample weighting, embeddings of all nodes in our model are separate samples weighted in the log-likelihood by their occurrence probabilities.
The regression learns to classify nodes based on all their embedding vectors, and at test time, a node is classified by weighted average of class probabilities predicted for each of its embedding vectors.
Thus, the
[Table]
Norm) [Table] multiple embedding vectors are treated separately instead of being combined in a simplistic weighted average. Three different datasets are used for the link prediction task.
GitHub : a social network where each node is a GitHub developer, links between nodes are mutual follow relations.
Wikipedia : a network of English Wikipedia pages.
Edges between pages reflect their mutual links.
Twitch : a user-user interaction network between gamers.
Edge between two nodes represents mutual friendship.
We evaluate the binary link classification by area under the curve (AUC).
Table 3 shows our model performs well on all datasets; the model with Gaussian likelihood works best.
5.2 TASK: NODE CLASSIFICATION
In this task, each node has a class.
The learned embedding vectors are used as input features to train a classifier to predict the class of each node.
Again, for Deepwalk, node2vec, struc2vec and EFGE, a logistic regression classifier is used.
For our model, the logistic regression with sample weights is used.
For Splitter, we take the same procedure with each embedding equally weighted.
Three different datasets are used for the node classification task.
LastFM Asia : a network of people living in Asia using the streaming site LastFM.
Links represent followership relations
The class of each node is its location.
CiteSeer : a scientific publication network from the CiteSeer digital library.
Each node belongs to 1 of 6 categories.
Yeast :a protein-protein interaction network.
The “Class” attribute of each protein is based on its function (e.g. energy
We evaluate the performance by Micro-averaged F1, reported in Table 4.
Our model outperforms other methods.
Rozemberczki et al. Additionaly, in general, our model took 2-4 hours to converge (depends on different tasks and settings) without GPU. The Splitter, which also learns multiple representations for each node, took 10+ hours on a GPU machine and 100+ hours without GPU. Our approach achieved better results with less resources.
We proposed nonparametric exponential family graph embedding, allowing multiple node representations, drawn both with a Dirichlet process prior, and also exploring uniform processes.
A tailored algorithm for efficient computation is provided.
The experiments demonstrate the learned multiple representations can enhance performance in two tasks.
We considered three classical exponential family distributions,
Bernoulli, Poisson, and Gaussian, which yielded promising results.
Our model can be adapted to other distributions such as Geometric and Chi-square with the proposed nonparametric framework.
In our experiments, the hyperparameter γ of the nonparametric prior was fixed for the nodes, which already yielded promising results in the standard tasks; having differing γ values could be useful for extending the model to scenarios such as learning multiple representations for under-represented nodes, or imbalanced classification tasks.
Acknowledgements
This work is supported by the Academy of Finland, decisions 312395 and 327352.