Learning Dynamic Graph Representations
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We address two fundamental questions that arise in learning over dynamic graphs:(i) How to elegantly model dynamical processes over graphs?
(ii) How to leverage such a model to effectively encode evolving graph information into lowdimensional representations?
We present DyRep - a novel modeling frameworkfor dynamic graphs that posits representation learning as a latent mediation processbridging two observed processes – dynamic of the network (topological evolution)and dynamic on the network (activities of the nodes).
To this end, we propose aninductive framework comprising of two-time scale deep temporal point processmodel parameterized by a temporal-attentive representation network and trainedend-to-end using an efﬁcient unsupervised procedure.
We demonstrate that DyRepsigniﬁcantly outperforms state-of-art baselines for dynamic link prediction andevent time prediction and provide extensive qualitative analysis of our framework.
Introduction
Representation learning over graph structured data has ubiquitous applicability in variety of domainssuch as social networks, bioinformatics, natural language processing, and relational knowledge bases.
Learning node representations to effectively encode high-dimensional and non-Euclidean graphinformation is a challenging problem but recent advances in deep learning has helped importantprogress towards addressing it [1, 2, 3, 4, 5, 6, 7], with majority of the approaches focusing onadvancing the state-of-art in static graph setting.
However, several domains (e g. social networkcommunications, ﬁnancial transaction graphs or longitudinal citation data) now present highlydynamic data that exhibit complex temporal properties in addition to earlier cited challenges.
Theserecent developments have created a conspicuous need for principled approaches to advance graphembedding techniques for dynamic graphs [8].
We focus on two pertinent questions fundamental torepresentation learning over dynamic graphs: (i) What can serve as an elegant model for dynamicprocesses over graphs?
(ii) How can one leverage such a model to learn dynamic node representationsthat are effectively able to capture evolving graph information over time?
As noted in [9], an important requirement to effectively learn over such dynamical systems is theability to express the dynamical processes at different scales.
We propose that any dynamic graph must be minimally expressed as a result of two fundamental processes evolving at different time scales: Association Process (dynamics of the network), that brings change in the graph structure andleads to long lasting information exchange between nodes; and Communication Process (dynamicson the network), that relates to node’s self evolution and activities between (not necessarily connected)nodes which leads to temporary information ﬂow between them [10, 11].
We, then, posit our goalof learning node representations as modeling a latent mediation process that bridges the above twoobserved processes such that learned embeddings drive the complex temporal dynamics of both processes and these processes subsequently lead to the nonlinear evolution of node representations.
Further, the information propagated across the graph is governed by the temporal dynamics of communication and association histories of nodes with its neighborhood (Figure 3 Appendix A )
Preliminaries
Related Work — Representation Learning approaches for static graphs either perform node embedding [1, 2, 3, 4, 5, 6, 7] or sub-graph embedding [12, 13, 14] which can also utilize convolutionalneural networks [15, 16, 17, 18].
Dynamic network embedding is pursued through various techniques such as matrix factorization [19], structural properties [20], CNN-based approaches [21],deep recurrent models [22], and random walks [23].
Literature on temporal modeling of dynamicnetworks [24], that focus on link prediction tasks is orthogonal to our work as they do not focuson representation learning.
Authors in [25, 26] proposed models of learning dynamic embeddingsbut none of them consider time at ﬁner level and do not capture both topological evolution andinteractions simultaneously.
Temporal Point Processes [27] have gained lot of attention recently formodeling dynamic systems [28, 29, 30, 31, 32].
In parallel, research on deep point process modelsinclude parametric approaches to learn intensity [33, 34] using RNN and GAN based approaches tolearn intensity functions [35].
Notations — Let G t =
( V t , E t ) denote graph G at time t , where V t isthe set of nodes and E t is the set of edges in G t .
An event at time t is represented as e = ( u, v, t, l, k ) ,where u, v are the two nodes involved in an event.
l ∈ { 0 , 1 } represent link status – l = 1 signifya structural edge and l = 0 signify the two nodes are not directly connected.
k ∈ { 0 , 1 } and k = 0signify an association event and k = 1 signify a communication event.
We then represent complete setof P observed events ordered by time in window [0 , T ] as O = { ( u, v, t, l, k ) p } Pp =1 .
Here, t p ∈ R + ,0 ≤ t p ≤ T .
Let z v ( t )
∈ R d represent d -dimensional representation of node v at time t .
We usez v (¯ t ) for most recently updated embedding of node v just before t .
Proposed Method: DyRep
The rationale behind our framework is that the observed set of events are the realizations of thenonlinear dynamic processes governing the changes in topological structure of graph and interactionsbetween the nodes in the graph.
Now, when an event is observed between two nodes, informationﬂows from the neighborhood of one node to the other and affects the representations of the nodesaccordingly.
While a communication event (interaction) only propagates local information acrosstwo nodes, an association event changes the topology and thereby has more global effect.
The goal isto learn node representations that encode information evolving due to such local and global [-effects.
-]effects.
Modeling Two-Time Scale Observed Graph Dynamics.
Given an observed event p =
( u, v, t, l, k ) ,we deﬁne a continuous-time deep model of temporal point process using the intensity functionλ u,vk ( t ) that models the occurrence of event p of dynamics k between nodes u and v at time t :λ u,vk ( t ) =
f k ( g k (¯ t )) .
The inner function g k (¯ t ) computes the compatibility of the most recentlyupdated representations of two nodes, z u (¯ t ) and z v (¯ t ) as follows:
g k (¯ t ) =
ω
Tk · [ z u (¯ t ); z v (¯ t )] .
Here,[;] signiﬁes concatenation and ω k ∈ R 2 d serves as the model parameter that learns time-scale speciﬁccompatibility.
The choice of outer function f k needs to: 1.) keep intensity positive.
account fordifference in time-scale of dynamics corresponding to communication and association processes.
To account for this, we use a modiﬁed version of softplus function parameterized by a dynamicsparameter ψ k to capture this timescale dependence: f k ( x ) =
ψ k log(1+exp( x/ψ k )) where, x = g (¯ t )in our case and ψ k is scalar time-scale parameter learned as part of training.
ψ k corresponds to therate of events arising from a corresponding process.
Learning latent Mediation Process Via Temporally Attentive Representation Network.
Afteran event has occurred, the representation of both the participating nodes need to be updated tocapture the effect of the observed event based on the principles of: Self-Propagation. , Exogenous
Drive.
and Localized Embedding Propagation.
Two nodes involved in an event form a temporary(communication) or a permanent (association) pathway for the information to propagate from theneighborhood of one node to the other node.
For any event at time t , we update the embeddings forboth nodes involved in the event using a recurrent architecture.
Speciﬁcally, for p -th event of node v ,we evolve z v as:
[Equation]
where, h ustruct ∈ R d is the output representation vectors obtained from aggregator function on node 87 u ’s neighborhood and z v ( t p − 1 ) is the recurrent state.
t p is time point of current event, ¯ t p signiﬁesthe timepoint just before current event and t p − 1 represent time point of previous event for node v .
W struct , W rec ∈ R d × d and W t ∈ R d are model parameters that govern the aggregate effect of all the three processes respectively.
The above formulation is ﬂexible in supporting various features.
Forexample, in case of dynamic heterogenous graphs, Eq. 1 can be extended to include edge type withcorresponding parameters.
Similarly, one can support high-dimensional attributes (including spatial)by adding them to Eq.
1 as a feature.
Temporally Attentive Aggregation.
We propose a novel Temporal Point Process based Attentional Mechanism that empowers the aggregate function (used to compute h struct ) to attend to the neighborsbased on node’s communication and association history.
Let A ( t ) ∈ R n × n be the adjacency matrix for graph G t at time t .
Let S ( t ) ∈
R n × n be a stochasticmatrix capturing the strength between pair of vertices at time t 2 .
One can consider S as a selectionmatrix that induces a natural selection process for a node – it would tend to communicate morewith other nodes that it wants to associate with or has recently associated with.
And it wouldwant to attend less to non-interesting nodes.
Formally, we perform localized attention for a givennode u and compute the coefﬁcients pertaining to the 1-hop neighbors i of node u as: q ui ( t ) = exp( S ui (¯ t )) (cid:80) [Equation]
S
[Equation] t )) , where q vi signiﬁes the attention weight for the neighbor i at time t and hence itis a temporally evolving quantity. These attention coefﬁcients are then used to compute the aggregateinformation h ustruct (¯ t ) for node v by employing an attended aggregation mechanism across neighborsas follows: h ustruct (¯ t ) = max (cid:0)(cid:8) σ (cid:0) q ui ( t ) · h i (¯ t ) (cid:1) , ∀ i ∈ N u (¯ t ) (cid:9)(cid:1) , where, h i (¯ t ) =
W h
z i (¯ t ) +
b hand W h ∈ R d × d and b h ∈ R
d
are parameters governing the information propagated by eachneighbor of u .
z i (¯ t ) ∈
R d is the most recent embedding for node i .
Efﬁcient Learning Procedure.
The complete parameter space for the current model is Ω ={ W struct , W rec , W t , W h , b h , { ω k } k =0 , 1 , { ψ k } k =0 , 1 } .
For a set O of P observed events, welearn these parameters by minimizing the negative log likelihood: L = − (cid:80) Pp =1 log ( λ p ( t ))
+(cid:82) T 0 Λ( τ ) dτ , where λ p ( t ) =
λ u p ,v p k p ( t ) represent the intensity of event at time t and Λ( τ ) =(cid:80) nu =1 (cid:80) nv =1 (cid:80) k ∈{ 0 , 1 } λ u,vk ( τ ) represent total survival probability for events that do not happen.
While it is intractable (will require O ( n 2 k ) time) to compute the integral in log-likelihood equationfor all possible non-events in a stochastic setting, we locally optimize L using mini-batch stochasticgradient descent where we estimate the integral using novel sampling technique 3 .
Experiments
We evaluate DyRep and baselines on two real world datasets: Social Evolution Dataset releasedby MIT Human Dynamics Lab —
#nodes: 100, #Initial Associations: 407, #Final Associations:809, #Communications: 2020554 and Clustering Coefﬁcient: 0.548.
Github Dataset availableat Github Archive — #nodes: 12328, #Initial Associations: 70640, #Final Associations: 166565,#Communications: 604954 and Clustering Coefﬁcient: 0.087.
We study the effectiveness of DyRepby evaluating our model on tasks of: (i) Dynamic link prediction – for a given test record ( u, v, t, l, k ) ,we replace v with other entities in the graph and compute the conditional density f u,vk ( t ) =
λ u,vk ( t ) ·exp (cid:16)(cid:82) t ¯ t λ ( s ) ds (cid:17) , where ¯ t is the time of the most recent event on either dimension u or v .
We thenrank all the entities in descending order of the density and report Mean Average Rank and HITS(@10)metric for dynamic link prediction.
(ii) Event Time Prediction – Given a pair of nodes ( u, v ) andevent type k at time t , we use the above density formulation to compute conditional density at time t .
The next time point ˆ t for the event can then be computed as: ˆ t = (cid:82) ∞ t tf u,vk ( t ) dt where the integraldoes not have an analytic form and hence we estimate it using Monte Carlo trick.
For a given testrecord ( u, v, t, l, k ) , we predict the next time this communication event may occur and report MAEagainst the ground truth.
For Dynamic Link Prediction task, we compare the performance of ourmodel against Know-Evolve [22], DynGem [36], DynTrd [20], GraphSage [18] and Node2Vec [2].
For Event Time Prediction, we compare our model against Know-Evolve which has the ability topredict time in a multi-relational dynamic graphs and Multi-dimensional Hawkes Process (MHP)
[37]model where all events in graph are considered as dyadic.
During evaluation, We divide our testsets into n (= 6) slots based on time and report the performance for each time slot, thus providingcomprehensive temporal evaluation of different methods. [Figure] [Figure] (unit is hrs).
Figure best viewed in pdf or colored print.
4.1 Experimental Results
Communication Event Prediction Performance.
Figure 1 (a-b) shows corresponding results.
For
Social Evolution dataset, our method signiﬁcantly and consistently outperforms all the baselines onboth metrics.
While the performance of our method drops a little over time, it is expected due tothe temporal recency affect on node’s evolution.
For Github dataset, we demonstrate comparableperformance with both Know-Evolve and GraphSage on Rank metric.
We would like to note thatoverall performance for all methods on rank metric is low.
Github dataset is very sparse with verylow clustering coefﬁcient and challenging dataset to learn.
It is expected that for a large number ofnodes with no communication history, most of the methods will show comparable performance butour method outperforms all others when there is some history available.
This is demonstrated byour signiﬁcantly better performance for HITS@10 metric where we are able to do highly accurateprediction for nodes where we learn better history.
Association Event Prediction Performance.
Association events are not available for all time slots so
Figure 1 (c-d) report the aggregate number for this task.
For both the datasets, our model signiﬁcantlyoutperforms the baselines for this task.
Speciﬁcally, our model’s strong performance on HITS@10metric across both datasets demonstrates its robustness in accurate learning from various propertiesof data.
On Social evolution dataset, the number of association events are very small (only 485) andhence our strong performance shows that the model is able to capture the inﬂuence of communicationevents on the association events through the learned representations (mediation).
On the Githubdataset, the network grows through new nodes and our model’s strong performance across both metricdemonstrates its inductive ability to generalize across new nodes across time.
Time Prediction Performance.
Figure 2 demonstrates consistently better performance than state-ofart baseline for event time prediction on both datasets.
While Know-Evolve models both processes astwo different relations between entities, it does not explicitly capture the variance in the time scales oftwo processes.
Further, Know-Evolve does not consider inﬂuence of neighborhood which may leadto capturing weaker temporal-structural dynamics across the graph.
MHP uses speciﬁc parametricintensity function which fails to account for intricate dependencies across graph.
We refer the readersto Appendix B for sample qualitative analysis of our framework.
Conclusion
We introduced a novel modeling framework for dynamic graphs that effectively and efﬁciently learnsnode representations by posing representation learning as latent mediation process bridging dynamicprocesses of topological evolution and node interactions.
We proposed a deep temporal point processmodel parameterized by temporally attentive representation network that models these complex andnonlinearly evolving dynamic processes and learns to encode structural-temporal information overgraph into low dimensional representations.
Our superior evaluation performance demonstrates theeffectiveness of our approach compared to state-of-arts. Our framework can support wide range ofdynamic graph characteristics which can potentially have many exciting adaptations. As a part ofour framework, we also propose a novel temporal point process based attention mechanism that canattend over neighborhood based on the history of communications and associations in the graph. Appendix
A Overview of DyRep Framework
B Sample Qualitative Performance
[Figure]
Qualitative Performance.
Figure 4 (a-b) shows the tSNE embeddings learned by Dyrep (left) and
GraphSage (right) respectively.
The visualization demonstrates that DyRep embeddings have morediscriminative power as it can effectively capture the distinctive and evolving structural features overtime as aligned with empirical evidence.
Figure 4 (c-d) shows use case of two associated nodes (19and 26) that has less communication at the two time points for above two methods.
DyRep keeps theembeddings nearby although not in same cluster (cos. dist. - 0.649) which demonstrates its abilityto learn the association and less communication dynamics between two nodes.
For GraphSage theembeddings are on opposite ends of cluster with (cos.
dist. - 1.964) 4 .