Learning Dynamic Graph Representations
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We address two fundamental questions that arise in learning over dynamic graphs:(i) How to elegantly model dynamical processes over graphs?
(ii) How to leverage such a model to effectively encode evolving graph information into lowdimensional representations?
We present DyRep - a novel modeling frameworkfor dynamic graphs that posits representation learning as a latent mediation processbridging two observed processes – dynamic of the network (topological evolution)and dynamic on the network (activities of the nodes).
To this end, we propose aninductive framework comprising of two-time scale deep temporal point processmodel parameterized by a temporal-attentive representation network and trainedend-to-end using an efﬁcient unsupervised procedure.
We demonstrate that DyRepsigniﬁcantly outperforms state-of-art baselines for dynamic link prediction andevent time prediction and provide extensive qualitative analysis of our framework.
Introduction
Representation learning over graph structured data has ubiquitous applicability in variety of domainssuch as social networks, bioinformatics, natural language processing, and relational knowledge bases.
Learning node representations to effectively encode high-dimensional and non-Euclidean graphinformation is a challenging problem but recent advances in deep learning has helped importantprogress towards addressing it [1, 2, 3, 4, 5, 6, 7], with majority of the approaches focusing onadvancing the state-of-art in static graph setting.
However, several domains (e g. social networkcommunications, ﬁnancial transaction graphs or longitudinal citation data) now present highlydynamic data that exhibit complex temporal properties in addition to earlier cited challenges.
Theserecent developments have created a conspicuous need for principled approaches to advance graphembedding techniques for dynamic graphs [8].
We focus on two pertinent questions fundamental torepresentation learning over dynamic graphs: (i) What can serve as an elegant model for dynamicprocesses over graphs?
(ii) How can one leverage such a model to learn dynamic node representationsthat are effectively able to capture evolving graph information over time?
As noted in [9], an important requirement to effectively learn over such dynamical systems is theability to express the dynamical processes at different scales.
We propose that any dynamic graph must be minimally expressed as a result of two fundamental processes evolving at different time scales: Association Process (dynamics of the network), that brings change in the graph structure andleads to long lasting information exchange between nodes; and Communication Process (dynamicson the network), that relates to node’s self evolution and activities between (not necessarily connected)nodes which leads to temporary information ﬂow between them [10, 11].
We, then, posit our goalof learning node representations as modeling a latent mediation process that bridges the above twoobserved processes such that learned embeddings drive the complex temporal dynamics of both processes and these processes subsequently lead to the nonlinear evolution of node representations.
Further, the information propagated across the graph is governed by the temporal dynamics of communication and association histories of nodes with its neighborhood (Figure 3 Appendix A )
Preliminaries
Related Work — Representation Learning approaches for static graphs either perform node embedding [1, 2, 3, 4, 5, 6, 7] or sub-graph embedding [12, 13, 14] which can also utilize convolutionalneural networks [15, 16, 17, 18].
Dynamic network embedding is pursued through various techniques such as matrix factorization [19], structural properties [20], CNN-based approaches [21],deep recurrent models [22], and random walks [23].
Literature on temporal modeling of dynamicnetworks [24], that focus on link prediction tasks is orthogonal to our work as they do not focuson representation learning.
Authors in [25, 26] proposed models of learning dynamic embeddingsbut none of them consider time at ﬁner level and do not capture both topological evolution andinteractions simultaneously.
Temporal Point Processes [27] have gained lot of attention recently formodeling dynamic systems [28, 29, 30, 31, 32].
In parallel, research on deep point process modelsinclude parametric approaches to learn intensity [33, 34] using RNN and GAN based approaches tolearn intensity functions [35].
Notations — Let G t =
( V t , E t ) denote graph G at time t , where V t isthe set of nodes and E t is the set of edges in G t .
An event at time t is represented as e = ( u, v, t, l, k ) ,where u, v are the two nodes involved in an event.
l ∈ { 0 , 1 } represent link status – l = 1 signifya structural edge and l = 0 signify the two nodes are not directly connected.
k ∈ { 0 , 1 } and k = 0signify an association event and k = 1 signify a communication event.
We then represent complete setof P observed events ordered by time in window [0 , T ] as O = { ( u, v, t, l, k ) p } Pp =1 .
Here, t p ∈ R + ,0 ≤ t p ≤ T .
Let z v ( t )
∈ R d represent d -dimensional representation of node v at time t .
We usez v (¯ t ) for most recently updated embedding of node v just before t .
Proposed Method: DyRep
The rationale behind our framework is that the observed set of events are the realizations of thenonlinear dynamic processes governing the changes in topological structure of graph and interactionsbetween the nodes in the graph.
Now, when an event is observed between two nodes, informationﬂows from the neighborhood of one node to the other and affects the representations of the nodesaccordingly.
While a communication event (interaction) only propagates local information acrosstwo nodes, an association event changes the topology and thereby has more global effect.
The goal isto learn node representations that encode information evolving due to such local and global effects.
Modeling Two-Time Scale Observed Graph Dynamics.
Given an observed event p =
( u, v, t, l, k ) ,we deﬁne a continuous-time deep model of temporal point process using the intensity functionλ u,vk ( t ) that models the occurrence of event p of dynamics k between nodes u and v at time t :λ u,vk ( t ) =
f k ( g k (¯ t )) .
The inner function g k (¯ t ) computes the compatibility of the most recentlyupdated representations of two nodes, z u (¯ t ) and z v (¯ t ) as follows:
g k (¯ t ) =
ω
Tk · [ z u (¯ t ); z v (¯ t )] .
Here,[;] signiﬁes concatenation and ω k ∈ R 2 d serves as the model parameter that learns time-scale speciﬁccompatibility.
The choice of outer function f k needs to: 1.) keep intensity positive.
account fordifference in time-scale of dynamics corresponding to communication and association processes.
To account for this, we use a modiﬁed version of softplus function parameterized by a dynamicsparameter ψ k to capture this timescale dependence: f k ( x ) =
ψ k log(1+exp( x/ψ k )) where, x = g (¯ t )in our case and ψ k is scalar time-scale parameter learned as part of training.
ψ k corresponds to therate of events arising from a corresponding process.
Learning latent Mediation Process Via Temporally Attentive Representation Network.
Afteran event has occurred, the representation of both the participating nodes need to be updated tocapture the effect of the observed event based on the principles of: Self-Propagation. , Exogenous
Drive.
and Localized Embedding Propagation.
Two nodes involved in an event form a temporary(communication) or a permanent (association) pathway for the information to propagate from theneighborhood of one node to the other node.
For any event at time t , we update the embeddings forboth nodes involved in the event using a recurrent architecture.
Speciﬁcally, for p -th event of node v ,we evolve z v as:
[Equation]
where, h ustruct ∈ R d is the output representation vectors obtained from aggregator function on nodeu ’s neighborhood and z v ( t p − 1 ) is the recurrent state.
t p is time point of current event, ¯ t p signiﬁesthe timepoint just before current event and t p − 1 represent time point of previous event for node v .
W struct , W rec ∈ R d × d and W t ∈ R d are model parameters that govern the aggregate effect of all the three processes respectively.
The above formulation is ﬂexible in supporting various features.
Forexample, in case of dynamic heterogenous graphs, Eq. 1 can be extended to include edge type withcorresponding parameters.
Similarly, one can support high-dimensional attributes (including spatial)by adding them to Eq.
1 as a feature.
Temporally Attentive Aggregation.
We propose a novel Temporal Point Process based Attentional
Mechanism that empowers the aggregate function (used to compute h struct ) to attend to the neighborsbased on node’s communication and association history.
Let A ( t ) ∈ R n × n be the adjacency matrix for graph G t at time t .
Let S ( t ) ∈
R n × n be a stochasticmatrix capturing the strength between pair of vertices at time t 2 .
One can consider S as a selectionmatrix that induces a natural selection process for a node – it would tend to communicate morewith other nodes that it wants to associate with or has recently associated with.
And it wouldwant to attend less to non-interesting nodes.
Formally, we perform localized attention for a givennode u and compute the coefﬁcients pertaining to the 1-hop neighbors i of node u as: q ui ( t ) =
exp( S ui (¯ t ))
(cid:80)
[Equation]
S
[Equation]
t )) , where q vi signiﬁes the attention weight for the neighbor i at time t and hence itis a temporally evolving quantity.
These attention coefﬁcients are then used to compute the aggregateinformation h ustruct (¯ t ) for node v by employing an attended aggregation mechanism across neighborsas follows: h ustruct (¯ t ) = max (cid:0)(cid:8) σ (cid:0) q ui ( t ) · h i (¯ t ) (cid:1) , ∀ i ∈ N u (¯ t ) (cid:9)(cid:1) , where, h i (¯ t ) =
W h
z i (¯ t ) +
b hand W h ∈ R d × d and b h ∈ R
d
are parameters governing the information propagated by eachneighbor of u .
z i (¯ t ) ∈
R d is the most recent embedding for node i .
Efﬁcient Learning Procedure.
The complete parameter space for the current model is Ω ={ W struct , W rec , W t , W h , b h , { ω k } k =0 , 1 , { ψ k } k =0 , 1 } .
For a set O of P observed events, welearn these parameters by minimizing the negative log likelihood: L = − (cid:80) Pp =1 log ( λ p ( t ))
+(cid:82) T 0 Λ( τ ) dτ , where λ p ( t ) =
λ u p ,v p k p ( t ) represent the intensity of event at time t and Λ( τ ) =(cid:80) nu =1 (cid:80) nv =1 (cid:80) k ∈{ 0 , 1 } λ u,vk ( τ ) represent total survival probability for events that do not happen.
While it is intractable (will require O ( n 2 k ) time) to compute the integral in log-likelihood equationfor all possible non-events in a stochastic setting, we locally optimize L using mini-batch stochasticgradient descent where we estimate the integral using novel sampling technique 3 .
Experiments
We evaluate DyRep and baselines on two real world datasets: Social Evolution Dataset releasedby MIT Human Dynamics Lab —
#nodes: 100, #Initial Associations: 407, #Final Associations:809, #Communications: 2020554 and Clustering Coefﬁcient: 0.548.
Github Dataset availableat Github Archive — #nodes: 12328, #Initial Associations: 70640, #Final Associations: 166565,#Communications: 604954 and Clustering Coefﬁcient: 0.087.
We study the effectiveness of DyRepby evaluating our model on tasks of: (i) Dynamic link prediction – for a given test record ( u, v, t, l, k ) ,we replace v with other entities in the graph and compute the conditional density f u,vk ( t ) =
λ u,vk ( t ) ·exp (cid:16)(cid:82) t ¯ t λ ( s ) ds (cid:17) , where ¯ t is the time of the most recent event on either dimension u or v .
We thenrank all the entities in descending order of the density and report Mean Average Rank and HITS(@10)metric for dynamic link prediction.
(ii) Event Time Prediction – Given a pair of nodes ( u, v ) andevent type k at time t , we use the above density formulation to compute conditional density at time t .
The next time point ˆ t for the event can then be computed as: ˆ t = (cid:82) ∞ t tf u,vk ( t ) dt where the integraldoes not have an analytic form and hence we estimate it using Monte Carlo trick.
For a given testrecord ( u, v, t, l, k ) , we predict the next time this communication event may occur and report MAEagainst the ground truth.
For Dynamic Link Prediction task, we compare the performance of ourmodel against Know-Evolve [22], DynGem [36], DynTrd [20], GraphSage [18] and Node2Vec [2].
For Event Time Prediction, we compare our model against Know-Evolve which has the ability topredict time in a multi-relational dynamic graphs and Multi-dimensional Hawkes Process (MHP)
[37]model where all events in graph are considered as dyadic.
During evaluation, We divide our testsets into n (= 6) slots based on time and report the performance for each time slot, thus providingcomprehensive temporal evaluation of different methods.
[Figure] [Figure]
(unit is hrs).
Figure best viewed in pdf or colored print.
4.1 Experimental Results
Communication Event Prediction Performance (Figure 1 (a-b) ).
For Social Evolution dataset,our method signiﬁcantly and consistently outperforms all the baselines on both metrics.
While theperformance of our method drops a little over time, it is expected due to the temporal recency affecton node’s evolution.
For Github dataset, we demonstrate comparable performance with both KnowEvolve and GraphSage on Rank metric.
It is notable that that overall performance for all methods onrank metric is low.
Github dataset is very sparse with very low clustering coefﬁcient and challengingdataset to learn.
It is expected that for a large number of nodes with no communication history, mostof the methods will show comparable performance but our method outperforms all others when thereis some history available.
This is demonstrated by our signiﬁcantly better performance for HITS@10metric where we are able to do highly accurate prediction for nodes where we learn better history.
Association Event Prediction Performance.
Association events are not available for all time slots so
Figure 1 (c-d) report the aggregate number for this task.
For both the datasets, our model signiﬁcantlyoutperforms the baselines for this task.
Speciﬁcally, our model’s strong performance on HITS@10metric across both datasets demonstrates its robustness in accurate learning from various propertiesof data.
On Social evolution dataset, the number of association events are very small (only 485) andhence our strong performance shows that the model is able to capture the inﬂuence of communicationevents on the association events through the learned representations (mediation).
On the Githubdataset, the network grows through new nodes and our model’s strong performance across both metricdemonstrates its inductive ability to generalize across new nodes across time.
Time Prediction Performance.
Figure 2 demonstrates consistently better performance than state-ofart baseline for event time prediction on both datasets.
While Know-Evolve models both processes astwo different relations between entities, it does not explicitly capture the variance in the time scales oftwo processes.
Further, Know-Evolve does not consider inﬂuence of neighborhood which may leadto capturing weaker temporal-structural dynamics across the graph.
MHP uses speciﬁc parametricintensity function which fails to account for intricate dependencies across graph.
We refer the readersto Appendix B for sample qualitative analysis of our framework.
Conclusion
We introduced a novel modeling framework for dynamic graphs that effectively and efﬁciently learnsnode representations by posing representation learning as latent mediation process bridging dynamicprocesses of topological evolution and node interactions.
We proposed a deep temporal point processmodel parameterized by temporally attentive representation network that models these complex andnonlinearly evolving dynamic processes and learns to encode structural-temporal information overgraph into low dimensional representations.
Our superior evaluation performance demonstrates theeffectiveness of our approach compared to state-of-arts.
Our framework can support wide range ofdynamic graph characteristics which can potentially have many exciting adaptations.
As a part ofour framework, we also propose a novel temporal point process based attention mechanism that canattend over neighborhood based on the history of communications and associations in the graph.
A Overview
DyRep
B Sample Qualitative Performance
[Figure]
Qualitative Performance.
Figure 4 (a-b) shows the tSNE embeddings learned by Dyrep (left) and
GraphSage (right) respectively.
The visualization demonstrates that DyRep embeddings have morediscriminative power as it can effectively capture the distinctive and evolving structural features overtime as aligned with empirical evidence.
Figure 4 (c-d) shows use case of two associated nodes (19and 26) that has less communication at the two time points for above two methods.
DyRep keeps theembeddings nearby although not in same cluster (cos. dist. - 0.649) which demonstrates its abilityto learn the association and less communication dynamics between two nodes.
For GraphSage theembeddings are on opposite ends of cluster with (cos.
dist. - 1.964) 4 .
C Contributions and More Related Work
Recent availability of dynamic graphs have created a conspicuous need for principled approaches toadvance graph embedding techniques for dynamic graphs [8].
We focus on two pertinent questionsfundamental to representation learning over dynamic graphs:
(i) What can serve as an elegant model for dynamic processes over graphs?
— A key modeling
choice in existing representation learning techniques for dynamic graphs [36, 20, 22, 23, 38] assume
that graph dynamics evolve as a single time scale process.
In contrast to these approaches, we
observe that most real-world graphs exhibit at least two distinct dynamic processes that evolve at
[Table]
different time scales — Topological Evolution : where the number of nodes and edges are expectedto grow (or shrink) over time leading to structural changes in the graph; and Node Interactions :which relates to activities between nodes that may or may not be structurally connected.
Modelinginterleaved dependencies between these non-linearly evolving dynamic processes is a crucial nextstep for advancing the formal models of dynamic graphs.(ii) How can one leverage such a model to learn dynamic node representations that are effectivelyable to capture evolving graph information over time?
— Existing techniques in this direction canbe divided into two approaches: a.) Discrete-Time Approach, where the evolution of a dynamicgraph is observed as collection of static graph snapshots over time [19, 36, 20].
These approachestend to preserve (encode) very limited structural information and capture temporal information ata very coarse level which leads to loss of information between snapshots and lack of ability tocapture ﬁne-grained temporal dynamics.
Another challenge in such approaches is the selection ofappropriate aggregation granularity which is often misspeciﬁed.
b.)
Continuous-Time Approach,where evolution is modeled at ﬁner time granularity in order to address the above challenges.
Whileexisting approaches have demonstrated to be very effective in speciﬁc settings, they either modelsimple structural and complex temporal properties in a decoupled fashion [22] or use simple temporalmodels (exponential family in [23]).
But several domains exhibit highly nonlinear evolution ofstructural properties coupled with complex temporal dynamics and it remains an open problem toeffectively model and learn informative representations capturing various dynamical properties ofsuch complex systems.
Contributions.
First, our work expresses dynamic graphs at multiple scales as follows: a.)
Dynamic”of” the Network: This corresponds to the topological changes in network – insertion or deletionof nodes and edges b.)
Dynamic ”on” the Network: This corresponds to various activities in thenetwork – self evolution of node’s interests/features, change in node’s features due to exogenousdrive (activities external to net-work), information propagation within network and within-networkinteractions between nodes which may or may not have direct edge between them.
This dichotomy ofdynamic network processes is well-known and has been subject of several studies [9, 10, 11, 39]in segregated manner.
But none of the existing machine learning approaches has jointly modeledthem for representation learning over dynamic graphs (our key objective) to the best of our knowledge.307 Next, we propose a novel representation learning framework for dynamic graphs, to modelinterleaved evolution of two observed processes through latent mediation process expressed aboveand effectively learn richer node representations over time.
Our framework ingests dynamic graphinformation in the form of association and communication events over time and updates the noderepresentations as they appear in these events.
We build a two-time scale deep temporal pointprocess approach to capture the continuous-time ﬁne-grained temporal dynamics of the two observedprocesses.
We further parameterize the conditional intensity function of the temporal point processwith a deep inductive representation network that learns functions to compute node representations.
Finally, we couple the structural and temporal components of our framework by designing a novel
Temporal Attention Mechanism , which induces temporal attentiveness over neighborhood nodesusing the learned conditional intensity function.
This allows to capture highly interleaved andnonlinear dynamics governing node representations over time.
We design an efﬁcient unsupervisedtraining procedure for end-to-end training of our framework.
Table 1 further provides qualitative comparison between state-of-arts and our framework.