Learning Dynamic Graph Representations
Abstract
We address two fundamental questions that arise in learning over dynamic graphs: (i) How to elegantly model dynamical processes over graphs?
(ii) How to leverage such a model to effectively encode evolving graph information into lowdimensional representations?
We present DyRep - a novel modeling framework for dynamic graphs that posits representation learning as a latent mediation process bridging two observed processes – dynamic of the network (topological evolution) and dynamic on the network (activities of the nodes).
To this end, we propose an inductive framework comprising of two-time scale deep temporal point process model parameterized by a temporal-attentive representation network and trained end-to-end using an efﬁcient unsupervised procedure.
We demonstrate that DyRep signiﬁcantly outperforms state-of-art baselines for dynamic link prediction and event time prediction and provide qualitative analysis of our framework.
Introduction
Representation learning over graph structured data has ubiquitous applicability in variety of domains such as social networks, bioinformatics, natural language processing, and relational knowledge bases.
Learning node representations to effectively encode high-dimensional and non-Euclidean graph information is a challenging problem but recent advances in deep learning has helped important progress towards addressing it [1, 2, 3, 4, 5, 6, 7], with majority of the approaches focusing on advancing the state-of-art in static graph setting.
However, several domains (e g. social network communications, ﬁnancial transaction graphs or longitudinal citation data) now present highly dynamic data that exhibit complex temporal properties in addition to earlier cited challenges.
These recent developments have created a conspicuous need for principled approaches to advance graph embedding techniques for dynamic graphs [8].
We focus on two pertinent questions fundamental to representation learning over dynamic graphs: (i) What can serve as an elegant model for dynamic processes over graphs?
(ii) How can one leverage such a model to learn dynamic node representations that are effectively able to capture evolving graph information over time?
As noted in [9], an important requirement to effectively learn over such dynamical systems is the ability to express the dynamical processes at different scales.
We propose that any dynamic graph must be minimally expressed as a result of two fundamental processes evolving at different time scales: Association Process (dynamics of the network), that brings change in the graph structure and leads to long lasting information exchange between nodes; and Communication Process (dynamics on the network), that relates to node’s self evolution and activities between (not necessarily connected) nodes which leads to temporary information ﬂow between them [10, 11].
We, then, posit our goal of learning node representations as modeling a latent mediation process that bridges the above two observed processes such that learned embeddings drive the complex temporal dynamics of both processes and these processes subsequently lead to the nonlinear evolution of node representations.
Further, the information propagated across the graph is governed by the temporal dynamics of communication and association histories of nodes with its neighborhood (Figure 3 Appendix A ).
Preliminaries
Dynamic Graph Setting — Let G t =
( V t , E t ) denote graph G at time t , where V t is the set of nodes and E t is the set of edges in G t .
An event at time t is represented as e = ( u, v, t, k ) , where u, v are the two nodes involved in an event.
k ∈ { 0 , 1 } and we use k = 0 to signify events from the topological evolution process (association) and k = 1 to signify events from node interaction process (communication).
We then represent complete set of P observed events ordered by time in window [0 , T ] as O = { ( u, v, t, k ) p } Pp =1 .
Here, t p ∈ R + , 0 ≤ t p ≤ T .
Let z v ( t )
∈ R d represent d -dimensional representation of node v at time t .
We use z v (¯ t ) for most recently updated embedding of node v just before t .
Proposed Method
The rationale behind our framework is that the observed set of events are the realizations of the nonlinear dynamic processes governing the changes in topological structure of graph and interactions between the nodes in the graph.
Now, when an event is observed between two nodes, information ﬂows from the neighborhood of one node to the other and affects the representations of the nodes accordingly.
While a communication event (interaction) only propagates local information across two nodes, an association event changes the topology and thereby has more global effect.
The goal is to learn node representations that encode information evolving due to such local and global effects.
Modeling Two-Time Scale Observed Graph Dynamics.
Given an observed event p =
( u, v, t, k ) , we deﬁne a continuous-time deep model of temporal point process using the intensity function λ u,vk ( t ) that models the occurrence of event p between nodes u and v at time t : λ u,vk ( t ) =
f k ( g k (¯ t )) .
The inner function g k (¯ t ) computes the compatibility of the most recently updated representations of two nodes, z u (¯ t ) and z v (¯ t ) as follows:
g k (¯ t ) =
ω
Tk · [ z u (¯ t ); z v (¯ t )] .
Here, [;] signiﬁes concatenation and ω k ∈ R 2 d serves as the model parameter that learns time-scale speciﬁc compatibility.
The choice of outer function f k needs to: 1.) keep intensity positive.
account for difference in time-scale of dynamics corresponding to communication and association processes.
To this end, we use a modiﬁed version of softplus function parameterized by a dynamics parameter ψ k to capture timescale dependence: f k ( x ) =
ψ k log(1 + exp( x/ψ k )) where, ψ k is scalar time-scale parameter learned as part of training.
ψ k corresponds to the rate of events arising from a corresponding process.
Learning latent Mediation Process via Temporally Attentive Representation Network.
After an event has occurred, the representation of both the participating nodes need to be updated to capture the effect of the observed event based on the principles of: Self-Propagation. , Exogenous
Drive.
and Localized Embedding Propagation.
Two nodes involved in an event form a temporary (communication) or a permanent (association) pathway for the information to propagate from the neighborhood of one node to the other node.
For any event at time t , we update the embeddings for both nodes involved in the event using a recurrent architecture.
Speciﬁcally, for p -th event of node v , we evolve z v as:
[Equation]
where, h ustruct ∈ R d is the output representation vectors obtained from aggregator function on node u ’s neighborhood and z v ( ¯ t vp ) is the recurrent state.
t p is time point of current event, ¯ t p signiﬁes the timepoint just before current event and ¯ t v p represent time point of previous event for node v .
W struct , W rec ∈ R d × d and W t ∈ R d are model parameters that govern the aggregate effect of all the three processes respectively.
The above formulation is ﬂexible in supporting various features.
For example, in case of dynamic heterogenous graphs, Eq. 1 can be extended to include edge type with corresponding parameters.
Similarly, one can support high-dimensional attributes (including spatial) by adding them to Eq.
1 as feature inputs to the model.
Temporally Attentive Aggregation.
We propose a novel Temporal Attention Mechanism to compute h struct by attending to the neighbors based on node’s communication and association history.
Let A ( t ) ∈ R n × n be the adjacency matrix for graph G t at time t .
Let S ( t ) ∈
R n × n be a stochastic matrix capturing the strength between pair of vertices at time t .
One can consider S as a selection matrix that induces a natural selection process for a node – it would tend to communicate more with other nodes that it wants to associate with or has recently associated with.
And it would want to attend less to non-interesting nodes.
Formally, we perform localized attention for a given node u and compute the coefﬁcients pertaining to the 1-hop neighbors i of node u as: q ui ( t ) =
exp( S ui [Equation]
[Equation]
i
[Equation]
u ( t ) exp( S ui (cid:48) (¯ t )) , where q ui signiﬁes the attention weight for the neighbor i at time t and hence it is a temporally evolving quantity.
These attention coefﬁcients are used to compute the aggregate information h ustruct (¯ t ) by employing an attended aggregation mechanism across neighbors as follows: h ustruct (¯ t ) = max (cid:0)(cid:8) σ (cid:0) q ui ( t ) · h i (¯ t ) (cid:1) , ∀ i ∈ N u (¯ t ) (cid:9)(cid:1) , where, h i (¯ t ) =
W h
z i (¯ t )
+ b h and W h ∈ R d × d and b h ∈ R
d
are parameters governing the information propagated by each neighbor of u .
z i (¯ t ) ∈
R d is the most recent embedding for node i .
Spatial Properties.
Our work considers spatial properties implicitly.
Concretely, the distance between the nodes of graph in embedded space effects the pattern of connections and communications between the nodes.
Simultaneously, the dynamics "of" and "on" the network affect the node distances in embedded space. If one wants to introduce explicit spatial features (e.g. location in transportation or mobile communication network), our model will support it by adding such features as part of Eq. Another approach could be to combine the temporal attentive operator with a local spatial operator (e.g. geometric deep learning) to build a sophisticated spatiotemporal operator. Efﬁcient Learning Procedure.
The complete parameter space for the current model is Ω = { W struct , W rec , W t , W h , b h , { ω k } k =0 , 1 , { ψ k } k =0 , 1 } .
For a set O of P observed events, we learn these parameters by minimizing the negative log likelihood: L = − (cid:80) Pp =1 log ( λ p ( t ))
+ (cid:82) T 0 Λ( τ ) dτ , where λ p ( t ) =
λ u p ,v p k p ( t ) represent the intensity of event at time t and Λ( τ ) = (cid:80) nu =1 (cid:80) nv =1 (cid:80) k ∈{ 0 , 1 } λ u,vk ( τ ) represent total survival probability for events that do not happen.
While it is intractable (will require O ( n 2 k ) time) to compute the integral in log-likelihood equation for all possible non-events in a stochastic setting, we locally optimize L using mini-batch stochastic gradient descent where we estimate the integral using novel sampling technique.
Experiments
We evaluate DyRep and baselines on two real world datasets: Social Evolution Dataset released by MIT Human Dynamics Lab — #nodes: 100, #Initial Associations: 407, #Final Associations: 809, #Communications: 2020554 and Clustering Coefﬁcient: 0.548.
Github Dataset available at Github Archive — #nodes: 12328, #Initial Associations: 70640, #Final Associations: 166565, #Communications: 604954 and Clustering Coefﬁcient: 0.087.
We study the effectiveness of DyRep by evaluating our model on tasks of: (i) Dynamic link prediction – for a given test record ( u, v, t, k ) , replace v with other entities in the graph and compute the conditional density f u,v k ( t ) =
λ u,v k ( t ) · exp (cid:16)(cid:82) t ¯ t λ ( s ) ds (cid:17) , where ¯ t is the time of the most recent event on either dimension u or v .
We then rank all the entities in descending order of the density and report Mean Average Rank and HITS(@10) metric for dynamic link prediction.
(ii) Event Time Prediction – Given a pair of nodes ( u, v ) and event type k at time t , we use the above density formulation to compute conditional density at time t .
The next time point ˆ t for the event can then be computed as: ˆ t = (cid:82) ∞ t tf u,vk ( t ) dt .
For a given test record, we predict the next time an event may occur and report MAE against the ground truth.
For Dynamic Link Prediction task, we compare the performance of our model against Know-Evolve [12], DynGem [13], DynTrd [14], GraphSage [15] and Node2Vec [2].
For Event Time Prediction, we compare our model against Know-Evolve which has the ability to predict time and Multi-dimensional
Hawkes Process (MHP) [16] model where all events in graph are considered as dyadic.
During evaluation, we divide our test sets into n (= 6) slots based on time and report the performance for each time slot, thus providing comprehensive temporal evaluation of different methods.
4.1 Experimental Results
Communication Event Prediction Performance (Figure 1 (a-b) ).
For Social Evolution dataset, our method signiﬁcantly and consistently outperforms all the baselines on both metrics. While the [Figure] [Figure]
(unit is hrs).
Figure best viewed in pdf or colored print.
performance of our method drops a little over time, it is expected due to the temporal recency affect on node’s evolution.
For Github dataset, we demonstrate comparable performance with both KnowEvolve and GraphSage on Rank metric.
It is notable that that overall performance for all methods on rank metric is low.
Github dataset is very sparse with very low clustering coefﬁcient and challenging dataset to learn.
It is expected that for a large number of nodes with no communication history, most of the methods will show comparable performance but our method outperforms all others when there is some history available.
This is demonstrated by our signiﬁcantly better performance for HITS@ metric where we are able to do highly accurate prediction for nodes where we learn better history.
Association Event Prediction Performance.
Association events are not available for all time slots so
Figure 1 (c-d) report the aggregate number for this task.
For both the datasets, our model signiﬁcantly outperforms the baselines for this task.
Speciﬁcally, our model’s strong performance on HITS@ metric across both datasets demonstrates its robustness in accurate learning from various properties of data.
On Social evolution dataset, the number of association events are very small (only 485) and hence our strong performance shows that the model is able to capture the inﬂuence of communication events on the association events through the learned representations (mediation).
On the Github dataset, the network grows through new nodes and our model’s strong performance across both metric demonstrates its inductive ability to generalize across new nodes across time.
Time Prediction Performance.
Figure 2 demonstrates consistently better performance than state-ofart baseline for event time prediction on both datasets.
While Know-Evolve models both processes as two different relations between entities, it does not explicitly capture the variance in the time scales of two processes.
Further, Know-Evolve does not consider inﬂuence of neighborhood which may lead to capturing weaker temporal-structural dynamics across the graph.
MHP uses speciﬁc parametric intensity function which fails to account for intricate dependencies across graph.
We refer the readers to Appendix B for sample qualitative analysis of our framework.
Conclusion
We introduced a novel modeling framework for dynamic graphs that effectively and efﬁciently learns node representations by posing representation learning as latent mediation process bridging dynamic processes of topological evolution and node interactions.
We proposed a deep temporal point process model parameterized by temporally attentive representation network that models these complex and nonlinearly evolving dynamic processes and learns to encode structural-temporal information over graph into low dimensional representations.
Our superior evaluation performance demonstrates the effectiveness of our approach compared to state-of-arts.
Our framework can support wide range of dynamic graph characteristics which can potentially have many exciting adaptations.
As a part of our framework, we also propose a novel temporal point process based attention mechanism that can attend over neighborhood based on the history of communications and associations in the graph.
Appendix A Overview of DyRep Framework
B Sample Qualitative Performance
[Figure]
Qualitative Performance.
Figure 4 (a-b) shows the tSNE embeddings learned by Dyrep (left) and
GraphSage (right) respectively.
The visualization demonstrates that DyRep embeddings have more discriminative power as it can effectively capture the distinctive and evolving structural features over time as aligned with empirical evidence.
Figure 4 (c-d) shows use case of two associated nodes ( and 26) that has less communication at the two time points for above two methods.
DyRep keeps the embeddings nearby although not in same cluster (cos. dist. - 0.649) which demonstrates its ability to learn the association and less communication dynamics between two nodes.
For GraphSage the embeddings are on opposite ends of cluster with (cos.
dist. - 1.964) [Figure]
C Contributions and Related Work
Recent availability of dynamic graphs have created a conspicuous need for principled approaches to advance graph embedding techniques for dynamic graphs [8].
We focus on two pertinent questions fundamental to representation learning over dynamic graphs:
(i) What can serve as an elegant model for dynamic processes over graphs?
— A key
modeling choice in existing representation learning techniques for dynamic graphs [13, 14, 12, 17, 18] assume that graph dynamics evolve as a single time scale process.
In contrast to these approaches, we observe that most real-world graphs exhibit at least two distinct dynamic processes that evolve at different time scales — Topological Evolution : where the number of nodes and edges are expected to grow (or shrink) over time leading to structural changes in the graph; and Node Interactions : which relates to activities between nodes that may or may not be structurally connected.
Modeling interleaved dependencies between these non-linearly evolving dynamic processes is a crucial next step for advancing the formal models of dynamic graphs. (ii) How can one leverage such a model to learn dynamic node representations that are effectively able to capture evolving graph information over time?
— Existing techniques in this direction can be divided into two approaches: a.) Discrete-Time Approach, where the evolution of a dynamic graph is observed as collection of static graph snapshots over time [19, 13, 14].
These approaches tend to preserve (encode) very limited structural information and capture temporal information at a very coarse level which leads to loss of information between snapshots and lack of ability to capture ﬁne-grained temporal dynamics.
Another challenge in such approaches is the selection of appropriate aggregation granularity which is often misspeciﬁed.
b.)
Continuous-Time Approach, where evolution is modeled at ﬁner time granularity in order to address the above challenges.
While existing approaches have demonstrated to be very effective in speciﬁc settings, they either model simple structural and complex temporal properties in a decoupled fashion [12] or use simple temporal models (exponential family in [17]).
But several domains exhibit highly nonlinear evolution of structural properties coupled with complex temporal dynamics and it remains an open problem to effectively model and learn informative representations capturing various dynamical properties of such complex systems.
Contributions — First, our work expresses dynamic graphs at multiple scales as follows: a.)
Dynamic ”of” the Network: This corresponds to the topological changes in network – insertion or deletion of nodes and edges b.)
Dynamic ”on” the Network: This corresponds to various activities in the network – self evolution of node’s interests/features, change in node’s features due to exogenous drive (activities external to net-work), information propagation within network and within-network interactions between nodes which may or may not have direct edge between them.
This dichotomy of dynamic network processes is well-known and has been subject of several studies [9, 10, 11, 20] in segregated manner.
But none of the existing machine learning approaches has jointly modeled them for representation learning over dynamic graphs (our key objective) to the best of our knowledge. Next, we propose a novel representation learning framework for dynamic graphs, to model interleaved evolution of two observed processes through latent mediation process expressed above and effectively learn richer node representations over time.
Our framework ingests dynamic graph information in the form of association and communication events over time and updates the node representations as they appear in these events.
We build a two-time scale deep temporal point process approach to capture the continuous-time ﬁne-grained temporal dynamics of the two observed processes.
We further parameterize the conditional intensity function of the temporal point process with a deep inductive representation network that learns functions to compute node representations.
Finally, we couple the structural and temporal components of our framework by designing a novel
Temporal Attention Mechanism , which induces temporal attentiveness over neighborhood nodes using the learned conditional intensity function.
This allows to capture highly interleaved and nonlinear dynamics governing node representations over time.
We design an efﬁcient unsupervised training procedure for end-to-end training of our framework.
Related Work — Representation Learning approaches for static graphs either perform node embedding [1, 2, 3, 4, 5, 6, 7] or sub-graph embedding [21, 22, 23] which can also utilize convolutional neural networks [24, 25, 26, 15].
Dynamic network embedding is pursued through various techniques such as matrix factorization [19], structural properties [14], CNN-based approaches [27], deep recurrent models [12], and random walks [17].
Literature on temporal modeling of dynamic networks [28], that focus on link prediction tasks is orthogonal to our work as they do not focus on representation learning. Authors in [29, 30] proposed models of learning dynamic embeddings but none of them consider time at ﬁner level and do not capture both topological evolution and interactions simultaneously. Temporal Point Processes [31] have gained lot of attention recently for modeling dynamic systems [32, 33, 34, 35, 36]. In parallel, research on deep point process models include parametric approaches to learn intensity [37, 38] using RNN and GAN based approaches to learn intensity functions [39].