R ECURRENT E VENT N ETWORK : G LOBAL S TRUCTURE I NFERENCE OVER T EMPORAL K NOWLEDGE G RAPH
Anonymous authors
Paper under double-blind review
A BSTRACT
Modeling dynamically-evolving, multi-relational graph data has received a surge of interests with the rapid growth of heterogeneous event data.
However, predicting future events on such data requires global structure inference over time and the ability to integrate temporal and structural information, which are not yet well understood.
We present
Recurrent Event Network (RE-N ET ) a novel autoregressive architecture for modeling temporal sequences of multi relational graphs ( e.g tem poral knowledge graph) which can perform sequential global structure inference over future time stamps to predict new events
RE-N ET employs a recurrent event encoder to model the temporally conditioned joint probability distribution for the event sequences, and equips the event encoder with a neighborhood aggregator for modeling the concurrent events within a time window associated with each entity
We apply teacher forcing for model training over historical data, and infer graph sequences over future time stamps by sampling from the learned joint distribution in a sequential manner.
We evaluate the proposed method via temporal link prediction on ﬁve public datasets
Extensive experiments 1 demonstrate the strength of
RE-N ET , especially on multi-step inference over future time stamps.
I NTRODUCTION
Representation learning on dynamically-evolving, graph-structured data has emerged as an important problem in a wide range of applications, including social network analysis (Zhou et al., 2018a;
Trivedi et al., 2019), knowledge graph reasoning (Trivedi et al., 2017; Nguyen et al., 2018;
Kazemi et al., 2019), event forecasting (Du et al., 2016), and recommender systems (Kumar et al., 2019; You et al., 2019).
Previous methods over dynamic graphs mainly focus on learning time-sensitive structure representations for node classiﬁcation and link prediction in single-relational graphs.
However, the rapid growth of heterogeneous event data (Mahdisoltani et al., 2014; Boschee et al., 2015) has created new challenges on modeling temporal, complex interactions between entities ( i.e. , viewed as a temporal knowledge graph or a TKG), and calls for approaches that can predict new events in different future time stamps based on the history— i.e. , structure inference of a TKG over time.
Recent attempts on learning over temporal knowledge graphs have focused on either predicting missing events (facts) for the observed time stamps (García-Durán et al., 2018; Dasgupta et al., 2018;
Leblay & Chekol, 2018), or estimating the conditional probability of observing a future event using temporal point process (Trivedi et al., 2017; 2019).
However, the former group of methods adopts an interpolation problem formulation over TKGs and thus cannot predict future events, as representations of unseen time stamps are unavailable.
The latter group of methods, including Know-Evolve and its extension, DyRep, computes the probability of future events using ground-truths of the proceeding events during inference time, and cannot model concurrent events occurring within the same time window—which often happens when event time stamps are discrete.
It is thus desirable to have a principled method that can infer graph structure sequentially over time and can incorporate local structural information ( e.g. , concurrent events) during temporal modeling.
To this end, we propose a sequential structure inference architecture, called Recurrent Event Network (RE-N ET ), for modeling heterogeneous event data in the form of temporal knowledge graphs.
Key ideas of RE-N ET are based on the following observations: (1) predicting future events can be viewed as a sequential (multi-step) inference of multi-relational interactions between entities over time; (2)
Grant diplomatic recognition at 1/1/
M a k e s t a t e m e n t a t 3 / 8 /
C on s u l t a t 4 / 10 / 18 M a k e a r eque s t a t 4 / 22 /
C r i t i c
i z e a t 5 / 5 /
V i s i t a t 5 / 6 /
M a k e s t a t e m en t a t 6 / 13 /(a)
An example of temporal KG ? ? ?
𝐺 "#$ 𝐺 "#% 𝐺 "#& 𝐺 "
Aggregate 𝑃(𝐺 " |𝐺 "#$:"#& )
Decoder Encoder
Local
Global (
[Figure] [Figure]
of the
[Figure]
architecture temporally adjacent events may carry related semantics and informative patterns, which can further help inform future events ( i.
, temporal information); and (3) multiple events may co-occur within the same time window and exhibit structural dependencies as they share entities ( i.e. , local structural information).
To incorporate these ideas, RE-N ET deﬁnes the joint probability distribution of all the events in a TKG in an autoregressive fashion, where it models the probability distribution of the concurrent events at the current time step conditioned on all the preceding events (see Fig.
1b for an illustration).
Speciﬁcally, a recurrent event encoder, parametrized by RNNs, is used to summarize information of the past event sequences, and a neighborhood aggregator is employed to aggregate the information of concurrent events for the related entity within each time stamp.
With the summarized information of the past event sequences, our decoder deﬁnes the joint probability of a current event.
Such an autoregressive model can be effectively trained by using teacher forcing.
Global structure inference for predicting future events can be achieved by performing sampling in a sequential manner.
We evaluate our proposed method on temporal link prediction task, by testing the performance of multi-step inference over time on ﬁve public temporal knowledge graph datasets.
Experimental results demonstrate that RE-N ET outperforms state-of-the-art models of both static and temporal graph reasoning, showing its better capacity to model temporal, multi-relational graph data with concurrent events.
We further show that RE-N ET can perform effective multi-step inference to predict unseen entity relationships in a distant future.
R ELATED W ORK
Our work is related to previous studies on temporal knowledge graph reasoning, temporal modeling on homogeneous graphs, recurrent graph neural networks, and deep autoregressive models.
Temporal KG Reasoning.
There are some recent attempts on incorporating temporal information in modeling dynamic knowledge graphs.
(Trivedi et al., 2017) presented Know-Evolve which models the occurrence of a fact as a temporal point process.
However, this method is built on a problematic formulation when dealing with concurrent events, as shown in Section F. Several embedding-based methods have been proposed (García-Durán et al., 2018; Leblay & Chekol, 2018;
Dasgupta et al., 2018) to model time information.
They embed the associate into a low dimensional space such as relation embeddings with RNN on the text of time (García-Durán et al., 2018), time embeddings (Leblay & Chekol, 2018), and temporal hyperplanes (Leblay & Chekol, 2018).
However, these models do not capture temporal dependency and cannot generalize to unobserved time stamps.
Temporal Modeling on Homogeneous Graphs.
There are attempts on predicting future links on homogeneous graphs (Pareja et al., 2019;
Goyal et al., 2018; 2019; Zhou et al., 2018b; Singer et al., 2019).
Some of the methods try to incorporate and learn graphical structures to predict future links (Pareja et al., 2019;
Zhou et al., 2018b; Singer et al., 2019), while other methods predict by reconstructing an adjacency matrix by using an autoencoder (Goyal et al., 2018; 2019).
These methods seek to predict on single-relational graphs, and are designed to predict future edges in one future step (i.e., for t + 1 ).
However, our work focuses on “multi-relational” knowledge graphs and aims for multi-step prediction (i.e., for t + 1 , ..., t + k ).
Recurrent Graph Neural Models.
There have been some studies on recurrent graph neural models for sequential or temporal graph-structured data (Sanchez-Gonzalez et al., 2018; Battaglia et al., 2018; Palm et al., 2018; Seo et al., 2017; Pareja et al., 2019).
These methods adopt a messagepassing framework for aggregating nodes’ neighborhood information ( e.g. , via graph convolutional operations).
GN (Sanchez-Gonzalez et al., 2018; Battaglia et al., 2018) and RRN (Palm et al., 2018) update node representations by a message-passing scheme between time stamps.
Some prior methods adopt an RNN to memorize and update the states of node embeddings that are dynamically evolving (Seo et al., 2017), or memorize and update the model parameters for different time stamps (Pareja et al., 2019).
In contrast, our proposed method, RE-N ET , aims to leverage autoregressive modeling to parameterize the joint probability distributions of events with RNNs.
Deep Autoregressive Models.
Deep autoregressive models deﬁne joint probability distributions as a product of conditionals.
DeepGMG (Li et al., 2018) and GraphRNN (You et al., 2018) are deep generative models of graphs and focus on generating homogeneous graphs where there is only a single type of edge.
In contrast to these studies, our work focuses on generating heterogeneous graphs, in which multiple types of edges exist, and thus our problem is more challenging.
To the best of my knowledge, this is the ﬁrst paper to formulate the structure inference (prediction) problem for temporal, multi-relational (knowledge) graphs in an autoregressive fashion.
P ROPOSED M ETHOD : RE-N ET
We consider a temporal knowledge graph (TKG) as a multi-relational, directed graph with timestamped edges (relationships) between nodes (entities).
An event is deﬁned as a time-stamped edge, i.e. , (subject entity, relation, object entity, time) and is denoted by a quadruple (s , r , o , t ) or (s t , r t , o t ) .
We denote a set of events at time t as G t .
A TKG is built upon a sequence of event quadruples ordered ascending based on their time stamps, i.e. , { G t } t = { (s i , r i , o i , t i ) }
i (with t i <
t j , ∀ i < j ), where each time-stamped edge has a direction pointing from the subject entity to the object entity.
The goal of learning generative models of events is to learn a distribution P ( G ) over temporal knowledge graphs, based on a set of observed event sets { G 1 , ..., G T } .
To model lasting events which span over a time range, i.e. , (s , r , o , [ t 1 , t 2 ]) , we simply partition such event into a sequence of time-stamp events { G t 1 , ..., G t 2 } .
We leave more sophisticated modeling of lasting events as future work.
3. R ECURRENT E VENT N ETWORK
Sequential Structure Inference in TKG.
The key idea in RE-N ET is to deﬁne the joint distribution of all the events G = { G 1 , ..., G T } in an autoregressive manner, i.e., P ( G ) =
(cid:81)
Tt =1 P ( G t | G t − m : t − 1 ) .
Basically, we decompose the joint distribution into a sequence of conditional distributions (e.g., P ( G t | G t − m : t − 1 ) ), where we assume the probability of the events at a time step, e.g. G t , only depends on the events at the previous m steps, e.g., G t − m : t − 1 .
For each conditional distribution P ( G t | G t − m : t − 1 ) , we further assume that the events in G t are mutually independent given the previous events G t − m : t − 1 .
In this way, the joint distribution can be rewritten as follows.
[Equation]
Intuitively, the generation process of each triplet (s t , r t , o t ) is deﬁned as below.
Given all the past events G t − m : t − 1 , we ﬁst generate a subject entity s t through the distribution P (s t | G t − m : t − 1 ) .
Then we further generate a relation r t with P (r t | s t , G t − m : t − 1 ) , and ﬁnally the object entity o t is generated by deﬁning P (o t | s t , r t , G t − m : t − 1 ) .
In this work
[Equation]
we assume that P (o t | s t , r t , G t − m : t − 1 ) and P (r t | s t , G t − m : t − 1 ) depend only on events that are related to s , and focus on modeling the following joint probability:
[Equation]
[Figure]
𝑁 𝑠 $%& 𝑠
Graph
[Figure]
𝑁 𝑠 $%’ 𝑠
Graph from 𝑁 𝑠 $
ReLu 𝑔(𝑁 𝑠 $ ) 𝑠
RGCN Aggregation 1-hop aggregator 2-hop aggregator 𝑟 ’ 𝑟 & 𝑟 , 𝑟 ’ 𝑟 & 𝑟 , 𝑟 ’ 𝑟 & 𝑟 , 𝑔(𝑁 𝑠 $%’ )
𝑔(𝑁 𝑠 $%& ) where G t becomes N ( s ) t which is a set of neighboring entities interacted with subject entity s under all relations at time stamp t .
For the third probability, the event sets should be considered since subject is not given.
Next, we introduce how we parameterize these distributions.
Recurrent Event Encoder.
RE-N ET parameterizes P (o t | s , r , G t − m : t − 1 ) in the following way:
[Equation]
where e s , e r ∈ R
d are learnable embedding vectors speciﬁed for subject entity s and relation r .
h t − 1 (s , r) ∈
R
d is a history vector which encodes the information from the neighbor sets interacted with s in the past, as well as the global information from graph structures of G t − 1: t − m .
Basically, [ e s : e r : h t − 1 (s , r)] is an encoding to summarize all the past information.
Based on that, we further compute the probability of different object entities o t by passing the encoding into a linear softmax classiﬁer parameterized by { w o t } .
Similarly, we deﬁne the probabilities for relations and subjects as follows:
[Equation]
) where h t − 1 (s) captures all the local information about s in the past, and H t − 1 ∈ R d is a vector representation to encode the global graph structures G t − 1: t − m .
For each time step t , since the hidden vectors h t − 1 (s) , h t − 1 (s , r) and H t − 1 preserve the information from the past events, and we update them in the following recurrent way:
[Equation]
where g is an aggregation function, and N (s) t stands for all the events related to s at the current time step t .
Intuitively, we obtain the current information related to s by aggregating all the related events at time t , i.e., g ( N (s) t ) .
Then we update the hidden vector h t (s , r) by using the aggregated information g ( N (s) t ) at the current step, the past value h t − 1 (s , r) and also the global hidden vector
H t .
The hidden vector h t (s) is updated in a similar way.
For the aggregation of all events g ( G t ) , we deﬁne g ( G t ) =
max( { g ( N ( s ) t ) } s ) , which is from the element-wise max-pooling operation over all g ( N (s) t ) .
We use Gated Recurrent Units Cho et al.
(2014) as RNN.
Details are described in Section A.
For each subject entity s , it can interact with multiple relations and object entities at each time step t .
In other words, the set N (s) t can contain multiple events.
Designing effective aggregation functions g to aggregate information from N (s) t for s is therefore a nontrivial problem.
Next, we introduce how we design g ( · ) in RE-N ET .
3. M ULTI - RELATIONAL G RAPH (RGCN) A GGREGATOR
Here we discuss the aggregate function g ( · ) , which capture different kinds of neighborhood information for each subject entity and relation, i.e. , ( s , r ).
We ﬁrst introduce two simple aggregation functions, i.e. , mean pooling aggregator and attentive pooling aggregator.
These two simple aggregators only collect neighboring entities under the same relation r .
Then we introduce a more powerful aggregation function, i.e. , multi-relational aggregator.
Mean Pooling Aggregator.
The baseline aggregator simply takes the element-wise mean of the vectors in { e o : o ∈ N (s , r) t } , where N (s , r) t is a set of objects interacted with s under r at t .
But the mean aggregator treats all neighboring objects equally, and thus ignores the different importance of each neighbor entity.
Attentive Pooling Aggregator.
We deﬁne an attentive aggregator based on the additive attention introduced in (Bahdanau et al., 2015) to distinguish the important entities for (s , r) .
The aggregate function is deﬁned as g ( N (s , r) t
[Equation]
α
[Equation]
e o , where α o = softmax ( v (cid:62) tanh( W ( e s ; e r ; e o ))) .
v ∈
R d and W ∈ R d × 3 d are trainable weight matrices.
By adding the attention function of the subject and the relation, the weight can determine how relevant each object entity is to the subject and relation.
Multi-Relational Aggregator.
Here, we introduce a multi-relational graph aggregator based on (Schlichtkrull et al., 2018).
This is a general aggregator that can incorporate information from multi-relational neighbors and multi-hop neighbors.
Formally, the aggregator is deﬁned as follows:
[Equation]
where initial hidden representations for each node ( h (0) o ) are set to trainable embedding vectors ( e o ) for each node.
Basically, each relation can derive a local graph structure between entities, which further yield a message on each entity by aggregating the information from the neighbors of that entity, i.e., (cid:80) o ∈ N (s , r) t
1 c s W ( l ) r h ( l ) o .
The overall message on each entity is further computed by aggregating all the relation-speciﬁc messages, i.e., (cid:80) r ∈ R (cid:80) o ∈ N (s , r) t
1 c s W ( l ) r h ( l ) o .
Finally, the aggregator g ( N (s) t ) is deﬁned by combining both the overall message and the information from past steps,
i.e., W ( l ) 0 h
( l ) s .
To distinguish between different relations, we introduce independent weight matrices { W ( l ) r } for each relation r .
Furthermore, the aggregator collects representations of multi-hop neighbors by introducing multiple layers of the neural network, with each layer indexed by l .
The number of layers determines the depth to which the node reaches to aggregate information from its local neighborhood.
We depict this aggregator in Fig.
The major issue of this aggregator is that the number of parameters grows rapidly with the number of relations.
In practice, this can easily lead to overﬁtting on rare relations and models of very large size.
Thus, we adopt the block-diagonal decomposition (Schlichtkrull et al., 2018), where each relation-speciﬁc weight matrix is decomposed into a block-diagonal by decomposing into lowdimensional matrices.
W ( l ) r in equation 9 is deﬁned as a block diagonal matrix, diag ( A ( l ) 1 r , ..., A ( l ) Br )
where
A ( l ) kr ∈ R
( d ( l +1) /B )
×
( d ( l ) /B ) and B is the number of basis matrices.
The block decomposition reduces the number of parameters and helps to prevent overﬁtting.
3. P ARAMETER L EARNING AND I NFERENCE OF RE-N ET
Parameter Learning via Event Prediction.
The (object) entity prediction given (s , r) can be viewed as a multi-class classiﬁcation task, where each class corresponds to one object entity.
Similarly, relation prediction given s and subject entity prediction can be considered as a multi-class classiﬁcation task.
Here we omit the notation for previous events.
To learn weights and representations for entities and relations, we adopt a multi-class cross-entropy loss to the model’s output.
The loss function is comprised of three losses and is deﬁned as:
[Equation]
where G is set of events, and λ 1 and λ 2 are importance parameters that control the importance of each loss term.
λ 1 and λ 2 can chosen depending on a task.
If the task aims to predict o given ( s, r ) , then we can give small values to λ 1 and λ 2 .
Each probability is deﬁned in equations 3, 4, and 5, respectively.
We apply teacher forcing for model training over historical data.
Multi-step Inference over Time.
At inference time, RE-N ET seeks to predict the forthcoming events based on the previous observations.
Suppose that the current time is t and we aim at predicting events at time t + ∆ t , then the problem of multi-step inference can be formalized as an inference problem, i.e., inferring the conditional probability P ( G t +∆ t | G : t ) .
The problem is nontrivial as we need to integrate over all G t +1:
t +∆ t − 1 .
To achieve efﬁcient inference, we draw a sample of G t +1:
t +∆ t − 1 , and estimate the conditional probability in the following way:
[Equation]
Such an inference procedure is intuitive.
Basically, one starts with computing P ( G t +1 | G : t ) , and drawing a sample ˆ G t +1 from the conditional distribution.
With this sample, one can further compute P ( G t +2 | ˆ G t +1 , G : t ) .
By iteratively computing the conditional distribution for G t (cid:48) and drawing a sample from it, one can eventually estimate P ( G t +∆ t | G : t ) as P ( G t +∆ t | ˆ G t +1:
t +∆ t − 1 , G : t ) .
In practice, we can improve the estimation by drawing multiple graph samples at each step, but RE-N ET already performs very well with a single sample, and thus we only draw one sample graph at each step for better efﬁciency.
Based on the estimation of the conditional distribution, we can further predict events which are likely to form in the future.
We summarize the detailed inference algorithm in Algorithm 1.
In Algorithm 1, we sample one graph at a time.
To obtain the graph, we ﬁrst sample M number of s (line 3) and pick top- k triples (line 4).
Then we have a knowledge graph at time t (cid:48)
(line 5).
Computational Complexity Analysis.
Here we analyze the time complexity of the graph generation algorithm 1.
To compute P (s t | G t − m : t − 1 ) (equation 5), it takes O ( | E | Lm ) , where | E | is the maximum number of triples among { G t − m , ..., G t − 1 } , L is the number of layers of aggregation, and m is the number of the past time steps since we unroll m time steps in RNN.
From this probability, we sample M number of subjects s .
To compute P (s t , r t , o t | G t − m : t − 1 ) (equation 2), it takes O ( D L m ) , where D is the maximum degree of entities.
To get probabilities of all possible triples given sampled subjects, it needs O ( M | R || O | D L m ) where | R | is the total number of relations and | O | is the total number of entities.
Thus, the time complexity for generating one graph is
O ( | E | Lm + M | R
|| O | ( D L m + log k ))
where k is the cutoff number for picking top- k triples.
The time complexity is linear to the number of entities and relations, and the number of sampled s .
E XPERIMENTS
Evaluating the quality of generated graphs is challenging, especially in knowledge graphs (Theis et al., 2015).
Instead, we evaluate our proposed method on a link prediction task on temporal knowledge graphs.
The task of predicting future links aims to predict unseen relationships with object entities given (s , r , ? , t ) (or subject entities given (? , r , o , t ) ), based on the observed events in the
TKG.
Essentially, the task is a ranking problem over all the events (s , r , ? , t ) (or (? , r , o , t ) ).
RE-N ET can approach this problem by computing the probability of each event in a distant future with the inference algorithm in Algorithm 1, and further rank all the events according to their probabilities.
We evaluate our proposed method on three benchmark tasks: (1) predicting future events on three event-based datasets; (2) predicting future facts on two knowledge graphs which include facts with time spans, and (3) studying parameter sensitivity and ablation of our proposed method.
Section 4.
[Table]
summarizes the datasets
[Table]
and the supplementary material contains additional information.
In all these experiments, we perform predictions on time stamps that are not observed during training.
4. E XPERIMENTAL S ET - UP
Datasets.
We use ﬁve datasets: 1) three event-based temporal knowledge graphs: ICEWS18 (Boschee et al., 2015), ICEWS14 (Trivedi et al., 2017), and GDELT (Leetaru & Schrodt, 2013); and 2) two knowledge graphs where temporally associated facts have meta-facts as (s , r , o , [ t s , t e ]) where t s is the starting time point and t e is the ending time point: WIKI (Leblay & Chekol, 2018) and
YAGO (Mahdisoltani et al., 2014).
The details of the datasets are described in Section B.
Evaluation Setting and Metrics.
For each dataset except ICEWS14, we split it into three subsets, i.e., train(80%)/valid(10%)/test(10%), by time stamps.
Thus, (times of train) < (times of valid) <
(times of test).
We report Mean Reciprocal Ranks (MRR) and Hits @1 / 3 / 10 , using the ﬁltered version and the raw version of the datasets.
Similar to the deﬁnition of ﬁltered setting in (Bordes et al., 2013), during evaluation, we remove from the list of corrupted triplets all the triplets that appear either in the train, dev, or test set.
Competitors.
We compare our approach to baselines for static graphs and temporal graphs: (1) Static Methods.
By ignoring the edge time stamps, we construct a static, cumulative graph for all the training events, and apply multi-relational graph representation learning methods including
TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016), RGCN (Schlichtkrull et al., 2018), ConvE (Dettmers et al., 2018), and RotatE (Sun et al., 2019).
(2) Temporal Reasoning Methods.
We also compare state-of-the-art temporal reasoning methods for knowledge graphs, including Know-Evolve 3 (Trivedi et al., 2017), TA-DistMult (García-Durán
et al., 2018), HyTE (Dasgupta et al., 2018), and TTransE (Leblay & Chekol, 2018).
TA-DistMult,
HyTE, and TTransE are for a interpolation task which is to make predictions at time t such that t 1 < t < t 2 , which is different from our setting.
We give random values or embeddings that are not observed during training.
To see the effectiveness of our recurrent event encoder, we use encoders of previous work and our MLP decoder as baselines; we compare Know-Evolve, Dyrep (Trivedi et al., 2019), and GCRN (Seo et al., 2017)
combined with our MLP decoder, which are called
Know-Evolve+MLP, DyRep+MLP, and R-GCRN+MLP.
The GCRN utilizes Graph Gonvolutional
Network (Kipf & Welling, 2016).
Instead, we use RGCN (Schlichtkrull et al., 2018) to deal with multi-relational graphs.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
(3) Variants of RE-N ET .
To evaluate the importance of different components of RE-N ET
[Table]
we varied our base model in different ways: RE-N ET w/o multi-step which does not update history during inference, RE-N ET without the aggregator (RE-N ET w/o agg.), RE-N ET with a mean aggregator (RE-N ET w. mean agg.), and RE-N ET with an attentive aggregator (RE-N ET w. attn agg.).
takes a zero vector instead of a representation of the aggregator.
RE-N ET w. GT (s , r) denotes RE-N ET with ground truth history or interactions during multi-step inference, and thus the model knows all the interactions before the time for testing.
It does not update history (or generate a graph) since it already has ground truth history.
Experiment settings and implementation details of RE-N ET and baselines are described in Section C.
4. P ERFORMANCE C OMPARISON ON T EMPORAL K NOWLEDGE G RAPHS .
In this section we compare our proposed method with other baselines.
The test results are obtained by averaged metrics over the entire test sets on datasets.
Performances on Event-based TKGs.
Table 1 summarizes results on three event-based datasets:
ICEWS18, GDELT, and ICEWS14.
Our proposed RE-N ET outperforms all other baselines on these datasets.
Static methods show good results but they underperform our method since they do not consider temporal factors.
Also, RE-N ET outperforms all other temporal methods, which demonstrates effectiveness of the proposed method.
The modiﬁed Know-Evolve with our MLP decoder (Know-Evovle+MLP) achieves the better performances than Know-Evolve, which shows effectiveness of our MLP decoder, but there is still a large gap from our model.
We notice that
Know-Evolve and DyRep has a gradient exploding issue on their encoder since their RNN-like structures keep accumulating embedding over time.
This issue degrades their performances.
Graph
Convolutional Recurrent Network (GCRN) is not for dynamic and multi-relational graphs, and is not capable of link prediction.
We modiﬁed the model to work on dynamic graphs and our problem setting by using RGCN instead of GCN, and our MLP decoder.
The modiﬁed model (R-GCRN+MLP) shows good performances but it does not outperform our method.
R-GCRN+MLP has a similar structure to ours in that it has a recurrent encoder and an RGCN aggregator but it lacks multi-step inference, global information, and sophisticated modeling for the recurrent encoder.
These results of the combined models suggest the our recurrent event encoder shows better performances in link prediction.
Importantly, all these temporal methods are not capable of multi-step inference, while
RE-N ET sequentially infers multi-step events.
Performances on Public KGs.
Previous results have proved the effectiveness of RE-N ET , and here we will compare the method on the Public KGs: WIKI and YAGO.
In Table 2, our proposed
RE-N ET outperforms all other baselines.
In these datasets, baselines show better results than in the
Event-based TKGs.
This is due to the characteristics of the datasets; they have facts that are valid within a time span.
However, our proposed method consistently outperforms the static and temporal
[Figure] [Figure]
methods.
which implies that RE-N ET effectively infers new events using a powerful event encoder and an aggregator, and provides accurate prediction results
[Figure]
Performances of
Prediction over Time.
Next, we further study performances of RE-N ET over time.
Figs.
3 shows the performance comparisons over different time stamps on the ICEWS18, GDELT,
WIKI, and YAGO datasets with ﬁltered Hits@3 metrics.
RE-N ET consistently outperforms baseline methods for all different time stamps.
Performances of each method ﬂuctuate since testing entities are different at each time step.
We notice that with the increase of time step, the difference between
RE-N ET and ConvE is getting smaller as shown in Fig.
This is expected since further future events are harder to predict.
Furthermore, we can think that the decline of the performances is due to the generation of a long graph sequence.
To estimate the joint probability distribution of all events in a distant future, RE-N ET should generates a long sequence of graphs.
The quality of generated graphs deteriorates when RE-N ET generates a long graph sequence.
4. A BLATION S TUDY
In this section, we study the effect of variations in RE-N ET .
To evaluate the importance of different components of RE-N ET , we varied our base model in different ways, measuring the change in performance on the link prediction task on the ICEWS18 dataset.
We present the results in Tables 1, 2, and Figs. 4.
Different Aggregators.
We ﬁrst analyze the effect of the aggregator.
In Tables 1, 2, we observe that
RE-N ET w/o agg.
hurts model quality.
This suggests that introducing aggregators make the model capable of dealing with concurrent events and aggregators improve the prediction performances.
Fig.
4a shows the performances of RE-N ET with different aggregators.
Among them, RGCN aggregator outperforms other aggregators.
This aggregator has the advantage of exploring multirelational neighbors not limited to neighbors under the same relation.
Also, RE-N ET with an attentive aggregator shows better performances than RE-N ET with a mean aggregator, which implies that giving different attention weights to each neighbor helps predictions.
Global Information.
We further observe that representations from global graph structures help the predictions.
Fig.
4b shows effectiveness of a representation of global graph structures.
We consider that global representations give information beyond local graph structures.
[Figure] [Figure]
Empirical Probabilities.
Here, we study the role of P (s t | G t − m : t − 1 ) and P (r t | s , G t − m : t − 1 ) .
We simply denote them as P (s) and P (r) for brevity.
Also, P (s t , r t | G t − m : t − 1 ) (or simply P (s , r) ) is equivalent to P (s) P (r) .
In Fig 4c, emp.
P (s) denotes a model with empirical P (s) (or P e (s) ) which is deﬁned as P e (s) =
(# of s -related triples) / (total # of triples).
Also, emp.
P (s , r) denotes a model with P e (s) and P e (r) which is deﬁned as P e (r) =
(# of r -related triples) / (total # of triples).
Thus, P e (s , r) =
P e (s) P e (r) .
RE-N ET use a trained P (s) and P (r) .
The results show that the trained P (s) and P (r) help RE-N ET for multi-step predictions.
Using P s (s) underperforms RE-N ET , and using P e (s , r) =
P e (s) P e (r) shows the worst performances, which suggests that training each part of the probability in equation 2 gives better prediction performances.
4. S ENSITIVITY A NALYSIS
In this section, we study the parameter sensitivity of RE-N ET including the length of history for the event encoder, cutoff position k for events to generate a graph.
Furthermore, we study the layers of RGCN aggregator.
We report the performance change of RE-N ET on the ICEWS18 dataset by varying the hyper-parameters in Table 5.
Length of Past History in Recurrent Event Encoder.
The recurrent event encoder takes the sequence of past interactions up to m graph sequences or previous histories.
Figure 5a shows the performances with varying length of past histories.
When RE-N ET uses longer histories, MRR is getting higher.
However, the MRR is not likely to go higher when the length of history is 5 and over.
This implies that long history does not make big differences.
Cut-off Position k at Inference Time.
To generate a graph at each time, we cut off top- k triples on ranking results.
Fig.
5b shows the performances with choosing different cutoff position k .
When k is 0, RE-N ET does not generate graphs for estimating P ( G t +∆ t | G : t ) , and it shows the lowest result.
which means RE-N ET performs single-step predictions, .
When k is larger, the performance is getting higher and it is saturated after 500.
We notice that the conditional distribution P ( G t +∆ t | G : t ) can be approximated by P ( G t +∆ t | ˆ G t +1:
t
+∆ t − 1 , G : t ) by using a larger cutoff position.
Layers of RGCN Aggregator.
We examine the number of layers in the RGCN aggregator.
The number of layers in the aggregator means the depth to which the node reaches.
Fig.
5c shows the performances according to different numbers of layers of RGCN.
We notice that 2-layered RGCN improves the performances considerably compared to 1-layered RGCN since 2-layered RGCN aggregates more information.
However, RE-N ET with 3-layered RGCN underperforms RE-N ET with 2-layered RGCN.
We conjecture that the bigger parameter space leads to overﬁtting.
C ONCLUSION
In this work, we studied the sequential graph generation on temporal knowledge graphs.
To tackle this problem, we proposed Recurrent Event Network (RE-N ET ) which models temporal, multi-relational, and concurrent interactions between entities.
A recurrent event encoder in RE-N ET summarizes information of the past event sequences, and a neighborhood aggregator collects the information of concurrent.
RE-N ET deﬁnes the joint probability of all events, and thus is capable of inferring global structures in a sequential manner.
We tested the proposed model on a link prediction task on temporal knowledge graphs.
The experiment revealed that the proposed RE-N ET outperforms all the static and temporal methods and our extensive experiments shows its strength.
Interesting future work includes modeling lasting events and performing inference on the long-lasting graph structures.
A R ECURRENT E VENT E NCODER
We deﬁne a recurrent event encoder based on RNN as follows:
[Equation]
We use Gated Recurrent Units (Cho et al., 2014) as RNN:
[Equation]
where : is concatenation, σ ( · ) is an activation function, and ◦ is a Hadamard operator.
The input is a concatenation of four vectors: subject embedding, object embedding, aggregation of neighborhood representations, and global information vector ( e s , e r , g ( N t (s)) , H t ).
h t ( s ) and H t are similarly deﬁned.
For h t ( s ) , a concatenation of subject embedding, aggregation of neighborhood representations, and global information vector ( e s , g ( N t ( s )) , H t ) is input.
For H t , aggregation of the whole graph representations g ( G t ) is input.
B D ATASET
We use ﬁve datasets: 1) three event-based temporal knowledge graphs and 2) two knowledge graphs where temporally associated facts have meta-facts as (s , r , o , [ t s , t e ]) where t s is the starting time point and t e is the ending time point.
The ﬁrst group of graphs includes Integrated Crisis Early
Warning System (ICEWS18 (Boschee et al., 2015) and ICEWS14 (Trivedi et al., 2017)), and Global
Database of Events, Language, and Tone (GDELT)
(Leetaru & Schrodt, 2013).
The second group of graphs includes WIKI (Leblay & Chekol, 2018) and YAGO (Mahdisoltani et al., 2014).
ICEWS18 is collected from 1/1/2018 to 10/31/2018, ICEWS14 is from 1/1/2014 to 12/31/2014, and
GDELT is from 1/1/2018 to 1/31/2018.
The ICEWS14 is from (Trivedi et al., 2017).
We didn’t use their version of the GDELT dataset since they didn’t release the dataset.
WIKI and YAGO datasets have temporally associated facts (s , r , o , [ t s , t e ]) .
We preprocess the datasets such that each fact is converted to { (s , r , o , t s ) , (s , r , o , t s + 1 t ) , ..., (s , r , o , t e ) } where 1 t is a unit time to ensure each fact has a sequence of events.
Noisy events of early years are removed (before 1786 for WIKI and 1830 for YAGO).
The difference between the ﬁrst group and the second group is that facts happen multiple times (even periodically) on the ﬁrst group (event-based knowledge graphs) while facts last long time but are not likely to occur multiple times in the second group.
Dataset statistics are described on table 3.
[Table]
C D ETAILED E XPERIMENTAL S ETTINGS
Model details of RE-N ET .
We use Gated Recurrent Units (Cho et al., 2014) as our recurrent event encoder, where the length of history is set as m = 10 which means saving past 10 event sequences.
If the events related to s are sparse, we check the previous time steps until we get m previous time steps related to the entity s .
We pretrain the parameters related to equations 5 and 8 due to large size of training graphs.
We use a multi-relational aggregator to compute H t .
The aggregator provides hidden representations for each node and we max-pool over all hidden representations to get H t .
At inference time, RE-N ET performs multi-step prediction across the time stamps in dev and test sets.
In each time step, we sample 1000 (= M ) number of subjects and save top-1000 (= k ) triples to use
[Table]
them as a generated graph .
We set the size of entity/relation embeddings to be 200 and embedding of unobserved embeddings are randomly initialized.
We use two-layer RGCN in the RGCN aggregator with block dimension 2 × 2 .
The model is trained by the Adam optimizer (Kingma & Ba, 2014).
We set λ 1 to 0.1, the learning rate to 0 .
001 and the weight decay rate to 0.00001.
All experiments were done on GeForce GTX 1080 Ti.
Experimental Settings for Baseline Methods.
In this section, we provide detailed settings for baselines.
We use implementations of TransE and DistMult 4 .
We implemented TTransE and TADistMult based on the implementation of TransE and Distmult, respectively.
For TA-DistMult, We use temporal tokens with the vocabulary of year, month and day on the ICEWS dataset and the vocabulary of year, month, day, hour and minute on the GDELT dataset.
We use a margin-based ranking loss with
L1 norm for TransE and use a binary cross-entropy loss for DistMult and TA-DistMult.
We validate the embedding size among 100 and 200.
We set the batch size to 1024, margin to 1.0, negative sampling ratio to 1, and use the Adam optimizer.
We use the implementation of ComplEx 5 Han et al.
We validate the embedding size among 50, 100 and 200.
The batch size is 100, the margin is 1.0, and the negative sampling ratio is 1.
We use the Adagrad optimizer.
We use the implementation of HyTE 6 .
We use every timestamp as a hyperplane.
The embedding size is set to 128, the negative sampling ratio to 5, and margin to 1.0.
We use time agnostic negative sampling (TANS) for entity prediction, and the Adam optimizer.
We use the codes for ConvE 7 and use implementation by Deep Graph Library 8 .
Embedding sizes are 200 for both methods.
We use 1 to all negative sampling for ConvE and use 10 negative sampling ratio for RGCN, and use the Adam optimizer for both methods.
We use the codes for Know-Evolve 9 .
For Know-Evolve, we ﬁx the issue in their codes.
Issues are described in Section F.
We follow their default settings.
We use the code for RotatE 10 .
The hidden layer/embedding size is set to 100, and batch size 256; other values follow the best values for the larger FB15K dataset conﬁgurations supplied by the author.
The author reports ﬁltered metrics only, so we added the implementation of the raw setting.
[Figure] [Figure]
Minutes
RE-Net
ConvE
TA-DistMult
[Figure] [Figure]
[Table]
D A DDITIONAL E XPERIMENTS D. R ESULTS WITH RAW METRICS .
Table 4 shows the performance comparison on ICEWS18, GDELT, ICEWS14 with raw settings.
Our proposed RE-N ET outperforms all other baselines.
Figs.
6 shows the performance comparisons over different time stamps on the ICEWS18, GDELT, WIKI and YAGO datasets with ﬁltered MRR.
Our proposed RE-N ET consistently outperform baselines over time.
D. C OMPARISONS WITH C ONV E+RGCN.
To examine aggregation techniques in other baselines, we combine ConvE and R-GCN.
We ﬁrst run 2-layered R-GCN over the training graph and then each node has its own transformed representations.
We run ConvE on this transformed representations.
As in Table 5, ConvE+R-GCN shows better performances than R-GCN and worse performances than ConvE, which implies that the aggregation technique is not helpful to ConvE.
However, aggregators in our framework is a complementary and necessary component to the temporal part, which shows superiority over the baselines.
D. C OMPARISONS WITH D YNAMIC M ETHODS .
Here we compare our method with dynamic methods on homogeneous graphs: EvolveGCN-O
(Pareja et al., 2019), DynGEM (Goyal et al., 2018), dyngraph2vecAE (Goyal et al., 2019), DynTriad (Zhou et al., 2018b), and tNodeEmbed (Singer et al., 2019).
These methods were proposed to predict interactions at a future time on homogeneous graphs, while our proposed method is for predicting interactions on multi-relational graphs (or knowledge graphs).
Furthermore, those methods predict links at one future time stamp, whereas our method seeks to predict interactions at multiple future time stamps.
We modiﬁed some methods to apply them on multi-relational graphs as follows.
Experimental Settings.
We adopt R-GCN (Schlichtkrull et al., 2018) for EvolveGCN-O and call it EvolveRGCN.
We convert knowledge graphs into homogeneous graphs for dyngraph2vecAE.
The idea of this method is to reconstruct an adjacency matrix using an auto-encoder and regard it as a future adjacency matrix.
If we keep relations, relation-speciﬁc adjacency matrices will be extremely sparse; the method learns to reconstruct near-zero adjacency matrices.
tNodeEmbed is a temporal method on homogeneous graphs.
To use this on multi-relational graphs, we ﬁrst train entity
Men
President of South Africa ( , Accuse, ? )
( Criticize, ) ( visit, )
( Statement, ) ( Military, Use force, ? )
( Use force, Businessman ) ( Arrest, Men )
( Arrest, Businessman ) ( Aid, Citizen ) ( , Invite, ? )
( Accuse, ) ( Use force, ) ( Employ weapons, )
Query (s, r, ?)
History (r, o) with same s at different times
Answer 𝑡 𝑡 −
[Figure]
embeddings with DistMult and set these as initial embeddings for entities in tNodeEmbed.
Also we give entity embeddings as input to LSTM of tNodeEmbed.
We concatenate output of LSTM and relation embeddings to predict objects.
We did not modiﬁed other methods since it is not trivial to extend the methods.
Results.
As shown in Table 5, RE-N ET signiﬁcantly outperforms the methods.
Furthermore, all the dynamic methods do not show good performances.
We conjecture that the methods are not designed for multi-relational graphs, and thus it is not capable of effectively handling multiple relations, which leads to degradation of their performances.
Also, DynGEM is not suitable for our setting since it predicts edges based on observed edges at future time stamps.
However, in our setting, we are not given any observed edges at future times stamps, so it shows poor performances.
E C ASE S TUDY
In this section, we study RE-N ET ’s predictions.
Its predictions depend on interaction histories.
We categorize histories into three cases: (1) consistent interactions with an object, (2) a speciﬁc temporal pattern, and (3) irrelevant history.
RE-N ET can learn (1) and (2) cases, so it achieves high performances.
For the ﬁrst case, static methods cannot predict the answer since it does not see past interactions.
However, RE-N ET can predict the answer because it consistently interacts with an object.
The second case shows speciﬁc temporal patterns on relations: ( Arrest, o ) → ( Use force, o ).
Without knowing this pattern, one method might predict “Businessman" instead of “Men".
RE-N ET is able to learn these temporal patterns so it can predict the second case.
Lastly, the third case shows irrelevant history to the answer and the history is not helpful to predictions.
RE-N ET fails to predict the third case.
F I MPLEMENTATION I SSUES OF K NOW -E VOLVE
We found a problematic formulation in the Know-Evolve model and codes.
The intensity function (equation 3 in (Trivedi et al., 2017)) is deﬁned as λ s,rr ( t | ¯ t ) =
f ( g s,rr (¯ t ))
( t − ¯ t ) , where g ( · ) is a score function, t is current time, and ¯ t is the most recent time point when either subject or object entity was involved in an event.
This intensity function is used in inference to rank entity candidates.
However, they don’t consider concurrent event at the same time stamps, and thus ¯ t will become t after one event.
For example, we have events e
=
( s, r, o 1 , t 1 ) , e 2 =
( s, r, o 2 , t 1 ) .
After e 1 , ¯ t will become t (subject s ’s most recent time point), and thus the value of intensity function for e 2 will be 0.
This is problematic in inference since if t = ¯ t , then the intensity function will always be 0 regardless of entity candidates.
In inference, all object candidates are ranked by the intensity function.
But all intensity scores for all candidates will be 0 since t = ¯ t , which means all candidates have the same 0 score.
In their code, they give the highest ranks (ﬁrst rank) for all entities including the ground truth object in this case.
Thus, we ﬁxed their code for a fair comparison; we give an average rank to entities who have the same scores.
G T HEORETICAL A NALYSIS
Here we analyze the model capacity of RE-N ET of capturing complex time-invariant local structure like (Hamilton et al., 2017), as well as the emerging global community structure as (You et al., 2018).
Theorem 1 Let { G t } τt =1 be the snapshot of temporal knowledge graph after τ time-steps.
Let h 0 v ∈ R d , v ∈ { s i } ∪ { o i } to be the input feature representation for Algorithm 1 of each entity node v .
Suppose that there exists a ﬁxed positive constant C ∈ R + such that
||
h 0 v − h 0 v (cid:48)
|| > C for all pair of all pair of entities v, v (cid:48) .
Then we have that ∀ (cid:15) > 0 , there exist a parameter setting Θ for RE-N ET s.t.
after K = 4 layers of aggregation,
[Equation]
where h Kv,τ are output values generated by RE-N ET and c v,τ are clustering coefﬁcients of { G i } τi =1 .
Observation 1 Consider a temporal graph under stochastic block model described in Section G.2.
Let h 0 v ∈ R d , v ∈ { s i } ∪ { o i } to be the input feature representation for Algorithm 1 of each node.
Suppose that a constant portion p c of input representations can be linearly separated by a hyperplane, while the representation of other nodes lies on the hyperplane.
There exists a parameter setting of
RE-N ET that can output the probability that new node j connected to node i .
G. P ROOF FOR T HEOREM
Using pooling aggregator of GraphSAGE, we can actually copy its behavior by setting recurrent weight matrix of the RNN model to be 0 .
In this case, we lose all time-dependency our RE-N ET and the representation model becomes time-invariant .
However, RE-N ET have exactly the same model capacity as GraphSAGE.
G. A NALYSIS FOR O BSERVATION
Here we deﬁne the generation process of our temporal graph.
Assume that the generation process of the graph follows a stochastic block model, and there are two communities in the graph.
Half of the nodes belong to community A and the other half belong to community B. Nodes within one community have probability p s to be connected while other pairs have p d < p s probability to be connected.
The edges in the graph are introduced into the graph in a time.
Suppose a sequence of time-steps, a new node is introduced to the community and each edge is added to the graph.
This observation follows from three facts: (1) For each node v j in the neighborhood N ( v ) , using pooling aggregator, we can detect their community assignment s j .
We assign the output of community
A to be +1 and the output of community B to be − 1 .
The error of incorrectly discerning the community of a node decrease exponentially with the number of links.
For example let the node v be in community A. Let the total number of nodes at time t to be n t , by Hoeffding’s inequality we have
[Equation]
(3) Given the correct community classiﬁcation, the relation classiﬁer is able to predict the probability of linking nodes.
Combining these three facts, RE-N ET is able to infer the community structure of the node.