RTFE : A Recursive Temporal Fact Embedding Framework for Temporal Knowledge Graph Completion (2021)

Abstract: Static knowledge graph (SKG) embedding (SKGE) has been studied intensively in the past years. Recently, temporal knowledge graph (TKG) embedding (TKGE) has emerged. In this paper, we propose a Recursive Temporal Fact Embedding (RTFE) framework to transplant SKGE models to TKGs and to enhance the performance of existing TKGE models for TKG completion. Different from previous work which ignores the continuity of states of TKG in time evolution, we treat the sequence of graphs as a Markov chain, which transitions from the previous state to the next state. RTFE takes the SKGE to initialize the embeddings of TKG. Then it recursively tracks the state transition of TKG by passing updated parameters/features between timestamps. Specifically, at each timestamp, we approximate the state transition as the gradient update process. Since RTFE learns each timestamp recursively, it can naturally transit to future timestamps. Experiments on five TKG datasets show the effectiveness of RTFE.

## introduction temporal knowledge graph (tkg) is an extension of static knowledge graphs (skgs) which introduce the time dimension. in skgs, facts are considered to be time-invariant @xcite . in reality, facts are not always true. for example, the triple (obama, president, united states) was true only from 2009 to 2016 and (obama, married, mitchell) since 1992. however, skgs do not reflect the change in facts over time. an example of tkg is shown in figure 1 . besides, facts on social networks, e-commerce platforms and trading platforms also change over time. therefore, tkgs have the potential to improve the performance of question answering, search, recommendation and prediction based on kgs @xcite . tkg can be expressed as a set of quadruples @xcite . different from skgs which ignore the time attribute of facts, the facts of tkgs are distributed in timestamps, which can reflect the dynamic change of entities and relationships over time. due to the limited coverage of kgs, tkgs are also incomplete. by completing tkg, missing and potential knowledge under specific timestamps can be found. in recent years, a lot of work @xcite @xcite @xcite @xcite has focused on kg completion by methods of graph embedding. these efforts have yielded good results, but most of them focused on skgs and required training in a large number of triples. however, the tkg under a certain timestamp is a sparse multi-relation graph @xcite , so it is necessary to absorb information from other timestamps. what's more, skge methods lacked the modeling of time attribute of relations, and were proposed based on the assumption that all facts occur at the same time. so they cannot reflect the temporal dependencies of facts. to handle these two problems, our rtfe passes parameters and features between timestamps in a recursive manner, which not only alleviates the sparsity problem of tkg, but takes advantage of the continuity and relevance characteristics of the fact as well. existing tkg completion methods @xcite @xcite follow the training pattern of skge, which shuffles facts (s, r ,o ,t) of different time randomly and learns all facts in a chaotic temporal order by minibatch gradient descent algorithm. in other words, they just take time as a parameter but ignore the correlations in time evolution. however, the early state may affect the later one and later facts tend to be dependent on early ones. in particular, the state at t i directly influences that at t i+1 . e.g., @xcite directly influences (obama, inaugurated as, president, 2009). it has been verified that the chronological order of events can be used to improve the performance of link prediction @xcite . based on this, we further find that the early training state can improve later one if we train facts in their chronological order. in order to capture changes in tkg's state transition, we think tkg as a sequence of dynamic graphs, not as a whole graph labeled with time information. besides, since new facts of future timestamps can be added to tkg, tkg is expanding dynamically. and the graphs of new timestamps may still be incomplete. however, existing tkg completion methods provide no solution to complete unseen future graphs. in their training pattern, facts of all timestamps are trained jointly to complete the graphs that have appeared. models may need to be retrained on facts of all timestamps when a new timestamp appears. in contrast, our rtfe embeds and completes tkg's each timestamp in a recursive way. by using the information of previous timestamps, rtfe can be naturally extended to future timestamps during the state transition of parameters/features. rtfe only needs to be trained on new emerging facts, which is light and immediate. skge has been studied for many years while tkge is still at birth. problems encountered in skge (e.g., diverse relation patterns) can also occur in tkge. thus the advantages of skg researches can be used to accelerate the development of tkgs if we bridge the gap between them. our rtfe provides a way to migrate skge methods to tkgs while preserving their excellent effects. further, existing tkg completion methods de-signed specifically for the characteristics of tkgs can also be enhanced using the training pattern of rtfe. to sum up, we have made the following contributions: to the best of our knowledge, we are the first to deal with the tkg evolution problem (i.e., new future timestamps are added to tkgs) in the tkg completion task. 4. the experimental results on 5 tkg datasets show that rtfe preserve skge models' excellent performance. and the predictive performance of state-of-the-art tkge models are further enhanced using rtfe. ## problem definition a temporal knowledge graph (tkg) can be represented as a sequence of graphs, i.e. @xmath0, . . . , g tn } where g t i is a set of quadruples that occured at timestamp t i , i.e. g t @xmath1, r, o, t i )} where v is the set of g's entities and s, @xmath2; r is the set of g's relations and @xmath3. we focus on the following task: given a training tkg g @xmath4, . . . , g tn } , to infer the missing quadruples (s, r, o, t) in test set g @xmath5, . . . , g tn } (i.e., assign high scores to true quadruples and low scores to false ones). as shown in figure 1 , missing facts with high probability are dotted. ## recursive temporal fact embedding (rtfe) framework the state of tkgs change with the change of entities and relations over time. skge models fail to capture correlations during state transition. and existing tkge models for tkg completion capture it implicitly. it can be observed that the tkg after the current change changes on the closest former state, which is similar to a first order markov chain (given the state at the current moment, the state at the next moment is independent of the state at the past moment). inspired by markov analysis, we use the time granularity of tkgs to discretely divide states. then the basic model of rtfe can be expressed as: the idea of rtfe is to dynamically adjust θ and x as the tkg changes while passing the information of each timestamp graph. we simply assume the features and parameters satisfy markov property: rtfe does not specify a model, but rather a training method for tkg completion. existing skge methods and tkge methods that follow the skge training pattern such as de-simple @xcite and tcomplex @xcite can potentially be utilized as the embedding component. the rtfe framework is illustrated in figure 2 . in section 3, we specify that rtfe how to use skge models for tkg completion. in section 4, we generalize rtfe to existing tkge models to enhance their performance. ## related work there are two kinds of facts in the tkgs: continuous facts and discrete events. continuous facts have temporal attributes (since. . . is true, until. . . ends) like @xcite . and discrete events have temporal attributes (happens at. . . ) like (obama, inaugurated as, @xcite . in recent years, some work @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite began to use the time information to improve the kg completion or directly complete the tkg. based on the fact or event they dealt with, we state representative tkge methods as follows. ## conclusion we propose a framework rtfe for tkg completion. we have transplanted skge models to tkgs and enhance the performance of existing tkge models. experiments show that on five tkg datasets rtfe outperformed baselines and is extensible for future timestamps to some extent. in the future, we will further deal with discrete events. since events with adjacent timestamps are correlated, we plan to modify rtfe so that it can learn correlations (especially causality) of events. by modeling spatio-temporal dependency of tkg, events in future timestamps can be forecasted. besides, we plan to deal with the task of predicting time validity of facts @xcite .