Re-Temp: Relation-Aware Temporal Representation Learning for Temporal Knowledge Graph Completion (2023)

Abstract: Temporal Knowledge Graph Completion (TKGC) under the extrapolation setting aims to predict the missing entity from a fact in the future, posing a challenge that aligns more closely with real-world prediction problems. Existing research mostly encodes entities and relations using sequential graph neural networks applied to recent snapshots. However, these approaches tend to overlook the ability to skip irrelevant snapshots according to entity-related relations in the query and disregard the importance of explicit temporal information. To address this, we propose our model, Re-Temp (Relation-Aware Temporal Representation Learning), which leverages explicit temporal embedding as input and incorporates skip information flow after each timestamp to skip unnecessary information for prediction. Additionally, we introduce a two-phase forward propagation method to prevent information leakage. Through the evaluation on six TKGC (extrapolation) datasets, we demonstrate that our model outperforms all eight recent state-of-the-art models by a significant margin.

## introduction a knowledge graph (kg) is a graph-structure database, composed of facts represented by triplets in the form of (subject entity, relation, object entity) such as (alice, is a friend of, bob). in this graph, entities serve as nodes, and relations are depicted as direct edges connecting the nodes. however, facts in a kg are not static but undergo continuous updates over time. to incorporate temporal information into the kg, temporal knowledge graphs (tkgs) are introduced. tkgs add the extra temporal information of each fact and extend each triple with a timestamp as a quadruplet (subject entity, relation, object entity, timestamp). a tkg can be represented as a sequence of snapshots, * corresponding author. caren.han@sydney.edu.au where each snapshot represents a static knowledge graph for one specific timestamp. temporal knowledge graph completion (tkgc) aims to predict the missing entity from a query (subject entity, relation, ?, timestamp) or (?, relation, object entity, timestamp). tkgc is difficult and even large-scale pre-trained language models such as chatgpt(openai, 2022) are prone to making factual errors @xcite . there are two main settings: interpolation and extrapolation setting. tkgc under the interpolation setting completes the facts in history, while tkgc under the extrapolation setting predicts facts at future timestamps. in this paper, we focus on tkgc in the extrapolation setting, which is more challenging and requires further improvement @xcite . enormous attention has been focused on static kgc problems, and numerous models have been employed to encode entities and relations. however, a key question remains: how can a static kgc model be extended to incorporate temporal infor-mation for tkgc tasks? recent works @xcite @xcite have utilised sequential graph neural networks (gnns) to the previous snapshots for encoding the entities and relations. then, they use a static score function as the decoder to assess the score of each candidate. sequential gnns are used because the facts shown in recent history can be helpful when making predictions in the future. an example is shown in figure 1 , the previous facts (kim jong-un, criticize, united states) three days before and (kim jong-un, make statement, donald trump) one day before may imply (donald trump, threaten with administrative sanction, kim jong-un) today. since no explicit timestamp value is used, we can call it "implicit temporal information". however, to effectively encode the timestamp, the temporal information, two additional considerations arise: first, explicit temporal information is crucial. for instance, the validity score of (donald trump, threaten with administrative sanction, kim jong-un) may differ between 2018 and 2023 as donald trump was the president in 2018 but not in 2023, affecting his ability to threaten another nation with administrative sanctions in 2023. the nature of entities can change over time, necessitating the consideration of explicit temporal information to encode time-dependent factors. second, not all the facts in the recent history are relevant. given historical facts (kim jong-un, criticize, united states, 2018-08-01),(donald trump, make a visit, switzerland, 2018-08-02) and @xcite , when calculating the score of (donald trump, threaten with administrative sanction, kim jongun, 2018-08-01), the second quadruplet visiting switzerland does not contribute to the prediction of the relation between donald trump and kim jong-un since switzerland is neutral. in such case, the model should find a way to skip the irrelevant snapshots based on the entity-related relation in the query. therefore, an optimal tkgc model should consider (1) explicit temporal information and (2) implicit temporal information with skipping irrelevant snapshots by considering the query. in this paper, we propose re-temp, an innovative tkgc model designed for extrapolation settings that incorporates relation-aware temporal representation learning. the encoder of re-temp utilises explicit temporal embedding for each entity, combining static and dynamic embedding. within the encoder, a sequential gnn is employed to capture the implicit temporal information with a skip information flow applied after each timestamp, taking into account the entity-related relation in the query. the main contributions of this paper can be summarised as follows: ## related work kgc models normally adopt an encoder-decoder framework @xcite , where the encoder generates the embedding of entities and relations and the score function plays as a decoder. most of the existing works extend the static kgc models into tkgc models by introducing temporal information. ## re-temp the overall architecture of re-temp can be found in figure 2 . section 3.1 describes the notations of a tkgc task. the input of the model is represented by a combination of static and dynamic entity embedding, in section 3.2, showing explicit temporal information. the encoder in section 3.3 uses a sequential multi-relational gnn to learn implicit temporal information and after each timestamp, a relation-aware skip information flow mechanism is applied to retain the necessary information for prediction. the convtranse decoder together with the loss function is introduced in section 3.4. to avoid information leaking, we apply a two-phase forward propagation method in section 3.5. ## conclusion we introduced re-temp, which integrates both explicit and implicit temporal information and applies a relation-aware skip information flow to adopt after each timestamp to remove unnecessary information for prediction by taking the entity-related relation in the query into consideration. the experimental results on six tkgc datasets present the superiority of our model, compared with eight baseline models. we also conduct a statistical analysis of the datasets to show the different nature between wiki and other datasets. it is hoped that re-temp presents insight into the importance of the relation in the query and both types of temporal information.