Learning Joint Structural and Temporal Contextualized Knowledge Embeddings for Temporal Knowledge Graph Completion (2023)

Abstract: Temporal knowledge graph completion that predicts missing links for incomplete temporal knowledge graphs (TKG) is gaining increasing attention. Most existing works have achieved good results by incorporating time information into static knowledge graph embedding methods. However, they ignore the contextual nature of the TKG structure, i.e., query-specific subgraph contains both structural and temporal neighboring facts. This paper presents the SToKE, a novel method that employs the pre-trained language model (PLM) to learn joint Structural and Temporal Contextualized Knowledge Embeddings.Specifically, we first construct an event evolution tree (EET) for each query to enable PLMs to handle the TKG, which can be seen as a structured event sequence recording query-relevant structural and temporal contexts. We then propose a novel temporal embedding and structural matrix to learn the time information and structural dependencies of facts in EET.Finally, we formulate TKG completion as a mask prediction problem by masking the missing entity of the query to fine-tune pre-trained language models. Experimental results on three widely used datasets show the superiority of our model.

## introduction knowledge graphs have facilitated many realworld applications, including question answering, dialogue systems and speech recognition @xcite . the rapidly growing facts on the knowledge graph often show dynamic relations or interactions of entities along the timeline, which creates the need for introducing the concept of temporal knowledge graph (tkg). such tkgs often suffer from incompleteness due to their own dynamic features. therefore, the temporal knowledge graph completion (tkgc) task that predicts missing links across these tkgs is gaining increasing attention from researchers @xcite . recent works on the tkgc task have primarily focused on static knowledge graph embedding (kge) methods. they extend kge models by encoding time information into conventional score functions in different ways @xcite @xcite @xcite , called temporal knowledge graph embedding (tkge). although tkge methods outperform static kge on this task, they mostly fail to consider the rich contextual information related to the query in the tkg structure. as a subgraph of tkg shown in figure 1 (a), when facing the query (china, sign formal agreements, ?, 2014-12-30), its relevant contexts include both concurrent facts within the same timestamp (solid red) and temporal adjacency facts at different timestamps (dashed blue). the structural contexts (china, intend to cooperate, japan, 2014-12-30) and (china, engage in diplomatic cooperation, @xcite suggests that china may sign formal agreements with either japan or iran. moreover, the temporal contexts illustrate that china has been expressing intent to cooperate with japan for some time, making japan a more reasonable answer to the query. hence, learning knowledge embeddings that could effectively capture queryrelevant structural and temporal contextual information facilitates the inference of missing facts. however, existing methods either only encode temporal adjacent facts residing in the query's local neighborhood @xcite , or only focus on concurrent events within the same timestamp and integrate entity representations of different timestamps with additional recurrence or attention mechanisms @xcite . they fail to encode two query-relevant contexts simultaneously only using graph neural network variants, which requires us to solve the above problem by introducing a new architecture. recent advances in pre-trained language models (plms) are able to learn word representations and graph representations through distinct structured contexts @xcite . intuitively, plms can capture contextual meanings of entity and relation representations along joint structural and temporal dimensions. inspired by this, we propose the stoke, a novel bert-based model to learn knowledge embeddings with joint structural and temporal contexts. another challenging problem is to draw connections of query-specific substructure to natural language sentences. to handle this issue, we construct an event evolution tree (eet) for each individual query as shown in figure 1(b) . the eet represents the temporal evolution in the order from "root" to "crown", where each layer corresponds to different timestamps. facts occurring at the same timestamp constitute the "branches" of this layer, and query's subject entity china acts as a "trunk" connecting different structural and temporal adjacent facts. eet is essentially a structured event sequence that helps plms better understand contextual information in the query-specific subgraph. besides, we add a novel temporal embedding in the initial embeddings to ensure the sequential order of facts appearing in the eet. considering the knowledge noise problem caused by introducing query-relevant contexts with different dimensions, we present a structural matrix to restrict the structure dependencies between facts. overall, this paper makes the following contributions: ## notations and task definition a temporal knowledge graph (tkg) g can be viewed as a multi-relational, directed graph with timestamped edges between nodes (entities). each event (fact) in the @xmath0, g 1 , . . . , g t } can be represented as a quadruple (s, r, o, t) or (s t , r t , o t ), corresponding to subject entity @xmath1, relation type @xmath2, object entity @xmath3, where e, r and t represent the sets of entities, relationships and timestamps, respectively. the purpose of the temporal knowledge graph completion task is to infer the missing object entity o given the query (s, r, ?, t), or missing subject entity s of the query (?, r, o, t), @xmath4, . . . , t }. ## method as shown in figure 2 , our model consists of two stages. the first step is to construct an event evolution tree (eet) for each query. eet can be viewed as a structured event sequence incorporating structural and temporal contextual information related to the predicted fact. second, we present our stoke model, which learns joint structural and temporal contextualized knowledge representations based on the constructed eet. ## conclusion in this paper, we propose a novel model for the temporal knowledge graph completion task named stoke, which learns contextualized knowledge representations in terms of joint structural and temporal dimensions. unlike other gnn-based methods, our model uses a unified bert architecture to simultaneously explore contextual information of the tkg substructure, i.e., query-relevant structural and temporal neighboring facts. to enable bert to handle the tkg, we construct an event evolution tree (eet) for each individual query, and introduce temporal embedding and structural matrix to ensure the temporal order and structural dependencies among facts in eet. through masking the missing entity of query to fine-tune the pre-trained bert, our model outperforms other methods on three widely used datasets.