Deja vu: Contrastive Historical Modeling with Prefix-tuning for Temporal Knowledge Graph Reasoning (2024)

Abstract: Temporal Knowledge Graph Reasoning (TKGR) is the task of inferring missing facts for incomplete TKGs in complex scenarios (e.g., transductive and inductive settings), which has been gaining increasing attention. Recently, to mitigate dependence on structured connections in TKGs, text-based methods have been developed to utilize rich linguistic information from entity descriptions. However, suffering from the enormous parameters and inflexibility of pre-trained language models, existing text-based methods struggle to balance the textual knowledge and temporal information with computationally expensive purpose-built training strategies. To tap the potential of text-based models for TKGR in various complex scenarios, we propose ChapTER, a Contrastive historical modeling framework with prefix-tuning for TEmporal Reasoning. ChapTER feeds history-contextualized text into the pseudo-Siamese encoders to strike a textual-temporal balance via contrastive estimation between queries and candidates. By introducing virtual time prefix tokens, it applies a prefix-based tuning method to facilitate the frozen PLM capable for TKGR tasks under different settings. We evaluate ChapTER on four transductive and three few-shot inductive TKGR benchmarks, and experimental results demonstrate that ChapTER achieves superior performance compared to competitive baselines with only 0.17% tuned parameters. We conduct thorough analysis to verify the effectiveness, flexibility and efficiency of ChapTER.

## introduction knowledge graphs (kgs) constitute structured representations of knowledge, storing substantial factual information in the form of @xcite . kgs have been an essential component of various nlp applications including question answering @xcite , recommendation @xcite , etc. considering facts inherently evolve in kgs over time, temporal knowledge graphs (tkgs) are constructed to describe the relationship between entities over time in the form of quadruple @xcite . while tkgs are usually incomplete, tkg reasoning (tkgr) aims to predict the missing facts from known ones. in this paper, we focus on the extrapolation task, which requires forecasting future events on tkgs with historical events. for instance, tkgr needs to answer the query (olivia rodrigo, release an album, ?, 2023-9-8) by matching and selecting from all candidate entities based on related historical events. to address the problem of tkg reasoning, many efforts have been made to capture temporal evolutional information in tkgs. due to the graph-like features of tkgs, previous methods @xcite @xcite work on designing the temporal-aware encoders referring to known history and mine evolutional patterns from query neighborhoods. recently, pretrained language models (plms) have been showing great abilities to model textual linguistic semantics, and some methods incorporate temporal information of tkgs into plms by designing manufactural prompts with fact texts @xcite , tuning plms with a time-specific masking strategy @xcite , etc. nevertheless, on the one hand, these manually designed prompts are explicitly based on a priori assumption. on the other hand, training an entire language model on the time-specific masking strategy is computationally expensive. though plms are equipped with strong linguistic inherence from the pre-training stage, they tend to exceedingly focus on the textual semantics, thus struggling to balance time-specific information and textual knowledge in tkgs. furthermore, given the highly dynamic essence of tkg, the continuous emergence of new unseen entities usually leads to the need for tkgr to predict entities in a more complex scenario (e.g. few-shot inductive scenario). thus, in this paper, we focus on the research question: can we efficiently integrate temporal history into textual knowledge in a unified plm-based framework and adapt to tkg reasoning tasks in various complex scenarios? to this end, we propose chapter, an efficient temporal-aware plm-based pseudo-siamese framework adaptable for tkgr in different scenarios. specifically, chapter first verbalizes the input queries and candidates with up-to-date historical contexts, and feeds them into the twotower model encoders individually to learn historycontextualized embeddings in a decoupled manner. then contrastive estimation is performed between them to strike a balance of temporal information and textual knowledge in representations. rather than training an entire model, chapter applies a two-tower prefix-based tuning method to enable frozen plms capable of performing tkg reasoning tasks under both transductive and few-shot inductive settings. to refrain from the dependency of entity-related prompts in previous method @xcite , we feed entity-agnostic virtual time prefix prompts into the frozen plms, empowering the model to tkgs with unseen entities. to summarize, our contributions are as follows: ## related work transductive tkg reasoning. most existing transductive tkg reasoning methods are embedding-based and some of them extended from previous kg reasoning methods. ttranse @xcite extends the distance-based method transe @xcite by incorporating extra temporal constraints among facts. tntcomplex @xcite extends com-plex @xcite by performing 4thorder tensor factorization to learn time-aware representations. besides, graph-based methods @xcite @xcite employ gcns to capture the structural information via message passing and model temporal correlations from knowledge graph snapshots with historical information. more recently, plm-based models have been utilized to incorporate external textual semantics for tkg reasoning. ppt @xcite converts tkgr task into a masked token prediction task by utilizing plm with manually designed prompts. ecola @xcite learns the contextualized representations by jointly optimizing the tkgr and the masked language modeling objectives. inductive tkg reasoning. tkg reasoning tasks under inductive setting aim to predict new emerging entities in tkgs, indicating that unseen entities in the test set are not contained in the train set. to handle unseen entities, gnn-based methods like grail @xcite and noodle @xcite extract enclosing subgraphs and learn entityagnostic local structural information. tlogic @xcite mines entity-independent logic rules to infer unseen entities. sst-bert @xcite conducts inductive relation prediction by applying a time masking mlm task to pre-train bert with structured sentences. filt @xcite adopts a meta-learning-based model with entity concepts to handle unseen entities in tkgs. prefix tuning. prompts are manually designed textual templates to query a language model, and they are beneficial to help language models solve different tasks with all parameters frozen. to alleviate the suboptimal performance caused by discrete prompting, continuous prompts with trainable embeddings are added to the embeddings of input sequence @xcite , which have been shown to achieve competitive performance across various nlp tasks. li and liang (2021) adds trainable prefix vectors to each transformer layer within frozed seq2seq plms, aiming to efficiently adapt plms to natural generation tasks. chen et al. (2023a) introduces conditional soft prompts to sufficiently incorporate textual se-t barack obama [sep] sign formal agreement feb-26 [sep] … head of government (afghanistan) feb-25 …… time head relation … … … … … … … e[cls] e0 e[sep] ek … … e[sep] en t[cls] t0 t[sep] tk … … t[sep] tn plm tn+1 en+1 … … query-related historical contexts t-2 t-n 1 t-1 … … (barack obama, sign formal agreement, afghanistan, 2014-02-27) temporal knowledge graph query encoder candidate encoder t t plm plm plm t t contrastive loss prefix (few-shot inductvie) … prefix (few-shot inductvie) … prefix (few-shot inductvie) … prefix (few-shot inductvie) … prefix (transductive) … prefix (transductive) … prefix (transductive) … flashback plm plm plm t t t t tunable frozen figure 1: overall illustration of the chapter model: 1) an example of chapter for the tkg input query (barack obama, sign formal agreement, ?, 2014-02-07) and corresponding candidate afganistan (right); 2) a detailed sketch about the structure and verbalized input of query encoder (left). mantics into structural information for kgc tasks. jin et al. (2023) integrates local structure information into transformer layer text encoding via virtual node tokens. ## experimental setup datasets. we evaluate chapter on tkgr task in both transductive and few-shot inductive settings. for transductive tkgr, we use four widely-used event-based tkg datasets: icews14, icews18, icews05-15 @xcite and icews14* @xcite . for few-shot inductive tkgr, we use three tkg few-shot oog benchmarks proposed in @xcite : icews14-oog, icews18-oog and icews0515-oog. for textual descriptions, existing icews datasets do not provide entity description texts, so we create them by combining corresponding country and sector entries for each entity. detailed dataset statistics are shown in appendix a.1. implementation details. all experiments are carried out on 24g rtx 3090. we adpot adamw optimizer with linear learning rate decay to train chapter. the query encoder and candidate encoder are initialized with parameters of bert-baseduncased. we truncate the description token length up to 50 for entities. the learnable temperature τ is initialized to 0.05 and the additive margin is set to 0.02. we formulate pre-batch negatives n pre from previous 2 batches. for the settings of all baselines, we adopt their default configurations. most of the transductive tkgr results are taken from @xcite and few-shot inductive tkgr results are taken from @xcite . for fairness of comparison, we reimplemented simkgc, kgt5, kgt5-context based on their open source codes to adequately incorporate temporal information. we report the metrics mrr (mean reciprocal rank) and hits@n (proportion of correct entity rank) to evaluate the performance of chapter. we calculate the model results under the time-aware filtered setting @xcite . more detailed implementation settings can be found in appendix a.3, a.4 and a.5. codes are avaliable at this website foot_0 . ## experimental results in this section, we first compare chapter against other competitive baselines in both transductive and few-shot inductive tkg reasoning tasks in sec 5.1. then we conduct ablation study in sec 5.2 to evaluate the effectiveness of each component in chapter. after that, we further analyze the efficiency and flexibility of chapter in sec 5.3. ## conclusion and future work in this paper, we propose chapter, a plm-based pseudo-siamese framework that models balanced textual knowledge and historical information. with the introduced time prefix tokens, chapter is ca-pable for tkg reasoning tasks in various complex scenarios through prefix-based tuning. experimental results on two tkgr tasks demonstrate the superiority of chapter compared to competitive baselines. thorough analysis shows the efficiency and flexibility of chapter. in the future, we would like to explore 1) bridging the gap of two towers with shared time prefix tuning; 2) extending our method to seq2seq plms to model temporal knowledge in a generative manner.