SALMON : A Structure-Aware Language Model with logicality and densification strategy for Temporal Knowledge Graph Reasoning (2024)

Abstract: Temporal knowledge graph reasoning (TKGR) is a crucial task that involves reasoning at known timestamps to complete the future facts and has attracted more and more attention in recent years. The current TKGR models are mainly based on graph neural networks or tensor decomposition techniques. Few works in TKGR focus on pre-trained language models (PLMs) which have powerful sequence modeling capabilities to capture the temporal associations between facts. In this paper, we propose a model SALMON: a Structure-Aware Language Model with logicality and densification strategy. Specifically, we design a PLM-based framework with a structure-aware layer inside to jointly capture the temporal evolving pattern and structural information in TKGs. To further enhance the model’s ability to infer causal associations of facts, we propose a logical judging module, which can guide the model to prioritize learning the most relevant evolving information of logical causal associations in TKGs during the training process. Moreover, we propose a densification strategy based on large language models, through a carefully crafted Chain of Thought prompt, to dig out some knowledge necessary for reasoning about fact associations, thereby making the model perform better. Extensive experimental results demonstrate the superiority of our model over the state-of-the-art baselines.

## introduction knowledge graphs (kgs) @xcite represent relations among real-world entities, enabling a broad spectrum of applications in natural language processing tasks. as knowledge evolves over time, timestamps are incorporated into kgs, giving rise to temporal kgs (tkgs) (cai et al., 1 code is available at https://github.com/linjh1118/ salmon . moreover, in order to further grasp the causal associations among events, another type of approaches based on rule such as xerte @xcite , tlogic @xcite have also been proposed. these methods mainly walk on tkgs through rules (the walking paths constitute a reasoning chain), and then reach the final entity to achieve tkgr. although the rule-based method can provide explainable reasoning, this walking method has relatively high requirements on the integrity of the reasoning chain in tkgs. when a key clue event related to the entity to be predicted is missing, the reasoning may be interrupted. in addition, with the excellent performance of language models (e.g., @xcite , gpt @xcite ) in natural language processing tasks, tkgr based on language models has gradually attracted attention. the earlier work kg-bert @xcite and lass @xcite attempt to leverage pre-trained language models (plms) to address traditional kg reasoning task by treating triples as textual sequences. the recent work ppt @xcite utilizes a plm with prompts to integrate temporal kg information into language models for achieving tkgr. due to the sparsity of tkgs, many of the facts are relatively independent and lack historical information related to them as shown in figure 1 . this limits the ability of ppt to fully explore the deeper reasons for associations among facts to a certain extent, resulting in limited performance. the potential of language models in tkgr task still has a great space to explore. to address these issues, in this paper we propose a novel model, salmon (structure-aware language model with logicality and densification strategy), which leverages the strengths of plms to enhance tkgr. built on top of the plm-based framework, salmon includes a dedicated structure-aware layer to jointly capture the evolving patterns and structural information in a tkg, leading to a more comprehensive understanding of the tkg. further, to enhance the model's reasoning capabilities, we design a logical judging module. instead of traditional rules-based walking, our logical judgment module aims to give priority to learning the most relevant evolving information of logical causal associations in the tkg during the training process of plms. to alleviate the sparsity of tkgs mentioned above, inspired by the potential of recent large language models (llms), we propose a densification strategy that harnesses the power of llms through a carefully designed prompt, to enhance the tkgr performance by digging out some evidence necessary for reasoning about associations among events. our contributions can be summarized as follows: ## problem formulation a temporal knowledge graph (tkg) is an assembly of facts, symbolized by g ⊂ e × r × e × t , where e, r and t denote the finite sets of entities, relations, and timestamps, respectively. a quadruple (s, r, o, t) describes that a fact of relation type @xmath0. boldfaced s, r, o, t represent their embeddings. the goal of the tkg reasoning (tkgr) task is to predict the missing object entity o via answering query like @xmath1, r q , ?, t q ) with only historical known facts {(s, r, o, t i )|t @xmath2. not that, without loss of generality, when predicting the missing subject of a query @xmath3, r q , o q , t q ), we can convert the query into @xmath4, r -1 q , ?, t q ). ## conclusion in this paper, we propose a novelty tkgr model salmon. specifically, we design a plm-based framework with a structure-aware layer inside to jointly capture evolving patterns and structural information in tkgs. moreover, we propose a logical judging module to give priority to learning the most relevant evolving information in tkgs during the training process. furthermore, we propose a llms-based densification strategy to alleviate the difficulty of the model in making accurate inferences about sparse events in tkgs. results on tkgr benchmarks demonstrate the effectiveness of our method.