Pre-trained Language Model with Prompts for Temporal Knowledge Graph Completion (2023)

Abstract: Temporal Knowledge graph completion (TKGC) is a crucial task that involves reasoning at known timestamps to complete the missing part of facts and has attracted more and more attention in recent years. Most existing methods focus on learning representations based on graph neural networks while inaccurately extracting information from timestamps and insufficiently utilizing the implied information in relations. To address these problems, we propose a novel TKGC model, namely Pre-trained Language Model with Prompts for TKGC (PPT). We convert a series of sampled quadruples into pre-trained language model inputs and convert intervals between timestamps into different prompts to make coherent sentences with implicit semantic information. We train our model with a masking strategy to convert TKGC task into a masked token prediction task, which can leverage the semantic information in pre-trained language models. Experiments on three benchmark datasets and extensive analysis demonstrate that our model has great competitiveness compared to other models with four metrics. Our model can effectively incorporate information from temporal knowledge graphs into the language models.

## introduction in recent years, temporal knowledge graphs(tkgs) have attracted much attention. tkgs describe each fact in quadruple @xcite . compared to static knowledge graphs, tkgs need to consider the impact of timestamps on events. for example, (donald trump, presidentof, america, 2018) holds while (donald trump, presi-dentof, america, 2022) does not. there are missing entities or relations in the tkgs, therefore, barack obama françois hollande usa discuss by telephone consult sergey viktorovich lavrov on the next day john kerry benjamin netanyahu discuss by telephone consult on the next day ## preliminary temporal knowledge graph g is a set of networks of entities and relations that contain timestamps. it can be defined as @xmath0, r, t , q}, where e is the set of entities, r is the set of relations and t is the set of timestamps. @xmath1, r, o, t)} ⊆ e × r × e × t is the quadruple set, where s and o are the subject entity (head entity) and object entity (tail entity), r is the relation between them at timestamp t. g @xmath2, r, o) ⊆ e × r × e} is called the tkg snapshot at t, and it can be taken as a static kg filtering the triple set from g at t. temporal knowledge graph completion (tkgc) is the task of predicting the evolution of future kgs given kgs of a known period. given a quadruple (s, r, ?, t n ) or (?, r, o, t n ), we have a set of known facts from tkg snapshots g (t @xmath3. the probability of prediction of missing the entity o in quadruple (s, r, ?, t n ) can be formalized as follows: ## methodology in this paper, we propose ppt, a novel plm-based model with prompts to solve tkgc task. the framework of our model is illustrated in figure 2 . we sample quadruples and convert them into pretrained language model inputs. the prediction of [mask] token is the completed result. ## conclusions this paper proposes a novel temporal knowledge graph completion model named pre-trained language model with prompts for tkgc (ppt). we use prompts to convert entities, relations, and timestamps into pre-trained model inputs and turn tkgc problem into a masked token prediction problem. this way, we can extract temporal information from timestamps accurately and sufficiently utilize implied information in relations. our proposed method achieves promising results compared to other temporal graph representation learning methods on three benchmark tkg datasets. for future work, we plan to improve the sampling method in temporal knowledge graphs to get more timespecific inputs. we are also interested in combining gnns and pre-trained language models in temporal knowledge graph representation learning.