Graph H awkes Transformer for Extrapolated Reasoning on Temporal Knowledge Graphs (2022)

Abstract: Temporal Knowledge Graph (TKG) reasoning has attracted increasing attention due to its enormous potential value, and the critical issue is how to model the complex temporal structure information effectively. Recent studies use the method of encoding graph snapshots into hidden vector space and then performing heuristic deductions, which perform well on the task of entity prediction. However, these approaches cannot predict when an event will occur and have the following limitations: 1) there are many facts not related to the query that can confuse the model; 2) there exists information forgetting caused by long-term evolutionary processes. To this end, we propose a Graph Hawkes Transformer (GHT) for both TKG entity prediction and time prediction tasks in the future time. In GHT, there are two variants of Transformer, which capture the instantaneous structural information and temporal evolution information, respectively, and a new relational continuous-time encoding function to facilitate feature evolution with the Hawkes process. Extensive experiments on four public datasets demonstrate its superior performance, especially on long-term evolutionary tasks.

## introduction knowledge graph (kg), a multi-relational directed graph database that stores human knowledge and facts, is widely used in downstream applications such as recommendation systems @xcite , web search @xcite and question answering @xcite . conventionally, kgs store each fact in the form of a triplet (𝑠𝑢𝑏 𝑗 𝑒𝑐𝑡, 𝑝𝑟𝑒𝑑𝑖𝑐𝑎𝑡𝑒, 𝑜𝑏 𝑗 𝑒𝑐𝑡). however, many facts may change over time and may contain event-based interaction data. to encode the temporal information, temporal knowledge graph (tkg) is proposed so that each temporal fact is stored as a quadruple (𝑠𝑢𝑏 𝑗 𝑒𝑐𝑡, 𝑝𝑟𝑒𝑑𝑖𝑐𝑎𝑡𝑒, 𝑜𝑏 𝑗 𝑒𝑐𝑡, 𝑡𝑖𝑚𝑒𝑠𝑡𝑎𝑚 𝑝). typically, we represent tkg as a sequence of static kg snapshots associated with timestamps. there are two types of reasoning for tkg. interpolated reasoning is designed to complete the missing facts on known historical snapshots, while extrapolated reasoning predicts future events or facts. this work focuses on extrapolated reasoning, a problematic but meaningful task, which can be used for crisis warning, user behavior prediction, supply chain management, etc. while most existing works @xcite perform link predictions on snapshots at the next timestamp, we consider predicting events that may occur over a long period in the future, which are more appropriate in the real-world setting. the task requires the assistance of known historical facts. a key feature required by the model is to retrieve key information from complex temporal structured data that can help answer the query and make correct judgments. recent methods of re-net @xcite and re-gcn @xcite ) employ r-gcn @xcite to capture structural information from historical snapshots and then use recurrent neural networks (rnns) to model the latent vector sequences. while the two methods work well on entity prediction task, they also meet some limitations. there are some early methods @xcite that could do both tasks simultaneously by using a temporal point process to represent the evolution of facts, with the key concept of conditional intensity function denoting the con-ditional probability of an event occurring over a period. nevertheless, these methods could not perform well on the entity prediction task as they did not capture the structural information from the graph snapshots. on the other hand, transformer @xcite has recently been widely used in various fields @xcite owing to its powerful modeling capabilities. however, to our knowledge, there exists no work that uses transformer or its variants to solve the tkg evolution problem. in this work, we propose a new model termed graph hawkes transformer (ght), which introduces transformer into the tkg evolution modeling and further integrates the neural temporal point process. specifically, we design two transformer variants to capture the structural and temporal information in tkgs by building conditional intensity function. one variant is used to aggregate multirelational graphs, capturing structural information for each timestamp and generating feature vectors. the model can learn which interactions are more critical in query with the attention mechanism. the other variant captures temporal information based on a sequence of feature vectors and simultaneously outputs a hidden state for each timestamp in the future. finally, the model uses the hidden state to calculate the conditional intensity and then gets the candidate entity score or the time probability density of the next event. our main contributions are as follows: not only can it simultaneously predict events of multiple timestamps in the future, but it can also predict when an event will occur. to our knowledge, this is the first transformer-based temporal point process model for the tkg evolutionary representation learning. ## the proposed model our method first obtains concurrent structural information from subgraphs and generates feature vectors for each source node. then it captures the temporal evolution information from the vector sequences and finally outputs the hidden state vectors to participate in the construction of the conditional strength function to complete the prediction task. compared with previous evolutionary representation learning methods, e.g., re-net @xcite and re-gcn @xcite , we expect the model to make up for the four limitations introduced in section 1. figure 1 illustrates the overall architecture of our model. ## conclusion we propose a new tkg reasoning model, called graph hawkes transformer (ght), a neural temporal point process model based on transformer. we first analyze four limitations of the previous stateof-the-art methods, re-gcn @xcite and re-net @xcite . then we design two transformer blocks, which are used to capture the structural and temporal information, respectively. based on hawkes process, the model can learn a conditional intensity function to solve the above issues with the attention mechanism and the proposed relational continuous-time encoding function. moreover, most previous works only evaluate their methods for entity prediction at the next future timestamp, while we evaluate the models comprehensively by setting different forecasting time window sizes. experimental results on four popular datasets demonstrate the superior performance of our model on both entity prediction and time prediction tasks. notably, our model performs much better under long-term evolution scenarios.