HTCCN : Temporal Causal Convolutional Networks with H awkes Process for Extrapolation Reasoning in Temporal Knowledge Graphs (2024)

Abstract: Temporal knowledge graphs (TKGs) serve as powerful tools for storing and modeling dynamic facts, holding immense potential in anticipating future facts. Since future facts are inherently unknowable, effectively modeling the intricate temporal structure of historical facts becomes paramount for accurate prediction. However, current models often rely heavily on fact recurrence or periodicity, leading to information loss due to prolonged evolutionary processes. Notably, the occurrence of one fact always influences the likelihood of another. To this end, we propose HTCCN, a novel Hawkes process-based temporal causal convolutional network designed for temporal reasoning under extrapolation settings. HTCCN employs a temporal causal convolutional network to model the historical interdependence of facts and leverages Hawkes to model link formation processes inductively in TKGs. Importantly, HTCCN introduces dual-level dynamics to comprehensively capture the temporal evolution of facts. Rigorous experimentation on four real-world datasets underscores the superior performance of HTCCN.

## introduction knowledge graphs (kgs) serve as graphstructured bases for storing human knowledge, promising numerous real-world scenarios in recommender systems @xcite , information retrieval @xcite , and q&a @xcite . typically, kgs express and store knowledge in the form of @xcite . however, facts undergo continuous changes over time, and facts exhibit interactions with one another. to effectively express this dynamic information, researchers construct temporal knowledge graphs (tkgs) en- * corresponding author abling the representation of each fact as a quadruple (subject, relation, object, timestamp). generally, a tkg is represented as a sequence of static kg snapshots, each associated with a timestamp indicating the occurrence of facts at a specific time. the tkg reasoning task involves inferring new facts from known facts and is generally approached in two main settings, i.e. interpolation and extrapolation @xcite . in the interpolation setting, the goal is to recover historically missing facts. conversely, in the extrapolation setting, historical facts are used to predict future facts @xcite . this paper specifically focuses on the extrapolation setting, which poses significant challenges and remains an unsolved problem. the prevailing interpolation reasoning method, as depicted in fig. 1(a) , typically extends static reasoning methods to tkgs. this method involves employing an encoder to derive embeddings of relations and entities within a particular time hyperplane. subsequently, a decoder is used to predict incomplete facts at historical timestamps @xcite . however, relying solely on representations of relations and entities based on historical hyperplanes can lead to the model conflating future facts within the candidate set, thereby impeding its ability to accurately reason about future facts. the key to the extrapolation reasoning task lies in fully understanding historical facts. as illus-trated in fig. 1 (b), many efforts center around extracting essential information from complex timestructured data to facilitate answering queries and making informed judgments @xcite . representative methods such as re-net @xcite and re-gcn @xcite ) utilize r-gcn @xcite to capture structural information from historical snapshots, while rnns are employed to model the temporal information of facts. more recent approaches, including cluster @xcite and tirgn @xcite , reason based on historically observed facts, enabling them to handle repetitive or periodic facts. however, existing rnn-based methods model temporal characteristics of facts as equidistant, which does not accurately reflect real-world fact sequences that often have varying temporal intervals @xcite . additionally, during the inference process, these methods tend to rank the probability scores of all candidate entities across the entire graph, disregarding the crucial aspect of factual evolution @xcite . we contend that fact evolution is highly essential when reasoning about future facts, while the fundamental concept of fact evolution lies in the conditional intensity function, which represents the probability of a fact occurring over time. to address extrapolation reasoning challenges in tkgs, we propose the hawkes process-based temporal causal convolutional network, denoted as htccn. this innovative framework rests upon three fundamental pillars: (1) the adept utilization of a temporal causal convolutional network, empowered by inflated convolutions, which adeptly captures historical fact dependencies and effectively addresses the complex temporal dynamics over diverse time intervals. (2) the seamless integration of hawkes processes, harnessed to inductively model the link formation processes within tkgs, concurrently establishing conditional intensity functions to capture structural and temporal information in tkgs. (3) the incorporation of dual-level dynamics, which comprehensively capture the evolution of facts. in summary, the main contributions of this paper are as follows: ## related work reasoning on tkgs has drawn a lot of attention, mainly including neural network-based methods and query-based methods. neural network-based approaches utilize deep neural networks to learn the underlying features of timestamps for temporal reasoning. re-net @xcite and tirgn @xcite combine recurrent neural networks and neighborhood aggregators to model fact sequences. re-gcn @xcite recursively models kg sequences using recursive evolutionary networks to learn an evolutionary representation of facts for each timestamp. tango @xcite employs neural ordinary differential equations to model the structural information of each candidate entity in a continuous time domain. cen @xcite ) develops an evolutionary model that represents all candidate entities by considering the history of a few recent timestamps. ght @xcite introduces two transformer variants to harness structural and temporal information in tkgs, alongside developing a conditional intensity function for temporal prediction. techs @xcite utilizes a graph convolutional network combined with temporal encoding and heterogeneous attention mechanisms to focus on attention-driven structural features. meanwhile, retia @xcite integrates a relational aggregation module with lstm to model relations and track the temporal progression of hyper-relations. however, these models model the temporal characteristics of facts as isometric, which fail to accurately reflect the dynamic evolutionary patterns of facts in the real world. query-based approaches in temporal reasoning concentrate on modeling entities and relations related to queries. for instance, cygnet @xcite reasons future facts by modeling queryrelated history facts through a copy-generation network. ghnn @xcite introduces a point-in-time process to model temporal information and considers the 1-hop subgraph of query entities. titer @xcite utilizes a temporalaware reinforcement learning strategy to continuously transfer query nodes to new nodes using relevant temporal facts and generate representation vectors of unseen entities using the im module. cluster @xcite combines reinforcement learning and graph convolutional networks for reasoning on tkgs. meanwhile, xerte @xcite addresses the tkg inference problem by using query subgraphs with time-aware neighbor sampling and attention propagation extensions, and additionally introduces a novel backward updating scheme to simulate human reasoning behavior. however, these approaches primarily focus on the history associated with a query, overlooking the complex dynamic interactions between facts. ## problem definition in this paper, we view a tkg as a sequence of timestamp ascension-based subgraphs, namely, based on the historical observed facts in g (1,τ ) , this paper concentrates on reasoning under the extrapolation setting, i.e., predicting potential object entity o or subject entity s via answering queries query (s, r, ?, t) or (?, r, o, t). ## conclusion in this paper, we propose htccn, a reasoning model tailored for tkgs under the extrapolation setting. htccn leverages hawkes processes to effectively model the link formation process inductively within tkgs. specifically, we introduce temporal causal convolutional networks based on hawkes processes, enabling htccn to discern the significance of concurrent facts in reasoning about future facts. additionally, we integrate nodelevel dynamics and fact-level dynamics, aiming to capture both collective features of nodes and individual features of facts, thereby achieving a more comprehensive understanding of the temporal evolution within tkgs. experimental results on four benchmark datasets unequivocally demonstrate the notable advantages and effectiveness of htccn in temporal entity and relationship prediction. furthermore, ablation experiments underscore the active roles played by fact-level dynamics and node-level dynamics in tkg inference.