Natural Evolution-based Dual-Level Aggregation for Temporal Knowledge Graph Reasoning (2024)

Abstract: Temporal knowledge graph (TKG) reasoning aims to predict missing facts based on a given history. Most of the existing methods unifiedly model the evolution process of different events and ignore their inherent asynchronous characteristics, resulting in suboptimal performance. To tackle this challenge, we propose a Natural Evolution-based Dual-level Aggregation framework (NEDA) for TKG reasoning. Specifically, we design a natural division strategy to group TKGs into different patches according to the occurrence of a given target entity. Then, we present a dual-level aggregation scheme to extract local representations from information within patches and then aggregate these representations with adaptive weights as the final entity representations. By assigning varying weights to different patches, this aggregation scheme can incorporate the asynchronous characteristics of event evolution for representation computation, thus enhancing prediction performance. Extensive experiments demonstrate the significant improvement of our proposed model.

## introduction temporal knowledge graphs (tkgs) are essential for effectively capturing temporal facts and hold remarkable values in diverse real-world applications, such as information retrieval @xcite , recommendation systems @xcite , and question answering @xcite . each fact in tkgs is represented as a quadruple (s, r, o, t), such as @xcite . due to the significant practical value, reasoning over tkgs has garnered multidisciplinary research interests @xcite @xcite . reasoning over tkgs primarily has two settings: interpolation and extrapolation. the former intends * corresponding author: chunjingxiao@gmail.com to infer missing facts that happened in the past. while the extrapolation attempts to predict facts that are likely to occur in the future. in this paper, we focus on predicting future facts (i.e., extrapolation). given a tkg from timestamp t 0 to t n , we aim to predict future facts at timestamp t with @xmath0, i.e., answer the queries such as (lebron james, play for, ?, 2024). to answer such queries, existing solutions @xcite @xcite have attempted to learn temporal evolutionary information by modeling structural dependencies and sequential patterns. however, they unifiedly model the evolution process of different events, ignoring evolution discrepancies (i.e., asynchronous characteristics) of events. whereas, in the real world, event evolution is typically asynchronous. different events often exhibit distinct life cycles (i.e., starting and ending time) and evolution types (i.e., continuous and discontinuous) @xcite @xcite . the unified processing schemes may incorporate irrelevant information into the reasoning process or overlook critical information, leading to inferior performance. considering asynchronous evolution for tkg reasoning is non-trivial and faces two main challenges: (1) for a given query on tkgs, identifying the correlation between the query and each snapshot is challenging. as an event typically evolves across multiple snapshots, each individual snapshot contains only limited topic information. this makes determining their relevance a complex task. however, to capture asynchronous evolution, it is vital to ascertain their correlation and further eliminate unrelated snapshots (e.g., those before the starting time) to accurately answer the query. (2) current approaches cannot accurately model the evolution of events with long life-cycles, as they fail to efficiently capture long-term historical information. these methods primarily employ gcns and grus to model event evolution, which limits them to capturing only a short history window to model structural dependencies and sequential patterns @xcite . however, short-range historical information is insufficient to encompass the entire evolution process of events with long life-cycles @xcite . therefore, capturing long-range historical information is radical for effectively modeling the evolution patterns of events with varying life-cycles. to address these issues, we propose a natural evolution-based dual-level aggregation framework (neda) for tkg reasoning, which can efficiently capture the asynchronous evolution characteristics of events to boost reasoning performance. this framework primarily involves the patch construction, which aims to group tkgs into different patches, as well as the first-level aggregation (i.e., intra-patch encoder) and the second-level aggregation (i.e., inter-patch attention), both of which intend to compute entity representations by combining the intra-patch and inter-patch information with adaptive weights. to summarize, the major contributions can be listed as follows: ## preliminaries in this section, we introduce the essential background of tkgs and formally formulate the task of tkg reasoning under the extrapolation setting. ## experiments in this section, we perform experiments on six tkg datasets to evaluate the performance of our neda. we aim to answer the following questions through experiments. how does neda perform compared with the state-of-the-art baselines? • q2: effectiveness. how do the different components affect the neda's performance? • q3: scalability. how does neda perform when there is limited training data available? • q4: sensitivity. how does the performance of neda fluctuate with different hyperparameters? • q5: asynchrony. how does neda capture the inherent asynchronous evolution characteristics of events? 4.1 basic setting datasets & evaluation metrics. we conduct experiments on four commonly used tkg datasets including icews14 (garcia-duran et al., 2018), icews18 (jin et al., 2020), icews05-15 (garcia-duran et al., 2018), and gdlet @xcite , as well as two benchmark datasets we newly collected, icews22 and icews23. the two datasets are obtained from the icews database foot_0 and sorted out using the typical processing method in the work @xcite . ## related work depending on the timing of predicted events, tkg reasoning models can be divided into two settings: interpolation and extrapolation @xcite . tkg reasoning under interpolation. the task in the interpolation setting aims to infer missing facts from the past @xcite @xcite . for example, ttranse @xcite integrates temporal information by treating both relations and time as translations of entities, whereas @xcite associates each timestamp with specific hyperplanes. chronor @xcite combines relation and temporal embeddings to create a comprehensive rotational embedding, which is then applied to the final entity representation. tilp @xcite introduces a constrained random walk mechanism and integrates an array of temporal operators into the learning framework. tkg reasoning under extrapolation. the extrapolation methods try to predict future events @xcite @xcite . following the work @xcite , these approaches can be divided into two main categories according to the historical structural information used for event prediction: candidate-based and query-based methods. candidate-based methods try to utilize the historical facts of all candidate entities without considering the query during the encoding process. the query information is incorporated only in the decoding stage @xcite @xcite . for instance, re-gcn @xcite learns the evolutionary representations for entities and relations by incorporating the gate recurrent unit to extract sequential patterns and taking externally injected static entity attributes into consideration. tirgn @xcite learns sequential, repetitive, and periodic historical facts from local and global perspectives, respectively. l 2 tkg @xcite exploits the latent relation between entities and learns latent relational graphs based on similarities between entities. query-based methods focus on modeling the query-specific historical dependencies for event prediction @xcite @xcite . among them, cygnet @xcite calculates the occurrence frequencies of events to narrow the candidate entities when making predictions, while cenet @xcite utilizes contrastive learning to identify whether an event has occurred before. da-net @xcite tries to capture distributed attention on repeated facts across various historical timestamps to better understand future events. in addition, some recent approaches attempt to utilize the powerful reasoning capabilities of llms for tkg reasoning @xcite . for example, gentkg @xcite formulates a generative forecasting framework with llms, and icl @xcite uses in-context learning to the potential of llms for tkg reasoning.