Learning Latent Relations for Temporal Knowledge Graph Reasoning (2023)

Abstract: Temporal Knowledge Graph (TKG) reasoning aims to predict future facts based on historical data. However, due to the limitations in construction tools and data sources, many important associations between entities may be omitted in TKG. We refer to these missing associations as latent relations. Most existing methods have some drawbacks in explicitly capturing intra-time latent relations between co-occurring entities and inter-time latent relations between entities that appear at different times. To tackle these problems, we propose a novel Latent relations Learning method for TKG reasoning, namely L2TKG. Specifically, we first utilize a Structural Encoder (SE) to obtain representations of entities at each timestamp. We then design a Latent Relations Learning (LRL) module to mine and exploit the intra- and inter-time latent relations. Finally, we extract the temporal representations from the output of SE and LRL for entity prediction. Extensive experiments on four datasets demonstrate the effectiveness of L2TKG.

## introduction temporal knowledge graphs (tkgs) play a vital role in capturing temporal facts in the real world. each fact in a tkg is represented as a quadruple (s, r, o, t), such as @xcite . reasoning over tkgs involves two primary settings: interpolation and extrapolation. in recent years, there has been significant interest in the extrapolation setting due to its practical value in event prediction @xcite , question answering @xcite , and other applications. in the extrapolation setting, the objective is to predict facts that occur at a time t with @xmath0, based on the historical information available in the tkg from t 0 to t n . * to whom correspondence should be addressed. most extrapolation models utilize the temporal and structural information available in the tkg for reasoning. for example, re-net @xcite and re-gcn @xcite incorporate recurrent neural networks and graph neural networks to capture the temporal and structural dependencies of historical tkg sequences. additionally, xerte @xcite and titer @xcite develop sub-graph search and path search strategies to predict target entities based on existing tkg structures, respectively. while these methods demonstrate promising results in tkg reasoning, they still face the challenge of missing associations within tkgs. specifically, the majority of tkg data is automatically identified and extracted from diverse news articles, such as icews data @xcite . many crucial associations between entities may be omitted from tkgs due to the limitations of construction tools and data sources. we refer to these missing associations as latent relations between entities. existing approaches fail to explicitly discover and utilize these latent relations, which manifest primarily in two aspects. firstly, existing methods fail to explicitly capture intra-time latent relations between co-occurring en-tities. during tkg reasoning, certain concurrent entities may lack direct connections but exhibit strong semantic correlations. figure 1 illustrates this phenomenon, where afghanistan and taliban are not connected in the tkg for may 2021. however, in reality, the taliban was involved in negotiations with afghanistan during that time, significantly impacting the situation in afghanistan. hence, it is crucial to model the critical latent relations among concurrent entities. most existing tkg reasoning models rely on relational graph neural networks (rgnns) @xcite to capture the semantic dependencies between entities at each timestamp. however, rgnns heavily depend on existing edges or associations, making it challenging to model critical semantic dependencies among indirectly connected entities. secondly, existing methods ignore the inter-time latent relations between entities appearing at different timestamps. some entities at various timestamps can exhibit strong semantic dependencies, providing essential auxiliary information for tkg reasoning. therefore, it is necessary to consider the associations between these entities. taking figure 1 as an example, the impact of the usa in may 2021 on afghanistan in august 2021 is significant. however, as these two entities appear at different times, they cannot be directly related in the tkg. existing tkg reasoning models primarily focus on modeling the semantic dependencies of the same entities at different times but fall short when addressing entities at distinct timestamps. to address the aforementioned challenges, we propose a novel latent relations learning method for tkg reasoning, l 2 tkg for brevity. the overall framework of l 2 tkg is presented in figure 2 . specifically, we first employ a structural encoder (se) to generate the representations of entities at each timestamp. inspired by graph structural learning @xcite @xcite , we design a latent relations learning module (lrl) for learning the two types of missing associations in tkg reasoning. utilizing the embeddings of entities at each timestamp, lrl enables the creation of new crucial associations between unconnected entities in a learnable manner and then encodes the learned latent relational graph to obtain more comprehensive representations of entities. finally, we extract temporal representations from the output of se and lrl components for the entity prediction task. in summary, our work makes the following main contributions: ## related work in this paper, we illustrate the related work about tkg reasoning under the extrapolation setting and graph structure learning. ## preliminaries in this section, we introduce the definition of tkg, formulate the task of tkg reasoning, and explain some notations used in this paper. definition 1 (temporal knowledge graph). let e and r represent a set of entities and relations. a quadruple q @xmath1, r, e o , t) represents a relation @xmath2. all quadruple occurring at time t constitute a knowledge graph g t . e t @xmath3. a temporal knowledge graph (tkg) g is defined as a sequence of knowledge graphs with different timestamps, i.e. definition 2 (temporal knowledge graph reasoning). this paper primarily emphasizes the entity prediction task within tkg reasoning. the objective of the entity prediction task is to forecast the missing object entity of (e s , r, ?, t + 1) or the missing subject entity of (?, r, e o , t + 1) based on the historical kg sequence let x @xmath4, where d represents the dimension. the general paradigm of tkg reasoning is to learn future representations of each entity for predicting g t+1 by using the historical kg sequences {g i } t @xmath5, along with static entity and relation embeddings x s and x r . the embeddings x s and x r serve as learnable parameters. ## methodology in this section, we present the proposed l 2 tkg. the overall framework of l 2 tkg is illustrated in figure 2 . there are three main components: (1) structural encoder (se), which captures the semantic dependencies among concurrent entities at each timestamp using the existing tkg structure. ## experiments in this section, we perform experiments on four tkg datasets to evaluate our model. we aim to answer the following questions through experiments. how does l 2 tkg perform compared with state-of-the-art tkg reasoning methods on the entity prediction task? • q2: how does l 2 tkg perform in learning missing associations? • q3: how do different components affect the l 2 tkg performance? • q4: how sensitive is l 2 tkg with different hyper-parameter settings? 5.1 experimental setup 5.1.1 datasets we evaluate our l 2 tkg on four representative tkg datasets in our experiments: icews14 (garcía-durán et al., 2018), icews18 (jin et al., 2020a), icews05-15 (garcía-durán et al., 2018), and gdelt @xcite . the first three datasets are from the integrated crisis early warning system @xcite and record the facts in 2014, 2018, and the facts from 2005 to 2015, respectively. the last one is from the global database of events, language, and tone @xcite . the details of data split strategy and data statistics are shown in appendix a. ## sensitivity analysis (rq4) the structural encoder (se) and latent relation learning (lrl) are two vital modules in our model. this section studies how hyper-parameters, the k value of the sparse operations (intra-time and intertime learning), and the layer numbers of lrl and se affect the performance of l 2 tkg. ## conclusion in this paper, we have proposed a novel method l 2 tkg for reasoning over tkg. we first obtain the embedding of each historical entity based on the structural encoder. then, a well-designed latent relations learning module is proposed to mine and encode the two types of latent relations, obtaining comprehensive entity embeddings. finally, we extract temporal representations of entities from the outputs of lrl and se for final prediction. experimental results on four benchmarks and extensive analysis demonstrate the effectiveness and superiority of l 2 tkg in tkg reasoning.