Temporal Knowledge Graph Reasoning with Low-rank and Model-agnostic Representations (2022)

Abstract: Temporal knowledge graph completion (TKGC) has become a popular approach for reasoning over the event and temporal knowledge graphs, targeting the completion of knowledge with accurate but missing information. In this context, tensor decomposition has successfully modeled interactions between entities and relations. Their effectiveness in static knowledge graph completion motivates us to introduce Time-LowFER, a family of parameter-efficient and time-aware extensions of the low-rank tensor factorization model LowFER. Noting several limitations in current approaches to represent time, we propose a cycle-aware time-encoding scheme for time features, which is model-agnostic and offers a more generalized representation of time. We implement our methods in a unified temporal knowledge graph embedding framework, focusing on time-sensitive data processing. The experiments show that our proposed methods perform on par or better than the state-of-the-art semantic matching models on two benchmarks.

## introduction knowledge graphs offer promising technologies to structure and organize common-sense and domainspecific knowledge and form the information basis for many anticipated technological foundations. their importance is signified by downstream applications, including speech recognition, sentiment analysis, and knowledge base question answering @xcite . in this context, event and temporal knowledge graphs prove worthy successors of static knowledge graphs, targeting the augmentation of static relational data with temporal meta information. fig. 1 presents a temporal sub-graph from wikidata @xcite , where we are interested in answering the question: who was the president of the u.s. in 1961? figure 1 : time-sensitive relationships of u.s. presidents with the democratic and republican parties along with their timestamps. temporal knowledge graph completion (tkgc) aim to perform time-aware reasoning over the kg to answer questions of type (?, is-a, @xcite ) with an answer j. f. kennedy. temporal information opens new opportunities for many time-sensitive domains, including timeseries forecasting, biomedical event extraction, and time-sensitive crime reconstruction. however, temporal knowledge graphs show many inconsistencies and lack data quality across different dimensions, including the accuracy, completeness, and timeliness of facts. the quality of evolving knowledge constitutes a challenging task due to the volatile nature of knowledge. to approach the problem of both completeness and correctness in temporal knowledge graphs, this work addresses the task of temporal link prediction and introduces time-aware extensions of the parameter efficient and expressive static embedding model lowfer @xcite . more precisely, we formulate the main contributions of this paper as follows: tkgc approaches are divided into (a) geometric embedding models using distance-based scoring functions, (b) semantic matching models using similarity-based scoring functions, and (c) deep learning models. in this work, we focus on the group of semantic matching models, commonly referred to as factorization-based models. the most prominent static models in this area are dist-mult @xcite , simple (kazemi and poole, 2018), complex @xcite and tucker @xcite . further, noting characteristic patterns throughout several timeaware extensions of static embedding models, we identified four distinct temporal extension themes fig. 2(b): (1) inclusion-based, (2) feature-based, (3) regularization-based, and (4) aggregation-based extensions. ## low-rank representation tensor factorization models decompose the order-3 and order-4 binary tensor into a compressed tensor and a set of factor matrices, respectively, for static and temporal kgs. tucker @xcite proposed a tucker decomposition model for static kgs and showed that existing semantic matching models could be subsumed in their formulation. noting the cubic growth of core tensor in tucker, lowfer @xcite proposed an efficient parameter initialization of the core tensor using low-rank factorized bilinear pooling. due to its ability to handle arbitrary relations (fully expressive), parameter efficiency, generalization abilities, and state-of-the-art performance in embedding-based models for static kgs @xcite , we extend it to the temporal kgs. ## cycle-aware time embedding in this section, we propose a novel extension technique for embedding time features (e t ), which relies on multi-recurrent cycle-aware (mrca) time decomposition and is model-agnostic. we first explain the concept of multi-recurrence and then show its application for cycle-aware encoding of time features. temporal recurrence denotes the concept of expressing time as recurrent component within a certain time frame e.g., a week occurs approximately four times per month or equally a year contains four seasons. in this context, we speak of multirecurrence, if a time frame is expressed in terms of multiple cycles, e.g., a year entails four seasons, 12 months, 52 weeks and 365 days. multirecurrence is expressible for all time concepts with one or more underlying recurrent cycles. even more, (long-term) cycles themselves are also expressible in terms of more fine-grained (short-term) cycles. our multi-recurrent cycle-aware (mrca) encoding uses a mapping ψ cyc of timestamps t to a set of recurrence encodings c m×l : and each time component c k p with 1 ≤ @xmath0: here, n denotes the number of timestamps, m is the number of time windows, and l is the number of recurrent subcycles. now, to generate the cycleaware multi-recurrent time embedding, we consider the following five components and their respective cycle decompositions foot_0 : where w, m, s, y, g denote the weekly, monthly, seasonal, yearly, and global components, respectively, and d, w, m, s, y denote the daily, weekly, monthly, seasonal, and yearly subcycles. the components c 1 , c 10 , c 100 , c 1000 denote the positional year component for single years, decades, centuries and milleniums, respectively. the final cycle-aware time encoding ψ m rca is then generated by summing over all cycle decompositions of each separate recurrence mapping φ rec w k : icews14 is a subset of the icews database, including facts from the year 2014. it consists of 7,129 distinct entities, 230 relations, and 365 timestamps with 24 hours (daily) temporal granularity. icews05-15 is another subset of the icews database, including facts between the start of 2005 and the end of 2015. it consists of 10,488 distinct entities, 251 relations, and 4,017 timestamps with a temporal granularity of 24 hours (daily). in ## conclusion in this work, we introduced time-lowfer, a family of time-aware extensions of the bilinear factorization model lowfer. following existing work in temporal link prediction, we extended lowfer using time-modulated relations (tnt). further, noting several limitations of modulation-based extensions, we proposed two feature-based extensions of lowfer, which are based on bilinear chaining (cfb) and trilinear fusion (ftp). in particular, we showed that the ftp represents a parameterefficient specialization of the cfb, while cfb offers state-of-the-art results among semantic matching models for temporal link prediction. in addition, we investigated four different approaches for time-aware extension of static embedding models and outlined, despite the increased popularity of time modulation techniques, the superiority of feature-based kge extensions. furthermore, we investigated the process of time encoding in representation learning and proposed a modelagnostic method (cte) for encoding timestamps based on multi-recurrent cycle-aware (mrca) time decomposition.