Temporal Knowledge Graph Completion using a Linear Temporal Regularizer and Multivector Embeddings (2021)

Abstract: Representation learning approaches for knowledge graphs have been mostly designed for static data. However, many knowledge graphs involve evolving data, e.g., the fact (The President of the United States is Barack Obama) is valid only from 2009 to 2017. This introduces important challenges for knowledge representation learning since the knowledge graphs change over time. In this paper, we present a novel time-aware knowledge graph embebdding approach, TeLM, which performs 4th-order tensor factorization of a Temporal knowledge graph using a Linear temporal regularizer and Multivector embeddings. Moreover, we investigate the effect of the temporal dataset’s time granularity on temporal knowledge graph completion. Experimental results demonstrate that our proposed models trained with the linear temporal regularizer achieve the state-of-the-art performances on link prediction over four well-established temporal knowledge graph completion benchmarks.

## introduction numerous large-scale knowledge graphs (kgs) including dbpedia @xcite , free-base @xcite and wordnet @xcite have been established in recent years. such kgs abstract knowledge from the real world into a complex network graph consisting of billions of triples. each triple is denoted as (s, r, o), where s is the subject entity, o is the object entity, and r is the relation between the entities. knowledge graph completion is one of the main challenges in the kg field since most kgs are incomplete. to tackle this problem, knowledge graph embedding (kge) approaches embed entities and relations into a low-dimensional embedding space and measure the plausibility of triples by inputting embeddings of the entities and their relation to a score function @xcite . for instance, complex @xcite has been proven to be a highly effective kge model, where entities and relations are represented as complex embeddings, and the score of a triple (s, r, o) is computed with the asymmetric hermitian dot product. some kgs involve temporal facts, e.g., the triple (barack obama, presidentof, usa) is only valid in a specific time period @xcite . temporal kgs like wikidata @xcite , yago3 @xcite and icews @xcite incorporate time information into triples. triples attached with time information are represented as quadruples, shaped like (s, r, o, t ), where t denotes the timestamp. traditional kge models disregard time information, leading to an ineffectiveness of performing link prediction on tkgs involving temporary relations, e.g., @xcite . recent researches show that the temporal knowledge graph embedding (tkge) models, which encode time information in their embeddings, have better performances on link prediction over tkgs than traditional kge models @xcite @xcite @xcite . in this paper, we present a novel temporal kg embedding approach telm. we move beyond the complex-valued representations and introduce more expressive multivector embeddings from 2grade geometric algebras to model entities, relations, and timestamps for tkge. at a high level, our approach performs 4th-order tensor factorization of a temporal kg, using the asymmetric geometric product. the geometric product provides a greater extent of expressiveness compared to the complex hermitian operator. specially, each relation is represented as a pair of dual multivector embeddings used to handle the beginning and the end of the relation. in this way, telm can adapt well to datasets where time annotations are represented in the various forms: time points, begin or end time, time intervals. moreover, we develop a new linear temporal regularization function for time representation learning which introduces a bias component in the temporal smoothing function and empirically study the effect of the time granularity for a tkg dataset on the performance of our models. experimental results on four well-established tkg datasets show that our approach outperforms the state-of-the-art tkge models, and the linear temporal regularization function improves the performance of our model compared to three common temporal regularization functions. ## related work tensor decomposition-based kge approaches have led to good results in static kg completion. such approaches @xcite @xcite @xcite ) model a static kg as a lowdimensional 3rd-order tensor and consider knowledge graph completion as a tensor decomposition problem. a typical tensor decomposition model complex @xcite has been proven to be fully expressive with complex embeddings. apart from tensor decomposition approaches, distance-based kge models are also commonly used for kg completion. however, distance-based kge models like transe @xcite and its variants @xcite @xcite have been proven to have limitations in modeling various relation patterns which does not lead to the state-of-the-art results on the current benchmarks. the above kge approaches achieve satisfactory results on link prediction over static kgs. recent research on tkg completion shows that the inclusion of time information can improve the performances of kge models on tkgs. ttranse @xcite , hyte @xcite , atise and tero @xcite propose scoring functions which incorporate time representations into a distancebased score function in different ways. further-more, rtge @xcite introduces the concept of temporal smoothness to optimize and learn the hyperplanes of adjacent time intervals jointly on the basis of hyte. @xcite utilize recurrent neural networks to learn timeaware representations of relations and use standard scoring functions from the existing kg embedding model, e.g. transe and distmult. de-simple @xcite uses diachronic entity embeddings to represent entities at different time steps and exploit the same score function as sim-ple to score the plausibility of a quadruple. time-plex @xcite and tcomplex @xcite extend the time-agnostic complex model in different ways. among them, tcom-plex performs a 4th-order tensor decomposition of a tkg using the quadranomial hermitian product which involves the embedding of timestamp t . similarly to rtge, tcomplex also uses the temporal smoothness to improve its performance. thanks to the strong expressiveness provide by the complex embeddings and the 4th-order tensor decomposition, tcomplex achieves state-of-the-art results on tkg completion. ## geometric algebras in this section, we provides a brief introduction to the 2-grade geometric algebra g 2 . the contents are sufficient to understand the rest of the work. members of g 2 are called 2-grade multivectors. the multivector space g 2 is build with vectors from the vector space r 2 . let {e 1 , e 2 } be an orthonormal basis of r 2 . the multivector space g 2 is based on two rules: e 1 e 1 = e 2 e 2 = 1 and e 1 e 2 = -e 2 e 1 . the multivector space g 2 is 4-dimensional with basis: a 2-grade multivector @xmath0. noteworthly, the unit bivectors from g 2 has similar algebraic properties as the imaginary unit i, i.e., thus, the complex numbers @xmath1. in other words, a 2-grade multivector @xmath2. the norm of a multivector @xmath3. taking a 2-grade multivector as an example, its norm is defined as: ||m || = a 2 0 + a 2 1 + a 2 2 + a 2 12 . geometric algebra also introduces a new product geometric product denoted as × n where n is the grade of multivectors, as well as three multivector involutions, space inversion, reversion and clifford conjugation. the geometric product of two 2-grade multivectors comprises of multiplications between scalars, vectors and bivectors. the product of two 2-grade multivectors the clifford conjugation of a 2-grade multivector m is a subsequent composition of space inversion m * and reversion m † as @xmath4, where space inversion m * is obtained by changing e i to -e i and reversion is obtained by reversing the order of all products i.e. changing e 1 e 2 to e 2 e 1 . thus, the clifford conjugation of an 2-grade multivector m @xmath5. note that the product of a multivector m a and its conjugation m 2 is a scalar, i.e., given a 2-grade multivector in our approach, the total number of parameters increases linearly with embedding dimension k, i.e., the space complexity of a telm model is o(k). since the score is computed with an asymmetric quadranomial geometric product between k-dimensional multivector embeddings, the time complexity is also equal to o(k), which are the same as some common kge models, e.g., transe and distmult. ## conclusion we propose a new time-aware approach for tkg completion, telm, which performs 4th-order tensor factorization of a temporal knowledge graph using multivector embeddings for knowledge graph representation and a linear temporal regularizer for learning time embeddings. compared to realvalued and complex-valued embeddings, multivector embeddings provides better generalization capacity and richer expressiveness with higher degree of freedom for tkge. moreover, the linear temporal regularizer provides better geometric meanings for time embeddings and improves the performances of telm compared to the temporal smoothness. additionally, two time division methods are used for different types of tkg datasets to study the effect of the time granularity on tkg completion. our proposed models trained with the linear temporal regularizer achieve the state-of-the-art results on time-wise link prediction over four wellknown datasets involving various forms of time information, e.g., time points, begin or end time, and time intervals. experimental results also show that choosing a reasonable time division method with an appropriate time granularity is helpful for tkg completion.