Temporal Knowledge Graph Completion with Approximated G aussian Process Embedding (2022)

Abstract: Knowledge Graphs (KGs) stores world knowledge that benefits various reasoning-based applications. Due to their incompleteness, a fundamental task for KGs, which is known as Knowledge Graph Completion (KGC), is to perform link prediction and infer new facts based on the known facts. Recently, link prediction on the temporal KGs becomes an active research topic. Numerous Temporal Knowledge Graph Completion (TKGC) methods have been proposed by mapping the entities and relations in TKG to the high-dimensional representations. However, most existing TKGC methods are mainly based on deterministic vector embeddings, which are not flexible and expressive enough. In this paper, we propose a novel TKGC method, TKGC-AGP, by mapping the entities and relations in TKG to the approximations of multivariate Gaussian processes (MGPs). Equipped with the flexibility and capacity of MGP, the global trends as well as the local fluctuations in the TKGs can be simultaneously modeled. Moreover, the temporal uncertainties can be also captured with the kernel function and the covariance matrix of MGP. Moreover, a first-order Markov assumption-based training algorithm is proposed to effective optimize the proposed method. Experimental results show the effectiveness of the proposed approach on two real-world benchmark datasets compared with some state-of-the-art TKGC methods.

## introduction knowledge graphs (kgs) provide an efficient way to store world knowledge. various kgs such as dbpedia @xcite , nell @xcite , @xcite and freebase @xcite have been constructed and benefited downstream applications such as information retrieval, question answering, etc @xcite . generally, a fact in kg can be represented as a triple (s, p, o), where s (subject) denotes head entity node, o (object) denotes tail entity node and p (predicate) denotes relation edge between them. however, in the real world, some facts are time-aware. for example, the fact (joe biden, presidentof, the united states) is not valid until january 21, 2021, the united states presidential inauguration of joe biden. therefore, some kgs store time-aware facts or events as the quadruple (s, p, o, t), where t is the timestamp. such kgs are referred as temporal knowledge graphs (tkgs), which mainly include yago3 @xcite , gdelt @xcite , wikidata @xcite and icews @xcite . temporal knowledge graph completion (tkgc), aiming at inferring the missing edges based on known facts, is a fundamental task for the incomplete real-world tkgs. a large class methods perform tkgc by mapping nodes and edges in tkgs into high-dimensional latent feature spaces while preserving the semantic and structural information as much as possible. in recent years, extensive research have been conducted on embedding-based tkgc with notable process, which is also known as temporal knowledge graph embeddings (tkges). early works extend the translation-based knowledge graph embedding (kge) approaches by mapping each timestamp into a specific time embedding @xcite . obviously, such approaches cannot deal with the unseen timestamps. to tackle this problem, some researchers model the entity and relation embeddings in tkge as the continuous functions of time @xcite , which can be categorised as deterministic-based approaches. recently, some researchers model tkge as multivariate gaussian density embeddings @xcite , which can be considered as density-based methods. although the notable progresses have been made, most of existing tkge methods still suffer from the following two disadvantages. (1) most of them are not flexible enough, the entity and relation embeddings are usually learned as the deterministic function of time, which is good at capturing the global trend while failing to model the surging local fluctuation. for example, as shown in figure 1 , the semantic meaning of obama should have a violent fluctuation on 2008 -11 -04 as he won the presidential election, which is hard to model solely based on the deterministic function of time. the above disadvantages could be naturally tackled by the stochastic process-based method. on the one hand, a stochastic process-based method is flexible to deal with the local fluctuations by modeling the correlations of embeddings at neighboured timestamps. on the other hand, a stochastic process-based method is expressive to model the dynamic changes of temporal uncertainties with the covariance matrix and kernel function. therefore, in this paper, we propose a novel method to learn flexible and expressive temporal knowledge graph embeddings based on approximated multivariate gaussian processes (tkgc-agp). in specific, each entity and relation in tkgs are mapped into a specific multivariate gaussian process. the evolution dynamics of the entities and relations are modeled using the mean function of multivariate gaussian process. the temporal correlation for each entity/relation are captured by the kernel function. the temporal uncertainty of the entities and relations is modeled by the entity/relation-specific covariance matrix. further more, a first-order markov assumption based algorithm is proposed to approximate the likelihood of multivariate gaussian process. to investigate the effectiveness of the proposed approach, extensive experiments have been conducted on two largescale tkg datasets. experimental results show the effectiveness of the proposed approach compared to various competitive baselines. in general, our contributions are listed as follows. our work is mainly related to two lines of research, described as follows. ## method in this section, we will discuss the details about the proposed method. we will start from the background knowledge of multivariate gaussian process. then we will talk about how to construct the entity and relation embeddings based on multivariate gaussian process. finally, the training and inference process of the proposed approach is explained. ## conclusion in this paper, we proposed tkgc-agp, a novel temporal knowledge graph completion method based on approximated gaussian process embeddings. with the flexibility and capacity, we can naturally model the global trends of entity and relation embeddings as well as the surging local fluctuations. moreover, the temporal uncertainties can be also naturally modeled with the kernel function and corvariance matrix of mgp. to training tkgc-agp, we employ the fisrt-order markov assumption to approximate the joint distribution of mgp as well as a time-batch-based training strategy to align the embeddings across the time. the experimental results demonstrate that the proposed method outperform various static and temporal kge baselines. further work could be done by taking a bayesian perspective to learn the proposed method.