Tucker Decomposition with Frequency Attention for Temporal Knowledge Graph Completion (2023)

Abstract: Temporal Knowledge Graph Completion aims to complete missing entities or relations under temporal constraints. Previous tensor decomposition-based models for TKGC only independently consider the combination of one single relation with one single timestamp, ignoring the global nature of the embedding. We propose a Frequency Attention (FA) model to capture the global temporal dependencies between one relation and the entire timestamp. Specifically, we use Discrete Cosine Transform (DCT) to capture the frequency of the timestamp embedding and further compute the frequency attention weight to scale embedding. Meanwhile, the previous temporal tucker decomposition method uses a simple norm regularization to constrain the core tensor, which limits the optimization performance. Thus, we propose Orthogonal Regularization (OR) variants for the core tensor, which can limit the non-superdiagonal elements of the 3-rd core tensor. Experiments on three standard TKGC datasets demonstrate that our method outperforms the state-of-the-art results on several metrics. The results suggest that the direct-current component is not the best feature for TKG representation learning. Additional analysis shows the effectiveness of our FA and OR models, even with smaller embedding dimensions.

## introduction knowledge graph (kg) contains a number of structured facts (h, r, t), where a fact expresses a directed relation r from a head entity h to a tail entity t. the complex kgs, such as freebase @xcite , dbpedia @xcite , and wikidata @xcite , are collected manually or automatically from structured or unstructured data on the web. such kgs are successfully applied to several downstream tasks, e.g., question answering @xcite and figure 1 : a toy example from the temporal knowledge graph shows an athlete's career. this example illustrates the temporal dependencies of facts that tom adeyemi plays for four teams from 2011 to 2017. toy example scores calculated by tucker-fa are in the appendix c. recommender system @xcite . however, those works ignore that many facts in the kgs change over time. temporal facts can be expressed as a quadruplet (h, r, t, τ ), with τ being the timestamp. the temporal knowledge graphs (tkgs), such as icews @xcite , gdelt @xcite and yago @xcite , are then built to handle these facts coupled with timestamps. one problem that hinders the application of tkgs in downstream tasks is the inevitable incompleteness or knowledge scarcity problem caused by missing entities or relations. thus, temporal knowledge graph completion (tkgc) aiming to complete the missing entities or relations over time has become an essential task in the research community. the previous methods for tkgc can be divided into four branches, in which the critical challenge is how to integrate timestamps into kgc modeling. time-dependent embedding method @xcite @xcite considers the temporal information as a transformation or an activation function for entities or relations. timestamp embedding method @xcite @xcite treats timestamps as additional learnable embeddings of the score function. experimental experience @xcite suggests that timestamp embeddings generally perform bet-ter than time-dependent embeddings. knowledge graph snapshots method @xcite aggregates multi-relational interactions of cropped subgraph to achieve more precise representations. historical context method @xcite model n-hop facts chain or repeat facts to increase the interpretability. the previous timestamp embedding methods model each quadruplet independently, which only captures the local temporal dependencies, ignoring the global temporal dependencies between one relation and entire timestamp. as shown in figure 1 , an athlete plays for different teams in different periods. we treat such events as a continuous line of events rather than as separate events. we propose a frequency attention (fa) model to address this issue. specifically, we treat each dimension in the timestamp embedding as a long-term signal and use discrete cosine transform (dct) to capture the frequency of the signal. furthermore, we take the frequency and part of the relation embedding as input to calculate attention weights for each timestamp. the proposed frequency attention model can easily apply to exist tensor decomposition methods. the previous tucker decomposition method is interpreted as a high-dimensional linear classifier to distinguish facts. tuckertnt @xcite , uses a simple l2 norm as the core tensor regularization. however, this regularization may be overstrict, leading the embedding to change sharply and risk vanishing or explosion. inspired by orthogonal regularization in @xcite , we propose two variants of orthogonal regularization (or) for the core tensor, i.e., excluding superdiagonal elements or diagonal elements of each slice matrix of the core tensor. this way, we achieve a balanced core tensor regularization, preventing the embedding norm from vanishing or exploding. in summary, our work makes the following contributions: ## preliminaries 3.1 problem definition to formally define the problem and describe the solution, we use consistent notations in the rest of the paper. we represent scalars with the lower case letters, e.g., d r , represent sets with the flower letters, e.g., e, represent vectors with the bold lower case letters, e.g., h i , denoting the i th entity embedding. we use bold upper letters h to denote the embedding matrix and represent the high order tensor with bold flower letters w. we use a ⊙ b to denote hadamard (elementwise) product of two vectors or matrix, a ⊗ b to denote the tensor outer product, [a|b] to denote the vector or matrix concatenation operator, || • || p to denote the p-norm of a vector or tensor. a temporal knowledge graph g consists of a set of facts {(h i , r j , t k , τ l )} ⊆ e × r × e × t , where e is a finite entity set, r is a finite relation set and t is a finite timestamp set. each quadruplet (h i , r j , t k , τ l ) respectively denotes a relation r j from the head entity h i to tail entity t k at a specific time τ l . a temporal knowledge graph uses a binary tensor @xmath0, 1} |e|×|r|×|e|×|t | to indicate whether the corresponding quadruplets occurs in the kg data set. |e|, |r| and |t | denote the number of entities, relations, and timestamps, respectively. although knowledge graphs contain large numbers of facts, they are still incomplete due to the complex nature of the real world. tkgc aims to predict the missing entity. we focus on the link prediction problem, aiming to predict the tail entity or head entity through the query (h i , r j , ?, τ l ) or (?, r j , t k , τ l ). the problem reduces to ranking a set of candidate entities to select the most likely entity that makes the partial quadruplet factual. the problem can be formulated as a ranking problem to learn a quadruplet score function x (e, r, e, t ) ∈ r to sort all candidate entities. ## model we propose a new framework, tucker-fa, combining the fa and or with the temporal tucker decomposition method. we input the timestamp embedding and relation embedding to the fa model to compute frequency attention weights, then weighted-sum the timestamp embedding and combine it with the relation embedding. the timestamp-enhanced relation feature and head/tail entity embedding compose the factor matrix of tucker decomposition. in learning progress, we include several regularization losses and the orthogonal regularization of the core tensor into the overall objective. ## limitation although our method has been shown effective, it has two limitations that may be improved in the future. first, the fa model has advantages in computation but relies on an effective frequency selection strategy, which is difficult to design. we just simply select some manual frequencies for different datasets by experience. the more effective frequency selection strategy needs further exploration. second, there is no theoretical guarantee that the orthogonal regularization can generalize to a 3-order tensor. our or terms are only formally consistent with matrix orthogonal regularization, which has been empirically shown effective. a parameter complexity we implement our model based on two previous models, tucker and tntcomplex. other baseline models mentioned above have yet to provide publicly available code. during the pre-processing data phase, we convert all time intervals into two different time points and consider them independent. the time intervals in yago15k look like "occursuntil/occursince 1994". the time interval is split into two parts, "occurssince" or "occursuntil" merged into a relation, and the time point transforms the timestamp. note that quadruplet without timestamps in yago15k also own a unique timestamp.