Time-dependent Entity Embedding is not All You Need: A Re-evaluation of Temporal Knowledge Graph Completion Models under a Unified Framework (2021)

Abstract: Various temporal knowledge graph (KG) completion models have been proposed in the recent literature. The models usually contain two parts, a temporal embedding layer and a score function derived from existing static KG modeling approaches. Since the approaches differ along several dimensions, including different score functions and training strategies, the individual contributions of different temporal embedding techniques to model performance are not always clear. In this work, we systematically study six temporal embedding approaches and empirically quantify their performance across a wide range of configurations with about 3000 experiments and 13159 GPU hours. We classify the temporal embeddings into two classes: (1) timestamp embeddings and (2) time-dependent entity embeddings. Despite the common belief that the latter is more expressive, an extensive experimental study shows that timestamp embeddings can achieve on-par or even better performance with significantly fewer parameters. Moreover, we find that when trained appropriately, the relative performance differences between various temporal embeddings often shrink and sometimes even reverse when compared to prior results. For example, TTransE (CITATION), one of the first temporal KG models, can outperform more recent architectures on ICEWS datasets. To foster further research, we provide the first unified open-source framework for temporal KG completion models with full composability, where temporal embeddings, score functions, loss functions, regularizers, and the explicit modeling of reciprocal relations can be combined arbitrarily.

## introduction the knowledge graph (kg), a graph-structured knowledge base, has gained increasing interest as a promising way to store factual knowledge. kgs represent facts in the form of triples (s, r, o), e.g., (bob, livesin, new york), in which s (subject) and o (object) denote nodes (entities) and r denotes the edge type (relation) between s and o. knowledge graphs are commonly static and store facts in their current state. in reality, however, the relations between entities often change over time. for example, if bob moves to california, the triple of (bob, livesin, new york) will be invalid. to this end, temporal knowledge graphs (tkgs) have been introduced to capture temporal aspects of facts in addition to their multi-relational nature. a tkg represents a temporal fact as a quadruple (s, r, o, t) by extending a static triple with time t, describing that this fact is valid at time t. figure 2 in the appendix depicts an exemplary temporal kg. to address the inherent incompleteness of temporal @xcite proposed the first tkg model. afterwards, a line of work emerged that extends static kg completion models by adding temporal embeddings, e.g., ttranse @xcite , @xcite , de-simple @xcite , tntcom-plex @xcite , cont @xcite , and many more. the models generally consist of two parts, a temporal embedding layer to capture the evolving features of tkgs and a score function to examine the plausibility of a given quadruple. temporal embeddings are crucial in temporal kg completion models for storing the evolving knowledge; without them, the temporal aspect cannot be captured. the pes can be generally categorized into three classes: (1) timestamp embeddings (tes): the models learn an embedding for each discrete timestamp in the same vector space as entities and relations @xcite @xcite . (2) time-dependent entity embeddings (tees): the models define entity embedding as a function that takes an entity and a timestamp as input and generates a time-dependent representation for the entity at that time @xcite @xcite . (3) deep representation learning (dtrs): the models incorporate temporal information into advanced deep learning models, e.g., recurrent neural network and graph neural network, to learn time-aware representations of entities and relations @xcite . in many cases, the introduction of new temporal embedding approaches went along with new score functions and new training methods (regularization, the explicit modeling of reciprocal relations, etc.). ablation studies were provided, but not investigated thoroughly. besides, some temporal embedding papers introduced new datasets. they commonly tune model architecture and hyperparameters of old temporal embedding approaches on new datasets using grid search on a small grid involving hand-crafted parameter ranges or settings known to work well from prior studies. a grid suitable for one dataset may be suboptimal for another, however. it is often difficult to attribute the incremental improvements in performance reported with each new state-of-the-art (sota) model to the proposed temporal embeddings or other components. in this work, we investigate the significance of previously reported temporal embeddings with several thousands of experiments and 19000 gpu hours. first, we aim to study which temporal embedding approach can generally outperform other temporal embedding approaches regardless of different score functions and different datasets. we choose one representative from bilinear score functions, i.e., @xcite , and one from translation-based score functions, i.e., transe @xcite . then we benchmark six temporal embedding approaches on two subsets of icews @xcite and a subset of gdelt @xcite with the two representative score functions through an extensive set of experiments. second, we performed an extensive benchmark study on wellknown temporal kg completion models using popular model architectures and training strategies in a unified experimental setup. following the work @xcite , we considered many training strategies as well as a large hyperparameter space, and we performed model selection using a quasi-random search followed by bayesian optimization, which has been shown to be able to find good model configurations with relatively low effort. regarding the first aim, we surprisingly find that the te proposed by @xcite outperforms other temporal embedding approaches on the icews subsets and achieves on-par results on gdelt. @xcite represent timestamps in the same vector space as entities and relations and learn embeddings for each discrete timestamp. while achieving better results, the te models only require about half of the model parameters as much of tees. however, the common belief is that the tees are more expressive and can better capture the evolving knowledge. recall that models with tees learn an embedding function for each entity that takes time as input and provides an entity representation as output. in particular, it has been proven that tees are fully expressive for tkg completion in combination with certain score functions @xcite , and thus, they should perform better than tes, which is in contrast to our findings. we argue that the sparsity of temporal kg data may cause the undesirable empirical performance of tees. every entity has the same dimensionality of time-dependent embeddings, but the majority of entities are only involved in a small number of quadruples. as a result, the tees may suffer from the overfitting problem. to verify our assumption, we learn a unique temporal embedding function for all entities instead of learning entity-specific embedding functions. we refer to it as utee. empirical study shows that the utee achieves similar or even better results than all other tees variants, emphasizing the overfitting problem of tees. besides, we empirically find that the performance of a fine-tuned baseline can by far exceed the performance observed in all previous studies. for example, t-transe @xcite , one of the first temporal kg completion models, achieves superior performance metric in our study that is more than doubled to that reported in recent papers @xcite @xcite . thus, it is competitive to or even outperforms current sota models such as de-simple @xcite and tcomplex @xcite . this suggests that training strategies significantly affect the performance of temporal kg models and are responsible for a substantial fraction of the progress made in recent years. thus, to fairly compare the effectiveness of different temporal kg models, it is nec-essary to evaluate them on a unified framework. to this end, our study realizes the first fair benchmarking by investigating the interplay between temporal kg interaction models, loss functions, regularization methods, the use of reciprocal relations, and other training techniques in a unified open-source framework foot_0 . to ensure the composability of the framework, the temporal embedding layer, score functions, and various training strategies are implemented as independent submodules. thus, one can easily assess the individual benefit of a novel temporal embedding approach via our framework. additionally, we perform an extensive experimental study in which well-known temporal kg models are fine-tuned by popular training strategies and a wide range of hyperparameter settings. the reported results can be directly used for comparison in future work. ## experimental study in this section, we first introduce the design of our unified framework that enables us to evaluate a large set of different combinations of interaction models, loss functions, regularization methods, the usage of of explicitly modeling reciprocal relations and position-aware entity embeddings. then we split our experimental study into two parts. in the first part, we examine six temporal embedding methods combined with two representative score functions by performing an extensive set of experiments using advanced training strategies and a wide range of hyperparameter settings via the unified framework. in the second part, we re-evaluate various well-known tkg models from prior studies. we provide evidence that several old tkg models can obtain results competitive to or even better than the sota when configured carefully. we present the best configuration of each model and report its best performance on each benchmark that future research can directly use for comparison. ## conclusion we assess well-known temporal embeddings of tkge models via an extensive experimental study. we found that when trained appropriately, the naive timestamp embedding approach performs similarly or even outperforms the more advanced timedependent entity embedding (tee) approaches, which is in contrast to the results in prior studies. we contribute to the community in at least two ways: i) we provide a unified framework to enable an insightful assessment for novel temporal embedding approaches; ii) reveal the weakness of tee approaches. ative quadruple. the margin η is a hyperparameter. goel et al. (2019); garcía-durán et al. (2018) treat the entity prediction task as a categorical classification problem and utilize the cross entropy (ce) loss to align the model distribution and the data distribution. @xcite proposed to use binary cross entropy (bce) loss that applies a sigmoid function to the score of each positive or negative quadruples and takes the cross entropy between the resulting probability and that quadruple's label as the loss. it has been shown in @xcite that the loss function has a significant impact on the performance of static kge models. to provide additional evidence on temporal kge models, we search the best choice of loss functions for each model on each dataset. regularization methods l2 regularization is widely used in literature @xcite . besides, @xcite proposed to use l1norm in the regularization term. and @xcite used l3-norm for cp-decomposition. additionally, @xcite proposed a smoothness regularization for timestamp embeddings that enforce neighboring timestamps to have close representations. moreover, @xcite used dropout in its hidden layers. aitse normalized the static embeddings e i , the trend component w e i to unit norm after each update. other hyperparameters for models with diachronic entity embedding as its temporal encoding heads, we extend the static feature ratio as an extra searchable hyperparameter. the negative sample ratio of the negative sampling policy is 500. namely, for each positive sample (s, p, o, t), we corrupt the subject and object entity via uniformly sampling from t , where @xmath0, p, o, t)|@xmath1, p, o, t)|@xmath2. we set our batch size to be 512. besides, since adam (kingma and ba, 2014) optimizer performs well for the majority of the models @xcite , we decided to progress only with adam in order to reduce the computational costs. additionally, for translational models, we set the margin γ to be 100 in the score function.