ECOLA : Enhancing Temporal Knowledge Embeddings with Contextualized Language Representations (2023)

Abstract: Since conventional knowledge embedding models cannot take full advantage of the abundant textual information, there have been extensive research efforts in enhancing knowledge embedding using texts. However, existing enhancement approaches cannot apply to temporal knowledge graphs (tKGs), which contain time-dependent event knowledge with complex temporal dynamics. Specifically, existing enhancement approaches often assume knowledge embedding is time-independent. In contrast, the entity embedding in tKG models usually evolves, which poses the challenge of aligning temporally relevant texts with entities. To this end, we propose to study enhancing temporal knowledge embedding with textual data in this paper. As an approach to this task, we propose Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations (ECOLA), which takes the temporal aspect into account and injects textual information into temporal knowledge embedding. To evaluate ECOLA, we introduce three new datasets for training and evaluating ECOLA. Extensive experiments show that ECOLA significantly enhances temporal KG embedding models with up to 287% relative improvements regarding Hits@1 on the link prediction task. The code and models are publicly available on https://github.com/mayhugotong/ECOLA .

## introduction knowledge graphs (kgs) have long been considered an effective and efficient way to store structural knowledge about the world. a knowledge graph consists of a collection of triples ps, p, oq, where s (subject entity) and o (object entity) correspond to nodes, and p (predicate) indicates the edge type (relation) between the two entities. common knowledge graphs @xcite ˚zhen han's work done prior to joining amazon. † equal contribution. ‡ https://github.com/mayhugotong/ecola @xcite assume that the relations between entities are static connections. however, in the real world, there are not only static facts but also temporal relations associated with the entities. to this end, temporal knowledge graphs (tkgs) @xcite were introduced that capture temporal aspects of relations by extending a triple to a quadruple, which adds a timestamp to describe when the relation is valid, e.g., (r.t. erdogan, visit, @xcite . if the temporal relationship lasts for several timestamps, most tkgs represent it by a sequence of quadruples, e.g., {(r.t. erdogan, visit, us, 2019-11-12), (r.t. erdogan, visit, us, 2019-11-13)}. conventional knowledge embedding approaches learn kgs by capturing the structural information, suffering from the sparseness of kgs. to address this problem, some recent studies incorporate textual information to enrich knowledge embedding. kg-bert @xcite takes entity and relation descriptions of a triple as the input of a pre-trained language model (plm) and turns kg link prediction into a sequence classification problem. similarly, kepler @xcite computes entity representations by encoding entity descriptions with a plm and then applies kg score functions for link prediction. however, they could not be applied to tkgs. specifically, existing approaches (e.g., kepler) encode an entity, no matter at which timestamp, with the same static embedding based on a shared entity description. in comparison, entity embeddings in tkg models usually evolve over time as entities often involve in different events at different timestamps. therefore, an entity might be aligned with different textual knowledge at different time. and it should be taken into account which textual knowledge is relevant to which entity at which timestamp. we name this challenge as temporal alignment between texts and tkgs, which is to establish a correspondence between textual knowledge and their tkg depiction. another challenge is that many temporal knowledge embedding models @xcite ) learn the entity representations as a function of time. however, the existing enhancement approaches cannot be naturally applicable to such tkg embedding. we refer to this challenge as dynamic embedding challenge. in this work, we propose to study enhancing temporal knowledge embedding with textual data. as an approach to this task, we develop enhanced temporal knowledge embeddings with contextualized language representations (ecola), which uses temporally relevant textual knowledge to enhance the time-dependent knowledge graph embedding. specifically, we solve the temporal alignment challenge using tkg quadruples as an implicit measure. we pair a quadruple with its relevant textual data, e.g., event descriptions, which corresponds to the temporal relations between entities at a specific time. then we use the event description to enhance the representations of entities and the predicate involved in the given quadruple. besides, ecola solves the dynamic embedding challenge using a novel knowledge-text prediction (ktp) task which injects textual knowledge into temporal knowledge embeddings. specifically, given a quadruple-text pair, we feed both the temporal knowledge embeddings of the quadruple and token embeddings of the text into a plm. the ktp task is an extended masked language modeling task that randomly masks words in texts and entities/predicates/timestamp in quadruples. with the help of the ktp task, ecola would be able to recognize mentions of the subject entity and the object entity and align semantic relationships in the text with the predicate in the quadruple. for training ecola, we need datasets with tkg quadruples and aligned textual event descriptions, which are unavailable in the existing temporal kg benchmarks. thus, we construct three new temporal knowledge graph datasets by adapting two existing datasets, i.e., gdelt @xcite and @xcite , and an event extraction dataset @xcite . to summarize, our contributions are as follows: (i) we are the first to address the challenge of enhancing temporal knowledge embedding with temporally relevant textual information while preserving the time-evolving properties of entity embedding. (ii) we construct three datasets to train the text-enhanced tkg models. specifically, we adapt three existing temporal kg completion datasets by augmenting each quadruple with a relevant textual description. (iii) extensive experiments show that ecola is model-agnostic and can be potentially combined with any temporal kg embedding model. ecola also has a superior performance on the temporal kg completion task and enhances temporal kg models with up to 287% relative improvements in the hits@1 metric. (iv) as a joint model, ecola also empowers plms by integrating temporal structured knowledge into them. we select temporal question answering as a downstream nlp task, demonstrating that ecola can considerably enhance plms. ## preliminaries and related work temporal knowledge graphs temporal knowledge graphs are multi-relational, directed graphs with labeled timestamped edges between entities (nodes). let e and p represent a finite set of entities and predicates, respectively. a quadruple q " pe s , p, e o , tq represents a timestamped and labeled edge between a subject entity e s p e and an object entity e o p e at a timestamp t p t . let f represent the set of all true quadruples, the temporal knowledge graph completion (tkgc) is the task of inferring f based on a set of observed facts o. specifically, tkgc is to predict either a missing subject entity p?, p, e o , tq given the other three components or a missing object entity pe s , p, ?, tq. we provide related works on temporal knowledge representations in appendix a. ## ecola in this section, we present the overall framework of ecola, including the model architecture in section 3.1 -3.3, a novel task designed for aligning knowledge embedding and language representation in section 3.4, and the training procedure in section 3.5. as shown in figure 2 , ecola implicitly incorporates textual knowledge into temporal knowledge embeddings by jointly optimizing the knowledge-text prediction loss and the temporal knowledge embedding loss. note that, at inference time, we only take the enhanced temporal knowledge embeddings to perform the temporal kg completion task without using plm and any textual data for preventing information leakage and keeping a fast inference speed. ## the model-agnostic property of ecola ecola is model-agnostic and can enhance different temporal knowledge embedding models. besides ecola-dyernie, we introduce here two additional variants of ecola. ecola-de enhances de-simple, which applies the diachronic embedding (de) function @xcite . de-function defines the temporal embeddings of entity e i at timestamp t as ecola-utee enhances utee @xcite ) that learns a shared temporal encoding for all entities to address the overfitting problem of de-simple on sparse datasets. compared to ecola-de, ecola-utee replaces equation 3 1 . gdelt is an initiative knowledge base storing events across the globe connecting people and organizations, e.g., (google, consult, the united states, 2018/01/06). for each quadruple, gdelt provides the link to the news report which the quadruple is extracted from. we assume each sentence that contains both mentions of the subject and object is relevant to the given quadruple, and, thus, temporally aligned with the subject and object at the given timestamp. we pair each of these sentences with the given quadruple to form a training sample. this process is similar to the distant supervision algorithm @xcite in the relation extraction task. the proposed dataset contains 5849 entities, 237 predicates, 2403 timestamps, and 943956 quadruples with accompanying sentences. duee is originally a human-annotated dataset for event extraction containing 65 event types and 121 argument roles. each sample contains a sentence and several extracted event tuples. we select 41 event types that could be represented by quadruples and reformat duee by manually converting event tuples into quadruples and then pairing quadruples with their corresponding sentence. wiki is a temporal kg dataset proposed by @xcite . following the postprocessing by dasgupta et al. (2018), we discretize the time span into 82 different timestamps. we align each entity to its wikipedia page and extract ¶ https://www.gdeltproject.org/data.html#googlebigquery || https://ai.baidu.com/broad/download ** https://www.wikidata.org/wiki/wikidata:main_page the first section as its description. to construct the relevant textual data of each quadruple, we combine the subject description, relation, and object description into a sequence. in this case, the knowledge-text prediction task lets the subject entity learn the descriptions of its neighbors at different timestamps, thus, preserving the temporal alignment between time-dependent entity representation and textual data. ## experiments we evaluate the enhanced temporal knowledge embedding on the temporal kg completion task. specifically, we take the entity and predicate embedding of ecola-dyernie and use equation 1 to predict missing links. the textual description of test quadruples could introduce essential information and make the completion task much easier. thus, to make a fair comparison with other temporal kg embedding models, we take the enhanced lookup table embedding of temporal kgs to perform the link prediction task at test time but use neither textual descriptions of test quadruples nor the language model. we report such results in baselines we include both static and temporal kg embedding models. from the static kg embedding models, we use transe @xcite , distmult @xcite , and simple (kazemi and poole, 2018). these methods ignore the time information. from the temporal kg embedding models, we compare our model with several stateof-the-art methods, including atise @xcite , tntcomple @xcite , dy-ernie † † @xcite , tero @xcite , and de-simple @xcite . we provide implementation details in appendix b and attach the source code in the supplementary material. 3a . ecola-sf is the static counterpart of ecola-de, where we do not consider the temporal alignment while incorporating textual knowledge. specifically, ecola-sf integrates all textual knowledge into the time-invariant part of entity representations. we randomly initialize an embedding vector ēi p r d for each entity e i p e, where ēi has the same dimension as the token embedding in the pre-trained language model. then we learn the time-invariant part ēi via the knowledge-text prediction task. for the temporal kg completion task, we combine ēi with temporal knowledge embeddings, moreover, figure 3b shows the results of different masking strategies on gdelt. the first strategy, e.g., masking e+r+w, allows to simultaneously mask predicate, entity, and subword tokens in the same training sample. the second strategy is masking e/r+w, where we mask 15% subword tokens in the language part, and either an entity or a predicate in the knowledge tuple. in the third strategy called masking e/r/w, for each training sample, we choose to mask either subword tokens, an entity, or the predicate. figure 3b shows the advantage of the second masking strategy, indicating that remaining adequate information in the knowledge tuple helps the model to align the knowledge embedding and language representations. ## discussion inference with textual data in section 6, we compared different tkg embedding models, where textual data of test quadruples is absent during inference time. however, if the textual descriptions of the test quadruples are given during inference, will the contextualized language model incorporate this information into tkg embeddings? we use the entity predictor of the knowledge-text prediction task to perform the tkg completion task on gdelt. as shown in figure 4a , the results show significant improvement across all metrics, specifically, 145% relatively higher regarding mrr of ecola-utee when given textual data during inference than not given. thus, the results confirm that ktp task is a good choice for successful alignment between knowledge and language space and ecola utilizes the pre-trained language model to inject language representations into temporal knowledge embeddings. masking temporal information in ktp as temporal alignment is crucial for enhancing temporal knowledge embeddings, we study the effect of masking temporal information by extending the existing ktp task with an additional time prediction task, where the timestamp in the input is masked, and the model learns to predict the original timestamp. the extended model is named tecola-utee and has significant performance gain on both gdelt and wiki datasets across all metrics as shown in temporal question answering although we focus on generating informative temporal knowledge embeddings in this work, joint models often benefit both the language model and the temporal kg model mutually. unlike previous joint models @xcite , we do not modify the transformer architecture, e.g., adding entity linkers or fusion layers. thus, the language encoder enhanced by external knowledge can be adapted to a wide range of downstream tasks as easily as bert. besides the tkg completion task, we evaluate the enhanced language model in ecola on the temporal question-answering task to study its enhancement. natural questions often include temporal constraints, e.g., who was the us president before jimmy carter? to deal with such challenging temporal constraints, temporal question answering over temporal knowledge base, formulated as tkgqa task, has become trendy since tkgs help to find entity or timestamp answers with support of temporal facts. @xcite introduced the dataset cronquestions containing natural temporal questions with different types of temporal constraints. they proposed a baseline cronkgqa that uses bert to understand the temporal constraints, followed by a scoring function for answer prediction. we apply ecola to enhance the bert in cronkgqa then plug it back into cronkgqa and finetune it on the question answering dataset. we name the enhanced model as ecola-cronkgqa. the models are evaluated with standard metrics hits@kpk p t1, 3uq: the percentage of times that the true entity or time candidate appears in the top k of ranked candidates. figure 4b shows that our proposed ecola considerably enhances cronkgqa, demonstrating the benefits of ecola to the language model. ## conclusion we introduced ecola to enhance time-evolving entity representations with temporally relevant textual data using a novel knowledge-text prediction task. besides, we constructed three datasets that contain paired structured temporal knowledge and unstructured textual descriptions, which can benefit future research on fusing temporal structured and unstructured knowledge. extensive experiments show ecola can improve various temporal knowledge graph models by a large margin.