Time-Aware Language Models as Temporal Knowledge Bases (2022)

Abstract: Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snapshots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum—those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently “refreshed” as new data arrives, without the need for retraining from scratch.

## introduction language models (lms) have been suggested as repositories of real-world knowledge @xcite and there is much interest in using them for tasks such as closed-book question answering (qa; @xcite , fact verification @xcite , and dialogue @xcite . many facts, however, change with time. this raises two questions: do pretrained lms learn the appropriate temporal scope for the facts they encode? and what is the best way to update temporally scoped knowledge in pretrained models? pretraining corpora for models such as bert @xcite , roberta @xcite , and gpt @xcite are typically derived from a snapshot of the web crawled at a specific moment in time @xcite . while the impact on language modeling itself has been highlighted in recent work (e.g., @xcite @xcite , there are several potential problems specific to the encoding of factual knowledge: temporally scoped facts are common in practice; however, qa datasets such as squad @xcite or natural questions @xcite focus on a single time period, even for questions whose answers are temporally scoped. thus, our first contribution in this paper is a diagnostic dataset, templama (short for temporal language model analysis), of fill-in-the-blank queries for probing time-sensitive knowledge in lms. the queries in templama are chosen such that the answer varies with time ( § 2.1). using this dataset, we find empirical evidence of the problems mentioned above ( § 3). as a first step towards addressing these problems, we propose a lightweight modification to pretraining. we parametrize the masked language modeling objective (mlm; @xcite with temporal information, p (y|x, t; θ), where y is a masked token or span, x is the textual context, and t is the time ( § 2.3). the parameters θ must learn a representation of both text and time. in the t5 framework @xcite , this can be accomplished by prefixing the input x with a string representation of t, for example, ''year: 2018''. in addition, we pretrain from documents that are uniformly sampled from the timespan of the training corpus which, in our case, consists of news articles ranging from 2010-2018 @xcite ( § 2.1). these interventions accomplish two goals: the model is exposed to facts from the entire time range instead of just the most recent one, which avoids forgetting certain temporally scoped facts, and it prevents averaging because the facts are assigned to different time buckets (in our case years). this leads to improved recall of facts from the timespan of the training corpus ( § 3.1). these interventions also improve the model's temporal calibration. we find that jointly modeling text and time improves perplexity on future years unseen during training. on templama, the joint model degrades more gracefully than a model unaware of time. we also examine the model's calibration farther into the future using hand-crafted sets of queries whose answer is likely to change frequently, rarely, or never. we find qualitative evidence that the entropy of models trained uniformly across the training timespan increases most rapidly for the frequently changing facts ( § 3.2). while calibration is desirable, models should be refreshed with new data when it becomes available. a standard practice for doing this is to combine the new and old data and retrain the model from scratch (e.g., @xcite , but retraining can be costly for large-scale models @xcite . on the other hand, finetuning only on the new data leads to catastrophic forgetting of the old data @xcite , since standard lms have no knowledge of what is ''new'' and what is ''old'', unlike a model trained with temporal context. we show that our temporally scoped pretraining procedure makes lms more amenable to post-hoc finetuning, as the data is implicitly bucketed into non-overlapping time slices. we observe a similar performance to models retrained from scratch with 30× fewer steps, and without degradation on the knowledge encoded by the older data ( § 3.3). ## methods we probe factual knowledge in masked lms using span prediction-given an input statement x with a span y replaced by a special character, the task is to reconstruct that span. additionally, we assume that each (x, y) pair has a timestamp t denoting the time at which it was written or a point in time at which its assertion is valid. in this paper, we discretize t into yearly buckets and leave more fine-grained groupings (e.g., at the level of months or days) for future work. for simplicity and efficiency, all of our models are text-to-text transformers @xcite initialized from publicly available t5 checkpoints @xcite and then adapted to more time-dependent datasets. we first describe these datasets, followed by the approaches for jointly modeling text and time. ## experiments we design several experiments to highlight the problems around temporally scoped knowledge in lms and to test whether they can be addressed by joint models of text and time. ## discussion and limitations our experiments have shown that current models have practical limitations in their ability to memorize the past and reasonably estimate the future. these limitations can be mitigated by providing the model the date at which a text was created. while our results show consistent advantages, they also represent a narrow understanding of time. in particular, the publication date of a news articles does not necessarily correspond to the temporal scope of all events described in the article. for example, articles may talk about historical events or discuss events scheduled to happen in the future. in customnews around 3.9% sentences explicitly mention a year between 2010 and 2018, and 2.1% mention the same year as the publication date of the article. this fraction is likely responsible for the improvement of the uniform model. the temporal model further assigns an approximate scope to the remaining 96% sentences and it is encouraging to see improvements from that. one avenue for future work is to explore better strategies for assigning dates to these sentences. we have focused on closed-book question answering, but temporal staleness of language models may have impacts in other applications as well. for example, in open-book question answering, it is still necessary to align the question with relevant text in the retrieved passage, and this could be challenging when the question cannot be properly encoded by a stale lm: for example, the query ''which countries were affected by the 2020 hurricane season?'' would not match the passage ''iota caused damages of $564 million in nicaragua'' in an lm that did not have access to training data mentioning ''iota'' as a hurricane. another limitation of our work is that temp-lama is constructed in a synthetic manner from wikidata. incomplete or incorrect facts in the kb can result in incorrect queries in templama; for instance, we assume a missing start date implies the fact is valid from the beginning of our time period of interest. we partition the templama and customnews dataset on the same yearly slices despite the nature of the datasets being quite different. moreover, we did not investigate using longer or shorter temporal partitions. additionally, we did not test the ability to model temporal expressions such as ''before'' or ''during'', and we did not investigate temporal commonsense (e.g., @xcite , temporal ordering (e.g., @xcite , or events (e.g., @xcite . lastly, it is worth noting that like all closedbook models the models presented in this paper are also likely to only memorize common facts about popular entities. this has the danger of reinforcing stereotypes and leading to unfair outcomes. additionally, training the multitude of large-scale language models presented in this paper required the use of 32 cloud tpu v3 cores for several hundred hours, which has a significant environmental impact @xcite . however, our hope is that efficient schemes for updating temporally sensitive knowledge in lms will eventually save energy costs in the long run. ## related work there is extensive prior work on learning diachronic embeddings of individual words (e.g., @xcite @xcite . particularly related is the approach of @xcite , who learn time-sensitive embeddings by concatenating each word token with the decade in which it appears. as contextualized embedding models have largely replaced non-contextual word embeddings @xcite , the main application of diachronic word embeddings is to detect and model lexical semantic changes (e.g., @xcite , rather than to improve temporal awareness on downstream tasks. our work fills this gap by adding a temporal component to t5, a pretrained language model that can complete multi-token spans. while @xcite use contextualized embeddings from bert to model lexical semantic changes post hoc, they do not add a time-sensitive component to the language model itself. thus, their approach cannot support time-aware fact completion. several studies have focused on degradation of models on test data from a different time period than their training data @xcite @xcite @xcite . @xcite introduced an lstm language model that conditions on dynamic author representations computed separately, and showed that it improves perplexity on both seen and unseen (future) time periods. most recently, @xcite analyzed the interplay between temporal adaptation during pretraining and finetuning, and concluded that while both stages benefit from adaptation separately, adaptation during pretraining does not help the downstream task. here we show that the benefits of adaptation can be achieved using a single model that conditions on time. we further show that the benefits of adaptation come, at least in part, from better memorization of time-sensitive facts. in production contexts, an important form of temporal generalization is the deployment of models trained on data up to a certain time t but applied on data after t : that is, the present. @xcite show that language models gradually degrade in performance under such a time-stratified setting, and propose dynamic evaluation @xcite as a potential mitigation. however, lms are frequently applied to past data as well, for example, for extracting representations, and here we show that updating on only the new data degrades performance on old data. our approach of conditioning on the temporal context alleviates this issue. a related line of work has explored editing neural predictions after training given a dataset of revised input and output pairs @xcite @xcite . here we introduce a different setting where we have access to new unlabeled text after model training, which must be used implicitly to update the factual predictions of the model. in this case the update procedure also needs to figure out which facts must be updated and which ones remain the same. @xcite introduced the lama benchmark for probing the factual knowledge memorized by lms, which consists of cloze queries about facts, for example, ''dante was born in x ''. follow up studies have introduced improved prompts for eliciting such knowledge @xcite as well as multilingual versions @xcite . however, all these benchmarks assume a static view of the knowledge inside an lm, and consider all answers across time to be correct for a given query. the templama dataset instead focuses on relations where the answers change with time and uses temporal scopes to determine the correct answer. templama is similar in spirit to kb-qa benchmarks which focus on temporal reasoning such as tempquestions @xcite and cronquestions @xcite . its format, however, mimics the masked lm task typically used in pretraining, since it is intended as a zero/few-shot probe. unlike those datasets, we further restrict the queries to subject and relation pairs for which multiple objects exist at different points in time, and ensure a balanced distribution over the entire time period of interest from 2010-2020. wikidata id relation # queries template p54 member of sports team 9033 <@xmath0. p39 position held 7343 <@xmath1. p108 employer 9049 <@xmath2. p102 political party 7324 <@xmath3. p286 head coach 4886 <@xmath4. p69 educated at 1672 <@xmath5. p488 chairperson 4190 <@xmath6. p6 head of government 4125 <@xmath7. p127 owned by 2688 <@xmath8.