Set the Clock: Temporal Alignment of Pretrained Language Models (2024)

Abstract: Language models (LMs) are trained on web text originating from many points in time and, in general, without any explicit temporal grounding. This work investigates the temporal chaos of pretrained LMs and explores various methods to align their internal knowledge to a target time, which we call “temporal alignment.” To do this, we first automatically construct a dataset containing 20K time-sensitive questions and their answers for each year from 2000 to 2023. Based on this dataset, we empirically show that pretrained LMs (e.g., LLaMa2), despite having a recent pretraining cutoff (e.g., 2022), mostly answer questions using earlier knowledge (e.g., in 2019). We then develop several methods, from prompting to finetuning, to align LMs to use their most recent knowledge when answering questions, and investigate various factors in this alignment. Our experiments demonstrate that aligning LLaMa2 to the year 2022 can enhance its performance by up to 62% according to that year’s answers. This improvement occurs even without explicitly mentioning time information, indicating the possibility of aligning models’ internal sense of time after pretraining. Finally, we find that alignment to a historical time is also possible, with up to 2.8\times the performance of the unaligned LM in 2010 if finetuning models to that year. These findings hint at the sophistication of LMs’ internal knowledge organization and the necessity of tuning them properly.

## introduction large-scale pretraining @xcite @xcite has enabled language models (lms) to learn extensive knowledge from unlabeled text @xcite . however, since pretraining corpora are constructed over a wide time period, they inevitably contain outdated and contradictory information @xcite . previous studies have found that the temporal misalignment between lm pretraining and deployment has a significant impact on models' performance @xcite @xcite , which motivates many studies on making model's knowledge up-todate, by either continual learning @xcite , knowledge editing @xcite , or retrieval augmentation @xcite . these approaches mainly focus on updating models with new knowledge and do not evaluate lm's internal temporal knowledge across time. in this work, we hypothesize that lms, after pretraining, encode a chaotic sense of time, which means that models do not know which timesensitive knowledge to use even if they have seen it during pretraining. we empirically investigate this temporal chaos and explore methods to do temporal alignment, which aims to align models' internal knowledge to a target time. we first introduce our temporalalignmentqa dataset (taqa, §2) consisting of 20,148 questions, each with at least five different answers between 2000 and 2023. this dataset is automatically constructed based on wikipedia tables that contain temporal information, making it easily scalable for future study as the world keeps changing. complementary to previous temporal qa datasets @xcite @xcite @xcite , taqa focuses on facts that changed multiple times during a recent period when most pretraining data is collected, so that it can be used to probe the temporal knowledge distribution of pretrained lms. figure 1 plots the f1 score of several representative lms relative to ground-truth answers from each year, demonstrating that they tend to use earlier knowledge to answer questions, even when they have a very recent pretraining cutoff date (e.g., next, we explore how to align lms to answer questions based on a target time. specifically, we first try aligning models to the most recent possible time, as given by their knowledge cutoff date ( §3). this is often desired when lms are deployed for general user populations who seek current answers to their questions. we propose three methods, including 1) time-aware prompting, where we prompt lms using time information and time-sensitive examples; 2) target-year finetuning, where we finetune lms with ground truth answers in a target year so that they can generalize and answer new questions based on that year; 3) adaptive finetuning, where we elicit the most recent knowledge each lm knows about a question (which can be earlier if the model does not know the updated version), and teach the model to answer adaptively based on the year proper for its own. experimental results ( §4) show that finetuning llama2 on taqa, even without explicit temporal information, can relatively improve the answer performance in 2022 by up to 62.2%. we additionally explore the possibility of aligning lms to a historical time ( §5) and find our finetuning alignment strategy can boost the qa performance by 2.8× when aligning to 2010. we also find that 2019 is the most readily alignable year for llama2, while aligning it to 2015 causes the fewest errors measured against all valid answers from 2000 to 2023. ## taqa: a qa dataset for studying lm temporal alignment to study the temporal chaos of pretrained lms and how to align them temporally, we curate the taqa dataset, featuring questions with frequently changing answers in a recent time period . we introduce the formulation of taqa in §2.1, detail the data construction process in §2.2, and further describe the evaluation metrics in §2.3. ## aligning lms towards recency in this section, we propose several methods for aligning lms to recent years. complementary to existing studies that aim to teach language models up-to-date knowledge @xcite , our alignment methods focus on how to steer models to reorganize their knowledge and align them to the most-recent possible time. ## experiments this section describes our experiments on aligning llama2 using our proposed methods on the taqa dataset. since the pretraining cutoff of llama2 is september 2022, we try aligning llama2 to both 2022, which it has seen partially, and 2021, which it should have seen entirely. we first introduce several baselines ( §4.1), present the effectiveness of aligning llama2 ( §4.2), ablate different data selection strategies ( §4.3), and finally conduct various analyses to verify the improvement ( §4.4). ## further analysis: to what year are lms most readily alignable? aside from aligning lms to a recent time, we are also curious about which year, historically speaking, an lm is most readily alignable to and achieve the best performance. we investigate this by finetuning llama-70b to different years. figure 6 shows that 2019 is the best year for aligning the llama2-70b model instead of the year of the pretraining cutoff. meanwhile, the performance of models finetuned with data whose answers are true in 2010, 2015, and 2019 are all better than in recent years (2022 and 2023). we believe this is because the information about the most recent factual knowledge is too short in existing pretraining corpora, so it is more challenging to align the internal knowledge of lms to recent times than slightly further to the past. furthermore, if considering the historical maximum f max , the target year of 2015 performs the best among all our target year choices. we argue that the reason could be factual knowledge in 2015 aligns the most with llama2-70b's internal (but unactivated) knowledge learned from pretraining, and therefore, finetuning the lm on taqa where the answers were true in 2015 causes the least contradiction and thus makes the lm the least likely to produce historically-incorrect answers. temporal qa and reasoning temporal qa and reasoning are tasks where the correct answers change through time @xcite , e.g., "who is the president of the united states". prior research tried using knowledge graphs to tackle these tasks @xcite @xcite . as modern lms can memorize factual knowledge, new benchmarks @xcite @xcite address the challenge of leveraging lms to answer questions with contextual temporal mentions provided (e.g., "in 2023", "after" certain events, etc.). however, such question structure can sound unnatural, and these studies do not explicitly focus on questions whose answers have changed in recent years. other news article-based datasets like realtimeqa @xcite and streamingqa @xcite , despite having more natural questions, still lack questions whose answers can change multiple times, rather than a single time. in contrast, our taqa dataset curates natural questions without temporal information but with at least five different answers after 2000. several approaches have been proposed to tackle the temporal qa and reasoning challenges @xcite , yet these approaches re-quire dedicated training for enhancing lms' understanding of temporal mentions, which are missing in most natural questions. ## conclusion we build taqa, a qa dataset with time-sensitive questions during a recent time period @xcite @xcite . using this dataset, we quantitatively show that lms can respond to questions chaotically in terms of the question's placement in time. we thus propose the concept of temporal alignment and a set of methods for it. extensive experiments demonstrate the possibility and effectiveness of aligning model internal knowledge temporally, both to a recent time and to a historical time.