T emporal W iki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022)

Abstract: Language Models (LMs) become outdated as the world changes; they often fail to perform tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment. This is especially a challenging problem because the research community still lacks a coherent dataset for assessing the adaptability of LMs to frequently-updated knowledge corpus such as Wikipedia. To this end, we introduce TemporalWiki, a lifelong benchmark for ever-evolving LMs that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation, respectively. The benchmark hence allows researchers to periodically track an LM’s ability to retain previous knowledge and acquire updated/new knowledge at each point in time. We also find that training an LM on the diff data through continual learning methods achieves similar or better perplexity than on the entire snapshot in our benchmark with 12 times less computational cost, which verifies that factual knowledge in LMs can be safely updated with minimal training data via continual learning.

## introduction large language models (lms) pretrained on a vast amount of text corpus have shown to be highly effective when finetuned or prompted to perform various downstream tasks @xcite @xcite . however, most of the datasets used to evaluate these lms are static benchmarks; the train and test data are both from similar points in time. on the other hand, in the real world, factual knowledge is frequently changed, added, or deprecated. for example, suppose a language model is asked what the most dominant coronavirus variant is (figure 1 ). the answer would have been the delta variant in the fall of 2021 but has changed to the omicron variant near the end of 2021. if lms remain unchanged and are not periodically trained to cope with the changing world, they will be outdated very quickly. this means downstream tasks that directly depend on or are finetuned from the lm will suffer from temporal misalignment @xcite , which refers to the misalignment in time between the train and test data. temporal misalignment becomes a critical problem, especially when using language models for knowledge-intensive tasks such as closed-book question answering @xcite @xcite since they rely solely on the knowledge stored in their parameters. furthermore, lms augmented with retrieval mechanism @xcite @xcite often suffer from hallucination even if they successfully retrieve up-to-date information @xcite @xcite . this means that the implicit knowledge stored in the model parameters has to be updated as well because it may cause conflicts with the explicit knowledge retrieved from external sources such as up-to-date knowledge bases and ultimately cause the lm to hallucinate. recently, @xcite ; @xcite have explored updating the internal knowledge of lms through continual pretraining on new and updated data as a solution for mitigating temporal misalignment. however, these datasets are still static in nature: as the world changes, they will eventually get outdated as well. in order to comprehensively measure the capability of ever-evolving lms on addressing temporal misalignment, automated periodic evaluation of the lms is crucial. in this paper, we introduce temporalwiki, a lifelong benchmark for training and evaluating everevolving lms in a periodic and automated manner, shown in figure 1 . the corpora used for updating lms are constructed by comparing articles from consecutive english wikipedia snapshots and retrieving only changed information, which we name as twiki-diffsets. the evaluation datasets are constructed in a similar manner by comparing english wikidata snapshots that correspond to the wikipedia snapshots in time and categorizing each factual instance into unchanged or changed. since wikidata updates may not exactly align with wikipedia updates, we only retain factual instances that can be grounded to articles in wikipedia, ensuring the quality of the data and name the resulting evaluation dataset as twiki-probes. the entire benchmark creation process is done without any human annotation, thus allowing it to be automated and lifelong as new english wikipedia and english wikidata snapshots are released by wikimedia 1 on a monthly basis. through temporalwiki, we aim to tackle the following research questions: how can we train ever-evolving lms efficiently and automate the evaluation of each update? how does updating lms only on updated data from wikipedia compare to updating lms on entire wikipedia snapshots, especially in scenarios requiring multiple updates? how problematic is catastrophic forgetting (mccloskey and cohen, 1989) when lms are updated only on new data, and how can we effectively mitigate catastrophic forgetting? our main 1 https://commons.wikimedia.org/ contributions are summarized as follows: we hope that temporalwiki will foster future research towards training ever-evolving lms. ## background recent works have introduced the need to tackle the issue of temporal misalignment, which refers to neural networks showing poor performance due to misalignment in time between the train and test data. temporal misalignment can be caused either by (1) the dynamic nature of language @xcite @xcite or (2) the update of factual information @xcite @xcite . @xcite have emphasized the effect of temporal misalignment on eight different nlp downstream tasks, asserting that misalignment between the train and test sets of the downstream tasks causes severe performance degradation that can be mitigate finetuning on the corpus from the target period. @xcite have argued this to be less of a concern when utilizing representations from pretrained lms and show that self-labeling on the downstream task is more effective than continued pretraining on more recent data for temporal adaptation. note that these works have focused on misalignment caused by the dynamic nature of language on tasks that are not knowledge-intensive, such as text classification. others have tackled the problem caused by the update of factual knowledge. @xcite have shown that lms deteriorate significantly in performance when there is a misalignment in time between the pretraining data and the downstream task and argued ever-evolving lms are necessary. @xcite have proposed explicitly including time information during pretraining as a potential solution. @xcite ; @xcite have implemented continual learning methods to mitigate catastrophic forgetting that occurs during continued pretraining on new data. despite the recent community interest in the need for ever-evolving lms, the community lacks widely-available resources to train and evaluate such lms. previous works have introduced benchmarks comprised of data sources from twitter feeds @xcite @xcite , recent news articles @xcite , and arxiv papers @xcite where the temporal adaptability of lms and the effectiveness of different methodologies of updating lms can be evaluated. however, these data sources are domain-specific and inherently static. on the other hand, wikipedia and wikidata are known to be great sources of general world knowledge and thus have been widely used by the community @xcite @xcite . 120k volunteer editors make 120 updates to the english wikipedia per minute and add hundreds of new article entries every day (logan iv et al., 2021) foot_0 . even though every wikipedia and wikidata update may not correspond to an actual change in the real world, temporalwiki leverages the dynamic nature of wikipedia and wikidata to provide a lifelong benchmark for developing and maintaining ever-evolving lms. ## temporalwiki in this section, we delve into the process of creating temporalwiki, which is comprised of training corpora (twiki-diffsets) and evaluation datasets (twiki-probes) sourced from english wikipedia and english wikidata, respectively. for efficiency, english is abbreviated when referring to english wikipedia and english wikidata throughout the paper. moreover, we clarify that not all wikipedia/wikidata updates equate to actual updates of world knowledge. in section 3.1, we first describe the process of constructing the training corpora from wikipedia snapshots. then in section 3.2, we describe the process of generating the evaluation datasets from wikidata snapshots. in section 3.3, we describe the quality control applied to the evaluation datasets. ## dataset statistics in this paper, we construct temporalwiki from 08.2021 to 12.2021 foot_1 and its statistics are discussed below. training corpora statistics statistics of wikipedia snapshots and twiki-diffsets are shown in ## conclusion in this paper, we provide answers to the four proposed questions in section 1. (1) how can we train ever-evolving lms efficiently and automate the evaluation of each update? we introduce tem-poralwiki, a lifelong benchmark that can be used for training and evaluating ever-evolving lms in an automated manner. it consists of twiki-diffsets as the training corpora for temporal language modeling and twiki-probes as the evaluation datasets for measuring the stability-plasticity trade-off. (2) how does updating lms only on new and updated data from wikipedia compare to updating lms on entire wikipedia snapshots, especially in scenarios with multiple updates? through experiments on temporalwiki, we show that updating lms on twiki-diffsets leads to better acquisition of new and updated knowledge than updating on entire wikipedia snapshots with much less computational cost (12 times less). (3) how serious is catastrophic forgetting when lms are updated only on new and updated data? we observe that temporal language modeling is a challenging problem, especially as the number of lm updates increases. however, results still show an overall enhancement in terms of stability-plasticity compared to updating with entire wikipedia snapshots, showing that temporal language modeling is an effective alternative. (4) how can we mitigate catastrophic forgetting? we find that continual learning methods (regularization, rehearsal, and parameter-expansion) for large language model training effectively mitigates forgetting and shows robust performance in terms of enhancing the overall trade-off between stability and plasticity on twiki-probes.