Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2022)

Abstract: Pretrained language models (PTLMs) are typically learned over a large, static corpus and further fine-tuned for various downstream tasks. However, when deployed in the real world, a PTLM-based model must deal with data distributions that deviates from what the PTLM was initially trained on. In this paper, we study a lifelong language model pretraining challenge where a PTLM is continually updated so as to adapt to emerging data. Over a domain-incremental research paper stream and a chronologically-ordered tweet stream, we incrementally pretrain a PTLM with different continual learning algorithms, and keep track of the downstream task performance (after fine-tuning). We evaluate PTLM’s ability to adapt to new corpora while retaining learned knowledge in earlier corpora. Our experiments show distillation-based approaches to be most effective in retaining downstream performance in earlier domains. The algorithms also improve knowledge transfer, allowing models to achieve better downstream performance over latest data, and improve temporal generalization when distribution gaps exist between training and evaluation because of time. We believe our problem formulation, methods, and analysis will inspire future studies towards continual pretraining of language models.

## introduction pretrained language models (ptlms) have achieved remarkable performance on benchmark datasets for a range of nlp tasks @xcite . however, when deployed in the wild, nlp systems must deal with emerging data that have constantly shifting data distribution, different from the text corpora they were initially pretrained on -for example, when new data domains are introduced (upper part of fig. 1 ) @xcite , or when the language uses and vocabulary change over time (lower part of fig. 1 ) @xcite . fine-tuning from a ## problem formulation here we present the problem formulation for lifelong pretraining of ptlm, provide details about the data stream construction process and downstream tasks, and introduce the evaluation protocol. ## methods lifelong language model pretraining introduces novel challenges because of the large training sets and more comprehensive evaluation protocols compared to classification tasks. we establish several strong baselines, and evaluate the performance of continual learning algorithms from different categories spanning over model-expansion, memorybased, and distillation-based approaches, we illustrate the approaches in figure 3 . ## results we summarize our findings over the created data streams. we ask whether lifelong pretraining and continual learning algorthms are effective base on our evaluation protocol proposed in sec. 2.3. ## conclusion in this paper, we formulated the lifelong language model pretraining problem and constructed two data streams associated with downstream datasets. we evaluated knowledge retention, adaptation to the latest data, and temporal generalization ability of continually pretrained language models. our experiments show distillation-based approaches being most effective in these evaluation setups. a limitation of the work is that it has not been fully addressed whether there exists a variant of distillation-based cl approach that consistently outperforms logit-kd. based on the current observation, we conclude the performance of different kd approaches for cl is highly task-dependent. it asks for more future works into continual learning algorithms within the proposed problem setup.