Lifelong Pretraining : Continually Adapting Language Models to Emerging Corpora
Abstract
Pretrained language models (PTLMs) are typically learned over a large, static corpus and further ﬁne-tuned for various downstream tasks.
However, when deployed in the real world, a
PTLM-based model must deal with data distributions that deviate from what the PTLM was initially trained on.
In this paper, we study a lifelong language model pretraining challenge where a PTLM is continually updated so as to adapt to emerging data.
Over a domain-incremental research paper stream and a chronologically-ordered tweet stream, we incrementally pretrain a PTLM with different continual learning algorithms, and keep track of the downstream task performance (after ﬁne-tuning).
We evaluate PTLM’s ability to adapt to new corpora while retaining learned knowledge in earlier corpora.
Our experiments show distillation-based approaches to be most effective in retaining downstream performance in earlier domains.
The algorithms also improve knowledge transfer, allowing models to achieve better downstream performance over the latest data, and improve temporal generalization when distribution gaps exist between training and evaluation because of time.
We believe our problem formulation, methods, and analysis will inspire future studies towards continual pretraining of language models.
Introduction
Pretrained language models (PTLMs) have achieved remarkable performance on benchmark datasets for a range of NLP tasks (Liu et al., 2019b;
Brown et al., 2020).
However, when deployed in the wild, NLP systems must deal with emerging data that have constantly shifting data distribution, different from the text corpora they were initially pretrained on — for example, when new data domains are introduced (upper part of Fig. 1)
(Gururangan et al., 2020), or when the language uses and vocabulary change over time (lower part of
Fig. 1) (Lazaridou et al., 2021).
Fine-tuning from a
[Figure]
Temporal
[Figure]
static and possibly “outdated" PTLM may limit the model performance on downstream tasks, as the
PTLM may no longer provide an effective model initialization (Beltagy et al., 2019; Müller et al., 2020).
Here we look to understand whether continuously adapting a PTLM to emerging data can yield gains on various downstream tasks, and how to achieve better downstream performance for such lifelong PTLM adaptation.
A number of recent works make attempts on adapting PTLMs to a new data domain.
Gururangan et al.
; Yao et al.
(2021) adapt language models to corpora of different genres and topics and observe performance improvement in domainspeciﬁc downstream tasks.
Arumae et al.
(2020) further show that by regularizing the parameters of PTLMs, the downstream tasks performance on the general domain can be preserved.
Another line of works focuses on temporal domain shift (Hombaiah et al., 2021), which analyzes the effect of pretraining over up-to-date data to the downstream tasks.
Röttger and Pierrehumbert (2021) further study vocabulary composition approaches for improving adaptation to up-to-date corpora.
However, these work focus their study on adapting PTLM to a single new domain; while in practice, corpora from distinct domains and time stamps may emerge sequentially.
Whether one can maintain a single, up-to-date PTLM remains an open problem.
Related to this, Lazaridou et al. (2021) study adaptation of PTLMs over temporal data streams, but solely focus on language modeling instead of ﬁne-tuning performance.
It is also important to understand multiple aspects of the utility of lifelong
PTLM pretraining, such as knowledge retention over all the seen data, and study what methods can improve the utility of PTLMs in such a continual pretraining process.
In this paper, we formulate a Lifelong Language
Model Pretraining task to simulate practical scenarios of maintaining and adapting a PTLM over emerging corpora, create a testbed (along with pretraining data streams and downstream tasks) for studying continual pretraining algorithms, and present a systematic evaluation protocol for measuring the progress made on this challenging problem (see Figure 2 for an illustration).
We consider two types of text corpus sequences when constructing pretraining data streams, each of which simulates a representative use case and that has slightly different focuses on the evaluation: continuously learning a single model that is applicable to both old and new domains; and improving the model’s ability to handle latest data.
Speciﬁcally, we construct 1) a domain-incremental text stream that consists of academic papers published in four research ﬁelds, and 2) a temporal tweet stream that consists of tweets collected from four different years.
By conducting systematic experiments on these two data streams, we look to answer a series of analysis questions: 1) whether continual pretraining retains ﬁne-tuning performance over earlier corpora compared to traditional ofﬂine pretraining, 2) whether pretraining improves downstream performance on the latest data, and 3) whether pretraining improves temporal generalization where training and evaluation have distribution gaps because of time.
To address the research questions above, we conduct a systematic evaluation of existing continual learning (CL) algorithms, spanning over modelexpansion based, memory-based, and distillationbased approaches.
Our results show distillationbased approaches are most effective in knowledge retention in the research paper stream, while simultaneously improve adaptation to latest data and temporal generalization in the tweet stream.
We believe our problem formulation, evaluation setup, methods and analysis can inspire more future work on continual pretraining of language models.
Problem Formulation
Here we present the problem formulation for lifelong pretraining of PTLM, provide details about the data stream construction process and downstream tasks, and introduce the evaluation protocol.
2. Lifelong Pretraining of PTLMs
We consider the scenario where one needs to deploy and/or maintain NLP models over a sequence of T data domains.
At each time step t the model visits an unlabeled text corpus D t from a domain with a data distribution P ( D t ) .
The data distribution P ( D t ) evolves as the time step t , forming a data stream D 1 ..T = { D 1 , D 2 , ...
D T } .
In practice, the data domain shift can refer to the topic change of the text content (from computer science research papers to biomedical papers), or temporal evolution of the text (from past to recent tweets).
The task of lifelong pretraining of PTLM looks to continuously adapt a language model f as the model visits (unlabeled) text corpus D t from the data stream D 1 ..T , in order to provide a good model initialization for ﬁne-tuning on downstream tasks from the same domain.
With slight abuse in notations, we also use
D t to directly refer to a data domain.
Here, we assume a language model f is updated sequentially over each pretraining corpora D t , without accessing the full earlier corpora { D i } i<t in the data stream D 1 ..T .
This aims to capture practical constraints such as privacy restriction for storing earlier data, or computation budget for training over all the text corpora in D 1 ..T .
We use f t to denote the language model right after updating on the domain D t .
In our study, f is a RoBERTa-base transformer (Liu et al., 2019b) and the model ( f 0 ) is initialized with pretrained RoBERTa weights.
The utility of the PTLMs { f t } is evaluated based on their ﬁne-tuned model performance on various downstream tasks.
After updating on a domain
D i , the model f i can be ﬁne-tuned over downstream tasks from visited domains D t where t ≤ i .
We note the set of downstream tasks related to domain D t as S t = { S jt } N t
j
=1 , assuming the number of downstream tasks is N t .
Note that in the ﬁnetuning stage, model f t has no access to any of the pretraining corpus D 1 ..T .
[Figure]
2. Data Streams & Downstream Datasets
We construct data streams to simulate two representative scenarios of data domain shifts in practice (also see Fig. 1): one domain-incremental stream to simulate the sequential changes of research paper areas; and one chronologically-ordered stream to simulate tweets emerging over time.
Domain-incremental Paper Stream.
This paper stream consists of the full text of research papers published in four research areas: biomedical, computer science, material science, and physics, ﬁltered from the S2ORC dataset 1 , which are presented sequentially to the model.
For each domain, we evaluate downstream performance over two datasets.
The downstream tasks span over various tasks such as relation extraction and named entity recognition, and are summarized in Table 1.
We detail these datasets in Appendix D.
Chronologically-ordered Tweet Stream.
This tweet data stream consists of tweets from the year 2014, 2016, 2018 and 2020, collected by the Archive Team 2 and preprocessed following Nguyen et al.
These four tweet corpora are presented sequentially to the language model following the chronological order of the tweet year.
For downstream tasks, we hold out 1M tweets from each year’s corpus to construct multi-label hashtag prediction datasets (Gong and Zhang, 2016) and single-label emoji prediction datasets (Barbieri
[Table]
et al., 2018).
On two datasets, we report label ranking average precision scores (a multi-label version of MRR) of models (Azeemi and Waheed, 2021) and Macro-F1 respectively.
The detailed dataset construction process is included in Appendix D.
2. Evaluation Protocol
We consider three key aspects for evaluating the utility of the language models { f t } that are continuously updated over the data stream D 1 ..T , also illustrated in Figure 2: 1) knowledge retention and transfer over the pretraining corpora seen earlier; 2) adaptation to the latest data domain, and 3) temporal generalization when training and evaluation data are from different time steps.
Knowledge Retention.
A key utility of continual language model pretraining is to obtain a single model applicable to all domains.
We focus on the evaluation of the ability with the domainincremental paper stream, because for the tweet stream, the practical need of performance over outdated data is limited.
Knowledge retention is measured with the downstream task performance from earlier or the current domains that the pretrained model has visited.
More formally, for each pretrained model checkpoint in { f i } , we ﬁne-tune f i over downstream tasks { S t } where t ≤ i and evaluate the corresponding test set performance.
It is important that the models do not suffer from catastrophic forgetting (Robins, 1995), i.e., signiﬁcantly reduced helpfulness when f i is ﬁne-tuned for downstream tasks S t from earlier domains with t < i .
Adaption to Latest Data Domain.
In certain scenarios, performance of downstream models over the latest data domain should be emphasized.
For example, classiﬁers in the tweet domain are usually trained and evaluated with up-to-date data for practical deployment.
Formally, we focus on the downstream task performance of models ﬁne-tuned from the ﬁnal pretrained model checkpoint f T , where the downstream tasks S T are also from the latest domain.
To succeed in these metrics, it is crucial for the model to transfer knowledge from earlier domains to the latest domain.
Temporal Generalization Ability.
We consider another practical ﬁne-tuning scenario in the tweet stream where the model is trained on outdated data and evaluated on the latest data (Rijhwani and Preotiuc-Pietro, 2020; Huang and Paul, 2018), referred to as the temporal generalization ability.
Formally, we ﬁne-tune the ﬁnal pretrained model checkpoint f T over the training set of downstream tasks S t from an earlier time step ( t < T ), and evaluate on the test set of the downstream tasks S T from the latest time step T .
Methods
Lifelong language model pretraining introduces novel challenges because of the large training sets and more comprehensive evaluation protocols compared to classiﬁcation tasks.
We establish several strong baselines, and evaluate the performance of continual learning algorithms from different categories spanning over model-expansion, memorybased, and distillation-based approaches, We illustrate the approaches in Figure 3.
3. Simple Baselines
We consider several simple baselines which continual learning algorithms will be compared against.
RoBERTa-base ( f 0 ) corresponds to not pretraining on any of the domain-speciﬁc corpora.
By separately pretraining f 0 on each corpus
D 1 , D 2 , ...
D T , we obtain T Task-Specific pretrained models.
We also pretrain f 0 sequentially over D 1 ..T , which we refer to as sequential pretraining .
While it allows knowledge transfer between domains compared to domain-speciﬁc models, without any continual learning algorithms, sequential pretraining is prone to catastrophic forgetting (Robins, 1995).
Finally, we randomly shufﬂe corpora from all domains D 1 ..T before pretraining, noted as Multi-Task Learning (MTL) .
MTL corresponds to an ofﬂine training paradigm that models new corpora by re-training over all corpora seen before.
The drawback is that it requires storing full data from earlier domains, and that it can be extremely costly to repetitively retrain over earlier data if new data keeps emerging. [Figure] 𝑘 Layer
[Figure]
{ 𝑥
[Figure]
𝑥
[Figure]
3. Model-expansion and Regularization-based Methods
We ﬁrst introduce model-expansion based approaches, which add small trainable modules ( e.g., multi-layer perceptron) to the model per new domain while keeping other parts of the model frozen.
The Adapter approach is a representative approach that learns a set of “adapter” layers g t = { g kt } Kk =1 for each domain D t and each of the K transformer layers (Houlsby et al., 2019).
We also experiment with a simple Layer Expansion approach, which learns separate top two layers of the transformer and the prediction head for each domain.
We also involve a regularization-based continual learning baseline, online EWC (Schwarz et al., 2018), which directly penalize change of model parameters.
3. Memory Replay Methods
We also experiment with Experience Replay (ER) (Chaudhry et al., 2019), which alleviates forgetting by storing a subset of earlier examples and periodically re-training (replaying) over them.
We maintain a ﬁxed-size memory M ( 100 k examples by default) and populate the memory M each time pretraining on a domain D t ﬁnishes with examples in the current domain.
We ensure M always contains a balanced sample of examples from all seen domains D 1 ..t .
We replay a mini-batch of examples from the memory every 10 training steps.
3. Distillation-based CL Methods
While knowledge distillation (KD) (Hinton et al., 2015) techniques have been studied intensively for pretrained language models (Sun et al., 2019), applying them to continual learning has been underexplored outside image classiﬁcation tasks (Li and
Hoiem, 2018; Rebufﬁ et al., 2017; Hou et al., 2018).
Distillation based CL approaches store one previ- ous model checkpoint of the model (noted as f t − 1 ) and regularize the differences between f t − 1 and the current model f t .
We adapt several existing knowledge distillation techniques to PTLMs and utilize them for continual learning.
We note, while individual distillation techniques are not original, their adaptation to CL algorithms can be novel.
We perform distillation with examples from the current domain D t and a replay memory M (similar to ER).
Despite the potential gap between D t and the training data of f t − 1 , the approach allows utilizing more data for distillation.
Formally, each time the model receives a mini-batch of stream examples x s or a draws mini-batch of memory examples x m from M (both noted as x ), we collect certain outputs of the model ( e.g. , output logits or intermediate representations) with f t − 1 and f t .
We compute a distillation loss (cid:96) KD ( x , f t − 1 , f t ) that penalizes the differences between the model outputs, and jointly optimize it with the masked language modeling loss (cid:96) MLM .
The ﬁnal objective is written as (cid:96) =
(cid:96) MLM + α(cid:96) KD , where α is a hyperparameter to weight the distillation loss.
Logit Distillation.
In logit distillation (Hinton et al., 2015), we collect the output logits of f t and f t − 1 , noted as y t and y t − 1 respectively.
The distillation loss is computed as D KL ( y t , y t − 1 ) , where
D KL is the Kullback–Leibler divergence function.
Representation Distillation.
We also consider minimizing the representational deviation of sentences between previous and current models (Sun et al., 2019; Jiao et al., 2020).
We extract the representation of each word of two models, noted as h 1: N t − 1 and h 1: N t , before the masked language modeling prediction head, where N is the length of the sentence.
Then, we compute MSE loss ||
h 1: N t − 1 − h 1: N t || 22 as the distillation loss.
Contrastive Distillation.
In addition to output logits and hidden representations, we further look into representational similarity within a batch of examples as additional knowledge to distill.
The approach is adapted from (Cha et al., 2021), which is originally studied for supervised image classiﬁcation tasks.
We brieﬂy introduce the adapted algorithm and leave the details in Appendix E. During continual pretraining, in addition to the language model pretraining objective, we add an unsupervised contrastive learning objective, namely the
SimCSE (Gao et al., 2021) objective to encourage sentence representations to reﬂect semantic similarities between sentences.
Then, we compute the intra-batch representational similarity matrices of sentence representations ( i.e. between each pair of examples in the mini-batch) with f t − 1 and f t , noted as B t − 1 and B t , and minimize the cross entropy loss (cid:96) distill
[Equation]
Self-Supervised Distillation (SEED).
SEED distillation proposed by (Fang et al., 2021) has a similar spirit as the contrastive distillation.
The only difference is that it distills representational similarity between the batch and a large set of other examples .
We leave the details of the algorithm in
Appendix E. We further combine SEED
Distillationwith logit distillation and refer to the approach as SEED-Logit Distillation.
Results
We summarize our ﬁndings over the created data streams.
We ask whether lifelong pretraining and continual learning algorthms are effective base on our evaluation protocol proposed in Sec. 2.3.
4. Experiment Settings
We use the RoBERTa-base model (Liu et al., 2019b), initialized with RoBERTa-base weights throughout the experiments.
We set the maximal sequence length to 128 and an effective training batch size of 2,048.
On the research paper stream, models are trained for 8 k steps in the ﬁrst domain and 4 k steps in the subsequent domains.
On the
Tweet stream, we train the models for 4 k steps in each domain.
These correspond to less than a single pass of data in each domain.
See Appendix A for detailed setups.
4. Domain Incremental Data Stream
As we introduced in Sec. 2.2, in the domain incremental research paper stream, we expect a model f t to perform well on all downstream tasks S 1 ..t from domains D 1 ..t .
In Table 2, we report the performance of models on all downstream tasks S 1 ..
T ﬁne-tuned from the ﬁnal pretraining checkpoint, f T .
We visualize more complete change of downstream task performance over different time steps of pretraining ( i.e., , f 1 , f 2 , f 3 , f 4 ) in Fig.
We also report the log perplexity of masked language modeling (MLM) in Table 2 as additional information.
With these results, we address the research questions below.
[Table]
Does lifelong pretraining help retain knowledge across different domain corpora?
We ﬁrst examine whether task-speciﬁc or lifelong pretraining improves performance over domain-speciﬁc downstream tasks.
Comparing Task-Speciﬁc LMs with
RoBERTa-base in Table 2, we notice consistent performance improvements, especially on Biomedical and Computer Science domains ( D 1 , D 2 ).
We also see Sequential Pretraining could consistently outperform RoBERTa-base.
However, the comparison between Sequential Pretraining and Task Speciﬁc
LMs are mixed: on D 1 , D 2 , D 3 , Sequential Pretraining could outperform Task-Speciﬁc LMs only except MNER; while on the earliest biomedical domain ( D 1 ), Sequential Pretraining achieves substantially lower performance.
From Figure 4, we see the performance of Sequential Pretraining on
Chemprot and RCT (from D 1 ) drops signiﬁcantly from t = 1 to 4 .
The results imply lifelong pretraining allows later domains to beneﬁt from knowledge transfer from earlier domains, but the performance on earlier domains is limited because of forgetting.
Does continual learning algorithms help retain knowledge in sequential pretraining?
Next, we compare different kinds of CL algorithms and investigate the effect of CL algorithms in alleviating forgetting and improving knowledge transfer.
Table 2 shows that Online-EWC slightly improves
MLM perplexity compared to Sequential PT, but brings no improvement to the ﬁne-tuning performance.
We hypothesize that regularization directly in the parameter space as in Online-EWC is not effective when the parameter space is very high dimensional.
Adapter improves downstream task
F1 scores on the bio-medical domain ( D 1 ) by 1.2% and 0.8%, but does not outperform Sequential Pretraining in other domains (similarly for Simple [Table]
[Table]
Layer Expansion approach), likely because a great portion of the model is kept frozen.
In contrast, the memory-replay based approach (ER) allows training the full parameters of the model and has been shown to be highly effective in continual learning of classiﬁcation tasks (Wang et al., 2019; Chaudhry et al., 2019).
However, we surprisingly ﬁnd that ER could hardly improve over
Sequential Pretraining except D 1 .
A similar pattern can be found in the MLM perplexity.
We hypothesize that the positive effect of example replay has diminished because of the overﬁtting to the memory examples.
Table 3 summarizes the effect of tuning hyperpameters in ER.
When we reduce the frequency of replay (from every 10 steps tosteps), the MLM performance improves, which implies reduced overﬁtting; however, the performance of downstream task performance does not improve.
When we increase the size of the memory | M | from 100 k to 10 M , the MLM perplexity also improves; still, there are still no improvements in downstream tasks.
It may imply ER itself is not an effective approach for continual pretraining.
Unlike ER, distillation approaches utilize richer information such as output logits or representation similarity to preserve past knowledge.
We ﬁnd either Logit KD or SEED-Logit KD to be most effective depending on the task, while Rep-KD and Contrastive-KD are less effective.
The best performing distillation approach improves F1 over
Sequential Pretraining on downstream tasks from
[Figure]
ER
[Figure] [Figure]
LogitKD
RepKD
[Figure] [Figure] ER Adapter
LogitKD [Figure] [Figure] [Figure]
D 1 , D 2 at least by 1.0%.
However, performance on
D 3 , D 4 , which come later in the data stream, does not improve over Sequential Pretraining, possibly because the distillation loss term makes the model rigid in obtaining new knowledge.
What is the gap between lifelong pretraining and multi-task learning across all the domains?
Multi-Task Learning refers to the ofﬂine training paradigm, which retrain PTLMs over all corpora ( D 1 ..t ) each time a new corpus D t becomes available.
We examine whether lifelong pretraining is comparable to multi-task pretraining in terms of performance.
From Table 2 and Figure 4, we see
Sequential Pretraining in general underperforms
MTL except for the ﬁnal domain.
However, certain
CL approaches, such as Logit-Distillation, could improve over MTL on all downstream tasks from the ﬁrst and the second domain.
We speculate the reason is that continual learning naturally provides a curriculum (Xu et al., 2020; Shi et al., 2015) to models where each individual task is easier to learn.
Downstream Training Instances
M a c r o F29.
RoBERTa-baseSequential-PT
MTLERLogit-KD
The results have a positive implication that lifelong pretraining is not only more computationally efﬁcient and requires less storage of past data, but may also improve the performance of pretraining.
Does lifelong pretraining make models more data efﬁcient?
In Table 5, we further examine the performance of ﬁnal pretrained models under different amounts of training examples.
We include full results in Appendix B. We ﬁnd in general, performance improvements are more signiﬁcant in the low-resource setup.
Computational Costs.
We analyze computational costs of different CL algorithms and present additional experiments with controlled computational costs.
We ﬁnd additional computational cost is necessary for performance improvement of distillation-based CL.
However, it is not possible to trade performance simply by investing more computation budget with arbitrary CL algorithms.
We leave detailed discussions in Appendix F.
4. Temporal Data Stream
We conduct analysis on pretraining PTLM on chronologically-ordered tweet corpora, to understand whether lifelong pretraining helps adaptation to the latest data and improves temporal generalization ability.
The results are summarized in Table 4.
Will LMs be outdated?
We compare the performance of Task-Speciﬁc (2014) to the Task-Speciﬁc models pretrained on the year of downstream datasets (noted as Task-Speciﬁc (Latest)) and notice consistent improvements in downstream tasks in 2018 and 2020 (ﬁrst two columns in Table 4).
Sequential Pretraining could also outperform the
Task-Speciﬁc (2014) model.
It veriﬁes that language models may get outdated, which can be addressed by task-speciﬁc or lifelong pretraining over the latest corpora.
Does lifelong pretraining help improve the downstream model’s performance on latest data?
We show that downstream model’s performance over
[Table]
later data ( D 3 , D 4 ) can be improved over TaskSpeciﬁc models when continual learning algorithms are applied.
From the ﬁrst two columns of Table 4, we see Logit-KD and SEED-KD improve Hashtag prediction score over data of years 2018 and 2020.
SEED-Logit KD further improves prediction F1 on Emoji prediction.
Note that these ﬁndings are in contrast to the research paper stream, where CL algorithms do not improve performance in the latest domain D 4 .
The reason can be the higher similarity between domains in the tweet corpora making the knowledge transfer easier, which is further discussed in Appendix H.
Does lifelong pretraining improve temporal generalization?
Temporal generalization evaluates downstream performance over latest test data when ﬁne-tuned over outdated training data.
We show lifelong pretraining brings clear improvement to temporal generalization.
From Table 4, we see even Sequential Pretraining could improve over the model pretrained merely on the year [-
 data(Task-Speciﬁc-]2020 data (Task-Speciﬁc (2020)) consistently.
We ﬁnd performance further improves with CL algorithms applied.
SEED-Logit-KD performs best in general on crossyear hashtag prediction tasks.
In crossyear emoji prediction, we ﬁnd Contrast-KD and SEEDKD perform best.
We also ﬁnd that SEED-LogitKD could slightly outperform Logit-KD.
Related Works Domain and Temporal Adaptation of Language Models.
Gururangan et al.
(2020) study adaptation of PTLMs to domain-speciﬁc corpora.
Arumae et al.
(2020) study algorithms to mitigate forgetting in original PTLMs, but does not investigate forgetting that happens over a sequence of domains.
Maronikolakis and Schütze (2021); Röttger and
Pierrehumbert (2021);
Luu et al.
(2021) proposes sequential pretraining over domains or emerging data, but did not investigate CL algorithms.
Several recent studies have demonstrated the necessity of adapting LMs over time (Lazaridou et al., 2021) while speciﬁcally focusing on factual knowledge (Dhingra et al., 2021; Jang et al., 2021).
Continual Learning Algorithms in NLP.
Continual learning in NLP has mainly been studied for classiﬁcation tasks.
An effective approach is to utilize a number of stored past examples (de Masson d’Autume et al., 2019;
Wang et al., 2020), or pseudo examples ( e.g., the ones generated with a
PTLM (Sun et al., 2020; Kanwatchara et al., 2021)).
Recent extensions of the algorithm (Chuang et al., 2020) perform knowledge distillation with generated pseudo examples.
Other lines of works focus on regularization over the sentence representations (Wang et al., 2019; Huang et al., 2021; Liu et al., 2019a) or directly merging models in the parameter space (Matena and Raffel, 2021).
Model expansion-based approaches (Liu et al., 2019a;
Pfeiffer et al., 2021), including learning domain speciﬁc expert models (Gururangan et al., 2021), are also actively studied.
Conclusion
In this paper, we formulated the lifelong language model pretraining problem and constructed two data streams associated with downstream datasets.
We evaluated knowledge retention, adaptation to the latest data, and temporal generalization ability of continually pretrained language models.
Our experiments show distillation-based approaches being most effective in these evaluation setups.
A limitation of the work is that it has not been fully addressed whether there exists a variant of distillation-based CL approach that consistently outperforms Logit-KD.
Based on the current observation, we conclude the performance of different
KD approaches for CL is highly task-dependent.
It asks for more future works into continual learning algorithms within the proposed problem setup.
. Multidomain pretrained language models for green NLP.
In Proceedings of the Second Workshop on Domain Adaptation for NLP , pages 1–8, Kyiv,
Ukraine.
Association for Computational Linguistics.
Michael Matena and Colin Raffel. 2021. Merging models with ﬁsher-weighted averaging. ArXiv , abs/2111.09832. Martin Müller, Marcel Salathé, and Per Egil Kummervold. 2020.
Covid-twitter-bert:
A natural language processing model to analyse covid-19 content on twitter.
ArXiv , abs/2005.07503.
Sheshera Mysore, Zachary Jensen, Edward Kim, Kevin
Huang, Haw-Shiuan Chang, Emma Strubell, Jeffrey Flanigan, Andrew McCallum, and Elsa Olivetti.
The materials science procedural text corpus:
Annotating materials synthesis procedures with shallow semantic structures.
In Proceedings of the 13th
Linguistic Annotation Workshop , pages 56–64, Florence, Italy.
Association for Computational Linguistics.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen.
BERTweet:
A pre-trained language model for English tweets.
In Proceedings of the
Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 9– 14, Online.
Association for Computational Linguistics.
Elsa A Olivetti, Jacqueline M Cole, Edward Kim, Olga
Kononova, Gerbrand Ceder, Thomas Yong-Jin Han, and Anna M Hiszpanski. 2020.
Data-driven materials research enabled by natural language processing and information extraction.
Applied Physics Reviews , 7(4):041317.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé,
Kyunghyun Cho, and Iryna Gurevych. 2021.
AdapterFusion: Non-destructive task composition for transfer learning.
In Proceedings of the 16th
Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 487–503, Online.
Association for Computational Linguistics.
Sylvestre-Alvise
Rebufﬁ,
Alexander
Kolesnikov,
Georg Sperl, and Christoph H. Lampert. 2017.
icarl:
Incremental classiﬁer and representation learning.
In 2017 IEEE Conference on Computer Vision and
Pattern Recognition, CVPR 2017, Honolulu, HI,
USA, July 21-26, 2017 , pages 5533–5542.
IEEE
Computer Society.
Shruti Rijhwani and Daniel Preotiuc-Pietro. 2020.
Temporally-informed analysis of named entity recognition.
In Proceedings of the 58th Annual
Meeting of the Association for Computational Linguistics , pages
Online
Association for
Computational Linguistics.
Anthony V. Robins re pseudorehearsal
Sci document classiﬁcation
Jonathan
Schwarz,
Wojciech
Czarnecki,
Jelena Luketina,
Agnieszka Grabska-Barwinska,
Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.
ICML 2018, Stockholmsmässan, Stockholm, Sweden
Research
PMLR.
Yangyang Shi, Martha Larson, and Catholijn M. Jonker.
Comput
Speech Lang. , 33:136–154.
Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee.
Addis Ababa,
Ethiopia, net.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu
Patient
BERT model
- pression
Empirical Methods in Natural Language Processing and the 9th
Natural Language Processing (EMNLP-IJCNLP) , pages 4323–4332, Hong Kong, China.
Association for
Computational Linguistics.
Jens Vindahl.
Chemprot-3.0: a global chemical biology diseases mapping.
Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo,
Shiyu Chang, and William Yang Wang.
Sentence embedding alignment for lifelong relation extraction.
In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 796–806, Minneapolis, Minnesota.
Association for
Computational Linguistics.
Zirui Wang, Sanket Vaibhav Mehta, Barnabas Poczos, and Jaime Carbonell. 2020.
Efﬁcient meta lifelonglearning with limited memory.
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 535–548,
Online.
Association for Computational Linguistics.
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan
Wang, Hongtao Xie, and Yongdong Zhang.
Curriculum learning for natural language understanding.
In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 6095–6104, Online.
Association for Computational Linguistics.
Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, and Furu Wei. 2021.
Adapt-and-distill: Developing small, fast and effective pretrained language models for domains.
In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 460–470, Online.
Association for Computational Linguistics.
[Figure]
A Detailed Experiment Settings
We use a linearly decreasing learning rate initialized with 5e-4 on the research paper stream and 3e-4 on the tweet stream.
On the research paper stream, we train the model for 8,000 steps in the ﬁrst task, and 4,000 steps in the subsequent tasks.
On the tweet stream, we train the model for 8, steps in all tasks.
We hold out 128,000 sentences from each corpus to evaluate MLM performance.
As the size of pretraining corpora is large, during training, each training example is visited only once.
We use the masked language modeling perplexity over held-out validation sets of the pretraining corpora as the metrics for hyperparameter tuning.
Common hyperparameters such as learning rate and batch sizes are tuned with Task-speciﬁc models with the ﬁrst task.
Hyperparameters that are speciﬁc to continual learning algorithms, such as the scale of the distillation loss, is tuned using the ﬁrst two domains in the stream according to the MLM performance over validation sets.
The weight of the distillation term α is set as 1.0 for logit distillation and 0.1 for other distillation algorithms.
By default, we replay or perform distillation with a mini-batch of examples from the replay memory every 10 training steps in ER and Distillationbased CL approaches.
We use the huggingface transformers library https://github.com/ huggingface/transformers for implementation.
B Low-Resource Fine-Tuning
Figure 6 summarizes the performance of ﬁne-tuned models from the ﬁnal model checkpoint ( t = 4 )
[Table]
using different amount of downstream training examples.
We see on Chemprot and SciERC, the beneﬁt of Sequential Pretraining over RoBERTa-base is more signiﬁcant in low-resource ﬁne-tuning setups.
Whenever Seqential Pretraining outperforms
RoBERTa-base, we notice Logit-KD could further improve over Sequential Pretraining.
C Full Results over the Tweet Stream
Tables 5 and 6 summarize full results over the
Tweet stream.
Compared to the table 4 in the main text, we add downstream performance over data from years 2014 and 2016 ( D 1 , D 2 ), and temporal generalization from year 2014 to 2020 ( D 1 → D 4 ).
D Dataset Details
The research paper stream consists of full text of 6.6M, 12.1M, 7.8M, and 7.5M research papers from the S2ORC (Lo et al., 2020) dataset.
We evaluate downstream ﬁne-tuning performance on two in-domain datasets for each research area:
Chemprot relation exaction dataset (Vindahl, 2016) and RCT abstract sentence role labeling dataset (Dernoncourt and Lee, 2017) for the bio-medical domain; ACL-ARC citation intent classiﬁcation dataset (Jurgens et al., 2018) and SciERC relation extraction dataset (Luan et al., 2018) for the
[Table]
computer science domain; relation extraction over
Synthesis procedures (Mysore et al., 2019) and named entity recognition over material science papers (MNER) (Olivetti et al., 2020) for material science domain; keyphrase classiﬁcation and hyponym classiﬁcation after ﬁltering out physics papers for the physics domain (Augenstein et al., 2017).
We report micro-averaged F1 on Chemprot,
RCT, MNER datasets following the evaluation metrics in the original work, and report macroaveraged F1 on all other datasets.
We use the ofﬁcial data splits for all datasets except for RCT, where we employ a low-resource training setup following Gururangan et al.
The pretraining corpora for the tweet stream consist of 25M tweets in each year.
For downstream tasks, we use a separate set of 1M tweets from each year to construct multi-label hashtag prediction (Gong and Zhang, 2016) datasets and singlelabel emoji prediction datasets (Barbieri et al., 2018).
We replace user names to special tokens.
For Hashtag prediction, the label space consists of tweets containing 200 most frequent hashtags in each year.
We independently sample 500 tweets per label (hashtag) as training, validation and test sets, which results 10 k examples in each of the data splits.
For emoji prediction, we construct 20way single-label emoji prediction datasets for each year following Barbieri et al.
(2018) with the 1M held out tweets.
We sample 5,000 tweets per emoji in each split, resulting in balanced datasets of the same size as the hashtag prediction datasets.
E Details of Continual Learning Algorithms E. Contrastive Distillation
During continual pretraining, in addition to the language model pretraining objective, we add a unsupervised contrastive learning objective, namely the SimCSE (Gao et al., 2021) objective, so that the similarity in the sentence representation better reﬂects the semantic similarity in the sentence.
We use the l 2 -normalized representation of the startof-sequence token at the ﬁnal layer as the sentence representation, noted as h .
Then, we distill the intra-batch representational similarity from the previous model f t − 1 to the current model f t .
Given a mini-batch of N examples x , we compute the representational dot-product similarity matrix between normalized sentence representations h between each pair of examples with f t − 1 and f t , noted as
B t − 1 and B t , where each element B ij is,
[Equation]
where τ is a temperature hyperparameter.
We specify a temperature τ t = 0 .
for the teacher model f t − 1 and a temperature τ s for the student model f t = 0 .
We compute the cross-entropy between
B t − 1 and B t as the distillation loss,
[Equation]
E. SEED Distillation
SEED distillation proposed by (Fang et al., 2021) has a similar spirit as the contrastive distillation with differences in the examples used for computing similarity matrices computes.
The algorithm distills representational similarity between the batch and a large set of other examples , maintained in an example queue Q .
As the number of target examples K can be much larger than the batch size, it allows distillation of richer information by regularizing similarities.
During pretraining, the method maintains a ﬁxed-size queue
Q to cache examples from the current domain
D t .
Given a mini-batch of training examples x , it computes cosine similarity between each pair of examples within the batch x and Q with f t −and f t , resulting in two similarity matrices B t − 1 ,
P y ∈ R | B |×| Q | .
Similar to the contrastive distillation, the distillation loss is the cross-entropy between two similarity matrices B t − 1 and B t computed in the same way as Eq. 2.
F Analysis and Controlled Experiments of Computational Costs
Computational cost is a crucial matter for online continual learning systems.
In this section, we analyze the computational costs of continual learning algorithms and perform controlled experiments of computational costs.
We quantify computational costs with the total number of forward ( C f ) and backward ( C b ) computations ( C = C f + C b ) over the PTLMs, which is easy to control; in practice, we ﬁnd the wall clock time of training was approximately linear to C .
We summarize the number of forward and backward passes and the wall clock time of training in Table 7.
In the visit of b batches from the training stream,
Sequential PT performs b forward and backward passes respectively over the PTLM, resulting in
C = 2 b .
Experience replay further replays 1 batch of examples every r steps over the training stream, which results in C =
(2 + 2 /k ) b .
In our main experiments, r is set to 10 (Sec. 3.3).
Logit-Distill and Rep-Distill require one additional forward pass over a frozen PTLM to compute the target of distillation, resulting in C
= (3 + 3 /k ) b .
Distillation algorithms that perform contrastive learning with SimCSE ( i.e. SEED-Distill and SEED-LogitDistill) additionally require one forward and backward pass using the same batch of examples with different dropout masks.
Therefore, for SEEDLogit-Distill, C = (5 + 5 /k ) b .
To control the number of forward and backward passes, we present approaches to compensate the lower computation costs compared to Distillation algorithms and one approach to shrink the computational cost of distillation algorithms: (1) for
Sequential PT, we train the models for 1.2 times more steps so that C = 2 .
4 b , noted as Sequential
PT b (cid:48)
=1 .
2 b ; (2) for ER, we increase the replay frequency k to 5 from the default setup 10, so that
C = 2 .
4 b .
We also decrease the cost of Logit-KD and SEED-Logit-KD by reducing the frequency of distillation from every 1 batch to every r (cid:48)
=
steps, while still replaying and distilling knowledge over 1 batch of memory examples every 10 training steps.
This results in C f
=
(1 + 2 /k + 1 /k (cid:48) )
b and C b =
(1 + 1 /k ) b , where
C = 2 .
4 b when both r and r (cid:48) are 10.
The approach is referred to as Sparse Logit-KD.
Finally, for SEED-Logit-KD, we remove the SimCSE loss from training and perform sparse distillation similar to Sparse-Logit-KD, which also results in C = 2 .
4 b .
The performance of the models is presented in
Table 8.
We notice that at the end of pretraining, increasing the number of training steps in Sequential
PT by 1.2 times does not lead to performance boost on the latest domain ( D 4 ), while the performance over tasks from earlier domains (Chemprot, ACLARC, SciERC) slightly dropped, possibly due to increased forgetting.
For ER, we notice replaying only slightly more frequently (ER k =5 ) than the default setup ( k =10) greatly increased the perplexity of MLM, implying signiﬁcantly increased overﬁtting to the memory; while the performance differences of downstream tasks compared to the default ER is mixed.
When we decrease the replay frequency of distillation, the performance on LogitKD and SEED-KD also decreased and does not outperform ER.
The results show additional computation costs can be necessary for continual learning algorithms such as Logit-KD and SEED-Logit-KD.
However, the results also show that there is no simple tradeoff between computational cost and performance.
We have seen that it is not always beneﬁcial to increase the number of training steps over the emerging data, as it increases forgetting in earlier domains.
Similarly, increasing the frequency of replay may lead to signiﬁcant overﬁtting to the replay memory.
Investigating into more effective continual learning algorithms, despite increased computation costs, allows us to obtain performance improvement that cannot be simply traded with more computation with arbitrary continual learning algorithms.
We leave more thorough studies into this topic as future work.
G Experiments with BERT on Tweet Stream After
In this section, we present an additional set of experiments on BERT-base (Devlin et al., 2019) model, which is originally pretrained with Wikipedia articles before 2019, with Tweets only after 2019.
The
[Table]
where k is the frequency of replay wall clock time is calculated over 4 k steps of training (which is the number of training steps of a single domain in the Research Paper stream) excluding the ﬁrst domain as no replay or distillation happens while learning the ﬁrst domain use 2 Quadro RTX 8000 GPUs for training
In the additional controlled experiments (described in Appendix.
F), we control the total number of forward and backward passes of different approaches.
This also yields approximately the same wall clock time for approaches
[Table]
[Table]
training corpora D 1 .. 4 consist of tweets from the ﬁrst half of 2019, the second half of 2019, the ﬁrst half of 2020, and the second half of 2020 respectively.
We accordingly construct hashtag prediction and cross-year hashtag prediction datasets.
The performance of downstream tasks ﬁne-tuned from the ﬁnal pretrained model is presented in Table [Table]
We see Sequential PT clearly outperforms BERTbase which is not continually pretrained, and that
Logit-KD generally improves hashtag prediction performance compared to Sequential PT except on the ﬁrst half of 2019.
We hypothesize the small temporal gap between D 1 .. 4 makes improvements less signiﬁcant than our main experiment setup.
We present temporal generalization performance in cross-year hashtag prediction tasks in Table 10.
Similarly, Logit-KD improves over Sequential PT
[Table]
in two out of three cross-year hashtag prediction setups.
H Analysis of Data Streams
In this section, we provide further analysis about the created research paper stream and the tweet stream.
We measure cosine distances d v of vocabulary distributions between each pair of different domains ( D 1 .. 4 ) and summarize the results in Figure 7.
The results indicate that the Tweet stream has a magnitude smaller vocabulary distribution gap between domains, which is in the scale of 1 e − 5 , compared to the research paper stream, which is in the scale of 1 e − 2 .
On the Tweet stream, we see the differences of vocabulary distributions align with the temporal gap between domains.
On the
[Figure]
research paper stream, we ﬁnd some domains to be more similar than others.
For example, Biomedical ( D 1 ) and Material Science domains ( D 3 ) have larger similarity in their vocabulary distributions, which explains general downstream performance increase on D 1 after the model is pretrained on D 3 (Fig. 4 (a,b)).
The differences in vocabulary distribution explain inconsistency in results between two data streams, speciﬁcally, whether lifelong pretraining improves downstream model performance on the latest domain, as we mentioned in Sec. 4.3.
Other than this, our main ﬁndings, such as the effect of distillation-based CL algorithms on reducing forgetting, are consistent over two datasets with such signiﬁcant differences in their changes of vocabulary distribution.
We believe it implies the conclusions in this paper should be reliable in diverse data streams.
I Ethic Risks
We would like to note that, in practice, continually pretrained models over real-world data streams would require identiﬁcation and removal of biased contents from pretraining corpora, which may affect the prediction of downstream models.
As
PTLMs are continuously updated, the bias in earlier pretraining may have a profound negative impact.
In future works, it is preferable to develop algorithms to “forget” certain biased knowledge from language models.
We further note that any data released in this paper, especially the tweet stream, should only be used for research purposes.