A N E MPIRICAL I NVESTIGATION OF THE R OLE OF P RETRAINING IN L IFELONG L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The lifelong learning paradigm in machine learning is an attractive alternative to the more prominent isolated learning scheme not only due to its resemblance to biological learning, but also its potential to reduce energy waste by obviating excessive model re-training.
A key challenge to this paradigm is the phenomenon of catastrophic forgetting.
With the increasing popularity and success of pre-trained models in machine learning, we pose the question: What role does pre-training play in lifelong learning, speciﬁcally with respect to catastrophic forgetting?
We investigate existing methods in the context of large, pre-trained models and evaluate their performance on a variety of text and image classiﬁcation tasks, including a large-scale study using a novel dataset of 15 diverse NLP tasks.
Across all settings, we observe that generic pre-training implicitly alleviates the effects of catastrophic forgetting when learning multiple tasks sequentially compared to randomly initialized models.
We then further investigate why pre-training alleviates forgetting in this setting.
We study this phenomenon by analyzing the loss landscape, ﬁnding that pre-trained weights appear to ease forgetting by leading to wider minima.
Based on this insight, we propose jointly optimizing for current task loss and loss basin sharpness in order to explicitly encourage wider basins during sequential ﬁne-tuning.
We show that this optimization approach leads to performance comparable to the state-of-the-art in task-sequential continual learning across multiple settings, without retaining a memory that scales in size with the number of tasks.
I NTRODUCTION
The contemporary machine learning paradigm concentrates on isolated learning (Chen & Liu, 2018)
i e., learning a model from scratch for every new task.
In contrast, the lifelong learning (LL) paradigm (Thrun, 1996) deﬁnes a biologically-inspired learning approach where models learn tasks in sequence, ideally preserving past knowledge and leveraging it to efﬁciently learn new tasks.
LL has the added beneﬁt of avoiding periodical re-training of models from scratch to learn novel tasks or adapt to new data, with the potential to reduce both computational and energy requirements (Hazelwood et al., 2018;
Strubell et al., 2019; Schwartz et al., 2020).
In the context of modern machine learning where state-of-the-art models are powered by deep neural networks, catastrophic forgetting has been identiﬁed as a key challenge to implementing successful LL systems (McCloskey & Cohen, 1989;
French, 1999).
Catastrophic forgetting is when deep networks forget knowledge learned in previous tasks as information relevant to the current task is incorporated, and mitigating or preventing this phenomenon is critical to achieving true LL.
At the same time, transfer learning (TL) has shown impressive results in both computer vision (CV; Zhuang et al. 2021) and natural language processing (NLP) applications (Howard & Ruder, 2018; Peters et al., 2018; Devlin et al., 2019).
The modern TL paradigm involves pre-training a ﬁxed architecture (like ResNet or BERT) using copious amounts of data, and then ﬁne-tuning the learnt parameters on target tasks.
Given the tremendous success of pre-trained models, there has been increased interest in understanding their role in improving generalization (Erhan et al., 2010;
Neyshabur et al., 2020), speed of convergence (Hao et al., 2019), successful transfer (He et al., 2019;
Pruksachatkun et al., 2020), and out-of-distribution robustness (Hendrycks et al., 2020; Tu et al., 2020).
Despite these efforts, the role of pre-trained initializations in lifelong learning settings has been under-explored.
In contemporary work, it has been shown that pre-trained models can be used as feature extractors (i.e., pre-trained weights are frozen) for task-sequential learning (Hu et al., 2021).
Because the pre-trained weights are explicitly frozen in this setting, the model undergoes no catastrophic forgetting.
In contrast, ﬁne-tuning pre-trained weights updates the pre-trained model parameters and is susceptible to severe forgetting.
This is typically the most accurate and thus common TL paradigm (Peters et al., 2019), and the one we consider in this work.
To the best of our knowledge, there is no prior work systematically analyzing the role of pre-trained initialization on catastrophic forgetting in lifelong learning scenarios.
[Figure]
Figure 1 shows that simply changing the network initialization to generic pre-trained weights can signiﬁcantly reduce forgetting on the ﬁrst task when doing sequential training on ﬁve tasks.
This observation motivates us to ask: Does pre-training implicitly alleviate catastrophic forgetting, and if so, why?
To answer this question we conduct a systematic study on existing CV and NLP benchmarks and observe that pre-training indeed leads to less forgetting.
We also investigate the effect of the type of pretrained initialization by analyzing the extent to which four pre-trained Transformer language model variants (Sanh et al., 2019; Devlin et al., 2019; Liu et al., 2019) undergo forgetting, observing that increasing the capacity of the model and diversity of the pre-training corpus play an important role in alleviating forgetting.
To further stress-test these models on a large number of diverse tasks, we introduce a dataset with 15 diverse NLP tasks and observe that forgetting consistently increases in this setting.
We hypothesize that pre-trained weights already have a good inductive bias to implicitly alleviate forgetting.
To explain this behavior we build upon two separate observations — Hao et al.
and Neyshabur et al.
(2020) show that in the context of transfer learning, pre-trained weights lead to a ﬂat basin in the loss landscape when ﬁne-tuning on a single task.
Mirzadeh et al.
(2020b) argues that the geometric properties of the local minima found for each task play an important role in forgetting, and they propose to modify the training regime (learning rate decay, batch size, dropout) to widen the tasks’ local minima.
To verify the above hypothesis, we analyze the loss landscape of the ﬁrst task while the model is training incrementally on subsequent tasks.
For pre-trained initializations, we see that minima obtained after training on a sequence of tasks still remain in the relatively low loss basin of the ﬁrst task when compared with random initialization.
These observations hint at the ﬂatness of the minima reached in the case of pre-trained initialized models.
To quantify the ﬂatness of the loss landscape, we evaluate the sharpness metric (Keskar et al., 2017) and verify that pre-trained weights indeed lead to ﬂat basins in comparison to random weights while training sequentially.
These analyses help us showcase that continual training from pre-trained weights induces wide task minima, which is shown to alleviate forgetting (Mirzadeh et al., 2020b).
To further alleviate forgetting when sequentially ﬁne-tuning the pre-trained weights, we explicitly seek ﬂat basins by minimizing the current task loss value and sharpness metric.
We use the existing Sharpness-Aware Minimization (SAM) procedure (Foret et al., 2021) for this purpose and report improved results across many experimental settings.
Our main contributions can be summarized as follows:
• We observe that initializing models with pre-trained weights results in less forgetting compared to random weights despite achieving higher performance on each task.
We bolster this observation with a systematic study validating that this behavior persists across applications (NLP, CV) and three existing approaches: Elastic weight consolidation (Kirkpatrick et al., 2017), A-GEM (Chaudhry et al., 2018), and episodic replay (Chaudhry et al., 2019).
• To understand the role of varying pre-trained initializations, we analyse a suite of pretrained Transformer language models and showcase that model capacity and diversity of the pre-training corpus do play a role in alleviating forgetting.
We also show that sequential training on diverse tasks is still challenging for models initialized with pre-trained weights by introducing a new, more challenging benchmark for lifelong learning in NLP consisting of 15 diverse NLP tasks.
• We hypothesize and verify empirically that pre-trained models alleviate forgetting as they have an implicit bias towards wider task minima.
The effect of these wider minima is that changes in weights from learning subsequent tasks result in a smaller change to the current task loss, which helps reduce forgetting.
We further show that explicitly seeking ﬂat basins during sequential ﬁne-tuning results in even less forgetting.
P RELIMINARIES 2. P ROBLEM S ETUP
We consider a setup where we receive a continuum of data from different tasks in sequential manner: ( x 1 , y 1 , t 1 ) , · · · , ( x i , y i , t i ) , · · · .
Each triplet ( x i , y i , t i ) consists of a task descriptor t i ∈ T , input data x
i ∈
D t i and target labels y i ∈
Y t i .
In our setup, we consider an explicit task descriptor t i because the same input x i can appear in multiple different tasks but with different labels.
For example, we can have a product review with positive sentiment along with the grammatical acceptability judgements.
Following Lopez-Paz & Ranzato (2017), we assume that the continuum is locally i.i.d, i.e., each triplet ( x i , y i , t i ) satisﬁes ( x i , y i )
iid ∼ P t i ( X, Y ) .
Based upon the observed data, our goal is to learn a predictor f : X × T → Y where we want to evaluate test pairs ( x, t ) from previously observed tasks (backward transfer), current task, and future tasks (forward transfer) at any time during the continual training of our model.
2. D ATASETS AND T ASKS
We perform extensive experiments on the widely adopted standard benchmarks (Chaudhry et al., 2019; Mirzadeh et al., 2020b; Wang et al., 2020) across both CV and NLP domains. Speciﬁcally, we use the following datasets:
Split CIFAR-100 This dataset is based on the CIFAR-100 image classiﬁcation dataset (Krizhevsky & Hinton, 2009).
The 100 classes are randomly split into 20 5-way classiﬁcation tasks, with each task containing 2500 train examples and 500 test examples.
Split CIFAR-
This dataset takes the ﬁrst 50 classes of the CIFAR-100 dataset, and randomly splits them into ﬁve 10-way classiﬁcation tasks.
Each task contains 5000 training examples andtest examples.
This dataset serves as a homogeneous counterpart to the diverse 5-dataset, so it was constructed to match the structure and length of 5-dataset.
5-dataset (Ebrahimi et al., 2020) consists of ﬁve 10-way image classiﬁcation datasets:
CIFAR- (Krizhevsky & Hinton, 2009), MNIST (LeCun, 1998), Fashion-MNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011), and notMNIST (Bulatov, 2011).
Split YahooQA This dataset is a 10-way topic classiﬁcation dataset (Zhang et al., 2015) to create ﬁve 2-way classiﬁcation tasks.
We randomly split topics into different tasks.
Each task contains around 279 , 000 training examples and 12 , 000 testing examples.
5-dataset-NLP consists of text classiﬁcation datasets (Zhang et al., 2015) from diverse domains and constitutes four tasks: (1) News classiﬁcation (AGNews, 4-way classiﬁcation); (2) sentiment analysis (Yelp, Amazon, 5-way classiﬁcation); (3) Wikipedia article classiﬁcation (DBPedia, 14-way classiﬁcation); and (4) question and answer topic categorization (YahooQA, 10-way classiﬁcation).
We follow the data processing procedure mentioned in (de Masson d’Autume et al., 2019) and have 115 , 000 training examples and 7 , 600 test examples per task.
15-dataset-NLP consists of 15 diverse text classiﬁcation tasks: (1) CoLA (Warstadt et al., 2019)
BoolQ (Clark et al et al et al., 2015); (6) Yelp (Zhang et al., 2015); (7) Event Factuality (Poliak et al., 2018); (8) Argument
Aspect Mining (Stab et al., 2018); (9) Explicit Discourse Marker Prediction (Prasad et al., 2019;
Kim et al., 2020); (10) QNLI (Wang et al., 2018); (11) Rocstory (Mostafazadeh et al., 2016); (12)
MNLI (Williams et al., 2018); (13) SciTail (Khot et al., 2018); (14) Implicit Discourse Relation Classiﬁcation (Prasad et al., 2019; Kim et al., 2020); and (15) Emotion Detection (Saravia et al., 2018). For more details on this dataset, see Appendix B. 2. E VALUATION
Let s t,τ denote the score (e.g., accuracy) on the task τ after training on task t .
After model ﬁnishes training on the task t , we compute the average accuracy ( A t ) , forgetting ( F t ) and learning accuracy ( LA t ) metrics as proposed by (Lopez-Paz & Ranzato, 2017; Riemer et al., 2019).
F t measures the inﬂuence of learning task t on the performance of all previously seen tasks τ, (1 ≤ τ < t ) .
LA t measures the learning capability when model sees the new task t .
Say we learn the t th task, then A t , F t and LA t are deﬁned as follows:
[Equation]
2. M ETHODS
We consider prominent approaches from the literature for our analysis.
We ﬁrst consider the ﬁnetune (FT) approach, where we simply ﬁne-tune the model on each task in sequence with no additional constraints on learning.
Elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) is a regularization-based approach that tries to mitigate forgetting by limiting learning for parameters important to previously learned tasks, as measured by the Fisher information matrix.
A-GEM (Chaudhry et al., 2018) and episodic replay (ER) (Chaudhry et al., 2019) methods augment the base model with episodic memory module which retains examples from the previously seen tasks.
We retain one example per task per class and randomly select examples for storage.
Stable SGD (Mirzadeh et al., 2020b) controls training dynamics by varying the learning rate, learning rate decay, dropout and batch size (see Appendix A).
(Prabhu et al., 2020; Hussain et al., 2021) show that simple
ER method outperforms all of the previous methods under realistic continual learning setting and we compare our results with ER.
D OES PRE - TRAINING IMPLICITLY ALLEVIATE FORGETTING ?
Having deﬁned the formal problem deﬁnition, evaluation metrics, and methods for alleviating the forgetting phenomenon, in this section we conduct experiments to tease apart the role of pre-training for lifelong learning.
We are interested in answering the following questions: (Q1)
How much does pre-training help in alleviating the forgetting?
(Q2)
Do pre-trained weights undergo similar forgetting on diverse (5-dataset, 5-dataset-NLP) and homogeneous tasks (Split CIFAR-50, Split YahooQA)?
(Q3)
How do different pre-trained initializations affect forgetting?
Experimental design.
To answer these questions convincingly, we select various datasets across text and vision domains.
For text classiﬁcation, we conduct experiments on Split YahooQA, 5dataset-NLP, and 15-dataset-NLP.
For image classiﬁcation, we consider Split CIFAR-50, Split
CIFAR-100, and 5-dataset.
We utilize the DistilBERT base (Sanh et al., 2019) architecture for text classiﬁcation and the ResNet-18 (He et al., 2016) architecture for image classiﬁcation.
To isolate the effect of pre-training, we consider two variants for each of these architectures: pre-trained models ( DistilBERT-PT , ResNet-18-PT ) and randomly initialized models ( DistilBERT-R , ResNet-18-R ).
For our study, we need to ensure that there are as few confounding factors as possible.
Therefore, we keep all other hyperparameters the same and vary only the initialization (for more details refer to
Appendix A).
To measure the severity of forgetting, we ideally want sufﬁcient training samples to ensure either a pre-trained model or randomly initialized model of the same capacity can achieve similar learning accuracy on each task.
To control for this behavior we either select a large training corpus whenever available (e.g., 279k examples for the Split YahooQA task) or run our experiments for multiple epochs (5 epochs for CV benchmarks unless explicitly speciﬁed).
Finally, the ResNet-18PT model was pretrained on a subset of Imagenet consisting of 733 classes where classes that were [Table] semantically similar to the classes in the CIFAR-100 dataset were removed (more details Appendix B).
3. H OW MUCH DOES PRE - TRAINING HELP IN ALLEVIATING FORGETTING ?
From Table 1, we see that models with pre-trained initializations (ResNet-18-PT, DistilBERT-PT) undergo signiﬁcantly less forgetting in comparison to models with random initializations (ResNet18-R, DistilBERT-R).
This trend holds across all three methods.
For text classiﬁcation tasks (Split
YahooQA, 5-dataset-NLP), we see that both models have comparable learning accuracy (LA) and signiﬁcantly less forgetting for DistilBERT-PT.
This can be completely attributed to the pre-trained initialization.
Now on 5-dataset, ResNet-18-PT (38.28) undergoes less forgetting when compared to
ResNet-18-R (51.51).
Speciﬁcally, despite task accuracy starting at a higher base for ResNet-18-PT, the absolute forgetting value is still lower compared to ResNet-18-R models .
Additionally, this effect also holds when considering a sequentially ﬁnetuned pre-trained model (with no additional regularization to alleviate forgetting) to a randomly initialized model trained with lifelong learning methods.
For example, on 5-dataset-NLP, sequentially ﬁnetuning DistilBERT-PT undergoes less forgetting ( 16 . 73 ) compared to competitive ER method ( 21 . 58 ) when applied to DistillBERT-R.
This raises an interesting research direction — explicitly focusing on learning generic features while training sequentially apart from just focusing on the forgetting aspect of lifelong learning.
3. D O PRE - TRAINED WEIGHTS UNDERGO SIMILAR FORGETTING ON DIVERSE AND HOMOGENEOUS TASKS ?
Most work in the lifelong learning literature evaluate algorithms on the Split MNIST, Split CIFAR-10,
Split CIFAR-100, and FewRelations benchmarks (Chaudhry et al., 2019; Wang et al., 2019), which are homogenous in nature; different tasks in these benchmarks are sourced from the same underlying data distribution.
From Table 1, we see that ResNet-18-PT does not undergo a signiﬁcant amount of forgetting when sequentially ﬁne-tuned on Split CIFAR-50, Split CIFAR-100 (homogenous tasks).
On Split CIFAR-50, forgetting is around 7 accuracy points.
Surprisingly, the competitive ER method also undergoes a similar amount of forgetting, thereby raising a question about the applicability of these datasets when studying forgetting in the context of the pre-trained ResNet models.
It may be possible to manually cluster tasks based upon semantic closeness, rendering severe interference to make these benchmarks more challenging (Ramasesh et al., 2020).
Given the generic nature of the pre-trained initialization, we ask: What happens when we train the model sequentially on diverse datasets, where diverse datasets are ones that span multiple sources?
To answer this question, we conduct experiments on 5-dataset and 5-dataset-NLP.
From Table 1, we empirically observe that pre-trained models are susceptible to forgetting when exposed to diverse tasks .
Particularly,
DistilBERT-PT undergoes a 16 .
73 point drop in accuracy when trained on 5-dataset-NLP.
Similarly,
ResNet-18-PT undergoes a 38 .
28 point drop in accuracy when trained on 5-dataset.
3. H OW DO DIFFERENT PRE - TRAINED INITIALIZATIONS AFFECT FORGETTING ?
To examine the impact of varying pre-trained initialization on forgetting, we choose to evaluate different pre-trained Transfomer models, DistilBERT (Sanh et al., 2019), BERT (Devlin et al., 2019),
RoBERTa (Liu et al., 2019), on text classiﬁcation tasks.
From the previous subsection, we observe that pre-trained models are relatively more susceptible to forgetting when sequentially training on diverse tasks.
In response, we conduct a thorough investigation on the 5-data-NLP dataset.
From
Table 2, we observe that when keeping the pre-training corpora the same and increasing the capacity of the model — DistilBERT (66M), BERT-base (110M), and BERT-large (336M) — we observe that larger models undergo less forgetting on sequential training of diverse NLP tasks.
Further, to explore the impact of the diversity of the pre-training corpora, we compare BERT-base (110M) with
RoBERTa-base (125M).
We observe that the RoBERTa-base model performs far superior to BERTbase, thus hinting at the necessity of diverse pre-training corpora to implicitly alleviate forgetting.
Further, to stress-test these models, we introduce a novel suite of 15 diverse text classiﬁcation tasks (for more details see Appendix B).
We observe that by increasing the number of tasks in the sequence, pre-trained models undergo severe forgetting.
Surprisingly, the RoBERTa-base model out-performs
BERT-Large despite having many fewer parameters.
We conclude based on the empirical results that diversity of pre-training corpora is highly relevant when it comes to easing catastrophic forgetting when training on a diverse sequence of tasks.
E XPLORING THE L OSS L ANDSCAPE
To better understand how pre-training reduces forgetting, we perform experiments analyzing where models are situated in the loss landscape after training on each task.
We denote model parameters after training on task k as w k .
If we deﬁne forgetting as the increase in loss for a given task during training (instead of decrease in accuracy), Mirzadeh et al.
(2020b) show that the forgetting can actually be bounded by:
[Equation]
where L 1 ( w ) represents the loss on Task 1 with parameters w , ∆ w = w 2 − w 1 , and λ maxis the largest eigenvalue of ∇ 2 L 1 ( w 1 ) .
The magnitude of the eigenvalues of L 1 ( w ) can be used to characterize the curvature of the loss function (Keskar et al., 2017), and thus λ maxcan be thought of as a proxy for the ﬂatness of the loss function (lower is ﬂatter).
From Equation 2, we can see that the ﬂatter the minima, the less forgetting occurs in the model.
We hypothesize that the one explanation of improvements from pre-training shown in the previous section might be because pre-training leads to a more favorable loss landscape.
Speciﬁcally, pretraining results in wider, ﬂatter minima for each task.
The effect of these wider minima is that the
[Table]
change in weights from learning on future tasks results in a smaller change on the actual loss for the current task, which leads to less forgetting.
We verify this idea in two parts.
First we use loss contours and then interpolate between model checkpoints to show that the ﬂat loss basins lead to smaller changes in loss.
Next we compute a sharpness metric to show that pre-training leads to ﬂat loss basins.
All models analyzed in this section are trained using the Finetune method.
4. L OSS C ONTOUR
In Figure 2, we visualize the contours of the test loss for the ﬁrst task.
On those contours, we plot the locations of the model ( w 1 , w 2 , w 3 ) after training on each of the ﬁrst three tasks.
Pre-training results in signiﬁcantly wider optima.
In fact, as the model is trained on tasks sequentially, the pre-trained model still remains mostly at the same loss level as compared to randomly initialized models, despite moving approximately the same (or even more) euclidean distance away from the original model.
For example, in the plot for the pre-trained model on 5-dataset (Figure 2e), the model after the second task ( w 2 ) remains at the same Task 1 loss level as after just training on Task 1 ( w 1 ).
It is a couple of loss levels higher for task 3 ( w 3 ).
For the randomly initialized model (Figure 2a), the euclidean distances between the model parameter vectors are approximately the same as for the pre-trained model, but the differences in Task 1 loss levels are signiﬁcantly higher.
We provide more instances of these visualizations along with loss interpolation plots in Appendix D.
4. S HARPNESS
As another measure of the wideness of the minima, we calculate a sharpness metric (Keskar et al., 2017) for the model on each task as it goes through training.
The metric tries to ﬁnd the maximum value of the loss in the neighborhood of the minima, and calculates the difference between the maximum and the minimum loss value, scaled by the loss value.
The maximization is performed in a subspace of the entire parameter space R n , speciﬁed by a projection matrix A ∈ R n × p .
For our experiments, we randomly sample our matrix A and set p = 100 as in Keskar et al.
The neighborhood of the metric is given by:
[Equation]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
(g) 5dataset-NLP w/PT 0.
www0.0 0.
(h) YahooQA w/PT
[Table]
where A + is the pseudo inverse of A , x is the parameter vector and (cid:15) is a hyperparameter controlling the size of the neighborhood.
The metric is then deﬁned as:
[Equation]
where f ( x ) denotes the loss value with parameters x .
We calculate the sharpness metric at (cid:15) = 5 × 10 − 4 and 10 − 3 .
After training on each task, we compute the sharpness values of the minimum reached by the model on that task.
We then take the sharpness value of the run to be the mean of the values across the sequence of tasks.
We present the mean and standard deviation across 5 runs.
The sharpness values for 5-dataset and Split-CIFAR50 are presented in Table 3 and for rest of the datasets see Table 6 and Table 7 (in Appendix C).
We see that for all datasets, the sharpness values for the pretrained initialized models are signiﬁcantly lower than the values for the randomly initialized models.
S HARPNESS A WARE M INIMIZATION FOR L IFELONG L EARNING
In the previous section, we looked at how initialization plays an important role in overcoming forgetting as pre-trained initializations favor ﬂat loss basins.
On the other hand, (Mirzadeh et al., 2020b) suggests modifying the training regime by varying learning rate decay, batch size, and dropout regularization such that inherent noise in the stochastic gradients leads to ﬂat basins in the loss landscape.
However, the procedure for tuning these hyperparameters before continual learning of tasks is ill-deﬁned, thereby rendering their strategy less helpful and quite expensive.
Doing a sweep over just the hyperparameter values suggested in the original paper requires 48 separate runs.
Furthermore, these hyperparameters cannot be reused across different architectures and datasets, and thus we need to do even more sweeps.
To alleviate these issues, we propose to explicitly seek ﬂat basins while sequential ﬁne-tuning of the pre-trained weights.
Towards this objective, we employ the
Sharpness-Aware Minimization (SAM) procedure (Foret et al., 2021) that seeks parameters that lie in the neighborhoods having uniformly low loss values by jointly minimizing the task loss value and sharpness metric.
SAM deﬁnes the sharpness of the loss function f at parameters x as:
[Equation]
5) where maximization region is an (cid:96) p ball with radius ρ for p = 2 in Equation (5).
SAM problem can be deﬁned in terms of the following minimax optimization:
[Equation]
The gradient of the result of the maximization problem can be approximated as
[Equation]
where
[Equation]
[Equation]
To make the optimization simpler, the second order term in the gradient is dropped, leaving us with
[Equation]
The full derivation of this gradient can be found in Foret et al.
From the results in Tables 1 and 2, we can see that SAM results in a consistent improvement in performance over non-SAM counterparts.
Simply adding SAM to the sequential ﬁnetune baseline makes the approach competitive with, and sometimes outperform, state-of-the-art baselines like ER and Stable SGD, and with minimal hyperparameter tuning required (we used the default value of the ρ = 0 .
05 parameter for all experiments).
Finally, since SAM is simply an optimizer, it can be combined with existing continual learning methods.
Combining SAM with ER results in a method that outperforms all existing baselines.
R ELATED AND F UTURE W ORK
We focus on several closely related lines of work studying the optimization and loss landscapes of deep learning and lifelong learning models here, and discuss some more distant related work in
Appendix E. Hao et al.
(2019) show that for single-task generalization, pre-training leads to wider optima for BERT models.
Keskar et al.
(2017) explore how larger batch sizes lead to sharper minima and worse generalization in the single-task learning setting.
Mirzadeh et al.
(2020b) look at how catastrophic forgetting can be impacted by the training regime, and show that certain hyperparameter settings produce wider minima which lead to less catastrophic forgetting.
Finally, Mirzadeh et al.
(2020a) compare minima that result from multitask learning and continual learning, and show that the the minima resulting from continual learning are linear mode connected to the optimal sequential multitask minima, but not to each other, which results in forgetting and a corresponding drop in performance.
These works all either explore the relation between pre-training and ﬂatness of minima in single-task settings, or between ﬂatness of minima and model generalization capability.
We extend this line of work by examining whether beneﬁts from pre-training can persist across training on several tasks, assessing the effects of pre-training on loss landscapes over the course of lifelong learning, and validating a hypothesis explaining the effects of pre-training on lifelong learning.
Based on our ﬁndings and results, a potential future work could explore where the multitask minima are in relation to the pre-trained initialization, as Mirzadeh et al.
(2020a) show that the sequential multitask minima are linear mode connected to minima after each task in lifelong learning.
The ﬂatness of the minima for every model starting from a pre-trained initialization could suggest a way to regularize the sequential training process with the pre-trained initialization such that that the model ends up at the multitask minima.
One ﬁnal takeaway from these results is that lifelong learning methods should focus on creating more general representations instead of simply reducing catastrophic forgetting, as more general representations appear to result in more robust learning.
E THICS S TATEMENT
Large neural network models are known to mimic and potentially emphasize biases found in the data, including biases that can negatively impact the individuals who interact with these increasingly ubiquitous models.
Our work does not analyze this aspect of these models as it pertains to lifelong learning, but without such analysis it is unknown whether lifelong learning could further exacerbate negative societal biases exhibited in these models.
We leave this important analysis to future work.
R EPRODUCIBILITY S TATEMENT
We include the code, and instructions needed to reproduce the main experimental results in the supplemental material.
We specify all the training details including the hyper-parameters for our method and baselines ones.
The relevant training details are provided in Sections 3, 4, 5 and Appendix
A.
Every experiment in the paper was run across 5 random seeds and is reported with error bars (See
Tables 1, 2 and Figures ?? and 3).
We cite all datasets and baselines used as a part of our experiments in Section 2.
In the supplemental material, we provide a way to aggregate our new 15-dataset-NLP.
A I MPLEMENTATION D ETAILS
Vision Experiments For all vision experiments, we use the full ResNet-
He et al.
(2016) architecture, with the ﬁnal linear layer replaced (the number of outputs corresponds to the total number of classes in all given tasks).
During inference, only the subset of outputs corresponding to the given task is considered.
All images are resized to 224 × 224 , and normalized with µ = (0 . 485 , 0 .
456 , 0 . 406) and σ = (0 . 229 , 0 .
We used a SGD optimizer with the learning rate set to . 01 for all methods (we did a hyperparameter search for both pre-trained and randomly initialized models and found the learning rate 0 .
01 resulted in a good learning accuracy for both pre-trained and randomly initialized models).
The batch size was set to 10 for the Split CIFAR-50 and Split CIFARexperiments and 64 for the 5-dataset experiments.
The memory per class for ER was set to 1, and the λ parameter for EWC was also set to 1.
For Stable SGD, we performed a hyperparameter sweep over the parameters speciﬁed in the original paper, namely:
• initial learning rate: [.25 (Split-CIFAR100-Random, Split-CIFAR50-Random, 5dataRandom), .1, .01 (Split-CIFAR100-PT, Split-CIFAR50-PT), .001 (5data-PT)] • learning rate decay: [0.9 (Split-CIFAR50-Random, 5data-Random, Split-CIFAR100-PT), 0.85 (Split-CIFAR100-Random, Split-CIFAR50-PT), 0.8 (5data-PT)] • batch size: [10 (all), 64] • dropout: [0.5 (5data-Random), 0.25 (Split-CIFAR100-Random, Split-CIFAR50-Random, Split-CIFAR100-PT, Split-CIFAR50-PT, 5data-PT)]
NLP Experiments For most of the text classiﬁcation experiments, we use the Transformer architecture based text encoder, DistilBERT-base (Sanh et al., 2019) to encode our input.
In a single sentence text classiﬁcation task, x t is an input sentence to be classiﬁed.
In a sentence-pair classiﬁcation task, concatenation of x 1 t and x 2 t sentences separated by a [ SEP ] symbol is considered as a input x t .
DistilBERT produces a contextual representation of each token in x t including a special beginning of the sentence token symbol [ CLS ] .
We use the representation of the [ CLS ] symbol from model as features for a linear task classiﬁer.
We have a separate classiﬁer for each task.
We mainly set hyper-parameters to default implementation from HuggingFace.
5 We use Adam as our optimizer, set dropout 0 .
1 , the base learning rate to 2 e − 5 , batch size to 32 and the maximum total input sequence length after tokenization to 128 .
For EWC, we set the regularization strength λ to 100 (as this ended up with comparable LA across other methods) and for ER, following (Chaudhry et al., 2019), the memory per class per task is set to 1 .
For SAM, we set ρ = 0 .
02 for all models (random as well as pre-trained) on 5-dataset-NLP and 15-dataset-NLP.
For SplitYahooQA we set ρ = 0 .
B D ATASETS
ImageNet pre-training corpus For a fair comparison between pre-trained and randomly initialized models, we wanted to explicitly control for and remove the overlap between pre-training and downstream tasks. Publicly available ResNet models are pre-trained on ImageNet that overlaps with CIFAR-100 in terms of class labels. Therefore, we make sure that the subset of the ImageNet corpus we use does not have any visually and semantically overlapping classes with the CIFAR-100 dataset.
We use the publicly available (Abdelsalam et al., 2021) two-level class hierarchies for ImageNet, where semantically and visually similar labels are grouped under one super-category.
We iterate over all CIFAR-100 labels and drop the complete super-category from ImageNet corresponding to each of these labels.
For example, CIFAR-100 contains a castle class and we have a building super-category in ImageNet that contains castle, palace, monastery, church, etc. .
We remove all building-related labels from our pre-training dataset.
In total, we remove 267 classes and pre-train the ResNet-18-PT model on the remaining subset of the ImageNet dataset.
[Table]
[Table]
5-dataset (Ebrahimi et al., 2020) consists of ﬁve 10-way image classiﬁcation datasets:
CIFAR- (Krizhevsky & Hinton, 2009), MNIST (LeCun, 1998), Fashion-MNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011)
[Table]
and notMNIST (Bulatov, 2011).
Table 4 reports statistics of the datasets in 5-dataset.
One of the objectives of our work is to study the role of different pre-trained initializations in lifelong learning.
To enable this study, we introduce 15-dataset-NLP , a novel suite of diverse tasks for lifelong learning.
It consists of ﬁfteen text classiﬁcation tasks covering a broad range of domains and data sources.
Although there exists a setup with 4 tasks spanning 5 datasets, 5-dataset-NLP (de Masson d’Autume et al., 2019), we show that our introduced benchmark proves more challenging (see Table 2 and Section 3.3) than the previous setup for the Transformer models (e.g., DistilBERT,
BERT, RoBERTa) considered in our study.
15-dataset-NLP benchmark consists of single sentence or sentence pair classiﬁcation tasks.
We design our benchmark from existing tasks such that (1) the overall dataset includes various domains, (2) different tasks are (dis)similar to each other, thereby, facilitating both transfer and interference phenomena.
All tasks under consideration differ in dataset size (from 8.5k-400k), so for our experiments, we only use between 8.5-14k training examples from each task.
Lifelong learning from highly imbalanced data is an interesting problem, and we feel that our introduced benchmark can be used to investigate this problem as well.
As our data is gathered from publicly available sources, for some tasks we do not have access to hidden test examples.
In such cases, we consider dev examples as test split and sample examples from train split for validation 6 .
We describe the tasks below and Tabledetails the evaluation metrics and train/dev/test split sizes for each task.
Linguistic acceptability aims at identifying whether the given sequence of words is a grammatical sentence.
The Corpus of Linguistic Acceptability ( CoLA ) ((Warstadt et al., 2019) consists of English sentences annotated with their grammatical judgements.
The data spans multiple domains, speciﬁcally books and journal articles.
Boolean QA is a reading comprehension task of answering yes/no questions for a given passage.
The Boolean Questions ( BoolQ ) (Clark et al., 2019) dataset consists of short passages with yes/no questions about the passage.
The questions are sourced from anonymous Google users and paired up with passages from Wikipedia articles.
Sentiment analysis is a binary classiﬁcation task of identifying the polarity (positive/negative sentiment) of a given text.
The Stanford Sentiment Treebank ( SST-2 ) (Socher et al., 2013) corpus consists of sentences from Rotten Tomatoes movie reviews annotated with their sentiment.
Paraphrase detection aims at identifying whether two sentences are semantically equivalent.
The Quora Question Pairs ( QQP ) corpus constitutes of question pairs from Quora 7 website annotated for semantic equivalence of question pairs.
Q&A categorization is a topic classiﬁcation task of categorizing question and answer text pairs into existing topics.
The Yahoo!
Answers Comprehensive Questions and Answers ( YahooQA ) (Zhang et al., 2015) corpus contains data corresponding to the ten largest categories from Yahoo!
Webscope program.
Review rating prediction is a ﬁve-way classiﬁcation task of predicting the number of stars the user has given in a review given the corresponding text.
The Yelp (Zhang et al., 2015) dataset contains business reviews obtained from the Yelp Dataset Challenge (2015).
Event factuality prediction is the task of determining whether an event described in the text occurred.
The factuality annotations from the Decomp corpus are recast into an NLI structure and we use the modiﬁed dataset from Diverse NLI Collection (Poliak et al., 2018).
Argument aspect mining is concerned with the automatic recognition and interpretation of arguments (assessing the stance, source, and supportability for a given topic).
The Argument Aspect Corpus ( AAC ) (Stab et al., 2018) has over 25,000 arguments spanning eight topics annotated with three labels (no argument, supporting argument, opposing argument).
Stab et al.
(2018) collected the data from web documents representing a range of genre and text types, including blogs, editorials, forums, encyclopedia articles.
The explicit discourse marker prediction task aims at classifying the discourse markers between sentences.
Speciﬁcally, words like ’and’, ’but’, ’because’, ’if’, ’when’, ’also’, ’while’, ’as’ mark the conceptual relationship between sentences ( DISCONN8 ) and are considered as labels for this task as discussed in (Prasad et al., 2019; Kim et al., 2020).
We use examples from the Penn Discourse TreeBank 3.0 marked for explicit discourse relationship for our experimentation
Question-answering NLI ( QNLI ) is a task adapted from the SQuAD by converting it into the sentence pair classiﬁcation task (Wang et al., 2018).
QNLI is a binary classiﬁcation task of detecting whether the context sentence contains the answer to the question.
Binary Sentence Ordering (BSO) is a binary classiﬁcation task to determine the order of two sentences.
This task is similar to pre-training objectives considered in recent works.
We use Roc Stories ( RocBSO ) (Mostafazadeh et al., 2016) corpus for constructing the dataset for this task.
Natural language inference (NLI) is a three-way classiﬁcation task of predicting whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral).
The Multi-Genre Natural Language Inference ( MNLI ) (Williams et al., 2018) corpus consists of sentence pairs from different sources (transcribed speech, ﬁction, and government report) annotated for textual entailment.
Multi-choice QA is a reading comprehension task wherein given a passage and question, models need to pick up the right option out of provides ones.
Khot et al.
(2018) cast the multiple-choice science exam questions into an NLI structure to convert them to the binary classiﬁcation task.
We use the SciTAIL (Khot et al., 2018) dataset released by them for our experimentation.
Implicit discourse relation classiﬁcation is a common task of identifying discourse relations between two text spans or arguments.
The Penn Discourse TreeBank 3.0 ( PDTB3L1 ) (Prasad et al., 2019; Kim et al., 2020) contains a hierarchical annotation scheme (top-level senses, ﬁne-grained level-2 senses) and we use top-level senses comprising of four labels (expansion, comparison, contingency, temporal) for our experimentation.
Emotion detection is a classiﬁcation task of detecting the emotions from a given text snippet.
We use Emotion dataset (Saravia et al., 2018) which contains Twitter messages with six emotions: anger, fear, joy, love , sadness, and surprise.
B. T ASK S EQUENCES
The task sequences for the Split CIFAR-50 and Split CIFAR-100 experiments were generated by randomly sampling classes without replacement for each task, similar to Chaudhry et al.
Thus, the sequences were different for every random seed, but since we ran each method with the same 5 seeds, each method was trained and tested on the same 5 sequences.
For Split YahooQA , we created 5 tasks by using disjoint groups of consecutive classes (e g.
These tasks were than randomly ordered for each task sequence, and each method was trained and tested using the same 5 random sequences.
For 5-dataset , we randomly select the following dataset orders:
Seq1 SVHN → notMNIST → Fashion-MNIST → CIFAR-10 → MNIST Seq2 SVHN → MNIST → notMNIST → Fashion-MNIST → CIFARSeq3 CIFAR-10 → SVHN → notMNIST → Fashion-MNIST → MNIST Seq4 notMNIST → Fashion-MNIST → CIFAR-10 → SVHN → MNIST Seq5 CIFAR-
→ MNIST → notMNIST → SVHN → Fashion-MNIST
For 5-dataset-NLP , we randomly select the following dataset orders (ﬁrst 4 are consistent with (de Masson d’Autume et al., 2019)):
Seq1 Yelp → AGNews → DBPedia → Amazon → YahooQA Seq2 DBPedia → YahooQA → AGNews → Amazon → Yelp Seq
Yelp → YahooQA
→ Amazon → DBpedia → AGNews Seq4 AGNews → Yelp → Amazon → YahooQA → DBpedia Seq5 YahooQA
→ Yelp → DBPedia → AGNews → Amazon
For 15-dataset-NLP , we randomly select and use the following 5 dataset orders: [Table]
Seq1 Decomp → BoolQ → AAC → Yelp → DISCONN8 → SST-2 → QQP → YahooQA → QNLI → RocBSO → MNLI → SciTAIL → CoLA → PDTB2L1 → Emotion Seq2 CoLA → QQP → MNLI → QNLI → Emotion → SST-2 → BoolQ → Decomp → AAC → SciTAIL → RocBSO → Yelp → PDTB2L1 → YahooQA → DISCONN Seq3 SciTAIL → BoolQ → SST-2 → AAC → DISCONN8 → YahooQA → QNLI → RocBSO → PDTB2L → Emotion → Decomp → MNLI → QQP → CoLA → Yelp
Seq4 DISCONN8 → QNLI → CoLA → YahooQA → AAC → SciTAIL → PDTB2L1 → Emotion → Decomp → RocBSO → QQP → Yelp → MNLI →
BoolQ → SSTSeq5 Emotion → SST-2 → RocBSO → YahooQA → AAC → MNLI → CoLA → DISCONN8 → QQP → QNLI → Decomp → PDTB2L1 → SciTAIL → Yelp →
[Table]
C S HARPNESS
The matrix A ∈ R n × p used for projecting the parameters onto a subspace is randomly sampled and then normalized row-wise.
Since this matrix is very large, the computation of the pseudo inverse
A + (required for calculating the bounds in Equation 3) is very memory intensive and unstable.
Instead, we directly calculate A + x by ﬁnding the least squares solution to Ab = x .
To ﬁnd the maximum referenced in Equation 4, we use the L-BFGS-B algorithm.
We set the maximum number of iterations for the algorithm to 10, and to speed up computation, we directly provide the gradients along with the loss to the algorithm, instead of using the default 2-point ﬁnite difference gradient estimation.
For ResNet-18 ( n = 11 M ), we set p = 100 .
However, for DistilBERT ( n = 66 M ) when we set p = 100 , we notice extremely small values for the sharpness metric.
With the increase in the number of parameters, n , we should ideally increase random subspace projection dimension p .
Setting larger p ( > 100) values for DistilBERT, however, leads to memory issues relating to allocating space for
A and computing the bounds (even with the more efﬁcient method discussed above).
So instead of evaluating the sharpness metric in a random manifold, we perform the maximization in the entire space R n (basically setting A = I n ).
According to Keskar et al.
(2017), when (cid:15) is small enough and
A = I n , the sharpness metric in Equation 4 relates to the largest eigenvalue of ∇ 2 f ( x ) .
In Table 7, we report sharpness values for DistilBERT on 5-dataset-NLP and Split YahooQA datasets for (cid:15) ∈ { 5 e − 5 , 1 e − 4 , 5 e − 4 } .
We see that values in the case of pre-trained models (w/ PT) are lower compared to randomly initialized models (w/o PT), thereby, validating the relative ﬂatness of the task minima in the case of pre-trained models.
/ PT) reduces the sharpness in comparison to DistilBERT-R (w/o PT).
[Table]
[Figure]
[Figure]
D L OSS L ANDSCAPE D.
L INEAR M ODEL I NTERPOLATION
Ideally [Figure]
to minimize forgetting, as the model sequentially trains on tasks [Figure] its loss on previous tasks would not change.
This would be satisﬁed if the loss surface between model checkpoints were ﬂat
[Figure]
To visualize this, we linearly interpolate between w 1 and the other checkpoints of the model, tracking the test loss on Task 1.
This can be interpreted as viewing a slice of the contour plots shown in
Figure 2 along the line that connects w 1 to each checkpoint.
The results are shown in Figure 3.
The pre-trained plots are shown in hues of blue, and the random plots are shown in hues of red.
These plots show that the pretrained initialized models experience a much more gradual increase in the loss compared to the randomly initialized models, even when interpolating to checkpoints created after training on several tasks.
In Figure 4 we present additional linear model interpolation visualizations.
Speciﬁcally, we track the Task 2 test loss as we interpolate from the model checkpoint after training on Task 2 (w 2 ) to the other checkpoints.
These plots show that the trends presented in Figure 3 holds for Task 2 as well, thereby verifying that pre-trained initialized models lead to ﬂatter task minima for subsequent tasks.
Pre-trained initialized models experience a much more gradual increase in the loss compared to the randomly initialized models.
D. L OSS C ONTOURS
In this section we present loss contours for Task 1 and Task 2 for all task sequences (refer to Section
B.1 for more details) for 5-dataset-NLP , Split YahooQA , Split CIFAR-50 , and 5-dataset .
In line with our observation from the sharpness and linear model interpolation analyses, pre-trained initialized models lead to ﬂatter task minima for subsequent tasks as well.
D.2. L OSS C ONTOURS : SAM AND ER
In Figure 5, we plot loss contours for Task 1 of Split CIFAR-50, under continual training from randomly initialized weights, and compare them across three different methods: Finetune, Finetune + SAM, and ER.
We notice that SAM leads to wide task minima (for Task 1) as compared to Finetune and ER methods. Moreover, from Table 3, we see that Finetune + SAM ( 14 . 95 ) undergoes relatively less forgetting in comparison to Finetune ( 23 . 68 ) and ER ( 20 . 63 ) methods. These results convincingly demonstrate the effectiveness of SAM optimization procedure over vanilla SGD optimization.
www0.
(a) Finetune 0.
www0. (b) Finetune + SAM 0. www0. (c) ER
[Table]
[Table] [Figure]
w
[Figure]
[Figure]
w
[Figure]
[Figure]
[Figure]
(c) Seq 3 w/o PT 0.
www 4 0.0 0.3 0.
(d) Seq 4 w/o PT 0.
www 4 0.0 0.3 0.
(e) Seq 5 w/o PT 0.
www 4 0.0 0.3 0.
(f) Seq 1 w/ PT 0.
www 4 0.0 0.3 0.
(g) Seq 2 w/ PT 0.
www 4 0.0 0.3 0.
(h) Seq 3 w/ PT 0.
www 4 0.0 0.3 0.
(i) Seq 4 w/ PT 0.
[Figure]
[Table]
www 3 0.0 0.2 0.
(g) Seq 2 w/ PT 0.
www 3 0.0 0.2 0.
(h) Seq 3 w/ PT 0.
www 3 0.0 0.2 0.
(i) Seq 4 w/ PT 0.
www 3 0.0 0.2 0.
(j) Seq 5 w/ PT
www
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
(a
[Figure]
Seq 1 w/o PT 0. www 4 0.0 0.2 0. [Table] [Table]
PT
[Table] [Figure] [Table] [Figure]
[Figure] [Figure]
[Figure]
PT
[Figure]
[Figure]
www 4 0.0 3.
(c) Seq 3 w/o PT 0.
www 4 0.0 3.
(d) Seq 4 w/o PT 0.
www 4 0.0 3.
(e) Seq 5 w/o PT 0.0 2.5 5.0 0.
www 4 0.0 3.
(f) Seq 1 w/ PT 0.0 2.5 5.0 7.5 0.
www 4 0.0 3.
(g) Seq 2 w/ PT 0.0 2.5 5.0 0.
www 4 0.0 3.
(h) Seq 3 w/ PT 0.0 2.5 5.0 7.5 0.
www 4 0.0 3.
(i) Seq 4 w/ PT 0.0 2.5 5.0 7.5 0.
www 4 0.0 3.
(j) Seq 5 w/ PT
[Table]
[Figure]
[Figure]
[Table]
[Figure]
E A DDITIONAL R ELATED W ORK
Transfer learning from generic pre-trained models has enabled signiﬁcant recent progress in ML (Zhuang et al., 2021).
This trend started in the CV ﬁeld with the ImageNet dataset (Deng et al., 2009).
Transfer learning in NLP has witnessed its own “ImageNet revolution” where large models pre-trained on self-supervised tasks have shown impressive results across many language understanding tasks (Peters et al., 2018; Howard & Ruder, 2018;
Radford et al., 2018;
Devlin et al., 2019; Raffel et al., 2019; Liu et al., 2019).
Lifelong learning approaches focus on the idea of mitigating the catastrophic forgetting phenomenon and can be categorized into three groups: (1) Regularization-based approaches that augment the loss function with extra penalty terms preventing important parameters learned on previous tasks from signiﬁcantly deviating while training on the new task (Kirkpatrick et al., 2017; Zenke et al., 2017); (2) memory-based approaches that augment the model with episodic memory for sparse experience replay of previous task examples (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018;
Wang et al., 2020) (3) network expansion-based approaches that dynamically expand the network based upon new tasks (Rusu et al., 2016; Aljundi et al., 2017; Sodhani et al., 2020).
We analyze regularization-based and memory-based approaches in this work.
Meta-learning involves creating models that learn to learn over time.
Several works have applied meta-learning approaches to the task of lifelong learning (Riemer et al., 2019;
Finn et al., 2019; Javed & White, 2019; Wang et al., 2020).
Caccia et al.
(2020) propose a two-phase continual learning scenario where the ﬁrst phase is pre-training (using MAML (Finn et al., 2017)) and the second phase involves continual deployment with task revisiting.
They make the point that in many scenarios (Lomonaco et al., 2019), it would be unrealistic to deploy agents with no pre-training in a lifelong learning setting.
Whereas some of these works do use pre-trained initializations for their models, many do not, and none have extensively studied the effect of pre-training on lifelong learning.