A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning
Abstract
With the success of pre-training techniques in representation learning, a number of continual learning methods based on pre-trained models have been proposed.
Some of these methods design continual learning mechanisms on the pre-trained representations and only allow minimum updates or even no updates of the backbone models during the training of continual learning.
In this paper, we question whether the complexity of these models is needed to achieve good performance by comparing them to a very simple baseline that we design.
We argue that the pre-trained feature extractor itself can be strong enough to achieve a competitive or even better continual learning performance on Split-CIFAR100 and CoRe 50 benchmarks.
To validate this, we conduct the baseline that 1) uses a frozen pre-trained model to extract image features for every class and computes their corresponding mean features during the training time, and 2) makes predictions based on the nearest neighbor distance between test samples and mean features of the classes; i.e., Nearest Mean Classiﬁer (NMC).
This baseline is single-headed, exemplar-free, and can be task-free by updating the mean features continually.
This baseline achieved 88 .
53% on 10-Split-CIFAR-100, surpassing most state-of-the-art continual learning methods, with all initialized by the same pre-trained transformer model.
We hope our baseline may encourage future progress in designing learning systems that can continually add quality to the learning representations even if they start from pre-trained weights.
Code is available https://github.com/Pauljanson002/pre-trained-cl.git
Introduction
Conventional machine learning models struggle to perform well when the i.i.d. assumption is violated in the real world, where data arrives sequentially from tasks with shifting distributions over time.
These models suffer from catastrophic forgetting of earlier tasks[15, 10].
Continual learning, in which the agent is expected to learn new tasks without forgetting the old ones, has been studied extensively as a potential solution to this issue.
Early approaches start training with a model from scratch, continually adapt the model for future tasks, and prevent forgetting by replaying the data, designing the penalization of the updates of the model parameters, and/or dynamically increasing model parameters to incorporate new knowledge.
The use of pre-training [24] on large-scale datasets, such as ImageNet-1k[23] and ImageNet-21k [22] has led to signiﬁcant advances.
With sufﬁcient and diverse data, the output features of a pre-trained model generalize well to a vast array of tasks, greatly increasing the performance of challenging learning scenarios such as continual learning.
As a result, some existing works directly deploy the pre-trained models as feature extractors, and apply continual learning techniques on the feature level.
These methods imply that pre-trained models provide general and coarse features that need to be task-speciﬁcally ﬁne-tuned.
However, [20] stated large-scale pre-trained models have excellent classiﬁcation performance to the data outside of the pre-trained distribution even without extra training.
We are questioning whether these pre-trained features are competitive for downstream tasks and whether sophisticated continual learning techniques are indeed needed to achieve a good performance on the studied continual learning benchmarks.
To answer these questions, we implement a simple baseline in continual learning.
We use a frozen pre-trained model to extract features for the training set and compute mean features to represent each class.
During the testing time, we make predictions based on the distance between the test samples and class mean features.
Our results are surprisingly outperforming SOTA methods using the same pre-trained backbone network.
We achieve 88 .
53 % Average Accuracy at the end of the learning sequence of Split CIFAR100 compared to 83 .
83% of L2P[27], and 83 .
23% on the evaluation task of
CoRe50 compared to 78 .
33% of L2P.
This implies that the pre-trained models provide quality and robust learning representations under distribution shifts.
We also present the experimental results on two more diverse benchmarks, 5-dataset and Split-ImageNet-R.
The results on these datasets also show competitive performance outperforming most methods.
Our proposed baseline is single-headed, task-free, and exemplar-free as an additional bonus.
We argue that this is a simple yet powerful baseline that every continual learning method should compare against.
The intentions of these properties are to make less use of task labels (which is unrealistic in practice) and replay data (which raises privacy concerns) in the techniques.
We observe that some prior works outperform this baseline, but at the sacriﬁce of one or more of these characteristics.
There are few comparable approaches with ours when all the conditions align.
We hope that this work sheds some light on examining the practicality of pre-training in continual learning and whether the new methods are improving the learned representation quality continually.
Related Work
Continual learning with pre-trained models Continual learning methods generally trained feature extractors from scratch and constrained the drift in feature representation [10, 16].
Recently, the usage of pre-trained models has attracted more attention in continual learning.
[2] viewed the ﬁrst task as a pre-training stage and froze the feature representation after the ﬁrst task.
[7] found that a larger size of data in the ﬁrst task and self-supervised pre-training helped to decrease catastrophic forgetting.
[12] empirically analyzed the effect of training the last layer with a ﬁxed feature extractor.
[18] studied foundation models and replay of frozen latent features to overcome catastrophic forgetting.
Recently [27] proposed to use prompt based ﬁne-tuning with a pre-trained transformer [5, 25] and compared it with other methods in the same initialization.
We adopt this strategy and point out that classiﬁer learning actually reduced the power of representations learned by the pre-trained model.
Task-aware/free continual learning Task incremental method requires the task identiﬁers of the samples during the inference, which reduces the practical application of these methods, whereas class-incremental methods do not require those.
However, task-aware/free methods suffer from class recency bias.
To overcome such recency bias, BiC[29] proposed to add a ﬁnal layer that reduces the ﬁnal bias.
LUCIR [9] proposed to use a balanced ﬁne-tuning to train the classiﬁer after freezing feature representation.
Our method also focuses on this problem and uses the simple nearest mean classiﬁer on top of pre-trained transformer.
Exemplar-free continual learning Earlier methods were proposed to store raw data, learned features, or generated features from previous tasks.
Since the replay mechanism such as ER [4] is orthogonal to architecture-based and regularization-based methods, they are widely adopted in other methods to improve performance.
Recently there has been an increased interest in exemplar-free continual learning to account for privacy concerns in storing raw samples and storage concerns.
iCaRL [21] introduced the method of using exemplars for continual learning and selected exemplars close to the class means.
Our baseline follows a similar strategy but only stores the class-mean in feature space which save the storage by reducing the number of latent variables to keep per class.
Methodology
Problem Setup We adopt the standard continual learning scenario where a model learns from a subset, x ti ∈ R w × h × c is an image input and
y ti ∈ Z is its corresponding label.
The goal of continual learning is to learn a function f θ which maps the input x to the label y from an arbitrary task seen so far.
We focus on two scenarios.
In the class-incremental setting, each subset D t contains a disjoint class label set.
In the domain-incremental setting, the subsets D 1 , . . . , D T share the class labels, but the input distributions varies over time.
Nearest Mean Classiﬁer (NMC)
We decouple the goal of continual learning f θ into two steps.
The ﬁrst step is to learn the representation h and the next is to learn the classiﬁer g .
We directly adopt a pre-trained vision transformer as our feature representation without training.
For the classiﬁer, inspired by [21, 17] we use the nearest mean classiﬁcation strategy.
During the training stage of task t , we calculate the mean features of a class in D t
[Equation]
where C k denotes the set of training samples belonging to class k .
Only class mean features are saved in the memory and will be used during evaluation.
At the test time of task t , the feature of a test sample is extracted by the pre-trained model, and the predicted class label is taken as the class whose mean features is the closest (over all the seen classes so far) to the feature of a test sample.
[Equation]
Experiments
We follow the experimental setup used in [27] to evaluate our method for a fair comparison.
We test our method in class incremental learning, where new sets of classes are introduced to the model, and in domain incremental learning, where classes remain the same and the domain changes; see Sec 3.
Datasets: We evaluate our baseline on four common continual learning benchmarks, Split- CIFAR100 [11] , 5-datasets [6] and Split-ImageNet-R [8] in the class incremental learning setting.
As proposed by [27] and [26].
Split-CIFAR-100 contains 10 tasks with 10 classes for each task.
The 5-dataset benchmark concatenates 5 datasets, MNIST, SVHN, notMNIST , FashionMNIST and CIFAR10, with each dataset forming one task.
Split ImageNet-R is a newly proposed dataset for continual learning by [26].
It consists of 200 classes which are randomly divided into 10 tasks.
It contains the same object types however presented in different styles such as cartoon , grafﬁti and origami.
These variations make the continual learning more challenging.
For the domain incremental learning setting, we use CoRe50 proposed by [14].
It contains 50 objects collected in 11 distinct domains(tasks).
8 domains were faced and learned incrementally while the test is performed on the remaining three domains.
Since a single test task is used, we do not report forgetting and joint training results in that scenario.
Evaluation Methods: For our approach ( Ours ), we employ the widely used ViT-B/16[5] model pre-trained on ImageNet-21k [24] provided by timm library [28].
We mainly compare our baseline with the recent L2P [27] which adopts the pretrained model as ours and learns a prompt pool with a prompt selection mechanism to modify the pretrained representations.
We also consider popular continual learning methods including regularization-based methods (LwF[13] , EWC [10] ) and rehearsal-based methods (ER[4], GDumb[19], BiC [29], DER++ [1] and Co 2 L [3]).
We present joint training results where the training data is, i.i.d.
distributed among the whole benchmark with no task split.
FT-frozen adds a fully-connected layer on top of the frozen feature extractor as the classiﬁcation head, and FT allows end-to-end training on the feature extractor.
Note that FT-frozen is different from our baseline, as we use NMC classiﬁer and build it incrementally.
Results: Table 1, 4, and 2 report the performance of continual learning in incremental setting in
Average Accuracy at the end of the learning sequence of Split-CIFAR100, Split-ImageNet-R, and the 5-dataset respectively.
Table 4 shows the results of the continual learning in domain incremental setting on CoRe50[14].
The results are grouped based on the use of replay sample.
Our simple baseline achieves the best performance on Split CIFAR-100 and CoRe50 while being the second best on Split-ImageNet-R and 5-dataset benchmarks.
Our results on Split-CIFAR-100 are even better than methods that use replay samples.
Concretely, our baseline achieves 88.53% with zero buffer size, which is 4 .
7% and 2 .
22% better than the runner-up and signiﬁcantly more complicated L2P with no exemplars and L2P with 50 exemplars/class buffer, respectively.
This suggests that pre-trained transformer offers a strong representation that achieves competitive performance.
We think that the main reason for a possible inferior performance could be the ineffective design of the
[Table] [Table]
All methods are initialized pretrained
[Table] [Table]
continual learning mechanisms compared to the robust features provided by the pretrained model.
Such a model, pretrained on a large and diverse dataset, may have already captured most of the distribution properties in the evaluation benchmarks.
Then a desired continual learning mechanism needs to further encourage the model to produce the task-invariant features speciﬁc to the deployed benchmark.
Our nearest neighbor baseline is also competent among exemplar-free methods on
Split-ImageNet-R and 5-datasets benchmarks, which is more diverse than CIFAR-100 and CoRe50.
However, methods with higher buffer sizes and ﬁnely designed continual learning mechanisms do improve the representations extracted from pretrained models.
Conclusion
In this work, we explore the representational capacity of large-scale pre-trained models in continual learning settings.
We provide simple nearest neighbor baseline experiments on four benchmarks, showing competitive performance to more sophisticated state-of-the-art continual learning methods which also leverage the same pretrained models.
We agree that using pretrained weights can be a reasonable practice even in continual learning.
However, to show real progress in continual learning systems, we need to focus more on building methods that can continually add quality to the learning representations.
A desired continual learning algorithm shall go signiﬁcantly beyond the knowledge embedded in the pretrained model.
Another important aspect is the considered benchmarks for evaluating continual learning methods.
Such benchmarks need to be sufﬁciently challenging and different from the data distributions employed for pretrained models.