Memory Efﬁcient Continual Learning with Transformers
Abstract
In many real-world scenarios, data to train machine learning models becomes available over time.
Unfortunately, these models struggle to continually learn new concepts without forgetting what has been learnt in the past.
This phenomenon is known as catastrophic forgetting and it is difﬁcult to prevent due to practical constraints.
For instance, the amount of data that can be stored or the computational resources that can be used might be limited.
Moreover, applications increasingly rely on large pre-trained neural networks, such as pre-trained Transformers, since the resources or data might not be available in sufﬁciently large quantities to practitioners to train the model from scratch.
In this paper, we devise a method to incrementally train a model on a sequence of tasks using pre-trained Transformers and extending them with Adapters.
Different than the existing approaches, our method is able to scale to a large number of tasks without signiﬁcant overhead and allows sharing information across tasks.
On both image and text classiﬁcation tasks, we empirically demonstrate that our method maintains a good predictive performance without retraining the model or increasing the number of model parameters over time.
The resulting model is also signiﬁcantly faster at inference time compared to Adapter-based state-of-the-art methods.
Introduction
Transformers [59], e.g. BERT [12], have shown their effectiveness in various natural language processing (NLP) tasks such as classiﬁcation [27], Natural Language Inference [43, 41] and Question
Answering [18].
Inspired by this achievement, some pioneering works have recently been introduced on adapting Transformers architectures to Computer Vision (CV).
Vision Transformers [13, 57] showed that a pure Transformer applied directly to a sequence of image patches can perform well on image classiﬁcation tasks.
Besides, some recent studies [69, 4, 32] showed that Transformers are generalized to new domains given only a few samples.
Transformers show a great ability to learn complex concepts but when confronted with a sequence of different tasks they tend to “overwrite” the previously learnt concepts.
In general, deep networks suffer heavily from this phenomenon, called catastrophic forgetting (CF) [37], impeding continual or lifelong learning.
In the last few years, a growing body of works attempted to tackle CF in continual learning (CL) [14, 21, 29, 48, 65, 67] but most of them are not able to meet the scale or accuracy requirements of real-world applications.
Moreover, adapting large-scale pre-trained Transformer models to downstream tasks via ﬁne-tuning is the method of choice in NLP applications, posing the need for methods that can directly work with pre-trained models instead of requiring the training of a new model from scratch [15].
In this work, we tackle both text and image classiﬁcation problems in a setting where the number of tags or classes associated to the input data grows over time.
In fact, the ability to continually extend the set of tags or classes used to categorize the content is a major problem in many applications.
For example, newspapers can tag news according to topics of interest such as “sport”, “politics”, “food” by using a pre-trained language model and reﬁning it using a few hundred pre-tagged articles.
New tags may appear over time, for example “COVID-19” was a completely unknown news category in 2019 but appeared frequently since it is emerged.
In these cases, retraining models from scratch is often impractical and can lead to inconsistencies in the labeling when compared to the one provided by the previous model.
In particular, we focus on incrementally extending classiﬁers based on pre-trained Transformer models given their ubiquity in NLP and the growing interest in CV.
To address the issue of incremental ﬁne-tuning of pre-trained Transformers in the sequential learning setting without CF, we propose Adaptive Distillation of Adapters (ADA).
ADA leverages
Adapters [22], a specialized neural network module that adds new parameters to the neural network and a distillation mechanism to consolidate the new information with the previously learnt knowledge in a ﬁxed amount of parameters with little amount of forgetting.
This method allows the user to control the memory consumption, while retaining state-of-the-art performance when running the algorithm on sequences of tens of tasks.
This tight memory control is important in industrial applications.
The alternative, a model growing in size with the number of tasks, would require a change in hardware to adapt to the growing memory requirements of the deployed model.
This would be problematic since a practitioner will incur into higher risk of system instability and be forced to make conservative hardware choices.
The main contribution of our work is ADA, an algorithm that can achieve high predictive performance on both text and image classiﬁcation in different continual learning scenarios.
ADA also provides lower inference time and uses an order of magnitude fewer parameters than state of the art methods such AdaptersFusion [41].
Additionally, we implemented Adapters for vision Transformers and empirically demonstrated their effectiveness.
Related Work
Adapters [22] were proposed for ﬁne-tuning of pre-trained language models and were studied for the multi-task setting.
AdapterFusion [41] provides state-of-the-art performance by composing the pre-trained Adapters and it can simply be repurposed for preventing CF in CL by learning one Adapter for every new task.
While it has been shown that the number of additional model parameters per Adapter is signiﬁcantly smaller than the number of parameters used in the pre-trained model [22] (e.g., 3 .
6% of the parameters of the pre-trained model), since both Adapters and AdapterFusion require to store all the model parameters, the memory consumption increases rapidly with the number of tasks.
In the case of a model being trained on 30 tasks, we would have to add more parameters than the number of pre-trained Transformer parameters (details in Section 5). The linear increase in memory and storage consumption making the method unsuitable for CL.
Recent work studied catastrophic forgetting [55, 10, 27, 35] and incremental learning [66] for NLP and CV [32, 15].
Pasanuru et al.
[40] focus on the few-shot setting where only a few data points are available for each task.
Ke et al.
[28] proposed an architecture to achieve both CF prevention and knowledge transfer.
This method has some similarity to AdapterBERT [22] since they insert
a
CL plug-in module in two locations in BERT.
A CL-plugin is a capsule network [52] that uses one separate capsule [20] (2-layer fully connected network) for each task, and like Adapters, memory increases linearly over the time.
In addition, this algorithm requires to learn task masks to address knowledge transfer, which is costly to compute.
Among those recent works, only a few [32, 15] have applied the Transformers architecture to CL on image datasets.
In [32], for each new task, the model is copied and ﬁxed to be used as the teacher model in the distillation phase.
The student model is trained on both new task samples together with the knowledge distillation loss that uses samples from old tasks which is stored in the rehearsal memory.
In [15], the authors aim to learn a uniﬁed model that will classify an increasingly growing number of classes by building upon a new architecture.
However, they need to train a new Transformer, where the process is very costly and contrast with our goal of using public pre-trained models.
To the best of our knowledge there is no method able to leverage public pre-trained Transformers while keeping the number of model parameters constant while retaining state-of-the-art predictive performance.
Problem Setup.
A sequence of classiﬁcation tasks { T 1 , . .
, T N } are given where each task T i contains a different set of data sample (text or image)-label training pairs ( x i 1: t , y i 1: t ) and contains c new classes namely Y i =
{ Y 1 i , . . .
, Y ci } with t examples for each new class.
The goal of the learner is to learn a set of parameters ˜ ⇥ such that 1 N
P i 2 { 1 ,...,N } loss ( T i ; ˜ ⇥ ) is minimized.
The task identiﬁer is provided to the learner with every new batch of data.
Moreover, in our speciﬁc case, ˜ ⇥ is composed of a set of parameters ⇥ provided by a pre-trained model and, depending on the algorithm, some additional parameters which need to be learned for each speciﬁc task.
In its simplest case, this additional set of model parameters can just be a head model, but some algorithms use signiﬁcantly more elaborate functions.
In the case of the labeling application described in Section 1, each task represents a label and the learner creates a new binary classiﬁer for each label.
For the training of task T i , the learner can only access the newly added examples and label names in this task.
To evaluate the learner, the test data consists of examples across all the previous tasks, where the potential label space for the test example is Y 1: c[ Y 1: c[ · · · [ Y 1: c N .
All methods that we deﬁne in the following sections receive as input a pre-trained model f ⇥ ( . ) , e.g., BERT [12], that is able to extract high quality representations from the input data.
Adapters.
Adapters were proposed by [22] as an alternative to ﬁne-tuning in NLP.
They add new modules between layers of a pre-trained network called Adapters .
These modules are feed-forward layers that project the original feature size to a smaller dimension and projects them to the original size thereafter, ensuring that the number of parameters stays substantially small as compared to the original model.
(See Appendix A.4 for the details of the Adapter architectures.)
Adapters share the pre-trained model parameters ⇥ across all tasks and introduce a small number of task-speciﬁc parameters  i without affecting previous ones.
The model is initialized with parameters of a pretrained model ⇥ .
For each of the task i 2 { 1 , . . . , N } where N is the total number of tasks, a set of new and randomly initialized Adapter parameters  i are introduced.
The parameters ⇥ are ﬁxed and only the parameters  i are trained when a new task is added.
This makes it possible to train
Adapters for all N tasks, and store the corresponding knowledge in designated parts of the model.
The objective for each task i 2 { 1 , . . .
, N } is of the form:  i arg min  L
i ( D i ; ⇥ ,  ) .
AdapterFusion [41], has been proposed to mitigate the lack of knowledge sharing across tasks.
It works in two phases: i) in the knowledge extraction stage, adapters, which encapsulate the taskspeciﬁc information, are learnt for each of the N tasks; while ii) in the knowledge composition stage, the set of N Adapters are combined by using additional parameters .
The additional parameters i for task i are deﬁned as: i arg min
L i ( D i ; ⇥ ,  1 , . . .
,  i , ) .
While this provides good predictive performance, in the CL setting, new tasks are added sequentially and storing a large set of
Adapters  1 , . .
,  N is practically infeasible.
Adaptive Distillation of Adapters (ADA)
To address the issues we mentioned in the previous sections, we propose Adaptive Distillation of
Adapters (ADA).
ADA keeps a ﬁxed amount of Adapters in memory and takes transferability of representations into account to effectively consolidate newly created Adapters with previously created ones.
ADA works in two steps: i) it trains a new Adapter and classiﬁcation head, which we refer as the new model, using the training dataset of the new task; ii) it consolidates the old model with the new model.
To better control the memory usage, ADA has a ﬁx budget for the number of Adapters
K that are stored in a pool of old models.
In the consolidation phase, the algorithm selects one of the models in the pool using scores that quantify the transferable information contained in the representations they provide.
In the following sections, we explain the components of ADA and how they work.
4.1 Distillation of Adapters
For each new task T n , the Adapter parameters  n are added to the model, while the pre-trained model parameters ⇥ are kept frozen and are never changed.
Only the task-speciﬁc Adapter parameters  n and the head model parameters h n are trained for the current task.
The model f n ( x ; ⇥ ,  n , h n ) , with parameters ⇥ ,  n and h n is called the new model.
The head model parameters are ﬁxed after training the new model and they are not updated during model consolidation.
When a prediction for a task T i is required, the corresponding Adapter   ( i ) and head model h i is called.
is a mapping from the task id to the corresponding Adapter in the pool or to the newly trained Adapter.
We abuse notation deﬁning f as the function returning the output ( logits ) on all tasks:
[Equation]
For the consolidation step, an Adapter from the pool  is selected as explained in Section 4.2 and new collection of Adapter pool  0 is created where the selected Adapter is replaced with  c .
c denotes the consolidated model parameters and preliminarily the parameters are randomly initialized.
Similarly, a copy of  is created to map the old tasks that are associated to the selected Adapter and the new task n to  c .
The consolidation then has the following objective:
[Equation]
where D distill denotes the unlabeled training data used for distillation, and the distillation loss is computed as the difference between the logits produced by the existing specialist models denoted by f ( x i ; ⇥ ,  , h ) and the consolidated model denoted by f ( x i ; ⇥ ,  0 , h ) based on L 2 loss.
After  c has been trained,  is swapped with  0 .
This is a high-level view of the mechanism, our implementation is optimized to avoid copying models when not necessary.
This procedure follows the double distillation loss [71] which is originally proposed for class incremental learning to train a new Adapter that is used with the pre-trained model to classify both old and new tasks.
The main idea is ﬁrst training a separate model for the new class(es) using labeled data, and then combining the new and old models using unlabeled distillation data via a double distillation training objective.
We generalize this solution to our case where we have a set of teacher models kept in Adapter pool  and train a student model  c .
Double distillation procedure and the alternative solutions for distillation are discussed in Appendix A.1 but this solution was the best performing one in our experiments.
While several different data sources can be used to populate the buffer, such as using auxiliary external data [71] or generating synthetic data [9], in this work we populate the buffer using covariates from previous tasks selected with Reservoir Sampling [60].
This simple mechanism may not be the most effective, but it will guarantee that no advantage is given to ADA in the experimental comparison.
4.2 Adapter Selection for Distillation
In the previous section, we assumed the Adapter to be consolidated as given but ADA keeps a pool of
Adapters and the selection of the Adapter to be distilled is an important part of the algorithm.
In fact, our empirical observations show that a random selection of the Adapter provides poor performance (see Section 5.3).
The intuition behind our selection mechanism is the following: since a specialized head for every task is created, we can assume that when the features provided by the associated
Adapter are highly informative, the updates (i.e., the gradients applied) will be small.
At the same time, training a new head with every Adapter in the pool in order to observe which one is the most effective would increase the amount of computation required and signiﬁcantly impact the usability of the method.
The problem of computing the information carried by a representation in an efﬁcient manner has been already studied in the transfer learning community [3, 58, 56].
While, the aim of that research is completely different and, to the best of our knowledge, there is no clear relation between transferability and forgetting, the mathematical foundation of this work are closely related to our intuition.
In fact, scores like TransRate [24] employ the mutual information between the features provided by a pre-trained model and the target labels for the task at hand.
When the mutual information is high, the transferability is high.
More speciﬁcally, the knowledge transfer from a source task T s to a target task T t is measured as:
[Equation]
where Y are the labels of target examples and Z = f
( X ; ⇥ ,   ( s ) ) are features of them extracted by the pre-trained model and the Adapter associated to the source task.
TransRate is not the only score designed to quantify transferability between a pre-trained model and a new task: Log Expected Empirical Prediction (LEEP)
[38] is a well-known alternative.
Also in this case, the score was designed with a different application in mind, but it leverages the conditional distribution of the target label given the source label to quantify the how informative the information provided by the source model is.
Speciﬁcally, LEEP is a three steps method.
At Step 1, it computes dummy label distributions of the inputs f ( X ; ⇥ ,   ( t ) , h t ) in the target data set
D .
At Step 2, it computes the empirical conditional distribution ˆ P ( y | z ) of target label y given the source label z .
At
Step 3, it computes LEEP using f ( X ; ⇥ ,   ( s ) , h s ) and ˆ P ( y | z ) :
[Equation]
where z is a dummy label randomly drawn from f ( X ; ⇥ ,   ( s ) , h s ) and y is randomly drawn from ˆ P ( y | z ) .
We selected TransRate and LEEP for their simplicity and their ability to provide a quantiﬁcation without training but practitioners can replace these scores with different ones as they see ﬁt.
4.3 Algorithm
Algorithm
Require: ⇥ : pre-trained model, K : adapters pool size Freeze ⇥ and create
=
Map () for n 1 to N do A task T n is received
Initialize  n Process T n , train new model f n ( x ; ⇥ ,  n ,h n ) Sample from T n and add to distillation data
D distill if n  K then Store f n in the pool else j ⇤ argmax j 2 { 1 ,...,K } TranScore( T n ,f j ) Add ( n,j ⇤ )
to
Consolidate model: f j ⇤ =
Distill( f j ⇤ ,f n , D distill ) end if
Serve predictions for any task i  n using f  ( i ) end for
Distill( f i ,f j , D distill ) : Get soft targets
ˆ y i from old model f i with D distill
Get soft targets ˆ y j from new model f j with D distill Initialize  c
Compute distillation loss and train model f ( x ; ⇥ ,  c ) as de- ﬁned in Eq.return f
ADA is detailed in Algorithm 1.
The graphical workﬂow of the algorithm is shown in Appendix 7.
For every new task, the algorithm trains a new adapter and head model (called  n and h n ).
If the adapters pool did not reach the maximum size yet (controlled by K ), it just adds it to the pool.
If the pool reached the maximum size, the algorithm is forced to select one of the adapters already in the pool and distill it together with the newly trained one.
In order to select which adapter to distill, ADA uses the transferability scores (e.g., LEEP or TransRate).
Once the adapter in the pool with the highest transferability score (called f j ⇤ ) is identiﬁed, it consolidates that adapter and the newly trained one into a new adapter and replaces the old one present in the pool.
In order to be able to make effective predictions, the algorithm also keeps a mapping  of which adapter in the pool must be used in combination with each of the task-speciﬁc heads.
Experiments
In this section, we empirically validate our adapter distillation approach on text and image classiﬁcation tasks and show that ADA achieves similar performance to AdapterFusion while consuming signiﬁcantly less memory.
We dedicate Section 5.3 to ablation studies providing further insights into the mechanisms implemented in ADA and their contribution.
Datasets and experimental setup.
We use three text datasets for multi-label text classiﬁcation: Arxiv
Papers [68] (paper classiﬁcation), Reuters (RCV1-V2) [31] (news classiﬁcation), Wiki-30K [73] (Wikipedia article classiﬁcation) and two dataset for image classiﬁcation: CIFAR100 [30] and
MiniImageNet [51].
Details about the datasets are given in Appendix A.3.
For the multi-label text classiﬁcation experiments, we ﬁrst sample a sequence of labels from the label space.
Then, we create a balanced binary classiﬁcation task for each label by sampling the same amount of positive data points from the label considered and negative data points from the labels preceding the current one in the sequence.
After splitting the data in training and test set, we provide the algorithm with the training set and subsequently measure its performance on the test set.
The algorithm never observes any data point in the test set and, more generally, every data point in the dataset is used only once.
For Arxiv Papers and Reuters datasets, we created 20 tasks and for
Wiki-30K 60.
We ﬁxed the number of training samples per task to 100.
The test set consists ofdata points on Reuters and of 100 data points on Arxiv and Wiki-30K.
For image classiﬁcation , we design two scenarios.
In the ﬁrst scenario, each new task is a balanced binary classiﬁcation problem.
Each class can be selected to be the positive class only once.
In the second scenario each task is a balanced multi-class classiﬁcation problem with 5 classes.
In both cases we provide the learner with 50 data points per class both at training and test time: in the ﬁrst scenario each task will have a training set of 250 data points and in the second case of 100 data points.
The total number of tasks is ﬁxed to 20 for both scenarios.
The distillation memory size is ﬁxed to 1000 for Wiki-30K which has a larger number of tasks, and to 500 for the others.
Metrics.
In [36], three metrics that we discuss in the following are deﬁned to evaluate the performance of a CL method.
We use these metrics to evaluate our methods.
It is considered that we have access to a test set for each of the N tasks in { T 1 , . . .
, T N } .
After the model ﬁnishes learning about the task T i , its test performance are evaluated on all N tasks.
By doing so, a matrix R 2 R N ⇥ N is constructed where R i,j is the test classiﬁcation accuracy of the model on task T j after observing the last sample from task T i .
Letting ¯ b be the vector of test accuracies for each task at random initialization, the three metrics are deﬁned: i) Average Accuracy = 1 N
P
Ni =1 R N,i , ii) Backward Transfer (BWT) =
N
P
N  1 i =1 R N,i  R
i,i and iii)
Forward Transfer (FWT) =
1 N
P
Ni =
2 R
i  1 ,i  ¯ b i .
(The larger these metrics, the better the model.)
All the results in this section are averaged over 5 runs.
Baselines.
We compare ADA the following baselines.
1) Fine-tuning head model (B1) :
We freeze the pre-trained representation and only ﬁne-tune the output layer of each classiﬁcation task.
The output layer is multiple-head binary classiﬁer that we also use for the other methods.
Fine tuning the full model (B2) :
We ﬁne-tune both the pre-trained representation and the output layer for each classiﬁcation task.
3) Adapters [22]: We train and keep separate Adapters for each classiﬁcation task as well as the head models.
4) AdapterFusion [41]: It is a two stage learning algorithm that leverages knowledge from multiple tasks by combining the representations from several task Adapters in order to improve the performance on the target task.
This follows exactly the solution depicted in Section 3. 5) Experience Replay (ER) [50]: ER is a commonly used baseline in Continual Learning that stores a subset of data for each task and then “replays” the old data together with the new one to avoid forgetting old concepts.
[11] propose to use such a memory module for sparse experience replay and local adaptation in the language domain.
This method stores all training examples, in order to achieve optimal performance.
To make this method comparable with adapter-based methods, we freeze pre-trained representation, add a single adapter parameters  and train the adapter by replaying examples from old tasks while training using data from the new task.
In order to keep baselines comparable we assign to ER the same amount of memory is used for the distillation buffer in ADA.
In addition to these baselines, we use one special case of ADA with K=1 as a baseline to demonstrate the advantage of effective consolidation of Adapters.
Adapter architectures.
We use pre-trained models from HuggingFace Transformers [63] as our base feature extractors.
We ran experiments with BERT base , DistilBERT base , RoBERTa base for text classiﬁcation and ViT-B and DeiT-B for image classiﬁcation.
We analyze the cases based on all these models, due to the space constraints, we present BERT base in this section and the rest in
Appendix A.5.
BERT base model uses 12 layers of Transformers block with a hidden size of 768 and number of self-attention heads as 12 and has around 110 M (440 MB) trainable parameters.
For the
Adapter implementation, we use Adapter-Hub [42], but no Adapter implementation was available for
Vision Transformers.
We deﬁne our architecture of Adapters for ViT and DeiT in Appendix A.4.
An
Adapter has a simple bottleneck architecture that contains fewer parameters than the attention and the feed-forward layers.
The Adapter size is the hyper-parameter that is tuned and it can be set to { 12 , 24 , 48 , 96 , 192 , 384 } for BERT base model.
For all the methods, we use the same conﬁguration for the Adapters, setting the size to 48 .
With this setting, an Adapter contains ⇠ 1 . 8 M parameters.
We also train a head model for each task, that has 768 parameters for BERT base (last hidden size of BERT base ⇥ output size, which equals to 1 for binary classiﬁcation).
The tables in Appendix A. reports the number of parameters used for baselines and ADA in our experiments.
5.1 Text Classiﬁcation
Predictive performance.
Figure 1 shows the comparison of ADA and the baseline methods.
It can be clearly seen that freezing all pre-trained model parameters, and ﬁne-tuning only the head models (B1) led to an inferior performance compared to adapter-based approaches.
The main reason is that the head models have small amount of parameters to train and ﬁne-tuning only the heads suffers from under-ﬁtting.
B2 performs good only for ﬁrst 2-3 tasks, since we keep training the complete model, it forgets the previously learned tasks very quickly.
As mentioned above, Adapters and AdapterFusion add ⇠ 1 . 8 M parameters for each task and train these parameters with new task data, and these parameters are ﬁxed after training.
So, they perform well on both new tasks and previous tasks.
The results on each dataset conﬁrm this.
Both ER and ADA K=1, perform closely with Adapters almost for half of the tasks.
The similar behavior of ER and ADA K=1 demonstrates that the distillation with soft labels works well and it is almost as good as training with the true labels.
Later the performance declines for both methods, because the capacity of the Adapter is exceeded.
ADA LEEP and ADA
TransRate results with K=4 Adapters show that selective consolidation of Adapters signiﬁcantly improves the performance.
Their performance is on par with AdapterFusion while the number of model parameters is signiﬁcantly lower.
We present BERT base results in this section while the rest is reported in Appendix A.9.
We also compute FWT and BWT scores for these methods.
We didn’t present B1 and Adapters in the plots, since both FWT and BWT are zero for them.
BWT is zero for AdapterFusion, since the fusion parameter is computed with available Adapters, and the Adapters trained later is not used for the previous tasks.
ADA-LEEP and ADA-TransRate minimizes negative backward transfer, while showing a positive forward transfer for all datasets.
Memory consumption.
Figure 2 shows the increase in terms of percentage in the number of parameters used by each method and their predictive performance.
We see that on Wikipedia, 200% of the base model parameters ( 220 M additional parameters) are added.
These results make clear that
ADA is signiﬁcantly more efﬁcient in terms of memory usage.
It can achieve predictive performance similar to the one of Adapters and AdapterFusion while requiring signiﬁcantly less model parameters.
On Reuters and Arxiv, it can store the parameters of only 5 Adapters (K=4 Adapters in the pool, and one Adapter for new task), against the 20 required by AdapterFusion (on Wikipedia it is 5 against 60).
Inference time.
When machine learning models are used to power customer-facing web sites, they are often required to provide predictions in a few milliseconds to keep the overall latency within requirements.
Moreover, in this kind of application the model will be trained once and make billions of predictions so a reasonable increase in the training time is irrelevant compared to a decrease in the inference time.
We report the inference time results of ADA and other Adapter based methods in Appendix A.6.
Results demonstrate that ADA provides a sufﬁciently fast inference for most applications and still offers opportunities to speed it up further, for example by employing smaller pre-trained Transformers (e.g. DistilBERT, see Appendix A.5).
Training time.
Distillation of Adapters brings an extra cost for ADA while learning fusion parameters brings an extra cost for AdapterFusion.
Computing transferability takes constant time which is negligible.
Distillation costs training an additional Adapter ( 1.6 % of full ﬁne-tuning time of BERT).
Figure 10d in AppendixA.6 reports the average training time comparison on Wiki-30K that is the largest difference with AdapterFusion given larger number of tasks.
We can clearly see that the difference is small (ADAis 3.37% more, ADA-TransRate is 5.6% more) while the difference between the inference time is signiﬁcant.
5.2 Image Classiﬁcation
For image classiﬁcation experiments, we add Elastic Weight Consolidation (EWC) [29] as an additional baseline since it is widely used in CL literature for image classiﬁcation.
EWC is a regularizationbased CL method that assumes that some weights of the trained neural network are more important for previously learned tasks than others.
During training of the neural network on a new task, changes to the weights of the network are made less likely the greater their importance.
Figure 3 shows the comparison of ADA and the baseline methods.
The results show the same behaviour with text classiﬁcation.
B1 leaded to an inferior performance compared to other approaches.
B2 performs well only for initial tasks and it forgets the previously learned tasks very quickly.
Results conﬁrm there is no forgetting for Adapters and AdapterFusion.
Although the careful tuning of regularization coefﬁcient, EWC cannot handle CF, especially for multi-class classiﬁcation problem.
ADA with K=1 shows that distillation alone doesn’t prevent forgetting.
In almost all cases, ER performs on-par with ADA K=1, providing evidence that a small amount of memory can actually improve performance compared to ﬁne-tuning or regularization, but the improvement is limited and does not last as the number of tasks increases.
ADA-LEEP and ADA-TransRate results with K=4 Adapters show that selective consolidation of
Adapters signiﬁcantly improves the performance.
For binary classiﬁcation, their performance are on par with AdapterFusion while the number of model parameters is signiﬁcantly lower.
For multi-class, their performance slightly declines after a certain number of tasks.
This is discussed in next section and the main reason is that the capacity of the Adapter is exceeded.
To validate the interoperability of ADA to different models, we run the same experiments on DeiT model and present the results in
Appendix A.10 due to space constraints.
5.3 Ablation studies
Comparison with larger distilled models.
In Section 5 we compared ADA with the special case of ADA with K=1 to evaluate the improvement provided by our approach over a distillation-only solutions.
We would like to provide additional observations of the superior performance of ADA by comparing its performance with the one of a distilled Adapter using more parameters.
Speciﬁcally, we run an experiment where we compare ADA with K=4 and ADA with K=1 as displayed before but in this case the “size” of the Adapter, which is 48 for Size ⇥ 1, is multiplied by 4 for Size ⇥ 4 Adapter to have a comparison where the different methods use the same number of model parameters.
Since
K=1 is a special case where a single Adapter is kept in the pool, the transferability metric is irrelevant and we can see ADA with K=1 as a method purely based on distillation like DMC [71].
The results reported in Figure 4 show that ADA can make a better use of the model parameters compared to a distillation-only method and that the intelligent selection of which Adapters to distill together makes once again a big difference.
It is also interesting to observe that the usage of additional model parameters brings a clear advantage but the mixed comparison between the ADA K=4 with random Adapter selection and ADA K=
with four times larger Adapters leaves some questions open regarding how far distillation can get in this setting.
Another ﬁnding is that TransRate outperforms
LEEP in most cases.
It is also demonstrated in the original paper [24] that TransRate has a strong correlation to the transfer learning performance and it outperforms LEEP and other metrics employed.
Impact of the Adapters pool size.
In our experiments we used a ﬁxed number of Adapters in the pool size, but more Adapters can be added to ADA’s pool as more tasks are processed.
This may actually be the preferred usage in some applications.
We already know that having an Adapter per task provides good performance and using multiple of them at the same time like in AdapterFusion provides a beneﬁt, but we would like to verify the sensitivity to this parameter.
The results reported in Figure 5 show a rapidly decreasing added value when the number of Adapters grows, a behavior which aligns well with our practical requirements of keeping the number of model parameters under control when the number of tasks increases.
See additional experiments in Appendix A.11.
Mixed Data Experiments.
We run experiments in a setting where we sample 200 tasks from
Arxiv, Reuters and Wikipedia (50/50/100) respectively (the order of the tasks are created randomly).
First we design experiment to see when the performance decline is observed for different values of
K . Figure 6 shows that we observe a little saturation only after the 150th task when K =4 and no saturation when K =8. So, we set K =8 for these experiments and show that ADA with TransRate comparable performance with Adapters and AdapterFusion even in a complicated setting.  Conclusion
In this paper we presented ADA, a method that allows neural text [16] and image [17] classiﬁers to learn new classes based on pre-trained Transformers while maintaining strict control of the memory usage and reaching state-of-the-art predictive performance.
The method has shown to be effective in different domains and allows users to leverage publicly available pre-trained Transformers for continual classiﬁcation tasks.
We empirically demonstrated that Adapters can give good results when used in combination with vision Transformers on CV tasks.
We evaluated ADA on different classiﬁcation tasks and demonstrated that the predictive performance is competitive with state-ofthe-art methods which use up to an order of magnitude parameters.
Moreover, ADA displayed lower latency at inference time and improved data efﬁciency for some speciﬁc settings (see Appendix A.8).
Transformers are very popular, but they are not the only models being widely used in practice.
We consider this the main weakness of our approach and we would like to further expand our activity to perform CL on other widely used pre-trained models such as ResNet.
Addressing multi-modal classiﬁcation using text and images together will be the other focus of our future research.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors.
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Our work is tied to the usage of Transform- ers, we remind it in the conclusions.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] We don’t have theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] We don’t have theorems.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[TODO]
We will provide the code if the paper is accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 5 and Appendix A.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Yes, all ﬁgures show the standard deviation (shaded regions).
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.2.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We used AdapterHub, that is cited in experiments and appendix sections.
(b) Did you mention the license of the assets?
[Yes] Yes, in Appendix A.3.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]