Memory Efﬁcient Continual Learning with Transformers
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In many real-world scenarios, data to train machine learning models becomesavailable over time.
Unfortunately, these models struggle to continually learn newconcepts without forgetting what has been learnt in the past.
This phenomenonis known as catastrophic forgetting and it is difﬁcult to prevent due to practicalconstraints.
For instance, the amount of data that can be stored or the computationalresources that can be used might be limited.
Moreover, applications increasinglyrely on large pre-trained neural networks, such as pre-trained Transformers, sincecompute or data might not be available in sufﬁciently large quantities to practitioners to train from scratch.
In this paper, we devise a method to incrementallytrain a model on a sequence of tasks using pre-trained Transformers and extending them with Adapters.
Different than the existing approaches, our method isable to scale to a large number of tasks without signiﬁcant overhead and allowssharing information across tasks.
On both image and text classiﬁcation tasks, weempirically demonstrate that our method maintains a good predictive performancewithout retraining the model or increasing the number of model parameters overtime.
The resulting model is also signiﬁcantly faster at inference time compared to
Adapter-based state-of-the-art methods.
Introduction
Transformers [57], e.g. BERT [12], have shown their effectiveness in various natural languageprocessing (NLP) tasks such as classiﬁcation [25], Natural Language Inference [41, 39] and Question
Answering [16].
Inspired by this achievement, some pioneering works have recently been introducedon adapting Transformers architectures to Computer Vision (CV).
Vision Transformers [13, 55]showed that a pure Transformer applied directly to a sequence of image patches can perform wellon image classiﬁcation tasks.
Besides, some recent studies [67, 4, 30] showed that Transformersgeneralize to new domains given only a few samples.
Transformers show a great ability to learncomplex concepts but when confronted with a sequence of different tasks they tend to “overwrite” thepreviously learnt concepts.
In general, deep networks suffer heavily from this phenomenon, calledcatastrophic forgetting (CF) [35], impeding continual or lifelong learning.
In the last few years, agrowing body of works attempted to tackle CF in continual learning (CL) [14, 19, 27, 46, 63, 65] butmost are not able to meet the scale or accuracy requirements of real-world applications.
Moreover,adapting large-scale pre-trained Transformer models to downstream tasks via ﬁne-tuning is themethod of choice in NLP applications, posing the need for methods that can directly work withpre-trained models instead of requiring the training of a new model from scratch [15].
In this work, we tackle both text and image classiﬁcation problem in a setting where the number oftags or classes associated to the input data grows over time.
In fact, the ability to continually extendthe set of tags or classes used to categorize content is a major problem in many applications.
Forexample, newspapers can tag news according to topics of interest such as “sport”, “politics”, “food” by using a pre-trained language model and reﬁning it using a few hundred pre-tagged articles.
Newtags may appear over time, for example “COVID-19” was a completely unknown news category in2019 but appeared frequently since.
In these cases, retraining models from scratch is often impracticaland can lead to inconsistencies in the labeling when compared to the one provided by the previousmodel.
In particular, we focus on incrementally extending classiﬁers based on pre-trained Transformermodels given their ubiquity in NLP and the growing interest in CV.
To address the issue of incremental ﬁne-tuning of pre-trained Transformers in the sequential learning setting without CF, we propose Adaptive Distillation of Adapters (ADA).
ADA leverages
Adapters [20], a specialized neural network module, to adapt part of the weights in the neuralnetwork and a distillation mechanism to consolidate the information previously learnt in a ﬁxedamount of network parameters with limited amount of forgetting.
This method allows the userto control the memory consumption, while retaining state-of-the-art performance when runningthe algorithm on sequences of tens of tasks.
This tight memory control is important in industrialapplications.
The alternative, a model growing in size with the number of tasks, would require achange in hardware to adapt to the growing memory requirements of the deployed model.
This wouldbe problematic since a practitioner will incur into higher risk of system instability and be forced tomake conservative hardware choices.
The main contribution of our work is ADA, an algorithm that can achieve high predictive performance on both text and images classiﬁcation in different continual learning scenarios.
ADA alsoprovides lower inference time and uses an order of magnitude fewer parameters than state of the artmethods such AdaptersFusion.
Additionally, we implemented Adapters for vision Transformers andempirically demonstrated their effectiveness.
Related Work.
Adapters [20] were proposed for ﬁne-tuning of pre-trained language models andwere studied for the multi-task setting.
AdapterFusion [39] provides state-of-the-art performanceby composing the pre-trained Adapters and it can simply be repurposed for preventing CF in CLby learning one Adapter for every new task.
While it has been shown that the number of additionalmodel parameters per Adapter is signiﬁcantly smaller than the number of parameters used in thepre-trained model [20] (e.g., 3 .
6% of the parameters of the pre-trained model), since both Adaptersand AdapterFusion require to store all the model parameters, the memory consumption increasesrapidly with the number of tasks.
In the case of a model being trained on 30 tasks, we would have toadd more parameters than the number of pre-trained Transformer parameters (details in Section 4),making the method unsuitable for CL.
Recent work studied catastrophic forgetting [53, 10, 25, 33] and incremental learning [64] for NLPand CV [30, 15].
Pasanuru et al.
[38] focus on the few-shot setting where only a few data pointsare available for each task.
Ke et al.
[26] proposed an architecture to achieve both CF preventionand knowledge transfer.
This method has some similarity to AdapterBERT [20] since they insert
a
CL plug-in module in two locations in BERT.
A CL-plugin is a capsule network [50] that uses oneseparate capsule [18] (2-layer fully connected network) for each task, and like Adapters, memoryincreases linearly over the time.
In addition, this algorithm requires to learn task masks to addressknowledge transfer, which is costly to compute.
Among those recent works, only a few [30, 15] haveapplied the Transformers architecture to CL on image datasets.
In [30], for each new taskthe modelis copied and ﬁxed to be used as the teacher model in the distillation phase.
The student model istrained on both new task samples together with the knowledge distillation loss that uses samples fromold tasks which is stored in the rehearsal memory.
In [15], the authors aim to learn a uniﬁed modelthat will classify an increasingly growing number of classes by building upon a new architecture.
However, they need to train a new Transformer, where the process is very costly and contrast withour goal of using public pre-trained models.
To the best of our knowledge there is no method able toleverage public pre-trained Transformers while keeping the number of model parameters constantwhile retaining state-of-the-art predictive performance.
Problem setup and Preliminaries
Problem Setup.
A sequence of classiﬁcation tasks { T 1 , . .
, T N } are given where each task T icontains a different set of data sample (text or image)-label training pairs ( x i 1: t , y i 1: t ) and containsc new classes namely Y i =
{ Y 1 i , . . .
, Y ci } with t examples for each new class.
The goal of thelearner is to learn a set of parameters ˜ ⇥ such that 1 N
P i 2 { 1 ,...,N } loss ( T i ; ˜ ⇥ ) is minimized.
The task identiﬁer is provided to the learner with every new batch of data.
Moreover, in our speciﬁc case, ˜ ⇥ iscomposed of a set of parameters ⇥ provided by a pre-trained model and, depending on the algorithm,some additional parameters which need to be learned for each speciﬁc task.
In its simplest case, thisadditional set of model parameters can just be a head model, but some algorithms use signiﬁcantlymore elaborate functions.
In the case of the labeling application described in Section 1, each taskrepresents a label and the learner creates a new binary classiﬁer for each label.
For the training of task T i , the learner can only access the newly added examples and label namesin this task.
To evaluate the learner, the test data consists of examples across all the previous tasks,where the potential label space for the test example is Y 1: c[ Y 1: c[ · · · [ Y 1: c N .
All methods that we 100 deﬁne in the following sections receive as input a pre-trained model f ⇥ ( . )
, e.g., BERT [12], that isable to extract high quality representations from the input data.
Adapters.
Adapters were proposed by [20] as an alternative to ﬁne-tuning in NLP.
They add newmodules between layers of a pre-trained network called Adapters .
These modules are feed-forwardlayers that project the original feature size to a smaller dimension and projects them to the originalsize thereafter, ensuring that the number of parameters stays substantially small as compared to theoriginal model.
(See Appendix A.4 for the details of the Adapter architectures.) Adapters sharethe pre-trained model parameters ⇥ across all tasks and introduce a small number of task-speciﬁcparameters  i without affecting previous ones.
The model is initialized with parameters of a pretrained model ⇥ .
For each of the task i 2 { 1 , . . . , N } where N is the total number of tasks, a setof new and randomly initialized Adapter parameters  i are introduced.
The parameters ⇥ are ﬁxedand only the parameters  i are trained when a new task is added.
This makes it possible to train Adapters for all N tasks, and store the corresponding knowledge in designated parts of the model.
The objective for each task i 2 { 1 , . . .
, N } is of the form:  i arg min  L
i ( D i ; ⇥ ,  ) .
AdapterFusion [39], has been proposed to mitigate the lack of knowledge sharing across tasks.
Itworks in two phases: i) in the knowledge extraction stage, adapters, which encapsulate the taskspeciﬁc information, are learnt for each of the N tasks; while ii) in the knowledge composition stage,the set of N Adapters are combined by using additional parameters .
The additional parametersi for task i are deﬁned as: i arg min
L i ( D i ; ⇥ ,  1 , . . .
,  i , ) .
While this provides goodpredictive performance, in the CL setting, new tasks are added sequentially and storing a large set of
Adapters  1 , . .
,  N is practically infeasible.
Adaptive Distillation of Adapters (ADA)
To address the issues we mentioned in the previous sections, we propose Adaptive Distillation of
Adapters (ADA).
ADA keeps a ﬁxed amount of Adapters in memory and takes transferability ofrepresentations into account to effectively consolidate newly created Adapters with previously createdones.
ADA works in two steps: i) it trains a new Adapter and classiﬁcation head, which we refer asthe new model, using the training dataset of the new task; ii) it consolidates an old model with thenew model.
To better control the memory usage, ADA has a ﬁx budget for the number of Adapters K that are stored in a pool of old models.
In the consolidation phase, the algorithm selects one of themodels in the pool using scores that quantify the information contained in the representations theyprovide.
In the following sections, we explain the components of ADA and how they work.
3.1 Distillation of Adapters
For each new task T n , the Adapter parameters  n are added to the model, while the pre-trained modelparameters ⇥ are kept frozen and are never changed.
Only the task-speciﬁc Adapter parameters  nand the head model parameters h n are trained for the current task.
The model f n ( x ; ⇥ ,  n , h n ) , withparameters ⇥ ,  n and h n is called the new model.
The head model parameters are ﬁxed after trainingthe new model and they are not updated during model consolidation.
When a prediction for a task T iis required, the corresponding Adapter   ( i ) and head model h i is called.
is a mapping from thetask id to the corresponding Adapter in the pool or to the newly trained Adapter.
We abuse notationdeﬁning f as the function returning the output ( logits ) on all tasks:
[Equation]
For the consolidation step, an Adapter from the pool  is selected as explained in Section 3.2 andnew collection of Adapter pool  0 is created where the selected Adapter is replaced with  c .
c denotes the consolidated model parameters and preliminarily the parameters are randomly initialized.
Similarly, a copy of  is created to map the old tasks that are associated to the selected Adapter andthe new task n to  c .
The consolidation then has the following objective:
[-P |U| i =
-][Equation]
where D distill denotes the unlabeled training data used for distillation, and the distillation loss iscomputed as the difference between the logits produced by the existing specialist models denoted byf ( x i ; ⇥ ,  , h ) and the consolidated model denoted by f ( x i ; ⇥ ,  0 , h ) based on L 2 loss.
After  c hasbeen trained,  is swapped with  0 .
This is a high-level view of the mechanism, our implementationis optimized to avoid copying models when not necessary.
This procedure follows the double distillation loss [69] which is originally proposed for classincremental learning to train a new Adapter that is used with the pre-trained model to classify bothold and new tasks.
The main idea is ﬁrst training a separate model for the new class(es) using labeleddata, and then combining the new and old models using unlabeled distillation data via a doubledistillation training objective.
We generalize this solution to our case where we have a set of teachermodels kept in Adapter pool  and train a student model  c .
Double distillation procedure and thealternative solutions for distillation are discussed in Appendix A.1 but this solution was the bestperforming one in our experiments.
While several different data sources can be used to populate the buffer, such as using auxiliary externaldata [69] or generating synthetic data [9], in this work we populate the buffer using covariates fromprevious tasks selected with Reservoir Sampling [58].
This simple mechanism may not be the mosteffective, but it will guarantee that no advantage is given to ADA in the experimental comparison.
3.2 Adapter Selection for Distillation
In the previous section, we assumed the Adapter to be consolidated as given but ADA keeps a pool of
Adapters and the selection of the Adapter to be distilled is an important part of the algorithm.
In fact,our empirical observations show that a random selection of the Adapter provides poor performance(see Section 4.3).
The intuition behind our selection mechanism is the following: since a specializedhead for every task is created, we can assume that when the features provided by the associated
Adapter are highly informative, the updates (i.e., the gradients applied) will be small.
At the sametime, training a new head with every Adapter in the pool in order to observe which one is the mosteffective would increase the amount of computation required and signiﬁcantly impact the usability ofthe method.
The problem of computing the information carried by a representation in an efﬁcientmanner has been already studied in the transfer learning community [3, 56, 54].
While, the aim of that research is completely different and, to the best of our knowledge, there is noclear relation between transferability and forgetting, the mathematical foundation of this work areclosely related to our intuition.
In fact, scores like TransRate [22] employ the mutual informationbetween the features provided by a pre-trained model and the target labels for the task at hand.
Whenthe mutual information is high, the transferability is high.
More speciﬁcally, the knowledge transferfrom a source task T s to a target task T t is measured as:
[Equation]
where Y are the labels of target examples and Z = f
( X ; ⇥ ,   ( s ) ) are features of them extracted bythe pre-trained model and the Adapter associated to the source task.
TransRate is not the only score designed to quantify transferability between a pre-trained model and 182 a new task: Log Expected Empirical Prediction (LEEP)
[36] is a well-known alternative.
Also inthis case, the score was designed with a different application in mind, but it leverages the conditional 184 distribution of the target label given the source label to quantify the how informative the informationprovided by the source model is.
Speciﬁcally, LEEP is a three steps method.
At Step 1, it computes 186 dummy label distributions of the inputs f ( X ; ⇥ ,   ( t ) , h t ) in the target data set
D .
At Step 2, itcomputes the empirical conditional distribution ˆ P ( y | z ) of target label y given the source label z .
At
Step 3, it computes LEEP using f ( X ; ⇥ ,   ( s ) , h s ) and ˆ P ( y | z ) :
[Equation]
where z is a dummy label randomly drawn from f ( X ; ⇥ ,   ( s ) , h s ) and y is randomly drawnfrom ˆ P ( y | z ) .
We selected TransRate and LEEP for their simplicity and their ability to provide aquantiﬁcation without training but practitioners can replace these scores with different ones as theysee ﬁt.
3.3 Algorithm
Require: ⇥ : pre-trained model, K : adapters pool size Freeze ⇥ and create
=
Map () for n 1 to N do A task T n is received
Initialize  n Process T n , train new model f n ( x ; ⇥ ,  n ,h n ) Sample from T n and add to distillation data
D distill if n  K then Store f n in the pool else j ⇤ argmax j 2 { 1 ,...,K } TranScore( T n ,f j ) Add ( n,j ⇤ )
to
Consolidate model: f j ⇤ =
Distill( f j ⇤ ,f n , D distill ) end if
Serve predictions for any task i  n using f  ( i ) end for
Distill( f i ,f j , D distill ) : Get soft targets
ˆ y i from old model f i with D distill
Get soft targets ˆ y j from new model f j with D distill Initialize  c
Compute distillation loss and train model f ( x ; ⇥ ,  c ) as de- ﬁned in Eq.return f
ADA is detailed in Algorithm 1.
Thegraphical workﬂow of the algorithm isshown in Appendix 6.
For every newtask, the algorithm trains a new adapterand head model (called  n and h n ).
If the adapters pool did not reach themaximum size yet (controlled by K ),it just adds it to the pool.
If the poolreached the maximum size, the algorithmis forced to select one of the adaptersalready in the pool and distill it togetherwith the newly trained one. In order toselect which adapter to distill, ADA usesthe transferability scores (e.g., LEEPor TransRate). Once the adapter in thepool with the highest transferability score(called f j ⇤ ) is identiﬁed, it consolidatesthat adapter and the newly trained oneinto a new adapter and replaces the oldone present in the pool. In order to beable to make effective predictions, thealgorithm also keeps a mapping  of whichadapter in the pool must be used in combination with each of the task-speciﬁc heads.
Experiments
In this section, we empirically validate our adapter distillation approach on text and image classiﬁcation tasks and show that ADA achieves similar performance to AdapterFusion while consumingsigniﬁcantly less memory.
We dedicate Section 4.3 to ablation studies providing further insights intothe mechanisms implemented in ADA and their contribution.
Datasets and experimental setup.
We use three text datasets for multi-label text classiﬁcation: Arxiv Papers [66] (paper classiﬁcation), Reuters (RCV1-V2) [29] (news classiﬁcation), Wiki-30K [71](Wikipedia article classiﬁcation) and two dataset for image classiﬁcation: CIFAR100 [28] and MiniImageNet [49].
Details about the datasets are given in Appendix A.3.
For the multi-label text classiﬁcation experiments, we ﬁrst sample a sequence of labels from thelabel space.
Then, we create a balanced binary classiﬁcation task for each label by sampling thesame amount of positive data points from the label considered and negative data points from thelabels preceding the current one in the sequence.
After splitting the data in training and test set, weprovide the algorithm with the training set and subsequently measure its performance on the test set.
The algorithm never observes any data point in the test set and, more generally, every data point inthe dataset is used only once.
For Arxiv Papers and Reuters datasets, we created 20 tasks and for Wiki-30K 60.
We ﬁxed the number of training samples per task to 100.
The test set consists of 40data points on Reuters and of 100 data points on Arxiv and Wiki-30K.
For image classiﬁcation , we design two scenarios.
In the ﬁrst scenario, each new task is a balancedbinary classiﬁcation problem.
Each class can be selected to be the positive class only once.
In thesecond scenario each task is a balanced multi-class classiﬁcation problem with 5 classes.
In bothcases we provide the learner with 50 data points per class both at training and test time: in the ﬁrstscenario each task will have a training set of 250 data points and in the second case of 100 data points.
The total number of tasks is ﬁxed to 20 for both scenarios.
The distillation memory size is ﬁxed to1000 for Wiki-30K which has a larger number of tasks, and to 500 for the others. Metrics.
In [34], three metrics that we discuss in the following are deﬁned to evaluate the performanceof a CL method. We use these metrics to evaluate our methods. It is considered that we have access toa test set for each of the N tasks in { T 1 , . . . , T N } .
After the model ﬁnishes learning about the task T i ,its test performance are evaluated on all N tasks. By doing so, a matrix R 2 R N ⇥ N is constructedwhere R i,j is the test classiﬁcation accuracy of the model on task T j after observing the last samplefrom task T i . Letting ¯ b be the vector of test accuracies for each task at random initialization, thethree metrics are deﬁned: i) Average Accuracy = 1 N P
Ni =1 R N,i , ii) Backward Transfer (BWT)
=1 N
P
N  1 i =1 R N,i  R
i,i and iii)
Forward Transfer (FWT) =
1 N
P
Ni =
2 R
i  1 ,i  ¯ b i .
(The largerthese metrics, the better the model.)
All the results in this section are averaged over 5 runs.
Baselines.
We compare ADA the following baselines.
1) Fine-tuning head model (B1) :
We freezethe pre-trained representation and only ﬁne-tune the output layer of each classiﬁcation task.
Theoutput layer is multiple-head binary classiﬁer that we also use for the other methods. 2)
Fine tuningthe full model (B2) :
We ﬁne-tune both the pre-trained representation and the output layer for eachclassiﬁcation task.
3) Adapters [20]: We train and keep separate Adapters for each classiﬁcation taskas well as the head models.
4) AdapterFusion [39]: It is a two stage learning algorithm that leveragesknowledge from multiple tasks by combining the representations from several task Adapters in orderto improve the performance on the target task.
This follows exactly the solution depicted in Section 2.5) Experience Replay (ER) [48]: ER is a commonly used baseline in Continual Learning that storesa subset of data for each task and then “replays” the old data together with the new one to avoidforgetting old concepts.
[11] propose to use such a memory module for sparse experience replayand local adaptation in the language domain.
This method stores all training examples, in order toachieve optimal performance.
To make this method comparable with adapter-based methods, wefreeze pre-trained representation, add a single adapter parameters  and train the adapter by replayingexamples from old tasks while training using data from the new task.
In order to keep baselinescomparable we assign to ER the same amount of memory is used for the distillation buffer in ADA.
In addition to these baselines, we use one special case of ADA with K=1 as a baseline to demonstratethe advantage of effective consolidation of Adapters.
Adapter architectures.
We use pre-trained models from HuggingFace Transformers [61] as ourbase feature extractors.
We ran experiments with BERT base , DistilBERT base , RoBERTa base fortext classiﬁcation and ViT-B and DeiT-B for image classiﬁcation.
We analyze the cases based onall these models, due to the space constraints, we present BERT base in this section and the rest in
Appendix A.5.
BERT base model uses 12 layers of Transformers block with a hidden size of 768 andnumber of self-attention heads as 12 and has around 110 M (440 MB) trainable parameters.
For the
Adapter implementation, we use Adapter-Hub [40], but no Adapter implementation was available for
Vision Transformers.
We deﬁne our architecture of Adapters for ViT and DeiT in Appendix A.4.
An
Adapter has a simple bottleneck architecture that contains fewer parameters than the attention andthe feed-forward layers.
The Adapter size is the hyper-parameter that is tuned and it can be set to{ 12 , 24 , 48 , 96 , 192 , 384 } for BERT base model.
For all the methods, we use the same conﬁgurationfor the Adapters, setting the size to 48 .
With this setting, an Adapter contains ⇠ 1 . 8 M parameters.
We also train a head model for each task, that has 768 parameters for BERT base (last hidden sizeof BERT base ⇥ output size, which equals to 1 for binary classiﬁcation).
The tables in Appendix A.5reports the number of parameters used for baselines and ADA in our experiments.
4.1 Text Classiﬁcation
Predictive performance.
Figure 1 shows the comparison of ADA and the baseline methods.
It canbe clearly seen that freezing all pre-trained model parameters, and ﬁne-tuning only the head models(B1) led to an inferior performance compared to adapter-based approaches.
The main reason is thatthe head models have small amount of parameters to train and ﬁne-tuning only the heads suffers fromunder-ﬁtting.
B2 performs good only for ﬁrst 2-3 tasks, since we keep training the complete model, itforgets the previously learned tasks very quickly.
As mentioned above, Adapters and AdapterFusionadd ⇠ 1 . 8 M parameters for each task and train these parameters with new task data, and theseparameters are ﬁxed after training.
So, they perform well on both new tasks and previous tasks.
Theresults on each dataset conﬁrm this.
Both ER and ADA K=1, perform closely with Adapters almostfor half of the tasks.
The similar behavior of ER and ADA K=1 demonstrates that the distillation withsoft labels works well and it is almost as good as training with the true labels.
Later the performancedeclines for both methods, because the capacity of the Adapter is exceeded.
ADA LEEP and ADA
TransRate results with K=4 Adapters show that selective consolidation of Adapters signiﬁcantlyimproves the performance.
Their performance is on par with AdapterFusion while the number ofmodel parameters is signiﬁcantly lower.
We present BERT base results in this section while the rest isreported in Appendix A.9.
We also compute FWT and BWT scores for these methods.
We didn’t present B1 and Adapters inthe plots, since both FWT and BWT are zero for them.
BWT is zero for AdapterFusion, since thefusion parameter is computed with available Adapters, and the Adapters trained later is not used forthe previous tasks.
ADA-LEEP and ADA-TransRate minimizes negative backward transfer, whileshowing a positive forward transfer for all datasets.
Memory consumption.
Figure 2 shows the increase in terms of percentage in the number ofparameters used by each method and their predictive performance.
We see that on Wikipedia, 200%of the base model parameters ( 220 M additional parameters) are added.
These results make clear that ADA is signiﬁcantly more efﬁcient in terms of memory usage.
It can achieve predictive performancesimilar to the one of Adapters and AdapterFusion while requiring signiﬁcantly less model parameters.
On Reuters and Arxiv, it can store the parameters of only 5 Adapters (K=4 Adapters in the pool, andone Adapter for new task), against the 20 required by AdapterFusion (on Wikipedia it is 5 against 60).
Inference time.
When machine learning models are used to power customer-facing web sites, theyare often required to provide predictions in a few milliseconds to keep the overall latency withinrequirements.
Moreover, in this kind of application the model will be trained once and make billionsof predictions so a reasonable increase in the training time is irrelevant compared to a decrease inthe inference time.
We report the inference time results of ADA and other Adapter based methodsin Appendix A.6.
Results demonstrate that ADA provides a sufﬁciently fast inference for mostapplications and still offers opportunities to speed it up further, for example by employing smallerpre-trained Transformers (e.g. DistilBERT
Training time.
Distillation of Adapters brings an extra cost for ADA while learning fusion parametersbrings an extra cost for AdapterFusion.
Computing transferability takes constant time which isnegligible.
Distillation costs training an additional Adapter ( 1.6 % of full ﬁne-tuning time of BERT).
Figure 9d in AppendixA.6 reports the average training time comparison on Wiki-30K that is thelargest difference with AdapterFusion given larger number of tasks.
We can clearly see that thedifference is small (ADAis 3.37% more, ADA-TransRate is 5.6% more) while the difference betweenthe inference time is signiﬁcant.
4.2 Image Classiﬁcation
For image classiﬁcation experiments, we add Elastic Weight Consolidation (EWC) [27] as an additional baseline since it is widely used in CL literature for image classiﬁcation.
EWC is a regularizationbased CL method that assumes that some weights of the trained neural network are more importantfor previously learned tasks than others.
During training of the neural network on a new task, changesto the weights of the network are made less likely the greater their importance.
Figure 3 shows the comparison of ADA and the baseline methods.
The results show the samebehaviour with text classiﬁcation.
B1 leaded to an inferior performance compared to other approaches.
B2 performs well only for initial tasks and it forgets the previously learned tasks very quickly.
Resultsconﬁrm there is no forgetting for Adapters and AdapterFusion.
Although the careful tuning ofregularization coefﬁcient, EWC cannot handle CF, especially for multi-class classiﬁcation problem.
ADA with K=1 shows that distillation alone doesn’t prevent forgetting.
In almost all cases, ERperforms on-par with ADA K=1, providing evidence that a small amount of memory can actuallyimprove performance compared to ﬁne-tuning or regularization, but the improvement is limited anddoes not last as the number of tasks increases.
ADA-LEEP and ADA-TransRate results with K=4 Adapters show that selective consolidation of
Adapters signiﬁcantly improves the performance.
For binary classiﬁcation, their performance are onpar with AdapterFusion while the number of model parameters is signiﬁcantly lower.
For multi-class,their performance slightly declines after a certain number of tasks.
This is discussed in next sectionand the main reason is that the capacity of the Adapter is exceeded.
To validate the interoperabilityof ADA to different models, we run the same experiments on DeiT model and present the results in
Appendix A.10 due to space constraints.
4.3 Ablation studies
Comparison with larger distilled models.
In Section 4 we compared ADA with the special caseof ADA with K=1 to evaluate the improvement provided by our approach over a distillation-onlysolutions.
We would like to provide additional observations of the superior performance of ADA bycomparing its performance with the one of a distilled Adapter using more parameters.
Speciﬁcally,we run an experiment where we compare ADA with K=4 and ADA with K=1 as displayed before but in this case the “size” of the Adapter, which is 48 for Size ⇥ 1, is multiplied by 4 for Size ⇥ 4 Adapterto have a comparison where the different methods use the same number of model parameters.
Since
K=1 is a special case where a single Adapter is kept in the pool, the transferability metric is irrelevantand we can see ADA with K=1 as a method purely based on distillation like DMC [69].
The results reported in Figure 4 show that ADA can make a better use of the model parameterscompared to a distillation-only method and that the intelligent selection of which Adapters to distilltogether makes once again a big difference.
It is also interesting to observe that the usage of additionalmodel parameters brings a clear advantage but the mixed comparison between the ADA K=4 withrandom Adapter selection and ADA K=
with four times larger Adapters leaves some questions openregarding how far distillation can get in this setting.
Another ﬁnding is that TransRate outperforms
LEEP in most cases.
It is also demonstrated in the original paper [22] that TransRate has a strongcorrelation to the transfer learning performance and it outperforms LEEP and other metrics employed.
Impact of the Adapters pool size.
In our experiments we used a ﬁxed number of Adapters in thepool size, but more Adapters can be added to ADA’s pool as more tasks are processed.
This mayactually be the preferred usage in some applications.
We already know that having an Adapter pertask provides good performance and using multiple of them at the same time like in AdapterFusionprovides a beneﬁt, but we would like to verify the sensitivity to this parameter.
The results reported in Figure 5 show a rapidly decreasing added value when the number of Adapters grows, a behaviorwhich aligns well with our practical requirements of keeping the number of model parameters undercontrol when the number of tasks increases.
See additional experiments in Appendix A.11.
Conclusion
In this paper we presented ADA, a new method that allows neural text and image classiﬁers tolearn new classes based on pre-trained Transformers while maintaining strict control of the memoryusage and reaching state-of-the-art predictive performance.
The method has shown to be effectivein different domains and allows users to leverage publicly available pre-trained Transformers forcontinual classiﬁcation tasks.
We empirically demonstrated that Adapters can give good resultswhen used in combination with vision Transformers on CV tasks.
We evaluated ADA on differentclassiﬁcation tasks and demonstrated that the predictive performance is competitive with state-ofthe-art methods which use up to an order of magnitude parameters.
Moreover, ADA displayed lowerlatency at inference time and improved data efﬁciency for some speciﬁc settings (see Appendix A.8).
Transformers are very popular, but they are not the only models being widely used in practice.
Weconsider this
the main weakness of our approach and we would like to further expand our activityto perform CL on other widely used pre-trained models such as ResNet.
Addressing multi-modalclassiﬁcation using text and images together will be the other focus of our future research.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to 572 answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You arestrongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of 574 your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist 579 section does not count towards the page limit.
In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Our work is tied to the usage of Transformers, we remind it in the conclusions.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] We don’t have theoreticalresults. (b) Did you include complete proofs of all theoretical results?
[N/A] We don’t have theorems.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[TODO]
We will provide the code if
the paper is accepted.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4 and Appendix A.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[Yes] Yes, all ﬁgures show the standard deviation (shaded regions).
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[Yes] See Appendix A.2.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We used AdapterHub, that iscited in experiments and appendix sections.
(b) Did you mention the license of the assets?
[Yes] Yes, in Appendix A.3.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
informa-
tion or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A](b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on
participant compensation?
[N/A]