P ROGRESSIVE P ROMPTS : C ONTINUAL L EARNING FOR L ANGUAGE M ODELS
A BSTRACT
We introduce Progressive Prompts – a simple and efficient approach for continual learning in language models.
Our method allows forward transfer and resists catastrophic forgetting, without relying on data replay or a large number of task-specific parameters.
Progressive Prompts learns a new soft prompt for each task and sequentially concatenates it with the previously learned prompts, while keeping the base model frozen.
Experiments on standard continual learning benchmarks show that our approach outperforms state-of-the-art methods, with an improvement > 20% in average test accuracy over the previous best-preforming method on T5 model.
We also explore a more challenging continual learning setup with longer sequences of tasks and show that Progressive Prompts significantly outperforms prior methods.
I NTRODUCTION
Learning a long sequence of tasks while gaining experience and avoiding forgetting remains a key feature of human-level intelligence.
Although pretrained language models have largely succeeded in learning on a single task, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as continual learning (CL) (de Masson D’Autume et al., 2019;
Huang et al., 2021).
Two major challenges arise in CL: (1) avoiding catastrophic forgetting , i e., loss of the knowledge acquired from previous tasks after learning new ones (McCloskey & Cohen, 1989; Ratcliff, 1990), and (2) allowing forward transfer , i e., leveraging the knowledge from past tasks for efficient learning of new tasks.
Typical CL approaches for language models train a model on all tasks, which ensures forward transfer but also leads to forgetting.
These methods use data replay or add regularization constraints (Huang et al., 2021; de Masson D’Autume et al., 2019; Sun et al., 2019), but they still suffer from forgetting due to inevitable changes in parameters shared between tasks.
Other approaches, such as progressive networks (Rusu et al., 2016), can alleviate catastrophic forgetting completely while supporting forward transfer, but are computationally expensive because they add a new copy of the model for each task.
This can be especially intractable for large-scale language models with billions of parameters, which have become a standard in the NLP field (Zhang et al., 2022).
In this paper, we introduce Progressive Prompts – a novel CL approach for language models that supports forward transfer without forgetting.
Our method is inspired by progressive networks, but is significantly more memory-efficient because it only learns a fixed number of tokens, or prompt , for each new task.
Learning a prompt to adapt language models on a single downstream task was introduced in prompt tuning (Lester et al., 2021), and was shown to match the performance of full model finetuning while training < 0.01% of the parameters.
In Progressive Prompts, we learn a separate prompt for each incoming task and sequentially concatenate it with previously learned prompts.
Importantly, we share input tokens across all tasks and progressively prepend new prompts while keeping previous prompts frozen (see Figure 1).
Our method can: 1) alleviate catastrophic forgetting by preserving the knowledge acquired in previous prompts, and 2) transfer knowledge to future tasks by sequentially learning new prompts given previous ones.
We also introduce a new technique for prompt embedding reparameterization (Li & Liang, 2021).
We show that by passing the prompt embeddings through a residual MLP we can stabilize prompt tuning and improve its performance.
We run extensive experiments on standard CL benchmarks for text classification, and show that
Progressive Prompts outperforms state-of-the-art approaches on both BERT and T5 architectures (Devlin et al., 2018; Raffel et al., 2020).
We show over 20% improvement over the current SOTA for T5 model (Qin & Joty, 2021).
Furthermore, we run experiments on a more challenging CL setup with longer task sequences, and show that our method outperforms prior approaches for T5 and BERT architectures.
Our main contributions in this paper are as follows:
• We propose a novel CL approach, Progressive Prompts , that alleviates catastrophic forgetting and supports knowledge transfer to future tasks – all while learning < 0 .
1% of the total parameters.
• Progressive Prompts is suitable for any transformer-based architecture.
We show that it significantly outperforms prior SOTA methods on standard CL benchmarks for both BERT and T5 models.
• We propose a more challenging CL setup encompassing 15 text classification tasks and show that our method significantly outperforms prior methods.
B ACKGROUND 2. F INETUNING
The predominant technique for adapting the language model to a downstream task T is finetuning , when all parameters Θ of the model (initialized from some pre-trained weights) are updated during task adaptation (Devlin et al., 2018;
Zhang et al., 2020).
Consider a classification task T with input text x , and output scalar label y .
Here p Θ is a probability distribution of output classes parameterized by the weights, Θ , of the language model.
In finetuning, we perform gradient updates according to the following log-likelihood objective:
[Equation]
Despite its simplicity and effectiveness, finetuning updates all model parameters, and when there are multiple downstream tasks it requires storing a separate finetuned model for each of them.
2. P ROMPT TUNING
Recent works explored prompt tuning (Lester et al., 2021) as a lightweight alternative for finetuning, when instead of training the whole model, a series of virtual tokens – or soft prompt – is trained.
In prompt tuning, a soft prompt P is prepended to the input text x , while keeping other parameters frozen.
In this case, the model parameters Θ are composed of two sets: 1) the pretrained language model parameters θ which are fixed on downstream tuning, and 2) prompt parameters θ p which are fine-tuned on the given task.
Since prompt P has its own dedicated parameters θ p , we are not restricted by the original model parametrization θ , and can perform gradient updates on the prompt parameters independently.
Given a task T and a probability distribution p θ,θ p of output classes parameterized by θ and θ p , the learning objective here is to maximize the log-likelihood of y given the input text x concatenated with the soft prompt P , or
[Equation]
2. C ONTINUAL L EARNING
In this work, we focus on a continual learning setup, where the language model is presented with a sequence of m text classification tasks ( T 1 , ..., T m ) .
In each task we have a set if i.i.d.
training examples ( x i , y i )
Ni =1 , where x i is an input text and
y i is a label from a set of predefined labels Y k associated with the task T k .
We assume that the model is parameterized by Θ and has access to the task identity in both training and inference.
Hence, the learning objective across all tasks becomes:
[Equation]
The most straightforward approach for continual learning is finetuning, which sequentially optimizes loss for task k , k ∈ 1 ..m by updating all model parameters:
L k (Θ) = − (cid:80) x,
y ∈
T
k log p ( y | x, Θ) .
Continual finetuning will support forward knowledge to future tasks, but will result in catastrophic forgetting, where performance on the earlier tasks decreases after learning new tasks, and eventually leads to a higher generalization loss (Kirkpatrick et al., 2017; de Masson D’Autume et al., 2019;
McCloskey & Cohen, 1989).
M ETHOD
Progressive Prompts We propose Progressive Prompts , a continual learning approach which progressively learns a prompt P k for each new task
T k
(Figure 1).
With Progressive Prompts we learn a separate prompt P k for task T k , and concatenate it with all previously learned prompts P i , i < k , before prepending to the input embeddings.
During training, parameters θ of the language model are always frozen, and parameters θ P k corresponding to prompt P k are only trainable during learning
T k , and frozen afterwards.
The training objective for task T k ( k ∈ { 1 ...m } ) is to find prompt parameters θ P k that minimize the negative log probability of training examples under our progressive prompt and the frozen base model:
[Equation]
Such a progressive setup allows to achieve two goals for efficient CL: (1) it successfully eliminates catastrophic forgetting , and (2) allows forward transfer for subsequent tasks.
Since Progressive
Prompts trains a separate prompt for each encountered task without modifying its parameters when new tasks are learned, old tasks do not suffer from forgetting.
In addition to that, prompts learned on previous tasks allow information re-use for future tasks.
A similar phenomenon has been shown by Vu et al.
(2021) – prompts learned on informative source tasks served as a good initialization for other downstream tasks.
Embedding reparameterization Li & Liang (2021) have shown that directly optimizing prompt parameters can lead to training instability, while reparameterizing prompt embeddings matrix through a multi-layer perceptron (MLP) can improve performance.
At the same time, Liu et al.
(2021) report that prompt embedding reparameterization can hinder performance on certain tasks and is highly sensitive to its initialization.
To solve this issue, we propose adding a residual connection to the
MLP reparameterization.
This residual connection improves optimization, because it avoids learning an identity mapping by the MLP.
More specifically, we reparameterize prompt P k for task T k as follows:
[Equation]
After training on task k is complete, the reparameterization parameters MLP can be discarded, and prompt embeddings P k can be replaced by their corresponding projection P ′ k .
E XPERIMENTAL S ETUP 4. D ATASETS
Continual Learning Benchmark
We first evaluate our approach on the widely adopted CL benchmark for language models, which includes five text classification datasets by Zhang et al.
AG News (4 classes, news classification), Amazon reviews (5 classes, sentiment analysis), Yelp reviews (5 classes, sentiment analysis), DBpedia (14 classes, Wikipedia text classification) and Yahoo Answers (10 classes, Q&A classification).
We provide the task details in Appendix A.1 and sequences of tasks used in our experiments in Appendix A.2.
Following previous works on CL for BERT model, including IDBR (Huang et al., 2021) and
MBPA++ (de Masson D’Autume et al., 2019), for BERT-based experiments we use four different orders five tasks
We use train
IDBR (Huang et al
++ (de al
Following CL setup for T5 model as in LFPT5 (Qin & Joty, 2021), we use three different orders of the AG News, Amazon, Yahoo and DBpedia datasets.
We replicate Qin & Joty (2021) few-shot CL setting and sample 16 examples per task for the training set and keep the test sets unchanged.
Large number of tasks While previous CL approaches mostly focused on relatively short task sequences of 3-5 tasks (Huang et al., 2021; de Masson D’Autume et al., 2019; Qin & Joty, 2021), a more realistic CL scenario would be long sequences encompassing more tasks.
Hence, we create a benchmark of 15 text classification tasks to evaluate the performance of Progressive Prompts along with widely adopted CL approaches.
Our benchmark consist of the aforementioned five datasets from CL benchmark, combined with four tasks from GLUE benchmark (MNLI, QQP, RTE, SST2) (Wang et al., 2018), five tasks from SuperGLUE benchmark (Wang et al., 2019) (WiC, CB, COPA,
MultiRC, BoolQ), and IMDB movie reviews dataset (Maas et al., 2011).
To evaluate performance under different dataset sizes, we create three different versions of each dataset, with 20, 200 and 1000 training samples sampled per class, and report test set performance for each of these settings.
Again, we follow Huang et al.
(2021) practice and for every task we randomly hold out 500 samples per class from the training set for validation, and use early stopping according to the validation accuracy on all seen tasks.
We provide the task details in Appendix A.1 and sequences of 15 tasks used in our experiments in Appendix A.2.
Transfer learning experiments We also run ablation experiments to understand the effect of forward transfer with Progressive Prompts.
Firstly, we perform experiments on six pairs of transfer learning tasks from similar domains (IMDb & SST2, Amazon & Yelp, QQP & MRPC) following
Hendrycks et al.
Secondly, we perform experiments on SuperGLUE benchmark (Wang et al., ) with Progressive Prompts and compare the performance against original per-task prompt tuning by Lester et al.
Following Lester et al.
(2021) setup, we use full training sets of eight
SuperGLUE tasks and report best validation performance.
4. B ASELINES
We consider nine baseline methods for comparison with ProgressivePrompts:
• Finetune (Wang et al., 2020; de Masson D’Autume et al., 2019; Huang et al., 2021): train all model parameters on a sequence of tasks (without adding any regularization constraints or replaying samples from the previous tasks).
• EWC
Kirkpatrick et al., 2017): finetune the whole model with a regularization loss that prevents updating parameters that could interfere with previously learned tasks.
• A-GEM
(Chaudhry et al., 2018): save examples from the past tasks and restrict the gradients used to update the model on new tasks based on the retrieved examples.
• Experience replay : finetune the whole model with a memory buffer, and replay samples from old tasks when learning new tasks to avoid forgetting.
• MBPA++ (de Masson D’Autume et al., 2019): augment BERT with an episodic memory that saves all seen examples.
Perform replay during training, and local adaptation during test time.
• IDBR
(Huang et al., 2021): BERT-specific approach which continuously trains the whole model using data replay and a regularization loss, which applies sentence representation disentanglement into task-specific and task-generic spaces.
Current SOTA on CL benchmark with BERT.
• Per-task prompts (Lester et al., 2021): train a separate soft prompt for each task, while keeping the original model frozen.
This setup will eliminate catastrophic forgetting, since per-task parameters do not change when new tasks are learned, but will not result in forward transfer.
• PromptTuning
(Lester et al., 2021;
Qin & Joty, 2021): train a shared soft prompt sequentially on all tasks, while keeping the original model parameters frozen.
• LFPT5 (Qin & Joty, 2021): continuously train a soft prompt that simultaneously learns to solve the tasks and generate training samples, which are subsequently used in experience replay.
Current SOTA on CL benchmark with T5.
4. I MPLEMENTATION DETAILS
Progressive Prompts is a model-agnostic CL method that can be used with any transformer-based model.
In our experiments, we use two language models adopted by the previous lines of works in
CL for NLP: encoder-only BERT model (Devlin et al., 2018) and encoder-decoder T5 model (Raffel et al., 2020).
To compare Progressive Prompts to the recent CL approaches implemented specifically with BERT, we use the pre-trained BERT-base model as in IDBR and MBPA++ methods (Huang et al., 2021; de Masson D’Autume et al., 2019).
For comparison with LFPT5 approach (Qin & Joty, 2021) we use pre-trained T5-large 1 model.
BERT Following Devlin et al.
(2018), to predict the class of input text x , we use the representation of its first token h
[ CLS ] (which is encoded by a special beginning-of-a-sentence symbol, [CLS]) as a sentence representation.
Here h is the whole-input representation matrix from BERT encoder.
We apply a linear transformation parametrized by w and a softmax function to obtain the classification probabilities over classes c ∈ { 1 ... C} :
[Equation]
In BERT experiments we train a separate linear head in addition to prompt embeddings for each task, using cross-entropy loss between the predicted classification probabilities and ground truth class labels.
T5 Following Raffel et al.
(2020) and Lester et al.
(2021), we adopt text-to-text formulation for all T5 experiments, where classification labels are mapped into words (e.g. 0/1 could be encoded as ”True”/”False”).
T5 model applies a multi-headed self-attention over the input tokens followed by position-wise feed-forward layers to produce an output distribution over target tokens.
We train prompt embeddings for T5 model using cross-entropy loss.
We report the rest of the implementation details in Appendix A.3.
Prompt length For all Progressive Prompt experiments on BERT, we set single-task prompt length to 20 tokens and apply prompt reparameterization with a two-layer residual MLP.
For T5 experiments, we use single-task prompt length to 10 tokens in case of long-sequence experiments andin case of T5 experiments (so that total Progressive Prompt length more closely matches LFPT prompt length of 300).
We report more experimental details in Appendix A.4.
E XPERIMENTAL R ESULTS
For all CL experiments we evaluate methods after training on all tasks and report averaged test set scores across all tasks.
Metrics used for different tasks are shown in Appendix A.1.
We report Progressive Prompts performance with BERT-base and T5-large models, and compare it to the existing
CL approaches used with these models.
[Table]
5. R ESULTS ON STANDARD C ONTINUAL L EARNING BENCHMARKS
T5 benchmark .
Table 1a compares Progressive Prompts performance on the few-shot CL benchmark for T5 model with the existing CL approaches, including previous SOTA – LFPT5 (Qin &
Joty, 2021).
Notably, Progressive Prompts achieve over 20% improvement compared to LFPT5.
This drastic increase in accuracy is explained by the fact that Progressive Prompts method does not experience forgetting and allows forward transfer, which is especially useful in few-shot regime.
Of note, Qin & Joty (2021) report better performance of LFPT5 in continual learning setting than
Adapter-Fusion (Pfeiffer et al., 2020) – a strong parameter-efficient CL approach based on adapters.
Hence, we did not include adapter baselines in this study, but instead focused on a more parameterefficient and better-performing method – LFPT5.
BERT benchmark .
Table 1b compares performance of Progressive Prompts across four different task orders with existing CL approaches for BERT model, including previous SOTA – IDBR (Huang et al., 2021).
We want to note that CL benchmark for BERT focuses on full dataset experiments, contrary to few-shot CL benchmark for T5.
Since Progressive Prompts are most efficient in few-shot setting, we observe a larger performance gap with LFPT5 compared to IDBR.
Overall, Progressive
Prompts further improve performance compared to IDBR, reaching 77.9 average score across all or- ders.
We also note that Progressive Prompts is a model-agnostic approach, contrary to IDBR, which is designed specifically for BERT.
Additionally, our method does not require storing data from the previous tasks for future replay, in contrast to the previous commonly used CL approaches for BERT – IDBR and MBPA++.
In all of our experiments we use the embedding reparameterization defined in Section 3.
We find that this reparameterization helps in stabilizing and accelerating training.
Due space limitations, we explore the effect of residual reparameterization in Appendix B.3).
5. P ERFORMANCE WITH LARGE NUMBER OF TASKS
Table 2 compares common CL approaches, including SOTA methods for T5 and BERT models (Qin & Joty, 2021;
Huang et al., 2021), on a sequence continual learning with 15 tasks.
We report averaged results across three task orders (8, 9 and 10) obtained with T5-Large and BERT-base models.
We provide the full non-averaged results for each order in Appendix B.1.
To investigate the effect of limited data settings, we perform training on different dataset sizes with 20, 200 and 1000 samples per class.
[Table]
Our approach, Progressive Prompts, consistently outperforms all other methods across different data limits, with the strongest improvement in the few-shot setting ofsamples per class: +21 .
9% and +33 .
3% over previous
SOTA approaches on T5 and BERT models respectively.
We also provide more fine-grained performance metrics defined by Lopez-Paz & Ranzato (2017), including forward transfer (FWT) and backward transfer (BWT) scores, as well as evolution of accuracy with learning new tasks in Appendix B.1.
Attention between prompts To investigate which prompts from the progressive pool are helpful for learning new tasks, we visualized between-prompt attentions (Figure 2).
We used Progressive Prompts trained on order 8 with T5 model.
For each of the 15 consecutive tasks, we passed all test examples through T5 encoder, and extracted attention matrices from its last layer (24th layer).
We averaged attention scores across all prompt tokens and all heads, and shown the results in Figure 2.
Interestingly, prompts learn to ignore irrelevant tasks and attend to the most similar or informative tasks.
For example, prompt learned on Amazon reviews task heavily attends to Yelp reviews prompt, and SST prompt attends to the prompt learned on IMDb - another sentiment analysis task.
5. F ORWARD T RANSFER EXPERIMENTS
In this section we study in more detail the forward transfer phenomenon achieved with Progressive
Prompts.
In particular, can a subsequent task prompt exploit the knowledge from a previously learned prompt in learning a new related task?
Transfer Learning on a pair of tasks We compare the performance of a single prompt of length 2 M versus a Progressive Prompt consisting of two M -length prompts, M = 50 (see Figure 3).
We follow a similar transfer learning experimental protocol by Hendrycks et al.
and use pairs of six tasks from similar domains: IMDb and SST2 (2-class sentiment classification), Amazon and
Yelp reviews (5-class sentiment classification), MRPC and QQP (paraphrase detection).
We use
T5-Large model, and study both settings: full-dataset and few-shot (2, 5 and 20 samples per class).
Our results are summarized in Table 3 and Figure 4.
Overall, we observe that Progressive Prompts achieve forward transfer and outperform the standard prompt tuning across all dataset sizes.
The strongest improvements happen in few-shot setting – +20 .
4% and +12 .
9% relative improvement under 5-shot and 2-shot setup.
[Table]
[Table]
GLUE → SuperGLUE transfer We perform additional experiments on SuperGLUE benchmark to compare the forward transfer effect of Progressive Prompts with standard prompt tuning.
In the original prompt tuning paper, a separate prompt P T is trained for each SuperGLUE task T (Lester et al., 2021).
P T is a trainable matrix of 100 × e dimension, where 100 is the prompt length and e if the embedding dimension.
P T is concatenated with the input text x and then fed into the model as [ P T ; x ] .
For Progressive Prompt we perform training in two steps.
First, we learn a progressive prompt on GLUE benchmark P GLUE .
P GLUE is composed of 6 per-task GLUE prompts with a length of 10 on random task order, and the final dimension of P GLUE is 60 × e .
After that, we learn a length-40 prompt [ P T Prog for every SuperGLUE task T and concatenate it with the GLUE prompt before appending to the input x : [ P T Prog ; P GLUE ; x ] (illustration of the experimental setup is in Appendix B.2). [Table]
Following Lester et al.
we train a separate soft prompt for each SuperGLUE task on its full train set with a fixed number of epochs, and report the validation set performance.
Similarly to Lester et al.
(2021), for each SuperGLUE task use metrics recommended by Wang et al.
(2019) and for tasks with two metrics we compute an average of the two.
Our results are shown in Table 4 – Progressive Prompt improves the standard prompt tuning performance by 2.7 points on SuperGLUE benchmark.
This confirms that progressive prompt concatenation allows knowledge reuse from the previous prompts that were learned on GLUE benchmark.
R ELATED WORK
Continual Learning Existing continual learning approaches can be broadly organized into three main categories: (i) replay-based , (ii) regularization-based , and (iii) architecture-based (de Masson D’Autume et al., 2019; Huang et al., 2021).
Replay-based methods store a subset of data from previous tasks for future rehearsal via experience replay, representation consolidation or constrained optimization.
The data can be either stored directly or synthesized by generative models.
Regularization-based approaches restrict changes of model’s parameters to avoid inference with previously learned tasks (Li & Hoiem, 2017; Kirkpatrick et al., 2017).
Architecture-based approaches learn different set of parameters dedicated for a separate task.
Recently, replay-based and regularization-based approaches have been successfully applied for continual learning in language models.
While replay-based approaches have shown strong results for classification, question answering and relation extraction tasks, they pose significant memory requirements due to storing large number of samples for rehearsal (Rolnick et al., 2019; Huang et al., 2021).
Additionally, for many applications such storage would not be feasible due to privacy settings, when access to past data is not available.
Regularization-based approaches are more memory-efficient than replay-based approaches, but suffer from catastrophic forgetting and are often not suitable for long task sequences (Kirkpatrick et al., 2017; Razdaibiedina et al., 2022).
In contrast to regularization-based and replaybased approaches, architectural CL approaches are more efficient in resolving catastrophic forgetting and, hence, are suitable for long sequences of tasks.
Parameter-efficient Learning Recent works on parameter-efficient learning have shown that by training a subset of parameters, we can achieve a full model performance (Houlsby et al., 2019;
Li & Liang, 2021; Karimi Mahabadi et al., 2021; Lester et al., 2021).
While this line of work has mostly focused on learning a single task, there have been some attempts on using parameter-efficient tuning for CL (Madotto et al., 2020; Qin & Joty, 2021; Wang et al., 2022b;a).
For instance, Madotto et al.
(2020) proposes AdapterCL which learns a separate adapter block for each task, and Qin & Joty (2021) proposes LFPT5 that learns a large (length 300) soft prompt that is continuously trained on all tasks.
While AdapterCL resolves catastrophic forgetting problem, it does not allow forward transfer.
In contrast, LFPT5 allows forward transfer but suffers from forgetting.
C ONCLUSION
This paper presents Progressive Prompts – a novel approach for CL that addresses catastrophic forgetting in pre-trained language models, while allowing knowledge reuse from previous tasks.
In contrast to many existing CL methods for NLP, the proposed approach does not require saving examples from the previous tasks for data replay.
Moreover, our method does not require storing a large number of task-specific parameters.
Progressive Prompts is a model-agnostic approach and our experiments with two commonly used language models demonstrate that Progressive Prompts outperforms baseline methods on a standard CL benchmark for text classification and our custom benchmark of longer CL sequences that spans 15 tasks. A CKNOWLEDGMENTS
We thank Jimmy Ba for many fruitful discussions and brainstorming of the ideas, which helped to improve this project. We also thank Victoria Lin and the FAIR team for their helpful comments. Finally, we thank the anonymous reviewers whose feedback helped to improve this paper Appendix A F URTHER IMPLEMENTATION DETAILS A. D ATASETS
Table 5 shows details of the 15 datasets we used for our CL experiments, along with their evaluation metrics.
Overall, we used datasets from CL benchmark (Zhang et al., 2015), GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) benchmarks [Table] and added IMDB movie reviews dataset.
Following common practive, for tasks that have two evaluation metrics we use the average of the two as the final performance metric.
[Table]
A. T ASK SEQUENCE ORDERS
We report task orders used for our CL experiments across BERT and T5 models in Table 6 below:
Order
Model Task Sequence
[Table]
A. I MPLEMENTATION
We use PyTorch (Paszke et al., 2019) and HuggingFace Transformers library (Wolf et al., 2019) for our implementation.
For the standard CL benchmark, we use official datasets provided by
Zhang et al.
(2015) available at http://goo.gl/JyCnZq , following de Masson D’Autume et al.
; Zhang et al.
We use HuggingFace datasets ( https://github.com/ huggingface/datasets ) to download data for GLUE tasks (Wang et al., 2018), SuperGLUE tasks (Wang et al., 2019) tasks, and IMDB movie reviews dataset (Maas et al., 2011), which we use for long-sequence CL experiments and/or ablation studies.
Following previous studies (Rao et al., 2019; de Masson D’Autume et al., 2019), for CL experiments, for each dataset we use the available validation set as a test set (since test data is not available), and hold out 500 samples from the train set to construct the validation set.
For our ablation studies, since we compare Progressive Prompts with the original prompt tuning (Lester et al., 2021), we follow their set up and report maximal validation set performance.
A. E XPERIMENT DETAILS
We use Adam optimizer (Kingma & Ba, 2014) and set batch size to 8 for all the experiments, except for MTL runs with a batch size of 2 (due to memory limitations).
We train each prompt betweenand 300 epochs, depending on the number of data points.
We use the prompt checkpoints with the best validation set score as our final prompts.
Prompts are initialized from randomly sampled tokens as in Lester et al.
(2021), hyperparametes are shown in the Table 7 below:
Hyperparameter ↓
CL benchmark
Long-sequence benchmark
Num.
samples → -
BERT
Epochs
Learning rate 1 e −1 e −1 e −1 e
[Table]
For all CL experiments we use early stopping as in Huang et al.
(2021), to save model checkpoint based on the best validation performance on the current task.
We report test set performance after training on all tasks as our final metric.
For SuperGLUE experiments, we report maximal validation set performance over the course of training as in Lester et al.
We measure the validation performance after every epoch and use metrics described in Appendix A.1.
We use 1% of samples per class for the replay approach (but no less than 1 sample per class), following Huang et al.
We use the same hyperparameter setting for all prompt-based approaches (Progressive Prompts, prompt tuning, per-task prompts), except for prompt tuning we use a longer shared prompt oftokens.
For all other approaches, we use hyperparameters provided in their corresponding papers.
B F URTHER EXPERIMENTAL RESULTS B. L ONG SEQUENCE EXPERIMENTS
[Table]
Here we report results for orders 8, 9 and 10 of continual learning experiments with long task sequences of 15 tasks.
We show test set performance averaged across all tasks for each method.
Test scores are calculated after training has been completed.
Results are shown in Table 9.
We also investigate improvement of Progressive Prompts compared to per-task prompt on the corresponding task in Figure 12.
Clearly, some tasks benefit from knowledge sharing from the progressively added prompts.
Additionally, we assessed if initializing new prompt from the previous task prompt would result in better performance of continual prompt tuning.
We observe that Progressive Prompts outperform this setup in both few-shot (20/class) and full-shot settings, see Table 8.
In addition to average performance [Table] we compute more fine-grained performance metrics defined by Lopez-Paz & Ranzato (2017) for different approaches under long-sequence experiments.
Specifically
[Table] we compute backward transfer and forward transfer metrics [Table] and evolution of average accuracy over learning new tasks.
Our results on FWT for task orders 8, 9 and 10 are shown in
Figure 5, Figure 6 and Figure 7 respectively.
Our results on BWT for task orders 8, 9 andare shown in Figure 8, Figure 9 and Figure 10 respectively.
Evolution of accuracies is shown in
Figure 11.
[Table]
B. S UPER GLUE EXPERIMENTS SETUP
Comparison of the original prompt tuning on SuperGLUE (Lester et al., 2021) and our Progressive
Prompt setup is shown in Figure
. B. E FFECT OF PROMPT REPARAMETERIZATION
We find that residual reparameterization allows to reach performance close to finetuning, and is especially helpful for BERT model.
Table 10 shows the results of regular prompt tuning, prompt tuning with MLP reparameterization and prompt tuning with residual MLP reparameterization on
BERT-base model.
As in all our experiments, we use a 2-layer MLP with the hidden layer dimension of 800.
We show the best performance on four different tasks with prompts of length 5 and 30.
Following Lester et al.
(2021), we report the maximal validation set performance for each dataset.
Our results show that residual MLP reparameterization results in performance improvement over standard prompt tuning, reaching close to finetuning performance (Table 10).
Notably, with length5 prompt, residual MLP improves accuracy by approximately 6% and 4% for IMDB and QQP datasets, matching full model tuning.
Regular MLP reparameterization generally leads to either smaller improvement than residual MLP or even worse performance than prompt tuning. [Figure]
Prompt tuning trains a single prompt of 100 tokens for WiC task
Progressive Prompts learns a prompt of 40 tokens is progressively appended to the six frozen prompts of 10 tokens learned on GLUE benchmark (with random task order) is
[Table]