Q -Tuning: Queue-based Prompt Tuning for Lifelong Few-shot Language Learning (2024)

Abstract: This paper introduces Q-tuning, a novel approach for continual prompt tuning that enables the lifelong learning of a pre-trained language model. When learning a new task, Q-tuning trains a task-specific prompt by adding it to a prompt queue consisting of the prompts from older tasks. To better transfer the knowledge of old tasks, we design an adaptive knowledge aggregation technique that reweighs previous prompts in the queue with a learnable low-rank matrix. Once the prompt queue reaches its maximum capacity, we leverage a PCA-based eviction rule to reduce the queue’s size, allowing the newly trained prompt to be added while preserving the primary knowledge of old tasks. In order to mitigate the accumulation of information loss caused by the eviction, we additionally propose a globally shared prefix prompt and a memory retention regularization based on information theory. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods substantially on continual prompt tuning benchmarks. Moreover, our approach enables lifelong learning on linearly growing task sequences while requiring constant complexity for training and inference.

## introduction in recent years, pretrained language models (lms) have achieved huge success in natural language processing @xcite @xcite @xcite , which popularizes the pretrainingfinetuning pipeline in applications. however, with the ever-growing parameter scale of modern lms (e.g., gpt-4 that may have 1.76 trillion parameters (openai, 2023)), it becomes increasingly difficult to finetune the whole model, leading to extensive attention to parameter-efficient finetuning (peft) technologies. prompt tuning (pt) @xcite has recently emerged as a leading peft solution. pt trains soft prompts and prepends them to the input of lms, while keeping the lm parameters frozen. existing works @xcite have shown that pt can achieve performance on par with finetuning, while requiring fewer than 0.01% of the total trainable parameters. continual prompt tuning (cpt) is a methodology that extends pt to the continual learning (cl) paradigm for learning new tasks that arrive in a sequential fashion. cpt encounters technical challenges akin to those faced by traditional cl methods, including the well-known catastrophic forgetting (cf) @xcite and forward knowledge transfer (fkt). to overcome these challenges, @xcite designed a dual prompt tuning framework including a globally shared prompt and a task-specific prompt. however, continuously optimizing the shared prompt for new tasks will make the learned knowledge from old tasks vanish, leading to less efficient fkt. to improve the fkt, progprompt was proposed by @xcite which maintains a prompt list for incoming tasks by progressively adding newly trained prompts while storing all previously trained prompts. following a similar strategy to extending the prompt list, @xcite proposed a prompt set expansion method by weighting the sum over a group of prompt components for each task. although these methods succeed in improving the fkt, they suffer from the same problem when the length of prompts grows linearly at a rate of o(n ) along with the number of tasks. this leads to an o(n 2 ) complexity for transformer-based models. consequently, the training and inference costs will become intractable as n increases and exceeds a finite computation resource limit. in this paper, we overcome the aforementioned challenges by proposing a novel continual prompt tuning technology named queue-based prompt tun-ing (q-tuning). q-tuning manages a queue-based prompt (q-prompt), which is stored in a finite-size data buffer. for learning a new task, q-tuning trains a new prompt combined with the fixed q-prompt that stores all previously learned prompts. upon the completion of tuning, the latest trained prompt will be added to the q-prompt for the tuning of the next task. once the number of tasks exceeds the queuesize limit, we will remove less informative prompts according to a principal component analysis (pca) based dequeue rule. this endows q-tuning with the ability to perform lifelong prompt tuning on extremely long task sequences. our key contributions and results can be summarized as follows: a novel dequeue rule based on pca is applied to trim the q-prompt when it is full. in addition, a globally shared prefix prompt with a memory retention (mr) technique is devised to mitigate the information loss due to dequeuing. to address the aforementioned limitations, we introduce q-tuning, which is data-buffer-free and enables anti-forgetting lifelong learning in the face of an ever-expanding stream of tasks. for each task, only the newly appended prompt is trainable, while the previously trained prompts are fixed. however, when n grows asymptotically (i.e., the model is set as a lifelong learner), training the extremely long prompt list becomes intractable due to the finite system resources. this motivates us to propose the q-tuning technique. fig. 1 illustrates the overall framework of the q-tuning technique. in q-tuning, we add a new trainable prompt to a prompt queue q that stores all previously trained prompts for old tasks. this updated q associated with a globally shared prefix prompt will be tuned for the new task while keeping the prior prompts in q frozen. this progressively appending approach enables forward knowledge transfer as the old task's information is saved in the q-prompt. we let @xmath0, where l is the length of a single prompt per task and q size is the maximum number of prompts in the queue. when reaching the capacity limit of q, the q-prompt will be trimmed using an eviction rule to remove less informative prompts and append new trainable prompts for future tasks. knowledge aggregation for fkt: in q-tuning, all prompts in the memory (i.e., the q-prompt q), as well as the pretrained lm model, are frozen when learning a new task. consequently, the lm model will be forced to take these fixed prompts in the queue as inputs without considering their relevance to the current task, leading to suboptimal performance. to address this problem, we propose a prompt-based knowledge aggregation mechanism. for task i, we use a trainable matrix w @xmath1, which is of the same dimension as the q-prompt q i , to scale q i by w i • q i (• denotes the hadamard product). here, for task i, we denote the total prompt length of q i by c @xmath2. since directly optimizing a large-scale matrix of size c i × d is costly, we propose a low-rank multiplicative method inspired by @xcite . the weight matrix w i can be expressed as w @xmath3, where u @xmath4, v @xmath5. clearly, w i is a rank-one matrix, and the number of trainable parameters is reduced to c i + d ≪ c i × d. we jointly optimize the newly appended prompt θ i p and the low-rank aggregation matrix w i by maximizing the cross-entropy loss as follows: de-q rule: our q-prompt design allows appending newly trained prompts until they reach the maximum length. once the q-prompt is full (denoted by q c ), a dequeuing (de-q) rule is executed to reduce the length of q c to cl to add the new prompt. however, this leads to a challenge: how to retain the most useful prompt information after trimming the q-prompt? straightforward de-q rules include random eviction and first in first out (fifo). however, these simple rules may discard valuable information in the queue, resulting in negative impacts on fkt. to address the above problem, we introduce a simple yet effective de-q rule named dq-pca based on principal component analysis (pca) @xcite . specifically, we first calculate the centered q-prompt @xmath6: @xmath7. then we perform singular value decomposition (svd). we extract the first cl principal components to obtain the trimmed q-prompt qc-@xmath8. this process can be written as follows: propose a prompt queue (q-prompt) and a globally shared prefix prompt θ i p * to achieve the forward knowledge transfer, where the superscript of θ i p * denotes the i-th status. moreover, we adopt a knowledge aggregation method to adaptively adjust the contribution of each fixed prompt [θ foot_1 p , θ foot_3 p , . . . , θ i-1 p ] in q-prompt by using a rank-one matrix w i . we parameterize the trainable soft prompt by a two-layer residual mlp. if the length of the q-prompt exceeds the limit, we apply a de-q rule to discard less informative prompts in the queue. ## experimental results we report q-tuning performance on t5-large and bert-base models and compare it to previous cl and prompt tuning approaches. we evaluate the methods after training on all tasks and report the averaged test set accuracy across all tasks. the detailed experimental metrics are reported in appendix c.1. all the experiments are conducted on a single 32gb nvidia v100 gpu. ## conclusion this paper introduces a new model-agnostic approach named q-tuning, which can pave the way to achieving lifelong continual prompt tuning for present and future lms with a rapid growth of parameters. in comparison with existing cl methods, q-tuning maintains a low-cost prompt queue instead of storing a large number of task-specific parameters or saving old data samples for replay. our extensive experiments demonstrate that q-tuning outperforms existing continual learning, prompt tuning and continual prompt tuning methods on the standard cl benchmarks for text classification. in addition, we verify the effectiveness of q-tuning on both short and long task sequences, including up to 70 tasks that mimic the case of lifelong learning. limitations: although q-tuning demonstrates a strong fkt capability, it does not enable the backward knowledge transfer as both the model and the previous q-prompts are frozen during the learning of a new task. besides, q-tuning requires the task identities to be known at test time. to address the more challenging cl scenario when the task identities are undisclosed at test time, for task i, we can assign a trainable query key k i to the corresponding q-prompt q i and jointly train k i to maximize the similarity between k i and the feature of each sample x from task i. during test time, given an input x ′ with an unknown identity, we will first locate the q-prompt that has the largest similarity between its key k j and the input x ′ , and then we can use the retrieved q-prompt q j to infer x ′ . we will address this problem in our future work.