Residual Prompt Tuning: improving prompt tuning with residual reparameterization (2023)

Abstract: Prompt tuning is one of the successful approaches for parameter-efficient tuning of pre-trained language models. Despite being arguably the most parameter-efficient (tuned soft prompts constitute <0.1% of total parameters), it typically performs worse than other efficient tuning methods and is quite sensitive to hyper-parameters. In this work, we introduce Residual Prompt Tuning - a simple and efficient method that significantly improves the performance and stability of prompt tuning. We propose to reparameterize soft prompt embeddings using a shallow network with a residual connection. Our experiments show that Residual Prompt Tuning significantly outperforms prompt tuning across T5-Large, T5-Base and BERT-Base models. Notably, our method reaches +7 points improvement over prompt tuning on SuperGLUE benchmark with T5-Base model and allows to reduce the prompt length by 10 times without hurting performance. In addition, we show that our approach is robust to the choice of learning rate and prompt initialization, and is effective in few-shot settings.

## introduction pre-trained language models have achieved remarkable performance on a variety of natural language understanding tasks @xcite @xcite . recent studies have shown that scaling up model size consistently leads to performance gains @xcite @xcite , and larger scale models are becoming increasingly more common, e.g. gpt-3, 175b parameters @xcite , mt-nlg, 530b parameters @xcite . despite the significant performance improvement achieved with larger-scale models, their applicability is limited due to their size. the standard practice of fine-tuning becomes prohibitively expensive 1 our code is available at https://github.com/ arazd/residualprompts . since it requires storing gradients and optimizer states for all model parameters. additionally, storing a separate copy of a fine-tuned model for each task is infeasible for billion-parameter models. to address the challenges associated with full model tuning, a line of research has focused on prompt design, where natural language prompts are used to query a frozen model @xcite . in this setup, all tasks are cast as language modeling tasks (e.g. 0/1 classes could be encoded as "true"/"false"), and manually selected prompts condition the frozen model to generate the desired output. despite the fact that prompt design can achieve strong few-shot performance, manually finding optimal prompts remains challenging and time-consuming @xcite . additionally, different prompt choices often lead to large variances in the final performance @xcite . recently, @xcite proposed prompt tuning -a method of learning soft prompts through gradient descent instead of designing the prompts manually. soft prompts are a series of continuous embeddings prepended to the input, which are updated throughout training, and typically @xmath0.1% of the total parameters. notably, prompt tuning has been shown to perform close to full model tuning when model size increases, closing the performance gap when the model contains over 11b parameters @xcite . nevertheless, prompt tuning still underperforms with smaller models, and its performance can vary significantly depending on the choice of hyperparameters, such as prompt initialization and learning rate @xcite . furthermore, prompt tuning generally requires long training and a large number of prompt tokens (over 100) to achieve stable performance @xcite . in this work, we present residual prompt tuning, a method that can significantly improve and stabilize prompt tuning performance through residual reparameterization of prompt embeddings (figure 1 ). residual prompt tuning passes soft prompt embeddings through a shallow network with a residual connection, and subsequently prepends reparameterized prompt to the input and feeds to the language model. this reparameterization gives the model more flexibility to decide between using a separate embedding for each prompt token versus the representation obtained from the shared reparameterization network. after training is completed, the reparameterization network can be discarded and original prompt embeddings can be replaced with their projections. we conduct extensive experiments on super-glue tasks with t5-large, t5-base and bert-base models @xcite and demonstrate that residual prompt tuning outperforms previous prompt tuningbased methods by a large margin, achieving +7 points improvement over prompt tuning on super-glue with t5-base. we also show that resid-ual prompt tuning reduces performance vari-ance under different learning rates or prompt initializations, and achieves strong performance with fewer training iterations. finally, we show that residual prompt tuning significantly improves over prompt tuning in few-shot settings. despite its effectiveness, fine-tuning updates all model parameters, which can be prohibitively expensive for large language models. prompt tuning. @xcite proposed prompt tuning as a lightweight alternative to fine-tuning. the main idea is to prepend a sequence of virtual token embeddings, or a soft prompt p , to the input text x, and learn only them on the downstream task while keeping other model parameters fixed. the model parameters θ are now composed of the frozen pre-trained language model parameters, and the additional soft prompt parameters θ p , which are tuned on the downstream task. the training objective becomes: prompt tuning offers an attractive parameterefficient solution to repurpose pre-trained models for many real-world applications. however, training soft prompts often requires extensive hyperparameter tuning and longer training time to achieve the desired performance @xcite . ## results we describe our main results showing the effectiveness of residual prompt tuning compared to other prompt tuning-based methods and parameterefficient methods in section 5.1. we study the robustness of our method to the choice of hyperparameters in sections 5.2 and 5.3. then, we explore the performance of residual prompt tuning in more extreme settings, including smaller prompt sizes (section 5.4) and few-shot data regime (section 5.5). ## related work parameter-efficient tuning methods. recent approaches have explored parameter-efficient tuning (peft) of language models, where only a subset of parameters is trained while the rest of the model is kept frozen. @xcite proposed adapters -small modules injected between each transformer layer. to improve over original adapter tuning, several works proposed to remove adapter modules from lower transformer layers @xcite , or use a composition of adapter modules @xcite . other works focused on low-rank adaptations (lora) @xcite , and prefix tuning (li and liang, 2021). similarly to adapters, lora injects additional trainable weight matrices into each transformer layer, requiring changes to the intrinsic model structure and adding a high number of extra parameters. prompt tuning methods. to overcome the drawbacks of traditional peft methods, @xcite introduced prompt tuning as a highly parameter-efficient approach, where tuned soft prompts @xmath1.1% of the total parameters and can be easily appended to the input without modifying the model. several methods were recently introduced to improve over prompt tuning. @xcite proposed adding soft prompts at every transformer layer. while their method improves performance, it requires much more trainable parameters (10x in some cases). other works explored transfer learning-based methods to find better prompt initialization through pre-training @xcite . these methods pre-train soft prompts on a collection of source tasks and subsequently use the learned prompt embeddings to initialize prompts for target tasks. reparameterization methods. although reparameterization has not been traditionally used with prompt tuning, li and liang (2021) explored reparameterization of embeddings as a way to improve the performance of prefix tuning, and @xcite explored reparameterizing injectable embeddings together full model tuning. with these approaches, prefix embeddings are passed through a shallow neural network, such as mlp (in prefix tuning) or lstm (in gpt2 tuning by @xcite ), before being concatenated to the input embeddings (or representations) and passed into a subsequent layer of the language model. @xcite explores mlp-based reparameterization for p-tuning v2. despite improvements on some tasks, @xcite finds that the reparameterization effect is not consistent across datasets and can hinder the performance of certain tasks. ## conclusion we propose residual prompt tuning, a new method for learning soft prompts under a frozen language model using residual reparameterization of prompt embeddings. our method enables efficient learning of soft prompts, without the need for extensive hyperparameter search, long training times, or pre-training on source tasks. the experiments show that residual prompt tun-ing significantly outperforms prompt tuning by @xcite and its two variations across three model architectures (t5-large, t5-base and bert-base) on superglue benchmark. furthermore, our method is robust to the hyperparameter choice (learning rate and prompt initialization), speeds up convergence and is highly effective in few-shot settings.