Decomposed Prompt Tuning via Low-Rank Reparameterization (2023)

Abstract: While prompt tuning approaches have achieved competitive performance with high efficiency, we observe that they invariably employ the same initialization process, wherein the soft prompt is either randomly initialized or derived from an existing embedding vocabulary. In contrast to these conventional methods, this study aims to investigate an alternative way to derive soft prompt. Our empirical studies show that the soft prompt typically exhibits a low “intrinsic rank” characteristic. With such observations, we propose decomposed prompt tuning, a novel approach that utilizes low-rank matrices to initialize the soft prompt. Through the low-rank reparameterization, our method significantly reduces the number of trainable parameters while maintaining effectiveness. Experimental results on the SuperGLUE benchmark in both high-resource and low-resource scenarios demonstrate the effectiveness of the proposed method.

## introduction pre-trained language models @xcite @xcite @xcite have achieved remarkable performance on various natural language understanding and generation tasks. the pretrain-thenfinetune paradigm has been adopted as a common approach to deal with downstream tasks. however, such a paradigm is often considered inefficient, especially in the era of large language models (llms), as it requires tuning a large number of model parameters and saving a separate model for each task. recently, parameter-efficient fine-tuning (peft) approaches @xcite @xcite @xcite @xcite have been proposed to address this challenge. the main idea t5-small t5-base t5-large model scale 60 65 70 75 80 superglue average score ours res pt pt figure 1 : average performance over all the datasets of superglue with t5 models. the number of trainable parameters used are 11.2k, 102k, and 925k with t5-large for dpt, vanilla prompt tuning (pt) @xcite , and residual pt (res pt) @xcite , respectively. more details are included in section 5.1. of these methods is to fine-tune only a subset of the model's parameters or additionally introduced parameters while freezing the majority of parameters of a pre-trained model. these approaches require saving only the trainable parameters for different tasks, striking a balance between performance and efficiency. the success of peft methods is also aligned with the previous findings that pre-trained models possess a low "intrinsic rank" @xcite . @xcite empirically showed that employing a low-dimensional reparameterization is equally effective as fullmodel fine-tuning. @xcite further demonstrated that weight updates during model training also exhibit a low "intrinsic rank". by only tuning the proposed low-rank adaptation module, their approach achieves high efficiency while maintaining competitive performance. another line of research work focuses on p*-tuning @xcite @xcite . specifically, @xcite proposed prefix tuning by prepending a sequence of virtual tokens to each transformer layer, and it updates the representations of these virtual tokens while keeping the pretrained model frozen. prompt tuning @xcite further simplified prefix tuning by updating only a sequence of continuous prompt tokens in the embedding layer. owing to its simplicity, subsequent studies @xcite continuously improve the vanilla prompt tuning approach. despite the substantial achievements of these prompt tuning methods, we observe that they invariably employ the same initialization process, wherein the soft prompt is either randomly initialized or derived from the existing embedding vocabulary. different from the previous approaches, this paper aims to explore an alternative approach to deriving soft prompt. our motivation stems from the observations that weight updates in the transformer layer during training have a low "intrinsic rank", as highlighted by @xcite . this leads us to inquire whether soft prompt also have a similar low "intrinsic rank" pattern. to this end, we conduct studies examining the "intrinsic rank" of soft prompt, and we include the details of the analysis in section 2. based on the studies, we find that the soft prompt indeed tends to exhibit a low "intrinsic rank" behavior. armed with this insight, we propose decomposed prompt tuning (dpt), a novel approach that employs low-rank matrices to initialize the soft prompt. specifically, we decompose the original soft prompt into the product of two compact matrices, and we include the detailed description in section 3. with this low-rank reparameterization, dpt significantly reduces the number of trainable parameters while achieving strong performance. a comparison with previous approaches, depicted in figure 1 , verifies the efficacy of our method. our contribution can be summarized as follows: ## emergence of "intrinsic rank" to investigate the rank of soft prompt, we design an analysis based on the vanilla prompt tuning approach on the sub-tasks within the superglue benchmark. specifically, we re-design the soft prompt, which facilitates a probe into its rank. ## method motivated by the observation of the soft prompt's low "intrinsic rank" behavior, as discussed in the previous section, we propose a parameterization of the soft prompt that explicitly constrains it to be of low rank. in this section, we provide the details of our approach, as illustrated on the right side of figure 2 . ## analysis 6.1 sensitivity of bottleneck size as mentioned in section 3.1, the rank of our soft prompt can be adjusted through the bottleneck b. in this section, we investigate how the size of the bottleneck impacts the performance of dpt. we conduct experiments using t5-large on the wic, cb, rte, and copa datasets, with bottleneck @xmath0, 6, 8, 10, 12, 14}. the results are shown in figure 5 . we can see that though our approach experiences performance fluctuations with different sizes of the bottleneck b, dpt outperforms the vanilla prompt tuning and residual prompt tuning in quite a few cases. intuitively, the size of the bottleneck plays a critical role in the expressiveness of our model. we observe a performance drop when the size of the bottleneck is getting smaller. we find similar behavior when t5-base is used, and the results can be found in appendix a.7. ## related work parameter-efficient fine-tuning (peft) methods @xcite @xcite @xcite have emerged as a popular approach to fine-tune language model which can largely reduce the number of trainable parameters while maintaining competitive performance. there are two primary paradigms for current peft methods: adapter-based and prompt-based approaches. ## conclusion in this work, we uncover the low "intrinsic rank" behavior inherent in the soft prompt through empirical examination. motivated by these findings, we introduce decomposed prompt tuning (dpt), a novel approach that reparameterizes the soft prompt using two compact matrices. by adjusting the bottleneck, dpt enables the effective manipulation of the soft prompt matrix, ensuring it maintains a low rank. notably, dpt attains strong performance across the superglue benchmark in high-resource and low-resource scenarios while substantially reducing the number of trainable parameters.