Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Product (2025)

Abstract: Prompt tuning (PT) offers a cost-effective alternative to fine-tuning large-scale pre-trained language models (PLMs), requiring only a few parameters in soft prompt tokens added before the input text. However, existing PT approaches face two significant issues: i They overlook intrinsic semantic associations between soft prompt tokens, leading to high discreteness and limited interactions, thus reducing the model’s comprehension and effectiveness in complex tasks. ii Due to the complexity of downstream tasks, long soft prompt is necessitated to improve performance, but prompt length correlates positively with memory usage and computational costs. Achieving high efficiency and performance remains an ongoing challenge. To address these issues, we propose a novel Low-parameters Prompt Tuning (LAMP) method, which leverages prompt decomposition and compressed outer product. Specifically, the prompt decomposition module employs Truncated SVD to reduce training parameters and significantly lower the dimensionality of the soft prompt parameter space. It then utilizes a compressed outer product module to facilitate multiple interactions among prompt tokens, exploring their intrinsic associations to enhance knowledge representation. Finally, LAMP uses average pooling to reduce memory usage and training/inference time. Extensive experiments across six architectures and eight datasets demonstrate that LAMP outperforms state-of-the-art PT-based and LoRA-based methods in performance and efficiency.

## introduction pre-trained language models (plms) possess powerful learning capabilities to extract complex features and patterns from vast amounts of data @xcite . in recent * the first two authors contributed equally. parameter-efficient fine-tuning (peft) methods attempt to bridge this gap by achieving performance comparable to full fine-tuning with minimal computational resources and time costs @xcite @xcite . among these methods, prompt tuning (pt) stands out for its efficiency and flexibility. it freezes the model parameters and exclusively trains the soft prompt tokens attached to the model's input, delivering performance on par with full fine-tuning @xcite @xcite . notably, pt's trainable parameters are much lower than other peft methods (e.g., adapter @xcite and @xcite ) and do not grow exponentially as the scale of plms. although these methods provide undeniable contributions, existing pt-based methods still suffer from two main challenges: first, in various tasks, pt initialization methods exhibit high discreteness, lacking exploration of the intrinsic semantic associations between tokens. the two mainstream prompt initialization methods-random initialization and initialization from sampled vocabulary (i.e., the 5,000 most common tokens)-help the model explore a broader parameter space during training. the sample vocabulary initialization, in particular, is widely utilized in pre-training for transfer learning @xcite and multi-task learning @xcite due to its more informative nature. nevertheless, as shown in figure 1 , we found that although this informative initialization provides prompt tokens with rich knowledge, these tokens remain isolated without establishing semantic connections after training. in nlp tasks, intrinsic semantic associations form the context and meaning of language, helping models better understand and represent complex language structures @xcite @xcite . prompt tokens leverage semantic knowledge to guide plms in producing outputs that better meet task requirements. clearly, this intrinsic semantic association is essential for pt. this high discreteness overlooks capturing intrinsic semantic associations among soft prompt tokens, limiting the model's knowledge representation capabilities. second, although pt does not require training the parameters of plms, adding soft prompts increases the total length of input embeddings. figure 2 reveals the relationship between computational cost, trainable parameters, and prompt length. previous research has shown that a long prompt 100 yields optimal pt performance, enabling plms to adapt to complex downstream tasks @xcite @xcite . how-ever, such a length renders pt inefficient. this inefficiency stems from the inherent high complexity of plms (e.g., the quadratic complexity of transformers) @xcite and the fact that the storage of gradients and optimizer states is closely related to the number of trainable parameters @xcite . while some approaches @xcite @xcite attempt to optimize standard pt, they still struggle with inefficiency and suboptimal performance. to address the aforementioned challenges, we propose a novel efficient and effective low-parameters prompt tuning (lamp) method through prompt decomposition and compressed outer product. our motivation stems from the soft prompt exhibiting high dispersion and the "intrinsic rank" @xcite in peft, which indicates that model fine-tuning can occur in a low intrinsic dimensionality space. specifically, lamp first employs truncated singular value decomposition (svd) with its inherent structure-two low-dimensional singular vectors and singular values-to transform the loosely related semantic knowledge in pt tokens into a more structured and interrelated form, while simultaneously reducing trainable parameters. it then aggregates the semantic knowledge from the truncated svd and leverages the compressed outer product to enable multi-level interactions of intrinsic semantics, enhancing the model's knowledge representation. finally, lamp reduces computational load by applying average pooling, which does not increase training parameters. figure 2 demonstrates that the longer the prompt length, the more significant the reduction in computational cost and memory usage achieved by lamp. the main contributions of this paper are: ## experiments in this section, we will answer these key research questions by conduct extensive experiments: rq1: how does our lamp performance compare with other sota baselines across different model scales and datasets? rq2: how do few-shot adaptability and hyper-parameters optimization influence the lamp? rq3: how will the feature space of the soft prompt change after considering the intrinsic semantic associations between tokens? ## related work parameter-efficient fine-tuning. parameterefficient fine-tuning achieves strong results by training a small subset of parameters, thereby reducing computational costs and improving efficiency. adapterdrop @xcite improves efficiency by removing unimportant adapters for a given task in each layer of the transformer. bitfit 2024) leverages multi-space projection and prompt fusion to refine soft prompt knowledge, enhancing flexibility and balancing accuracy with computational efficiency for diverse tasks. nevertheless, these pt-based methods need more efficiency and task-specific knowledge richness when dealing with long soft prompt. lamp provides the ability to tailor prompts more precisely and effectively to the specific requirements of various tasks. ## conclusions in this work, we observed that soft prompt tokens initialized randomly from the vocabulary lack intrinsic semantic associations to enhance knowledge representation. additionally, pt-based methods face challenges balancing knowledge richness and computational cost in different tasks. based on these issues, we approximate soft prompt by proposing a low-parameter prompt tuning (lamp) method, which utilizes two singular vectors and singular values. lamp facilitates semantic knowledge interaction, allowing the soft prompt to incorporate more task-specific knowledge. it can serve as an efficient plugin for various pt-based tasks. experimental results across three model scales (t5-small, t5-base, t5-large) demonstrate that lamp achieves high effectiveness and robustness with fewer trainable parameters. superglue benchmark dataset #train #dev type domain #metric cb 250 56 natural language inference various accuracy wsc 259 104 common sense reasoning fiction books accuracy copa 400 100 question answering blogs, etc. accuracy rte 2,490 277 natural language inference news, wikipedia accuracy wic 5,428 638 word sense disambiguation lexical databases accuracy boolq 9,427 3,270 question answering wikipedia accuracy multicrc 27,243 4,848 question answering various f1 record 100,730 10,000 common sense reasoning news (cnn, daily mail) f1 mnli 392,702 19,647 nli various accuracy qnli 103,743 6,463 nli wikipedia accuracy sst-2 66,349 1872 sentiment movie reviews accuracy mrpc 3,668 408 paraphrase news accuracy descriptions of all lora-based baselines are as follows: