SM o P : Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts (2023)

Abstract: Prompt tuning has emerged as a successful parameter-efficient alternative to the full fine-tuning of language models. However, prior works on prompt tuning often utilize long soft prompts of up to 100 tokens to improve performance, overlooking the inefficiency associated with extended inputs. In this paper, we propose a novel prompt tuning method SMoP ( S parse M ixture- o f- P rompts) that utilizes short soft prompts for efficient training and inference while maintaining performance gains typically induced from longer soft prompts. To achieve this, SMoP employs a gating mechanism to train multiple short soft prompts specialized in handling different subsets of the data, providing an alternative to relying on a single long soft prompt to cover the entire data. Experimental results demonstrate that SMoP outperforms baseline methods while reducing training and inference costs. We release our code at https://github.com/jyjohnchoi/SMoP.

## introduction prompt tuning @xcite has recently gained attention as a parameterefficient alternative to the full fine-tuning of language models. by freezing the original language model parameters and solely tuning the soft prompts (i.e., learnable token embeddings) added to the model input, prompt tuning achieves comparable performance to full fine-tuning while largely reducing the number of trainable parameters. moreover, prompt tuning stands out for its conceptual simplicity and flexibility among other parameterefficient fine-tuning methods @xcite @xcite , as it does not require modifications to the model structure. since the proposal of prompt tuning, there has been active research to enhance its efficiency and effectiveness. on one hand, several approaches propose to improve the performance of prompt tuning by integrating soft prompts into activations in each for prompt tuning @xcite , increasing soft prompt length improves accuracy, but also results in a significant increase in memory usage. smop outperforms prompt tuning while preserving memory usage by sparsely activating short (length 5) prompts. to this end, we propose smop (sparse mixtureof-prompts), a novel prompt tuning method that utilizes short soft prompts during training and in- ## conclusion we have presented smop (sparse mixture-of-prompts), a novel prompt tuning method that utilizes short soft prompts for efficient training and inference while maintaining performance gains associated with increased prompt length. to achieve this, we have employed a gating mechanism in smop that routes each instance to one of the multiple short soft prompts. experimental results have demonstrated that smop has outperformed prompt tuning while reducing training and inference costs through the utilization of short soft prompts.