SPC : Soft Prompt Construction for Cross Domain Generalization (2023)

Abstract: Recent advances in prompt tuning have proven effective as a new language modeling paradigm for various natural language understanding tasks. However, it is challenging to adapt the soft prompt embeddings to different domains or generalize to low-data settings when learning soft prompts itself is unstable, task-specific, and bias-prone. This paper proposes a principled learning framework—soft prompt construction (SPC)—to facilitate learning domain-adaptable soft prompts. Derived from the SPC framework is a simple loss that can plug into various models and tuning approaches to improve their cross-domain performance. We show SPC can improve upon SOTA for contextual query rewriting, summarization, and paraphrase detection by up to 5%, 19%, and 16%, respectively.

## introduction prompting has emerged as a new paradigm for tuning language models (lms), drawing increasing research attention. it shows impressive utility in stimulating an lm' hidden knowledge without finetuning it @xcite . simply adding some descriptive words to the input can steer a generalpurpose lm to a specific task in a zero-shot fashion. albeit the impressive utility of these prompting techniques, they have some limitations that encourage further research. one major drawback is that the prompt templates significantly impact model performance but require handcrafting, which is why there is increasing prompt engineering and sharing of "good" templates @xcite . chosen unwisely, a prompt template would not work or even hurt the model inference @xcite . further, one has to design prompts for each task, making it inefficient and less robust to transfer across models or tasks @xcite . the other drawback of prompting is that it works best for large models @xcite . however, training and serving large lms are known to be costly. these limitations drive us to study new prompting methods that require little manual effort and work for large and small models. soft prompting is one of these approaches. unlike prompting that adds words to the input, soft prompting adds embeddings to the model, such as injecting embeddings to the token embedding layer or other model layers @xcite . such embeddings, or soft prompts, can be automatically learned. many studies have shown that soft prompting works well for smaller-sized models and can be as effective as fine-tuning large lms with orders of magnitude fewer trainable parameters @xcite @xcite . soft prompting is a promising approach to further improve cross-domain generalization and task adaptation for all-sized models. however, some barriers need to be removed. first, learning soft prompts lacks guidance, and traditional learning objectives (such as cross-entropy) often lead to unstable training, sub-optimality, and unstable performance. further, the unstable performance is amplified in cross-domain settings. soft prompts are often biased by a specific domain and hence less effective for other domains. in low-data settings, it is particularly difficult to learn soft prompts with consistent cross-domain performance @xcite @xcite . our objective is to derive a principled paradigm for learning domain-adaptable soft prompts, for which we propose the soft prompt construction (spc) framework. using this spc framework, we further derive a simple loss: the spc loss. the loss supervises the optimization of soft prompts without re-tuning the entire model. the learned soft prompts demonstrate superior cross-domain generalization than other fine-tuning and prompt tuning approaches. our contributions are the following: ## related work prompting adds token-level templates to the model input, often designed manually @xcite . autoprompt automates prompt design via gradient-based search @xcite . other approaches like @xcite may use lms to generate the prompts. two prompting techniques are particularly useful for large lms (llms): in-context learning (icl) and chain-of-thought (cot). icl uses a few examples to demonstrate the task, and the llm can perform the task in a few-shot manner @xcite . cot instructs an llm to reason step-by-step to perform a task @xcite . cot can be seen as a special case of icl as it often provides examples to demonstrate step-by-step reasoning, such as react @xcite , self-consistency @xcite . token-based prompts' performance is highly dependent on prompt selection and constraint on natural language words. on the other hand, soft prompt tuning uses embedding-level prompts and is optimized by gradients. the prompt embeddings do not necessarily represent natural language words and can generalize to different domains. dart @xcite treats input token embeddings as prompt templates. @xcite optimize soft prompts via perturbing a prompt mixture. prefixtuning (li and liang, 2021) prepends prompt embeddings to the model layers and uses additional networks to re-parameterize them. p-tuning v2 @xcite extends prefixtuning and achieves performance comparable to fine-tuning. such works often use deep prompts (adding prompt embeddings to all model layers) or reparameterization, introducing a large number of new parameters @xcite . @xcite add task-specific soft prompts to the model input and obtain performance comparable to finetuning large lms, especially on low-resource domain adaptation tasks. however, soft prompts' performance is unstable and sensitive to initialization @xcite . learning soft prompts across tasks is challenging, especially in low-data scenarios @xcite . to this end, @xcite construct soft prompts via knowledge injection and implicit relation constraint. @xcite pretrain soft prompts on varied tasks to improve generalization but require prompts to be added during pre-training. @xcite study the generalization of soft prompts via (nonlinear) subspace decomposition. however, @xcite find that the prompt dis- encoder decoder who directed the movie @xmath0: model diagrams for (a) fine-tuning, (b) prefixtuning, (c) naive prompt tuning, and (d) spc + prompt tuning. spc loss can plug into any tuning methods and language models. spc prompt embeddings are shown as prefixes, but they can be placed anywhere. tance cannot be appropriately quantified by euclidean or cosine distances, implying a nonlinear and (potentially) non-metrizable prompt embedding space. hence, constructing the prompt embedding space via decomposition is challenging. 3 soft prompt construction using the examples in consider the data on cross-domain tasks @xmath1. for text generation tasks, for example, each element in x or y represents a sequence of tokens. given the generality of the tasks and data structures, we omit the dimensionality in the notations when there is no confusion. the objective of a language model is to maximize the expected log-likelihood to maximize (2), we need to find a maximizer of the lower-bound (4). taking the partial derivative of (4) w.r.t. p and setting it to zero, one of the feasible maximizers p * is obtained by ( 5 ) when using soft prompts [p, x] f θ → y ⇒ [e p , e x ] f θ → e y where e x , e y , e p are embeddings of x, y, and p, relation (5) naturally extends to the embedding representations e * @xmath2 for those interested, we also provide an algebraic view for our methodology in appendix b. implementation details a backbone language model is used and its parameters are frozen. the prompt embeddings e p have fixed length and are concatenated to the input token embeddings e x , effectively increasing the input length. prompt length and concatenation are explained in section 4. output embeddings e y are taken from the last output layer. during training, the spc loss (7) is directly added to the original loss (e.g., cross-entropy loss). the prompt embeddings and the linear weights of the spc loss are learned. at inference time, the linear weights are discarded, and only the learned prompt embeddings are concatenated to the input token embeddings. only one soft prompt is learned, and it is universal for all domains and tasks. an illustration is shown in figure 1 . code 1 shows the code for computing the spc loss. ## experiments we design experiments to validate the main claims of our spc framework: (1) spc loss can be plugged into any model or tuning method, and (2) improve the zero/few-shot in/cross-domain generalization. we will explain our selected tasks, data, model, implementation, and results. ## conclusions this paper proposes a soft prompt construction (spc) framework to learn domain-adaptable soft prompts. the derived spc loss is model/taskagnostic and can plug into other models and tuning methods to improve their cross-domain performance. experiments on three tasks (contextual query rewriting, abstractive summarization, paraphrase detection) demonstrate that spc improves significantly over strong sota: prefixtuning (li and liang, 2021), prompt tuning @xcite , and fine-tuning in zero/few-shot crossdomain/task settings across different lms (bart, t5 family). various properties of spc are also analyzed. this study paves the way for loss-guided domain-adaptable prompt learning.