Fixed Input Parameterization for Efficient Prompting (2023)

Abstract: Recent works have shown that attaching prompts to the input is effective at conditioning Language Models (LM) to perform specific tasks. However, prompts are always included in the input text during inference, even when they are fixed, thus incurring substantial computational and memory overhead. Also, there is currently no straightforward method of utilizing prompts that are longer than the maximum input length of the LMs without incurring additional costs during inference. We formally define Fixed Input Parameterization (FIP) problem that focuses on injecting the fixed prompt into the parameters of an LM to be an efficient alternative to attaching fixed prompts to the input. We show that in scenarios with long fixed prompts, FIP can be up to 280 times more efficient in terms of total FLOPs than previous approaches. We further explore methodologies for FIP and show promising results in persona-dependent conversation, semantic parsing, and zero-shot learning with task instructions. Through these explorations, we show that FIP can be a promising direction for conditioning language models, in scenarios with long and fixed prompts.

## introduction contemporary works on language models (lms) @xcite @xcite have shown that attaching prompts to the input is effective at conditioning lms to perform specific tasks. note that the prompt in this work refers to a broader aspect of prompts which includes both the prompts used to induce specific behavior as well as prompts used to provide some contextual knowledge such as persona for dialogue agents. lms are trained to condition on the given prompts in hopes of generalizing to unseen prompts during inference. unseen prompts can be a persona for persona-dependent conversation @xcite , database schema for semantic parsing @xcite , and task instruction for zero-shot learning with task instructions @xcite . in these tasks, a new prompt is fixed to the input at every inference. for instance, in persona-dependent conversation, a persona description is appended to the dialogue history, so that the lm can always be conditioned on the persona. for another example, in semantic parsing, the lm is conditioned on the database schema as well as natural language questions to generalize to a new database. lastly, zero-shot learning with task instructions involves adding natural language instructions to the inputs for adapting lms to novel tasks. however, concatenating prompts to input sequences for prompt-dependent inference has two major limitations. (1) during inference, prompts are always included in the input text and thus incur computational and memory overhead @xcite . (2) it is challenging to fit a long text such as the detailed description of a persona as a prompt into transformer-based models whose input lengths are often fixed @xcite . for instance, in persona-dependent conversation, the model constantly refers to the persona description along with the dialogue history @xcite , as shown in the left side of figure 1. moreover, in real-world scenarios, a persona may consist of a long detailed text description of a character or person, not just a few profile sentences. naively concatenating long prompts to the input sequences is challenging due to the quadratic cost in time and memory of transformer-based architectures with regard to the input sequence length. other approaches specialized for processing long inputs @xcite @xcite , or those that augment the lm with a retrieval mechanism @xcite al., 🧑 hello! how are you today? great, thanks ! my children and i were just about to watch game of thrones. i am an artist. i have four children. i recently got a cat. i enjoy walking for exercise. i love watching game of thrones. hello! how are you today? 🤖 lm with explicit prompt (prev) i am an artist. i have four children. i recently got a cat. i enjoy walking for exercise. i love watcing game of thrones. 🧑 hello! how are you today? great, thanks ! my children and i were just about to watch game of thrones. hello! how are you today? lm with prompt injection (ours) agent's persona 💉 inject persona in advance concat persona to input agent's persona 2022) may be used but still come with increased overall memory and computations, ultimately leading to a delay in generating responses. this problem becomes critical in situations where the lms are deployed, and fast inference speed is required. in this work, we formally define fixed input prarameterization (fip) problem, where we focus on injecting a given fixed prompt into the parameters of an lm to address the two limitations mentioned above. with fip, lms can produce promptdependent outputs without the computational overhead of appending fixed prompts at inference time (the right side of figure 1 ), and it also enables the injection of longer prompts in a wholistic way. more specifically, we first show that fixed input prarameterization (fip) is much more efficient (up to 280 times) in terms of total flops compared to previous approaches that may be used for handling long prompts such as fusionin-decoder @xcite or linear transformer @xcite . next, we explore different methodologies as baselines for fip, including the continued pre-training approach on the prompt as well as a novel distillation approach called pseudo-input generation (ping) for successful fip. we apply these fip methods to three different tasks with fixed prompts: persona-dependent conversation, semantic parsing, and zero-shot learning with instructions. we compare the methods against lms with explicit prompts as the upper bound as well as the lm without both the prompt and fip as the lower bound. experimental results show meaningful improve-ments with respect to the lower bound, but also exhibit a non-trivial gap with the upper bound. despite the performance and efficiency trade-off, we still believe that fip is a direction worth exploring considering its computational benefit, especially when inference costs are critical in real-world applications. in sum, our main contributions are three folds: with the help of appropriate prompts, one can exploit knowledge learned by a pre-trained lm and manipulate the lm's behavior. however, for the in-context learning scenario, processing prompts that involve many training examples for each inference incurs substantial computational and memory overhead @xcite . given training data, @xcite replace in-context learning with fine-tuning a small set of parameters for tackling the above issue. we tackle the same issue but assume a stricter scenario where there is no training data for the given prompt. efficient transformers one can consider using efficient transformer-based @xcite architectures for handling long input sequences @xcite . the main challenge of using a vanilla transformer architecture is the quadratic cost in time and memory with regard to the input sequence length due to the self-attention operation. there has been a surge of recent works addressing this problem @xcite @xcite @xcite . they are primarily dedicated to improving either the efficiency of the selfattention mechanism or the general efficiency of the transformer architecture through sparse models. also, there has been an attempt to distill a unique prompt to handle long inputs @xcite . our fixed input prarameterization (fip) approach tackles the efficiency problem of performing prompt-dependent tasks by keeping the input sequences short (without prompts), bounding the time and memory complexity to a constant invariant of the length of the prompt. in contrast to @xcite , our work focuses on formally framing the problem into a more general and realistic setting since we aim to inject new prompts with no corresponding training data instead of only one prompt with corresponding training data. ## fixed input prarameterization in this section, we formally define fixed input prarameterization (fip) as a task and describe the benefits of the formulation. prompt-dependent generation is a task of generating an output sequence y that is a proper response to the input sequence x and coherent to the prompt z. utilizing the prompt during inference, the generated sentence is obtained by @xmath0, x) where f denotes an lm such as t5 and gpt-2. fixed input prarameterization (fip), i.e., parameterization of prompts, allows lms to perform prompt-dependent generation without using prompts during inference. to achieve this, we need to design a fip method h to inject a prompt z into an lm f . the process of fip can be represented as the above formulation can be seen as a high-level abstraction of iterative fip that we aim to approximate. in practice, in order to fully inject z 1:n , we repeat ( 2 )-( 4 ) multiple times (i.e., multiple epochs). why is fixed input prarameterization necessary? fip brings advantages in terms of efficiency when applied to prompt-dependent tasks. the previous approach of appending prompts to the input sequences has the drawback of the model repeatedly referring to the prompt at each inference time. this becomes critical in scenarios requiring long prompts, as transformer architecture has quadratic computational and memory costs due to the limitation of the self-attention operation. we propose fip as a solution to this computation bottleneck. once a prompt is injected into the lm in advance, the lm no longer needs to refer to the prompt during inference. as a result, the model's input length remains independent of the length of prompts and is able to utilize prompts of any length efficiently. we discuss the efficiency gain of fip in section 6.1. evaluation metric for fip fip can be evaluated by the evaluation metric of the fixed promptdependent task at hand. we also introduce a metric called the fip score (fip score) to measure the degree of injection. the metric is agnostic of the target task by comparing the results with that of an lm given actual prompts during inference. let x w/ prompt denote the lm's task score with the prompt as an additional input (upper bound) and x w/o prompt denote the lm's task score without the prompt (lower bound). we define fip score as the min-max scaling score of x f ip , where x f ip represents the score of the lm on the target task after fip, i.e., fip @xmath1, x f ip -x w/o prompt ) / (x w/ prompt -x w/o prompt ). we limit using fip only in situations where x w/ @xmath2. even if the range of individual task scores may vary from task to task, fip score represents the overall injection effectiveness of the fip methods, agnostic of the individual task score range. ## methods for fixed input prarameterization in this section, we explore methods of fixed input prarameterization (fip) that can address promptdependent tasks without accessing the prompt during inference. to achieve this, the model should be trained to store the prompt in its parameters. this can be seen as parameterizing the prompt into the model instead of feeding the prompt explicitly to the model. this is challenging as the prompt is unseen to the model and has no corresponding training data. in section 4.1, a baseline method by continued pre-training is introduced, followed by a method for improving the baseline with curriculum learning. section 4.2 presents a novel distillation-based method called pseudo-input generation (ping) that learns to generate pseudo-inputs to inject novel prompts. ## experimental setup in this section, we explain the experimental setups in detail. experiments are performed with the t5base @xcite (220m parameters) model unless noted otherwise. ## experimental results in this section, we first explore the inference efficiency of models performing prompt-dependent tasks and show that fixed input prarameterization (fip) leads to a meaningful gain in computational efficiency. then the baseline and proposed methods are tested and compared on datasets discussed in section 5.2. the results indicate that the pseudo-input generation (ping) method achieves the best performance among fip methods, sometimes even outperforming the upper bound, which uses explicit prompts during inference. in section 6.3, we provide a concrete instance of injecting a real persona description into a conversational model, demonstrating the feasibility of long prompt injection. ## conclusion in this paper, we formally define fixed input prarameterization (fip) problem that focuses on injecting the prompt into the parameters of an lm, as an efficient alternative to attaching fixed prompts to the inputs for prompt-dependent tasks. through experiments, we show that fip is much more computationally efficient (up to 280 times) in terms of total flops for handling long prompts compared to the previous alternatives. we further explore baseline methodologies for fip and find that pseudo-input generation (ping), a distillation-based approach, shows promising results in personadependent conversation, semantic parsing, and zero-shot learning with task instructions. through the explorations, we show that fip can be a promising direction for conditioning language models efficiently, in scenarios with long and fixed prompts. limitations while fixed input prarameterization (fip) enables performing prompt-dependent tasks efficiently, there are limitations that need to be addressed in future work. in particular, the current fip methods cause task performance degradation. moreover, the computational cost needed for the injection of prompts and the storage required to store the parameters of every injected model have not been extensively considered. for example, when considering previous conversation history as the prompt to be injected in a long-term conversation setting, fast injection may also be a requirement for real-world application. updating or adding a relatively small number of parameters @xcite ) may be a potential avenue for addressing the problems.