P rompt I ntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuning (2024)

Abstract: Recent advances in fine-tuning large language models (LLMs) have greatly enhanced their usage in domain-specific tasks. Despite the success, fine-tuning continues to rely on repeated and lengthy prompts, which escalate computational expenses, require more resources, and lead to slower inference. In this paper, we present a novel approach, PromptIntern, which internalizes prompt knowledge during model fine-tuning to achieve efficient inference and save costs. Instead of compressing the prompts for a vanilla model, PromptIntern aims to embed the recurrent prompt directly into the model parameters. We design a fine-tuning pipeline that includes instruction template compression, few-shot example absorption, and a progressive internalization strategy, effectively diminishing the need for intricate prompts during inference. Comprehensive experiments on challenging NL2Code tasks demonstrate that our method reduces input tokens by more than 90%, accelerates inference by 4.2 times, and reduces monetary inference costs by 88.3%.

## introduction large language models (llms) have become pivotal in numerous natural language processing (nlp) applications, such as natural language generation @xcite , reasoning @xcite , and code generation @xcite @xcite . to enhance the predictive accuracy of llms in domain-specific tasks, recent techniques in fine-tuning, such as parameter-efficient fine-tuning (peft) @xcite @xcite , have been developed for pretrained models to excel in specific tasks by adjusting their parameters to better align with targeted † the contributions by jiaru zou and tao li have been conducted and completed during their internships at microsoft. datasets @xcite . many of these finetuning approaches typically adopt prompts that are optimized and integrated with detailed instructions, examples, and retrieved documents through prompt engineering techniques such as chain-of-thought @xcite , few-shot prompting @xcite , and retrieval-augmented generation @xcite . although these advancements enhance the capabilities of llms during fine-tuning, they also present new challenges: prompt engineering often necessitates longer prompts, and directly integrating lengthy prompts into the training process further increases computational costs during inference @xcite . this increase in cost precludes llms in many cost-sensitive scenarios where computational resources are constrained. several prompt compression methods @xcite @xcite have been proposed to reduce text redundancy. they design various prompt compression systems and strive to preserve maximum information between original and compressed prompts. while these methods ensure the retention of original prompt information, they mainly focus on the prompt perplexity and overlook the adaption of target llms during compression @xcite . for challenging tasks that require model fine-tuning @xcite , these approaches struggle to establish connections between compressed tokens and dynamically adjusted model parameters. consequently, naively applying these compression methods often leads to large performance degradation, as relevant information may be inadvertently removed or distorted during the compression process. in this paper, we propose a novel prompt internalization approach, namely promptintern, which internalizes prompt input during model fine-tuning and enables efficient inference. unlike prompt compression which removes tokens based solely on prompt information entropy, we aim to transfer various types of prompt knowledge into updated model parameters, thereby directly enhancing llms' understanding. our idea is motivated by the human learning process as illustrated in figure 1 : during internship on-boarding, human interns need detailed instructions, examples, and documents to learn new tasks. as they internalize these materials and become familiar with their duties, they master the necessary skills and no longer require extra guidance. similarly, when specific prompt information (e.g. task constraints, output formats/styles) is repeatedly exposed to an llm during fine-tuning, the model can gradually internalize the knowledge into its updated parameters. such repeated information can be progressively eliminated from prompt inputs since it becomes unnecessary for inference of the master llm. we dub our approach promptintern to regard llms as human interns and internalize prompt knowledge progressively. our approach consists of several key steps: initially, we classify an input prompt into three components: the template, examples, and query. we start by setting a schedule to linearly decrease both the template compression rate and the number of few-shot examples across training stages. following the schedule, we implement template compression and example absorption to pre-process the input prompts. we then introduce a comprehensive pipeline that enables llms to progressively internalize template and ex-ample components into model parameters during fine-tuning and efficiently perform inference using query-only prompts. we assess our method on challenging nl2code tasks @xcite that are widely recognized as benchmarks for model fine-tuning. our experiments evaluate promptintern on three key metrics: accuracy, token usage, and inference speed. the results indicate that under identical fine-tuning settings, our method not only surpasses prompt compression methods but also achieves comparable accuracy to direct fine-tuning. moreover, it accelerates the inference process by a factor of 4.2 and reduces token usage by over 90% compared to direct fine-tuning. these enhancements demonstrate that our approach successfully balances efficiency and effectiveness, making it well-suited for optimizing llm performance across various cost-saving scenarios. we further quantify the total monetary cost savings and conduct detailed analyses to elucidate the efficacy of our approach and provide insights into its underlying mechanisms. our main contributions can be summarized as follows: the experimental results show that our approach reduces token usage by over 90%, speeds up inference time by 4.2 times, and achieves 88.3% cost savings across a broad spectrum of llms. template (instructions & doc) 10-shot examples query 0.3x template 5-shot examples 0-shot no template template compression example absorption scheduled epochs template compression example absorption @xmath0% query query progressive fine-tuning progressive fine-tuning preprocess scheduled epochs scheduled epochs progressive fine-tuning inference / !23 figure 2 : overview of promptintern framework. we structure the input prompt into three components: the template, examples, and query. by employing template compression and example absorption, we efficiently preprocess each component based on schedule s tmp , s egs . we then use a progressive fine-tuning strategy to gradually incorporate prompt knowledge into the model parameters θ, facilitating efficient inference without sacrificing performance. @xcite use learnable tokens to condense prompts. conversely, the task-agnostic methods utilize metrics such as information entropy to eliminate redundant prompt information, with systems like llmlingua @xcite estimating token importance using a smaller model. despite the demonstrated effectiveness of these methods, producing compressed text that can generalize across different tasks and be effectively integrated into training scenarios remains a challenge. model fine-tuning adopts pre-trained llms to specific tasks by modifying parameters. based on the assumption that fine-tuning adds less new information to the model pre-trained on large internet-scale datasets, parameter-efficient fine-tuning (peft) methods aim to curtail the costs of tuning large models by adjusting a subset of parameters. existing peft methods can be broadly categorized into three main approaches: 1) adapter-based methods @xcite : introduce trainable modules within a static "backbone" network, offering flexibility but potentially increasing model size. ## problem formulation let an input prompt as @xmath1, x egs , x que ), where each input prompt x is considered as a tuple of three components: x tmp as the template such as fixed instructions, api docs, etc., x egs as the examples, and x que as the query. typically, x tmp and x egs are relatively fixed and lengthy but essential for complex tasks. let f θ (•) denote the neural network function of a llm model, typically transformer @xcite , parameterized by θ. the generated output by llm can be represented as f θ (x). we then consider the following problem of prompt internalization. given a training dataset d @xmath2, y i )} n @xmath3, x i is an input prompt defined above, and y i is the corresponding groundtruth output. our goal is to internalize the knowledge contained in templates and examples of each input prompt i.e. {(x tmp i , x egs i )} n @xmath4, enabling efficient inference while maintaining high prediction performance through {x que i } n @xmath5. formally, the prompt internalization objective can be formulated as follows: ## methodology in this section, we introduce our method promptintern in detail. we first present the template compression to compress the entire fixed template part inside a prompt. then we show the example absorption to effectively absorb demonstration examples into model parameters. finally, we introduce a tailored training strategy for promptintern. the overall framework is shown in figure 2 . ## experiment in this section, we evaluate the performance of promptintern across various benchmarks on the nl2code task. the nl2code task is widely recognized for its utility in evaluating llms on both finetuning efficacy and cost-effectiveness in real-world applications @xcite . following this, our experiments primarily focus on two key perspectives: 1) effectiveness: assessing the performance accuracy of promptintern during inference phases; 2) efficiency: quantifying the reduction in token usage and corresponding cost savings achievable through promptintern. ## conclusion in this paper, we propose promptintern, a prompt internalization method that internalizes repetitive prompt knowledge into llms parameters. we develop specific compression strategies for different components of the prompt, accompanied by a tailored progressive fine-tuning pipeline. extensive experiments demonstrate that our method maintains comparable performance effectiveness while accelerating inference speed with less token usage.