Generative Prompt Internalization (2025)

Abstract: Prompts used in recent large language model based applications are often fixed and lengthy, leading to significant computational overhead. To address this challenge, we propose Generative Prompt Internalization (GenPI), a lightweight method that employs a joint training approach. GenPI not only replicates the behavior of models with prompt inputs but also generates the content of the prompt along with reasons for why the model’s behavior should change accordingly. We demonstrate that our approach effectively internalizes complex prompts across various agent-based application scenarios. For effective training without interactions with the dedicated environments, we introduce a data synthesis technique that autonomously collects conversational datasets by swapping the roles of the agent and environment. This method is especially useful in scenarios where only a predefined prompt is available without a corresponding training dataset. By internalizing complex prompts, Generative Prompt Internalization enables high performance and efficient inference without the need for explicit prompts.

## introduction in the real world, for inference in large language model (llm) based applications (e.g. chatgpt), fixed and complex prompts are often used repeatedly. although advanced prompts can improve performance, their dependence on lengthy prompts raises concerns regarding computational efficiency for service providers and cost-effectiveness for users. longer prompts can considerably increase computational overhead, especially during multiturn inference, making it a potential bottleneck in practical applications. therefore, finding effective strategies to optimize these prompts while maintaining performance is crucial for the practical deployment of llm-based applications. to address this issue, existing methods can be categorized into two approaches. one practical approach only for efficient prompt computation is compressing prompts into external embeddings @xcite @xcite or compressed text @xcite @xcite . however, these methods still require additional tokens, limiting their ability to fully internalize prompts and effectively modify the model's behavior as intended. alternatively, fine-tuning @xcite or distillation @xcite @xcite , have been explored as canonical approaches for internalizing prompts. these methods adjust the language model's behavior to follow the prompt's intentions without requiring a prompt during inference. however, these internalization methods are limited in that the model cannot reference the prompt's content during training. instead, they rely on indirect training based on the model's output @xcite or distribution @xcite @xcite of the original model when the prompt is provided. this leads to significant performance degradation in information-rich tasks or specific requirements, such as agent's action requirements or schema grounding @xcite . to overcome the limitations, we propose generative prompt internalization (genpi), a method that is trained to generate the target prompt, rather than merely using it as input. genpi employs joint loss training, combining two approaches: 1) mimicking the teacher's output to guide behavior, similar to distillation approaches @xcite @xcite , and 2) generating the content of the prompt while inferring why the output should change based on that prompt. genpi employs a lightweight internalization process for each prompt, requiring only 0.5% additional parameters on 1,000 samples. focusing on scenario where only prompts are available without a corresponding training dataset for internalization @xcite , we also introduce a data synthesis technique to generate a multi-turn pseudo conversational dataset. by simply swapping the roles of user and assistant within the given context, the model simulates both sides of conversation, enabling it to autonomously collect conversational datasets. we evaluate our method in agent application scenarios with lengthy prompts using agent-bench @xcite . genpi maintains strong performance even without the need for prompt input, while outperforming other distillation or compression-based baselines. genpi achieves 100% performance retention on os interaction agent task, while maintaining at least 82% performance on web-based agent tasks with over 1,000 tokens. furthermore, genpi demonstrates 39% efficiency improvement in handling environments with lengthy prompts, outperforming other compressionbased methods. prompt compression is one of the practical approach to reducing the computational overhead caused by lengthy prompts. in case of the apibased large language models (llms) services, systematic caching solutions allow frequently used prompts to be stored between api calls. 23 for users of api-based llms, text-based prompt compression methods are proposed, where key segments of long prompts are selected on a token-by-token basis to generate a compressed version of the original prompt @xcite @xcite . although this approach reduces prompt length, it often still results in relatively long token sequences, as essential tokens must be retained. on the other hand, embedding-based prompt compression methods generate cached token embeddings for prompts, which can be utilized as a more efficient representation in llms @xcite @xcite . these methods offer the advantage of using fewer token embeddings compared to text-based compression but often require modifications to the model architecture, making it challenging to leverage the compressed vectors across different models. ## problem definition following choi et al. (2023), we assume a scenario where an application-specific prompt p is predetermined. our goal is to guide the model to behave as if the prompt is given, even in its absence. similar to the previous distillation approaches @xcite @xcite , we define the teacher model t and the student model s based on whether the prompt p is provided to the same model θ. the teacher model t is defined as a function that takes the prompt p and the input x i at turn i, generating the teacher's output: y t @xmath0, p) = f t (x i , p; θ). the student model s is a function that takes the input x i at turn i, generating the student's output: we approximate the student model's behavior to match that of the teacher model, conditioned on the prompt p over multiple turns, as shown in equation 1 , ## generative prompt internalization in this section, we demonstrate our novel method, generative prompt internalization (genpi), which internalizes the given prompt by generating its own contents. basically, genpi involves training the student model to mimic the behavior of the teacher model given the prompt p. we finetune the student model on the multi-turn outputs generated by the teacher model (y t i , @xmath1, 1, ..., n }), inspired by sequence-level knowledge distillation approaches @xcite which follow the hard label distribution from the teacher model. however, the model is still unable to learn the content of the prompt directly; it only learns indirectly through the teacher's output. we introduce an additional loss function, prompt generation loss (pg), where the loss is calculated directly on the prompt. this loss involves training the student model to understand why its behavior should align with the teacher's behavior based on the prompt content. as illustrated in figure 1 , for a given input x, the model is trained to generate a prompt p along with a reason r for why the output should be changed. in this process, the student's output y s i is considered the initial state ("as-is"), while the teacher's output y t i represents the desired state ("to-be"). this process is formalized in equation 3 : we utilize a hyperparameter λ to combine the losses into a joint loss function, resulting in the final joint loss function: our goal is to enable lightweight training and inference for each prompt, while also adapting to changes in the prompt effectively. to achieve this, we employ qlora @xcite to learn prompt-specific adaptors, allowing us to tailor our approach to each individual prompt. ## components for prompt internalization following choi et al. (2023), we assume a realistic scenario where the predetermined prompt has never been encountered before and is not included in any training dataset for prompt internalization. consequently, we first generate the components as a pseudo training dataset from the prompt p. pseudo user input. similar to typical query generators @xcite @xcite , we simply utilize a large language model to generate pseudo user input x by prompting to generate a probably questionable user input from the given prompt p. in this paper, we generate only 1,000 pseudo user inputs. implementation details and examples are provided in appendix a.1 and appendix a.2. pseudo conversational outputs. in many taskspecific applications, such as agent applications, the model interacts with the real environment across multiple turns @xcite @xcite . this means that collecting multi-turn conversation outputs requires interacting with the actual environment at every turn, which is a non-trivial problem. as our primary objective is to gather the teacher model's behavioral patterns, rather than to optimize the task performance, we introduce a simple method: self role-playing conversation. this method simply involves reversing the role between agent and environment. as illustrated in figure 2 , we provide the model θ with a role-reversed context to simulate the task environment. by swapping the agent and environment roles in the prompt, a single model θ can effectively embody both environment and agent personas, enabling the collection of self-conversational outputs y. we report the examples and additional quality evaluation in appendix a.3. implementation details and examples are provided in appendix a.1 and appendix a.2. reason. to collect the supervision for the reason r, we utilize a large language model by prompting to generate from the prompt p, user input x, and outputs y s , y t . as illustrated in figure 1 , the reason explains why the student's output ("as-is") should be changed to teacher's output ("to-be"). implementation details and examples are provided in appendix a.1 and appendix a.2. ## efficiency to compare the efficiency between generative prompt internalization and other baselines that require fixed prompts, we sample an example with more than 5 turns from a web shopping task containing over 1200 tokens. 5 we then compare the macs (multiply-accumulate operations), flops (floating point operations), and latency to llama-3-8b-instruct (dubey et al., 2024) based baselines. all metrics were measured by deep-speed profile foot_3 . as shown in figure 3 , the overhead for baselines that require prompts accumulates and increases with each turn. while llmlingua-2 (pan et al., 2024) exhibits slightly better performance due to the reduced number of prompt tokens, it still incurs a larger overhead compared to ours. furthermore, when considering compression overhead, the llmlingua-2 (pan et al., 2024) which is based on xlm-roberta-large (conneau et al., 2020), shows relatively low macs and flops than inferring with the prompt. however, it requires high compression latency proportional to the prompt length, resulting in extra latency in the first turn. to simulate a more realistic scenario used in the real-world chat application, we conduct an additional experiment inspired by recent kv caching (pope et al., 2023) works across multi-turn dialogues @xcite @xcite . we assume that the previous turn's attention values are pre-cached to reduce the latency. the prompt is also considered as a previous turn's conversation and is pre-cached. in this setting, the overhead between each turns is independent, and we report the conversation-level total overhead over 5 turns. in figure 4 , the prompt caching time executed before the first turn incurs higher computational overhead due to the use of a larger model compared to the compression model. from the second turn onward, although caching previous conversation histories eliminates redundant computations, the computational overhead still increases proportionally to the length of the prompt. even without considering in caching time, genpi demonstrates a 39% improvement in macs and flops, along with a 17% improvement in latency. consequently, despite caching, baselines inevitably require more computational overhead compared to generative prompt internalization. ## discussion in this section, we discuss the strengths and limitations of two prompt internalization strategies: generalized prompt-agnostic compression and promptspecific internalization approach. prompt compression approaches, such as llmlingua-2 (pan et al., 2024) and icae @xcite , effectively compress short instruc-tions without requiring additional training for new prompts @xcite @xcite . however, as shown in prompt internalization approach, such as genpi, overcome this limitation by explicitly internalizing prompts through targeted training. while genpi requires training for each new prompt, this overhead is minimal in practical applications where only a limited set of prompts is needed. once trained, switching between adapters incurs negligible computational cost, making the approach both flexible and scalable. each adapter requires only 1,000 data samples for training. even when generating the dataset with a 70b llm, entire process can be completed in under an hour using four a6000 gpus with vllm @xcite , while training takes approximately three hours on a single a6000 gpu. ## conclusion in this paper, we propose a novel prompt internalization method, generative prompt internalization (genpi), which generates both the contents of the prompt and the reasoning behind changes to the model's output while mimicking the behavior of the teacher model. to address the challenging scenario where only predetermined prompts are available without additional training data, we introduce self role-playing conversation, a method that generates a pseudo-conversational training dataset from the given prompt. our approach demonstrates that even in scenarios with lengthy agent prompts, genpi maintains high performance without relying on the prompt itself. moreover, genpi improves efficiency by fully internalizing the prompt without requiring any additional tokens.