Prompt Compression for Large Language Models: A Survey (2025)

Abstract: Leveraging large language models (LLMs) for complex natural language tasks typically requires long-form prompts to convey detailed requirements and information, which results in increased memory usage and inference costs. To mitigate these challenges, multiple efficient methods have been proposed, with prompt compression gaining significant research interest. This survey provides an overview of prompt compression techniques, categorized into hard prompt methods and soft prompt methods. First, the technical approaches of these methods are compared, followed by an exploration of various ways to understand their mechanisms, including the perspectives of attention optimization, Parameter-Efficient Fine-Tuning (PEFT), modality integration, and new synthetic language. We also examine the downstream adaptations of various prompt compression techniques. Finally, the limitations of current prompt compression methods are analyzed, and several future directions are outlined, such as optimizing the compression encoder, combining hard and soft prompts methods, and leveraging insights from multimodality.

## introduction as task complexity increases, prompts become longer for llms to accommodate more detailed requirements, contextual information, and in-context learning (icl) examples. lengthy prompts reduce inference speed, increase memory costs, and negatively impact user experience. current methods for improving llm efficiency can be broadly categorized into model-centric and prompt-centric methods @xcite . model-centric approaches, such as parameter pruning @xcite and quantization @xcite , focus on optimizing the model itself. in contrast, prompt-centric methods, including prompt compression @xcite and prompt design @xcite , aim to improve the efficiency of llm by lowering the complexity of input. prompt-centric methods typically introduce minimal or no changes to the parameters of the llm, allowing them to be used in a modular way. thus, these methods, especially for prompt compression as shown in figure 1 , have gained increasing attention. however, the optimal architectures and underlying mechanisms of prompt compression remain unclear, highlighting an important area for further investigation. this survey aims to introduce llm prompt compression to people with prior knowledge of attention mechanisms, transformers, and llms. unlike previous surveys on general efficient prompting @xcite , this paper specifically focuses on prompt compression. the preliminary knowledge is introduced in section 2, covering prompting and efficiency. this is followed by a detailed discussion of hard prompt methods in section 3 and soft prompt methods in section 4. the key elements driving the evolution of prompt compression models are examined, and insights into various methods are provided. their downstream adaptations are discussed in section 5 as well. finally, section 6 analyzes the challenges of current prompt compression techniques and proposes several potential solutions. an overview of prompt compression methods and their downstream adaptations is shown in figure 2 . this survey highlights that: • in addition to the exploration of existing prompt compression methods, we provide our interpretation of their mechanisms: filtering or paraphrasing in hard prompt methods, and attention modification, peft, modality integration, or a new synthetic language in soft prompt methods. ## preliminary prompts are important for llms, broadly classified into hard and soft prompts @xcite . they serve as input instructions to guide llms in performing tasks such as summarization, classification, translation, and question answering without the need for fine-tuning @xcite . the flexibility and effectiveness of prompts leverage the generalization abilities of llms, influenced by elements such as wording, in-context examples, clarity, and accuracy, making prompt design an important area @xcite . prompt structures in this survey consist of two main components: an instruction or context and an input or question, depending on the task. the instruction, input, output format is commonly used in instruction fine-tuning datasets such as alpaca @xcite and pwc @xcite for tasks such as content creation and language translation. the context, question, answer format is prevalent in qa and reading comprehension datasets such as squad @xcite , hotpotqa @xcite , and race @xcite . in few-shot scenarios, several demonstrations with these formats are prepended at the beginning of the prompt, which serves as additional instruction or context to guide the behavior of the llm. prompts can vary significantly in length, with different prompt compression methods targeting some or all of these components. hard prompts are natural language prompts made up of tokens from the vocabulary set of the llm, corresponding to specific words or subwords @xcite . these prompts can be generated by either humans or models. while natural language prompts are interpretable and provide transparency, their inherent ambiguity can make it difficult to fully capture intent in a concise manner. this limitation reduces the utility of hard prompts in diverse contexts and scenarios. additionally, creating effective and precise hard prompts requires considerable human effort, and can involve training a model to refine or optimize these prompts. furthermore, variations in hard prompts can lead to differences in llm performance for the same task. soft prompts are trainable, continuous vectors that share the same dimensions as token embeddings in the dictionary of the llm @xcite . different from hard prompts, the vectors in soft prompts are trained to convey nuanced meanings that cannot be captured by discrete tokens in the predefined vocabulary. when tuned on diverse datasets, soft prompts are expected to help the llm perform well across different tasks. however, as the dataset size grows, the computational resources needed increase as well. additionally, soft prompts are less explainable by humans compared to hard prompts, as their tokens are not directly readable. prompt compression aims to reduce the length of prompts, thereby improving the efficiency of processing llm inputs @xcite . there are two primary approaches to prompt compression: removing unnecessary or low-information content (hard prompt methods) and learning continuous prompt vectors of the prompt information in the embedding space (soft prompt methods) @xcite . hard prompt methods act as a form of filtering, still using natural language tokens, although the resulting prompts may be less fluent and grammatically correct, and can potentially generalize to llms with different embeddings. soft prompt methods, on the other hand, employ encoding to convert prompt information into continuous prompt vectors, resulting in special embeddings that cannot be understood by humans. different from kv compression, prompt compression only manipulates on the input tokens, but not changes the kv values for the input tokens during encoding. thus, the compression ratio is calculated by the number of original input tokens divided by the number of compression tokens. the evaluation of prompt compression methods typically follows two approaches: comparing their performance under the same compression ratio, or examining their achievable compression ratios while maintaining comparable performance levels. ## hard prompt methods hard prompt methods remove unnecessary tokens from the original prompt while maintaining the use of natural language words or sub-words @xcite . these methods are particularly useful for llms that only accept natural language inputs, such as close api models, rather than word embeddings. there are three representative hard prompt methods: selectivecontext @xcite and llmlingua @xcite for filtering, and nano-capsulator @xcite for paraphrasing, as shown in figure 3 . selectivecontext identifies and deletes redundant or less informative parts of an input prompt by quantifying the informativeness of lexical units using self-information @xcite . to maintain text coherence, the syntactic parsing capabilities of spacy foot_1 are used to group individual tokens into noun phrases based on dependency parsing. se-lectivecontext does not rely on external models or additional parameters, and can be applied to any model architecture. however, there are two main disadvantages: (1) it relies on accurate phrase boundary detection using spacy. (2) currently, there are no methods for merging verb phrases. figure 3 : architectures for various prompt compression models by hard prompt methods. for selectivecontext and llmlingua, the bottom language models filter the prompt tokens without modifying them, serving as selection mechanisms. in nano-capsulator, the bottom llm generates a paraphrased version of the input prompt which then serves as input for the llm above. "slm" means "small language model". "close llm" refers to closed-source language models that only accept natural language inputs through api calls. llmlingua employs a smaller language model, such as gpt-2 @xcite , to calculate the self-information of content, similar to selec-tivecontext, and removes redundant tokens from the natural language prompt before it is fed to the llm @xcite . llmlingua operates on prompts structured as {instruction, input (demonstrations), question}, initially selecting key demonstrations based on self-information. it then applies token-level filtering across the prompt, allowing for breaking words into sub-word units and avoiding noun phrase merging by spacy. for key elements such as numbers and units, llmlingua incorporates token preservation algorithms that prioritize these elements within instructions and questions. achieving compression ratios up to 20x, the compression process of llmlingua is managed by the smaller external language model, allowing it to work with close llms. however, there are two limitations: (1) the smaller language model requires additional memory and may use a different tokenizer than the larger llm. (2) since not all prompts contain a large portion of in-context examples, it is important to differentiate between prompt compression and in-context example selection. nano-capsulator summarizes the original prompt into a concise natural language version, which is then input into the llm @xcite . this process removes irrelevant information and makes the prompt into fluent sentences. the compression model, a fine-tuned vicuna-7b, operates independently of the llm. different from standard summarization, nano-capsulator includes a semantic preservation loss to retain key meanings important for downstream tasks and a reward function to optimize the utility of the prompt for the llm. this targeted approach yields better task performance by enhancing semantic relevance. how-ever, the memory cost of the compression model is not negligible. additionally, the compression process is akin to a pre-generation step, requiring more computational resources due to the need for additional inference rather than simple encoding. in addition to the methods mentioned above, general hard prompt methods include longllm-lingua, which has a longer compression window than llmlingua by applying document reordering and subsequence recovery @xcite , and adacomp, an adaptive compression method that dynamically selects relevant documents based on query difficulty and retrieval quality @xcite . these general methods have been further improved in different ways. for example, llmlingua-2 uses data distillation to create a compressed dataset and trains a classifier to retain essential tokens @xcite . both pcrl (jung and kim, 2024) and taco-rl @xcite apply rl for token selection: pcrl is modelagnostic, while taco-rl is task-specific. cpc @xcite and tcra-llm @xcite reduce tokens by leveraging embeddings: cpc ranks sentence relevance with context-aware embeddings, while tcra-llm uses embeddings for summarization and semantic compression. ## downstream adaptions prompt compression has a wide range of applications in domains such as general qa, rag @xcite @xcite @xcite @xcite , icl @xcite , role playing @xcite , agent-based @xcite , and interdisciplinary tasks @xcite . in general qa, prompt compression methods are generally used to compress instructions @xcite or contexts @xcite to perform various instruction-following tasks @xcite . in rag, xrag exemplifies a method that answers questions based on text embeddings encoded from retrieved documents rather than processing the entire text @xcite . for icl, uniicl is an example method that compresses in-context examples to a smaller number of embedding tokens, helping with the selection of relevant examples @xcite . in agent, api documentation can be compressed to enable more efficient tool use @xcite . finetuning problems. although some soft prompt methods, such as icae, 500xcompressor, xrag, and uniicl, do not need to fine-tune the decoder llm, the input soft prompt functions similarly to prompt tuning or prefix tuning for the decoder, leading to problems related to fine-tuning. previous work has shown that fine-tuning base models can result in forgetting, overfitting, and model drift, which can decrease the generalization performance of the base llm @xcite . to avoid these problems, prompt compression models must be trained on large, diverse datasets, which is computationally expensive. furthermore, the trained encoders are specific to the corresponding decoder llms, meaning that when the llm is updated, for example, from llama-2 @xcite to llama-3 @xcite , the encoder must be re-trained. hard prompts face challenges as well: filtered hard prompts may decrease grammatical correctness and provide an unfamiliar input distribution to the llm, potentially influencing its performance. ## conclusions this survey provides a comprehensive overview of the previous prompt compression methods, from the perspectives of hard and soft prompt approaches. in addition to discussing the technical approaches of these models, different perspectives on understanding the compression process and their applications are explored. furthermore, we also discuss the challenges of the existing prompt compression methods and suggest the potential future development directions. we hope our survey offers a comprehensive overview of existing methods, providing deeper insights into their motivations. we also aim for our suggested future directions to inspire the community and support future developments in the field.