Understanding and Improving Information Preservation in Prompt Compression for LLM s (2025)

Abstract: Recent advancements in large language models (LLMs) have enabled their successful application to a broad range of tasks. However, in information-intensive tasks, the prompt length can grow fast, leading to increased computational requirements, performance degradation, and induced biases from irrelevant or redundant information. Recently, various prompt compression techniques have been introduced to optimize the trade-off between reducing input length and retaining performance. We propose a holistic evaluation framework that allows for in-depth analysis of prompt compression methods. We focus on three key aspects, besides compression ratio: (i) downstream task performance, (ii) grounding in the input context, and (iii) information preservation. Using our framework, we analyze state-of-the-art soft and hard compression methods and show that some fail to preserve key details from the original prompt, limiting performance on complex tasks. By identifying these limitations, we are able to improve one soft prompting method by controlling compression granularity, achieving up to +23% in downstream performance, +8 BERTScore points in grounding, and 2.7× more entities preserved in compression. Ultimately, we find that the best effectiveness/compression rate trade-off is achieved with soft prompting combined with sequence-level training.

## introduction recent advancements in large language models have enabled their successful application to a broader range of tasks that require long-context input, detailed instructions, and in-context learning (icl) demos @xcite @xcite @xcite . the resulting prompts can become very long, especially for tasks requiring extensive contextual information. even though state-of-the-art llms are able to process hundreds of thousands of tokens @xcite @xcite @xcite , providing them with long input context introduces computational inefficiencies @xcite . recent research has focused on developing methods for prompt compression to reduce the length of the llms inputs while preserving the information needed to successfully complete the task @xcite . the two main approaches to compression are: soft-prompting, which compresses the context into dense memory slots by learning continuous representations of the information in the latent space @xcite @xcite ; and hard-prompting, which operates on the surface form and involves removing unnecessary or lowinformation content, e.g., by pruning parts of the original text, or summarizing it @xcite @xcite . the resulting hard-compressed prompts still use natural language, but they may be less fluent. soft compression methods allow for significantly higher compression rates @xcite , as they operate in the embedding space. however, it is not easy to understand the information that is contained in soft prompts since, in contrast to hard prompting, these are not directly interpretable. a limitation of the work done on prompt compression is that it focuses mostly on downstream performance on tasks like question answering @xcite ) or reading comprehension @xcite ). we argue that downstream performance itself is insufficient to evaluate the quality and limitations of a compression method, and to understand its ability to preserve information from the original context. to address this gap, we propose a holistic evaluation framework to analyze the performance of prompt compression ap-proaches, which considers three key dimensions, beyond compression rate: (i) downstream task performance, (ii) grounding of the compressed context to the original text, and (iii) type and amount of information preserved during compression. the framework includes the evaluation on various text generation tasks with different complexity and context lengths (from 850 to 6.5k tokens), including multi-hop reasoning, conversation qa, long document summarization, and mathematical reasoning. we use our newly proposed framework to analyze a representative hard-prompting method, llmlingua @xcite , and two softprompting approaches, xrag @xcite and @xcite . all these methods are limited in handling long-context scenarios, exhibiting performance drops across various generation tasks and an increased number of ungrounded responses. we also demonstrate that the target llm struggles to accurately reconstruct the original context from soft prompts, particularly for xrag, often hallucinating information and failing to regenerate crucial details. based on these observations, we explore modifications of xrag, where we incorporate fine-grained data samples during pre-training and operate on more granular data at generation. our results show substantial gains: 23% improvement on downstream tasks, up to 8 bertscore f1 points in response grounding, and 2.7× more entities preserved after compression. still, we see limitations in the ability to integrate information across separate units. our results show that this gap can be filled by soft prompting methods such as pisco, which achieve the best tradeoff between compression rate and effectiveness. in summary, this paper provides the following contributions: (i) it reveals major limitations of state-of-the-art soft and hard prompt compression methods; (ii) it advocates for a holistic framework to evaluate prompt compression methods beyond downstream task performance; (iii) it presents promising directions to address the limitations of soft prompting methods, showing that manipulating the granularity of the compressed content is key to improving performance. ## related work prompt compression is a technique to improve the efficiency of llms by creating an approximate representation of a prompt with reduced size @xcite . approaches can be categorized into two main groups: hard prompting and soft prompting @xcite . hard prompting compression methods focus on producing a shorter text version of the input prompt, with minimal token usage to retain the key information. these methods operate at various granularity levels, from omitting low-information instruction content and icl demonstrations to refining token selection @xcite . approaches include optimizing task definitions @xcite , dynamic compression allocation @xcite , syntax-guided compression @xcite and demonstration selection via diversity-based sampling or relevance scoring @xcite . demonstration ordering can mitigate position bias @xcite , while token classification ensures faithfulness to the original text @xcite . however, hard prompting may disrupt grammar, introduce unfamiliar input distributions, and require re-encoding, impacting efficiency. additionally, extreme compression is challenging, and compressing long inputs can be computationally expensive @xcite . soft prompting methods compress a given text into continuous representations @xcite . soft prompts can achieve higher compression rates (i.e., fewer input tokens for the llms), but fall short in explainability-the content encoded in the soft prompt is not directly interpretable by humans. approaches like autocompressors recursively generate summary vectors @xcite , while gisting condenses prompts into transformer activations using virtual gist tokens @xcite . icae @xcite and 500xcompres @xcite encode long contexts into compact memory slots: icae leverages a loraadapted encoder and 500xcompressor uses keyvalue representations for richer information retention. xrag @xcite projects dense retrieval embeddings into the llm's representation space via a trainable modality bridge. it employs a frozen embedding model as the encoder, with a lightweight adapter between the encoder and the decoder llm as the only trainable component. unlike other soft prompting methods, xrag keeps both the encoder and decoder frozen, making it a modular and efficient solution for rapid development. @xcite is another compressor-decoder model relying on knowledge distillation from document-based questions. while # dataset ## prompt compression methods analysis for our experiments, we select two soft prompting and one hard prompting method, using the same target llm -mistral-7b instruct v0.2 @xcite ) -across all setups, with the nocompression case as an upper bound. ## improving xrag the results presented in section 4 show a large disproportion between pisco and the remaining two prompt compression methods. in particular, we highlight xrag's inability to preserve detailed information from the original input in long-context scenarios. we here explore directions to mitigate these limitations of xrag, while still retaining the advantage of high compression rates. we use our prompt compression evaluation framework to verify whether the modifications lead to improvements over the original xrag method. we re-train an xrag model using the data and code released with its paper to ensure that the models are fully comparable (details in appendix d). we denote this reproduced version as xrag (reproduced). ## conclusions in this paper, we propose a framework for assessing the quality of prompt compression methods on long-context generation tasks. we shed light on both the capabilities and limitations of the stateof-the-art hard (llmlingua) and soft (xrag and pisco) compression approaches. our findings reveal that some existing compression techniques fail to preserve detailed information, especially in longcontext scenarios. we explore several directions to improve the xrag soft compression method, showing an increase in model performance -up to +23% on downstream tasks, up to 8 bertscore points in grounding, and preserving 2.7x more entities. foot_5