QUITO - X : A New Perspective on Context Compression from the Information Bottleneck Theory (2025)

Abstract: Generative large language models ( LLMs) have achieved remarkable success in various industrial applications, owing to their promising In-Context Learning capabilities. However, the issue of long context in complex tasks poses a significant barrier to their wider adoption, manifested in two main aspects: (i) The excessively long context leads to high costs and inference delays. (ii) A substantial amount of task-irrelevant information introduced by long contexts exacerbates the “lost in the middle” problem. Existing methods compress context by removing redundant tokens using metrics such as self-information or perplexity ( PPL ), which is inconsistent with the objective of retaining the most important tokens when conditioning on a given query. In this study, we introduce information bottleneck theory (IB) to model the problem, offering a novel perspective that thoroughly addresses the essential properties required for context compression. Additionally, we propose a cross-attention-based approach to approximate mutual information in IB, which can be flexibly replaced with suitable alternatives in different scenarios. Extensive experiments on four datasets demonstrate that our method achieves a 25% increase in compression rate compared to the state-of-the-art, while maintaining question answering performance. In particular, the context compressed by our method even outperform the full context in some cases.

## introduction in recent years, llms @xcite have been widely applied to various tasks in multiple domains, such as text classification @xcite , question answering systems @xcite , and etc.. as one of the most promising capabilities of these models, in-context learning (icl) (brown, 2020) plays a critical role by enabling the effective use of large language models without requiring additional training. however, in complex tasks, the need to guide the model's adaptation to the task or provide supplementary knowledge often results in excessively long context, leading to high computational costs, increased inference latency, and the "lost in the middle" problem @xcite . therefore, how to compress context while maintaining model performance has become a widely studied topic. in the literature, @xcite utilize language models to compress context in a generative manner, while other methods select the most important lexical units (tokens, words, or sentences) from the original context in an extractive manner. specifically, the generative-based compression methods typically construct compressors by finetuning models to generate summaries of the original text, but they are often constrained by inherent limitations of language models, such as restricted context windows, hallucination phenomena, and the "lost in the middle" problem. the extractive-based compression methods is to design appropriate metrics (e.g., self-information @xcite , perplexity (ppl), self-attention) to assign importance scores to each unit, thereby identifying and removing less salient units. however, the metrics used in previous works are not aligned with the optimization goals of the compressor, which may lead to suboptimal results. for example, these metrics often place excessive emphasis on nouns, while overlooking other crucial elements like prepositional phrases, quantifiers or verbs, which may have lower information entropy. however, neglecting such information can result in highly fragmented compression that is difficult to understand, ultimately leading to incorrect model outputs, as shown in figure 2 . in this paper, we formulate this problem from an information bottleneck (ib) @xcite perspective, deriving mutual information as our metric. we also provide a mathematical proof that using mutual information is equivalent to maximizing the likelihood of the compressed output, which is precisely the compressor's optimization objective. in summary, our contributions are twofold: ## conclusion in this paper, we aim to tackle the challenge of context compression. leveraging information bottleneck theory, we derive mutual information as the optimization objective, which we prove to be equivalent to maximizing likelihood. our method significantly outperforms strong baselines in both inference latency and performance. furthermore, it excels on long texts, occasionally surpassing models that utilize the original context, likely by eliminating inherent redundancy in the context. more effective chunking strategies for long texts are left for future exploration.