A ttn C omp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation (2025)

Abstract: Retrieval-augmented generation improves the factual accuracy of Large Language Models (LLMs) by incorporating external context, but often suffers from irrelevant retrieved content that hinders effectiveness. Context compression addresses this issue by filtering out irrelevant information from context before LLM generation. However, existing methods struggle to adaptively adjust compression rates for different context, maintain low latency and integrate information across multiple documents. To overcome these limitations, We introduce AttnComp, an adaptive, efficient and context-aware compression framework. By leveraging the attention mechanism of LLMs to identify relevant information, AttnComp employs a Top-P compression algorithm to retain the minimal set of documents whose cumulative attention weights exceeds a predefined threshold. In addition to compression, AttnComp estimates response confidence by assessing the overall relevance of the retrieved content, enabling users to gauge response reliability. Experiments demonstrate that AttnComp outperforms existing compression methods and uncompressed baselines, achieving higher accuracy with substantial compression rates and lower latency.

## introduction retrieval-augmented generation (rag) enhances the factual accuracy and reliability of large language models (llms) in knowledge-intensive tasks by integrating retrieved context into their generation process @xcite @xcite @xcite . however, practical rag applications often grapple with retrieved content containing substantial irrelevant information, even entirely unrelated to the query @xcite . this gives rise to three primary issues: first, llms can * corresponding author: yixuan cao and ping luo. to mitigate these issues, context compression has emerged as a promising solution to filter out irrelevant information before generation. existing methods can be categorized into abstractive and extractive approaches. abstractive methods leverage llms to summarize or rewrite retrieved content via autoregressive generation @xcite @xcite . while achieving high compression rates, they incur significant latency due to token-by-token decoding. extractive methods instead select relevant spans from the original content, offering greater efficiency @xcite @xcite . however, current extractive methods typically only assess the relevance of individual sentence or document to the query, limiting their ability to integrate information across broader context. furthermore, many such approaches rely on fixed compression rates or target lengths @xcite , ignoring the variable proportion of relevant content and risking under-or over-compression. consequently, we posit that an effective context compression method should exhibit three key properties: (1) adaptive: it should dynamically adjust the compression rates based on the proportion of relevant information within the context. (2) efficient: it should maintain low computational cost and latency, ensuring rapid processing for real-time applications. (3) context-aware: it should integrate and synthesize information from the entire retrieved content to accurately identify relevant segments. however, to the best of our knowledge, no existing compression method simultaneously satisfies all three of these properties. to bridge this gap, we introduce attncomp (attention-guided context compression), an adaptive, efficient and context-aware extractive compression method that leverages the inherent attention mechanisms of llms. as illustrated in figure 1 , the attncomp pipeline consists of two stages: (1) attention computation. given a prompt that combines the instruction, retrieved documents and query, we compute attention weights from middle layers of the llm to quantify the relevance of each text segment to the query. (see sec. 4.1 for details). (2) top-p compression. we aggregate the attention weights to compute scores for the instruction and each document. documents are then ranked by score, and the top ones are retained until their cumulative score, combined with that of the instruction, reaches a predefined threshold. compared to fixed-length compression, this approach adjusts the retained content based on attention distribution, allowing for flexible selection ranging from no documents to all documents (see sec. 4.2 for details). we observe that while the attention mechanisms in llms inherently capture relevance, they can still assign high attention to irrelevant content. this is particularly evident when all retrieved documents are irrelevant, as the model fails to shift attention away from the documents, with some irrelevant ones consistently receiving high attention. to ad-dress this, we fine-tune the cross-attention layer of the model to direct attention to relevant documents when present, or to the instruction when all documents are irrelevant (see sec. 4.3 for details). experimental results on multiple qa datasets highlight the superior performance of attncomp. it achieves a 1.9 point accuracy improvement over the uncompressed baseline, while other compression methods incur at least a 3 point decrease. this advantage is even more pronounced in multi-hop question answering, which requires integrating information from multiple documents and thus places higher demands on context-aware compression capabilities. here, our method yields at least a 5.4 point improvement over other sentence-level compression methods. beyond accuracy, attncomp achieves a 17x compression rate, outperforming all other evaluated extractive methods, and significantly reduces the rag system's end-to-end latency to 49% of the uncompressed baseline. beyond its primary role in compression, at-tncomp also offers a valuable capability for estimating the confidence of rag responses by leveraging the attention assigned to the instruction (see sec. 4.4 for details). after training, the attention allocated to the instruction correlates with the quality of retrieved documents, serving as an indicator of answer reliability. experiments show a strong positive correlation between the confidence score and actual answer accuracy, enabling users to assess response trustworthiness and mitigate risks from low-quality retrieval. furthermore, this capability also suggests a possible avenue for future research on autonomous iterative rag @xcite @xcite . in summary, our contributions are as follows: ## observations in this section, we present our observations on the attention patterns within llms. our analysis is conducted on qa datasets, where inputs are constructed by concatenating the context before the query. we then compute the attention score from the query to different context segments as follows: ## attncomp inspired by our observations, we propose a novel compression framework attncomp. in this section, we provide a comprehensive explanation of the framework. problem formulation given a query q, a rag system retrieves a set of k documents @xmath0, d 2 , . . . , d k }. a language model m then generates an output y conditioned on the retrieved documents and the query, i.e., m (y | d, q). our objective is to filter irrelevant documents from d, yielding a reduced subset d ′ ⊆ d such that the size of d ′ is minimized while maintaining or even improving the quality of generated answer m (y | d ′ , q). ## analysis we evaluate attncomp for its adaptiveness (sec. 6.1), efficiency (sec. 6.2), context-awareness (sec. 6.3), and robustness (sec. 6.4). we also present an ablation study (sec. 6.5) and validate the reliability of its confidence estimation (sec. 6.6). ## conclusion we introduce attncomp, a novel framework that leverages the attention mechanism to adaptively compress retrieved documents. additionally, at-tncomp provides a confidence estimation capability for evaluating rag responses. extensive experiments demonstrate that attncomp outperforms existing compression methods and uncompressed baseline, offering higher accuracy with significant compression rates and lower end-to-end latency.