EXIT : Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generation (2025)

Abstract: We introduce EXIT, an extractive context compression framework that enhances both the effectiveness and efficiency of retrieval-augmented generation (RAG) in question answering (QA). Current RAG systems often struggle when retrieval models fail to rank the most relevant documents, leading to the inclusion of more context at the expense of latency and accuracy. While abstractive compression methods can drastically reduce token counts, their token-by-token generation process significantly increases end-to-end latency. Conversely, existing extractive methods reduce the latency but rely on independent, non-adaptive sentence selection, failing to fully utilize contextual information. EXIT addresses these limitations by classifying sentences from retrieved documents—while preserving their contextual dependencies—enabling parallelizable, context-aware extraction that adapts to query complexity and retrieval quality. Our evaluations on both single-hop and multi-hop QA tasks show that EXIT consistently surpasses existing compression methods and even uncompressed baselines in QA accuracy, while also delivering substantial reductions in inference time and token count. By improving both effectiveness and efficiency, EXIT provides a promising direction for developing scalable, high-quality QA solutions in RAG pipelines. Our code is available at https://github.com/ThisIsHwang/EXIT.

## introduction retrieval-augmented generation (rag) @xcite is the task of enhancing the responses of large language models (llms) with relevant external contexts or documents. by grounding answers in evidence, rag systems have gained much attention for mitigating hallucination issues @xcite and improving factual reliability @xcite . however, they face significant challenges in practical deployment, as retrieval models sometimes fail to rank the most relevant documents at the top @xcite . one potential solution is to retrieve a larger set of documents to ensure the coverage of the necessary information, but this approach compromises both effectiveness and efficiency. specifically, for effectiveness, llms often struggle with processing long contexts, overlooking critical information located in the middle of contexts @xcite . additionally, irrelevant information in retrieved documents can act as distractors, significantly degrading the overall qa performance @xcite @xcite . from an efficiency perspective, increasing the context size raises inference latency-due to quadratic complexity in attention computation @xcite -and api costs tied to input length foot_0 . moreover, the context window limitations inherent in llm architectures set strict upper bounds on the maximum input size @xcite . to address these challenges, context compression has emerged as a promising solution, condensing essential information from multiple retrieved contexts through either abstractive or extractive approaches. while they can reduce inference time and filter out irrelevant information, both approaches still have notable drawbacks. specifically, abstractive compression methods-often implemented via autoregressive generation-summarize or rewrite documents into a single condensed passage @xcite @xcite @xcite , significantly increasing end-to-end latency due to their tokenby-token generation process. for instance, as illustrated in figure 1 , compact @xcite , a representative abstractive approach, takes over 8 seconds to process just five documents for a single query, whereas using the original document without compression takes only 1 second. on the other hand, extractive compression approaches can offer a more efficient alternative @xcite @xcite , by selecting relevant textual segments (e.g., sentences, or even token-level excerpts) directly from retrieved documents. this strategy reduces both compression time and overall latency. however, current extractive methods have yet to reach their full potential in terms of effectiveness @xcite . they often rely on rigid selection criteria that do not adapt well to variations in query complexity or the quality of retrieved documents, and they frequently neglect to fully leverage the broader context when choosing which tokens or sentences to retain. specifically, as illustrated in figure 1 , while extractive approaches such as recomp-extr @xcite achieve minimal compression time, their inability to dynamically adjust selection processes results in suboptimal qa performance. therefore, in this work, we propose a novel compression framework for rag, extractive context compression (exit), designed to enhance both effectiveness and efficiency by improving efficiency through an extractive compression and by enhancing effectiveness through dynamic, context-aware sentence selection. specifically, as shown in figure 2 , exit operates in three stages: (1) splitting retrieved documents into sentences, (2) performing parallelizable binary classification ("yes" or "no") on each sentence to assess its relevance while considering its full document context, rather than evaluating them independently, and (3) recombining selected sentences while preserving their original order. therefore, as shown in figure 1 , exit frames context compression as a sentence classifi-cation problem, enabling it to outperform both compression methods and the uncompressed baseline in terms of speed. specifically, it reduces processing time from several seconds to about 1 second. moreover, by leveraging context-aware and adaptive sentence selection, exit also surpasses other extractive methods in accuracy. also, we note that exit operates as a plug-and-play module that can be seamlessly integrated into any existing rag pipeline without architectural modifications. we evaluate exit on both single-hop qa tasks (nq @xcite , tqa @xcite ) and multi-hop qa tasks (hqa @xcite , 2wikimultihopqa @xcite ). experimental results demonstrate that exit not only improves effectiveness over both abstractive and extractive compression baselines but also significantly reduces latency compared to abstractive methods and the uncompressed baseline. our contributions are as follows: retrieval-augmented generation. rag enhances llms by grounding responses in retrieved external documents @xcite @xcite . however, increasing retrieved content often leads to longer contexts, higher inference costs @xcite , and retrieval noise @xcite , impacting efficiency and accuracy. to mitigate these challenges, ad-hoc methods have been proposed at different stages of the retrieval pipeline, including pre-retrieval and postretrieval methods. pre-retrieval methods, such as query . . . . . . . . . first, the retrieved document is split into sentences. next, each sentence is classified as either "yes" or "no" using the compressor. finally, sentences with scores above the threshold are recombined in their original order to complete the compression. landmark embedding @xcite and late chunking @xcite , improve document representations to enhance retrieval effectiveness but lack adaptive filtering mechanisms after retrieval and may be difficult to integrate orthogonally with existing retrieval methods. post-retrieval methods, including reranking (nogueira and cho, 2019; @xcite and context compression @xcite @xcite , refine retrieved documents through reordering or pruning. however, most post-retrieval methods overlook query complexity and operate on a fixed number of retrieved items, limiting their adaptability in balancing effectiveness and efficiency. we introduce exit, a novel post-retrieval compression framework that adaptively filters queryrelevant sentences while discarding redundancy. unlike existing methods, exit dynamically adjusts to query complexity and retrieval quality, balancing efficiency and effectiveness. its lightweight, adaptive design seamlessly integrates into rag pipelines while remaining orthogonal to pre-retrieval methods. context compression. context compression is a practical remedy for handling increasingly extensive prompt lengths in rag pipelines. existing methods commonly fall into soft or hard compression. soft compression focuses on embedding prompts into compact continuous representations @xcite @xcite @xcite , but requires extensive training and architectural changes, making it unsuitable for black-box llms. hard compression, by contrast, removes non-essential textual content directly @xcite , offering a plug-and-play solution even compatible with api-based models such as chatgpt @xcite . hard compression are further divided into abstractive and extractive methods. abstractive methods employ autoregressive models to generate query-focused summaries, thus reducing token counts at the cost of additional latency and potential hallucinations @xcite . for example, recomp-abst @xcite uses a t5-based summarizer for token reduction but requires dataset-specific training and slows inference. compact @xcite and refiner @xcite take this method further by leveraging even larger llms with 7b parameters, compounding latency issues and increasing resource demands. extractive methods, such as recomp-extr @xcite , directly select salient segments (e.g., sentences or tokens) from retrieved documents. this avoids the autoregressive bottleneck and mitigates hallucinations. however, their static and contextagnostic selection of only a few sentences per document limits its performance. similarly, token-level methods such as llmlingua family @xcite @xcite can distort semantic coherence by removing key entities or splitting essential facts. in short, abstractive methods offer strong com-pression but suffer from latency and potential hallucinations, while extractive methods are often rigid and lack context awareness. our work addresses these limitations by proposing a parallelizable, context-aware extractive compression framework. it adaptively selects sentences at scale, optimizing accuracy and speed in complex retrieval scenarios. ## method in this section, we first present our problem formulation, the rag pipeline with a compression stage, and our novel compression framework, exit, which is designed to extract key evidence for answering in a parallel manner. ## experiment setups we conduct comprehensive experiments to evaluate exit's effectiveness and efficiency in context com-pression for rag systems. more implementation details of our experiments are in appendix b. datasets. we evaluate on both single-hop and multi-hop question answering datasets: natu-ralquestions (nq) @xcite and triviaqa (tqa) @xcite for singlehop qa; hotpotqa (hqa) @xcite and 2wikimultihopqa (2wiki) @xcite for multi-hop qa. we use the test set for tqa evaluations and development sets for all other datasets. for the train dataset for the classifier, we exploit the train split of hqa, which has relevant annotations to each sentence in the multiple documents for a query. model configuration. our system consists of three primary components. the retriever employs contriever-msmarco @xcite , a dense retriever fine-tuned on msmarco @xcite . the exit compressor utilizes gemma-2b-it @xcite , optimized for efficient parallel processing. for the reader model, we exploit two scales of instruction-tuned models: llama3.1-{8, 70}b-instruct @xcite . baselines. we compare exit against the following context compression methods: 1) original documents serves as uncompressed baseline. for abstractive methods, 2) recomp-abs @xcite uses t5-based (775m) summarization tuned for nq, tqa, hqa (hqa model for 2wiki), 3) compact @xcite implements mistral-7b-based iterative compression with 5-segment blocks, and 4) refiner @xcite uses llama2-7b-based compression. for extractive methods, 5) recomp-extr @xcite employs contriever-based (110m) sentence-level extraction tuned for nq, tqa, hqa (hqa model for 2wiki), and 6) longllmlingua @xcite uses llama2-7b-chat for token-level extraction with 0.4 dynamic compression rate. evaluation metrics. we use three metrics: exact match (em) and f1 score to measure effectiveness in question answering, and end-to-end inference latency (lat.) in seconds to assess efficiency. endto-end latency captures the total wall-clock time required to generate an answer for a single query, including both compression time and llm inference time. we report the average latency per sample, computed over the entire evaluation set. this provides a more holistic efficiency metric than token count alone, as it directly reflects the practical runtime performance of the full rag pipeline. ## main results the benefits of exit become more pronounced at larger scales. using the 70b reader, exit surpasses the accuracy of all competing methods, averaging a 3.7-point improvement in em and a 3.3-point improvement in f1 over the uncompressed baseline. on hqa, it achieves a 3.3-point em gain while maintaining an efficient 3.5s latency-faster than using uncompressed documents and still competitive with the previously fastest method, recomp-extr, but with significantly higher accuracy. exit's effectiveness and efficiency, especially with larger models, makes it a practical solution for large-scale qa applications. ## analyses we conduct a series of analyses examining exit's robustness, classification performance, latency factors, and design choices under various configurations. additional experimental results and analyses are provided in appendix c. ## conclusion we present exit, an efficient context compression framework for rag systems that leverages parallel processing and context-aware, adaptive sentence selection. our experiments demonstrate that exit achieves superior performance across both singlehop and multi-hop qa tasks while maintaining practical inference speeds. despite being trained only on hqa, exit shows a strong zero-shot generalization ability and proves effective across a wide range of open-source models of varying sizes. these results show that parallel extraction with smaller models outperforms larger abstractive methods, making it practical for real-world rag.