EC o RAG : Evidentiality-guided Compression for Long Context RAG (2025)

Abstract: Large Language Models (LLMs) have shown remarkable performance in Open-Domain Question Answering (ODQA) by leveraging external documents through Retrieval-Augmented Generation (RAG). To reduce RAG overhead, from longer context, context compression is necessary. However, prior compression methods do not focus on filtering out non-evidential information, which limit the performance in LLM-based RAG. We thus propose Evidentiality-guided RAG, or ECoRAG framework. ECoRAG improves LLM performance by compressing retrieved documents based on evidentiality, ensuring whether answer generation is supported by the correct evidence. As an additional step, ECoRAG reflects whether the compressed content provides sufficient evidence, and if not, retrieves more until sufficient. Experiments show that ECoRAG improves LLM performance on ODQA tasks, outperforming existing compression methods. Furthermore, ECoRAG is highly cost-efficient, as it not only reduces latency but also minimizes token usage by retaining only the necessary information to generate the correct answer. Code is available at https://github.com/ldilab/ECoRAG.

## introduction llms @xcite have excelled in tasks such as odqa by leveraging external knowledge through rag @xcite . however, rag inevitably increases context length, which incurs higher computational cost and also hinders generation quality @xcite @xcite . while adopting existing context compression @xcite may look promising, such a baseline presents two main challenges. first, llms are known to be vulnerable to irrelevant contents that @xcite , applying recomp @xcite , and applying ecorag on the natural questions @xcite test set. experiments were conducted using flan-ul2 @xcite . second, it is challenging to determine the desirable compression ratio for each question. failure to do so may lead to compressing too much, which results in losing crucial information, or compressing too little, which produces overly long contexts that degrade generation quality @xcite @xcite and increase computational costs. thus, it is necessary to find the desirable compression ratio that enables the llm to generate the correct answer for each question. our distinction is using evidentiality to address both challenges and proposing evidentialityguided compression and retrieval-augmented generation (ecorag) framework: ours compresses retrieved documents to retain only the information necessary to support the answer. to overcome the first challenge, evidentiality @xcite is used to determine whether each sentence in the retrieved documents supports the correct answer to a question. it can be quantified for each sentence by measuring how much it contributes to the model to generate the correct answer. we train the compressor using this as training signals. to address the second challenge, ecorag reflects on compression as a collective, where it contains sufficient evidence. we begin by forming the smallest possible collective unit of compression and assess whether it is evidential. if not, it means that it is compressed too much, which we adjust adaptively by collecting more, until it is sufficient. through this reflection process, ecorag finds the desirable compression ratio that enables the llm to generate the correct answer with minimal tokens. by applying these methods, ecorag has two advantages when dealing with long contexts as the number of documents increases. first, ec-orag improves performance by retaining only the information necessary for generating the correct answer and removing distracting content. this results in gains on odqa datasets such as natural questions (nq) @xcite , triv-iaqa (tqa) @xcite , webquestions (wq) @xcite . second, by compressing the long context to only what is needed, it reduces computational costs. our contributions to this work can be summarized as follows: (1) evidentiality-guided compression: we developed a method that compresses retrieved documents based on evidentiality. (2) evidentiality reflection for adaptive compression: our framework evaluates compressed content for evidentiality and adaptively adjusts the length of compression. (3) experiments show that our approach significantly improves retrieval-augmented llm performance on odqa datasets. (4) our approach is also cost-efficient, as it quickly compresses long context, reducing latency and tokens. ## proposed method in this section, we describe how ecorag adaptively adjusts the compression length to ensure that the llm generates the correct answer. to achieve this, we focus on: (1) compressing retrieved documents by sorting them based on evidentiality (section 3.1), and (2) evaluating whether the compressed documents is sufficiently evidential, and if not, adaptively incorporating more information (section 3.2), and figure 2 provides an overview. ## analysis in addition to the main results, we verified the effectiveness of our framework by addressing the following research questions: ## conclusion ecorag is a framework designed to compress long context by focusing on evidentiality in llms, defined as whether information supports generating the correct answer. evidentiality-guided compression effectively filters out irrelevant content and retains necessary evidence. through adaptive compression, ecorag determines the optimal compression length for each question, ensuring efficient use of context. as a result, ecorag demonstrates both superior performance and efficiency in handling long context, outperforming other compression methods.