Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation (2025)

Abstract: Retrieval-augmented generation (RAG) improves large language models (LMs) by incorporating non-parametric knowledge through evidence retrieved from external sources. However, it often struggles to cope with inconsistent and irrelevant information that can distract the LM from its tasks, especially when multiple evidence pieces are required. While compressing the retrieved evidence with a compression model aims to address this issue, the compressed evidence may still be unfamiliar to the target model used for downstream tasks, potentially failing to utilize the evidence effectively. We propose FaviComp (Familarity-Aware Evidence Compression), a novel training-free evidence compression technique that makes retrieved evidence more familiar to the target model, while seamlessly integrating parametric knowledge from the model. Experimental results show that FaviComp consistently outperforms the most recent evidence compression baselines across multiple open-domain QA datasets, improving accuracy by up to 28.1% while achieving high compression rates. Additionally, we demonstrate the effective integration of both parametric and non-parametric knowledge during evidence compression.

## introduction retrieval-augmented generation (rag) has become a common paradigm for large language models (lms) to leverage external knowledge beyond their inherent knowledge boundaries to perform better in knowledge-intensive tasks such as opendomain question answering (qa) @xcite @xcite and fact-checking @xcite . in particular, incorporating multiple evidence pieces is crucial in solving complicated tasks such as multi-hop and complex reasoning @xcite @xcite , which require various sources of information to solve the questions. nevertheless, rag often struggles to cope with inconsistent and irrelevant information from the multiple evidence pieces, which can interfere with downstream tasks @xcite . this highlights the need for evidence compression to identify and retain only the essential information for lms to utilize effectively. traditionally, evidence compression has focused on reranking documents or sentences by relevance and then incorporating a top-ranked subset @xcite @xcite or compressing the documents into a compact form that retains only essential context @xcite @xcite . however, the compressed evidence might be unfamiliar to the lm employed for the downstream task (referred to as the target model), particularly due to discrepancies in the internal knowledge and prompt preferences between the compression model and the target model @xcite @xcite . when lms encounter unfamiliar contextual information, they often fail in balancing parametric and non-parametric knowledge, either by overly relying on their parametric knowledge @xcite @xcite or by utilizing retrieved evidence without considering its relevance to the input @xcite . to address these challenges, we propose familiarity-aware evidence compression (favicomp), an inference-time evidence compression method that consolidates multiple evidence into an abstractive summary that is more familiar to the target model, while seamlessly integrating parametric knowledge from the model. inspired by the prior findings that an lm's familiarity with a prompt is generally reflected by low perplexity figure 1 : an overview of favicomp. instead of relying solely on compressed evidence from the compression model (upper), favicomp familiarizes the compressed evidence to the target model while integrating parametric knowledge through ensemble decoding, resulting in improved downstream performance (lower). @xcite @xcite , favicomp proactively composes the compressed evidence in a way to lower the perplexity of the target model. specifically, instead of directly selecting the highest probability token from the compression model at each decoding step, favicomp selects the token from the ensemble of the token probabilities from both the compression and target models. this ensemble decoding therefore constrains the token search space of the compression model to those with lower perplexity for the target model, making the context more familiar to the target model @xcite . furthermore, favicomp potentially synergizes the retrieved knowledge with the target model's parametric knowledge introduced during ensemble decoding. it can effectively discern when to leverage internal or external knowledge, which is particularly beneficial in the presence of noisy contextual evidence in complex tasks such as multi-document or multi-hop qa @xcite . our experiments show that favicomp outperforms most recent evidence compression baselines in five open-domain qa datasets, improving accuracy by up to 28.1% while maintaining high compression rates. additionally, we conduct ablation studies by varying the degree of decoding ensemble and analyzing its impact on performance and context perplexity. moreover, we investigate how favicomp effectively integrates parametric and non-parametric knowledge during evidence compression. ## method we present favicomp, a inference-time evidence compression method that familiarizes retrieved evi-dence with the target model while synergizing them with the model's parametric knowledge. we first illustrate the motivation for favicomp in §2.1 and provide the preliminaries of evidence compression in rag §2.2, followed by a detailed definition of our proposed framework in §2.3. ## experimental settings we assess the effectiveness of favicomp on knowledge-intensive qa tasks. in this section, we delve into the details of the experimental settings. ## experimental results in this section, we compare the overall performance of favicomp with other baselines across the five datasets ( §4.1), explore the impact of ensemble coefficient α on performance and perplexity ( §4.2), investigate how effectively favicomp incorporate parametric and non-parametric knowledge ( §4.3), and compare the compression rates with other baselines ( §4.4). ## case study in both examples, raw document fails to produce the correct answer, even though the evidence contains the necessary information, highlighting the need for effective evidence compression. in the first example, while the difference between the compressed evidence from zero-shot summarization and favicomp appears subtle, favicomp delivers the correct answer with a lower perplexity in compression, underscoring the significance of evidence familiarization. the second example highlights the importance of parametric knowledge when the retrieved evidence set lacks complete information. since the evidence set does not mention "skeptic", zero-shot summarization introduces irrelevant information ("philanthropy magazine"), ultimately leading to an incorrect answer. in contrast, favicomp integrates parametric knowledge about "skeptic" and incorporates it into the evidence compression. notably, favicomp selects the arg max token from the target model only when the token's probability is higher than that of the compression model, demonstrating that favicomp draws on parametric knowledge only when necessary-potentially when the compression model is uncertain about the next token. question: this film is an adaption of a jacques offenbach's opera that was written by a hungarian british screenwriter? methods (compressed) evidence prediction perplexity raw document the tales of hoffmann is a 1951 british technicolor film adaptation of jacques offenbach's opera "the tales of hoffmann", written, produced and directed by the team of michael powell and emeric pressburger ... emeric pressburger ✗ 12.429 zero-shot summarization the 1951 film "the tales of hoffmann" is an adaptation of jacques offenbach's opera, written, produced, and directed by michael powell and emeric pressburger. emeric pressburger ✗ 2.298 favicomp the 1951 film "the tales of hoffmann" is an adaptation of jacques offenbach's opera, written by emeric pressburger, a hungarian-british screenwriter, and directed by michael powell and emeric pressburger. the tales of hoffmann ✓ 1.959 question: which magazine was first published earlier, the chronicle of philanthropy or skeptic? raw document the chronicle of philanthropy is a magazine that covers the nonprofit world. ... it was founded in 1988 by editor phil ... first published as a newsletter in 1987, "philanthropy" became a glossy magazine in 1996. philanthropy ✗ 4.856 zero-shot summarization the chronicle of philanthropy was founded in 1988, while philanthropy magazine was first published as a newsletter in 1987 and became a glossy magazine in 1996. philanthropy magazine ✗ 3.196 favicomp the chronicle of philanthropy was first published in 1988, while skeptic was first published in 1992. the chronicle of philanthropy ✓ 1.345 ## conclusion in this study, we introduce favicomp, a trainingfree, inference-time evidence compression method designed to enhance rag performance by consolidating retrieved evidence set to be more familiar to the target model, while seamlessly integrating parametric knowledge. our extensive experiments validate the effectiveness of favicomp on opendomain qa tasks, showing significant improvements over recent evidence compression baselines in multiple datasets. additionally, favicomp's model-agnostic nature allows it to be incorporated into various rag workflows at inference time, making it a versatile tool for enhancing lms in complex tasks.