How Does Knowledge Selection Help Retrieval Augmented Generation? (2025)

Abstract: Retrieval-augmented generation (RAG) is a powerful method for enhancing natural language generation by integrating external knowledge into a model’s output. While prior work has demonstrated the importance of improving knowledge retrieval for boosting generation quality, the role of knowledge selection, a.k.a. reranking or filtering, remains less clear. This paper empirically analyzes how knowledge selection influences downstream generation performance in RAG systems. By simulating different retrieval and selection conditions through a controlled mixture of gold and distractor knowledge, we assess the impact of these factors on generation outcomes. Our findings indicate that the downstream generator model’s capability, as well as the complexity of the task and dataset, significantly influence the impact of knowledge selection on the overall RAG system performance. In typical scenarios, improving the knowledge recall score is key to enhancing generation outcomes, with the knowledge selector providing limited benefit when a strong generator model is used on clear, well-defined tasks. For weaker generator models or more ambiguous tasks and datasets, the knowledge F1 score becomes a critical factor, and the knowledge selector plays a more prominent role in improving overall performance.

## introduction retrieval-augmented generation (rag) has become a pivotal technique in natural language generation, enhancing a language model's ability to produce relevant, informed output by incorporating external knowledge @xcite @xcite . this approach complements the model's internal knowledge, which is inherently constrained by the information that was available during training, by retrieving up-to-date, * work performed outside of aws ai labs. the quality of the retrieved knowledge directly influences the quality of rag outputs. previous studies have consistently shown that improving knowledge retrieval leads to a direct enhancement in generation performance @xcite @xcite @xcite . similarly, effective knowledge selection, a.k.a. reranking or filtering, has been observed to improve generation quality by filtering out retrieved information that is less relevant to the generation target @xcite @xcite @xcite @xcite @xcite . however, we observe that knowledge selection is barely used for tasks other than dialogue generation, and there are notably fewer llm-based rag works that use knowledge selection, compared to fine-tuned rag models (see section 2). we hypothesize that knowledge selectors may not always improve downstream generation performance, and that there may be a selection bias where only positive results involving knowledge selector modules are published, while experiments where knowledge selectors are not helpful simply do not report those results. further, prior works focus on proposing specific knowledge selection approaches, offering only narrow, case-specific insights into the relationship between knowledge and generation. as a result, readers are often left with anecdotal observations, such as "using model a in scenario x improves performance", without a global picture of how knowledge selection impacts generation and when it is most effective. therefore, in this work, we perform a systematic empirical analysis * of how knowledge selectors with various performances affect downstream rag performance. by blending gold knowledge with distractor knowledge in varying ratios, we simulate different selection outcomes and examine their impact on the overall performance of rag. we find that the generator model's capability, as well as the complexity of the task and dataset, significantly influence rag system performance. in typical scenarios, improving knowledge recall via the knowledge retriever is key to enhancing generation outcomes, with the knowledge selector providing limited additional benefit when a strong generator model is used on clear, well-defined tasks. for weaker generator models or more ambiguous tasks and datasets, knowledge f1 becomes a critical factor, and the knowledge selector plays a more prominent role in improving overall performance. ## related work retrieval-augmented generation (rag) has been extensively studied in recent years. early works @xcite @xcite jointly fine-tuned a dense retriever (e.g. dpr @xcite ) and generator (e.g. bart @xcite ), which required dedicated training datasets. more recently, the introduction of large language models (llms) has made rag more convenient to implement due to their strong generation performance, in-context learning ability @xcite , and drastically longer context windows -from bart's 1024 tokens to more than a million for modern llms @xcite . as a result, recent rag research has shifted to using llms @xcite @xcite ; in this work, we follow this trend to focus on llmbased rag. knowledge selector for knowledge-grounded dialogue generation. dialogue generation @xcite @xcite is one of the major applications of rag, where the target response is conditioned on retrieved knowledge, and a knowledge selection step is often added to further refine the retrieved knowledge @xcite @xcite . for example, @xcite al. (2020) @xcite and trivia qa @xcite . however, while these works demonstrate the advantages of their approaches through ablation studies, it is unclear whether their performance improvement via knowledge selection specifically is transferable to other datasets, domains, or tasks. moreover, despite the popularity of llms for rag, there are dramatically fewer llm-based works that include the knowledge selection step, and it is likewise omitted in other rag tasks @xcite @xcite . to our knowledge, there is no prior work explaining this gap in knowledge selection. the closest works to ours are @xcite , @xcite and @xcite , which study the impact of knowledge retrieval on rag. we hypothesize that there is a selection bias effect where knowledge selection modules may have been experimented with, but ultimately not reported, in tasks or settings where it is not helpful. in this paper, we study the general effect of knowledge retrieval and selection on llm-based rag via hundreds of simulations, rather than being restricted to a few specific, anecdotal retrievers or selector implementations like prior works. ## meta-experimental results overall we observe several consistent trends across meta-experiments, except for mistral-7b-instruct, which is a relatively weak-performing llm. note that each of the plots in figures 2 3 4 corresponds to a meta-experiment, and each point corresponds to a full experiment on the entire test set. the standard error mean values in tables 1 & 2 indicate that the answer f1 scores are robust across all experiments and the overall observed trends are significant. rag for llm is beneficial. as tables 1 & 2 show, generators without any retrieved knowledge ("no knowledge") perform poorly on wow and hotpotqa, even though they may have been pretrained on the wikipedia articles that wow and hotpotqa are derived from. this finding indicates the llms are not over-fitted on the wow and hot-potqa datasets, and applying rag is beneficial. interestingly, our hotpotqa results in figure 2 show that distractor knowledge significantly harms performance; the cyan dots show that generators receiving mostly distractor knowledge underperform the "no knowledge" setting, which uses only the llm's internal knowledge. this trend is not observed with wow, supporting our observation that hotpotqa's distractor knowledge is truly irrelevant, while the "distractor" knowledge in wow may only be less relevant than the single gold sentence, but not completely irrelevant. this difference between datasets is further explored in appendix a.4. full knowledge setting without knowledge selection is a strong baseline. tables 1 & 2 show that the "full knowledge" setting, which corresponds to knowledge retrieval with perfect recall, but no knowledge selection, is a very strong baseline. this finding mostly contrasts with those of prior knowledge selection works @xcite @xcite @xcite @xcite . for example, in ## conclusion and discussion in this study, we systematically examined the behavior of rag generation in relation to the performance of knowledge retrieval and selection.