B lend F ilter: Advancing Retrieval-Augmented Large Language Models via Query Generation Blending and Knowledge Filtering (2024)

Abstract: Retrieval-augmented Large Language Models (LLMs) offer substantial benefits in enhancing performance across knowledge-intensive scenarios. However, these methods often struggle with complex inputs and encounter difficulties due to noisy knowledge retrieval, notably hindering model effectiveness. To address this issue, we introduce BlendFilter, a novel approach that elevates retrieval-augmented LLMs by integrating query generation blending with knowledge filtering. BlendFilter proposes the blending process through its query generation method, which integrates both external and internal knowledge augmentation with the original query, ensuring comprehensive information gathering. Additionally, our distinctive knowledge filtering module capitalizes on the intrinsic capabilities of the LLM, effectively eliminating extraneous data. We conduct extensive experiments on three open-domain question answering benchmarks, and the findings clearly indicate that our innovative BlendFilter surpasses state-of-the-art baselines significantly.

## introduction generative large language models (llms) have shown remarkable proficiency in various applications, such as summarization @xcite , dialogue systems @xcite , and question answering @xcite . nonetheless, the finite scope of their pre-training corpora imposes inherent limitations, preventing llms from capturing and maintaining comprehensive worldly knowledge, especially given its dynamic nature. this limitation has spurred interest in retrieval-augmented generation strategies that integrate external knowledge sources, like wikipedia, to refine the quality of llm-generated content. typically, retrieval-augmented generation methods @xcite @xcite feed a task input, such as a user query or a question in open-domain question answering, into a retriever to obtain related knowledge documents. subsequently, the llm generates content based on the initial input and the information retrieved. nevertheless, this direct retrieval strategy faces challenges with intricate task inputs @xcite . while straightforward queries enable effective identification of relevant information, multifaceted and complex questions may not cover some essential keywords, complicating the retrieval of pertinent documents. to enhance the retrieval for complex task inputs, recent studies have proposed methods to enrich the original input. these approaches encompass question decomposition @xcite , query rewriting @xcite , and query augmentation @xcite . they utilize knowledge memorized by llms or sourced from external databases to supplement the input with additional information, thereby explicitly incorporating additional keywords and substantially facilitating the retrieval process. among these, query augmentation is particularly noteworthy and achieves state-of-the-art performance because it processes all retrieved knowledge collectively while generating answers and it does not require the training of an additional language model for query rewriting. however, current query augmentation methods still suffer from some limitations. these techniques have typically relied on a single source of augmentation, either llm internal knowledge or an external knowledge base. on one hand, for certain complex inputs, this single source of augmentation may not be able to cover all the keywords and thus lead to insufficient augmentation. furthermore, existing work excludes original input but only rely on the augmented query, which could further exacerbate information loss. another major problem of existing methods is that the incorporated content fetched by the retriever could contain irrelevant or misleading information. usually top-k returned documents by the retriever will be used as augmentation, but there is no guarantee that all the top-k documents are relevant and helpful for the task. correspondingly, incorporating such noise information into the augmented query can potentially lead to inaccuracies in the llm's output @xcite . to mitigate the noise in retrieved knowledge documents, previous studies @xcite @xcite have suggested various strategies. unfortunately, these existing noise reduction methods in knowledge document retrieval are dependent on the llm's confidence levels, which can be imprecise @xcite . additionally, these methods often require an extra language model to determine the need for retrieval, which incurs significant computational costs. to tackle the aforementioned complex question and noisy retrieved knowledge challenges, we propose blendfilter, a novel framework that advances retrieval-augmented large language models by integrating query generation blending and knowledge filtering, as illustrated in fig. 1 . our framework, blendfilter, is structured around three core components: 1) query generation blending module, 2) knowledge filtering module, and a 3) answer generation module. the query generation blending module is dedicated to enhancing input queries through diverse augmentation strategies, essentially forming a composite of queries, to handle the complex question challenge. this module incorporates both external and internal knowledge sources for augmentation. these augmented queries, including the original, external knowledge-augmented, and internal knowledgeaugmented, are then employed by the retriever to collect pertinent information. in order to tackle the noise retrieved knowledge challenge, our proposed knowledge filtering module, aims to eliminate irrelevant retrieved knowledge and could operate autonomously without needing an extra language model, leveraging the innate filtering prowess of the llm. in the final phase, the llm integrates the filtered knowledge with the original query to generate the final answer. the contributions are summarized as follows: 1) we introduce a novel query generation blending approach that integrates various augmentation sources. in contrast to existing work that relies on one source only, the proposed method enriches queries by using a variety of knowledge sources, which lead to a more comprehensive coverage of pertinent knowledge. 2) we present a novel and effective knowledge filtering module designed to eliminate irrelevant knowledge. we are the first to propose the utilization of the llm itself as a filter in retrieval-augmented generation tasks. 3) we conduct extensive experiments across three opendomain question answering benchmarks. the results demonstrate that our proposed model, blend-filter, significantly surpasses the baseline models across three distinct backbones. ## related work retrieval-augmented generation enhances large language models (llms) by leveraging external knowledge to improve generation quality. initial approaches, as discussed in @xcite @xcite , portrayed llms as passive recipients of retrieved knowledge, lacking interactive dynamics with retrievers. however, due to the inherent challenges in accurately capturing relevance between inputs and documents, these direct methods often yield only marginal improvements. addressing this, recent advancements @xcite @xcite @xcite @xcite @xcite have empowered llms to engage actively with retrievers, thereby enhancing relevance modeling. the integration of llms into the retrieval process broadly falls into three categories: 1) question decomposition, 2) query rewriting, and 3) query augmentation. for question decomposition, as exemplified by @xcite and @xcite , llms break down a complex question into simpler components, leveraging both previous interactions and retrieved knowledge. this decomposition facilitates more straightforward reasoning by llms. however, the success of this approach heavily depends on the llm's capabilities. insufficiently powerful llms might generate misleading sub-questions. moreover, this method requires maintaining a historical context, potentially leading to lengthy dialogues and increased computational costs. in the realm of query rewriting, models are trained, often utilizing reinforcement learning, to reformulate the original question into a version more conducive to retrieval @xcite . these revised questions typ- external knowledge augmented query 𝒒𝑖𝑛 input query 𝒒 knowledge base … retrieved knowledge input query 𝒒 llm llm response input query 𝒒 input query 𝒒 llm internal knowledge augmented query 𝒒𝑖𝑛 llm response input query 𝒒 input query 𝒒 knowledge base llm … retrieved knowledge input query 𝒒 filtered knowledge knowledge base llm … retrieved knowledge input query 𝒒 filtered knowledge knowledge base llm … retrieved knowledge input query 𝒒 filtered knowledge external knowledge augmented query 𝒒𝑖𝑛 internal knowledge augmented query 𝒒𝑖𝑛 llm input query 𝒒 union of filtered knowledge final answer internal knowledge augmentation external knowledge augmentation query generation blending knowledge filtering answer generation ically yield improved generation outcomes. nevertheless, training an additional model for rewriting is a resource-intensive process. the third approach, query augmentation, involves enriching queries with knowledge from either llm internal databases or external sources @xcite . a limitation of this method is its reliance on a single source of augmentation and often overlooking the original query, thus constraining overall model performance. the aforementioned studies directly utilize retrieved knowledge, yet recent research @xcite highlights that such knowledge can sometimes be irrelevant or even detrimental to llms when answering queries. to solve this challenge, @xcite suggests an initial assessment to determine if llms need to retrieve knowledge, utilizing a classifier that could be based on bert-like models or the llm itself. however, this approach requires additional training data, which poses challenges in zero-shot or few-shot learning scenarios, and the llm's self-evaluation may not always yield reliable results. @xcite introduces a self-reflective method to ascertain the necessity of retrieval and to assess the relevance between the retrieved knowledge and the input. a critical limitation of this method, as noted by @xcite , is its dependence on training an auxiliary language model to produce text with reflection tokens, incurring extra costs. additionally, @xcite employs a strategy of comparing the average negative likelihood of answers with and without external knowledge to guide decision-making. nevertheless, this measure may not be a precise indicator of model confidence and is not universally applicable across models, with certain models like gpt-3.5-turbo and gpt-3.5-turbo-instruct currently unable to access this feature. we summarize the differences between the proposed blendfilter and other baselines in ## methodology given a pre-trained large language model (llm) m(•), a knowledge base @xmath0, a retriever r(•), and a query q, our objective is to utilize the knowledge base to facilitate accurate responses from the llm without fine-tuning. ## experiment in this section, we evaluate the proposed blendfilter and answer the following research questions: rq1) how does blendfilter perform compared to state-of-the-art retrieval-augmented baselines? rq2) can the proposed blendfilter generalize well with respect to different backbones and retrievers? rq3) is the llm effective to filter unrelated knowledge documents? rq4) what are the roles of the original query, external knowledge-augmented query, and internal knowledge-augmented query in model performance improvements respectively? rq5) how does the performance change with varying numbers of knowledge documents? rq6) will the proposed blendfilter be improved by sampling multiple times with different temperatures? ## conclusion in this paper, we introduce blendfilter, a comprehensive framework developed to enhance retrievalaugmented generation within llms. our methodology distinctively incorporates query generation blending and knowledge filtering techniques, effectively tackling the intricacies of complex inputs and significantly reducing noise in retrieved knowledge. the amalgamation of external and internal knowledge augmentation fosters a resilient and allencompassing retrieval mechanism. additionally, our innovative self-reliant knowledge filtering module exploits the inherent capabilities of the llm to refine and purify the retrieved knowledge by eliminating extraneous content. we conducted extensive experiments on three benchmarks, and the results demonstrate that blendfilter outperforms state-of-the-art baselines. moreover, blendfilter can be generalized well for different kinds llms, including gpt3.5-turbo-instruct, vicuna 1.5-13b and qwen-7b.