Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLM s (2024)

Abstract: The integration of retrieved passages and large language models (LLMs), such as ChatGPTs, has significantly contributed to improving open-domain question answering. However, there is still a lack of exploration regarding the optimal approach for incorporating retrieved passages into the answer generation process. This paper aims to fill this gap by investigating different methods of combining retrieved passages with LLMs to enhance answer generation. We begin by examining the limitations of a commonly-used concatenation approach. Surprisingly, this approach often results in generating “unknown” outputs, even when the correct document is among the top- k retrieved passages. To address this issue, we explore four alternative strategies for integrating the retrieved passages with the LLMs. These strategies include two single-round methods that utilize chain-of-thought reasoning and two multi-round strategies that incorporate feedback loops. Through comprehensive analyses and experiments, we provide insightful observations on how to effectively leverage retrieved passages to enhance the answer generation capability of LLMs. On three open-domain question answering datesets, NQ, TriviaQA and SQuAD, our multi-round approaches outperform traditional concatenation approach, achieving over a 10% improvement in answer EM.

## introduction large language models (llms), such as gpts @xcite , have found extensive applications, but often struggle with limited knowledge representation, resulting in inaccuracies and insufficient specificity in open-domain question answering. to overcome these limitations, the integration of retrieval-based techniques @xcite @xcite has emerged as a promising solution. by incorporating relevant passages during the answer generation, llms can leverage external information to provide more accurate and detailed responses. nevertheless, effective strategies for incorporating retrieved passages into the llms remains a challenging and relatively understudied area. our analysis (fig. 1 ), conducted under the oracle setting where one of the top-k retrieved passages contains the answer, reveals that a simple concatenation of the passages into llms often leads to "unknown" responses -instances where the provided context fails to answer the question -accounting for about 20% of all responses. an alternative method, where the passages are individually provided as input to llms and the majority vote determines the final answer, reduces the rate of "unknown" generation to 2-7% depending on the number of passages. however, this method introduces a new challenge: the correct answer does not align with the majority vote in the answer pool. particularly, when more passages are incorporated from 5 to 50, the error rate of the majority vote increases from 12% to 22%. thus, both of the methods have their own weaknesses and more suitable approaches for the integration of retrieved passages and llms remain to be investigated. transformer-based llms have shown the capability to utilize attention mechanisms @xcite for discovering token-level relevance. however, they may not always attend to the relevant parts within the context, leading to a potential oversight of important information present in the retrieved passages @xcite . this challenge becomes more pronounced when dealing with extensive corpora like wikipedia, which contains over 21 million passages, making it a formidable task to identify the most relevant passages for a question. furthermore, retrieved passages that are closely related to the question's topic can act as distractors, potentially misleading the model @xcite . if the model mistakenly directs its attention towards these distractor passages, it can introduce noise that negatively impacts the answer prediction process. in this paper, we explore the integration of retrieved passages with llms like chatgpts to enhance their ability to generate correct answers. in particular, we examine situations where the retrieved passages contain the correct answer, yet the model fails to generate the correct response, indicating areas for improvement. initially, we focus on two chain-of-thought (cot) @xcite @xcite strategies that incorporate in-context learning. we introduce a pruning strategy and a summarization strategy for the retrieved passages to guide the answer generation process of the llms. subsequently, we investigate two multi-round methods with feedback: post-fusion as the fallback: in the initial round, this method employs the concatenation approach to generate an answer. if the llm generates "unknown" responses with the inputs, it proceeds to use post-fusion in the second round, generating candidate answers. the final answer is chosen via majority vote. concatenation as the distiller: this approach starts by leveraging post-fusion to produce a pool of potential answers and to identify relevant passages. in the subsequent round, only the unfiltered passage is concatenated with the question and answer candidates from the first round. this consolidated input is then fed into the llm to derive the final answer. through extensive experiments on three singlehop open-domain question-answering datasets, we showcase the enhanced performance of our proposed methods, achieved with a minimal additional resource cost. our findings provide a foundation for the development of more advanced retrievalintegration methods aimed at further enhancing the capabilities of these models. ## problem setup this study focuses on the question answering task under the open-domain setting. it remains a open problem to retrieve the most relevant context for question answering. therefore, a common practice is to include multiple top ranked passages, which serves as the supplementary context for the llms. the number of supplementary passages, denoted as k, can vary based on the desired input length m of the llm. typically, k can be set to 5, 10, or 20, ensuring that the total length of k passages, each having a maximum length of l, remains within the maximum input length m of the llm (i.e., k * @xmath0. by incorporating these supplementary passages, the llm is provided with a more comprehensive and informative context, which has the potential to enhance its accuracy. ## methods we adopt a two-stage pipeline for open-domain qa. it consists of two black-box components, a retriever and a llm such as @xcite . we aim to methodically investigate the optimal methods for transferring the top-k retrieval results to the llms for generating factoid answers. our investigation begins with a focus on various single-round strategies, wherein the retrieved passages are directly fed into the llms. subsequently, we delve into several multi-round approaches, involving the initial supply of retrieved passages to the llms, gathering feedback, and then modifying the interaction process with the llms based on that feedback. ## related work the recent proliferation of llm-powered applications, such as chatgpt/gpt4 (openai, 2023), bing chat, and copilot, has highlighted both the impressive performance and certain limitations of llms. these limitations include a high compute and data demand, making it a challenge to continually update llms both efficiently and effectively @xcite . llms also tend to generate plausible yet non-factual texts, a phenomenon known as "hallucination" @xcite . in response to these issues, the field is witnessing a trend towards augmenting llms with specialized tools @xcite , such as code interpreters @xcite @xcite or search engines (park and @xcite . the goal is to delegate specific tasks to more proficient systems or to enrich the llms' input context with more pertinent information. augmentation of language models with pertinent data retrieved from diverse knowledge bases has demonstrated its effectiveness in enhancing opendomain question answering performance @xcite @xcite . the process typically involves using the input query to (1) command a retriever to fetch a document set (essentially, token sequences) from a corpus, after which (2) the language model integrates these retrieved documents as supplemental information, guiding the final prediction. the interleaving between the retriever and llm could be considered a reciprocal process. various studies have been conducted on generationaugmented retrieval (gar), which involves revising or supplementing queries with generated background information to enhance the retrieval of relevant content. well-known examples of this approach include gar @xcite and hyde @xcite . with regard to complex multi-step reasoning questions, work involving llms often necessitates the retrieval of segmented knowledge @xcite @xcite . this chain-ofthought reasoning process @xcite @xcite is followed by conducting partial reasoning to generate the next question, then retrieving further information based on the outcome of that partially formed next question, and repeating this cycle as needed @xcite . our work primarily focuses on a specific scope: once the output from the retriever is determined, we aim to identify the most effective method of inputting this data into llms for answer generation. ## conclusion in this study, we identified two key challenges associated with integrating llms and retrieved passages: the occurrence of "unknown" responses when feeding llms with concatenated passages and the erroneous majority when using the post-fusion approach. to overcome these challenges, we proposed four improved approaches, including two cot-related strategies and two multi-round methods incorporating llm's feedback. through our experimental results and token usage analysis, we observed that it is advantageous to first employ a concatenation strategy to generate an answer. in the case of an "unknown" response, we recommend transitioning to the post-fusion approach to obtain the final answer through a majority vote.