RE - RAG : Improving Open-Domain QA Performance and Interpretability with Relevance Estimator in Retrieval-Augmented Generation (2024)

Abstract: The Retrieval Augmented Generation (RAG) framework utilizes a combination of parametric knowledge and external knowledge to demonstrate state-of-the-art performance on open-domain question answering tasks. However, the RAG framework suffers from performance degradation when the query is accompanied by irrelevant contexts. In this work, we propose the RE-RAG framework, which introduces a relevance estimator (RE) that not only provides relative relevance between contexts as previous rerankers did, but also provide confidence, which can be used to classify whether given context is useful for answering the given question. We propose a weakly supervised method for training the RE simply utilizing question-answer data without any labels for correct contexts. We show that RE trained with a small generator (sLM) can not only improve the sLM fine-tuned together with RE but also improve previously unreferenced large language models (LLMs). Furthermore, we investigate new decoding strategies that utilize the proposed confidence measured by RE such as choosing to let the user know that it is “unanswerable” to answer the question given the retrieved contexts or choosing to rely on LLM’s parametric knowledge rather than unrelated contexts.

## introduction in recent years, the retrieval augmented generation framework has shown promising progress in natural language generation, specifically on knowledgeintensive tasks. this approach has been studied in many forms, from traditional rag @xcite , which aggregates answers from multiple contexts using document relevance scores as weights, to approaches like ralm @xcite , which simply utilizes concatenated context as an in-context learning approach for largelanguage models (llms). retrieval augmented generation enhances the model's faithfulness and reliability by leveraging nonparametric knowledge on top of parametric knowledge @xcite . in particular, the rag framework has the advantage of being easily adaptable to modern llms @xcite . these advantages have sparked a significant amount of new research @xcite @xcite focused on the rag framework. despite the great potential of the retrieval augmented generation framework, if the language model is provided with contexts that are not relevant to the query, it will be distracted by these inappropriate contexts, negatively affecting the accuracy of the answers @xcite . while retrievers or re-rankers in existing research have been effective at measuring the relative ranking across contexts to a query, these modules often fail to determine whether top-ranked contexts are actually relevant to the query or not. furthermore, if a precise relevance score is not used in the traditional rag framework, it can cause problems such as directing attention to documents that are less likely to answer the query. in this work, we propose the re-rag framework, which extends traditional rag by incorporating a relevance estimator (re) to simultaneously measure the precise relative relevance between retrieved contexts and evaluate their confidence, which can be used to classify whether given context is useful for answering the given question. by more accurately measuring the relative relevance between contexts, re computes precise relevance scores suitable for weighted aggregated answers in the traditional rag framework and also acts as an efficient reranker. re trained on a small generator (slm) not only benefits slm fine-tuned together with re but can also be separated and applied to llms as well, benefiting both. by explicitly classifying whether the context is useful for answering the query, the confidence of context measured by re provides various decoding strategies. if the retrieved context set is irrelevant, we can choose to classify the query as "unanswerable", while maintaining most of the accuracy for the answerable set. additionally, if a low-confidence context set is retrieved, which will likely result in wrong answers by parroting the context as is @xcite , we can instead selectively leverage the llm's parametric knowledge to improve answer accuracy in most cases. the main contributions of our work are: ## method in this section, after reviewing the traditional rag framework, we present the re-rag model combined with our relevance estimator. ## experimental setup we evaluated the performance of our model on an open-domain qa dataset. in this section, we describe the dataset we used in our experiments and the details of our experiments. ## experiment results we investigate the qa performance of the rag system with our newly proposed relevance estimator (re). in addition to the qa performance of the whole system, we also examine the performance of the re independently. ## related works previous research has shown that the performance of question answering systems can be improved by utilizing external knowledge about questions @xcite . methods for more accurate retrieval of external knowledge @xcite @xcite have been studied to make these systems more efficient. in open-domain qa, models that extract and use answers from retrieved documents have been studied @xcite @xcite , but studies that utilize generative models such as t5 @xcite or bart @xcite have become more common @xcite . rag and fid achieved powerful performance in open-domain qa using different methods. subsequently, models @xcite @xcite that leverage and improve upon the structural advantages of fid have been proposed. for atlas @xcite , state-of-the-art performance was achieved through an improved retriever @xcite and scaling up the model. in the case of rag, there is a study that improved performance by introducing a bert @xcite -based reranker @xcite , but it utilized additional data and high-quality label data when training the reranker. recently, large language models (llms) such as gpt @xcite and llama @xcite , which have been developed in recent years, face limitations with fid methods that require encoded data. consequently, research on rag models, which can directly input context, has received renewed attention. @xcite @xcite these approaches have achieved performance improvements by training a retriever, which can also be applied to llms, or by performing the review of questions and context within the model itself. ## conclusion we propose the re-rag framework, which extends traditional rag by incorporating re that can measure the relative relevance and confidence of contexts. we demonstrate that the re-rag framework can enhance the performance of traditional rag. we show that the re module, as a detachable component, can be combined with modern large language models (llms) to improve their performance. furthermore, we exploree some decoding strategies that leverage the confidence information measured by the re module to either answer "unanswerable" or selectively utilize the parametric knowledge of the llms when a low confidence context set is retrieved. we hope that our research will inspire the exploration of various additional modules for retrieval-augmented generation.