Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learning (2024)

Abstract: Retrieval-Augmented Language Models (RALMs) have significantly improved performance in open-domain question answering (QA) by leveraging external knowledge. However, RALMs still struggle with unanswerable queries, where the retrieved contexts do not contain the correct answer, and with conflicting information, where different sources provide contradictory answers due to imperfect retrieval. This study introduces an in-context learning-based approach to enhance the reasoning capabilities of RALMs, making them more robust in imperfect retrieval scenarios. Our method incorporates Machine Reading Comprehension (MRC) demonstrations, referred to as \textit{cases} , to boost the model’s capabilities to identify unanswerabilities and conflicts among the retrieved contexts. Experiments on two open-domain QA datasets show that our approach increases accuracy in identifying unanswerable and conflicting scenarios without requiring additional fine-tuning. This work demonstrates that in-context learning can effectively enhance the robustness of RALMs in open-domain QA tasks.

## introduction retrieval augmented language models (ralms) have demonstrated remarkable performance in the field of open-domain question answering (qa). by leveraging external knowledge to generate answers, ralms enhance accuracy and enable language models to respond to queries beyond their training data. @xcite @xcite typically, ralms operate in two stages: the retrieval step, which involves fetching relevant contexts from external knowledge sources, and the generation step, where answers are generated based on the retrieved contexts. recent research has shown that using frozen large language models (llms) without additional fine-tuning during the generation step can also be effective. @xcite however, a critical issue in open-domain qa is the reliance of ralms on the quality of external knowledge. figure 1 illustrates common imperfect retrieval scenarios in ralms. in unanswerable scenario where the retrieved contexts do not contain the correct answer, ralms cannot provide an accurate response. additionally, when contexts are retrieved from various sources, such as search engines, conflicting information may arise. in such scenario, ralms may struggle to determine the correct information, leading to reliance on their parametric knowledge or potential hallucination. figure 2 : an overview of our approach. conventional ralm generates answers by providing the llm with context retrieved from a knowledge source. in contrast, our method simultaneously inputs cases that enhance the llm's reasoning capability, allowing it to generate answers. this leads to more robust reasoning compared to conventional ralm. to address these challenges, we propose the in-context learning @xcite based approach to enhance the reasoning capabilities of llms, thereby increasing robustness in such imperfect retrieval scenarios. unlike previous approaches that depend on extensive fine-tuning @xcite @xcite @xcite , our method leverages the in-context learning capability of llms, demonstrating that providing simple examples to llms can improve robustness in opendomain qa without additional training. figure 2 provides an overview of our approach. unlike conventional ralm, our method retrieves demonstrations (referred to as cases) that assist in answering a given query. by concatenating these retrieved cases to the llm's input during retrieval-augmentation, we enhance the llm's reasoning abilities through in-context learning. this enables the ralms to perform more robust reasoning. our experiments show that providing llms with machine reading comprehension (mrc) demonstrations enhances accuracy and the ability to detect unanswerability. additionally, presenting llms with simple examples that simulate conflicts among retrieved contexts improves their ability to identify such conflicts. our contributions and key findings are summarized as follows: ## method our objective is to enhance the reasoning capabilities of llms in open-domain qa scenarios, particularly in detecting unanswerable scenarios where no answer exists within the retrieved contexts, and conflict scenarios where contradictions exist among retrieved contexts. our approach follows the in-context ralm method @xcite , which concatenates retrieved contexts as inputs to a frozen llm for retrieval-augmentation. to further enhance the llm's reasoning capability, we will add demonstrations to the ralms by simply concatenating demonstrations to the existing ralm input. typically, in-context learning provides examples of the same task @xcite , but our demonstrations are based on machine reading comprehension (mrc) datasets, which have a single shorter context, rather than generating answers from multiple documents as in odqa. we refer to these demonstrations as cases. ## experiments in these experiments, we aim to investigate how effectively our constructed cases can help llms identify unanswerability and conflicts in open-domain qa scenarios. ## conclusion we conducted experiments leveraging the incontext learning capabilities of llms, using sim-ple mrc examples to improve robustness in opendomain qa scenarios. these results show that providing mrc examples as demonstrations improves accuracy for both answerable and unanswerable examples in unanswerable scenarios. in conflict scenarios, providing demonstrations similar to conflict situations enhances the ability to identify conflicts. our experiments suggest that well-designed examples can significantly improve llms' robustness in open-domain qa without additional finetuning, indicating that simple examples can help solve complex tasks.