Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise (2024)

Abstract: Most existing retrieval-augmented language models (LMs) assume a naive dichotomy within a retrieved document set: query-relevance and irrelevance. Our work investigates a more challenging scenario in which even the “relevant” documents may contain misleading or incorrect information, causing conflict among the retrieved documents and thereby negatively influencing model decisions as noise. We observe that existing LMs are highly brittle to the presence of conflicting information in both the fine-tuning and in-context few-shot learning scenarios. We propose approaches for handling knowledge conflicts among retrieved documents by explicitly fine-tuning a discriminator or prompting GPT-3.5 to elicit its discriminative capability. Our empirical results on open-domain QA show that these approaches significantly enhance model robustness. We also provide our findings on incorporating the fine-tuned discriminator’s decision into the in-context learning process, proposing a way to exploit the benefits of two disparate learning schemes. Alongside our findings, we provide MacNoise, a machine-generated, conflict-induced dataset to further encourage research in this direction.

## introduction the general framework of retrieval-augmented language models (lms) for question answering (qa) consists of retrieving documents related to a question using a sparse @xcite or a dense @xcite retriever, and processing the retrieved documents using encoder @xcite or decoder @xcite ) models to derive an answer. despite being used in many practical applications, most retrieval-augmented lms @xcite @xcite ; lewis et al., 2021) are predicated on a naïve assumption: the retrieved documents are either relevant or irrelevant to the query. however, such a dichotomous view overlooks the fact that in real-world scenarios, the documents purportedly relevant to the query may not consistently offer accurate or reliable information, leading to conflicts among the retrieved documents. such conflicts, as noise, can adversely affect the models that heavily rely on the veracity of the provided information. inconsistencies caused by conflicting information may occur for various reasons such as updated/outdated or fabricated/hallucinated information, with the latter being a significantly growing concern due to documents generated by large language models (llms) flooding the web. we study the robustness of retrieval-augmented lms in the presence of noise and the ensuing knowledge conflict in open-domain question answering (odqa). to facilitate a controllable study, we adopt the widely-used @xcite 's framework that deliberately perturbs the retrieved documents, which is also used in previous works on knowledge conflict @xcite . this deliberate perturbation causes conflict among the documents, as shown in figure 1 , which undermines the model's reliability even in the presence of a gold document. our empirical results show existing models such as fid @xcite and gpt-3.5 (text-davinci-003) @xcite are highly susceptible to conflicting information. to alleviate this problem, we propose inducing the discrimination capabilities and exploiting them in the fine-tuned (fid; §3.1) and in-context learned (gpt-3.5; §3.2) models to let them focus on reliable information. we demonstrate that (i) the fine-tuned lm achieves high precision in discerning authentic from counterfactual documents, and (ii) large language models (llms) leverage rich parametric knowledge to perform tasks with limited training data, but exhibit weakness in distinguishing noisy documents ( §4). based on our findings, we combine the strengths of fine-tuning and prompting, highlighting the potential benefits of leveraging lightweight fine-tuned lms to assist llms. furthermore, while previous works @xcite @xcite also leverage @xcite to emulate knowledge conflict scenarios, the simple entity-swap technique faces several limitations regarding the verisimilitude of the perturbed texts. to this end, we also release a set of llm-generated contradictory documents using @xcite to enable a more realistic and challenging study ( §5). we hope this can further encourage future works to explore conflict resolution in the retrievalaugmented lms. our contributions include: ## method: discern and answer we hypothesize that injecting inductive bias @xcite about whether a document may be perturbed or not into a retrievalaugmented lm improves model robustness to con-flicting information in qa. we equip a qa model with a discriminator learned jointly with a qa task, to interpolate the discriminative features with the encoder embeddings so the decoder can capture such a bias when deriving an answer (figure 2 (a)). besides fine-tuning, we explore the potential to elicit gpt-3.5's discriminability through incontext instruction, by letting the model explicitly discern before answering (figure 2 (b) ) or injecting fine-tuned model's output into a prompt (figure 2 (c)). ## evaluation under entity replacement framework @xcite we measure the performance of fid and gpt-3.5 (text-davinci-003) in the following settings. the parametric (w/o retrieval) setting relies on only rich parametric knowledge @xcite to answer a question. the semi-parametric setting uses retrieved documents and parametric knowledge; we measure how the infused conflicting information affects the models' performance. our methods with discrimination (disc) capabilities are denoted as semi-parametric + disc: the fine-tuned discriminator is superscripted as disc fid and the purely prompt-based discrimination as disc inst . to fit the maximum length of gpt-3.5, we use the top 5 documents for the dev and test sets for both gpt-3.5 and fid for a fair comparison. due to the api budget constraint, we sample 256 dev set as in @xcite , while using the full test set. the generated outputs from gpt-3.5 are ensembled over the k instances (appendix d.4) to mitigate the in-context sample sensitivity observed in @xcite . details are in appendix c. ## evaluation on new machine-generated noise (macnoise) benchmark to extend our evaluation scope beyond the entity replacement, we present macnoise, a machine-generated noise dataset for odqa containing knowledge conflicts among evidence documents. macnoise aims to provide more realistic knowledge conflict scenarios compared to the previous context mismatch question answerability document length counterfactuality perturbation type entity replacement 27.5% 100.0% 106 100.0% er (100.0%) macnoise 0.0% 100.0% 123 91.8% ac (8.9%) | gr (21.9%) | lr (45.2%) | er (24.0%) ## conclusion this work investigates the robustness of retrievalaugmented lms when the retrieved documents include conflicting information. we show that (i) both the fine-tuned lms and in-context learned llms are brittle to the presence of misleading information, and (ii) our perturbation discriminating approach significantly enhances the lms' ability to handle conflicts. furthermore, we find that (iii) combining the fine-tuned discriminator's output with in-context learning improves the llms' stability and robustness, creating a new avenue for future work to utilize the advantages of both learning paradigms. we also release macnoise, an llmgenerated knowledge conflict dataset for odqa, to facilitate further research.