Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence (2022)

Abstract: Question answering models can use rich knowledge sources — up to one hundred retrieved passages and parametric knowledge in the large-scale language model (LM). Prior work assumes information in such knowledge sources is consistent with each other, paying little attention to how models blend information stored in their LM parameters with that from retrieved evidence documents. In this paper, we simulate knowledge conflicts (i.e., where parametric knowledge suggests one answer and different passages suggest different answers) and examine model behaviors. We find retrieval performance heavily impacts which sources models rely on, and current models mostly rely on non-parametric knowledgein their best-performing settings. We discover a troubling trend that contradictions among knowledge sources affect model confidence only marginally. To address this issue, we present a new calibration study, where models are discouraged from presenting any single answer when presented with multiple conflicting answer candidates in retrieved evidences.

## introduction traditionally, qa models have relied on retrieved documents to provide provenance for their answers @xcite . recent studies @xcite have shown that large language models are able to retain vast amounts of factual knowledge seen during pretraining, and closed-book qa systems @xcite build upon this foundation by memorizing facts from qa finetuning. retrieval-based generation approaches @xcite emerge as the best of both worlds -generating free-form answers from the question paired with retrieved evidence documents. they further combine these parametric knowledge sources with a large number of retrieved evidence documents, achieving state-of-the-art performances on open retrieval qa datasets @xcite . norway? germany? the u.s? the u.s. team had a historic winter games, winning an unprecedented 37 medals. figure 1 : models can use both parametric and nonparametric knowledge sources. in this example, the answer could be the u.s./norway/germany. we investigate for a given question which knowledge source was the most influential to output an answer. the model should be able to abstain from answering for these examples, as it is difficult for the model to decide which answer candidate is correct. understanding how retrieval-based generation models combine information from parametric and non-parametric knowledge sources is crucial for interpreting and debugging such complex systems, particularly in adversarial and complex real world scenarios where these sources may conflict with each other (see an example in figure 1 ). this can aid both developers to debug such models and for users to estimate how much they should trust an answer @xcite . thus, we focus on the following core question: when provided with numerous evidence passages and a pretrained and finetuned language model, which knowledge source do models ground their answers in? a recent study @xcite investigated this in a limited single evidence document setting. we expand this study to consider a more realistic scenario, where models consider multiple evidence passages (up to 100 passages), and observe results diverging from their reported heavy reliance on parametric knowledge. we further sim-ulate a setting where a subset of evidence passages are perturbed to suggest a different answer to reflect the realistic scenario where retrieval returns a mixed bag of information. such scenarios are common in settings where some passages are updated with new information, while other passages remain outdated @xcite . such conflicts can also occur when passages are adversarially edited to contain false information @xcite , or when passages are authored by multiple people who have differing opinions about an answer @xcite . our extensive studies on two datasets @xcite and two models @xcite exhibit that retrieval-based generation models are primarily extractive and are heavily influenced by a few most relevant documents instead of aggregating information over a large set of documents. learning that models mostly rely on evidence passages rather than parametric knowledge, we evaluate how sensitive models are toward semantic perturbation to the evidence documents (e.g., adding negation). we find retrieval-based generation models behave similarly to extractive models, sharing their weakness of returning answer candidates with high confidence, even after the context is modified to no longer support the answer @xcite . what should models do when confronted with conflicting knowledge sources? we propose a new calibration setting (section 5), where a model is encouraged to abstain from proposing a single answer in such scenarios. we find that teaching models to abstain when there are more than one plausible answers is challenging, and training a separate calibrator with augmented data helps moderately. to summarize, we empirically test how qa models @xcite use diverse knowledge sources. we present the first analysis of knowledge conflicts where (1) the model uses multiple passages, (2) knowledge conflicts arise from ambiguous and context-dependent user queries, and (3) there are knowledge conflicts between different passages. our findings are as follows: when provided with a high recall retriever, models rely almost exclusively on the evidence passages without hallucinating answers from parametric knowledge. when different passages suggest multiple conflicting answers, models prefer the answer that matches their parametric knowledge. lastly, we identify various weaknesses of retrievalbased generation models, including its confidence score not reflecting the existence of conflicting answers between knowledge sources. our initial calibration study suggests that dissuading models from presenting a single answer in the presence of rich, potentially conflicting, knowledge sources is challenging, and demands future study. ## background we first describe the task setting, qa models, and calibrator used in our study. we study open retrieval qa, where the goal is to find an appropriate answer y * for a given question q. systems for open retrieval qa may also be provided with access to a knowledge corpus consisting of a large number of passages, p, which is used to help answer the question. we use the open retrieval split @xcite of the nat-uralquestions dataset (nq-open) @xcite and triviaqa @xcite , and use wikipedia as our knowledge corpus. foot_0 ## when do retrieval-based generation models rely on parametric knowledge? as an initial step investigating whether retrievalbased generation models ground their answers in the retrieval corpus or in the pretrained language model's parametric knowledge, we evaluate whether models generate a novel answer that is not present in a set of evidence documents. unlike extractive qa models @xcite , generation based approaches @xcite do not require the evidence documents to contain the gold answer span. thus, we first analyze whether they actually generate novel answer spans not found in the retrieved passages. revisiting knowledge conflict study in @xcite this observation stands at odds with an earlier study on knowledge conflict @xcite which simulates knowledge conflict by substituting the existing answer with a new answer candidate in the evidence passage (see question: when was the last time the bills won their division? contradicts with knowledge from lm. the model produced the original answer 17% of the time, even when the answer no longer appears in the passage. we identify that the main difference in their experimental setup is in using a single evidence passage rather than multiple evidence passages. we re-visit their study, as single document setting is impractical. most open-retrieval qa models @xcite @xcite are trained with multiple passage to make up for imperfect passage retrieval. according to the answer recall in following their setup, we only evaluate on examples that the model has correctly answered (as perturbing examples where models are already confused is unnecessary) and where the answer is an entity. 6 we then substitute every answer entity mention in all evidence passages with a random entity of same type sampled from the training data. 7 all manipulation was done only at inference, and after the passages are retrieved. we report the exact match score to the original answer. prior to perturbation, the exact match score against the original answer is 100%. we also report the exact match score to the substituted answer and memorization ratio (m @xmath0, and p s is the fraction of examples predicting the substitute answer. takeaway retrieval-based reader models exhibit little memorization when the retriever has a high # pass. ans. exact match mr train / inf. r orig. sub. ## simulating mixed bag of evidence passages having identified that retrieval-based generation models rely heavily on evidence passages, especially when paired with a high-performance retriever, we study how models make use of multiple evidence passages when different passages suggest different answers. this happens frequently in real life, as questions can be ambiguous based on different, valid interpretations of the question @xcite or different extra-linguistic contexts (zhang and choi, 2021). we introduce two perturbations -an entity substitution perturbation @xcite (section 4.1) and adversarial semantic perturbation @xcite (section 4.2) -both will dissuade model from returning the original answer in the evidence passage (see ## re-calibrating models given a mixed bag of evidence when presented with a mixed bag of evidence, systems should inform users of the multiple, conflicting answers. while there are many of approaches for relaying this information to users (e.g., composing a paragraph aggregating answer candidates, or providing set of answers mapped to documents supporting them), a necessary prerequisite to all such systems is the ability to detect when there are conflicting answers in the evidence. thus, we explore creating systems that can detect and abstain from predicting on instances with conflicting evidence. questions should only be answered if (1) there is no knowledge conflict in its evidence set and (2) model's predicted answer matches the annotated answer. we report calibrator's binary calibration accuracy following prior work @xcite . we explain four evaluation settings here. original we use the original nq development set as is to provide a reference for the performances of calibrators. in the following settings, we only look at examples where the original fid model correclty answers. thus, the calibrator should only abstain for knowledge conflict. we construct three different types of knowledge conflict set where calibrator should abstain on half of the examples because of the knowledge conflict. to construct these set, we use the original question, 100 evidence passage set (where model should present its answer), and augment one perturbed example, where 100 evidence passage set is perturbed to have multiple answer candidates to the same question. we discuss three ways to introduce perturbed evidence set, with more than one valid answer candidate now. partial substitution we use the sets of conflicting evidence passages constructed in section 4.1 (randomly sampling 50% of the retrieved passages to substitute a new entity in). ambigqa instead of random new entity, we sample valid alternative answer to the question taken from a different interpretation of the same question from ambigqa @xcite dataset. instead of simply replacing answer in existing passage, we retrieve new passage for each rewritten, disambiguated version of the question. situatedqa we sample valid alternative answer from either corpus taken from a different time period from situatedqa (zhang and choi, 2021) dataset. we use the same query, but retrieve over two different snapshots of the same corpus (the wikipedia dump from 2018 and from 2021). we evenly combine retrieved passages from conflicting answer sets, using the top retrieved passages that contain the respective answer and backing off to the passages with high retrieval scores if not enough passages contain the answer string. calibrator as a baseline, we use same calibration model from our prior study in section 2.2. we also retrain separator calibrators for each of our three substituted answer types, which are trained by applying the same data augmentation process that was applied to the evaluation set (described above) to training portion of filtered nq-open dataset. ## related work recent analysis @xcite pointed the overlap in training and evaluation dataset inflates question answering performances. @xcite showed that the reader model tend to memorize entity answers despite the answer mentions are substituted by another entity. we showed that memorization do occur when the model can only have access to one passage, but can be reduced significantly if the model is trained with multiple passages. concurrent work @xcite investigates qa models' robustness to misinformation by providing contradicting contexts. they focus on generating conflicting passages, while we focus on understanding how models behave under such settings, including in-depth study of their confidence score. recent works evaluated robustness by minimally perturbing input examples @xcite to identify models that are invariant under distributional shift. prior work explored automatically generating such perturbed input (counterfactual data) with heuristics @xcite or learned models @xcite @xcite . recent work @xcite studies knowledge poisoning for a related task, fact checking. our perturbation methods are rule-based similar to ribeiro et al. (2020), but designed specifically for qa task. ## conclusion we summarize our findings: do models ground their answers from retrieved document or parametric knowledge? (section 3) current sota models ground their answers mostly from retrieved passages, when paired with a high recall retriever ( how do models use multiple passages when different passages suggest different answers? (section 4.1) models rely on a few, most relevant passages ( how do models behave if some passages are perturbed not to support an answer? (section 4.2) models largely ignore semantic perturbations and outputs potential answer entity in the retrieved passages ( how is the model's confidence score affected by knowledge conflicts? confidence score is not sensitive to knowledge conflicts ( can we train a model to refrain from returning a single answer when there is conflicting evidence? if we train a calibrator on the conflicting evidence set, calibrator can learn to refrain, but does not generalize to different types of conflicting evidence sets ( what should the model do when there is conflicting evidence? we present a partial solution of training a calibrator which learns to abstain from answering when provided conflicting evidence. future work can explore summarizing and comparing different answers suggested by diverse passages. overall, models' limited ability to aggregate conflicting information among its rich knowledge sources encourage future work in this domain.