What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems (2023)

Abstract: NLP systems have shown impressive performance at answering questions by retrieving relevant context. However, with the increasingly large models, it is impossible and often undesirable to constrain models’ knowledge or reasoning to only the retrieved context. This leads to a mismatch between the information that the models access to derive the answer and the information that is available to the user to assess the model predicted answer. In this work, we study how users interact with QA systems in the absence of sufficient information to assess their predictions. Further, we ask whether adding the requisite background helps mitigate users’ over-reliance on predictions. Our study reveals that users rely on model predictions even in the absence of sufficient information needed to assess the model’s correctness. Providing the relevant background, however, helps users better catch model errors, reducing over-reliance on incorrect predictions. On the flip side, background information also increases users’ confidence in their accurate as well as inaccurate judgments. Our work highlights that supporting users’ verification of QA predictions is an important, yet challenging, problem.

## introduction with the advent of large language models pretrained on massive data, question answering (qa) systems are able to reason about information that is external to their input context, based on their implicit factual or commonsense knowledge @xcite or by employing shortcuts to get to the correct answer @xcite @xcite . however, in such incomplete information settings, users may lack important background knowledge required to assess the correctness of model predictions: consider the example of figure 1 . here, a user interacts with a qa model to seek information about which non-swedish actress starred in the movie "light between oceans." the qa model retrieves the context (left) about the movie and predicts alicia vikander as an answer based on this context. in such workflows, the retrieved context serves as supporting evidence for the predicted answer. however, in order to assess the accuracy of this prediction, the user must already know that vikander is swedish, a piece of information that is not present in the provided context. this leads to a knowledge gap between the information that is required to answer the question and the information that is made available to the user. chat-based interfaces are becoming increasingly popular for information-seeking. in the case of factoid questions, these systems commonly surface retrieved or generated information that supports the provided answer. this supporting information is commonly seen through the lens of "information necessary for the underlying model to make the correct prediction." however, this does not necessarily translate to "information sufficient for users to assess the prediction." recent work in explainable nlp @xcite argues that to support users in decision-making and foster appropriate reliance, we should focus on providing users with information that aids them in assessing the correctness of model predictions. however, the effects of including such additional information on users' reliance on a qa model are not well studied. in this work, we design a wizard-of-oz experiment to study how users interact with a qa model when some of the information required to assess the correctness of the model prediction is missing. further, we investigate how users' reliance on model predictions and confidence in their judgments change when the requisite background-the information required to assess the correctness of the prediction-is provided to the user. the background serves as an extension of the model's input context and prediction where the goal is not to justify the prediction but rather to provide additional context that may establish a common ground between users and the model, a prerequisite for effective collaboration @xcite . further, we study whether the cognitive effort in consuming background mediates the effect of background on users' reliance on model predictions and confidence in their judgments. we employ highlights to surface the important parts of the context and the background to the user, making these easier to consume. we assess whether this highlighting of the relevant content helps users better rely on the model and calibrate their confidence. we control the sufficiency of the information shown to users by employing a multi-hop questionanswering dataset. we consider a qa model that predicts the answer to a complex question based on an input context that alone is insufficient to answer the question without additional background knowledge. using this model, we design a userstudy where we test how the user's agreement with the predicted answers changes when only part or all of the information is disclosed to the users. our findings are summarized as follows: ## research questions we conduct an online user-study foot_0 based on wizardof-oz experiments to examine how users' reliance on model predictions and confidence in their judgments change based on whether they are given the background information required to verify the predictions of a qa model. our study targets four key research questions: rq1: how do users interact with the model predictions in the absence of sufficient information to assess their correctness? human-ai decisionmaking often relies on humans to critically assess model predictions instead of simply agreeing with the model @xcite @xcite . users may frequently lack the necessary background to assess the correctness of the model. in such cases, we might expect users to either disagree with the model or have low confidence in their judgments. we study whether users indeed calibrate their confidence and reliance on the model when they likely lack enough information to assess the veracity of its predictions. rq2: does adding requisite background information allow users to calibrate reliance and confidence on model predictions? building on rq1, we study whether having access to information required to assess model predictions helps users make more informed decisions with higher confidence. we examine whether users are able to appropriately calibrate their reliance on models when provided with the relevant information to discern correct and incorrect model predictions. rq3: are users able to calibrate their reliance and confidence even when not all background provided is perfect? extending rq2, which explores users' reliance and confidence when provided with a background that enables perfect verification of model predictions, we study whether users are able to calibrate their reliance and confidence in model predictions when the background is sometimes, but not always, perfect. this reflects a more realistic scenario where users might be at times provided with background information that is ungermane: on topic, but not directly useful for answering the question. comparing ungermane and germane backgrounds also helps in unconfounding the effect of the presence of more information vs the effect of useful background. rq4: does highlighting important parts of context and background improve users' reliance and confidence calibration? background information, although potentially useful, also increases the information load and cognitive burden on the end user @xcite . thus, the effect of the presence of background is potentially mediated by the effort required in consuming the said background. to assess the mediating effect of the effort required to consume background, we introduce highlights in both the context and the background, identifying sentences that are crucial to answering the question. ## designing a wizard-of-oz system by using a multi-hop dataset we design a wizard-of-oz study with insufficient (no background) and sufficient (with background) information conditions. we employ a multi-hop question answering dataset, hotpotqa @xcite , which requires multiple pieces of evidence to reason the answer. we repurpose the hotpotqa dataset to control the information available to the model by only surfacing partial information to examine rq1 and providing the missing information to examine rq2. to approximate a near-realistic setting, we consider a subset of hotpotqa on which the model predictions remain unchanged when the background is added to the input to the model. this mimics a scenario where a qa model can make a prediction based on partial information using heuristic shortcuts @xcite @xcite or implicit knowledge acquired through pre-training @xcite , but users would lack sufficient background to assess its correctness. each question in the hotpotqa dataset is associated with two or more relevant context paragraphs: the context paragraph containing the gold-standard answer to the complex question and the other intermediate paragraphs providing background context corresponding to the required reasoning steps implicit in the question. we use the context paragraph with the gold-standard answer as the supporting context in the no background condition (rq1) and include the background context along with the supporting document in the with background condition (rq2). we only consider questions that involve exactly two documents to ensure that the relevant background is sufficient to assess the model prediction. for the mixed background condition (rq3), we sample a distractor document that is retrieved using the question, but that is not the required supporting document for answering the multi-hop question as the ungermane background. further, the hotpotqa dataset identifies the sentences in each paragraph that are essential supporting facts for answering the question. we use these supporting facts to highlight the context and the background information in the with background and highlights condition (rq4). see in realistic scenarios, the background information and highlights would be automatically retrieved or generated. however, for the purposes of this study, automating background and highlight extraction confounds the utility of the information presented to the end-user with the performance of the automated system. the proposed wizard-of-oz setting allows us more control over the background and highlights exposed to the end-user. this allows us to make stronger conclusions on whether and how users make use of additional information when it is known to be necessary and sufficient to assess model predictions in advance of future research on how to best extract and evaluate such information automatically. we choose a question answering model with the following desiderata: we want a model that (1) is a realistic representative of a standard qa model and (2) can make correct predictions based on its implicit background knowledge (as acquired from pre-training data), even when given questions and partial context, that is insufficient to answer the questions. based on this criteria, we select a 336m parameter bert model @xcite fine-tuned on the stanford question answering dataset (squad) @xcite , which consists of extractive question answering pairs over wikipedia context paragraphs. we employ a singlehop question answering model, as opposed to a model fine-tuned on a multi-hop qa task, as the underlying assumption is that the model is able to answer the complex question, even without the background information. the bert model finetuned on squad achieves an accuracy of ∼76% on the development set of hotpotqa both in the presence and absence of complete input context that is, in principle, needed to answer the question. ## related work despite the promise of ai systems to assist humans in decision-making tasks @xcite , recent studies show that human-ai collaboration exhibits a persistent failure mode: humans tend to over-rely on ai assistance rather than using their own insights and reasoning when performing tasks @xcite . this overreliance on ai systems can lead to the acceptance of incorrect suggestions @xcite @xcite tan, with background with background and highlights conditions 0.0 0.2 0.4 0.6 0.8 1.0 user agreement @xmath0.0 * @xmath1.0 * @xmath2.87 @xmath3.32 inappropriate agreement appropriate agreement with background with background and highlights conditions 3.0 3.5 4.0 4.5 5.0 user confidence @xmath4.0 * @xmath5.0 * @xmath6.75 @xmath7.23 inaccurate judgment accurate judgment figure 4: user agreement rate (mean and standard error) with correct vs. incorrect model predictions (left) and user confidence (mean and standard error) in their own accurate vs. inaccurate judgments (right) in the conditions with background, no highlight and the conditions with background, supporting facts highlighted. the rate of agreement is higher for correct model predictions (appropriate agreement) than for incorrect model predictions (inappropriate agreement), both with and without highlights. however, highlights do not help alleviate users' over-reliance on incorrect model prediction or overconfidence in their inaccurate judgments (rq4). foot_5 to address the issue of over-reliance, recent efforts have focused on the development of explainable ai methods, with the aim of enhancing users' comprehension of ai decisions and, in turn, promoting more appropriate reliance @xcite . however, an extensive body of humancentered evaluations of current explainability methods has revealed that explanations can inadvertently increase trust in ai models, exacerbating overreliance @xcite @xcite @xcite @xcite , particularly among non-expert users @xcite . recent research in the field of explainable nlp has advocated for explanations that prioritize users' decision-making @xcite . @xcite extensively review the current literature on the role of explanations in supporting human-ai decision-making, consolidating their findings into a compelling argument that explanations are most valuable when they empower users with the essential information required to critically evaluate the correctness of model predictions. this growing body of work underscores the pivotal function of explanations in facilitating users to scruti-nize the accuracy of model predictions. nonetheless, the concept of explainability is frequently framed as providing "information necessary for the underlying model to make the correct prediction" which may not inherently align with "information sufficient for users to assess the model prediction." this misalignment is particularly relevant in contemporary question-answering (qa) systems, powered by large language models pretrained on extensive datasets. these models can extrapolate knowledge beyond their immediate input context, drawing on implicit factual or commonsense knowledge @xcite and employing shortcuts to provide accurate answers @xcite @xcite . while this ability enhances the performance of qa systems, it introduces complexities in the realm of explainability and user interaction with ai decisions. our work extends the landscape of explainability and human-ai decision-making by moving beyond model explanations and exploring the implications of providing users with relevant background information as a way to enhance verifiability and foster appropriate reliance on ai, in the context of question-answering. openai, 2023), lamda @xcite , palm @xcite , and others, have propagated into informationseeking workflows of a general audience. a vast host of existing and ongoing work in nlp examines the deficiencies of these language models, ranging from hallucinated generations @xcite , lack of transparency @xcite , reasoning gaps @xcite , and more. however, close examination of the other piece of the puzzle-the user-is fairly sparse. literature in explainable nlp considers how nlp systems, and their underlying reasoning process, can be made available to the user to allow for better, more informed use of these systems and their predictions @xcite . extending this view of explainability, we considered the question of aiding users in their decision-making process, not by explaining the model prediction but by providing them with relevant pieces of information to assess the prediction. as argued by the contemporary work by @xcite , explanations in the form of the model's internal reasoning are rarely useful for supporting human decision-making @xcite @xcite . instead, explanations should aim to help the users assess the model prediction. our study takes a step in this direction by analyzing users' trust and reliance on qa models by providing information external to the model's input to enable verification of model predictions. we conducted our study in a multi-hop questionanswering setup. we designed our study such that some part of the information required to perform the reasoning is missing. we studied how users rely on model predictions in the absence of such information and how the presence of background affects their reliance and confidence in the qa systems. this background information, although not necessarily faithful to the model's reasoning process, may still be helpful to users in assessing the correctness of model predictions @xcite . such background explanations can take many shapes-implicit information encoded in the pre-trained models used for reasoning or external information required to fill in knowledge gaps. our study revealed that users' reliance on model predictions is fairly high (over 80%), much higher than the accuracy of the underlying qa system (76%), even without sufficient information to as-sess the correctness of the predictions (that is, without the relevant background). this indicates unwarranted trust @xcite in the model to some extent (rq1). we found that although background information did not affect reliance on correct model predictions, it significantly reduced over-reliance on incorrect model predictions (rq2). this indicates that even though users trust the model without sufficient information, they are able to catch model errors better when provided with the relevant information. we further find that users are able to calibrate their reliance on model prediction even when the background is sometimes, but not always, perfect (rq3). on the flip side, our study revealed that the addition of background also increased users' confidence in their judgments of the model predictions, both those that are accurate and, unfortunately, those that are inaccurate. this is especially concerning as even if users are better at the task when the background information is available to them, uncalibrated confidence might be detrimental in critical decision-making tasks. lastly, we found that surfacing relevant pieces to ease users' perusal of the background is not sufficient to alleviate overconfidence stemming from the background, even when the background and highlights pointed out the inaccuracies in users' judgments (rq4). our work highlights the utility and pitfalls of adding more information with model predictions. we studied user interaction with such insufficient background information in a wizard-of-oz experimental setting. more work is needed to study how such information can be gathered or generated automatically and made available to the user. research in nlp frequently adopts the reverse strategybuilding the systems first, followed by testing if the systems are useful to the end-users. our work aims to put humans front and center, studying their interactions with model predictions to inform the gaps in ai-assisted decision-making. our work highlights the issue of overconfidence stemming from more information, even when that information provides users with evidence to make accurate judgments. this indicates that explanations alone are not sufficient to garner appropriate trust and reliance. more efforts are needed to educate the lay audience to inspect explanations critically and diligently.