Characterizing LLM Abstention Behavior in Science QA with Context Perturbations (2024)

Abstract: The correct model response in the face of uncertainty is to abstain from answering a question so as not to mislead the user. In this work, we study the ability of LLMs to abstain from answering context-dependent science questions when provided insufficient or incorrect context. We probe model sensitivity in several settings: removing gold context, replacing gold context with irrelevant context, and providing additional context beyond what is given. In experiments on four QA datasets with six LLMs, we show that performance varies greatly across models, across the type of context provided, and also by question type; in particular, many LLMs seem unable to abstain from answering boolean questions using standard QA prompts. Our analysis also highlights the unexpected impact of abstention performance on QA task accuracy. Counter-intuitively, in some settings, replacing gold context with irrelevant context or adding irrelevant context to gold context can improve abstention performance in a way that results in improvements in task performance. Our results imply that changes are needed in QA dataset design and evaluation to more effectively assess the correctness and downstream impacts of model abstention.

## introduction question-answering (qa) in scientific settings is often defined as a context-dependent task, where models answer questions based on provided context or relevant context it identifies. when the provided or retrieved context is itself unreliable or inconsistent, however, the correct model response should be to abstain from answering. prior work has studied the abstention capabilities of llms @xcite and proposed approaches to improve abstention when presented with insufficient context @xcite figure 1 : our framework to probe the context sensitivity of llms for science qa. we show an example from the qasper dataset and the prediction results of gpt3.5 under different context perturbation settings. while the model fails to abstain when context is removed, it abstains appropriately when random context is provided. @xcite , but these approaches focus on general domain question-answering (e.g., squad2) against a narrow range of models (e.g., chatgpt). as we show in our experiments ( §5.2), general domain settings produce highly divergent results from scientific qa settings, and different model architectures also exhibit differing abilities to abstain. further, as new llms continue to be developed and released, we need an extensible way to measure their ability to abstain from answering questions when provided context is irrelevant or uncertain. to address these challenges, we introduce a framework to assess llm abstention in science question-answering by removing, replacing, and augmenting provided contexts to control the answerability of questions. using this framework, we probe abstention performance of six llms (llama2, vicuna, flan-t5, mistral, gpt3.5, and gpt4o) on one general domain qa dataset (squad2) and three scientific qa datasets (pub-medqa, bioasq, qasper). several of these datasets include unanswerable questions by design, allowing us to analyze interactions between question answerability and noisy context. our work aims to answer the following: (i) how well do llms abstain from answering questions when the correct context is not provided? (ii) how do context perturbations impact task performance and abstention performance? and (iii) how does question type impact task performance and abstention performance? we summarize our contributions below: ## related work prompting llms for science qa the few-shot capability of llms have been applied successfully to knowledge-intensive tasks like question answering @xcite @xcite . prompting strategies such as chainof-thought @xcite , least-to-most @xcite , and others @xcite significantly improve llms' zero or few-shot abilities on diverse qa benchmarks in the general @xcite and scientific @xcite domains. @xcite adopt retrieve-than-read on the qasper dataset @xcite , showing that current retrievers are the main bottleneck and readers (llms) are already performing at the human level. inspired by this finding, we study how irrelevant or incorrect contexts (mimicking retrieval errors) can impact llm performance on context-intensive qa tasks. liao et al. (2022) introduced a prompt-tuning strategy to enhance performance on unanswerable questions by mapping questions to specific templates. beyond prompting strategies, @xcite demonstrated that sampling methods are more effective at modulating uncertainty compared to relying on the model's likelihood. other approaches, such as data augmentation, have been used to steer models toward abstention @xcite . @xcite provided a detailed analysis of language models' abstention capabilities, identifying paragraph selection and answerability prediction as key areas for improvement. recent work has introduced new datasets to investigate whether llms are aware of their limitations @xcite , while @xcite revealed that differences in llm hidden states can delineate the boundary between known and unknown information. abstention is also closely linked to out-of-distribution (ood) detection and selective generation. ood scores derived from conditional language models @xcite or task representations @xcite have been employed to abstain from generating low-quality outputs. in this work, we systematically characterize llm abstention capabilities in context-dependent science question-answering tasks. ## datasets we conduct experiments on four qa datasets: squad2 @xcite , pubmedqa @xcite , bioasq @xcite , and qasper @xcite . dataset statistics are provided in unanswerable questions three of these datasets contain unanswerable questions (proportions in ## framework we describe our framework in terms of prompting strategies, context perturbation methods, the choice of models and handling of model output, and evaluation metrics. ## results we report baseline task performance ( ## error analysis we perform error analysis on a consistently high performing model in our analysis, gpt3.5, and summarize main reasons why the model may fail to abstain. we sample 20 failure cases each from among answerable questions and unanswerable questions, for both gold context and no context settings. we present our findings by dataset. squad2 answerable questions are either general (e.g., "what german poet was descended from huguenots?") or could be answered without context (e.g., "what types of pumps are typically used in industrial boilers?"); the model tends to answer these regardless of whether context is provided. for unanswerable questions, half of failure cases are very open-ended like "what effect do technologies and resources generate?" and removing context will cause gpt3.5 to answer rather than abstain. another 20% of questions contain popular entities but have no correct answer, e.g., "what lake contains part of the rhine falls?" misleads the model squad2 qasper context computational complexity theory is a branch of the theory of computation in theoretical computer science that focuses on classifying computational problems according to their inherent difficulty, and relating those classes to each other... table tabref35 show the comparisons between tree and sequential based methods. we can see that, if we don't deploy cnn, simple tree lstm yields better result than traditional lstm, but worse than bidirectional lstm... prompt q: what is a manual application of mathematical steps? q: do they separately evaluate performance of their learned representations (before forwarding them to the cnn layer)? given context unanswerable no ✗ no context calculation ✗ yes ✗ random context unanswerable unanswerable noisy context unanswerable unanswerable ✗ ground truth unanswerable yes pubmedqa gpt3.5 is strongly inclined to answer "yes" for all failure cases among both answerable and unanswerable questions under the no context setting, perhaps due to the phrasing of automatically constructed questions. bioasq around 90% of failure cases we sampled were answered correctly by gpt3.5 without context, including both "yes" and "no" answers. another 10% of cases are affected by the model's tendency to hallucinate "yes", which is consistent with cases in pubmedqa. we manually substitute antonyms for the word in brackets on failure cases such as "is apelin usually [decreased] in diabetes?", "does vesatolimod [inhibit] tlr7?" etc., and gpt3.5 still always answers "yes". interestingly, for these factoid questions, substituting gold with random context skews the model strongly towards answering "no". these two types of hallucination behavior require further investigation. our observations also raise concerns for assessing llm performance using qa benchmarks containing boolean questions. qasper for answerable questions, 50% of failures seem to be caused by gpt3.5's ignorance of the ambiguity resulting from anaphora; for instance, gpt3.5 should not be able to resolve terms in questions such as "they","this study", and "the models" without context, yet the model answers these questions anyway under the no context setting. around 30% of questions are very general, such as "what is a soft label?" and "why is ner for tweets more challenging as the number of entities increases?"these questions lead the model to answer rather than abstain. for unanswerable questions, the distribution of failure reasons is similar. ## discussion & conclusion our study investigates the impacts of context removal and perturbation on llm performance on scientific qa. while lack of correct context should result in a model abstaining from answering a question, our results highlight that there are inconsistent patterns of model behavior based on model pretraining paradigms, question types, and the contextdependence of various qa datasets. for example, perturbing given context by replacing with random irrelevant context or adding noisy context would be expected to reduce task performance, but in some cases, the improvements in abstention that result can negate any reductions in task performance and potentially lead to gains (e.g., vicuna and gpt4o on pubmedqa, llama2 and vicuna on qasper). additionally, we find that abstention varies greatly by question type, with all models in our experiments struggling to abstain on boolean questions. future work for enhancing models' abstention ability could investigate the impact of (i) different prompting strategies: since model abstention ability is sensitive to constrained prompts (such as boolean instructions) as shown in section 5.2, how to select good prompting strategies to produce the best trade-off between task performance and abstention performance remains an interesting problem; our results have direct implications for dataset curators and model developers. benchmark qa datasets with mixtures of unanswerable and answerable questions were designed to facilitate assessment of abstention ability, yet our experiments show conflation and a lack of clear assessment of either abstention ability or performance accuracy. in other words, while unanswerable questions were motivated by the need to measure abstention, when aggregated with other questions during task performance evaluation, this distinction is obscured. additionally, unanswerable questions in different datasets measure different phenomena, e.g., in squad2, they measure model sensitivity to irrelevant context passages, while in qasper, they indicate that a question is unanswerable based on the given document, neither of which clearly map to the notion of abstaining under insufficient information. given the importance of assessing model abstention capabilities, separating task and abstention assessment, coupled with changes in dataset construction, is needed to better align model performance on these tasks with human expectations. beyond dataset curation, other variables-such as the task defined by the dataset, the types of questions posed, the architecture of models, instruction tuning techniques, in-context learning, and domain-specific pretraining-can affect a model's ability to effectively abstain and provide accurate answers. in this work, we attempt to disentagle some of these factors, though not all. future studies could explore the extent to which in-domain pretraining and abstention-specific instruction tuning techniques impact model abstention performance. on the other hand, our results also hold implications for downstream builders/users of interactive systems who rely on llms as question-answering tools. we show that in some cases (e.g., boolean questions), llms exhibit little to no ability to abstain, even for highly context-dependent questions such as those in qasper. system designers should be cautious of these limitations and when/how to infer model uncertainty and communicate this uncertainty to users. for communicating abstention capabilities to users, confidence scores or similar indicators that reflect model certainty, accompanying explanations, or interpretation guidance can empower users to make more informed decisions.