( QA ) ^2 : Question Answering with Questionable Assumptions (2023)

Abstract: Naturally occurring information-seeking questions often contain questionable assumptions—assumptions that are false or unverifiable. Questions containing questionable assumptions are challenging because they require a distinct answer strategy that deviates from typical answers for information-seeking questions. For instance, the question “When did Marie Curie discover Uranium?” cannot be answered as a typical “when” question without addressing the false assumption “Marie Curie discovered Uranium”. In this work, we propose (QA)2 (Question Answering with Questionable Assumptions), an open-domain evaluation dataset consisting of naturally occurring search engine queries that may or may not contain questionable assumptions. To be successful on (QA)2, systems must be able to detect questionable assumptions and also be able to produce adequate responses for both typical information-seeking questions and ones with questionable assumptions. Through human rater acceptability on end-to-end QA with (QA)2, we find that current models do struggle with handling questionable assumptions, leaving substantial headroom for progress.

## introduction naturally occurring information-seeking questions often contain assumptions that are false or unverifiable. for example, the question when did marie curie discover uranium? contains a false assumption that marie curie discovered uranium, and the question when is olive garden shutting down? assumes that the chain has plans to shut down, which cannot be verified as of may 2023. such questions containing false or unverifiable assumptions (questionable assumptions henceforth 1 ) pose a unique challenge to qa because the type of answer requested by the form of the question is often the wrong answer without further elaboration. for example, when did marie curie discover uranium? cannot be answered as a true when question (i.e., with a specific point in time or a timeframe); the false assumption must be addressed in some way for an answer to this question to be acceptable. while questionable assumptions anecdotally seem to remain challenging even for extremely capable general-purpose models (figure 1 ), few systematic evaluations have been conducted. @xcite offer one analysis, examining false or unverifiable presuppositions in natural questions (nq; @xcite . they conduct verification of automatically extracted presuppositions of nq questions against a single wikipedia article, following the closed-book setup of @xcite where the context for each question is restricted to a single article. hence, unverifiable presuppositions under their definition may not necessarily be unverifiable in the real world. to this end, we focus on questions containing questionable assumptions that are not verifiable in the open domain setting-a more naturalistic setup for information-seeking qa. we construct (qa) foot_0 , a dataset of naturally occurring information-seeking questions with questionable assumptions, based on frequent search engine queries (the same domain as nq). the answers and relevant assumptions of these questions are annotated by expert annotators and crowdworkers. this dataset is primarily intended to be an evaluation set with a goal of testing robustness to questionable assumptions, with an adaptation set (instead of a full training set) consisting of only a handful of examples that are intended for few-shot tuning or in-context learning. 2 using (qa) 2 , we test the capacity of several strong qa models to handle questionable assumptions. we find that large language models including flan-t5-xxl @xcite and recent gpt-3 and 3.5 @xcite releases show modest performance, with the best-performing models at 56% human-judged abstractive qa acceptability, and 64% and 72% on the binary classification subtasks of questionable assumption detection and verification, respectively. ## questions with questionable assumptions presuppositions before defining questionable assumptions, we briefly discuss an important relevant concept: presuppositions. 3 presuppositions are backgrounded meanings associated with a linguistic utterance, as opposed to directly asserted content of the utterance. information that is presupposed is taken for granted by all discourse participants, 4 and this presupposed content must be true in order for the utterance to be appropriate (note that various definitions of "inappropriate" exist: @xcite @xcite . for example, if a speaker uttered my cat is cute, it is taken for granted that the speaker owns a cat, and the speaker must in fact own a cat for this utterance to be appropriate. in the context of questions, the veracity of the presupposed content is important because it determines whether the question can be answered by the answer type requested by the form of the question, as discussed in section 1. this is because a matching answer type necessarily leads to an endorsement of the false presupposition @xcite -for example, any numeric answer (with no other qualification) to the question how old was mark zuckerberg when he founded google? endorses the false presupposition mark zuckerberg founded google. an appropriate answer must address the presupposition failure in the question, such as he did not found google. ## (qa) 2 dataset we introduce the question answering with question assumptions ((qa) 2 ) dataset, which consists of 602 naturally occurring, expert-annotated questions. half of the questions contain questionable assumptions, and the other half are typical information-seeking questions that do not contain any questionable assumptions. we set aside 32 examples (16 with questionable assumptions, 16 without) as an adaptation set that can be used for few-shot tuning or as in-context demonstrations. we intend (qa) 2 to be an evaluation dataset that serves as a robustness check for qa models, rather than a dataset to fully train or augment the training of qa models. ## results and discussion the results are summarized in ## related work questions with questionable assumptions have been a part of benchmark qa datasets for some time, most often categorized under unanswerable questions (e.g., @xcite ; also see @xcite . more recently, @xcite have pointed out that treating questions with questionable assumptions under the banner of unanswerable is unsatisfactory, and argued for an approach that makes use of presuppositions in explanations to better address their unanswerability. furthermore, they have shown that even under a closed-book setup, assumption verification is a challenging task. we extend their analysis to open-domain settings and reach a similar conclusion that qa with questionable assumptions is difficult. even when the difficulty of identifying assumptions is removed, factual verification itself is not completely solved, although recent improvements in general-purpose lms do seem to contribute to better verification. we view (qa) 2 as a robustness check for questionable assumptions for qa models, in the spirit of @xcite . the closest work in terms of motivation is the challenge set of @xcite , in that we intend (qa) 2 to be used as an evaluation-only dataset (with minimal adaptation if necessary). their challenge set is also a small-scale, evaluation-focused open-domain qa dataset like ours, but we substantially extend the coverage specifically for questions with questionable assumptions, given that their challenge set only contains 3 examples of presupposition failure. finally, concurrent work by @xcite introduces an open-domain qa dataset capturing a naturally occurring distribution of presupposition failures. their findings demonstrate that our observation about the difficulty of the subproblems (detection and verification) of handling questionable assumptions generalizes across distinct source domains of questions (search engine queries vs. reddit questions). ## conclusion we propose (qa) 2 , an evaluation set for qa systems for the robust handling of questionable assumptions in the open domain. (qa) 2 consists of a small adaptation set that can be used for fewshot tuning or as in-context demonstrations, and the main evaluation set. evaluating existing qa models on three tasks derived from our dataset reveals that questionable assumptions pose substantial challenges to current qa, echoing the findings of @xcite in an open-domain setting. moreover, we observe that better assumption verification does not necessarily translate into better endto-end qa performance, suggesting a failure in the actual deployment of knowledge already present in the models. this implies that the detection of assumptions underlying a question is a critical bottleneck, and indeed, detection performance better correlates with end-to-end qa. going forward, we hope that (qa) 2 serves as a useful checkpointing tool in the steps towards building the next generation of qa systems that are sensitive to implicit layers of meaning.