Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering (2021)

Abstract: Many Question-Answering (QA) datasets contain unanswerable questions, but their treatment in QA systems remains primitive. Our analysis of the Natural Questions (Kwiatkowski et al. 2019) dataset reveals that a substantial portion of unanswerable questions (~21%) can be explained based on the presence of unverifiable presuppositions. Through a user preference study, we demonstrate that the oracle behavior of our proposed system—which provides responses based on presupposition failure—is preferred over the oracle behavior of existing QA systems. Then, we present a novel framework for implementing such a system in three steps: presupposition generation, presupposition verification, and explanation generation, reporting progress on each. Finally, we show that a simple modification of adding presuppositions and their verifiability to the input of a competitive end-to-end QA system yields modest gains in QA performance and unanswerability detection, demonstrating the promise of our approach.

## introduction many question-answering (qa) datasets including natural questions (nq) @xcite and squad 2.0 @xcite contain questions that are unanswerable. while unanswerable questions constitute a large part of existing qa datasets (e.g., 51% of nq, 36% of squad 2.0), their treatment remains primitive. that is, (closed-book) qa systems label these questions as unanswerable without detailing why, as in (1): unanswerability in qa arises due to a multitude of reasons including retrieval failure and malformed questions @xcite . we focus on a subset of unanswerable questions-namely, questions containing failed presuppositions (background assumptions that need to be satisfied). questions containing failed presuppositions do not receive satisfactory treatment in current qa. under a setup that allows for unanswerable as an answer (as in several closed-book qa systems; figure 1 , left), the best case scenario is that the system correctly identifies that a question is unanswerable and gives a generic, unsatisfactory response as in @xcite . under a setup that does not allow for unanswerable (e.g., open-domain qa), a system's attempt to answer these questions results in an inaccurate accommodation of false presuppositions. for example, google answers the question which linguist invented the lightbulb? with thomas edison, and bing answers the question when did marie curie discover uranium? with 1896 (retrieved jan 2021). these answers are clearly inappropriate, because answering these questions with any name or year endorses the false presuppositions some linguist invented the lightbulb and marie curie discovered uranium. failures of this kind are extremely noticeable and have recently been highlighted by social media @xcite , showing an outsized importance regardless of their effect on benchmark metrics. we propose a system that takes presuppositions into consideration through the following steps (figure 1, right): our contribution can be summarized as follows: ## presuppositions presuppositions are implicit assumptions of utterances that interlocutors take for granted. for example, if i uttered the sentence i love my hedgehog, presuppositions are often associated with specific words or syntactic constructions ('triggers'). we compiled an initial list of presupposition triggers based on @xcite and van der sandt (1992), foot_0 and selected the following triggers based on their frequency in nq (» means 'presupposes'): presuppositions and unanswerability. questions containing failed presuppositions are often treated as unanswerable in qa datasets. an example is the question what is the stock symbol for mars candy? from nq. this question is not answerable with any description of a stock symbol (that is, an answer to the what question), because mars is not a publicly traded company and thus does not have a stock symbol. a better response would be to point out the presupposition failure, as in there is no stock symbol for mars candy. however, statements about negative factuality are rarely explicitly stated, possibly due to reporting bias @xcite . therefore, under an extractive qa setup as in nq where the answers are spans from an answer source (e.g., a wikipedia article), it is likely that such questions will be unanswerable. our proposal is based on the observation that the denial of a failed presupposition (¬p) can be used to explain the unanswerability of questions (q) containing failed presuppositions (p), as in (2). an answer that refers to the presupposition, such as ¬p, would be more informative compared to both unanswerable (1-b) and an extractive answer from documents that are topically relevant but do not mention the false presupposition. ## analysis of unanswerable questions first, to quantify the role of presupposition failure in qa, two of the authors analyzed 100 randomly selected unanswerable wh-questions in the nq development set. 2 the annotators labeled each question as presupposition failure or not presupposition failure, depending on whether its unanswerability could be explained by the presence of an unverifiable presupposition with respect to the associated document. if the unanswerability could not be explained in terms of presupposition failure, the annotators provided a reasoning. the cohen's κ for inter-annotator agreement was 0.586. we found that 30% of the analyzed questions could be explained by the presence of an unverifiable presupposition in the question, considering only the cases where both annotators were in agreement (see ## user study with oracle explanation our hypothesis is that statements explicitly referring to failed presuppositions can better foot_3 speak to the unanswerability of corresponding questions. to test our hypothesis, we conducted a side-by-side comparison of the oracle output of our proposed system and the oracle output of existing (closedbook) qa systems for unanswerable questions. we included two additional systems for comparison; the four system outputs compared are described below (see ## model components given that presupposition failure accounts for a substantial proportion of unanswerable questions (section 3) and our proposed form of explanations is useful (section 4), how can we build a qa system that offers such explanations? we decompose this task into three smaller sub-tasks: presupposition generation, presupposition verification, and explanation generation. then, we present progress towards each subproblem using nq. 7 we use a templatic approach for the first and last steps. the second step involves verification of the generated presuppositions of the question against an answer source, for which we test four different strategies: zero-shot transfer from natural language inference (nli), an nli model finetuned on verification, zero-shot transfer from fact verification, and a rulebased/nli hybrid model. since we used nq, our models assume a closed-book setup with a single document as the source of verification. ## related work while presuppositions are an active topic of research in theoretical and experimental linguistics @xcite @xcite , comparatively less attention has been given to presuppositions in nlp (but see @xcite and @xcite ). more recently, @xcite discuss automatically detecting presuppositions, focusing on adverbial triggers (e.g., too, also...), which we excluded due to their infrequency in nq. @xcite investigate whether inferences triggered by presuppositions and implicatures are captured well by nli models, finding mixed results. regarding unanswerable questions, their importance in qa (and therefore their inclusion in benchmarks) has been argued by works such as @xcite and @xcite . the analysis portion of our work is similar in motivation to unanswerability analyses in @xcite and @xcite -to better understand the causes of unanswerability in qa. @xcite ; @xcite ; @xcite consider answer-ability detection as a core motivation of their modeling approaches and propose components such as independent no-answer losses, answer verification, and answerability scores for answer spans. our work is most similar to @xcite in proposing to consider implicit assumptions of questions. furthermore, our work is complementary to qa explanation efforts like @xcite that only consider answerable questions. finally, abstractive qa systems (e.g., @xcite were not considered in this work, but their application to presupposition-based explanation generation could be an avenue for future work. ## conclusion through an nq dataset analysis and a user preference study, we demonstrated that a significant portion of unanswerable questions can be answered more effectively by calling out unverifiable presuppositions. to build models that provide such an answer, we proposed a novel framework that decomposes the task into subtasks that can be connected to existing problems in nlp: presupposition identification (parsing and text generation), presupposition verification (textual inference and fact verification), and explanation generation (text generation). we observed that presupposition verification, especially, is a challenging problem. a combination of a competitive nli model, finer-tuning and rule-based hybrid inference gave substantial gains over the baseline, but was still short of a fully satisfactory solution. as a by-product, we showed that verified presuppositions can modestly improve the performance of an end-to-end qa model. in the future, we plan to build on this work by proposing qa systems that are more robust and cooperative. for instance, different types of presupposition failures could be addressed by more fluid answer strategies-e.g., violation of uniqueness presuppositions may be better handled by providing all possible answers, rather than stating that the uniqueness presupposition was violated.