Yes, No or IDK : The Challenge of Unanswerable Yes/No Questions (2022)

Abstract: The Yes/No QA task (Clark et al., 2019) consists of “Yes” or “No” questions about a given context. However, in realistic scenarios, the information provided in the context is not always sufficient in order to answer the question. For example, given the context “She married a lawyer from New-York.”, we don’t know whether the answer to the question “Did she marry in New York?” is “Yes” or “No”. In this paper, we extend the Yes/No QA task, adding questions with an IDK answer, and show its considerable difficulty compared to the original 2-label task. For this purpose, we (i) enrich the BoolQ dataset (Clark et al., 2019) to include unanswerable questions and (ii) create out-of-domain test sets for the Yes/No/IDK QA task. We study the contribution of training on other Natural Language Understanding tasks. We focus in particular on Extractive QA (Rajpurkar et al., 2018) and Recognizing Textual Entailments (RTE; Dagan et al., 2013), analyzing the differences between 2 and 3 labels using the new data.

## introduction the ability to know whether a claim is true or false given a context is an important component of language comprehension. one main way to study this ability is the yes/no question answering task, for which a large-scale dataset, boolq, has been proposed @xcite . the boolq dataset includes paragraphs together with naturally occurring questions whose answer is either "yes" or "no". however, in realistic scenarios, the information needed to answer a yes/no question can be missing. for example, figure 1 shows yes/no questions given the context "jane, who is a native of los angeles, married a lawyer from nyc". while the questions in (a) and (b) can be answered respectively by "yes" and "no" (jane was born in los 1 all the datasets and the code are available at http:// cogcomp.org/page/publication_view/975 . angeles so she was not born in france), the question "did jane marry in nyc?" in (c) cannot be answered given the context. indeed, the ability to extract information from text only addresses one aspect of the expectations we have from a comprehension system. another main aspect concerns the ability to identify that a given information is not in the text, a witness of understanding in human comprehension. in this paper we extend the yes/no qa task to include idk questions and show the considerable difficulty of the extended task compared to the twolabel setting. for this purpose we first enrich the boolq dataset by including unanswerable questions, along with creating two other, out of domain, datasets, to test the ability to answer yes/no/idk questions in realistic scenarios (section 3). experimenting with a system based on bert-large @xcite , we show that the performance on the dev set drops from 72.88 f1 to 33.64 f1 when moving from a two-label to a three-label setting and observe similar results on out-of-domain test sets (section 4). we then explore the contribution of other natural language undertanding tasks to the performance, focusing on extractive qa @xcite , using the squad 2.0 dataset @xcite and recognizing textual entailments (rte, @xcite , using the mnli dataset 1075 @xcite . we obtain that, similar to what has been observed in the two-label setting @xcite , leveraging squad 2.0 does not improve the performance while training on mnli achieves better performance. as the improvement is limited, we separately analyze the transfer from mnli in the two-label and three-label settings. we conclude that current systems that achieve high performance on the boolq dataset are not adapted to the task of yes/no qa where unanswerable questions are involved. using the rte data is helpful but not sufficient for a good performance. in this paper, we provide new datasets that allow a more realistic evaluation of the yes/no task by (1) addressing unanswerable questions, which appear in real-world scenarios; (2) compiling test sets to evaluate the performance of the systems on domains that are different from the ones seen in training and finetuning. using the new data, we explore the performance of current systems on this task and show its considerable difficulty compared to its two-label version. the selective question answering task in out-ofdomain settings @xcite is related to the identification of unanswerable questions. however, it targets the ability of a system to refrain from answering in some of the cases in order to avoid errors in out-of-domain settings, independently from the presence of the answer in the context. ## conclusion in this paper we aim to allow a more realistic evaluation of the yes/no qa task. for this purpose, we (i) enrich the boolq dataset to include unanswerable questions and (ii) compile out-of-domain test sets. using the new data, we show the difficulty of current systems to address the task both by training directly on the task-specific data and by leveraging other nlu tasks.