Do We Know What We Don’t Know? Studying Unanswerable Questions beyond SQ u AD 2.0 (2021)

Abstract: Understanding when a text snippet does not provide a sought after information is an essential part of natural language utnderstanding. Recent work (SQuAD 2.0; Rajpurkar et al., 2018) has attempted to make some progress in this direction by enriching the SQuAD dataset for the Extractive QA task with unanswerable questions. However, as we show, the performance of a top system trained on SQuAD 2.0 drops considerably in out-of-domain scenarios, limiting its use in practical situations. In order to study this we build an out-of-domain corpus, focusing on simple event-based questions and distinguish between two types of IDK questions: competitive questions, where the context includes an entity of the same type as the expected answer, and simpler, non-competitive questions where there is no entity of the same type in the context. We find that SQuAD 2.0-based models fail even in the case of the simpler questions. We then analyze the similarities and differences between the IDK phenomenon in Extractive QA and the Recognizing Textual Entailments task (RTE; Dagan et al., 2013) and investigate the extent to which the latter can be used to improve the performance.

## introduction extractive question answering (extractive qa) has attracted a lot of interest in recent years with the creation of large-scale datasets @xcite and has seen large improvements with the use of contextualized language models such as bert @xcite and roberta @xcite . however, the ability to extract information from text only addresses one aspect of the expectations we have from a comprehension system. another main aspect concerns the ability to identify that a given information is not in the text, a witness of understanding in human comprehension. the ability to answer "idk" allows one to address more realistic situations in reading comprehension, both as an end task and as an intermediary step for other nlp applications, such as qa-based event extraction @xcite or qa-based summarization evaluation @xcite . to begin addressing this important phenomenon, @xcite added unanswerable questions to squad 1.1 @xcite , providing a useful resource for identifying idk cases in the extactive qa case (squad 2.0). however, as we show, the performance of a top system trained on squad 2.0 considerably drops on out-of-domain simple questions. in this paper, we show that squad 2.0 alone is not sufficient to address idk questions in practical situations. for this purpose, we introduce a new evaluation dataset of very simple questions on single-sentence contexts that we compile based on an event extraction corpus (ace, @xcite . in particular, we propose to separately evaluate the performance of a qa model on two types of idk questions: (i) cases where the context includes an entity of the same type as the expected answer such as q1 in figure 1 where "new york" is a location appearing in the context 1. we call this type of questions competitive idk questions and (ii) cases where the context includes no entity of the same type as the expected answer such as q2 in figure 1 where the question expects a time mention while the context does not include time. evaluating on the new dataset, we find that a top squad 2.0 model obtains low scores. even in the case of the simpler, non-competitive idk questions, only reaching 28.46 f1 (section 4). we then explore the use of another natural language understanding (nlu) task that also includes an idk option. we focus on the recognizing textual entailments (rte, @xcite task and find that leveraging it considerably improves the results in the case of non-competitive idk questions but is not sufficient for reaching a good performance in the competitive idk cases. ## related work unanswerable questions have been first addressed in the context of the annual trec competition for open-domain qa @xcite , where the subtask of span extraction has some similarities with extractive qa, although in the former the goal is to answer a question from a large collection of documents. in extractive qa, a system being able to answer "i don't know" has been proposed by @xcite in the framework of the relation extraction task which is formulated in qa terms. another example is the use of qa systems for event extraction, as recently proposed by @xcite who modified a bert-based qa system to predict an argument role in a clozed test format. in this work we evaluate our extract qa systems on eventbased questions questions derived from the ace corpus @xcite , focusing on the location and time argument types. we differ from @xcite by experimenting in an out-of-domain setting and by preserving the qa format. we also distinguish between easier idk cases when there is no entities of the argument type expected by the question and harder cases where an entity of the same type appears in the sentence (see section 3). @xcite enriched the squad 1.1 corpus by including unanswerable questions for the same paragraphs via crowdsourcing, resulting in squad 2.0, that we are using in this paper for the extractive qa task. we show that training on squad 2.0 is not sufficient to address idk in out-of-domain settings (focusing on simple, eventbased questions) and that the rte data can be useful to address a particular type of idk questions. @xcite experimented on squad 2.0 using the bidaf-no-answer (bna) model proposed by @xcite and the documen-tqa no-answer (docqa) model from @xcite . these models learn to predict the probability that a question is unanswerable, in addition to a distribution over answer choices. this also holds in the bert implementation we use here. an alternative way for training and prediction in the case of unanswerable questions has been advanced by @xcite who proposed to first predict whether there is an answer in the context. tan et al. (2018) also used a predict+validate approach, which is also explored by @xcite who added a separately trained answer verifier for no-answer detection. we do not modify the training and prediction used in the bert paper approach but rather explore the performance in outof-domain scenarios as well as the use of rte to improve the performance. the selective question answering task in out-ofdomain settings @xcite ) is related to the identification of unanswerable questions. however, it targets the ability of a system to refrain from answering in some of the cases in order to avoid errors in out-of-domain settings, independently from the presence of the answer in the context. the authors show that selective prediction methods do not identify unanswerable questions, suggesting that an explicit labeling of idk in the training data is necessary in our case. in the rte task @xcite , the idk option is instantiated by the "neutral" category. in some of the rte works @xcite , "contradiction" and "neutral" are unified in a "non-entailed" joint category. demszky et al. ( 2018 ) proposed a conversion of extractive qa datasets to 2-label rte format. we instead leverage the rte task for extractive qa via additional pretraining and compare between the presence and the absence of an idk label in the rte data (see section 5). ## test datasets we leverage the ace event extraction @xcite 2 dataset to derive questions asking about the argument that participates in an event, given the trigger. this allow us to experiment on idk answers that result from the fact that one of the event arguments is missing. for this purpose, we first select sentence fragments that include a location or a time mention according to the ace annotation. to generate the wh-questions, we automatically generate candidate questions based on the event structure by asking both where and when did t happen, where t is the event trigger. the answer is labeled "i don't know" when the entity type is missing. to generate additional idk questions, we select more sentences from the ace dataset that do not necessarily include time/location mentions. all the questions are manually validated to ensure both grammatical and logical correctness. we compile two types of idk questions. the first concerns 2 https://catalog.ldc.upenn.edu/ ldc2006t06 idk questions where there is an entity in the context that has the same type as the expected answer; this creates competition and makes the prediction harder (compet. idk). for creating this type of examples, we manually modify the context sentence to add an entity that has the same type as the expected answer. we choose the entity randomly from a set of time/locations entities appearing in the dataset. for example, given the context "she went to mexico after she lost her seat in the 1997 election", a compet. idk question is "where is the loss?". the second type of questions (non-compet. idk) concerns cases where there is no candidate of the same type in the sentence. in this case too, we use manual modifications. for example, given the context "he was arrested for his crimes", a noncompet. idk question is "when was the arrest?". the resulting corpus, called ace-whqa includes three sub-corpora: "has answer", "compet. idk" and "non-compet. idk" with 238, 250 and 246 examples respectively. more examples are presented in figure 3 . ## training on squad 2.0 is not sufficient we finetune the bert-large-cased representation on the squad 2.0 dataset and evaluate on ace-whqa. 3 we also report the score on the squad 2.0 dev set (80.96 f1). the evaluation on the ace-whqa dataset is presented in the first column of ## exploring the use of the rte task similarities and differences the recognizing textual entailment (rte) task @xcite consists of classifying a sentence pair composed of a premise p and a hypothesis h into three classes, according to the relation between the two sentences: "entailment", "contradiction" and "neutral", which corresponds to the idk option. a common idea, there are also considerable differences. first, the format of a wh-question is missing some content which is already present in a corresponding rte hypothesis; for example, the location entity in a "where" question. therefore, a wh-question cannot be directly converted to an rte hypothesis, independently from the context. this format difference is also related to the fact that rte can be seen as a classification task, while extract qa involves span extraction. second, a conversion between the formats will not always preserve the idk label, as illustrated in h1 and q1 in figure 2 . in particular, an idk instance in extractive qa can correspond also to a "contradiction" in rte. finally, while short paragraphs are used in squad 2.0, the premises in the mnli corpus for the rte task are single sentences. while this is not inherent in the definition of the respective tasks, the available datasets impact the models used by the community. experimental setting here we consider extractive qa as a target task. rte is the auxiliary task. our baseline system consists in the bert-large-cased representation fine-tuned on the squad 2.0 train corpus. we experiment with the following systems: (i) mnli + squad 2.0 where we first finetune bert-large on mnli, remove the classification layer and further finetune on squad 2.0. (ii) c(mnli) + squad 2.0: 2-label pretraining on mnli, where we only consider the "contradiction" and "non-contradiction" classes. 4 in all cases we evaluate the system on the squad 2.0 dev as well as the three sub-corpora of ace-whqa introduced in section 3: questions that have an answer (has answer), questions that do not have an answer but there is an entity in the sentence of the same type as the expected answer (competitive idk) and questions that do not have an answer and there is no entity of the same type (non-competitive. idk). for training on mnli with the bert-large-cased representation, we use batch size of 32 and 3 training epochs. we fine tune over three possible learning rate values: 2e-5, 3e-5 and 5e-5. for training on squad 2.0, we use the same hyperparameters as in section 4. for each of the training settings, we choose the hyperparameter combination that maximizes the accuracy for the target task on the squad 2.0 dev set. ## conclusion we studied the idk phenomenon, which is essential in language comprehension, in extractive qa, going beyond the evaluation on squad 2.0. we designed an out-of-domain evaluation dataset, composed of two main types of idk questions. we show that idk in extractive qa is a major challenge for current nlp systems. we further explore the use of the rte dataset and observe a considerable improvement in the case of non-competitive questions. future work concerns the use of additional natural language understanding tasks and 5 one-sided t-test, @xmath0.05 data for idk and the improvement of the ability to face adversarial idk questions.