Challenges in Information-Seeking QA : Unanswerable Questions and Paragraph Retrieval (2021)

Abstract: Recent pretrained language models “solved” many reading comprehension benchmarks, where questions are written with access to the evidence document. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging. We analyze why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on Natural Questions and TyDi QA. Our controlled experiments suggest two headrooms – paragraph selection and answerability prediction, i.e. whether the paired evidence document contains the answer to the query or not. When provided with a gold paragraph and knowing when to abstain from answering, existing models easily outperform a human annotator. However, predicting answerability itself remains challenging. We manually annotate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct per-category answerability prediction, revealing issues in the current dataset collection as well as task formulation. Together, our study points to avenues for future research in information-seeking question answering, both for dataset creation and model development. Our code and annotated data is publicly available at https://github.com/AkariAsai/unanswerable_qa .

## introduction addressing the information needs of users by answering their questions can serve a variety of practical applications. to answer such informationseeking queries -where users pose a question because they do not know the answer -in an unconstrained setting is challenging for annotators as they have to exhaustively search over the web. to reduce annotator burden, the task has been simplified as reading comprehension: annotators are tasked with finding an answer in a single document. recent pretrained language models surpassed estimated human performance @xcite in many reading comprehension datasets such as squad @xcite and coqa @xcite , where questions are posed with an answer in mind. however, those state-of-the-art models have difficulty answering information-seeking questions @xcite . in this work, we investigate what makes information-seeking question answering (qa) more challenging, focusing on the natural questions (nq; @xcite and tydi qa @xcite datasets. our experimental results from four different models over six languages on nq and tydi qa show that most of their headroom can be explained by two subproblems: selecting a paragraph that is relevant to a question and deciding whether the paragraph contains an answer. the datasets are annotated at the document level, with dozens of paragraphs, and finding the correct paragraph is nontrivial. when provided with a gold paragraph and an answer type (i.e., if the question is answerable or not), the performance improves significantly (up to 10% f1 in nq), surpassing that of a single human annotator. after identifying the importance of answerability prediction, in section 4, we compare a question only baseline, state-of-the-art qa models, and human agreement on this task. for comparison, we also evaluate unanswerability prediction in a reading comprehension dataset including unanswerable questions @xcite . while all datasets contain a large proportion of unanswerable questions (33-59%), they differ in how easily models can detect them. this motivates us to further investigate the source of unanswerability. to this end, we quantify the sources of unanswerability by annotating unanswerable questions from nq and tydi qa; we first classify unanswerable questions into six categories and then further annotate answers and alternative knowledge sources when we can find the answers to the unanswerable questions. despite the difficulty of annotating questions from the web and crowdsourcing bilingual speakers, we annotated 800 examples across six typologically diverse languages. our analysis shows that why questions are unanswerable differs based on the dataset or language. we conduct per-category answerability prediction on those annotated data, and found unanswerable questions from some categories are particularly hard to be identified. we provide a detailed analysis for alternative sources of an answer beyond wikipedia. grounded in our analysis, we suggest avenues for future research, both for dataset creation and model development based on the analysis. our contributions are summarized as follows: ## background and datasets we first define the terminology used in this paper. in this work, we focus on a reading comprehension setting, where reference documents (context) are given and thus retrieval is unnecessary, unlike open retrieval qa @xcite . information-seeking qa datasets contain questions written by a human who wants to know the answer but doesn't know it yet. in particular, nq is a collection of english google search engine queries (anonymized) and tydi qa is a collection of questions authored by native speakers of 11 languages. the answers are annotated post hoc by another annotator, who selects a paragraph with sufficient information to answer (long answer). alternatively, the annotator can select "unanswerable" data % answerable % avg. long only short un-ans # of p nq 14.7 35.2 50.1 131.3 tydi qa 5.4 34.8 59.9 41.1 squad 2.0 -66.6 33.4 1.0 if there is no answer on the page, or if the information required to answer the question is spread across more than one paragraph. if they have identified the long answer, then the annotators are tasked to choose the short answer, a span or set of spans within the chosen paragraph, if there is any. questions are collected independently from existing documents, so those datasets tend to have limited lexical overlap between questions and context, which is a common artifact in prior reading comprehension datasets @xcite . reading comprehension datasets such as squad @xcite , by contrast, have been created by asking annotators to write question and answer pairs based on a single provided paragraph. squad 2.0 @xcite includes unanswerable questions that are written by annotators who try to write confusing questions based on the single paragraph. as shown in in the following sections, we focus on informationseeking qa datasets, investigating how to improve the answer coverage of those questions that are currently labeled as unanswerable through several controlled experiments and manual analysis. ## qa performances with gold answer type and gold paragraph we quantify how the two aforementioned subproblems in information-seeking qa -deciding answer type, also referred to as answer calibrations @xcite or answerability prediction, and finding a paragraph containing the answer -affect the final qa performance. we conduct oracle analysis on existing models given two pieces of key information: gold paragraph and gold type. in the gold paragraph setting, we provide the long answer to limit the answer space. in the gold type setting, a model outputs the final answer following the gold answer type t @xmath0, long only, unanswerable}, which correspond to the questions with short answers, 3 questions with long answers only, and questions without any answers, respectively. this lifts the burden of answer calibration from the model. ## answerability prediction we first quantitatively analyze how easy it is to estimate answerability from the question alone, and then we test the state-of-the-art models' performance to see how well our complex models given question and the gold context perform on this task. we conduct the same experiments on squad 2.0, to highlight the unique challenges of the information-seeking queries. each example consists of a question q i , a list of paragraphs of an evidence document d i , and a list of answer annotations a i , which are aggregated into an answer type t @xmath1, long, unanswerable}. ## annotating unanswerability in this section, we conduct an in-depth analysis to answer the following questions: (i) where the unanswerability in information-seeking qa arises, (ii) whether we can answer those unanswerable questions when we have access to more knowledge sources beyond a single provided wikipedia article, and (iii) what kinds of questions remain unanswerable when these steps are taken. to this end, we annotate 800 unanswerable questions from nq and tydi qa across six languages. then, we conduct per-category performance analysis to determine the types of questions for which our models fail to predict answerability. ## related work analyzing unanswerable questions. there is prior work that seeks to understand unanswerability in reading comprehension datasets. @xcite analyzes unanswerable questions in squad 2.0 and two conversational reading comprehension datasets, namely coqa and quac, while we focus on information-seeking qa datasets to understand the potential dataset collection improvements and quantify the modeling challenges of the stateof-the-art qa models. @xcite compare unanswerable factors between nq and a qa dataset on privacy policies. this work primarily focuses on a privacy qa, which leads to differences of the categorizations of the unanswerable questions. we search alternative knowledge sources as well as the answers to understand how we could improve answer coverage from dataset creation perspective and connect the annotation results with answerability prediction experiments for modeling improvements. answer calibrations. answerability prediction can bring practical values, when errors are expensive but abstaining from it is less so @xcite . while predicting answerability has been studied in squad 2.0 @xcite , the unanswerability in squad 2.0 has different characteristics from unanswerability in information-seeking qa as we discussed above. to handle unanswerable questions in informationseeking qa, models either adopt threshold based answerable verification @xcite , or introduce an extra layer to classify unanswerablity and training the model jointly @xcite . @xcite observes the difficulty of answer calibrations, especially under domain shift. ## conclusion we provide the first in-depth analysis on information-seeking qa datasets to inspect where unanswerability arises and quantify the remaining modeling challenges. our controlled experiments identifies two remaining headrooms, answerability prediction and paragraph selection. observing a large percentage of questions are unanswerable, we provide manual analysis studying why questions are unanswerable and make suggestions to improve answer coverage: (1) going beyond wikipedia textual information as the only source of information, (2) addressing ambiguous queries instead of simply marking and leaving the questions as is, (3) enable accessing multiple documents and introducing abstractive answers for non-factoid questions. together, our work shed light on future work for information-seeking qa, both for modeling and dataset design.