IIRC : A Dataset of Incomplete Information Reading Comprehension Questions (2020)

Abstract: Humans often have to read multiple documents to address their information needs. However, most existing reading comprehension (RC) tasks only focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a potential lack of sufficient information and locating sources for that information. To fill this gap, we present a dataset, IIRC, with more than 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. The questions were written by crowd workers who did not have access to any of the linked documents, leading to questions that have little lexical overlap with the contexts where the answers appear. This process also gave many questions without answers, and those that require discrete reasoning, increasing the difficulty of the task. We follow recent modeling work on various reading comprehension datasets to construct a baseline model for this dataset, finding that it achieves 31.1% F1 on this task, while estimated human performance is 88.4%. The dataset, code for the baseline system, and a leaderboard can be found at https://allennlp.org/iirc .

## introduction humans often read text with the goal of obtaining information. given that a single document is unlikely to contain all the information a reader might need, the reading process frequently involves identifying the information present in the given document, and what is missing, followed by locating a different source that could potentially contain the missing information. most recent read-figure 1 : an example from iirc. at the top is a context paragraph which provides only partial information required to answer the question. the bold spans in the context indicate links to other wikipedia pages. the colored boxes below the question show snippets from four of these pages that provide the missing information for answering the question. the answer is the underlined span. on the other hand, open-domain question answering tasks @xcite @xcite present a model with a ques-tion by itself, requiring the model to retrieve relevant information from some corpus. however, this approach loses grounding in a particular passage of text, and it has so far been challenging to collect diverse, complex question in this setting. alternatively, complex questions grounded in context can be converted to open-domain or incomplete-information qa datasets such as hot-potqa @xcite . however, they do not capture the information-seeking questions that arise from reading a single document with partial information @xcite . we present a new dataset of incomplete information reading comprehension questions, iirc, to address both of these limitations. iirc is a crowdsourced dataset of 13441 questions over 5698 paragraphs from english wikipedia, with most of the questions requiring information from one or more documents hyperlinked to the associated paragraphs, in addition to the original paragraphs themselves. our crowdsourcing process (section 2) ensures the questions are naturally information-seeking by decoupling question and answer collection pipelines. crowd workers are instructed to ask follow-up questions after reading a paragraph, giving links to pages where they would expect to find the answer. this process results in questions like the one shown in figure 1 . as illustrated by the example, this setup results in questions requiring complex reasoning, with an estimated 39% of the questions in iirc requiring discrete reasoning. moreover, 30% of the questions in iirc require more than one linked document in addition to the original paragraph and 30% of them are unanswerable even given the additional information. when present, the answers are either extracted spans, boolean, or values resulting from numerical operations. to evaluate the quality of the data, we run experiments with a modified version of @xcite , a state-of-the-art model from drop @xcite , chosen because a significant portion of questions in iirc require numerical reasoning similar to that found in drop. because drop uses only a single paragraph of context, we add a two-stage pipeline to retrieve necessary context for the model from the linked articles. the pipeline first identifies which links are pertinent, and then selects the most relevant passage from each of those links, concatenating them to serve as input for the model (section 3). this baseline achieves an f 1 score of 31.1% on iirc, while the estimated human performance is 88.4% f 1 . even giving the model oracle pipeline components results in a performance of only 70.3%. taken together, these results show that substantial modeling is needed both to identify and retrieve missing information, and to combine the retrieved information to answer the question (section 4). we additionally perform qualitative analysis of the data, and find that the errors of the baseline model are evenly split between retrieving incorrect information, identifying unanswerable questions, and successfully reasoning over the retrieved information. by construction, all examples in iirc require identifying missing information. even though current model performance is quite low, a model trained on this data could theoretically leverage that fact to achieve artificially high performance on test data, because it does not have to first determine whether more information is needed. to account for this issue, we additionally sample questions from squad 2.0 @xcite and drop @xcite , which have similar question language to what is in iirc, putting forward this kind of combined evaluation as a challenging benchmark for the community. predictably, our baseline model performs substantially worse in this setting, reaching only 28% f 1 on the iirc portion of this combined evaluation (section 5). ## building iirc we used wikipedia to build iirc and relied on the fact that entities in wikipedia articles are linked to other articles about those entities, providing more information about them. our goals were to build a dataset with naturally information-seeking questions anchored in paragraphs with incomplete information, such that identifying the location of missing information is non-trivial, and answering the questions would require complex cross-document reasoning. we ensured that the questions are informationseeking by separating question and answer collection processes, and by not providing the question writers access to the contexts where the answers occur. this process also ensured that we get questions that have minimal lexical overlap with the answer contexts. we used wikipedia paragraphs with many outgoing links to increase the difficulty of identifying the articles that provide the missing information. to ensure complex cross-document retrieval: linked context only was tip o'neill working as a politician the year o'donnell provided testimony to arlen specter? university of geneva the university of geneva is a public research university located in geneva, switzerland ben carré in the 1920s, carré worked as a freelance art director designing sets for the red lily, directed by fred niblo and starring ramon novarro and designing the catacombs for the phantom of the opera. carré worked on a string of films for the newly formed metro-goldwyn-mayer, starting with the masked bird and including la bohème, directed by king vidor. kenneth o'donnell on may 18, 1964, o'donnell provided testimony to norman redlich and arlen specter, assistant counsel for the warren commission. o'donnell stated that it was his impression that the shots fired at kennedy came from the right rear … in his 1987 autobiography man of the house, former house speaker tip o'neill wrote that he had dinner with o'donnell and powers in 1968, and that both men indicated that two shots were fired from behind the fence on the grassy knoll at dealey plaza jaya prada she became a huge star in 1976 with major hit films. director k. balachander's black-and-white film anthuleni katha (1976) showcased her dramatic skills; k. viswanath's color film siri siri muvva (1976) showed her playing a mute girl with excellent dancing skills; and her title role as sita in the big-budget mythological film seetha kalyanam confirmed her versatility. in 1977, she starred in adavi ramudu, which broke box office records and which permanently cemented her star status. filmmaker vijay introduced her to kannada cinema in his 1977 super-hit movie sanadi appanna alongside kannada matinee idol raj kumar. thomas bain bain was born in london. he lived kingston upon thames attending prep school at highfield school (liphook, hampshire). he suffered from dyslexia … he completed m. phil at the geneva-based iuee (institute for european studies), and later attended the doctoral seminars of wlad godzich in the university of geneva. adavi ramudu adavi ramudu is a 1977 telugu action film directed by k. raghavendra rao how many different directors did prada work with in 1976 and 1977? seeta kalyanam seeta kalyanam is a 1976 telugu epic, mythological, drama film directed by bapu answer type: numeric answer: 5 directors in what country did bain attend the doctoral seminars of wlad godzich? answer type: span answer: switzerland tip o'neill thomas phillip "tip" o'neill jr. was an american politician, representing northern boston, massachusetts, as a democrat from 1953 to 1987 answer type: binary answer: yes what was the first film metro-goldwyn-mayer released? metro-goldwyn-mayer mgm produced more than 100 feature films in its first two years. answer type: none answer: n/a link prediction: hard retrieval: bridge reasoning: discrete-numeric link prediction: medium retrieval: bridge reasoning: non-discrete link prediction: hard retrieval: cross context reasoning: discrete-temporal link prediction: easy reasoning: non-discrete figure 2: examples from iirc, labeled with what kinds of processing are required to answer each question. see ## combined evaluation by construction, all the questions in iirc require more than the original paragraph to answer. this means that a reading comprehension model built for iirc does not actually have to detect whether more information is required than what is in the given paragraph, as it can always assume that this is true. in order to combat this bias, we recommend an additional, more stringent evaluation that combines iirc with other reading comprehension datasets that do not require retrieving additional information. this is in line with recentlyrecommended evaluation methodologies for reading comprehension models @xcite . in this section, we present the results of one such evaluation. noting that iirc has similar properties to both squad 2.0 @xcite and drop @xcite , and even similar question language in places, we sample questions from these datasets to form a combined dataset for training and evaluating our baseline model. sampling from squad 2.0 and drop to construct the data for the combined evaluation, we sample an additional 3360 questions from squad 2.0 and drop, so that they make up 20% of the questions in the new data. we sample from squad 2.0 and drop with a ratio of 3 : 1 in order to match the distribution of numeric questions in iirc and used a wikifier @xcite to identify the links to wikipedia articles in them. ## related work questions requiring multiple contexts prior multi-context reading comprehension datasets were built by starting from discontiguous contexts, and forming compositional questions by stringing multiple facts either by relying on knowledge graphs as in qangaroo @xcite , or by having crowdworkers do so, as in hotpotqa @xcite . it has been shown that many of these questions can be answered by focusing on just one of the facts used for building the questions @xcite . in contrast, each question in iirc was written by a crowdworker who had access to just one paragraph, with the goal of obtaining information missing in it, thus minimizing lexical overlap between questions and the answer contexts. additionally, iirc provides a unique question type: questions requiring aggregating information from many related documents, such as the second question in figure 2 . ## conclusion we introduced iirc, a new dataset of incompleteinformation reading comprehension questions. these questions require identifying what information is missing from a paragraph in order to answer a question, predicting where to find it, then synthesizing the retrieved information in complex ways. our baseline model, built on top of state-ofthe-art models for the most closely related existing datasets, performs quite poorly in this setting, even when given oracle retrieval results, and especially when combined with other reading comprehension datasets. iirc both provides a promising new avenue for studying complex reading and retrieval problems and demonstrates that much more research is needed in this area. and gifts from the sloan foundation and allen institute for ai. authors would also like to thank members of the allen institute for ai, uw-nlp, and the h2lab at the university of washington for their valuable feedback and comments.