STARC : Structured Annotations for Reading Comprehension (2020)

Abstract: We present STARC (Structured Annotations for Reading Comprehension), a new annotation framework for assessing reading comprehension with multiple choice questions. Our framework introduces a principled structure for the answer choices and ties them to textual span annotations. The framework is implemented in OneStopQA, a new high-quality dataset for evaluation and analysis of reading comprehension in English. We use this dataset to demonstrate that STARC can be leveraged for a key new application for the development of SAT-like reading comprehension materials: automatic annotation quality probing via span ablation experiments. We further show that it enables in-depth analyses and comparisons between machine and human reading comprehension behavior, including error distributions and guessing ability. Our experiments also reveal that the standard multiple choice dataset in NLP, RACE, is limited in its ability to measure reading comprehension. 47% of its questions can be guessed by machines without accessing the passage, and 18% are unanimously judged by humans as not having a unique correct answer. OneStopQA provides an alternative test set for reading comprehension which alleviates these shortcomings and has a substantially higher human ceiling performance.

## introduction assessment of reading comprehension is of paramount importance in education and science and is a key component of high-stakes evaluations such as the sat examinations. reading comprehension tasks are also central to nlp, where extensive efforts are invested in developing systems that try to match human-level performance. despite the proliferation of nlp work on reading comprehension and the increasing number of large-scale reading comprehension datasets, key quality assurance issues such as question guessability, unwanted dataset biases, and the considerable success of simple pattern matching and slot filling heuristics remain open challenges for ensuring that evaluation benchmarks capture genuine reading comprehension. further, existing annotation frameworks have very limited support for reading behavior analyses which go beyond simple accuracy statistics. in this work, we introduce starc, a new annotation framework for multiple choice reading comprehension, which addresses these shortcomings. our framework aims to ensure high annotation quality and supports detailed probing and comparisons of human and machine reading comprehension behavior. the following are the primary novel characteristics of our annotation scheme. ## starc annotation scheme starc is a new annotation framework accompanied by a protocol for increasing annotation quality and reducing annotation biases which can be exploited by either humans or machines for solving reading comprehension datasets without performing the intended task. the annotation scheme aims for the questions to be on a high difficulty level. importantly, starc tries to minimize the possibility of answering questions correctly using simple string-matching strategies, as well as guessing the correct answer without reading the passage. to focus on testing language comprehension, as opposed to other types of skills and knowledge, it aims to avoid questions that rely on numerical reasoning and substantial external world knowledge. it also refrains from questions that require the reader to speculate (for example, given some information on person x, ask about their likely position issue y when this position is not stated in the text). reading comprehension questions have four answers, structured in the following manner. a is the correct answer. answering a question correctly requires comprehending information from a text span in the passage called the critical span. importantly, with exceptions when necessary, the correct answer should not appear in the critical span in verbatim form. b is an incorrect answer which represents a plausible misunderstanding of the critical span. c is an incorrect answer which refers to an additional span in the passage, called the distractor span. this answer can be anchored in the distractor span in various ways. for example, it may borrow keywords, or contain a correct fact that is stated in the distractor span but is not the correct answer to the question. d is an incorrect answer which is plausible a-priori, but has no support in the passage. note that to be plausible, d often appeals to the reader's general world knowledge. neither the critical span nor the distractor span have to adhere to sentence boundaries, and both can be non-continuous. this structure introduces well-defined and con-sistent relations between the answers and the passage. further, the answers are ordered by degree of comprehension, whereby a represents correct comprehension, b reflects the ability to identify the crucial information for answering the question but failure to comprehend it, c reflects some degree of attention to the passage's content, and d provides no evidence for text comprehension. the utilization of b-type answers in particular enables probing comprehension at a deep level. the overall answer structure can support new types of error analyses beyond the correct/incorrect distinction by examining specific types of miscomprehension and their relation to the text. in order to reduce the effectiveness of answer elimination strategies, we developed additional guidelines on the joint form and content of the answers. these include a quality ranking of answer patterns, where the most preferred structures are those in which all answers have either similar phrasings or distinct phrasings. for all other patterns (e.g. three similarly worded answers and an outstanding answer), the answer types for the pattern should be distributed equally across questions. the guidelines also list dispreferred content relations between answers, such as b being the opposite of a. finally, the guidelines specify that the answers across, and whenever possible within questions should be of comparable length. ## onestopqa dataset we implemented the starc annotation framework in a new reading comprehension dataset, onestopqa. the textual materials of onestopqa are drawn from the onestopenglish corpus @xcite , which contains guardian news lessons articles from the english language learning portal onestopenglish.com by macmillan education. we chose articles that have non-repetitive content, and collectively represent a diverse range of topics. the texts were cleaned from errors stemming from the conversion process from the original pdfs to plain text, and manually converted from british to american english spelling. each article has three versions, corresponding to three text difficulty levels: advanced, intermediate and elementary. the advanced version is the original guardian article. the intermediate and elementary articles are simplified versions of the original article created by professional editors at onestopenglish.com. @xcite are heuristic estimates of the number of education years required to fully comprehend the text. advanced a major international disagreement with wide-ranging implications for global drugs policy has erupted over the right of bolivia's indigenous indian tribes to chew coca leaves, the principal ingredient in cocaine. bolivia has obtained a special exemption from the 1961 single convention on narcotic drugs, the framework that governs international drugs policy, allowing its indigenous people to chew the leaves. bolivia had argued that the convention was in opposition to its new constitution, adopted in 2009, which obliges it to "protect native and ancestral coca as cultural patrimony" and maintains that coca "in its natural state ... is not a dangerous narcotic." elementary a big international disagreement has started over the right of bolivia's indigenous indian tribes to chew coca leaves, the main ingredient in cocaine. this could have a significant effect on global drugs policy. bolivia has received a special exemption from the 1961 convention on drugs, the agreement that controls international drugs policy. the exemption allows bolivia's indigenous people to chew the leaves. bolivia said that the convention was against its new constitution, adopted in 2009, which says it must "protect native and ancestral coca" as part of its cultural heritage and says that coca "in its natural state ... is not a dangerous drug." q what was the purpose of the 1961 convention on drugs? a regulating international policy on drugs b discussing whether indigenous people in bolivia should be allowed to chew coca leaves c discussing the legal status of bolivia's constitution d negotiating extradition agreements for drug traffickers onestopqa has 30 articles, with 4 to 7 paragraphs per article, and a total of 162 paragraphs. each paragraph has 3 to 12 sentences. further statistics on onestopqa and race articles along with readability estimates for the different text difficulty levels are presented in ## related work a considerable number of reading comprehension datasets have been introduced in nlp. a large fraction of these datasets can be broadly divided into three tasks: cloze @xcite @xcite , span identification qa @xcite @xcite @xcite and multiple choice qa @xcite . our approach primarily falls into the third category. the basic 4-answer format we use is identical to race @xcite , which enables training models on race and evaluating them on onestopqa. our dataset is considerably smaller than race, but is of appropriate size for robust evaluations and error analyses. as demonstrated in this work, onestopqa annotations are of substantially higher quality than race, and enable analyses which are not possible with race. mctest @xcite was created with a similar purpose to race, but has a low text difficulty level suitable for 7-year-olds. span identification qa is a task in which the correct answer to the question is one or more textual spans which the reader is required to mark. this task differs from multiple choice reading comprehension in its focus on information retrieval, which limits the range of question types (e.g. forces the answers to be primarily named entities) and their difficulty level. while our approach contains span annotations, our notion of span is different from that in span identification qa: spans are not considered as answers but rather as text regions that contain the critical information for the respective answer. this difference enables a higher difficulty degree and a wider scope of question types. the combination of this approach with a multiple choice answer structure which always has a span misinterpretation distractor facilitates deeper probing of text understanding and is designed to allow for more robustness to simple pattern matching. prior work has explored both manual and automatic auxiliary span annotations for correct answers in multiple choice qa datasets @xcite . our framework extends such annotations to include multiple distractor types, with b distractors providing an additional guarantee that simply identifying the critical span is not sufficient for answering the question correctly. we further demonstrate the utility of our distractor structure for automatic verification of annotation quality through ablation experiments, as well as detailed error comparisons between human and machine readers. ## discussion we introduce a new annotation framework for reading comprehension and an accompanying highquality dataset. we leverage the novel structure of our annotations to develop a methodology for automatic validation of annotations and to perform detailed comparisons between human and machine reading comprehension. our experiments further demonstrate substantial quality assurance issues with race, which are alleviated in our new dataset. our results demonstrate the promise of our annotation framework and dataset in supporting a wide range of reading behavior analyses, as well as the feasibility of developing automated question validation tools for reading comprehension examinations for humans as exciting directions for future work.