What Makes Reading Comprehension Questions Difficult? (2022)

Abstract: For a natural language understanding benchmark to be useful in research, it has to consist of examples that are diverse and difficult enough to discriminate among current and near-future state-of-the-art systems. However, we do not yet know how best to select text sources to collect a variety of challenging examples. In this study, we crowdsource multiple-choice reading comprehension questions for passages taken from seven qualitatively distinct sources, analyzing what attributes of passages contribute to the difficulty and question types of the collected examples. To our surprise, we find that passage source, length, and readability measures do not significantly affect question difficulty. Through our manual annotation of seven reasoning types, we observe several trends between passage sources and reasoning types, e.g., logical reasoning is more often required in questions written for technical passages. These results suggest that when creating a new benchmark dataset, selecting a diverse set of passages can help ensure a diverse range of question types, but that passage difficulty need not be a priority.

## introduction state-of-the-art systems have shown performance comparable with humans on many recent natural language understanding (nlu) datasets @xcite , suggesting that these benchmarks will no longer be able to measure future progress. to move beyond this, we will need to find better ways of building difficult datasets, ideally without sacrificing diversity or coverage @xcite . to obtain such humanwritten examples at scale, there are active lines of crowdsourcing research on protocols of worker handling and feedback @xcite and the design of the collection task @xcite . however, we do not have clear mctest: tony walked home from school on his birthday. he was surprised to see a lot of cars in front of his house. when he opened the door and entered the house, he heard a lot of people yell, "surprise!" it was a surprise party for his birthday. his parents called all his friends' parents and invited them to come to a party for tony. [...] q: who were invited to the party and by who? tony's parents invited only his friends tony invited his friends and their parents tony's parents invited his friends' parents tony's parents invited his friends and their parents reclor: humanitarian considerations aside, sheer economics dictates that country x should institute, as country y has done, a nationwide system of air and ground transportation for conveying seriously injured persons to specialized trauma centers. timely access to the kind of medical care that only specialized centers can provide could save the lives of many people. [...] q: what is the economic argument supporting the idea of a transportation system across the nation of country x? building the transportation system creates a substantial increase of jobs for the locals increasing access to specialized medical centers can lower the chance of the workforce population dying transportation ticket prices directly contribute to the government's revenue country y was successful with their attempts to potentially save lives so country x should try it as well information on what aspects of text sources affect the difficulty and diversity of examples. crowdsourced datasets in reading comprehension use passages taken from a variety of sources, such as news articles, exams, and blogs, about which questions are written @xcite @xcite . the first example in figure 1 is from mctest @xcite , the passages of which are written in grade-school-level english. the second example is from @xcite , which consists of passages and questions written for graduate and law school admission examinations. we hypothesize that difficult passages, such as those in the second example, are more suitable for crowdsourcing challenging questions. passages that are linguistically complex and have dense information could help facilitate the writing of questions that require understanding a wide range of linguistic and world knowledge, following intricate events, and comprehending logical arguments. in contrast, easy passages, as in children's stories, likely talk about common situations and simple facts, which might prevent workers from writing difficult questions. in this work, we crowdsource multiple-choice reading comprehension questions to analyze how question difficulty and type are affected by the choice of source passage. using passages extracted from seven different sources, we ask crowdworkers to write questions about the given passages. we compute the difference between human and machine accuracy, using it as a measure of the question difficulty, to investigate whether there is a correlation between the question difficulty and linguistic aspects of the passage, such as their source, length, and readability. in addition to a standard setting where we directly accept crowdworkers' submissions, we use an adversarial setting in which they have to write questions that fool a strong reading comprehension model @xcite . previous work finds that questions that require numerical reasoning frequently appear in the adversarial data collection of the extractive qa task on wikipedia articles @xcite , but our aim is to see whether we observe a similar trend in multiple-choice questions written for different passage sources or if the adversarial setting is useful for collecting especially diverse questions. to our surprise, we find that the difficulty of collected questions does not depend on the differences of passages in linguistic aspects such as passage source, passage length, flesch-kincaid grade level @xcite , syntactic and lexical surprisal, elapsed time for answering, and the average word frequency in a passage. our main positive finding comes through our manual annotation of the types of reasoning that each question targets, where we observe that questions that require numerical reasoning and logical reasoning are relatively difficult. in addition, we find several trends between the passage sources and reasoning types. for example, logical reasoning is more often required in questions written for technical passages, whereas understanding of a given passage's gestalt and the author's attitude toward it are more frequently required for argumentative and subjective passages than expository passages. these results suggest that when creating a new benchmark dataset or choosing one for evaluating nlu systems, selecting a diverse set of passages can help ensure a diverse range of question types, but that passage difficulty need not be a priority. our collected datasets could be useful for training reading comprehension models and for further analysis of requisite knowledge and comprehension types in answering challenging multiplechoice questions. foot_0 2 related work crowdsourcing nlu datasets crowdsourcing has been widely used to collect human-written examples at scale @xcite . crowdworkers are usually asked to write questions about a given text, sometimes with constraints imposed to obtain questions that require specific reasoning skills such as multi-hop reasoning @xcite or understanding of temporal order, coreference, or causality @xcite . in this study, to analyze naturally written examples, we do not consider specific constraints on questions or answer options. current benchmark datasets constructed by crowdsourcing may not be of sufficient quality to precisely evaluate human-level nlu. for example, ribeiro et al. (2020) reveal that state-ofthe-art models in traditional nlp benchmarks fail simple behavioral tests of linguistic capabilities (checklists). @xcite and @xcite show that questions in multi-hop reasoning datasets such as hotpotqa by @xcite do not necessarily require multi-hop reasoning across multiple paragraphs. to investigate how to collect high-quality, challenging questions through crowdsourcing, @xcite compare different sourcing protocols and find that training workers and providing feedback about their submissions improve the difficulty and quality of their reading comprehension questions. to encourage workers to write difficult examples, @xcite propose to collect questions using a model-in-the-loop setting. although this adversarial approach enables us to collect challenging questions efficiently, @xcite point out that the collected examples might be bi-ased towards the quirks of the adversary models. bowman and dahl (2021) extend this argument, and point out that adversarial methods can systematically eliminate coverage of some phenomena. this is also supported by @xcite , but their findings are limited to extractive qa for wikipedia articles. our motivation is to see if this argument is applicable to the multiple-choice format with a wide range of passage sources for which we expect crowdworkers to write linguistically diverse questions and answer options. sources of nlu datasets reading comprehension datasets are often constructed with a limited number of passage sources. @xcite sample about five hundred articles from the top 10,000 articles in pagerank of wikipedia. similarly, @xcite curate passages from wikipedia articles containing numeric values to collect questions for mathematical and symbolic reasoning. @xcite construct a dataset in which questions are written for various passage sources such as news articles, science textbooks, and narratives. however, we cannot use their questions for our analysis of the variation of naturally written questions because they are designed to require local multi-sentence reasoning (such as coreference resolution and paraphrasing) by filtering out questions answerable only with a single sentence. similarly to our work, @xcite find that readability metrics and question difficulty do not correlate in reading comprehension datasets. our study differs in the following two points, which could cause different findings: first, their observational study of existing datasets has fundamental confounding factors because the questions they examine are constructed using different sourcing methods (e.g., automatic generation, expert writing, and crowdsourcing), which could have an impact on the question difficulty. we aim to investigate uniformly crowdsourced examples across seven different sources to obtain insights for future data construction research using crowdsourcing. second, they define question difficulty using human annotations alone, but this does not necessarily reflect the difficulty for current state-of-the-art models. in this study, we define the question difficulty as the human-machine performance gap using eight recent strong models, which enables a more finegrained analysis of the collected questions for a better benchmark of current models. fisch et al. (2019) propose a shared task consist-ing of different in-domain and out-domain datasets. however, they combine datasets in different task formats and sourcing methods, which prevents us from comparing questions across passage sources alone. in contrast, our focus is to compare questions collected by crowdsourcing for the same task format to analyze the question difficulty for current state-of-the-art models. we adopt the multiplechoice format because, as discussed by huang et al. (2019), it allows us to evaluate both human and machine performance easily. ## linguistic analysis we analyze how the linguistic aspects of the collected examples correlate with the human-model performance gap computed in the experiments. to get a better estimate of human performance, we use the high-agreement examples @xcite . for ease of comparison, we split these examples into two subsets: easy (∆ ≤ 20%) and hard (∆ ≥ 40%). these subsets have 1,970 and 547 examples, respectively. appendix f provides the frequency of easy and hard examples across the passage sources and collection methods. ## conclusion to make an nlu benchmark useful, it has to consist of examples that are linguistically diverse and difficult enough to discriminate among state-ofthe-art models. we crowdsource multiple-choice reading comprehension questions for passages extracted from seven different sources and analyze the effects of passage source on question difficulty and diversity. although we expect that the difficulty of a passage affects the difficulty of questions about that passage, the collected questions do not show any strong correlation between the human-machine performance gap and passage source, length, or readability measures. our manual annotation of comprehension types reveals that questions requiring numerical or logical reasoning are relatively difficult. we also find several trends between passage sources and comprehension types. these results suggest that when creating a new benchmark dataset, we need to select passage sources carefully, so that the resulting dataset contains questions that require an understanding of the linguistic phenomena that we are interested in. this is especially important in the adversarial setting because it could concentrate the distribution of questions towards a few specific question types.