E nglish Machine Reading Comprehension Datasets: A Survey (2021)

Abstract: This paper surveys 60 English Machine Reading Comprehension datasets, with a view to providing a convenient resource for other researchers interested in this problem. We categorize the datasets according to their question and answer form and compare them across various dimensions including size, vocabulary, data source, method of creation, human performance level, and first question word. Our analysis reveals that Wikipedia is by far the most common data source and that there is a relative lack of why, when, and where questions across datasets.

## datasets all datasets and their properties of interest are listed in ## related work other mrc surveys have been carried out @xcite @xcite @xcite @xcite . there is room for more than one survey since each survey approaches the field from a different perspective, varying in the criteria used to decide what datasets to include and what dataset properties to describe. some focus on mrc systems rather than on mrc datasets. the surveys closest to ours in coverage are zeng et al. (2020) and @xcite . the survey of @xcite has a wider scope, including datasets for any language, multimodal datasets and datasets which do not fall under the canonical mrc passage/question/answer template. an emphasis in our survey has been on relatively fine-grained dataset processing. for example, instead of reporting only train/dev/test sizes for each dataset, we calculate the length of each question, passage and answer. we examine the vocabulary of each dataset, looking at question words and applying language identification tools, in the process discovering question imbalance and noise (html tags, misspellings, etc.). we also report data re-use across datasets. ## concluding remarks and recommendations this paper provides an overview of english mrc datasets, released up to and including 2020. we compare the datasets by question and answer type, size, data source, creation method, vocabulary, question type, "solvedness", and human performance level. we observe the tendency of moving from smaller datasets towards large collections of questions, and from synthetically generated data through crowdsourcing towards spontaneously created. we also observe a scarcity of why, when, and where questions. gathering and processing the data for this survey was a painstaking task, foot_18 from which we emerge with some very practical recommendations for future mrc dataset creators. in order to 1) compare to existing datasets, 2) highlight possible limitation for applicable methods, and 3) indicate the computational resources required to process the data, some basic statistics such as average passage/question/answer length, vocabulary size and frequency of question words should be reported; the data itself should be stored in consistent, easyto-process fashion, ideally with an api provided; any data overlap with existing datasets should be reported; human performance on the dataset should be measured and what it means clearly explained; and finally, if the dataset is for the english language and its design does not differ radically from those surveyed here, it is crucial to explain why this new dataset is needed. for any future datasets, we suggest a move away from wikipedia given the volume of existing datasets that are based on it and its use in pretrained language models such as bert @xcite . as shown by @xcite , its use in mrc dataset and pre-training data brings with it the problem that we cannot always determine whether a system's ability to answer a question comes from its comprehension of the relevant passage or from the underlying language model. the medical domain is well represented in the collection of english mrc datasets, indicating a demand for understanding of this type of text. datasets may be required for other domains, such as retail, law and government. some datasets are designed to test the ability of systems to tell if a question cannot be answered, by including a "no answer" label. building upon this, we suggest that datasets be created for the more complex task of providing qualified answers based on different interpretations of the question.