Benchmarking Machine Reading Comprehension: A Psychological Perspective (2021)

Abstract: Machine reading comprehension (MRC) has received considerable attention as a benchmark for natural language understanding. However, the conventional task design of MRC lacks explainability beyond the model interpretation, i.e., reading comprehension by a model cannot be explained in human terms. To this end, this position paper provides a theoretical basis for the design of MRC datasets based on psychology as well as psychometrics, and summarizes it in terms of the prerequisites for benchmarking MRC. We conclude that future datasets should (i) evaluate the capability of the model for constructing a coherent and grounded representation to understand context-dependent situations and (ii) ensure substantive validity by shortcut-proof questions and explanation as a part of the task design.

## introduction evaluation of natural language understanding (nlu) is a long-standing goal in the field of artificial intelligence. machine reading comprehension (mrc) is a task that tests the ability of a machine to read and understand unstructured text and could be the most suitable task for evaluating nlu because of its generic formulation @xcite . recently, many large-scale datasets have been proposed, and deep learning systems have achieved human-level performance for some of these datasets. however, analytical studies have shown that mrc models do not necessarily achieve humanlevel understanding. for example, @xcite use manually crafted adversarial examples to show that successful systems are easily distracted. @xcite show that a significant part of already solved questions is solvable even after shuffling the words in a sentence or dropping content words. these studies demonstrate that we cannot explain what type of understanding is required by the datasets and is actually acquired by models. although benchmarking mrc is related to the intent behind questions and is critical to test hypotheses from a top-down viewpoint @xcite , its theoretical foundation is poorly investigated in the literature. in this position paper, we examine the prerequisites for benchmarking mrc based on the following two questions: (i) what does reading comprehension involve? (ii) how can we evaluate it? our motivation is to provide a theoretical basis for the creation of mrc datasets. as @xcite indicate, interpreting the internals of a system is closely related to only the system's architecture and is insufficient for explaining how the task is accomplished. this is because even if the internals of models can be interpreted, we cannot explain what is measured by the datasets. therefore, our study focuses on the explainability of the task rather than the interpretability of models. we first overview mrc and review the analytical literature that indicates that existing datasets might fail to correctly evaluate their intended behavior (section 2). subsequently, we present a psychological study of human reading comprehension in section 3 for answering the what question. we argue that the concept of representation levels can serve as a conceptual hierarchy for organizing the technologies in mrc. section 4 focuses on answering the how question. here, we implement psychometrics to analyze the prerequisites for the task design of mrc. furthermore, we introduce the concept of construct validity, which emphasizes validating the interpretation of the task's outcome. finally, in section 5, we explain the application of the proposed concepts into practical approaches, highlighting potential future directions toward the advancement of mrc. regarding the what question, we indicate that datasets should evaluate the capability of the situation model, which refers to the construction question foundation requirements future direction what is reading comprehension? representation levels in human reading comprehension: (a) surface structure, (b) textbase, and (c) situation model. how can we evaluate reading comprehension? construct validity in psychometrics: (1) content, (2) substantive, (3) structural, (4) generalizability, (5) external, and (6) consequential aspects. ## mrc on psychometrics in this section, we provide a theoretical foundation for the evaluation of mrc models. when mrc measures the capability of reading comprehension, validation of the measurement is crucial to obtain a reliable and useful explanation. therefore, we focus on psychometrics-a field of study concerned with the assessment of the quality of psychological measurement @xcite . we expect that the insights obtained from psychometrics can facilitate a better task design. in section 4.1, we first review the concept of validity in psychometrics. subsequently, in section 4.2, we examine the aspects that correspond to construct validity in mrc and then indicate the prerequisites for verifying the intended explanation of mrc in its task design. ## future directions this section discusses future potential directions toward answering the what and how questions in sections 3 and 4. in particular, we infer that the situation model and substantive validity are critical for benchmarking human-level mrc. ## conclusion in this paper, we outlined current issues and future directions for benchmarking machine reading comprehension. we size indicates the size of the whole dataset including training, development, and test sets. in the question source column, crowd indicates questions written by crowdworkers and query indicates questions collected from searchengine queries.