To Test Machine Comprehension, Start by Defining Comprehension (2020)

Abstract: Many tasks aim to measure machine reading comprehension (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension—a “Template of Understanding”—for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.

## introduction over the past few years, neural models (e.g., @xcite @xcite have begun to match or even exceed human performance on machine reading comprehen-sion (mrc) benchmarks. in these tasks, systems demonstrate their comprehension of a passage by answering questions about it. yet despite recent successes, mrc appears far from solved: systems continue to make basic, sometimes baffling mistakes, and they fail to generalize to new data. such shortcomings have motivated a flurry of new mrc tasks, each designed to confront systems with questions deemed challenging for current methods. for example, tasks may ask questions requiring commonsense reasoning @xcite , multihop reasoning @xcite , or inferences based on a second passage @xcite . this line of research assumes that ever-more-"difficult" question-answering tasks will ultimately lead to more robust and useful reading comprehension. we argue that, while the question-answering * equal contributions. like athletic training, mrc tasks are not an end in themselves; ultimately, they are meant to lead to real-world applications. current tasks may suffice for sufficiently similar applications-e.g., chatbots that look up customer questions in product documentation. but many proposed nlp applications hinge on deeper comprehension. early work (e.g., @xcite pointed to examples like assistance with legal disputes and service contracts; more recent work suggests applications such as summarizing a patient's clinical timeline @xcite . for such complex applications, machines will need to manipulate rich models of the world evoked by the text-e.g., to compare a claimant's narrative to legal standards, or to build a causal model of a patient's condition. from this broader perspective, the current paradigm falls short. specifically, we claim that in the quest for difficulty, task designers overlook the issue of what content-what information expressed, implied, or relied on by the passage-systems should comprehend. mrc datasets are usually constructed by having humans cast about for supposedly tricky questions, most often questions based on reasoning. but the questions that result are scattershot, offering little assurance that even a high-scoring system has achieved a useful and robust understanding. we advocate for a different approach. we propose that the first step in defining mrc tasks should be specifying what content a system would likely need to understand for a given class of applica-tions. only then can tasks systematically compile questions to probe for the internal model that the machine ought to have constructed. this paper demonstrates such an approach for applications that involve understanding narratives. 1 after reviewing existing approaches to constructing mrc datasets ( §2), we argue for narratives as a valuable mrc testbed ( §3.1). then, inspired by cognitive science research on reading comprehension, we propose a "template of understanding" (tou) for stories-an account of what an internal model of a story should minimally contain ( §3.2). we also suggest ways to operationalize our tou as a story comprehension task ( §4). finally, we show evidence from a pilot tou-based task that current mrc models are not up to the challenge ( §5). ## existing mrc dataset designs this paper addresses how mrc tests can be made more systematic. accordingly, we review existing tasks grouped by their data collection methods. we argue that each category falls short of testing a useful body of content in a satisfying way. ## defining deep story understanding our approach starts from the content of a passage, which we define as the information it expresses, implies, or relies on. specifically, we propose that task designers lay out a minimal body of content that mrc systems should demonstrate they understand. exactly what that content is will vary from passage to passage, of course, but the key is to define a template of understanding (tou): a set of question templates that can be filled in with specific events and entities for any given passage. the answers to the fleshed-out questions will constitute a floor of understanding for the passage-a plausible lower bound on what content machines ought to comprehend. the natural next question is what content the tou should cover. system needs will vary by application. to advance mrc writ large without limiting ourselves to a single application, we propose selecting a class of texts where one could reasonably predict a priori what content would be useful for applications. in the rest of this section, we endorse fictional narratives as a particularly promising class of texts and propose a tou for them. 3 ## towards a story understanding task our tou provides a conceptual framework for stating what a machine should understand from a story. ## current mrc systems do not comprehend stories if current systems performed well on the tou, our argument would be moot. this section presents evidence that they do not.