Is the Understanding of Explicit Discourse Relations Required in Machine Reading Comprehension? (2021)

Abstract: An in-depth analysis of the level of language understanding required by existing Machine Reading Comprehension (MRC) benchmarks can provide insight into the reading capabilities of machines. In this paper, we propose an ablation-based methodology to assess the extent to which MRC datasets evaluate the understanding of explicit discourse relations. We define seven MRC skills which require the understanding of different discourse relations. We then introduce ablation methods that verify whether these skills are required to succeed on a dataset. By observing the drop in performance of neural MRC models evaluated on the original and the modified dataset, we can measure to what degree the dataset requires these skills, in order to be understood correctly. Experiments on three large-scale datasets with the BERT-base and ALBERT-xxlarge model show that the relative changes for all skills are small (less than 6%). These results imply that most of the answered questions in the examined datasets do not require understanding the discourse structure of the text. To specifically probe for natural language understanding, there is a need to design more challenging benchmarks that can correctly evaluate the intended skills.

## introduction machine reading comprehension (mrc) is concerned with the automatic extraction and generation of answers over unstructured textual data. due to its complexity, the task is seen as suitable for evaluating natural language understanding (nlu) @xcite . while neural mrc systems achieve impressive performance @xcite , it has been revealed by some research efforts that existing mrc benchmarks might be insufficient to establish model performance, i.e., that the models are not being assessed for their capabilities to read and comprehend @xcite @xcite @xcite @xcite @xcite . these analyses provide insights into the weaknesses of modern mrc gold standards. nonetheless, to stimulate the development of robust mrc systems with generalisable nlu capabilities, it is necessary to investigate the strengths and weaknesses of mrc datasets on a deeper level. in the task of mrc, it is assumed that questions test a cognitive process which involves various skills, such as retrieving stored information and performing inferences @xcite . therefore, considering metrics that reflect skills required to answer questions is useful for analysing the capabilities of mrc datasets to benchmark nlu @xcite . this leads to the following intuition: if a question is solvable even after removing features (e.g., specific words) associated with an mrc skill, the question does not require the skill. @xcite examined 10 datasets with regard to multiple requisite skills for answering questions. one of the identified 12 skills is the understanding of adjacent discourse relations, which relies on information given by the sentence order in a passage. by randomly shuffling the order of the sentences in the context and comparing model performance on the original and the modified dataset, they concluded that most existing mrc datasets might be inadequate for benchmarking adjacent discourse relations understanding. discourse relations describe how two segments of discourse are logically connected to one another. understanding them is key to answering reading comprehension questions correctly. though the findings in @xcite are useful to understand mrc datasets with respect to discourse relations understanding, we argue that it is not enough to only consider inter-sentential relations as discourse relations also widely exist within sentences foot_1 . furthermore, there also exist various types of relations and senses. hence, to comprehensively assess the capacity of mrc datasets to benchmark discourse relations understanding, we assert that further research is needed. in this paper, our aim is to provide a fine-grained analysis of the level of discourse relations understanding that is needed to answer questions in existing mrc datasets. specifically, we focus on explicit discourse relations, which are expressed using explicit connectives. this allows us to perform analysis that goes beyond shuffling sentence order. in our work, we identify seven mrc skills that represent different aspects of understanding explicit discourse relations. with these, we examine three datasets using two strong mrc models. our results show that these datasets might be insufficient for evaluating the understanding of explicit discourse relations. this work can potentially encourage the development of more challenging benchmarks that evaluate mrc models with respect to nlu capabilities that require discourse relations understanding. ## requisite skills as mentioned above, we identified a set of seven reasoning-related skills that require the understanding of explicit discourse relations, as shown in skill s 1 is inspired by @xcite , which aims to evaluate whether the understanding of adjacent explicit discourse relations is required in answering questions. different from their proposed method (i.e., randomly shuffling the order of the sentences in a passage), we only shuffle those containing explicit connectives. the selection of skills s 2 to s 7 is informed by the annotation scheme of the pdtb 3.0 corpus, which is annotated with information on discourse relations @xcite . the scheme defines 36 different senses of discourse relations. in the corpus, more than 24, 000 explicit connectives were annotated and categorised according to these senses. based on this, we obtained a distribution of explicit connectives over the 36 senses (see appendix a). afterwards, we selected a subset of them (6 senses) based on the number of unique explicit connectives, total number of explicit connectives for which each sense was recorded, and the exclusiveness of these explicit connectives. the identification process is detailed in appendix b. in the following, we provide an overview of skills s 2 to s 7 . skills s 2 and s 3 are for the understanding of asynchronous temporal relations. specifically, s 2 focuses on precedence while s 3 tests succession. skill s 4 evaluates the understanding of causal relations, which are explicitly marked in the passage by connectives such as because and due to. meanwhile, our motivation for selecting skill s 5 is to reveal whether explicit conditional reasoning is required to answer questions. different from s 4 , s 6 is for the understanding of negative causality, in which a causal relation expected on the basis of the first argument is negated by the situation described in the other. finally, s 7 assesses expansions which provide further detail to an argument. ## methodology for each of the seven identified skills, we defined an ablation method, as shown in ## experiments in this section, we describe our experimental settings, present the results of our experiments and provide insights drawn from experimentation under an extreme setting whereby all explicit connectives were dropped. ## conclusion in this paper, we proposed a methodology to assess the capabilities of mrc datasets to benchmark the understanding of explicit discourse relations. with seven fine-grained skills and corresponding ablation methods, we examined three large-scale datasets. the experimental results demonstrated that explicit discourse relations are not sufficiently evaluated by them, and thus there is a need to develop more challenging datasets so that their questions can correctly benchmark our defined skills. as for future work, we will develop a machine learning-based system that can recognise various senses of implicit discourse relations in the passage and further reveal whether the awareness of implicit discourse relations is required to do well on contemporary mrc benchmarks.