Coreference Reasoning in Machine Reading Comprehension (2021)

Abstract: Coreference resolution is essential for natural language understanding and has been long studied in NLP. In recent years, as the format of Question Answering (QA) became a standard for machine reading comprehension (MRC), there have been data collection efforts, e.g., Dasigi et al. (2019), that attempt to evaluate the ability of MRC models to reason about coreference. However, as we show, coreference reasoning in MRC is a greater challenge than earlier thought; MRC datasets do not reflect the natural distribution and, consequently, the challenges of coreference reasoning. Specifically, success on these datasets does not reflect a model’s proficiency in coreference reasoning. We propose a methodology for creating MRC datasets that better reflect the challenges of coreference reasoning and use it to create a sample evaluation set. The results on our dataset show that state-of-the-art models still struggle with these phenomena. Furthermore, we develop an effective way to use naturally occurring coreference phenomena from existing coreference resolution datasets when training MRC models. This allows us to show an improvement in the coreference reasoning abilities of state-of-the-art models.

## introduction machine reading comprehension is the ability to read and understand the given passages and answer questions about them. coreference resolution is the task of finding different expressions that refer to the same real-world entity. the tasks of coreference resolution and machine reading comprehension have moved closer to each other. converting coreference-related datasets into an mrc format improves the performance on some coreferencerelated datasets @xcite . there are also various datasets for the task of reading comprehension on which the model requires to perform coreference reasoning to answer some of the questions, e.g., drop @xcite , duorc @xcite , multirc @xcite , etc. quoref @xcite is a dataset that is particularly designed for evaluating coreference understanding of mrc models. figure 1 shows a qa sample from quoref in which the model needs to resolve the coreference relation between "his" and "john motteux" to answer the question. recent large pre-trained language models reached high performance on quoref. however, our results and analyses suggest that this dataset contains artifacts and does not reflect the natural distribution and, therefore, the challenges of coreference reasoning. as a result, high performances on quoref do not necessarily reflect the coreference reasoning capabilities of the examined models and answering questions that require coreference reasoning might be a greater challenge than current scores suggest. in this paper, we propose two solutions to address this issue. first, we propose a methodology for creating mrc datasets that better reflect the coreference reasoning challenge. we release a sample challenging evaluation set containing 200 examples by asking an annotator to create new question-answer pairs using our methodology and based on existing passages in quoref. we show that this dataset contains fewer annotation artifacts, and its distribution of biases is closer to a coreference resolution dataset. the performance of state-of-the-art models on quoref considerably drops on our evaluation set suggesting that (1) coreference reasoning is still an open problem for mrc models, and (2) our methodology opens a promising direction to create future challenging mrc datasets. second, we propose to directly use coreference resolution datasets for training mrc models to improve their coreference reasoning. we automatically create a question whose answer is a coreferring expression m 1 using the bart model @xcite . we then consider this question, m 1 's antecedent, and the corresponding document as a new @xcite tuple. this data helps the model learning to resolve the coreference relation between m 1 and its antecedent to answer the question. we show that incorporating these additional data improves the performance of the state-of-the-art models on our new evaluation set. our main contributions are as follows: ## creating an mrc dataset that better reflects coreference reasoning there is a growing trend in using adversarial models for data creation to make the dataset more challenging or discard examples that can be solved using surface cues @xcite @xcite @xcite @xcite . quoref is also created using an adversarial data collection method to discard examples that can be solved using simple lexical cues. the assumption is that it is hard to avoid simple lexical cues by which the model can answer questions without coreference reasoning. therefore, an adversarial model (a) is used to discard examples that contain such lexical cues. while this adversarial filtering removes examples that are easy to solve by a, it does not ensure that the remaining examples do not contain shortcuts that are not explored by a. first, the adversarial model in quoref is trained on another dataset, i.e., squad. thus, the failure of a on quoref examples may be due to ( 1 ) quoref having different lexical cues than those in squad, or (2) domain shift. second, and more importantly, as argued by @xcite , making the task challenging by focusing on examples that are more difficult for existing models is not a solution for more useful reading comprehension. 7 we instead propose a methodology for creating question-answer pairs as follows: we provide this guideline to a student from the 7 as put by them: "the dominant mrc research paradigm is like trying to become a professional sprinter by glancing around the gym and adopting any exercises that look hard". 8 proper names are more informative than common nouns, and they are more informative than pronouns @xcite . computer science department for generating new qa pairs from the existing passages in the quoref development set. we use quoref passages to ensure that the source of performance differences on our dataset vs. quoref is not due to domain differences. this results in 200 new qa pairs. ## improving coreference reasoning while we do not have access to many coreference annotations for the task of coreference-aware mrc, there are various datasets for the task of coreference resolution. coreference resolution datasets contain the annotation of expressions that refer to the same entity. in this paper, we hypothesize that we can directly use coreference resolution corpora to improve the coreference reasoning of mrc models. we propose an effective approach to convert coreference annotations into qa pairs so that models learn to perform coreference resolution by answering those questions. in our experiments, we use the 9 we examine 50 randomly selected examples from our challenge set, and they were all answerable by a human. @xcite ) that is the largest annotated dataset with coreference information. ## conclusions we show that the high performance of recent models on the quoref dataset does not necessarily indicate that they are adept at performing coreference reasoning, and that qa based on coreference reasoning is a greater challenge than current scores suggest. we then propose a methodology for creating a dataset that better presents the coreference reasoning challenge for mrc. we provide our methodology to an annotator and create a sample dataset. our analysis shows that our dataset contains fewer biases compared to quoref, and the performance of state-of-the-art quoref models drops considerably on this evaluation set. to improve the coreference reasoning of qa models, we propose to use coreference resolution datasets to train mrc models. we propose a method to convert coreference annotations into an mrc format. we examine the impact of incorporating this coreference data on improving the coreference reasoning of qa models using two topperforming qa systems from the quoref leaderboard. we show that using coreference datasets improves the performance of both examined models on our evaluation set, indicating their improved coreference reasoning. the results on our evaluation set suggest that there is still room for improvement, and reading comprehension with coreference understanding remains a challenge for existing qa models, especially if the coreference relation is between two distant expressions.