Tracing Origins: Coreference-aware Machine Reading Comprehension (2022)

Abstract: Machine reading comprehension is a heavily-studied research and test field for evaluating new pre-trained language models (PrLMs) and fine-tuning strategies, and recent studies have enriched the pre-trained language models with syntactic, semantic and other linguistic information to improve the performance of the models. In this paper, we imitate the human reading process in connecting the anaphoric expressions and explicitly leverage the coreference information of the entities to enhance the word embeddings from the pre-trained language model, in order to highlight the coreference mentions of the entities that must be identified for coreference-intensive question answering in QUOREF, a relatively new dataset that is specifically designed to evaluate the coreference-related performance of a model. We use two strategies to fine-tune a pre-trained language model, namely, placing an additional encoder layer after a pre-trained language model to focus on the coreference mentions or constructing a relational graph convolutional network to model the coreference relations. We demonstrate that the explicit incorporation of coreference information in the fine-tuning stage performs better than the incorporation of the coreference information in pre-training a language model.

## introduction machine reading comprehension (mrc), a task that automatically identifies one or multiple words from a given passage as the context to answer a specific question for that passage, is widely used in information retrieving, search engines, and dialog systems. several datasets on mrc that limit the answer to one single word or multiple words from the passage are introduced, including trec context: frankie bono, a mentally disturbed hitman from cleveland, comes back to his hometown in new york city during christmas week to kill a middle-management mobster, troiano. ...first he follows his target to select the best possible location, but opts to wait until troiano isn't being accompanied by his bodyguards. ... losing his nerve, frankie calls up his employers to tell them he wants to quit the job. unsympathetic, the supervisor tells him he has until new year's eve to perform the hit. question: what is the first name of the person who has until new year's eve to perform a hit? answer: he ->frankie question: what is the first name of the person who follows their target to select the best possible location? answer: he ->frankie human texts, especially long texts, are abound in deictic and anaphoric expressions that refer to the entities in the same text. these deictic and anaphoric expressions, in particular, constrain the generalization of the models trained without explicit awareness of the coreference. the quoref dataset @xcite is specifically designed to validate the performance of the models in coreferential reasoning, in that "78% of the manually analyzed questions cannot be answered without coreference" @xcite . the example in pre-trained language models, including bert @xcite , roberta @xcite and xlnet @xcite , that are trained through self-supervised language modeling objectives like masked language modeling, perform rather poorly in the quoref dataset. we argue that the reason for the poor performance is that those pre-trained language models do learn the background knowledge for coreference resolution but may not learn adequately the coreference information required for the coreference-intensive reading comprehension tasks. in the human reading process, as shown in the empirical study of first-year english as a second language students during the reading of expository texts, "anaphoric resolution requires a reader to perform a textconnecting task across textual units by successfully linking an appropriate antecedent (among several prior antecedents) with a specific anaphoric referent" and "students who were not performing well academically were not skilled at resolving anaphors" @xcite and the direct instruction on anaphoric resolution elevated the readers' comprehension of the text @xcite . in addition, the studies on anaphor resolution in both adults using eye movement studies @xcite and children @xcite evidenced a twostage model of anaphor resolution proposed by @xcite . the first stage is "an initial lexically driven, contextfree stage known as bonding, whereby a link between the anaphor and a potential antecedent is made, followed by a later process known as resolution, which resolves the link with respect to the overall discourse context" @xcite . the pre-trained language models only capture the semantic representations of the words and sentences, without explicitly performing such text-connecting actions in the specific coreferenceintensive reading comprehension task, thus they do not learn adequate knowledge to solve the complex coreference reasoning problems. explicitly injecting external knowledge such as linguistics and knowledge graph entities, has been shown effective to broaden the scope of the pre-trained language models' capacity and performance, and they are often known as xaware pre-trained language models @xcite @xcite . it is plausible that we may imitate the anaphoric resolution process in human's two-stage reading comprehension of coreference intensive materials and explicitly make the text-connecting task in our fine-tuning stage as the second stage in the machine reading comprehension. as an important tool that captures the anaphoric relationship between words or phrases, coreference resolution that clusters the mentions of the same entity within a given text is an active field in natural language processing @xcite @xcite @xcite , with neural networks taking the lead in the coreference resolution challenges. the incorporation of the coreference resolution results in the pre-training to obtain the coreferenceinformed pre-trained language models, such as corefbert and corefroberta @xcite , has shown positive improvements on the quoref dataset, a dataset that is specially designed for measuring the models' coreference capability, but the performance is still considerably below the human performance. in this paper, we make a different attempt to leverage the coreference resolution knowledge and complete the anaphoric resolution process in reading comprehension. we propose a fine-tuned corefaware model that directly instructs the model to learn the coreference information 1 . our model can be roughly divided into three major components: 1) pre-trained language model component. we use the contextualized representations from the pretrained language models as the token embeddings for the downstream reading comprehension tasks. 2) coreference resolution component. neuralcoref, an extension to the spacy, is applied here to extract the mention clusters from the context. 3) coreference enrichment component. we apply three methods in incorporating the coreference knowledge: additive attention enhancement, multiplication attention enhancement, and relationenhanced graph-attention network + fusing layer. in this paper, we show that by simulating the human behavior in explicitly connecting the anaphoric expressions to the antecedent entities and infusing the coreference knowledge our model can surpass that of the pre-trained coreference language models on the quoref dataset. ## coreference-aware machine reading comprehension our model, inspired by the anaphoric connecting behavior in the human reading comprehension process, consists of four parts, namely, a pretrained language model, a coreference resolution component, a graph encoder and a fusing layer. context in the machine reading comprehension task is first processed by a coreference resolution model to identify the underlying coreference clusters, which are formed by dividing the entities and anaphoric expressions in the context into disjoint groups on the principle that the mentions of the same entity should be in the same group. then we use the coreference clusters to construct a coreference matrix that labels each individual cluster and identifies each element in the same cluster with the same cluster number. meanwhile, the context is tokenized by the tokenizer defined in the pre-trained language model and the embeddings for each token are retrieved from that model. we propose three methods for connecting the anaphoric expressions and their antecedent entity: 1) adding the coreference matrix with each attention head in the additional coreference encoder layer; 2) multiplying the coreference matrix with each attention head in the additional coreference encoder layer; 3) constructing a graph neural network based on the coreference matrix with the edges corresponding to the coreference relations and then fusing the graph representation in the graph neural network with the embeddings of the context, as shown in figure 1 . the final representations from either one of the three methods are fed into the classifier to calculate the start/end span of the question. ## conclusion in this paper, we present intuitive methods to solve coreference-intensive machine reading comprehension tasks by following the reading process of human in which people connect the anaphoric expressions with explicit instructions. we demonstrate that all our three fine-tuning methods, including coref gnn , coref addatt and coref multiatt , are superior to the pre-trained language models that incorporate the coreference information in the pretraining stage, such as corefroberta large . as the fine-tuning methods rely on the coreference resolution models supplied by other researchers, their performance is also constrained by the accuracy of those coreference resolution models. in addition, the questions that require multistep reasoning, span multiple entities or contain multiple answer items also pose the challenges to our models. in the future, with more in-depth study on human reasoning in reading comprehension and more progress in graph neural networks, the gnnbased coreference graph can be enriched with more edge types and diverse structures to leverage more linguistic knowledge and gain better performance.