Graph-combined Coreference Resolution Methods on Conversational Machine Reading Comprehension with Pre-trained Language Model (2022)

Abstract: Coreference resolution such as for anaphora has been an essential challenge that is commonly found in conversational machine reading comprehension (CMRC). This task aims to determine the referential entity to which a pronoun refers on the basis of contextual information. Existing approaches based on pre-trained language models (PLMs) mainly rely on an end-to-end method, which still has limitations in clarifying referential dependency. In this study, a novel graph-based approach is proposed to integrate the coreference of given text into graph structures (called coreference graphs), which can pinpoint a pronoun’s referential entity. We propose two graph-combined methods, evidence-enhanced and the fusion model, for CMRC to integrate coreference graphs from different levels of the PLM architecture. Evidence-enhanced refers to textual level methods that include an evidence generator (for generating new text to elaborate a pronoun) and enhanced question (for rewriting a pronoun in a question) as PLM input. The fusion model is a structural level method that combines the PLM with a graph neural network. We evaluated these approaches on a CoQA pronoun-containing dataset and the whole CoQA dataset. The result showed that our methods can outperform baseline PLM methods with BERT and RoBERTa.

## introduction in recent years, using a large-scale pre-trained language model (plm) as a backbone for various challenging machine comprehension tasks @xcite has become fundamental, especially in conversational machine reading comprehension (cmrc) @xcite . cmrc tasks not only require a model to fully understand the given articles but also propose to mimic the way humans seek information in conversations through questionanswering. most plm utilize attention mechanism and achieve positive results on a broad range of cmrc datasets @xcite , 2019). plms generally use an end-to-end approach trained from questions to answers. however, the explainability of the answers generated through the intrinsic multi-head self-attention mechanism remains insufficient. although these plms have demonstrated great advantages in terms of solving questions that simply need semantic matching, limitations in logical comprehension @xcite such as in coreference resolution still exist. coreference resolution such as for anaphora @xcite is commonly found in cmrc tasks. anaphora can be described as a pronoun word (anaphor) contained in a current question, in which its referential entity (antecedent) has already been introduced earlier in the conversation history or article context. as shown in figure 1 , to answer the current question "did she have any visitors?", the model requires that the pronoun "she" be resolved as an anaphor referring to the entity "jessica" as its antecedent, on the basis of the given context and conversation history. therefore, cmrc models require mechanisms that can resolve referential dependencies to properly understand the intent of current questions. considering the shortcomings of the plm approach in logical comprehension such as in coref-erence resolution, research on how to better adapt models to learn reasoning is gradually gaining attention @xcite @xcite . flowqa @xcite was proposed to add a reasoning layer between questions and answers to incorporate intermediate representations of a conversation history. the question rewriting (qr) model @xcite was proposed to rewrite current questions on the basis of a conversation history. specifically, the qr model simplifies complex multi-turn questionanswering (qa) tasks into single-turn qa tasks, which can solve a current question without a conversation history. however, because these models are built through the embeddings of a conversation history @xcite , they generally suffer from two drawbacks in coreference reasoning for cmrc tasks. (1) since the input length of a conversation history is limited by the plm's structure, the current question sometimes contains pronouns whose referential entity does not appear in the conversation history, so the model cannot accordingly resolve referential dependencies. (2) to achieve coreference reasoning, a cmrc model also needs to seek information from the context of articles. due to the sequence nature of the plm and the multiple referential dependencies in the context of an article, these models cannot handle each referential dependency precisely, as shown in figure 2 's context part in different colors. in this paper, we propose solving the coreference of a target pronoun through additional mined information to enhance plms' coreference reasoning ability for cmrc. a novel graph approach is proposed that integrates the coreferences of given text into graph structures, which we call the coreference graph. the coreference graph is constructed separately by using the conversation history and article context as text information. each entity in the graph holds a unique place label in accordance with the text information, which can be used to pinpoint every pronoun's referential dependency precisely. to better implement the coreference graph as an enhanced component into plms, we propose two graph-combined methods: the evidence-enhanced method and the fusion model method. these two methods integrate graph information from the textual and structural levels of the plm architecture, respectively. the evidence-enhanced method involves two textual level methods that enrich the plm's input information for coreference reasoning: an evidence generator (eg) generates new text to elaborate pronouns, and an enhanced question (eq) rewrites a pronoun into a referential entity in a question. the fusion model is a structural level method that combines the plm with a graph neural network. this model treats the plm as an encoder to extract sequence features of pronouns and referential words from input. after that, the graph features of the corresponding words are computed by graph neural networks on the basis of the connectivity of the coreference graph. these two features are integrated using learnable weights to enhance the plm's coreference reasoning ability. for the experiments, we used questions from coqa @xcite that contained pronouns to compose a new dataset (pronoun-containing dataset) specialized for the coreference reasoning ability of the cmrc model. we evaluated various combinations of our proposed methods on different plms, and we also compared them with the existing qr approach. the results showed that our methods can greatly outperform in terms of f1 score on the coqa pronoun-containing dataset, 2.6 on bert @xcite and 0.7 on roberta @xcite . we also used the whole coqa dataset to evaluate the fusion model, which achieved the best performance in our methods, to compare its overall performance with roberta. the contributions of this paper are as follows. ## propsed methods in this section, we describe the architecture of our methods as an enhanced plm component, as illustrated in figure 2 . the model contains two stages. ## results & analysis the results are shown in by 0.1 on the roberta baseline. therefore, we concluded that while dealing with anaphora's coreference resolution, both the eg and eq were effective as enhanced components of the plm baseline model with bert. comparing eg and eq approaches comprehensively for bert and roberta, the eg one had generally higher scores. one possible reason is that generating additional evidence behind a question as input maintains the integrity of the original question. although the eq approach also achieved relevant performance, the textual substitution of pronouns may alter the intention of the question and mislead the model to make erroneous answer predictions. to measure the effectiveness of the evidenceenhanced approach for each question, we compared the f1 scores of the answers produced by the baseline (bert) and our evidence-enhanced model with the best performance ("eg+hist.&cont.," as shown in from the results for the fusion model, we found that the fusion model achieved a further improvement (by up to 0.7 on roberta) compared with the baseline and evidence-enhanced methods. this model also showed improvement on the coqa-all dataset, which contains samples that are not needed for coreference resolution (without pronouns in questions), compared with the baseline. this indicates that the fusion model can effectively use coreference graph information. it can solve coreference resolution and maintain the ability to solve no-coreference questions. therefore, compared with the evidence-enhanced approach, the fusion model has higher robustness. through comparing the two different graph embedding methods, gat and gcn, we found that gcn generally outperformed gat in terms of score in each category. we assume one reason is that the processed graphs always hold the same structure (a vertex containing multiple one-hop neighbor nodes), and such a simple structure is not adequately learned by gat's multi-head attention, which is suitable for capturing features from the spatial domain. in contrast, gcn captures the graph features of each neighbor by using convolution layers, so it performed better in this experiment. ## case study we investigated how our approaches improve the coreference reasoning ability of the roberta baseline approach. to compare the differences in answer prediction, we used roberta-base as the baseline. roberta-base + hist.&cont. + eg had the best performance in an example is shown in figure 4 . in this example, the coreference graph resolves that "he" refers to "joseph aloisius ratzinger." because of the absence of coreference resolution, the baseline incorrectly predicted the answer at the wrong place. eg resolved the referential dependencies, so the prediction's meaning was close to the correct answer. however, the fusion model could integrate the coreference information and predict the answer span accurately. ## conclusion in this paper, we proposed the coreference graph, which can integrate coreferences from text into a graph structure. to use the information retrieved from a coreference graph, we introduced the evidence-enhanced method, which comprises two textual-level coreference resolution approaches to leverage bert's performance on cmrc. however, the results showed that the improvement for roberta is still limited. therefore, we proposed the fusion model, using graph neural networks to incorporate the coreference graph into plm structure. in comparison with the baseline and evidenceenhanced methods, the fusion model showed further improvement on roberta, maintaining relatively higher robustness when learning coreference resolution. we confirmed that in conversational reading comprehension, a graph-structured representation of the article context and conversational history can both be an information supplement for answering a current question, especially with different plms. rather than the end-to-end method in plms, our approaches can generate readable text as evidence when answering a question, which strengthens the interpretability of plms.