Incorporating Centering Theory into Neural Coreference Resolution (2022)

Abstract: In recent years, transformer-based coreference resolution systems have achieved remarkable improvements on the CoNLL dataset. However, how coreference resolvers can benefit from discourse coherence is still an open question. In this paper, we propose to incorporate centering transitions derived from centering theory in the form of a graph into a neural coreference model. Our method improves the performance over the SOTA baselines, especially on pronoun resolution in long documents, formal well-structured text, and clusters with scattered mentions.

## introduction coreference resolution is the task to find all expressions which refer to the same entity. the coreferential mentions could occur anywhere in the discourse. in recent years, many transformerbased models @xcite @xcite achieved improvements on the conll benchmark @xcite . in contrast to using transformers such as bert @xcite which learn the text input sequentially in limited chunks, how knowledge about the structure of discourse can benefit coreference resolution is less explored in the neural nlp era. coreference plays an essential role in discourse coherence. a referring expression using a reduced linguistic form (e.g., pronoun) indicates a referential relation to its antecedent in previous utterances. the referring expression connects utterances and contributes to discourse coherence implicitly. on the other hand, coreference resolution can benefit from a coherent discourse. it has long been acknowledged that coherence structure can impose constraints on referential accessibility from a linguistic perspective @xcite . centering theory @xcite ; grosz @xcite @xcite is a method to formally describe discourse coherence by using attentional state (i.e., the focus of attention of the participants at each utterance of the discourse). figure 1 shows how the coherence structure of an example text is built by means of tracking the changes in the local attentional state. 2 by applying centering theory, gordon and scearce (1995) investigate how local coherence influences the interpretation of ambiguous pronouns. from reading-time experiments, they observe that utterances with pronouns were read faster in the centering continue than in the shift status, while utterances with noun phrases containing rich lexical information were read more quickly in the centering shift than in the continue status. we conjecture that this pattern could contribute to coreference resolution. in this work, we explore the effect of changes in attentional state in the discourse on entity coreference resolution in a neural approach. inspired by jeon and strube (2020), we capture the most salient mentions of each sentence as centers to compute the local centering transition relations in accordance with centering theory. we then extend the coherence structure globally in the form of a graph. it makes the centering transitions available between any two sentences. lastly, we fuse the novel discourse structure into a neural coreference model @xcite . from the results, our proposed method improves the sota models up to 80.9 f1 score. our extensive analysis shows that our approach performs better on pronoun resolution in long documents, formal well-structured text such as magazine and newswire genres, and documents with scattered mentions of clusters. overall, we observe that incorporating discourse structure derived from centering theory can benefit coreference resolution. centering theory. since centering theory is a linguistic theory, a great number of early works @xcite @xcite were dedicated to transform it to a computational one for various languages. lately, jeon and strube (2020) is the first work that applies centering theory in a neural model for discourse coherence explicitly. they capture the relationships sentence by sentence for assessing text coherence. in the coreference resolution task, coreferent mentions could occur anywhere in the discourse rather than the adjacent sentence ## experiments settings. we train and evaluate our models on the english ontonotes 5.0 dataset @xcite . the results are reported using the conll f1 score -the average of muc @xcite , b3 @xcite , ceafe @xcite -and lea (moosavi and @xcite . in our experiments, we examine two models, s2e+se and s2e+se_ct, as explained in §3. we have 8 self-attention heads for the attention mechanisms, select the top 5 mentions for c f of each sentence, and set the threshold of cosine similarity as 0.8 for computing the centering transition relations. following the baseline, we also use longformer @xcite ) as our pretrained model, which can process a sequence up to 4096 tokens. we set the other parameters the same as the baseline @xcite . 5 all our experiments are performed on a single nvidia tesla v100 32g gpu. results. analyses. first, we check the performance of our model for pronoun resolution on: (1) gap dataset @xcite ; and (2) ontonotes test dataset in which we only keep the resolved clusters containing pronouns in both gold and system outputs. second, we investigate how models perform on different genres. the conll-2012 data contains broadcast conversation (bc), broadcast news (bn), magazine genre (mz), newswire genre (nw), pivot text (pt), telephone conversation (tc), and web data (wb) genres. in ## conclusion we present a neural coreference model incorporating discourse structure information based on centering theory. the model captures the centering transition relationships between sentences. each sentence is encoded with all neighbour sentences in a weighted graph. our approach outperforms the baseline with 80.9 f1 score. especially, it helps resolving pronoun in long documents, text in formal genres and clusters with scattered mentions.