Exploring Span Representations in Neural Coreference Resolution (2020)

Abstract: In coreference resolution, span representations play a key role to predict coreference links accurately. We present a thorough examination of the span representation derived by applying BERT on coreference resolution (Joshi et al., 2019) using a probing model. Our results show that the span representation is able to encode a significant amount of coreference information. In addition, we find that the head-finding attention mechanism involved in creating the spans is crucial in encoding coreference knowledge. Last, our analysis shows that the span representation cannot capture non-local coreference as efficiently as local coreference.

## introduction coreference resolution, the task of grouping all referring expressions that point to the same entity into a cluster, plays a key role for various higher level nlp tasks that involve natural language understanding such as information extraction, question answering, machine translation, text summarisation, and textual entailment. referring expressions or mentions can be common nouns, proper nouns, or pronouns, which refer to a real-world entity known as the referent. with the breakthrough of end-to-end neural systems @xcite , current coreference resolution systems are for the most part neural based. contrary to previous architectures which identified mentions and then took coreferential decisions in two separate steps, these systems jointly learn the two. a typical system requires different levels of semantic representation of the input sentences, usually done by computing representations at the span level given the word embeddings. in another area, a wave of recent work has tried to inspect neural nlp models by associating neural † shared first authorship network components with distinct linguistic phenomena by means of probing tasks @xcite @xcite . targeting the coreference task, in this paper, we build a probing model @xcite to find out what degree of coreference information is encoded in the span representations as first proposed by @xcite . specifically, we generate mention-span representations with bert embeddings fine-tuned on the ontonotes dataset @xcite and train a probing model to predict coreference arcs between two mentions from the mention-span representations alone. moreover, we explore how fine-tuning bert @xcite on coreference resolution affects the linguistic knowledge learned by the span representations. given the well-documented difficulty in modelling long-distance coreference relations, we also measure the robustness of the span representations at different distance ranges between mentions. our probing models consistently @xmath0, suggesting that span representations encode a significant amount of coreference information. besides, they show that fine-tuning a bert model greatly helps with encoding coreference relations. by ablating components of the span representation, we also find that the head-finding attention mechanism plays a crucial part in encoding important coreference information. finally, we show that despite using a fine-tuned bert, the span representations cannot capture non-local coreference relation efficiently. our implementation is publicly available foot_0 . ## conclusion and future work in this paper, we quantify the coreference information in the span representation by how well they can do on the coreference arc prediction task. we demonstrate that using mention-span representations as inputs, a simple probing model can be used to predict coreference for pairs of mention spans with accuracy and f1 score over 90%. this suggests that a significant amount of coreference information is encoded in mention-span representations obtained from bert embeddings, which are fine-tuned on the ontonotes dataset. consistently with non-neural architectures, our analysis also shows that non-local coreference is challenging for span representations. furthermore, we show that the head-finding attention mechanism encodes essential coreference-related features in span representations, even when using original pre-trained bert embeddings. the findings we report are solely based on an english corpus. other pieces of research @xcite suggest that such positive results might be more challenging to achieve for morphologically or syntactically complex languages. although we work with the ontonotes dataset, there are other challenging coreference resolution datasets that focus on ambiguous pronouns (gap by @xcite ) or commonsense reasoning (winogrande by @xcite ), which can be used to understand coreference information in span representations better. moreover, we would like to probe span representations derived from other pre-trained language models such as roberta @xcite and spanbert @xcite . alternative transformer-based architecture that is better at handling long sequences such as longformer @xcite also seems promising to explore, as it might improve span representations capability to model long-range coreference. lastly, instead of building span representations from the final layer of a pre-trained bert model, one can opt to use the activations from the intermediate layers as well as elmo-style scalar mixing @xcite . we leave this to future work.