Using Type Information to Improve Entity Coreference Resolution (2020)

Abstract: Coreference resolution (CR) is an essential part of discourse analysis. Most recently, neural approaches have been proposed to improve over SOTA models from earlier paradigms. So far none of the published neural models leverage external semantic knowledge such as type information. This paper offers the first such model and evaluation, demonstrating modest gains in accuracy by introducing either gold standard or predicted types. In the proposed approach, type information serves both to (1) improve mention representation and (2) create a soft type consistency check between coreference candidate mentions. Our evaluation covers two different grain sizes of types over four different benchmark corpora.

## introduction coreference resolution (cr) is an extensively studied problem in computational linguistics and nlp @xcite @xcite @xcite . solutions to this problem allow us to make meaningful links between concepts and entities within a discourse and therefore serves as a valuable pre-processing step for downstream tasks like summarization and questionanswering @xcite @xcite . recently, multiple datasets including ontonotes @xcite , litbank @xcite , @xcite , and wikicoref @xcite have been proposed as benchmark datasets for cr, especially in the sub-area of entity anaphora @xcite . entity anaphora is a simpler starting place for work on anaphora because unlike abstract anaphora @xcite , entity anaphora are pronouns or noun phrases that refer to an explicitly mentioned entity in the discourse rather than an abstract idea that must be constructed from a repackaging of information revealed over an extended text. an affordance of entity anaphora is that they have easily articulated semantic types. most of the entity cr datasets are extensively annotated for syntactic features (like constituency parse etc.) and semantic features (like entity-types). however, none of the published sota methods @xcite @xcite ) explicitly leverage the type information. in this paper, we present a proof of concept to portray the benefits of using type information in neural approaches for cr. named entities are generally divided generically (e.g. person, organization etc.) or in a domain-specific manner (e.g. symptom, drug, test etc.). in this work, we consider cr datasets that contain generic entitytypes. one challenge is that the different corpora do not utilize the same set of type tags. for example, ontonotes includes 18 types while emailcoref includes only 4. thus, we evaluate the performance of the proposed modeling approach on each dataset both with the set of type tags germaine to the dataset as well as a common set of four basic types (person, org, location, facility) inspired from research on named entity recognition (ner) @xcite . our motivation is similar to @xcite , which used a structured crf with handcurated features to jointly-model the tasks of cr, entity typing, and entity linking. their joint architecture showed an improved performance on cr over the independent baseline. however, our work differs from there's as we show the benefits of entity-type information in neural models that use contextualized representations like bert @xcite . some prior art @xcite argues that contextual- (1) type information is concatenated with the mention span representation created by their model; and (2) a consistency check is incorporated that compares the types of two mentions under consideration to calculate the coreference score. please refer to section 3 for details. our contribution is that we evaluate the impact of the introduction of type information in neural entity coreference at two different levels of granularity (which we refer to as original vs common), demonstrating their utility both in the case where gold standard type information is available, and the more typical case where it is predicted. ## model in this section, we explain how we introduce type information into a neural cr system. ## datasets we gauge the benefits of using entity-type information on the four datasets discussed below. litbank. this dataset @xcite ) contains coreference annotations for 100 literary texts. foot_0 this dataset limits the markable mentions to six entity-types, where majority of the mentions (83.1%) point to a person. emailcoref. this dataset (dakle et al., 2020) comprises of 46 email threads with a total of 245 email messages. 2 similar to litbank, it considers a mention to be a span of text that refers to a real-world entity. in this work, we filter out pronouns that point towards multiple entities in the email (e.g. we, they) thus only focusing on singular mentions. ontonotes. from this multi-lingual dataset, we evaluate on the subset (english) from ontonotes that was used in the conll-2012 shared task @xcite . 3 it contains 2802 training, 343 development, and 348 test documents. the dataset differs from litbank in its annotation scheme with the biggest difference being the fact that it does not annotate singletons. it contains annotations for 18 different entitytypes. however, unlike litbank and emailcoref, not all mentions have an associated entity-type. for example, none of the pronoun mentions are given a type even if they act as anaphors to typed entities. we partially ameliorate this issue by extracting gold coreference clusters that contain at least one typed mention and assigning the majority type in that cluster to all of its elements. for example, in figure 1 , if los angeles is typed place, and it is in the gold coreference cluster of los angeles (no other element in the cluster), then it is also assigned the type place. wikicoref. this corpus, released by (ghaddar and langlais, 2016), comprises 30 documents from wikipedia annotated for coreference resolution. 4 the annotations contain additional metadata, like the associated freebase rdf link for each mention (if available). we use this rdf entry to extract the mention's entity types from freebase dump. mentions that do not get any type are marked na. the first 24 documents are chosen for training, the next 3 for development, and the rest for testing. the above-discussed datasets differ in the number as well as the categories of entity-types they originally annotate ( ## experiments and results in this section, we provide the results of our empirical experiments. evaluation metrics: we convert all three datasets into the conll 2012 format and report the f1 score for muc, b 3 , and ceaf metrics using the conll-2012 official scripts. the performances are compared on the average f1 of the abovementioned metrics. for emailcoref, ontonotes, and wikicoref, we report the mean score of 5 independent runs of the model with different seeds. whereas, for litbank, we present the 10-fold cross-validation results. ## predicted types results shown in the previous section assume the presence of gold standard types during training as well as inference, which is often impractical in the real-world. most of the new samples that a cr model would encounter would not include type information about the candidate mentions. therefore, we set up an additional experiment to gauge the benefits of type information using predicted types. we introduce a baseline approach to infer the type of the mentions and then use these predictions in the +et models, in place of the gold types, for coreference resolution. ## conclusion in this work, we show the importance of using entity-type information in neural coreference resolution (cr) models with contextualized embeddings like bert. models which leverage type information, annotated in the corpus, substantially outperform the baseline on four cr datasets by reducing the number of type mismatches in detected coreference clusters. since, these datasets vary in number and categories of the types they define, we also experiment with mapping the original corpus types to four common types (per, org, loc, fac) based on previous ner research that can be learnt more easily through large ner datasets. models which use these common types perform slightly worse than original types but still show significant improvements over the baseline systems. the presence of gold standard types during cr inference is unlikely in practice. therefore, we propose a model that infers the type of a mention given the mention span and its immediate context to use along side the proposed cr approach. in our evaluation, we find that using types predicted by our model for cr still performs significantly better than the baseline, thus offering stronger evidence that type information holds the potential for practical improvements for cr.