Conundrums in Entity Coreference Resolution: Making Sense of the State of the Art (2020)

Abstract: Despite the significant progress on entity coreference resolution observed in recent years, there is a general lack of understanding of what has been improved. We present an empirical analysis of state-of-the-art resolvers with the goal of providing the general NLP audience with a better understanding of the state of the art and coreference researchers with directions for future research.

## introduction the advent of the neural nlp era has revolutionized virtually all areas of nlp research. for entity coreference, many issues that were once thought to be important no longer appear to be particularly relevant to the current research agenda. specifically, while a decade ago coreference researchers have focused on developing computational models that are complex (e.g., structured models @xcite @xcite ) and knowledge-rich (e.g., those that encode world knowledge @xcite @xcite ), nowadays virtually all state-of-the-art resolvers employ a simple model (i.e., the mentionranking model, which was developed more than a decade ago (denis and baldridge, 2008)) 1 and a fairly simple input representation (i.e., contextualized word embeddings) in conjunction with a mechanism for learning representations of entity mention spans such that coreferent mentions have similar representations @xcite @xcite . despite significant progress in the past few years in terms of performance numbers, what seems to be missing is an understanding of what has been improved. the lack of understanding has long been a concern shared by coreference researchers, even before the neural revolution in nlp. this has led to several attempts to analyze coreference resolvers over the years @xcite . with the development of neural resolvers, however, this concern has become more serious than ever: the fact that significant progress can be made via learning mention representations with a simple neural mention-ranking model that employs a fairly simple input representation for a task as challenging as coreference resolution (cr) is somewhat contrary to common wisdom. in light of this apparent conundrum, we present an empirical analysis of state-of-the-art entity coreference resolvers through four major sets of experiments in this paper, with the goal of gaining insights into their behaviors. we believe that our analysis will not only provide the general nlp audience with a better understanding of the state of the art, but also provide coreference researchers with directions for future research. ## evaluation setup in this section, we describe the datasets, the evaluation metrics, the state-of-the-art resolvers and the hyperparameters used in our experiments. datasets. we report results on three coreference datasets. the nist-sponsored ace evaluations resulted in several datasets. we use ace 2005 @xcite , the last one in the series. the ace 2005 organizers have only made the official training set (but not the official test set) publicly available, so previous work defined different traintest splits over the official training set. we employ the same train-test split as @xcite . kbp is another series of nist-sponsored evaluations in the mid 2010s. kbp does not have any evaluations on entity cr, but to support high-level information extraction tasks (e.g., event extraction, event cr), the organizers have made available several corpora that include entity coreference annotations. for training, we use three such corpora (ldc2015e29, ldc2015e68, and ldc2016e64). for evaluation, the kbp 2017 organizers have made available the official test set for the event cr task (ldc2017e51), which also include entity coreference annotations. we use it as our test set. ontonotes @xcite , which was developed circa 2006, is the most widely-used dataset for entity coreference evaluations. it has a standard train-dev-test split. unlike in ace and kbp, singleton clusters are not annotated in ontonotes. the key difference between these three corpora is that ontonotes supports "unrestricted" cr, meaning that coreference links are annotated between entity mentions without regard to their entity types. in contrast, coreference links are only annotated between mentions belonging to one of the seven entity types in ace and one of the five entity types in kbp. statistics on these corpora are shown in systems. we evaluate five variants of three stateof-the-art neural resolvers, all of which employ a ranking model where all candidate antecedents are ranked against each other for a given anaphor. the first resolver, the stanford neural resolver @xcite 3 , takes as input a set of entity mentions identified for a given document by a rule-based md system and trains using reinforcement learning a simple mention ranker consisting of three hidden layers of relu units and a final layer that is fully-connected. the other two resolvers are developed by @xcite 4 and @xcite 5 . both are span-based models, which have two key characteristics. first, mention spans are identified as part of cr, so this mitigates the propagation of errors from md to cr. second, representations of entity mention spans are learned so that coreferent mentions have similar representations. the key differences between these resolvers are: (1) in @xcite . the input instances correspond to the sentences in the given document, whereas in joshi et al. the input instances correspond to fixed-length non-overlapping segments of the input document foot_4 ; (2) lee et al. use a lstm, whereas joshi et al. use a transformer; and (3) the pretrained embeddings are different. we use the publicly-available implementation of each of these resolvers. there is one caveat, however. recall that the span-based resolvers were all evaluated on ontonotes. since singleton clusters are not annotated in ontonotes, all singleton clusters predicted by a resolver are removed from its output before it is sent to the scoring program. in contrast, singleton clusters that contain mentions belonging to one of the ace/kbp entity types are annotated in ace/kbp, so these mentions should not be removed from a resolver's output. however, span-based resolvers cannot distinguish between spans that correspond to entity mentions and those that do not. to address this problem, we extend the span-based models so that they are jointly trained to predict entity mention spans and coreference links. specifically, the feedforward neural network that is responsible for scoring a span in these models currently do not receive direct feedback on whether a span corresponds to an entity mention. we first turn it into a mention detector by training it in a supervised manner using the negative cross entropy loss, so that it predicts a positive mention score for a span if and only if the span corresponds to an entity mention. then, to jointly learn md and cr in the span-based resolvers, we employ a loss function that is the unweighted sum of the coreference loss and the md loss. hyperparameter tuning. to ensure a fair comparison of the resolvers, we tune their hyperparameters to maximize the conll score on development data. note, however, that the authors of stanford, elmo, spanbert-b, and spanbert-l reported the best hyperparameter settings on ontonotes in the original papers @xcite , so we simply use them in our experiments and focus on tuning the hyperparameters for the remaining cases. we adopt the set of hyperparameters to be tuned from the original papers. for stanford, there are three hyperparameters to tune: α w l , α f a , and α f n . these are the weights associated with three different types of mistakes made by the coreference model. following @xcite , we fix α w @xmath0.0 and search for α f a and α f n out of {0.1, 0.2, . . ., 1.5} using a variant of grid search. for ace, (α w l , α f a , α f n ) = (1.0, 0.5, 1.0) is the best configuration, and for kbp, the best configuration is (1.0, 0.5, 0.8). for ontonotes, we use the configuration found by clark and manning, which is (1.0, 0.5, 0.8). for glove and elmo, we have five hyperparameters to tune. specifically, we search for: (1) max span width (i.e., maximum number of words in a candidate span) out of {10, 20, 30}; (2) max top antecedents (i.e., maximum number of candidate antecedents) out of {35, 40, 45, 50}; (3) max training sentences out of {25, 50, 75, 100}; (4) task learning rate out of {5e-4, 1e-3, 2e-3}; and (5) top span ratio (i.e., the fraction of top spans that survive the filtering) out of {0.3, 0.35, 0.4, 0.45, 0.5}. for the two spanbert resolvers, we have seven hyperparameters to tune. for three of the hyperparameters (max span width, max top antecedents, and top span ratio), the ranges are the same as those used in glove and elmo. for the remaining four, we search for: (1) max training segments out of {3, 4, 5}; (2) max segment length out of {128, 256, 384, 512}; (3) spanbert learning rate out of {1e-5, 2e-5}; and (4) task learning rate out of {1e-4, 2e-4, 3e-4}. ## performance across datasets we first provide the reader with a high-level understanding of the state of the art by analyzing the five resolvers' performance on the three datasets. performance across datasets. results on the three datasets, which are reported in terms of the conll score, are shown in the conll column in in particular, the use of elmo embeddings enables elmo to outperform glove by 3.8-4.9% points. spanbert-b outperforms elmo by 1.6-4.4% points, and spanbert-l further outperforms spanbert-b by 2.2-4.5% points. source of performance improvements. do the above improvements stem from improved recognition of coreference links, or improved recognition of singleton clusters, or both? to understand whether these resolvers have improved in terms of link prediction, we examine the muc f-scores (see the muc column), which are computed solely on coreference links. as we can see, the muc scores are consistently increasing down the table across all datasets, meaning that later systems are indeed doing better at identifying coreference links. to understand whether later resolvers are also better at identifying singleton clusters, we show in the singleton column the percentage of singleton clusters that are correctly recalled. again, the scores are increasing down the table, and the degree of improvement is particularly large from glove to elmo and from spanbert-b to spanbert-l. ## using oracles can the performance of coreference resolvers be further improved if we improve md? being able to answer this kind of questions is important: if further improvements in md can result in significant gains in coreference performance, then future research efforts should perhaps be focused on md. to answer this kind of questions, we perform oracle experiments. specifically, we provide a resolver with a particular type of perfect information (e.g., using gold mentions as input) and see how much performance improvement can be obtained. ## results on resolution classes to gain additional insights into the state-of-theart resolvers, we analyze their performance on different types of entity mentions. more specifically, motivated by @xcite , we partition the gold mentions into different resolution classes. while previous work has focused mainly on three coarse-grained resolution classes (namely, pronouns, names, and nominal mentions), we employ the 13 fine-grained resolution classes defined by @xcite , as discussed below. names. four classes are defined for gold names. first, if we consider only the three coarsegrained resolution classes, the results are perhaps not surprising: name resolution is the easiest and nominal resolution is the hardest. second, consider the 13 fine-grained resolution classes. by design, the names and the nominals in the 'e' class should be easier to resolve than those in 'p', which in turn should be easier to resolve than those in 'n'. the results are consistent with this intuition. results on the anaphoric pronoun classes are also consistent with our intuition: 3rd person gendered pronouns are the easiest to resolve, followed by 1st/2nd person gendered pronouns and then ungendered 3rd person pronouns. third, these results reveal that the difficulty of anaphoricity determination stems primarily from pronouns: while resolution accuracies on nonanaphoric names and nominal mentions are above 89%, those on non-anaphoric pronouns are only between 65.9% and 77.6%. note that we consider a non-anaphoric mention correctly "resolved" if it is resolved to the dummy antecedent. finally, spanbert-l has better resolution accuracies than elmo for all resolution classes on all datasets. encouragingly, the harder a resolution class is, the bigger the improvement is. these results clearly show that we are making progress on resolving anaphors that are traditionally considered difficult to resolve. note that part of this improvement can be attributed to improved md, which increases the likelihood that the correct antecedent of an anaphor is present in its list of candidate antecedents. additional experiments are needed to determine the impact of improved md on improvement in resolution accuracies, however. ## sensitivity to perturbed inputs next, we conduct a series of experiments that involve perturbing the input. in each experiment, we (1) replace a certain kind of words/phrases in each training document with other words/phrases, (2) train a coreference model on these perturbed training documents, and (3) evaluate the output. our goal is to gain insights into the behavior of state-ofthe-art resolvers by examining how sensitive their performance is to perturbations in the input. specifically, if performance drops significantly when a particular kind of words/phrases is replaced, that means the replaced words/phrases are important in the model learning process. note that perturbations are only applied to the training documents; no changes are made to the test documents. we divide the different kinds of perturbations into two broad categories, mention-internal perturbations and mention-external perturbations. ## conclusions while space limitations preclude a reiteration of all the observations we have made, we believe the key conclusions are: (1) the relative performances of the resolvers are consistent across datasets; (2) for each resolver, higher mention detection performance always yields better coreference performance; (3) the newest resolvers perform better because of not only improved mention detection, but also improved mention span representations, and they improved the resolution of both easyand difficult-to-resolve anaphors; (4) all resolvers can be improved by improving mention detection, anaphoricity determination, and entity type detection; and (5) our perturbation results suggest that coreference performance is most sensitive to those words/phrases in the input that have the greatest impact on mention detection performance. rui zhang, cícero nogueira dos santos, michihiro yasunaga, bing xiang, and dragomir radev. 2018. neural coreference resolution with deep biaffine attention by joint mention detection and mention clustering. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 102-107.