L ing M ess: Linguistically Informed Multi Expert Scorers for Coreference Resolution (2023)

Abstract: Current state-of-the-art coreference systems are based on a single pairwise scoring component, which assigns to each pair of mention spans a score reflecting their tendency to corefer to each other. We observe that different kinds of mention pairs require different information sources to assess their score. We present LingMess, a linguistically motivated categorization of mention-pairs into 6 types of coreference decisions and learn a dedicated trainable scoring function for each category. This significantly improves the accuracy of the pairwise scorer as well as of the overall coreference performance on the English Ontonotes coreference corpus and 5 additional datasets.

## introduction coreference resolution is the task of clustering textual mentions that refer to the same discourse entity. this fundamental task requires many decisions. in this work, we argue that different kinds of decisions involve different challenges. to illustrate that, consider the following text: to correctly identify that the pronoun "he" refers to "lionel messi", models need to model the discourse, while linking "my" to "i" may rely more heavily on morphological agreement. likewise, linking "the argentinian footballer" to "lionel messi" requires world knowledge, while linking "messi" to "lionel messi" may be achieved by simple lexical heuristics. ("lionel messi", "he"). indeed, pre-neural coreference resolution works often considered the types of a mention-pair, either by incorporating this information as model features, or by tailoring specific rules or specific models for each mention pair (see related work section). however, neural-network based coreference models are all based on a single pairwise scorer that is shared for all mention pairs, regardless of the different challenges that needs to be addressed by each pair type @xcite @xcite @xcite @xcite @xcite @xcite . in this work, we suggest that modeling different mention pairs by different sub-models (in our case, different learned scoring functions) depending on their types is beneficial also for neural models. we identify a set of decisions: (a) linking compatible category co-referring example non co-referring example pron-pron-c a couple of my law clerks were going to ... and i was afraid i was going to... the lord god said to my lord: "sit by me at my right side , and i will put your enemies ... " pron-pron-nc "i made a similar line and i produced it cheaper" , he says. she is my goddess ... ent-pron spain, argentina, thailand and indonesia were doing too little to prevent ... across their borders. tonight, to kick off the effort, cnn will premiere its first prime -time newscast in years. match ... says paul amos, cnn executive vice president for programming. accordingly, cnn is ... hertz and avis can not benefit budget's programs, " said bob wilson, budget's vice president ... ## background: the s2e model the s2e model @xcite achieves the current best coreference scores among all practical neural models. 4 given a sequence of tokens x 1 , . . . , x n , each mention pair (c, q) is scored using a scoring function f s (c, q) foot_3 described below, where c is a "candidate span" and q is a "query span" which appears before c in the sentence. the span encodings are based on contextualized word embeddings obtained by a longformer encoder, see @xcite for details. these pairwise scores are then used to form coreference chains (see "inference" below). the scoring function f s is further decomposed: ## lingmess clustering coreferring entities typically involves many different phenomena, which we argue should be addressed in a different manner. therefore, our core contribution is proposing to allocate a dedicated scorer f t a (c, q) for each phenomenon type t, in addition to the shared pairwise scorer f a (c, q). the overall architecture of our model is shown in figure 1 . concretely, we extend the s2e model with six additional antecedent scorers f t a where @xmath0, pron-pron-nc, ent-pron, match, contains, other}, the six categories we list in the pairwise scoring function now becomes: the pairwise scoring function f (c, q) scoring c as the antecedent of q, is now composed of a shared scorer f a (c, q) and an "expert" scorer f t a (c, q) which differs based on the type of the pair c, q. each of the seven pairwise scoring functions (f a and the six f t a ) is parameterized separately using its own set of matrices. the transformer-based encoder and the mention scorer f m are shared between all the antecedent scorers. see appendix a.2 for full model architecture. training for each span q, our model optimizes the objective function l coref over the sum of probabilities of all true antecedents of q: here, p (ĉ | q) is a softmax over f (ĉ, q) scores, that is our new score function described in figure 1 . this scorer is also the one used in inference. however, this objective does not explicitly push each category ("expert") to specialize. for example, for the pron-pron-c cases, it would be useful to explicitly train the model to distinguish between the possible antecedents of that type only (without regarding other antecedents), as well as to explicitly distinguish between a pronoun antecedent and a null antecedent. to this end, we extend the training objective by also training each "expert" separately: note that for l t (q) we replace c(q) with c t (q), considering only the potential antecedents that are compatible with the span q for the given type. for example, for l match and a span q, we will only consider candidates c which appear before q with the exact same content words as q. our final objective for each mention span q is thus: inference at inference time, we compute the score of each mention based on the shared scorer as well as the per-type scorer matching the mention type. we then form the coreference chains by linking each mention q to its most likely antecedent c according to f (c, q). we do not use higher-order inference as it has been shown to have a marginal impact @xcite . ## experiments our baseline is the s2e model trained on the english ontonotes 5.0 dataset by its authors @xcite . we train lingmess also on ontonotes, and evaluate both models on ontonotes, wiki-coref, gap, winogender, winobias and bug. implementation details are described in appendix b. performance ## related work many pre-neural works consider the various linguistic phenomena involved in coreference resolution as a different challenge. the early coreference system by @xcite divided the antecedents candidates into distinct coreference categories (e.g., name alias, apposition, definite noun, and a few more) and defined tailored rules for each category. later, lee et al. ( 2013 ) proposed the multi-sieves deterministic model, where each sieve adds coreference links between mention pairs from a specific linguistic category (e.g string match, compatible pronoun, etc.). haghighi and klein (2009) performed an error analysis of their coreference model according to different types of antecedent decisions, such as proper noun-pronoun, pronoun-pronoun, etc. based on this analysis, they focus on fixing the pronoun antecedent choices by adding syntactic features. more recently, lu and ng (2020) analyze empirically the performance of neural coreference resolvers on various finegrained resolution categories of mentions (e.g gendered pronoun vs. 1st and 2nd pronoun). they find that while models perform well on name and nominal mention pairs with some shared content words, they still struggle with resolving pronouns, particularly relative pronouns. early supervised statistical models train a feature-based classifier that incorporates the type of antecedent decision (e.g. pronoun-entity, string match) as features at the mention-pair level @xcite @xcite . subsequently, denis and baldridge (2008) demonstrate that training separate classifiers that specialize in particular types of mentions (e.g third person pronouns, speech pronouns, proper names, definite descriptions, and all other) provides significant performance improvements. lassalle and denis (2013) took that observation a step further and proposed a more advanced method for model specialization by learning to separate types of mention into optimal classes and their proper feature space. in our work, we make progress in coreference systems specialization direction, and show that the incorporation of linguistic information is helpful also in the context of end-to-end neural models. ## conclusion we present lingmess, a coreference model that significantly improves accuracy by splitting the scoring function into different categories, and routing each scoring decision to its own category based on a deterministic, linguistically informed heuristic. this indicates that while end-to-end training is very effective, linguistic knowledge and symbolic computation can still be used to improve results.