Separately Parameterizing Singleton Detection Improves End-to-end Neural Coreference Resolution (2024)

Abstract: Current end-to-end coreference resolution models combine detection of singleton mentions and antecedent linking into a single step. In contrast, singleton detection was often treated as a separate step in the pre-neural era. In this work, we show that separately parameterizing these two sub-tasks also benefits end-to-end neural coreference systems. Specifically, we add a singleton detector to the coarse-to-fine (C2F) coreference model, and design an anaphoricity-aware span embedding and singleton detection loss. Our method significantly improves model performance on OntoNotes and four additional datasets.

## introduction coreference resolution (cr) is the task of identifying and clustering linguistic expressions that refer to the same real-world entity. recent progress in cr has been led by various end-to-end (e2e) neural models @xcite @xcite @xcite which significantly outperform older pipelined systems. many of these e2e models follow the design of @xcite , jointly training both a mention detector that extracts candidate mentions from all text spans and a mention linker that assigns the antecedent to each candidate mention. despite their impressive performance, these e2e cr models are far from perfect: replacing either the mention detector or linker with an oracle results in a substantial improvement of the entire model @xcite . this indicates room for improving the mention detector and linker in the current joint systems. indeed, modeling cr as mention detection followed by mention linking is not a clear decomposition because mention linking itself is composed of two sub-tasks: singleton detection and antecedent 1 our code is available at https://github.com/ xiyuanzou/c2f-sd linking. singletons are mentions that refer to entities which only appear once in the discourse and are often removed from model predictions because they are not coreferring. it is important to correctly distinguish anaphoric mentions from singletons since singletons account for the majority of mentions: over 80% of mentions in the development set of ontonotes are singletons @xcite . nevertheless, prior work shows that current mention detectors lack the ability to make such anaphoricity decisions @xcite . thus, the mention linker in current joint systems performs two tasks: it not only links anaphora with antecedents, but also identifies singletons by linking them to the empty antecedent. singleton detection and antecedent linking, however, are two disparate tasks that may require different representations and relying on a single module hurts their performance. @xcite further note that the mention linker increases its confidence in assigning coreference scores when not tasked with singleton detection. incorporating an extra singleton detector is a straightforward solution and has been extensively investigated in the pre-neural era for pipelined systems @xcite @xcite . in this work, we show that it is also effective for neural end-to-end cr models. we extend the coarse-to-fine (c2f) model @xcite by adding a separately parameterized singleton detector between the mention detector and linker. the singleton detector takes in the top-scoring candidate mentions extracted by the mention detector and predicts a singleton score for each candidate mention. candidate mentions with the highest singleton scores are pruned out before being fed into the mention linker. it is notable that the anaphoricity decision is more challenging than the mention decision because the former requires not only the information from the mention itself but also contextual clues from its potential antecedents. we concatenate the mention embedding with an anaphoricity-aware span embedding produced from a span-level attention that explicitly attends to itself and all of its preceding candidate mentions. in addition, we add a singleton detection loss to explicitly supervise the singleton detector during the joint training. we show in sec. 4 that these components are necessary, and there is little performance gain without them. @xcite show that using separate encoders for each sub-task greatly improves the overall task of entity and relation extraction. inspired by their work, we set up an expert representation learner for each of the mention detection, singleton detection, and antecedent linking tasks. in addition, we add a shared representation learner between these three sub-tasks as these sub-tasks are related and may benefit from certain shared features during training. the overall architecture of our model is shown in figure 1 . although c2f is used as the base, our method is general and can be applied to any cr model that follows the mention detector-linker architecture. we show that our model gains significant improvements on ontonotes and four additional datasets compared to the base model and achieves a new sota among all detector-linker models. we also scale up our model to 2b parameters, outperforming the 11b asp model @xcite and approaching the current sota seq2seq model @xcite while being considerably smaller in size and faster at inference. ## background: the c2f model in this section, we introduce the c2f model @xcite , which is one of the first e2e neural cr models. it was later outperformed by the lingmess model @xcite which is the current best model that follows the mention detector-linker architecture. recently proposed seq2seq approaches @xcite @xcite also achieve substantially higher accuracy, but they require significantly more resources and have slower inference speed. we will show that with our method of separately parameterizing a singleton detector, the old c2f model can significantly outperform the lingmess model and narrow the gap with the sota seq2seq model. the c2f model computes a span embedding v q for each text span q. let x i be the contextual representation of the ith token produced by the hf attention mention detector hf attention vmd i vsd i sl attention vana i singleton detector candidate mentions hf attention antecedent linker candidate nonsingleton mentions val i mention detector singleton detector antecedent linker input document d language model encoder token representation x i ffnn md ffnn sd ffnn al ffnn shared encoder xmd i xsd i xal i figure 1 : a high-level overview of our model. hf attention refers to the head-finding attention proposed by @xcite and sl attention is the span-level attention to make anaphoricity-aware embeddings. llm encoder. each span embedding consists of the representation of the start token x qs , the end token x qe , and the weighted average of all tokens within the span xq computed via a head-finding attention, and a feature vector ϕ q encoding the span width: c2f first consists of a mention detector, which is essentially a feed forward network that computes a mention score s m for each span based on the span embedding: those spans with the highest mention scores are retained as candidate mentions. for each candidate mention i, c2f roughly scores the possible antecedents of i by a lightweight bilinear function and keeps a constant number of top-scoring spans as its candidate antecedents y(i). the antecedent linker, which is another feed forward network, then computes an antecedent score s a between i and each of its candidate antecedents j: during training, c2f optimizes the marginal loglikelihood of each candidate mention i being assigned all of its unpruned gold antecedents @xmath0: ## methodology our core contribution is to add a singleton detector to the c2f architecture. to exploit the similarities and differences between distinct sub-tasks of cr, we build an expert representation learner for each of the mention detector (md), singleton detector (sd), and antecedent linker (al) and also a general representation learner shared between them. the representation of each token x i for each sub-task t is the concatenation of the expert and the shared representation: we follow the same approach as c2f to create a span embedding v tq for each sub-task t. additionally, we make an anaphoricity-aware embedding to improve the ability of the singleton detector to make anaphoricity decisions. for this, we use additive attention @xcite , but applied on the span-level where each candidate mention i attends to itself and all of its preceding unpruned candidate mentions: the singleton detector computes a singleton score s s for each candidate mention using both the anaphoricity-aware embedding and the original span embedding: the top k percentile of spans with highest singleton scores are identified as singletons and pruned out. we keep the antecedent linker unchanged and the final pairwise coreference score s(i, j) now becomes the sum of the mention scores and the antecedent score minus the singleton scores. we further introduce a singleton detection loss to explicitly supervise the singleton detector: ## experiments dataset we train and evaluate on the ontonotes 5.0 english dataset @xcite and four additional datasets: wikicoref @xcite , ontogum @xcite , gap @xcite and winobias @xcite . these datasets do not annotate singletons and thus require models to filter out any potential singletons. baseline we re-implement and re-train the c2f model @xcite as a baseline and build our model upon it. the original c2f model comes with a higher-order inference step which we do not include as it marginally affects performance @xcite . we also re-implement the recently developed lingmess model @xcite as a stronger baseline. in addition, we compare our model to the asp @xcite at 11b and the link-append model at 13b parameters @xcite . unfortunately, we do not have enough resources to train these large seq2seq pretrained encoder we use deberta-large @xcite as the pretrained encoder for the c2f baseline and our model since deberta outperforms other pretrained encoder models for cr @xcite . to compete with seq2seq models that are considerably larger, we scale up our model by using deberta-v2-xxl. main results although there is still a gap of 0.7 to the 13b link-append model, our model is about 6.5 times smaller and 95 times faster in inference speed, thus more practical to use. importance of singleton detector to assess that the improvement of our model is due to the independent parameterization of singleton detection rather than the added parameters, we increase the parameter count of the original c2f model by adding extra layers to its mention linker to match the number of parameters of our model. we observe that simply adding more parameters to the mention linker without separately parameterizing singleton detection surprisingly results in a 0.2 absolute drop of conll f1 score on ontonotes. ## conclusion we decouple the singleton detection and the antecedent linking in the current detector-linker cr models by separately parameterizing a singleton detector. the effectiveness of our method shows that a separate singleton detection step benefits neural end-to-end cr systems. this also points out a future research direction: how to build a stronger singleton detector in end-to-end systems.