Major Entity Identification: A Generalizable Alternative to Coreference Resolution (2024)

Abstract: The limited generalization of coreference resolution (CR) models has been a major bottleneck in the task’s broad application. Prior work has identified annotation differences, especially for mention detection, as one of the main reasons for the generalization gap and proposed using additional annotated target domain data. Rather than relying on this additional annotation, we propose an alternative referential task, Major Entity Identification (MEI), where we: (a) assume the target entities to be specified in the input, and (b) limit the task to only the frequent entities. Through extensive experiments, we demonstrate that MEI models generalize well across domains on multiple datasets with supervised models and LLM-based few-shot prompting. Additionally, MEI fits the classification framework, which enables the use of robust and intuitive classification-based metrics. Finally, MEI is also of practical use as it allows a user to search for all mentions of a particular entity or a group of entities of interest.

## introduction coreference resolution (cr) is the task of finding text spans that refer to the same entity. cr is a fundamental language understanding task relevant to various downstream nlp applications, such as question-answering @xcite , building knowledge graphs @xcite , and summarization @xcite . despite the importance of cr and the progress made by neural coreference models @xcite @xcite , domain generalization remains an issue even with the bestperforming supervised models @xcite . the lack of domain generalization in cr models can largely be attributed to differences in annotation guidelines of popular cr benchmarks, 1 code for the paper is available at https://github.com/ kawshikmanikantan/mei ## supervised mei models we propose meira, major entity identification via ranking, which draws inspiration from the entity ranking formulation @xcite and maintains an explicit representation for entities. the meira models consist of 3 steps: encoding the document, proposing candidate mentions, and an identification (id) module that tags mentions with major entities or the null entity. document encoding is performed using a longformer-large @xcite , ϕ, that we finetune for the task. mentions (or spans) are encoded as m @xmath0, d) by concatenating the first, last, and an attention-weighted average of the token representations within the mention span. in mei, an additional input is the set of designative phrases p for the major entities. since each phrase is derived from the document itself, we also obtain its encoding using the backbone: e @xmath1, d). mention detection. similar to prior efforts @xcite , we use a mention proposal network that predicts high-scoring candidate mentions. this step finds all mentions m all and not just the ones corresponding to the major entities m.training a model to only detect mentions of major entities would confuse it leading to poor performance. identification module. as illustrated in fig. 2 , we initialize a working memory e @xmath2. given a mention m i , the id module computes the most likely entity as: following @xcite , the mention detection and identification modules are trained endto-end using separate cross-entropy loss functions. ## few-shot mei with llms we propose a prompting strategy to leverage llms for mei, addressing their challenges in cr. mention detection challenges. cr or mei can be addressed using separate few-shot prompting strategies for mention detection and mention clustering/identification. however, @xcite found that this strategy faced significant challenges with mention detection, performing worse than a deterministic mention detector. thus, they assume access to an oracle mention detector and focus on evaluating llms' linking capabilities. an alternative is to use an external supervised mention detector instead of the oracle. however, this requires annotated training data and may not align with a true few-shot llm prompt paradigm. additionally, supervised mention detectors often fail to generalize across cr datasets due to annotation variability @xcite . mei with llms. we demonstrate that transitioning from cr to mei addresses this gap in mention detection and proposes an end-to-end, few-shot prompting approach for mei. inspired by dobrovolskii (2021), we develop a prompting strategy that first performs mei at word-level (rather than span), followed by a prompt to retrieve the span corresponding to the word. in addition to the document d and the set of phrases p, we also provide entity identifiers (e.g. #1, #2) to the llm. we will use the following example: document: that lady in the bmw is alice's mom. major entities: 1. alice; 2. alice's mother. prompt 1. word-level mei. mention detection with llms is challenging due to the frequent occurrence of nested mentions. we overcome this by prompting the llm to tag each word. specifically, through few-shot examples, we ask the llm i is above the threshold τ , m i is associated with the highest scoring major entity e * i or discarded. in meira-s, the entity memory e w remains static. for meira-h (blue path), the assigned entity's working memory is updated, and both the static (top half) and updated working memory (bottom half) are utilized to compute a final score. to detect and tag the syntactic heads foot_0 (e.g., lady, alice, mom) of mentions that refer to the major entities. other words are left untagged (implicitly assigned to ∅, the null entity). to create the fewshot examples, a contiguous set of words annotated with the same entity is considered as a span and its syntactic head is extracted using spacy @xcite . the ideal output for the example above is: "that lady#2 in the bmw is alice#1's mom#2..". note that, even though the span "bmw" might be a valid mention, it is not annotated as it does not refer to one of the major entities. the exact prompt used for this is provided in the appendix, prompt 2. head2span retrieval. the entity tagged heads are passed to the head2span (h2s) module, along with the document to retrieve the span. the prompt consists of the document pre-annotated with the positions of the head, where each candidate head-word is followed by a "#" and is instructed to be replaced by the complete span (including any existent determiners and adjectives). for the input: that lady# in the bmw is alice#'s mom#. the expected ideal output is that lady (that lady in the bmw) in the bmw is alice(alice's)'s mom (alice's mom). preserving structure. we pose mei as a structured generation task, prompting llms to reproduce documents and generate mei tags at specific locations. proprietary models like gpt-4 generally reproduce documents faithfully but for rare failures, we use the needleman-wunsch algorithm @xcite to align documents and extract tags in the case of open-source models, we employ regular expression-based constrained decoding with the outlines library (willard and louf, 2023) macro-f1 is the average f1-score of entities across the dataset, while micro-f1 is the frequencyweighted f1-score of entities within a document, averaged across the dataset. ## related work neural models for cr have become the de facto choice in supervised settings @xcite @xcite . efforts to enhance model efficiency include reducing candidate mentions to word-level spans @xcite and using single dense representations for entity clusters @xcite . generalization in cr remains a lingering problem @xcite @xcite . current solutions include feature addition @xcite , joint training @xcite , and active learning @xcite @xcite . rather than relying on additional training data, we argue for an alternative formulation where the burden of domain adaptation is offloaded from training to inference. llm evaluation on referential tasks has largely been conducted in limited settings, such as the sentence-level winograd schema challenges (wsc) @xcite , clinical pronoun resolution @xcite and instance-level q&a @xcite . le and ritter (2023) conducted the first document-level evaluation of llms for cr but assumed an oracle-mention detector. in contrast, we conduct end-to-end evaluations. entity-centric tasks similar to mei include character identification, where either annotations are restricted to a subset of entities @xcite or custom models are developed to extract mentions of specific characters from tv show transcripts @xcite . we differ from these works by adopting a generalized task formulation independent of annotation strategies and entity selection. another task, entity linking @xcite extracts distinct entities from a document and links them to external knowledge bases. in contast, mei focuses on retrieving all mentions (including nominals and pronominals) of a specific set of key entities, extracted solely from the context of the document. ## conclusion cr models are limited in their generalization capabilities owing to annotation differences and general challenges of domain adaptation. we propose mei as an alternative to cr, where the entities relevant to the input text are provided as input along with the text. our experiments demonstrate that mei is more suited for generalization than cr. additionally, mei can be viewed as a classification task that enables the use of intuitive metrics. a trivially parallelized variation (meira-s), gives a 25x speedup over a comparable cr model, making it more suitable for longer narratives. unlike cr, the formulation of mei allows few-shot prompted llms to effectively compete with trained models. our novel two-stage prompting and robust baseline methods empower top-performing llms like gpt-4 to achieve this. our analysis indicates that this task holds promise for effectively evaluating the long-context referential capabilities of llms in an end-to-end manner. potential applications of mei include domains such as film and literature, where metadata about salient entities can be sourced from external databases like imdb or sparknotes. additionally, mei can be applied to the analysis of documents like of financial and legal reports, when the user is familiar with the relevant entities. lastly, recent research @xcite indicates that llms can assist or automate the extraction of salient entities, a direction we intend to explore in future work.