I dentify M e: A Challenging Long-Context Mention Resolution Benchmark for LLM s (2025)

Abstract: Recent evaluations of LLMs on coreference resolution have revealed that traditional output formats and evaluation metrics do not fully capture the models’ referential understanding. To address this, we introduce IdentifyMe, a new benchmark for mention resolution presented in a multiple-choice question (MCQ) format, commonly used for evaluating LLMs. IdentifyMe features long narratives and employs heuristics to exclude easily identifiable mentions, creating a more challenging task. The benchmark also consists of a curated mixture of different mention types and corresponding entities, allowing for a fine-grained model performance analysis. We evaluate both closed- and open-source LLMs on IdentifyMe and observe a significant performance gap (20-30%) between the state-of-the-art sub-10B open models vs. closed ones. We observe that pronominal mentions, which have limited surface information, are typically harder for models to resolve than nominal mentions. Additionally, we find that LLMs often confuse entities when their mentions overlap in nested structures. The highest scoring model, GPT-4o, achieves 81.9% accuracy, highlighting the strong referential capabilities of state-of-the-art LLMs while also indicating room for further improvement.

## introduction coreference resolution (cr) consists of identifying the entity mentions and clustering them based on the entity identity. it is a fundamental task for text comprehension and can therefore be used to assess a model's textual understanding. while llms have made tremendous strides on a wide array of nlp tasks @xcite @xcite , their performance on cr has been relatively underwhelming. it remains instruction: read the text given below. the text has an entity mention marked within """ {{mention}} (#this is the marked mention) """. extract the mention and find who/what the mention refers to in the text. text: the residence of mr. peter pett , the well-known financier , on riverside drive is one of the leading eyesores of that breezy and expensive boulevard . . . . . . for the thousandth time he felt himself baffled by this calm , goggle-eyed boy who treated him with such supercilious coolness . " you ought to be out in the open air this lovely morning , " he said feebly . " all right . let 's go for a walk . i will if you will . " " i -i have other things to do , " said mr. pett , recoiling from the prospect . " well , this fresh-air stuff is overrated anyway . where 's the sense of having a home if you do n't stop in it ? " " when i was your age , i would have been out on a morning like this -er -bowling my hoop . " " and look at you now ! " " what do you mean ? " " martyr to lumbago . " " i am not a martyr to lumbago , " said mr. pett , who was touchy on the subject . " have it your own way . all i know is -" " never mind ! " " i 'm only saying what mother . . . " " be quiet ! " ogden made further researches in the candy box . " have some , pop ? " " no . " " quite right . got to be careful at your age . " " what do you mean ? " " getting on , you know . not so young as you used to be . come in , pop , if you 're coming in . there 's a draft from that door . " mr. pett retired , fermenting . he wondered how another man would have handled this situation . the ridiculous inconsistency of the human character infuriated him . why should he be a totally different man on riverside drive from the person he was in pine street ? why should he be able to hold his own in pine street with grown men -whiskered , square-jawed financiers -and yet be unable on riverside drive to eject {{a fourteen-year-old boy}} (#this is the marked mention) from an easy chair ? . . . . . . ## identifyme benchmark identifyme is an mcq-based benchmark where, given a text document with a marked mention, the task is to identify the entity the mention refers to. we derive these mentions from two coreference datasets focused on literary texts: litbank and fantasycoref. these datasets provide long contexts (1700 words on average for fantasycoref and 2000 words for litbank) and feature multiple entities with rich inter-dependencies (e.g., mr. and mrs. pett) that make resolving mentions more challenging. while litbank offers diverse writing styles and linguistic structures, fantasycoref includes entities that often take on different forms (e.g., disguises and transformations), or undergo title change (e.g., prince rudolph is called the emperor after his coronation), which further complicates entity mapping. coreference annotations cluster mentions that refer to the same entity, but creating an mcq requires a representative phrase for each entity cluster. we use gpt-4o-mini (see to prevent confusion, we discard and avoid labeling clusters that: (i) contain annotation errors (e.g., due to cluster merging or splitting (kummerfeld and klein, 2013)); (ii) are too small (< 3 mentions) or difficult or ambiguous to label (e.g., entitites like some money); (iii) are plural, as they often lack explicit surface forms that can be derived from mentions. an mcq is created from a document using mentions from labeled clusters, with all labeled entities provided as options. to ensure benchmark quality, we exclude short context documents (< 1000 words) or those where the discarded entities represent more than 50% of the mentions. ## experiments models. among closed-source models, we evaluate gpt-4o (openai, 2024a), gpt-4o-mini (openai, 2024b), and gemini-1.5-flash 3 (gemini @xcite . due to computational constraints, we limit the evaluation of open-source models to sub-10b variants: llama-3.1-8b (meta-ai, 2024) and mistral-7b @xcite . ## conclusion we present identifyme, a challenging mcq benchmark designed for the evaluation of the referential capabilities of llms. our analysis reveals several key challenges for llms, including: (i) pronominal resolution which has limited surface form information, (ii) questions where "none of the above" is the correct answer, and (iii) nested mentions that require distinguishing between overlapping spans. gpt-4o scores 81.9% on identifyme, highlighting the strong referential capabilities of frontier llms while still leaving ample room for improvement. we believe the identifyme benchmark, with its curated mix of diverse and challenging mentions, will serve as an effective tool for fine-grained assessment of stateof-the-art llms' referential capabilities.