D ynamic ER : Resolving Emerging Mentions to Dynamic Entities for RAG (2024)

Abstract: In the rapidly evolving landscape of language, resolving new linguistic expressions in continuously updating knowledge bases remains a formidable challenge. This challenge becomes critical in retrieval-augmented generation (RAG) with knowledge bases, as emerging expressions hinder the retrieval of relevant documents, leading to generator hallucinations. To address this issue, we introduce a novel task aimed at resolving emerging mentions to dynamic entities and present DynamicER benchmark. Our benchmark includes dynamic entity mention resolution and entity-centric knowledge-intensive QA task, evaluating entity linking and RAG model’s adaptability to new expressions, respectively. We discovered that current entity linking models struggle to link these new expressions to entities. Therefore, we propose a temporal segmented clustering method with continual adaptation, effectively managing the temporal dynamics of evolving entities and emerging mentions. Extensive experiments demonstrate that our method outperforms existing baselines, enhancing RAG model performance on QA task with resolved mentions.

## introduction in the real world, large amounts of textual information are constantly being generated at an incredible rate. the dynamic nature of human language, characterized by the continuous emergence of new expressions, presents multiple significant challenges @xcite . the way we refer to named entities changes over time, influenced by the shifts that include the use of metaphors, adoption of slang, creation of euphemisms, and other linguistic evolutions @xcite . for example, consider the named entity "elon musk". over time, various mentions are used to refer to him, such as "the tesla ceo", or "the tech billionaire". emerging slang or metaphoric expressions like "real-life figure 1 : motivation of our dynamicer benchmark. new mentions referring to the same entity are constantly created over time: as shohei ohtani transfers from the la angels to the la dodgers, he is referred to by new mentions such as 'the dodgers' number 17.' we contribute a dynamic entity resolution dataset, along with two benchmark tests: traditional entity linking and entity-centric question-answering in the rag context. iron man" or "mars man" also appear. as a dynamic entity, his attributes change over time as he was initially known as "paypal co-founder", later as "hyperloop visionary", and more recently as "twitter owner". all of these expressions refer to the same entity at different time points, yet a system must be able to recognize them despite the dynamic linguistic landscape. therefore, it is crucial for a system to resolve these new expressions, accurately linking them to evolving entities in a continuously updating knowledge base (kb). recently, kbs have been critically utilized in the retrieval-augmented generation (rag) framework @xcite @xcite , where a retriever fetches relevant documents from kbs, allowing a generator like large language models (llms) to provide accurate answers. as llms become capable of learning with prompt, studies have utilized rag for knowledge-intensive open-domain tasks. following this trend, efforts @xcite @xcite have been made to manage dynamically evolving knowledge with rag. in particular, some studies @xcite have shown that in time-sensitive knowledge-intensive benchmarks, llms inevitably produce outdated answers if retrieval fails. one significant reason for retrieval failure is the emergence of new mentions, which prevents the retriever from functioning properly @xcite . however, to the best of our knowledge, no approach has tackled the challenge of resolving new expressions of named entities evolving over time for the rag framework. to overcome this limitation, we call for a dynamic entity resolution task that links emerging mentions to dynamic entities. we contribute the dynamicer (dynamic entity resolution for emerging mentions) benchmark, designed to study the resolution of continuously evolving entities and their newly appearing mentions. as illustrated in figure 1 , dynamicer annotates emerging mentions found in social media documents, linking these mentions to corresponding named entities within a kb. dynamicer is structured as a sequence of time segments to evaluate models resolving mentions that are not recognized in earlier time steps. we also introduce a dynamic entity-centric question-answering (qa) task where named entities in question are substituted into emerging mentions over time. this qa task examines the impact of emerging mentions on the retriever's accuracy and aims to evaluate the end-to-end performance of the rag methods. when a new mention first appears, it may be challenging to resolve it using only a single document since it is likely to have high lexical variation and insufficient context. although prior works have used mentions in multiple documents collectively @xcite @xcite , there is a potential risk when jointly handling mentions from different time steps. the definition or attributes of entities may evolve across time steps, and neighbor mentions referenced together also change accordingly. to address this issue, we propose a method that continuously clusters entities and a set of mentions at each time step and updates the evolving entity cluster representations. our contributions are summarized as follows: ## approach resolving new expressions based solely on the document where they first appear can be challenging due to their low lexical similarity to the entity name and the ambiguity of context. thus, it can be advantageous to consider multiple documents that share similar contexts and expressions to resolve these mentions jointly. previous works have studied this joint clustering approach @xcite @xcite , but they assume a static scenario and resolve the mentions without considering the time dimension. in our problem, on the other hand, entities evolve over time, with changes in definitions or attributes such as status, role, affiliation, or characteristics. as exemplified in figure 1 , it is hard to find the coreference between "the angels' superstar" and "the dodgers' number 17" for shohei ohtani. instead, clustering mentions that appear at similar time steps could be feasible since they share events or contexts at similar time steps. therefore, we propose a temporal segmented clustering approach with continuous adaptation (tempcca), as shown in figure 2 . our approach follows the joint clustering methods as prior works but continuously clusters the emerging mentions at each time step and further utilizes the cluster representation to resolve mentions in the next time step. ## experiments we investigate the following research questions. 1. how well does our method resolve emerging mentions compared to existing entity linking and coreference methods? training and inference. we divide our entity linking dataset into disjoint training and test time steps in timeline; specifically, documents dated from 2023-05-01 to 2023-10-31 constitute the training time steps, while documents from 2023-11-01 to 2024-04-30 form the test time steps. for each training time step, we further split the data into training and validation sets. for soccer, across the time steps 0506, 0708, and 0910, we use 6681, 7042, and 7783 mentions for training, and 1574, 1775, and 1963 mentions for validation, respectively. for baseball, we use 1417, 2648, and 2592 mentions for training, and 308, 631, 662 mentions for validation, respectively. tempcca and c-spel undergo continuous training and inference. specifically, we divide the documents into two-month intervals, resulting in three cycles (0506, 0708, 0910) of continuous training and three cycles (1112, 0102, 0304) of continuous inference. ## conclusion in this work, we addressed the challenge of resolving new linguistic expressions in the dynamic and ever-evolving landscape of human language. we introduced dynamicer to evaluate the ability of models to resolve emerging mentions. our benchmark proposes entity linking tasks for resolving emerging mentions and entity-centric qa tasks for rag evaluation. to address the temporal dynamics of emerging mentions, we proposed a temporal segmented clustering method with continual adaptation. our exhaustive experiments demonstrated that our method surpassed existing baselines in resolving new expressions, particularly when there is less lexical overlap. future work may extend to updating kb using entity resolution, which can directly handle the retrieval failures caused by mention dynamics. through dynamicer, we provide a resource for the research community to further explore and improve dynamic entity resolution. we hope this fosters further research towards developing more robust models capable of handling the continuous emergence of new expressions.