Handling Missing Entities in Zero-Shot Named Entity Recognition: Integrated Recall and Retrieval Augmentation (2025)

Abstract: Zero-shot Named Entity Recognition (ZS-NER) aims to recognize entities in unseen domains without specific annotated data. A key challenge is handling missing entities while ensuring accurate type recognition, hindered by: 1) the pre-training assumption that each entity has a single type, overlooking diversity, and 2) insufficient contextual knowledge for type reasoning. To address this, we propose IRRA (Integrated Recall and Retrieval Augmentation), a novel two-stage framework leveraging large language model techniques. In the Recall Augmented Entity Extracting stage, we built a perturbed dataset to induce the model to exhibit missing or erroneous extracted entities. Based on this, we trained an enhanced model to correct these errors. This approach can improve the ZS-NER’s recall rate. In the Retrieval Augmented Type Correcting stage, we employ Retrieval-Augmented Generation techniques to locate entity-related unannotated contexts, with the additional contextual information significantly improving the accuracy of type correcting. Extensive evaluations demonstrate the state-of-the-art performance of our IRRA, with significant improvements in zero-shot cross-domain settings validated through both auto-evaluated metrics and analysis. Our implementation will be open-sourced at https://github.com/DMIRLAB-Group/IRRA .

## introduction zero-shot named entity recognition (zs-ner) aims to achieve efficient named entity recognition in unseen domains without requiring training on specific annotated datasets. zs-ner allows for effective entity detection and classification in various domains @xcite . the existing leading zs-ner approaches involve end-to-end training of a robust language model on large ner corpora to enhance its cross-domain generalization capabilities. * corresponding author, hpakyim@gmail.com figure 1 : the upper part shows the difference between typical dataset and perturbed dataset. the typical dataset usually assigns one single gold type to each entity, while the perturbed dataset assigns multiple types to each entity. the dashed boxes represent the scope of entity types sets corresponding to these entity types. the lower part illustrates that the entity types of the retrieved entities may be not precise enough. although promising results were reported, we observe that end-to-end solutions have two notable limitations that adversely impact their generalization performance: (i) the lack of diversity in the data annotation process. existing ner corpora typically assign only one type to each entity, whereas entities in real world may correspond to multiple types. this restricts the model's ability to learn the diversity of entities and understand the connections between different types during end-to-end training. as shown in figure 1 , given a sample similar in form to the training sample but with entities of different types, the semantic distance between the types "train institute", "institution" and "university" could lead to "pmi" being neither extracted nor recognized, or (ii) the lack of sufficient contextual information for entity recognition. as shown in figure 1 , unlike in the given training set sample, the context surrounding the entity "pmi" may be insufficient to determine its specific type. the key to overcome these limitations lies in decoupling the end-to-end processes to introduce specific augmentation mechanisms. instead of utilizing end-to-end methods, we propose the irra (integrated recall and retrieval augmentation), a two-stage zs-ner framework that strategically leverages large language models (llms) techniques. specifically, irra is featured with a recall augmented entity extracting stage and retrieval augmented type correcting stage. in recall augmented entity extracting stage, we introduce a data perturbation method by enriching each entity's type with synonyms and hypernyms, thereby constructing a perturbed dataset which model will make inference errors on it. we then use these errors to build an enhanced dataset to train an enhanced extractor, which can recall more entities. through the above process, as shown in figure 1 , the entity "stanford" in the training sample matched to multiple types, also increases the probability that the test sample entity "pmi" to be extracted, due to the introduction of the types "educational institution" and "institution", which are more closely related to "training institute". in retrieval augmented type correcting stage, we em-ploy retrieval-augmented generation (rag) techniques to retrieve contexts in the database related to the entities extracted in the previous stage. then we instruct a frozen llm to correct the entity's type based on the contextual information brought by the retrieved contexts. the main contributions of the paper are summarized as follows: ## methodology the overall irra framework is shown in figure 2 . the entire process of this framework is divided into two stages: recall augmented entity extracting stage and retrieval augmented type correcting. in recall augmented entity extracting stage, we first deploy an llm for entity extraction, then construct a perturbed dataset and train an llm on it to enhance the initial extraction results. this stage is responsible for providing entity extraction results for the next stage while also enabling multilabel recognition. in retrieval augmented type correcting stage, based on the extracted entities, corresponding queries are generated to retrieve and rerank contexts from the database. these contexts are then provided as additional contextual information to the type correcting model. ## experiment this section presents experiments conducted under zero-shot settings to validate the effectiveness of irra. the details of the experimental settings are described in the following parts. ## conclusion in this paper, we propose a two-stage irra framework for zero-shot named entity recognition (ner), strategically leveraging large language model techniques. the core idea of irra is to supplement missing information in entity extraction and recognition through the recall augmented entity extracting and retrieval augmented type correcting stages. experimental results show significant improvements over state-of-the-art approaches, highlighting the efficacy of multi-stage processing in large model applications.