Z ero NER : Fueling Zero-Shot Named Entity Recognition via Entity Type Descriptions (2025)

Abstract: What happens when a named entity recognition (NER) system encounters entities it has never seen before? In practical applications, models must generalize to unseen entity types where labeled training data is either unavailable or severely limited—a challenge that demands zero-shot learning capabilities. While large language models (LLMs) offer extensive parametric knowledge, they fall short in cost-effectiveness compared to specialized small encoders. Existing zero-shot methods predominantly adopt a relaxed definition of the term with potential leakage issues and rely on entity type names for generalization, overlooking the value of richer descriptions for disambiguation. In this work, we introduce ZeroNER, a description-driven framework that enhances hard zero-shot NER in low-resource settings. By leveraging general-domain annotations and entity type descriptions with LLM supervision, ZeroNER enables a BERT-based student model to successfully identify unseen entity types. Evaluated on three real-world benchmarks, ZeroNER consistently outperforms LLMs by up to 16% in F1 score, and surpasses lightweight baselines that use type names alone. Our analysis further reveals that LLMs derive significant benefits from incorporating type descriptions in the prompts.

## introduction named entity recognition (ner) involves identifying entity mentions and assigning them to predefined types (classes). despite the upswing in harnessing the expressive power of pre-trained language models, current solutions mostly demand large-scale datasets and prioritize a handful of commonly occurring types (e.g., person, location, organization) @xcite . real-world ner applications frequently grapple with specialized domains and lack readily available labels for nonstandard types, with new targets constantly emerging. in this context, adopting or periodically retraining state-of-the-art (sota) models becomes impractical. thus, the impetus shifts to zero-shot learning (zsl) @xcite @xcite , where networks must generalize to entity types not experienced during training by effectively transferring knowledge gleaned from observed types. in this paradigm, prior knowledge emerges as an essential asset @xcite . simultaneously, creating substantial ground truths remains an expensive and time-consuming activity, prone to human errors and inconsistencies that might affect model effectiveness. the advent of large language models (llms) has revolutionized zero-shot ner by enabling recognition of arbitrary entity types through natural language instructions @xcite . this represents a significant departure from the rigidity of traditional models. however, llms demand significant computational power. although some llms are accessible through apis @xcite , large-scale usage can incur prohibitive costs. to address these challenges, we introduce ze-roner, a novel framework designed to enhance zero-shot ner capabilities in small language models (slms). zeroner operates by distilling both annotations and entity type descriptions from llms (figure 1 ). rather than depending on large autoregressive models, our approach employs a more efficient bert-based cross-encoder that incorporates textual descriptions as contextual information to accurately identify any named entity across domains. first, we generate a domaindiverse distillation dataset through a frozen llm. then, to ensure data quality, we implement an llm-based self-correction mechanism combined with heuristic filtering. the slm is fine-tuned on our general-domain silver dataset, which encapsulates broad ner knowledge, and is then directly evaluated on the test set of the target dataset. we experiment with the zero-shot-adapted versions of three widely used ner datasets: medmentions @xcite , ontonotes 5.0 @xcite , and legalner @xcite . unlike recent approaches, we adopt a hard zero-shot evaluation protocol to ensure realistic performance assessment. our evaluation adheres to the strict constraints established by @xcite : (1) complete separation of entity types across (silver, general-domain) training, @xcite test splits, (2) reservation of the rarest labels for development and test sets, and (3) class imbalance accounting via macro-averaged per-class metrics. our slms not only outperform all evaluated llms with up to 8b parameters but also exceed the performance of alternative slms that benefit from data contamination, i.e., training exposure to entity classes used for benchmarking. our contributions are as follows: entity type descriptions. a growing body of work explores the use of textual metadata to improve generalization in ner. obeidat et al. (2019) and aly et al. (2021) demonstrated that injecting manually crafted definitions of entity types into encoders supports a more accurate classification. nguyen et al. (2021) further extended this idea by incorporating structured information from knowledge graphs, allowing the model to capture crossdomain relationships between entities. recently, interest has moved toward adapting llms to follow explicit entity type definitions within the prompt. methods such as gollie and slimer @xcite fine-tune llms to align with natural language guidelines, achieving strong zero-shot performance on ner tasks. however, such gains come with considerable computational costs that are not applicable in resourceconstrained environments. to the best of our knowledge, openbioner @xcite is the only prior work that, like ours, investigates the use of entity type descriptions within encoder-based models for open-domain objectives. however, (i) its focus is restricted to the biomedical domain, (ii) it foresees a second-stage in-domain fine-tuning, and (iii) it does not account for type leakage between silver train annotations and evaluation splits. knowledge distillation. knowledge distillation (kd) centers on transferring knowledge from large, high-capacity teacher models to more compact student models @xcite . in this work, we draw upon llm-based kd in a self-supervised setting @xcite , where a teacher model is used to annotate unlabeled data. recent efforts have applied this procedure to a wide array of downstream tasks @xcite , including ner. two main directions have emerged in this space: generative and encoder-based student models. generative student models employ autoregressive architectures to model ner labels as output texts. instructuie @xcite fine-tunes flan-t5-xxl @xcite on a mixture of information extraction datasets, yielding strong zeroshot ner performance. universalner @xcite fine-tunes llama using data annotated by chatgpt, often matching or surpassing chatgpt's own zero-shot ner results. importantly, it requires extracting one entity type at a time, increasing llm inference costs for standard multi-class problems. gner @xcite incorporates negative instances and adopts a bio-tagging scheme for finer-grained supervision. however, all generative approaches are inherently limited by slow, tokenby-token decoding and high resource demands. in contrast, encoder-based student models resort to more computationally efficient architectures as backbones for kd. gliner @xcite distills universalner annotations into a de-berta @xcite student model. it formulates ner as a span-entity type matching problem in embedding space and has demonstrated superior zero-shot results compared to both chatgpt and fine-tuned llms. @xcite pretrain roberta @xcite on chatgpt annotations through contrastive-learning, achieving results on par with llms in few-shot settings. despite these advances, a recurring limitation across current distillation frameworks lies in the use of silver datasets that include entity types overlapping with those in evaluation benchmarks. this results in evaluations conducted under soft zeroshot conditions, where type-level contamination can lead to overly optimistic assessments of generalization capabilities (see §d). as our work is grounded in hard zero-shot ner, direct comparisons with these prior approaches are not fair and therefore fall outside the scope of this paper. ## experimental settings datasets. to gauge zeroner holistically, we conduct experiments on three heterogeneous english-language ner datasets: medmentions-st21pv (mohan and li, 2019), ontonotes 5.0 @xcite , and legalner @xcite . medmentions (cc0 license) holds 4k biomedical abstracts and a fine-grained class ontology rooted in the unified medical language system (umls). ontonotes (ldc license) is a largescale multigenre corpus targeting domain-general entity types, including values. legalner (mit license) annotates indian court judgments published between 1950 and 2022. building on the recommendations of @xcite , we create zero-shot adapted versions of these benchmarks (identified by the "-zs" suffix). in pursuit of this, we adhere to the conversion protocol suggested by @xcite , alternatively leaving the rarest classes in the dev and test sets. the chosen types are guaranteed to be not trivial to recognize; we omit classes whose surface form displays regular patterns (e.g., percent, date). after filtering, the total number of entity types within medmentions, ontonotes, and legalner is 21, 11, and 13, respectively. annotations on splitdisconnected classes are eliminated. an overview of the data distribution is shown in baselines. we compare zeroner with slms and llms on pure zero-shot. metrics. we evaluate models with macroaveraged precision, recall, and f 1 metrics at the span level. per-class averaging prevents frequent entity types from dominating performance. vram. the model was trained on the distillation dataset for 3 epochs. each epoch took approximately 18 hours to complete, amounting to a total training time of 54 hours. at each epoch, we saved the best checkpoint according to the dev set of our benchmarks. we used a batch size of 8 and a constant learning rate of 2e-5, the latter selected following manual hyperparameter tuning. the entity masking probability was set to 0.3, following @xcite . we leave 42 as the default training seed. for llm inference, we leveraged the vllm library 6 (version 0.6.3.post1). more details about both zeroner training and llm inference are discussed in §f. ## benchmark contamination to assess potential data contamination between training corpora of the considered gliner baselines and the benchmarks contemplated in this study, we implement a multi-tiered detection approach with increasing levels of semantic overlap tolerance. we first identify exact matches where entity type labels appear identically in both datasets. fuzzy matches leverage python's 6 github.com/vllm-project/vllm difflib.sequencematcher to detect semantically similar labels (e.g., organization vs. organisation) using a similarity threshold of 0.9, capturing morphological variations and minor spelling differences. word boundary matches identify cases where entity types share common semantic components (e.g., immigration judge overlapping with judge), indicating conceptual overlap. finally, substring matches detect direct containment relationships between labels, revealing hierarchical contamination patterns. each contamination type receives a severity weight (exact: 1.0, fuzzy: 0.8, word boundary: 0.6, substring: 0.4) to compute an overall contamination severity score, ensuring entities are categorized only once to prevent doublecounting. ## results and discussion our core results are detailed in zeroner versus smxm. compared to smxm @xcite , our method achieves substantially higher results, although it relies on a smaller bert backbone and does not utilize in-domain supervised training. zeroner registers a 100% increase in average performance. this gap highlights the advantage of using distilled annotations and entity descriptions, even when the training data distribution differs from the test benchmarks, as well as the strength of silver data in the absence of direct supervision. ## conclusions we present zeroner, a knowledge distillation framework specifically designed for hard zero-shot ner. after rigorous evaluation on heterogeneous datasets, zeroner proves capable of recognizing unseen entities using only a textual description of them, eliminating the need for re-