Familiarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data (2025)

Abstract: Zero-shot named entity recognition (NER) is the task of detecting named entities of specific types (such as Person or Medicine) without any training examples. Current research increasingly relies on large synthetic datasets, automatically generated to cover tens of thousands of distinct entity types, to train zero-shot NER models. However, in this paper, we find that these synthetic datasets often contain entity types that are semantically highly similar to (or even the same as) those in standard evaluation benchmarks. Because of this overlap, we argue that reported F1 scores for zero-shot NER overestimate the true capabilities of these approaches. Further, we argue that current evaluation setups provide an incomplete picture of zero-shot abilities since they do not quantify the label shift (i.e., the similarity of labels) between training and evaluation datasets. To address these issues, we propose Familarity, a novel metric that captures both the semantic similarity between entity types in training and evaluation, as well as their frequency in the training data, to provide an estimate of label shift. It allows researchers to contextualize reported zero-shot NER scores when using custom synthetic training datasets. Further, it enables researchers to generate evaluation setups of various transfer difficulties for fine-grained analysis of zero-shot NER.

## introduction zero-shot named entity recognition (ner) is the task of recognizing instances of named entities of specific types (such as person, organization, or medicine) without any training examples. current state-of-the-art models, such as gliner @xcite and gollie @xcite , are initially trained on datasets that contain a large set of different entity types @xcite . this allows the models to identify mentions of previously unseen entity types by leveraging their general language understand-figure 1 : impact of training data on zero-shot performance of the current state-of-the-art approach (gliner). each synthetic dataset is characterized by the label overlap (yellow column) and the total number of entity mentions (purple column). while zeroshot performance (red line, macro-averaged f1 across 7 benchmarks) has significantly improved, we note a concerning increase in entity type overlaps between training and testing data. as figure 1 shows, the advent of large synthetic training datasets has significantly improved reported zero-shot f1 scores. however, as the figto address this, we introduce familiarity, a metric that quantifies and accounts for label shift, enabling more accurate and fair comparisons across models. we rather argue that given the advancements of llms and their potential to generate highquality datasets, accepting custom synthetic training datasets is inevitable. we therefore propose to measure the transfer difficulty between the labels of a training and an evaluation dataset, referred to as label shift @xcite . contributions. with this paper, we identify a critical issue with current zero-shot ner evalu-ations caused by the growing availability of largescale synthetic training datasets. to address this issue, we propose familiarity, a novel metric that quantifies the similarity between the sets of entity types in training and evaluation data, allowing us to assess the transfer difficulty of an evaluation setup (cf. figure 2 ). we summarize our contributions as follows: to enable the research community to efficiently compute familiarity and incorporate it into future research, we make all code publicly available as open source foot_0 . further, we publish three benchmark scenarios on the hugging face hub foot_1 for different levels of transfer difficulty to aid researchers in fine-grained analysis of zero-shot ner. ## the impact of synthetic datasets on current evaluations as shown in figure 2 , we hypothesize that label shift between fine-tuning and evaluation datasets affects transfer performance, particularly in zero-shot ner settings. we define this transfer as the process of fine-tuning a model θ on a dataset d with entity types l d and subsequently evaluating it on one or more benchmarks z 1,...,n , each with its own set of entity types l z 1,...,n , such that @xmath0 however, the entity type sets of the training and evaluation datasets may overlap due to the broad coverage of entity types, particularly in synthetic training datasets: l z ⊆ l d . we further note that it is possible that l z ∩l @xmath1. however, given that llms can generate finetuning datasets with thousands of entity types, we dataset # sent. # ent. ## familiarity the previous experiments show two key challenges in current zero-shot ner evaluations: (1) overlapping entity types inflate the transfer evaluations of zero-shot models, and (2) llms may generate ideal datasets for fixed evaluation settings, undermining the concept of low-resource evaluations. therefore, future evaluations must distinguish between improvements coming from sophisticated datasets and those achieved through new data-efficient approaches that do not depend on overlapping entity types. to address these challenges, we introduce fa-miliarity to quantify label shift between finetuning datasets and evaluation benchmarks based on the semantic similarity of the respective entity type sets. familiarity considers two key factors: to compute semantic similarity, we use a sentence-transformer @xcite to embed evaluation and training entity types, calculate cosine similarity, and clip negative values to keep the metric within a 0 to 1 range. for the second factor, we introduce a hyperparameter, k, which limits the number of support examples considered. in our experiments, we set @xmath2, meaning that up to 1000 closest training entity types are considered, measured by their support. we further weight these similarities by a zipfian distribution @xcite , prioritizing the most similar entity types, as they are likely to have the greatest impact on transfer performance. definition. let l d and l z represent the sets of all entity types in the fine-tuning dataset and the zeroshot benchmarks, respectively. additionally, let c denote the set of counts for each entity type ℓ @xmath3, and let θ represent the all-mpnet-base-v2 sentence-transformer model. for any entity type ℓ @xmath4, we calculate the clipped cosine similarity as follows: we then repeat each element in s ℓ z according to the corresponding support c @xmath5: we then sort the repeated set of all similarities between the evaluation entity type ℓ z and all training entity types and select the top-k similarities. once we determined the top-k similarities for evaluation entity type ℓ z , we compute the weighted average using the position k of each similarity value: finally, we marco-average familiarity for each ℓ @xmath6, resulting in an aggregated score for the entire transfer setting. to account for the number of mentions of each training entity type ℓ d i , we weight each element in s ℓ z by the corresponding probability distribution vector p ℓ d , which represents the relative frequency of each training entity type: where s @xmath7, ℓ d i ) and the distribution vector p ℓ d ensures that entity types with higher mention counts contribute proportionally more to the similarity calculation. we then sort the weighted set of all similarities between the evaluation entity type ℓ z and all training entity types and select the top-k similarities: once we have determined the top-k similarities for evaluation entity type ℓ z , we compute the weighted average using the position k of each similarity value: finally, we macro-average familiarity for each ℓ @xmath8, resulting in an aggregated score for the entire transfer setting. ## experiments we evaluate familiarity in various settings to assess its ability to measure label shift in zero-shot ner transfer scenarios. we examine its correlation with traditional transfer performance, the impact of design choices (embedding model and top-k similarities), and how familiarity can be used to create ner tasks of varying difficulty. ## related work the problem of ner can be formulated in many ways such as span classification @xcite , question answering @xcite , and text generation @xcite . the emergence of large language models has recently transformed many downstream nlp tasks through natural language prompting @xcite , including ner @xcite @xcite @xcite . our work contributes to this line of research by measuring the label shift of entity type prompts. similarity metrics. many works exist on evaluating outputs generated by a model with the target sequence using similarity metrics such as bertscore @xcite , bartscore @xcite , or semscore @xcite as well as task-specific similarity metrics such as sem-f1 @xcite or sas @xcite . we follow this idea by comparing the semantic similarity between fine-tuning and zero-shot entity types. zero-shot ner. we have recently observed increasingly capable ner systems trained on largescale datasets @xcite @xcite . these works stand out because they have been fine-tuned on datasets covering thousands of entity types. considering the progress of llms, we expect more contributions generating tailored datasets @xcite @xcite for downstream tasks. our work supports this line of research to better evaluate future contributions by explicitly measuring the label shift. ## conclusion this paper explores how the label shift between synthetically produced training datasets affects the performance of zero-shot ner as evaluated in current benchmark scnearios. as llms advance, creating improved datasets that align with the chosen zero-shot benchmarks to enhance transfer performance becomes more accessible. as a consequence, evaluation settings become less comparable. thus, we introduce familiarity to quantify the connection between fine-tuning and zero-shot datasets and show how it can achieve fairer comparisons. although the automatic generation of datasets holds promise for future ner research, it is crucial to foster data-efficient research by conducting zero-shot ner in scenarios where finetuning datasets do not contain closely related entity types. to enable the research community to efficiently compute familiarity and incorporate it into future research, we make all code publicly available as open source. further, we publish three benchmark scenarios for different levels of transfer difficulty to aid researchers in fine-grained analysis of zero-shot ner.