Random Splitting Negatively Impacts NER Evaluation: Quantifying and Eliminating the Overestimation of NER Performance (2025)

Abstract: In named entity recognition (NER), models are evaluated on their ability to identify entity mentions in text. However, standard evaluation methods often rely on test sets that contain named entities already present in the training data, raising concerns about overestimation of model performance.This work investigates the impact of varying degrees of entity contamination on a dataset level on the generalization ability and reported F1 scores of three state-of-the-art NER models.Experiments on five standard benchmarks show that F1 scores for contaminated entities statistically significantly inflate reported F1 scores as contamination rates increase, with F1 performance gaps ranging from 2-10% compared to entities not seen during training.To address these inflated F1 scores, we additionally propose a novel NER dataset splitting method using a minimum cut algorithm to minimize train-test entity leakage.While our splitting method ensures near-zero entity contamination, we also compare new and existing dataset splits on named entity sample counts.

## introduction named entity recognition (ner) is a critical component of information extraction that involves the identification of named entities (nes) in unstructured text. while significant progress has been made in ner in recent years, there is still a pressing need for more accurate evaluation of model performance. the existence of test set contamination is a key aspect of this evaluation. this occurs when the model is evaluated using documents that contain nes present during training. this leads to an overestimation of the model's performance, which does not accurately reflect the generalization ability of an ner model. despite the availability of many ner datasets, there has been surprisingly little research on the degree of their test set contamination. most of the ex-figure 1 : example of entity contamination (here: idaho) in the conll04 @xcite splits by @xcite between a training and test sample. we investigate the effect of increasing contamination rates on model generalization, i.e., the ability of the model to generalize to previously unseen nes, through a series of studies using three fundamentally different state-of-the-art ner architectures. we found that even small rates of contamination statistically significantly inflate reported test f1 scores, thereby overestimating ner performance. to address this issue, we propose a method to quantify the degree of overestimation as a delta of the original f1 score. furthermore, based on our findings and the lack of existing research and approaches, we identify a need for creating ner dataset splits that focus on creating a test set that fairly evaluates a model's ability to generalize. we also argue that ideally, a test set should contain as many unseen nes as possible to fairly test the model's generalization ability @xcite . in order to test the generalization performance of ner models separately from their memorization ability, we publish two subsets of contaminated and clean entities for the test sets of all datasets. these allow the calculation of the ∆f1 for each dataset. we also provide and demonstrate a method for generating clean training and test splits on all datasets using a minimum cut algorithm @xcite . in brief, our main contributions are as follows: the dataset splits and code for reproducing experiments, analyzing contamination of other ner datasets, and the algorithm for creating clean ner dataset splits are available on github. ## related work the difficulty of ner models to generalize to unseen nes has been studied and identified previously @xcite @xcite . in particular, @xcite focus on evaluating ner performance on unseen samples, and find lower f1 and recall scores in various genres, and a tendency to overfit to previously seen tokens. together with bernier-colborne and langlais (2020) and lin et al. (2021) they report percentages of unseen nes on several datasets. however, they all fail to analyze how different rates of seen nes for a given dataset statistically affect the f1 score on previously seen and unseen nes, especially on current transformer and state-of-theart systems. agarwal et al. (2021) analyze transformers but find no improvement in bert (devlin et al., 2019)'s generalization ability to unseen nes compared to glove-based (pennington et al., 2014) approaches, while elmo (peters et al., 2018) and flair (akbik et al., 2019) are better at generalizing to different domains and contamination rates -referred to in fu et al. (2020) as entity coverage ratio. they also find that whether the context of the test entity has been seen before affects the generalization ability of ner models, not just the previously seen surface shapes of nes. most existing approaches to generalization testing focus on extending the training or test data to create unseen nes @xcite @xcite , while others test on different domains @xcite @xcite . for example, lin et al. (2021) used the semantic classes in wikidata @xcite and bert to replace target entities for a given input to create unseen samples and provide statistics on entities seen during training for the ontonotes @xcite dataset. while these approaches are easy to use, they require additional steps and do not address the root cause of contamination. a common approach to splitting a newly created ner dataset into training, validation, and test sets is to randomly split it in an 80-10-10 or similar ratio (see creating a method for generating clean training and test sets would solve the aforementioned overfitting and scoring on the same nes problems. this method would ensure fair evaluation and a better approximation of the model's real-world per-formance (see section 5. 1 and augenstein et al. (2017) ). most importantly, this method would allow for a better partitioning of future ner datasets. ## methodology in the following, we first lay the theoretical groundwork for our analysis and define the terms contamination and clean (section 3.1), introduce a new measure to quantify the overestimation of ner model f1 scores (section 3.2), and frame dataset contamination as a minimum cut problem in order to reduce the contamination of ner dataset splits (section 3.3). ## datasets while it is known for ace05 and conll04 that both datasets were created using random splitting @xcite , we assume a similar splitting strategy for the other datasets in our analysis (ade, conll03, genia, nyt, and scierc), as random splitting is considered standard practice @xcite , resulting in similar patterns of contamination across all datasets, as shown in figure 3 . this is further justified by the split ratios for all datasets in ## experiments we outline the experimental setup for three main investigations: contamination training across multiple ner models, the application of the minimum cut algorithm for dataset splitting, and the quantification of state-of-the-art model overestimation due to contamination. the main results and evaluations are also presented, focusing on correlations between model evaluation and contamination, overestimation of f1 values in the state of the art, and analysis of datasets generated using minimum cut. ## conclusion and future work our work extends and statistically substantiates the effects of entity contamination identified in previous research. based on our results -specifically the correlation between contamination and the model performance gap between clean and contaminated nes and the associated inflation of reported f1 scores -and the calls for better scoring @xcite , we create a novel method for splitting ner datasets that can help other researchers create clean splits and scores for existing and future datasets. performance on new entities is quantified by the introduced metrics f 1 clean and ∆f 1, which are easy to calculate for future researchers and provide intuitive insights into the memorizationgeneralization gap. furthermore, we share contaminated and clean test splits for all datasets for fairer and better generalization comparisons between existing and future models. future work could analyze possible relationships between contamination rate and different training and model sizes. while achieving 0% contamination should be considered ideal, alternative data splitting techniques could be explored to account for ne types while minimizing contamination as much as possible. although expensive and highly complex, one could explore relation extraction, analyze overlapping triples and relations, and apply similar data splitting methods to this area of research.