Token-Level Metrics for Detecting Incorrect Gold Annotations in Named Entity Recognition (2025)

Abstract: Annotated datasets for supervised learning tasks often contain incorrect gold annotations, i.e. label noise. To address this issue, many noisy label learning approaches incorporate metrics to filter out unreliable samples, for example using heuristics such as high loss or low confidence. However, when these metrics are integrated into larger pipelines, it becomes difficult to compare their effectiveness, and understand their individual contribution to reducing label noise. This paper directly compares popular sample metrics for detecting incorrect annotations in named entity recognition (NER). NER is commonly approached as token classification, so the metrics are calculated for each training token and we flag the incorrect ones by defining metrics thresholds. We compare the metrics based on (i) their accuracy in detecting the incorrect labels and (ii) the test scores when retraining a model using the cleaned dataset. We show that training dynamics metrics work the best overall. The best metrics effectively reduce the label noise across different noise types. The errors that the model has not yet memorized are more feasible to detect, and relabeling these tokens is a more effective strategy than excluding them from training.

## introduction state-of-the-art approaches for some nlp tasks still require supervision in the form of labeled training data @xcite . one example is named entity recognition (ner), the task of identifying and classifying named entities in text. for ner we need sentences in which named entities are marked and their correct type is assigned. however, available datasets for ner are affected by label noise, meaning that a certain percentage of labels are incorrect @xcite @xcite ). 1 github repository: https://github.com/elenamer/ tokenmetrics such erroneous annotations can deteriorate model quality in machine learning tasks, including ner. one promising approach involves selecting a subset of clean samples for training @xcite and excluding the ones with an error. this requires a robust way to detect the incorrectly labeled training samples. noisy label learning approaches often incorporate measures to select reliable samples as part of complex pipelines, for example heuristics such as low loss or high confidence @xcite . some studies @xcite , propose new measures to characterize dataset samples based on their learning dynamics. however, when these measures are integrated into larger pipelines, it becomes difficult to isolate their impact, compare their effectiveness, and understand their individual contributions to reducing label noise. furthermore, many existing approaches rely on computationally expensive techniques such as consistency checks, self-training, and ensemble methods @xcite . while effective, their efficiency is often a limiting factor. in this paper, we focus on evaluating sample metrics that can identify label errors from a single training run. we include both static metrics, calculated from a single epoch and training dynamics metrics, which aggregate information over multiple epochs. additionally, we assess the potential of layer-wise metrics, which have not been utilized for annotation error detection before. our goal is to assess the potential of these metrics as diagnostic tools for detecting mislabeled samples and to directly compare them. ner is commonly approached as a token classification task, where each token is assigned a bio tag, as illustrated in figure 1 . the confidence metric shown is the softmax probability of the observed label, averaged across epochs @xcite and is calculated for each token in the training set. as illustrated, using confidence as an indicator for noise entails defining a threshold (for @xmath0.4), and then flagging all samples below that threshold as erroneous. additionally, the flagged tokens can be relabeled with the model's predicted tags. there exist other metrics besides confidence, and we want to know which one is best. ner covers two subtasks: mention detection (tagging a token as either o or as belonging to a mention) and mention classification (tagging a token with the correct entity type). due to this, it is challenging to define universal sample selection rules. instead, we propose a disaggregated approach, where we handle four different categories of tokens independently. figure 2 shows the separation between the correct and incorrect tokens, both by handling all tokens at once in figure 2a and with our category approach in figure 2b . in this paper, we evaluate the use of sample metrics for detecting erroneous tokens in noisy ner datasets. following are our main contributions: ## token-level metrics we explore three types of token-level metrics: static metrics, training dynamics metrics, and layer agreement metrics. these metrics characterize each training sample based on the model's outputs. ## approach in this section, we present our approach for cleaning noisy ner data using token-level metrics. section 3.1 describes how we select the most effective metrics and parameter settings. section 3.2 introduces a disaggregated method that separates tokens into four categories to improve error detection. finally, section 3.3 outlines two strategies -relabeling and masking -for handling detected label errors. the full pipeline is illustrated in figure 3 . ## experiment 1: evaluating selection performance in this experiment, we determine which metrics are most effective as noise indicators. we also compare the ideal parameter configuration for each noise type with a single configuration that performs well on average. this allows us to assess both the maximum selection potential and the generalizability of metric performance. ## experiment 2: evaluating re-training performance this experiment evaluates how effective the best metrics from experiment 1 are in cleaning noisy annotations and how much test scores improve when re-training with these modified annotations, compared to using the original noisy sets. ## related work detecting and correcting label errors. the annotation error detection paper @xcite gives an extensive overview of different strategies for detecting label errors. they include sample metrics (scorer methods) in the comparison @xcite , but also methods ensembling multiple models @xcite @xcite and other approaches @xcite . their ner evaluation is quite limited and only detection strategies are explored, without considering relabeling. noise-robust learning. cleaning the dataset @xcite ) by removing or relabelling the incorrect annotations is one strategy for learning under label noise. other options include reweighting the potentially erroneous samples @xcite , delaying memorization @xcite and multi-stage pipelines @xcite @xcite . recent noise-robust learning studies have analysed the use of validation sets @xcite , suggesting that the use of large or clean validation data leads to overly optimistic evaluation. ## conclusion this paper explores how token-level metrics can be used to correct label annotations for ner. we assess if each metric can be used as a separation feature between clean and noisy samples. for this, we look at four token categories separately. when comparing different metrics, we see an added benefit of using metrics training dynamicsbased, as opposed to only well-established estimates like the maximum softmax probability. also, we found that layer agreement metrics perform worse than the other two types. we compare two strategies to deal with the erroneous samples: relabeling and masking. with relabeling, we see good improvements in the resulting noise shares. we are able to successfully reduce the noise share across all datasets and variants by 3.3 percentage points on average by either correcting missing mention or non-entity errors. as a next step, we re-trained a model with the modified dataset, to evaluate the impact of this approach on the final test scores. we see that the reduction in noise share translates in modest improvements with retraining for all datasets and noise types apart from expert noise. the token metrics approach works for misclassified-o and misclassified-entity and it fails for learned categories. this shows that once a model has memorized a label error, it is much more challenging to detect. finally, our experiments span three different languages and show that the metric and threshold selection generalizes well.