N oise B ench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition (2024)

Abstract: Available training data for named entity recognition (NER) often contains a significant percentage of incorrect labels for entity types and entity boundaries. Such label noise poses challenges for supervised learning and may significantly deteriorate model quality. To address this, prior work proposed various noise-robust learning approaches capable of learning from data with partially incorrect labels. These approaches are typically evaluated using simulated noise where the labels in a clean dataset are automatically corrupted. However, as we show in this paper, this leads to unrealistic noise that is far easier to handle than real noise caused by human error or semi-automatic annotation. To enable the study of the impact of various types of real noise, we introduce NoiseBench, an NER benchmark consisting of clean training data corrupted with 6 types of real noise, including expert errors, crowdsourcing errors, automatic annotation errors and LLM errors. We present an analysis that shows that real noise is significantly more challenging than simulated noise, and show that current state-of-the-art models for noise-robust learning fall far short of their achievable upper bound. We release NoiseBench for both English and German to the research community.

## introduction named entity recognition (ner) is the task of detecting and classifying named entities in text, such as the names of organizations or locations. current state-of-the-art approaches for ner still require supervision in the form of labeled training data @xcite , i.e. sentences in which named entities are marked and assigned their correct type. however, prior work found that available datasets for ner and other supervised tasks are affected by label noise, meaning that a certain percentage of entity labels are incorrect. for instance, the common ner dataset conll-03 @xcite was estimated in various studies to have noise shares of between 5 and 7% @xcite @xcite . other ner datasets have also been found to contain a share of incorrect labels, with ontonotes4 estimated around 8% and wnut-17 around 18% @xcite . label noise introduces inconsistencies during training, which may significantly deteriorate model quality @xcite . to address this issue, prior work proposed approaches for noise-robust learning aimed at mitigating the negative effects of the noisy training signal @xcite . however, the evaluation of these approaches has two main limitations. limitation 1: simulated noise is too easy. most current research in noise-robust learning relies on experiments with simulated label noise @xcite . while this allows for evaluation in a controlled setting, it has been shown that simulated noise, even though it can model noise well to some extent, is much easier for deep learning models to disregard than real label noise @xcite . refer to figure 1 for an illustrative comparison between real and simulated noise for three example sentences, including different types of errors that occur in ner datasets. these examples demonstrate that simulated noise can introduce similar errors as real noise, however the choice of spans to mislabel is random and as a result often less plausible. this means that an approach shown to be robust to simulated noise may not in fact be robust to real noise in practice. limitation 2: distinct types of real noise. additionally, there exist many possible sources of "real" noise. for instance, expert labelers may make different mistakes than crowd workers @xcite . next to human label- (a) example of a partial match error induced by noise. real noise makes a plausible mistake by labeling "un" as org (organization), whereas simulated noise implausibly caused "council" to be labeled. correct label noisy label from crowdsourcing simulated noisy label misc misc misc loc misc loc misc misc ... swiss grand prix cycling race on sunday... ... swiss grand prix cycling race on sunday... ... swiss grand prix cycling race on sunday... correct label noisy label from crowdsourcing simulated noisy label misc loc loc misc ... in zurich, the olympic champion... ... in zurich, the olympic champion... ... in zurich, the olympic champion... figure 1 : examples of text snippets with correct labels (top row) and two types of noise: real noise from crowdsourcing (middle row) and simulated class-dependent noise (bottom row). this introduces different types of errors: (a) partial matches of correct entity mentions, (b) a wrong type and a non-entity mention and (c) a missing entity. we qualitatively find real noise to be more plausible than simulated noise. we postulate that these types of real noise differ in their characteristics, meaning that a noise-robust learning approach shown to perform well on one type of noise may not perform well on another. for this reason, we argue there is a need for evaluating noise-robustness across multiple label noise types. contributions. with this paper, we present noisebench, a new benchmark for measuring the impact of label noise in the training data on the prediction quality of trained ner models. in more detail, our contributions are: our analysis finds that no single current approach works best for all types of real noise, and that all current approaches fall far short of their theoretical upper bound. to enable the research community to leverage our benchmark in their evaluations, we publicly release all data and implementation. ## noisebench our benchmark is derived from a subset of the classic conll-03 dataset for ner in english, annotated with entities belonging to four classes. we chose this dataset since it has been extensively studied in the field, allowing us to integrate various prior works. we derive a similar benchmark for ner in german, in section 5 and appendix a. noisebench consists of the following parts: (1) a noise-free test split to evaluate trained models. the training split contains 5,885 sentences from 400 documents, covering 9,685 entity mentions. the test split contains 3,427 sentences from 231 documents, covering 5,725 entity mentions. ## comparing real and simulated noise we first use noisebench to investigate how real label noise affects ner model performance in comparison to simulated noise. for this, we conduct two experiments: the first one addresses the impact of each type of training noise on the clean test set performance, and the second one compares training dynamics under real and simulated label noise to highlight the differences in noise memorization. ## evaluating noise-robust learning having established the difficulty of real noise, we now use noisebench to perform a comparative evaluation of widely-used noise-robust learning approaches. our goal is to determine their effectiveness in the presence of real label noise, and to establish upper bounds of what noise-robust learning could ideally achieve. ## ablation: noisebench for german using the german sentences in conll-03, we created a noisy label benchmark for german to confirm our findings for a different language. following the english counterpart described in section 2, it consists of (1) a noise-free test split to evaluate trained models and (2) three variants of the training split, where two are annotated with different types of noise and one is the ground truth. the two types of noise include expert labels, with 16.2% noise and llm labels, with 54% noise. more details can be found in appendix a.1. ## related work there are a few benchmarks for learning with label noise and related areas. the wrench benchmark @xcite focuses only on weak supervision labels for multiple tasks, with the emphasis on combining multiple weak labelling sources. @xcite compare a large number of methods for the detection of annotation errors. multiple tasks are included, including ner on conll-03, where they evaluate the detection of expert errors, concluding that most approaches are not successful at this. similarly, @xcite evaluate annotation error detection on datasets with noise only from crowdsourced labels, for part-of-speech tagging and natural language inference tasks. @xcite propose a benchmark for text classification under label noise, where they re-annotate an exist-ing sentiment classification dataset and construct noisy label sets according to annotator disagreements; however, they do not publish these label sets. noisywikihow, a benchmark for intention identification has also been presented @xcite , where the authors propose a method to simulate realistic noise that imitates human errors by producing heterogeneous and instance-dependent errors. for ner in estonian, @xcite introduce the noisyner, which includes multiple noise levels obtained from distant supervision approaches with varying quality. multiconerv2 @xcite addresses textual noise in the input data itself (e.g. typos), instead of label noise. ## conclusion in this paper, we address the issue of label noise in the ner task. we introduce a new benchmark, based on the commonly used ner dataset conll-03, for evaluating the impact of 6 distinct types of real label noise on the same set of sentences, with varying degrees of difficulty. we demonstrated that real noise causes transformer-based language models to immediately memorize the noise pattern, making real label noise a more challenging problem than simulated label noise, even in the case of oracle class-dependent noise informed by the characteristics of real noise. we further presented an evaluation of popular noise-robust learning approaches. our experiments indicate that current methods fall far short of what can potentially be achieved on the noise types in noisebench and that approaches that focus on automatically identifying a clean subset of labels have the highest potential. we hope that noisebench aids other researchers in the further development of more effective noise-robust approaches.