Self-Cleaning: Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances (2024)

Abstract: To achieve state-of-the-art performance, one still needs to train NER models on large-scale, high-quality annotated data, an asset that is both costly and time-intensive to accumulate. In contrast, real-world applications often resort to massive low-quality labeled data through non-expert annotators via crowdsourcing and external knowledge bases via distant supervision as a cost-effective alternative. However, these annotation methods result in noisy labels, which in turn lead to a notable decline in performance. Hence, we propose to denoise the noisy NER data with guidance from a small set of clean instances. Along with the main NER model we train a discriminator model and use its outputs to recalibrate the sample weights. The discriminator is capable of detecting both span and category errors with different discriminative prompts. Results on public crowdsourcing and distant supervision datasets show that the proposed method can consistently improve performance with a small guidance set.

## introduction deep learning methods have notably improved the performance of named entity recognition (ner), but need large-scale high-quality labeled data @xcite . in practice, collecting large-scale labeled data via crowdsourcing @xcite or distant supervision @xcite ) is far more cost-effective. however, such data is usually too noisy for direct use without further treatment @xcite @xcite . extensive efforts have been dedicated to develop data denoising techniques and learning strategies specifically tailored for noisy ner data. @xcite suggested fine-tuning pre-trained language models (plms) on such data, employing early stopping and self-training techniques to mitigate overfitting induced by noisy labels. @xcite extended the approach by using a frozen plm to generate augmented pseudo labels for selftraining. @xcite further eliminated selftraining labels with low estimated label confidence. yet these denoising methods do not have a mechanism to guide error correction, thus suffer from confirmation bias @xcite , where the learner struggles to correct its own mistakes. ## method: self-cleaning in this section, we introduce self-cleaning in detail. we begin with the key component of self-cleaning: the prompt-based discriminator, explained in section 3.1. we then present the training procedure (as shown in figure 3 ), which consists of three stages: stage i: demonstrative self-training. in this stage, high-confidence predictions from the ner model are used as pseudo labels to iteratively refine itself, a process often referred to as self-training @xcite . to enhance the robustness of the pseudo labels, we propose a mechanism called clean demonstration, in which entities from the guidance set serve as demonstrations to elicit robust predictions from the ner encoder entity head cls head discriminator the train is from bangkok to chiang mai [x]. bangkok is a [mask] entity. [x]. bangkok is a [mask] location entity. [x]. chiang mai is a [mask] entity. [x]. chiang mai is a [mask] person entity. ## conclusion in this paper, we study how to improve ner models trained on noisy labeled data with a guidance set consisting of a small number of clean instances. our research is grounded on the noise pattern analysis on the real-world noisy ner data. we identify two ner-specific error types: span error and category error. to address these errors, we propose to use a dedicated discriminator to guide the training of the ner model. this discriminator is tailored to detect the aforementioned errors using pre-defined discriminative prompts, and its outputs are used to reweight the samples for training the ner model. we design a three-stage training procedure to unleash the power of clean instances in guiding noisy ner learning. we evaluate the proposed method on a rich set of ner benchmarks with crowdsourcing labels and distant supervisions. the results show that with a few clean instances, the proposed method can boost the performance significantly.