U se C lean: learning from complex noisy labels in named entity recognition (2023)

Abstract: We investigate and refine denoising methods for NER task on data that potentially contains extremely noisy labels from multi-sources. In this paper, we first summarized all possible noise types and noise generation schemes, based on which we built a thorough evaluation system. We then pinpoint the bottleneck of current state-of-art denoising methods using our evaluation system. Correspondingly, we propose several refinements, including using a two-stage framework to avoid error accumulation; a novel confidence score utilizing minimal clean supervision to increase predictive power; an automatic cutoff fitting to save extensive hyper-parameter tuning; a warm started weighted partial CRF to better learn on the noisy tokens. Additionally, we propose to use adaptive sampling to further boost the performance in long-tailed entity settings. Our method improves F1 score by on average at least 5 10% over current state-of-art across extensive experiments.

## introduction named entity recognition (ner) aims to recognize mentions of rigid designators from text belonging to predefined semantic types such as a person, location, organization, etc. ner not only acts as a standalone tool for information extraction (ie), but also plays an essential role in a variety of natural language processing (nlp) applications such as text understanding, information retrieval, automatic text summarization, question answering, machine translation, and knowledge base construction, etc. recent progress in deep learning has significantly advanced ner performances (e.g. @xcite @xcite ). however, in the presence of noisy labels, training dnns is known to be vulnerable to noisy labels because the significant number of model parameters allow dnns easily overfit to even corrupted labels. this problem first raised attention in computer vision (cv): @xcite demonstrated that dnns can easily fit an entire training dataset with any ratio of corrupted labels, which eventually resulted in poor generalizability on a test dataset. unfortunately, popular regularization techniques, such as data augmentation, weight decay, dropout, and batch normalization do not completely overcome the overfitting issue caused by noisy labels. many endeavors have been put into handling noisy labels. note that this is a fundamentally different problem than general feature-level noise @xcite @xcite . except for the specific techniques in certain science domains @xcite , most of those methods are first designed for computer vision or instance-level classification tasks in nlp like text classification. denoising methods in the ner domain are generally under-explored and rendered harder: for ner, only correct detection of both the entity boundary and entity class are rendered as one correct prediction. therefore, the label noise in ner is more complex than those in cv or text classification. for example, human annotators could produce mis-specified entity boundaries; other automatic labels generation like distant supervision @xcite from the dictionary or database often generate incomplete annotations, meaning some entity words are wrongly named as non-entity simply because they are not recorded in the database; others like transfer learning or domain adaptation @xcite @xcite from one domain to another domain could cause wrongly labeled classes for many entity words, as same words could have different semantic types in different domains. due to the lack of clean data resources, the majority of denoising literature is unwilling to use any clean validation data or anchor points for denoising, regardless of the fact that most of them often require massive computation cost or extensive hyper-parameter tuning @xcite . in fact, those supervision-free methods often suffer from error propagation as the error incurred by false correction/filtering will be accumulated due to lack of supervision, especially when the number of classes or the number of mislabeled examples is large @xcite . to overcome those obstacles, maintaining multiple dnns or training a dnn in multiple rounds is frequently used (e.g. @xcite ), but these approaches significantly degrade the efficiency of the learning pipeline @xcite . in industrylevel applications, meta gold datasets (i.e. highquality/clean datasets) are commonly available, to guarantee direct and reliable evaluation of methods and therefore stable and supreme user experience; also the amount of available data rapidly increases in big companies. more attention should be paid to how to best design and leverage the meta gold dataset to do efficient, effective, and stable label denoising. motivated by the above, in this paper, we study the label denoising problem in ner, and we contribute in the following three aspects: through this system we find out that the baseline methods foot_0 are already agnostic to some noise types; while for the other, the noise rate influences the effectiveness of denoising rather than noise type. in the following, we will introduce the related work in more detail in section 2 and provide a formal problem and method description in section 3. we describe our experiment setting and corresponding results in section 4, where we also provide a careful ablation study to narrow down the bottleneck of the current state-of-arts method. in the end, we summarize our findings and contributions and some possible future directions in section 5. ## related work learning on noisy labels most of the denoising methods are designed for computer vision @xcite , that is, instance level classification. they can generally be categorized into the following four categories: 1) noise modeling: adding a noise adaptation layer at the top of an underlying dnn to learn the transition between clean and noisy labels, e.g. @xcite @xcite ); 2) regularization: enforcing a dnn to overfit less to false-labeled examples explicitly or implicitly, e.g. @xcite @xcite @xcite ; 3) sample reweighting: adjusting the loss value according to the trust-level of a given sample, e.g. @xcite @xcite ; 4) sample selection: identifying true-labeled examples from noisy training data via multi-network or multi-round learning, e.g. @xcite @xcite @xcite @xcite . from prior work and our investigation, we generally note that, noise modeling type of methods often estimate the transition matrix with large error when only noisy training data is used or when the noise rate is high; regularization type of methods often introduce sensitive model-dependent hyperparameters and therefore hard to stably work in practice; sample reweighting is often more useful for instance level classification, which is not the case in ner problem domain where often a graphical model is adopted for classification; sample selection is well motivated and works well in general, also its has more interpretability and light-weights. for industry-level application considerations: we hope to seek solutions that are more lightweight, stable, and easy to tune. therefore we focus on the line of methods using sample selection. semi-supervised learning for ner task an inherent limitation of sample selection is to discard all the un-selected training examples, thus resulting in a partial exploration of training data. to exploit all the noisy examples, researchers have attempted to combine sample selection with other orthogonal ideas. the most prominent method in this direction is combining a specific sample selection strategy with a specific semi-supervised learning model @xcite . for example, the most promising method in this direction is combining a specific sample selection strategy with a specific semi-supervised learning model like partial crf @xcite . ## method: useclean our method useclean is built upon a well-known ner modeling called conditional random field (crf). specifically, consider a sentence of words u : [u 1 , . . . , u s ], and a corresponding sequence of tags y : [y 1 , . . . , y s ], where y @xmath0:= {1, . . . , k}, crf @xcite models the conditional probability of y given u as: here h denotes the encoder hidden representation, linear(•) denotes a linear layer that converts h into the network estimation for the possibility of y i at word i given utterance u; and the transition score t ij to model the transition from i-th label to j-th for a pair of consecutive time steps, and it is position independent. dynamic programming can be used efficiently to compute t and inference optimal tag sequences @xcite . in the following, we will introduce our two-stage method useclean built upon this encoder-crf model. figure 1 shows the whole working flow of our useclean method. ## conclusion and discussion in conclusion, we propose an effective and efficient method called useclean, which includes a simple two-stage framework to avoid error accumulation, a novel confidence score utilizing minimal clean supervision to increase predictive power in sample selection, an automatic cutoff fitting to save exten-sive hyper-parameter tuning and finally weighted semi-supervised learning with warm start to learn better on the noisy tokens. additionally, we propose to use adaptive sampling to construct better clean supervision for a further performance boost. despite simple, our method improves f1 score by on average at least 5 ∼ 10% over current stateof-art without extensive hyper-parameter tuning or heavy computation, and is effective across a broad type of noise types and noise levels. we admit that most of the performance gain comes from the minimal clean supervision in small gold data. without it, the sota method nlnce suffers from error accumulation and heavy computation. still, we argue that the clean supervision we need is very minimal, like just about 100 samples, while the stable improvement and efficiency it can bring is fairly large. in fact, we suspect that it is often necessary to guarantee success in real applications, and how to best construct and leverage clean supervision is nontrivial and important.