Noisy-Labeled NER with Confidence Estimation (2021)

Abstract: Recent studies in deep learning have shown significant progress in named entity recognition (NER). However, most existing works assume clean data annotation, while real-world scenarios typically involve a large amount of noises from a variety of sources (e.g., pseudo, weak, or distant annotations). This work studies NER under a noisy labeled setting with calibrated confidence estimation. Based on empirical observations of different training dynamics of noisy and clean labels, we propose strategies for estimating confidence scores based on local and global independence assumptions. We partially marginalize out labels of low confidence with a CRF model. We further propose a calibration method for confidence scores based on the structure of entity labels. We integrate our approach into a self-training framework for boosting performance. Experiments in general noisy settings with four languages and distantly labeled settings demonstrate the effectiveness of our method.

## introduction recent progress in deep learning has significantly advanced ner performances @xcite . while most existing works assume clean data annotation, real-world data inevitably involve different levels of noise (e.g., distant supervision from the dictionary @xcite , or weak supervision from the web @xcite . figure 1 gives an example of such noisy labels. to train robust models with high performance, it is fundamentally critical to tackle the challenges associated with noisy data annotation. in this work, we propose a confidence estimation approach for ner with noisy labels. we motivate our approach with important empirical observations of the training dynamics of clean and noisy labels: usually, clean data are easier to fit with faster convergence and smaller loss values @xcite @xcite . consequently, loss values (probabilities or scores of labels) can serve as strong indicators for the existence of noise, which we utilize to build our confidence estimation. the key contribution of this work is a confidence estimation method with calibration. we use probabilities of labels as confidence scores and apply two estimation strategies based on global or local normalization that assume different dependency structures about how the noisy labels are generated. we further calibrate the confidence score for positive labels (labels representing entity parts, e.g., b-loc) based on the structure of these labels: we separately estimate scores for the position part (e.g., b in b-loc) and the type part (e.g., loc in b-loc). such fine-grained calibration leads to a more accurate estimation and better performance in our experiments. we apply our method in a crf model @xcite , marginalize out labels we do not trust, and maximize the likelihood of trusted labels. we use a self-training approach @xcite that iteratively estimates confidence scores in multiple training iterations and re-annotates the data at each iteration. experiments show that our approach outperforms baselines on a general noisy-labeled setting with datasets in four languages and shows promising results on a distantly-labeled setting with four datasets. ## method given a sentence @xmath0, ..., x n ] and its tag sequence ŷ1 , ..., ŷn , n is the sentence length. we model the conditional probability of y with a bidirectional lstm-crf @xcite : where h denotes lstm states, linear(•) denotes a linear layer, φ(y) denotes the potential (weight) evaluated for tag sequence y, z denotes the partition function, α denotes the forward variables, and forward(•) denotes the forward algorithm @xcite . the advantage of the crf model is that it gives us a probabilistically uniform way to handle labels we do or do not trust by partial marginalization, which we discuss later. ## related works state-of-the-art ner models @xcite @xcite are all under the traditional assumption of clean data annotation. the key motivation of this work is the intrinsic gap between the clean data assumption and noisy real-world scenarios. we believe that the noisy label setting is fundamentally challenging in ner and all related supervised learning tasks. previous works on ner with noise could be organized into two threads: (a) some works treat this task as learning with missing labels. @xcite propose a missing label crf to deal with partial annotation. @xcite propose a self-training framework with marginal crf to re-annotate the missing labels. (b) other works treat missing labels as noise and try to avoid them in the training process. for example, mayhew et al. (2019 ## conclusion in this work, we propose a calibrated confidence estimation approach for noisy-labeled ner. we integrate our method in an lstm-crf model under a self-training framework. extensive experiments demonstrate the effectiveness of our approach. our method outperforms strong baseline models in a general noise setting (especially for larger noise ratios), and shows promising results in a distant supervision setting.