Addressing NER Annotation Noises with Uncertainty-Guided Tree-Structured CRF s (2023)

Abstract: Real-world named entity recognition (NER) datasets are notorious for their noisy nature, attributed to annotation errors, inconsistencies, and subjective interpretations. Such noises present a substantial challenge for traditional supervised learning methods. In this paper, we present a new and unified approach to tackle annotation noises for NER. Our method considers NER as a constituency tree parsing problem, utilizing a tree-structured Conditional Random Fields (CRFs) with uncertainty evaluation for integration. Through extensive experiments conducted on four real-world datasets, we demonstrate the effectiveness of our model in addressing both partial and incorrect annotation errors. Remarkably, our model exhibits superb performance even in extreme scenarios with 90% annotation noise.

## introduction named entity recognition (ner) is a fundamental natural language processing task that aims to find entities with certain types in texts @xcite . generally, building a high-performance ner model requires to obtain high-quality labeled data. however, due to the complexities of the labeling process, real-world ner datasets often contain annotation noises @xcite . for example, figure 1 gives two typical annotation errors in the conll03 benchmark @xcite -the person entity "newcombe" is missed by annotators, and "loc angeles lakers" is incorrectly labelled as loca-tion. indeed, the noise rate is high (2%-8%) in real-world datasets @xcite , and this issue can significantly impede model learning. to date, many studies have been proposed to address ner annotation noises. for example, for the * equal contribution. partial annotation noise, previous works have employed re-weighting mechanisms to mitigate the impact of false negatives @xcite . for the incorrect annotation noise, previous works have suggested that the use of reinforcement learning and teacher-student framework is effective @xcite . nevertheless, most works focus on only one type of noise, and there has been limited success in using a unified approach to handle diverse types of noises @xcite . additionally, due to the combination of different components, previous methods may suffer from error propagation @xcite . in this paper, we present a unified approach for addressing annotation noises in ner. specifically, as shown in figure 2 , our method structures ner as a constituency tree parsing problem, with each node's label representing the entity type of a textual span. to learn with noises, we introduce uncertain nodes in the tree, indicating that the annotation for a span may be unreliable (this is compatible with both partial and incorrect annotation noises), and we use tree-structured conditional random fields (crfs) @xcite to average such uncertainties for learning, relieving the model from over-training on the noises. we present an monte carlo dropout (mc-dropout) mechanism @xcite for evaluating uncertainty and demonstrate that the overall framework is compatible with an iterative colearning framework. to verify the effectiveness of our method, we have conducted extensive experiments on realworld and simulated datasets. according to the results, on real-world datasets including youku @xcite and weibo @xcite , our approach outperforms previous stateof-the-art methods by up to 3.27% in absolute f1 score ( § 5.1), demonstrating remarkable effectiveness in addressing both partial and incorrect annotations. on simulated data sets, our method exhibits advantages under extreme conditions. for example, in the case of 90% partial or incorrect annotations on the taobao dataset @xcite , our approach achieves an impressive f1 score of 77.3%/25.36%, outperforming previous state-ofthe-art methods by a margin of 10.4%/2.13% ( § 5.2 and § 5.3). in summary, our contributions are as follows: ## approach our approach, depicted in figure 3 , consists of three main components: ## discussion to further investigate the effectiveness of our approach, we conduct more in-depth studies, divided into ablation study and case study. ## conclusion this study introduces a novel and unified solution to effectively address the noisy annotations in ner. our model integrates tree crfs with uncertainty evaluation, and comprehensive experiments validate the effectiveness of our approach in handling partial and incorrect annotations. although our primary focus was ner, we anticipate that our method holds promise for broader applications in various natural language processing tasks involving annotation noises.