A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition (2023)

Abstract: Existing models for named entity recognition (NER) are mainly based on large-scale labeled datasets, which always obtain using crowdsourcing. However, it is hard to obtain a unified and correct label via majority voting from multiple annotators for NER due to the large labeling space and complexity of this task. To address this problem, we aim to utilize the original multi-annotator labels directly. Particularly, we propose a CONfidence-based partial Label Learning (CONLL) method to integrate the prior confidence (given by annotators) and posterior confidences (learned by models) for crowd-annotated NER. This model learns a token- and content-dependent confidence via an Expectation–Maximization (EM) algorithm by minimizing empirical risk. The true posterior estimator and confidence estimator perform iteratively to update the true posterior and confidence respectively. We conduct extensive experimental results on both real-world and synthetic datasets, which show that our model can improve performance effectively compared with strong baselines.

## introduction named entity recognition (ner) plays a fundamental role in many downstream natural language processing (nlp) tasks, such as relation extraction @xcite , event extraction @xcite . recently, by leveraging deep learning models, existing ner systems have witnessed superior performances on ner datasets. however, these models typically require a massive amount of labeled training data, such as msra @xcite , ontonotes 4.0 @xcite , and resume @xcite . in real applications, we often need to consider new types of entities in new domains where we do not have existing annotated. the majority way to label the data at a lower cost ˚corresponding author, jie_zhou@fudan.edu.cn. the crowd-annotated datasets are always low quality for the following two reasons. first, as an exchange, crowd annotations are always nonexperts. various annotators may have different interpretations of labeling guidelines. moreover, they may make mistakes in the labeling process. it is hard to require a number of annotations to reach an agreement. for example, annotator 1 labels "david and jack" as a per entity, while the correct label is "david" and "jack" under our guidelines ( there are two main ways to utilize crowdannotated data. one simple and important way to obtain high-quality annotations for each input instance is majority voting. as shown in to address this problem, we translated this task into a partial label learning (pll) problem, which annotator 1 per hkkkkkkkkikkkkkkkkj david and jack will hold a meeting time hkkkkkkkkkkkkkkikkkkkkkkkkkkkkj tomorrow at 10:00 a.m. in the place hkkkkkkkkkkkkkkikkkkkkkkkkkkkkj company ( room 1003 ) . annotator 2 per hkkikkj david and per hk kik kj jack will hold a meeting time hkkkkkikkkkkj tomorrow at time hkkkkkikkkkkj 10:00 a.m. in the place hkkkkikkkkj company ( place hkkkkkikkkkkj room 1003 ). annotator 3 david and jack will hold a meeting time hkkkkkkkkkkkikkkkkkkkkkkj tomorrow at 10:00 a.m. in place hkkkkkkkkkkkkkkkikkkkkkkkkkkkkkkj the company ( room 1003 ). in this paper, we propose a confidence-based partial label learning (cpll) model for crowdannotated ner. for c1, we treat the classes' labeled number for each sample as prior confidence provided by the annotators. also, we learn the confidence scores via an expectation-maximization (em) algorithm @xcite . we estimate the real conditional probability p py " y|t " t, x " xq via a true posterior estimator based on the confidence that consists of the prior and posterior confidences. for c2, we learn a token-and content-dependent confidence via a confidence estimator to consider both the token t and sequence input x, because the candidate labels are always token-dependent and content-dependent. in fact, our model can be applied to all the sequence labeling tasks, such as word segment, part of speech, etc. we conduct a series of experiments on one real-world dataset and four synthetic datasets. the empirical results show that our model can make use of the crowd-annotated data effectively. we also explore the influence of annotation inconsistency and balance of prior and posterior confidences. the main contributions of this work are listed as follows. ## our approach in this section, we first give the formal definition of our task. then, we provide an overview of our proposed cpll model. finally, we introduce the main components contained in our model. ## experimental setups in this section, we first introduce one real-world and four synthetic datasets we adopted to evaluate the performance (section 3.1). then, we list the selected popular baselines to investigate the validity of our cpll model (section 3.2). finally, we present the implementation details and metrics to replicate the experiment easily (section 3.3). ## experimental results in this section, we conduct a series of experiments to investigate the effectiveness of the proposed cpll model. specifically, we compare our model with three kinds of strong baselines (section 4.1) and do ablation studies to explore the influence of the key parts contained in cpll (section 4.2). also, we investigate the influence of annotation inconsistency (section 4.3) and hyper-parameter α, which controls the balance of posterior confidence and prior confidence (section 4.4). ## related work in this section, we mainly review the most related works about named entity recognition (section 5.1) and partial label learning (section 5.2). ## conclusion and future work in this paper, we translate crowd-annotated ner into a pll problem and propose a cpll model based on an em algorithm. to rectify the model's prediction, we design a confidence estimator to predict token-and content-dependent confidence by incorporating prior confidence with posterior confidence. we conduct the experiments on one real-world dataset and four synthetic datasets to evaluate the performance of our proposed cpll model by comparing it with several state-of-theart baselines. moreover, we do ablation studies to verify the effectiveness of the key components and explore the influence of annotation inconsistency. in the future, we would like to investigate the performance of our model on other sequence labeling tasks.