Improving Distantly-Supervised Named Entity Recognition with Self-Collaborative Denoising Learning (2021)

Abstract: Distantly supervised named entity recognition (DS-NER) efficiently reduces labor costs but meanwhile intrinsically suffers from the label noise due to the strong assumption of distant supervision. Typically, the wrongly labeled instances comprise numbers of incomplete and inaccurate annotations, while most prior denoising works are only concerned with one kind of noise and fail to fully explore useful information in the training set. To address this issue, we propose a robust learning paradigm named Self-Collaborative Denoising Learning (SCDL), which jointly trains two teacher-student networks in a mutually-beneficial manner to iteratively perform noisy label refinery. Each network is designed to exploit reliable labels via self denoising, and two networks communicate with each other to explore unreliable annotations by collaborative denoising. Extensive experimental results on five real-world datasets demonstrate that SCDL is superior to state-of-the-art DS-NER denoising methods.

## introduction named entity recognition (ner) is the task of detecting entity spans and then classifying them into predefined categories, such as person, location and organization. due to the capability of extracting entity information and benefiting many nlp applications (e.g., relation extraction @xcite , question answering @xcite ), ner appeals to many researchers. traditional supervised methods for ner require a large amount of high-quality corpus for model training, which is extremely expensive and time-consuming as ner requires token-level labels. therefore, in recent years, distantly supervised named entity recognition (ds-ner) has been proposed to automatically generate labeled training set by aligning entities in knowledge bases (e.g., freebase) or gazetteers to corresponding entity mentions in sentences. this labeling procedure is based on a strong assumption that each entity mention in a sentence is a positive instance of the corresponding type according to the extra resources. however, this assumption is far from reality. due to the limited coverage of existing resources, many entity mentions in the text cannot be matched and are wrongly annotated as non-entity, resulting in incomplete annotations. moreover, two entity mentions with the same surface name can belong to different entity types, thus simple matching rules may fall into the dilemma of labeling ambiguity and produce inaccurate annotations. as illustrated in figure 1 , the entity mention "jack lucas" is not recognized due to the limited coverage of extra resources and "amazon" is wrongly labeled with organization type owing to the labeling ambiguity. recently, many denoising methods @xcite @xcite @xcite have been developed to handle noisy labels in ds-ner. for example, @xcite obtained high-quality phrases through autophrase @xcite and designed autoner to model these phrases that may be potential entities. @xcite proposed a positive-unlabeled learning algorithm to unbiasedly and consistently estimate the ner task loss, and @xcite used negative sampling to eliminate the misguidance brought by unlabeled entities. though achieving good performance, most studies mainly focus on solving incomplete annotations with a strong assumption of no inaccurate ones existing in ds-ner. meanwhile, these methods aim to reduce the negative effect of noisy labels by weakening or abandoning the wrongly labeled instances. hence, they can at most alleviate the noisy supervision and fail to fully mine useful information from the mislabeled data. intuitively, if we can rectify those unreliable annotations into positive instances for model training, a higher data utilization and better performance will be achieved. we argue that an ideal ds-ner denoising system should be capable of solving two kinds of label noise (i.e., incomplete and inaccurate annotations) and making full use of the whole training set. in this work, we strive to reconcile this gap and propose a robust learning framework named scdl (self-collaborative denoising learning). scdl co-trains two teacher-student networks to form inner and outer loops for coping with label noise without any assumption, as well as making full exploration of mislabeled data. the inner loop inside each teacher-student network is a self denoising scheme to select reliable annotations from two kinds of noisy labels, and the outer loop between two networks is a collaborative denoising procedure to rectify unreliable instances into useful ones. specifically, in the inner loop, each teacher-student network selects consistent and high-confidence labeled tokens generated by the teacher to train the student, and then updates the teacher gradually via exponential moving average (ema) 2 based on the re-trained student. and as for the outer loop, the high-quality pseudo labels generated by one network's teacher are used to update the noisy labels of the other network thanks to the stability of ema and different noise sensitivities between two networks. moreover, the inner and outer loop procedures will be performed alternately. obviously, a successful self denoising process (inner loop) can generate high-quality pseudo labels which benefit the collaborative learning procedure (outer loop) a lot and a promising outer loop will promote the inner loop by refining noisy labels, thus handling the label noise in ds-ner effectively. we evaluate our method on five ds-ner datasets. experimental results indicate that scdl consistently achieves superior performance over previous competing approaches. extensive valida-2 a momentum technique that has been explored in several studies, e.g., adam (kingma and ba, 2015), semi-supervised @xcite and selfsupervised @xcite learning. ## related work many studies have obtained reliable performance in ner. for example, bilstm-crf @xcite and bert @xcite based methods become the paradigm in ner due to their promising performances. however, most of these works rely on high-quality labels, which are quite expensive. to address this issue, several studies attempted to annotate tokens via distant supervision @xcite . they matched unlabeled sentences with external gazetteers or knowledge graphs (kgs). despite the success of distant supervision, it still suffers from noisy labels (i.e., incomplete and inaccurate annotations in ner). ## task definition given the training corpus d where each sample is a form of (x i , y i ), x @xmath0, x 2 , ..., x n represents a sentence with n tokens and y @xmath1, y 2 , ..., y n is the corresponding tag sequence. each entity mention @xmath2, ..., x j (0 ≤ @xmath3, associated with an entity type, e.g., person, location. in this paper, we use the bio scheme following @xcite . in detail, the begin token of an entity mention is labeled as b-type and others are i-type. the non-entity tokens are annotated as o. the traditional ner problem is a supervised learning task by fitting a sequence labeling model based on the training dataset. however, we mainly explore the practical scenario when the labels of training data are contaminated due to the distant supervision. in other words, the revealed tag y i may not correspond to the underlying correct one. the challenge posed in this setting is to reduce the negative influence of noisy annotations and generate high-confidence labels for them to make full use of the training data. ## methodology in this section, we give a detailed description of our self-collaborative denoising learning framework, which consists of two interactive teacher-student networks to address both the incomplete and inaccurate annotation issues. as illustrated in figure 2 , each teacher-student network contributes to an inner loop for self denoising and the outer loop between two networks is a collaborative denoising scheme. these two procedures can be optimized in a mutually-beneficial manner, thus improving the performance of the ner system. ## experiments in this section, we evaluate the performance of scdl, compared with several comparable baselines. additionally, we conduct lots of auxiliary experiments and provide comprehensive analyses to justify the effectiveness of scdl. ## conclusion and future work this paper proposes scdl to handle two kinds of label noise in ds-ner without any assumption. with devised teacher-student network and co-training paradigm, scdl can not only exploit more reliable annotations to avoid the negative effect of noisy labels but also explore more useful information from the mislabeled data. experimental results confirm its effectiveness and robustness in dealing with the label noise. for future work, data augmentation is worth exploring in our framework. besides, scdl can also be adapted to other nlp denoising tasks, e.g., classification and matching.