Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training (2021)

Abstract: We study the problem of training named entity recognition (NER) models using only distantly-labeled data, which can be automatically obtained by matching entity mentions in the raw text with entity types in a knowledge base. The biggest challenge of distantly-supervised NER is that the distant supervision may induce incomplete and noisy labels, rendering the straightforward application of supervised learning ineffective. In this paper, we propose (1) a noise-robust learning scheme comprised of a new loss function and a noisy label removal step, for training NER models on distantly-labeled data, and (2) a self-training method that uses contextualized augmentations created by pre-trained language models to improve the generalization ability of the NER model. On three benchmark datasets, our method achieves superior performance, outperforming existing distantly-supervised NER models by significant margins.

## introduction named entity recognition (ner), which aims at identifying real-world entity mentions (e.g., person, location and organization names) from texts, is a fundamental task in natural language processing with a wide range of applications, including question answering @xcite , knowledge base construction @xcite , text summarization @xcite and dialog systems @xcite . in recent years, deep neural models @xcite @xcite have achieved enormous success for ner, thanks to their strong representation learning power that accurately captures the entity semantics in textual contexts. however, a common bottleneck of applying deep learning models is the acquisition of abundant high-quality human annotations, and this is especially the case for training ner models, which require every entity mention to be labeled in a sequence. to eliminate the need for human annotations, one direction is to use distant supervision for automatic generation of entity labels. the common practice is to match entity mentions in the target corpus with typed entities in external gazetteers or knowledge bases. unfortunately, such a distant labeling process inevitably introduces incomplete and noisy entity labels, because (1) the distant supervision source has limited coverage of the entity mentions in the target corpus, and (2) some entities can be matched to multiple types in the knowledge bases-such ambiguity cannot be resolved by the context-free matching process. figure 1 shows that some "person" mentions may be partially labeled (or not labeled at all in other cases), and some entities with multiple possible types may be mislabeled. due to the existence of such noise, straightforward application of supervised learning to distantlylabeled data will yield deteriorated performance, because neural models have the strong capacity to fit to the given (noisy) data. some previous studies on distantly-supervised ner directly treat distant labels as if ground truth for model training and rely on simple tricks such as applying early stopping @xcite and labeling entities with multiple types @xcite to handle the noise. others require an additional manually labeled training set for building a noise classification model @xcite . in this paper, we study the distantly-supervised ner problem without requiring any human annotations. our method consists of two steps: (1) noiserobust learning, and (2) language model augmented self-training. in the first step, we explicitly address the label noise by using a noise-robust loss function and removing noisy labels. in the second step, we use the model's high-confidence predictions for self-training to improve generalization, wherein a pre-trained language model is used to not only initialize the ner model, but also generate contextualized augmentations. our method is named roster, for robust learning and self-training for distantly-supervised entity recognition. the contributions of this paper are as follows: ## method in this section, we (1) briefly describe how to obtain distantly-labeled data, (2) introduce our noiserobust learning scheme and ## related work the effectiveness of deep neural models for ner usually comes with the cost of annotating large amounts of training data. to alleviate the human annotation burden when applying deep models, several studies propose to train ner models with weakly/distantly-labeled data. for weaklysupervised ner, previous studies have explored cross lingual knowledge transfer from high resource languages to low resource languages @xcite @xcite , aggregating multiple weak labeling functions @xcite or leveraging sentence-level labels @xcite . few-shot approaches @xcite have also been explored to leverage very few labeled data for ner model training. our work is more closely related to distantlysupervised ner which uses external gazetteers or knowledge bases to automatically derive entity labels. along this line, different methods have been proposed to leverage the distant supervision, such as propagating reliable type information on graphs @xcite , designing new model components to handle multiple possible labels @xcite , employing additional models to classify noisy data (onoe and durrett, distant match: shanghai-ek [chor] per is jointly owned by the shanghai automobile corporation and [ek chor] per china motorcycle. ground truth: [shanghai-ek chor] org is jointly owned by the [shanghai automobile corporation] org and [ek chor china motorcycle] org . autoner: shanghai-ek [chor] per is jointly owned by the shanghai automobile corporation and [ek chor] per [china] loc motorcycle. bond: [shanghai-ek chor] per is jointly owned by the [shanghai] loc [automobile corporation] org and [ek chor] per [china motorcycle] org . roster: [shanghai-ek chor] org is jointly owned by the [shanghai automobile corporation] org and [ek chor china motorcycle] org . our study is also related to data augmentation techniques. in nlp, data augmentation is well developed for text classification, by either creating real text sequences @xcite via back translation @xcite or in the hidden states of the model via perturbations @xcite or interpolations by mixing up labels @xcite . however, these techniques cannot be readily used for the ner task. @xcite study a set of simple augmentation methods for the ner task, like synonym replacement, mention replacement or segment shuffling. nevertheless, these augmentations are context-free which may generate unreasonable sequences or require additional sources like the wordnet. our proposed augmentation method is unsupervised and contextualized, generating high-quality sequences thanks to the pre-trained knowledge of plms and reliably improving model generalization. ## conclusion and future work in this paper, we study the distantly-supervised ner problem without using any human annotations but only distantly-labeled data. for better model training with noisy data, we propose a noiserobust learning scheme, consisting of a new loss function and a noisy label removal step. to further improve the model generalization, we propose a self-training method that guides model refinement with its own high-confidence predictions and enforces the model to make consistent predictions on original and augmented sequences generated by plms. our method achieves strong performance on three benchmark datasets, outperforming previous distantly-supervised ner methods. the techniques proposed in this paper are generalizable for future studies: the noise-robust learning scheme may also be applied to other nlp problems where labels may contain noise (e.g., obtained via crowdsourcing from non-experts); the augmentation and self-training method may be helpful for other settings like semi-supervised or few-shot learning. one may also consider exploring larger pre-trained language models (e.g., roberta-large) or more recent pre-trained language models (e.g., coco-lm @xcite ) for the distantlysupervised ner task.