Noise-Robust Training with Dynamic Loss and Contrastive Learning for Distantly-Supervised Named Entity Recognition (2023)

Abstract: Distantly-supervised named entity recognition (NER) aims at training networks with distantly-labeled data, which is automatically obtained by matching entity mentions in the raw text with entity types in a knowledge base. Distant supervision may induce incomplete and noisy labels, so recent state-of-the-art methods employ sample selection mechanism to separate clean data from noisy data based on the model’s prediction scores. However, they ignore the noise distribution change caused by data selection, and they simply excludes noisy data during training, resulting in information loss. We propose to (1) use a dynamic loss function to better adapt to the changing noise during the training process, and (2) incorporate token level contrastive learning to fully utilize the noisy data as well as facilitate feature learning without relying on labels. Our method achieves superior performance on three benchmark datasets, outperforming existing distantly supervised NER models by significant margins.

## introduction named entity recognition (ner) is a fundamental task in natural language processing, which aims at locating entity mentions in a given sentence and assign them to certain types, and it has a wide range of applications @xcite @xcite . in recent years, deep neural networks have achieved great performance on ner due to their strong ability to learn from large amount of labeled data. however, the acquisition of abundant highquality human annotated data is costly and difficult. a major solution to this problem is distant supervision, a method for automatic generation of entity labels, the common practice of which is to match entity mentions in an unlabeled dataset with typed entities in external gazetteers or knowledge bases. unfortunately, this method inevitably introduces noise while generating labeled dataset, leading to a deterioration of the ner models' performance. there are many works that try to improve the performance of ner networks on distantly supervised dataset with the existence of such noise. some studies use training tricks like applying early stopping @xcite and labeling entities with multiple types @xcite to handle the noise, and some studies build an additional classification model to distinguish noisy labels from the ground truth labels @xcite relying on additional labeled data. sample separation @xcite is a dominant method in noise-robust learning, trying to filter out the noisy samples from the clean ones based on the small-loss criterion @xcite . roster @xcite applies sample separation to distantly-supervised ner, and they also uses generalized cross entropy (gce) @xcite loss for noise-robust training. in this paper, we also use the framework of sample separation and propose a distantly-supervised ner training scheme that uses dynamic gce loss to optimize the model training, and we incorporate contrastive learning to lower the risk of noisy label overfitting and fully utilize the data with untrusted labels. the contributions of this paper are as follows: ## methodology in this section, we (1) briefly describe how to obtain distantly-labeled data, (2) introduce our noiserobust learning scheme with dynamic gce loss and (3) present how to incorporate contrastive learning into our framework. we use the pre-trained roberta @xcite as our backbone model, but our proposed methods can be integrated with other architectures as well. the overall framework of our noise-robust training is showed in figure 1 . ## conclusion in this paper, we study the distantly-supervised ner problem. we propose a noise-robust training scheme using dynamic loss to adapt to changing noise distributions caused by sample selection mechanism. to further improve the model's generalization and robustness, we incorporate contrastive learning to facilitate feature learning without relying on labels and fully utilize the data with noisy labels. we conduct experiments on three datasets and our method outperforms all previous distantlysupervised ner methods.