MP roto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition (2023)

Abstract: Distantly supervised named entity recognition (DS-NER) aims to locate entity mentions and classify their types with only knowledge bases or gazetteers and unlabeled corpus. However, distant annotations are noisy and degrade the performance of NER models. In this paper, we propose a noise-robust prototype network named MProto for the DS-NER task. Different from previous prototype-based NER methods, MProto represents each entity type with multiple prototypes to characterize the intra-class variance among entity representations. To optimize the classifier, each token should be assigned an appropriate ground-truth prototype and we consider such token-prototype assignment as an optimal transport (OT) problem. Furthermore, to mitigate the noise from incomplete labeling, we propose a novel denoised optimal transport (DOT) algorithm. Specifically, we utilize the assignment result between *Other* class tokens and all prototypes to distinguish unlabeled entity tokens from true negatives. Experiments on several DS-NER benchmarks demonstrate that our MProto achieves state-of-the-art performance. The source code is now available on Github.

## introduction named entity recognition (ner) is a fundamental task in natural language processing and is essential in various downstream tasks @xcite @xcite . most state-of-the-art ner models are based on deep neural networks and require massive annotated data for training. however, human annotation is expensive, time-consuming, and often unavailable in some specific domains. to mitigate the reliance on human annotation, distant supervision is widely applied to automatically generate annotated data with the aid of only knowledge bases or gazetteers and unlabeled corpus. despite its easy accessibility, the distantlyannotated data is rather noisy, severely degrading the performance of ner models. we observe two types of issues in the distantly-supervised named entity recognition task. the first issue is incomplete labeling. due to the limited coverage of dictionaries, entities not presented in dictionaries are mistakenly labeled as other class (denoted as o class). for instance, in figure 1 , "micheal" is not covered in the dictionary and thus cannot be correctly labeled by distant supervision. these unlabeled entities will misguide ner models to overfit the label noise, particularly hurting recall. the second issue is intra-class variance. as illustrated in figure 1 , tokens of the same class (e.g., "mainstay" and "funds") may fall into different sub-clusters in feature space due to semantic differences. traditional single-prototype classifiers do not consider the semantic difference within the same entity type. they set only one prototype for each type, which suffers from intra-class variance. various methods have been proposed to mitigate the noise of incomplete labeling. for example, autoner @xcite tries to modify the standard crf-based classifier to adapt to the noisy ner datasets. self-learning-based methods @xcite @xcite learn by soft labels generated from a teacher model for denoising. positive-unlabeled learning methods treat tokens of o class as unlabeled samples and optimize with an unbiasedly estimated empirical risk @xcite . some other studies utilize negative sampling to avoid ner models overfitting label noise in o class @xcite . despite these efforts, the issue of intra-class variance has not yet been studied in previous ds-ner research. unlike previous methods, we propose a novel prototype-based network named mproto for distantly-supervised named entity recognition. in mproto, each class is represented by multiple prototypes so that each prototype can represent a certain sub-cluster of an entity type. in this way, our mproto can characterize the intra-class variance. to optimize our multiple-prototype classifier, we assign each input token with an appropriate ground-truth prototype. we formulate such tokenprototype assignment problem as an optimal transport (ot) problem. moreover, we specially design a denoised optimal transport (dot) algorithm for tokens labeled as o class to alleviate the noise of incomplete labeling. specifically, we perform the assignment between o tokens and all prototypes and regard o tokens assigned to o prototypes as true negatives while tokens assigned to prototypes of entity classes as label noise. based on our observation, before overfitting, unlabeled entity tokens tend to be assigned to prototypes of their actual classes in our similarity-based token-prototype assignment. therefore, unlabeled entity tokens can be discriminated from clean samples by our dot algorithm so as not to misguide the training. our main contributions are three-fold: ## method in this section, we first introduce the task formulation in section 2.1. then, we present our mproto network in section 2.2. to compute the crossentropy loss, we assign each token with a groundtruth prototype, and the token-prototype assignment will be discussed in section 2.3. besides, to mitigate the incomplete labeling noise in o tokens, we propose the denoised optimal transport algorithm which will be specified in section 2.4. figure 2 illustrates the overview of our mproto. ## related work named entity recognition is a fundamental task in natural language processing and has been applied to various downstream tasks @xcite @xcite @xcite @xcite . ner methods can be divided into two main categories: tagging-based and span-based. tagging-based methods @xcite predict a label for each token, which perform well at detecting flat named entities while failing at detecting nested named entities. span-based methods @xcite @xcite perform classification over span representations, which performs well on the nested ner task but is inferior in computational complexity. tan et al. (2021); shen et al. (2022); wu et al. (2022a) design a query-based ner framework that optimizes entity queries using bipartite graph matching. recently, some generative ner methods @xcite @xcite have been proposed with superior performance on various ner tasks. these supervised ner methods require a large amount of annotated data for training. ## conclusion in this paper, we present mproto, a prototype-based network for the ds-ner task. in mproto, each category is represented with multiple prototypes to model the intra-class variance. we consider the token-prototype assignment problem as an optimal transport problem and we apply the sinkhornknopp algorithm to solve the ot problem. besides, to mitigate the noise of incomplete labeling, we propose a novel denoised optimal transport algorithm for o tokens. experiments show that our mproto has achieved sota performance on various ds-ner benchmarks. visualizations and detailed analysis have confirmed that our mproto can successfully characterize the intra-class variance, and the denoised optimal transport can mitigate the noise of incomplete labeling.