Distantly Supervised Named Entity Recognition via Confidence-Based Multi-Class Positive and Unlabeled Learning (2022)

Abstract: In this paper, we study the named entity recognition (NER) problem under distant supervision. Due to the incompleteness of the external dictionaries and/or knowledge bases, such distantly annotated training data usually suffer from a high false negative rate. To this end, we formulate the Distantly Supervised NER (DS-NER) problem via Multi-class Positive and Unlabeled (MPU) learning and propose a theoretically and practically novel CONFidence-based MPU (Conf-MPU) approach. To handle the incomplete annotations, Conf-MPU consists of two steps. First, a confidence score is estimated for each token of being an entity token. Then, the proposed Conf-MPU risk estimation is applied to train a multi-class classifier for the NER task. Thorough experiments on two benchmark datasets labeled by various external knowledge demonstrate the superiority of the proposed Conf-MPU over existing DS-NER methods. Our code is available at Github.

## introduction named entity recognition (ner) aims to detect entity mentions from text and classify them into predefined types. it is a fundamental task in information extraction and many other downstream tasks @xcite @xcite . however, the necessity of extensive human efforts to annotate a large amount of training data imposes restrictions on the state-of-the-art supervised deep learning methods, especially in professional fields. to address this problem, distantly supervised methods spring up @xcite @xcite @xcite @xcite , which aim to train ner models using automatically annotated training data based on external knowledge such as dictionaries and knowledge bases. observed by previous ds-ner methods @xcite , distant labels provided by reliable dictionaries are usually of high precision. however, such distant labeling suffers a major drawback -incomplete labeling. this is due to the fact that most existing dictionaries and knowledge bases have limited coverage on entities. hence simply treating unlabeled samples (i.e., unmatched tokens) as negative ones will introduce a high false negative rate (e.g., "neutropenia" in figure 1 ) compared with humanannotated training data, and further mislead a supervised ner model to overfit to false negative samples and seriously affect its recall. recently, binary positive and unlabeled (pu) learning is applied to ds-ner tasks for this challenge @xcite . pu learning performs classification using only limited labeled positive data and unlabeled data, thus naturally suitable for handling distant supervision, where external knowledge often has a limited coverage on positive samples. however, binary pu learning has several drawbacks in real ds-ner tasks. it applies the one-vs-all strategy to convert a multi-class classification problem into multiple binary classification problems, and thus suffers from two weaknesses. first, it is not efficient, especially in the case where there are many entity types. for a ner task with n entity types, n binary classifiers need to be trained. second, the scale of predicted confidence values may differ among those binary classifiers, which may not guarantee a mutually beneficial inference for the final prediction @xcite . furthermore, the pu learning theory is built on a fundamental assumption of data distribution that unlabeled data can accurately reveal the overall distribution (i.e., the marginal distribution of the target field) @xcite . in ds-ner tasks, the distantly annotated training data may not fit the assumption well: it depends on the coverage of used dictionaries or knowledge bases on the entities. our empirical studies validate that violation of this assumption can significantly impact the performance of pu learning. to address these challenges in ds-ner tasks and pu learning, we propose a confidence-based multi-class positive and unlabeled (conf-mpu) learning framework. the proposed conf-mpu can handle different levels of false negative rates brought by dictionaries of various coverage and does not overfit to the distantly labeled training data. it consists of two steps. specifically, given the distantly labeled training data, we first carry out a token-level binary classification to estimate the confidence score (a probability value in [0, 1]) of a token being an entity token (i.e., a token of a named entity). then, we perform the ner classification using a neural network model with the proposed conf-mpu risk estimator, which incorporates the confidence scores obtained from the first step in the risk estimation, to alleviate the impact of annotation imperfection. it is worth noting that the two-step strategy of conf-mpu needs to train only two classifiers for any ds-ner tasks with arbitrary number of entity types, which is more efficient than previous binary pu learning. in summary, our main contributions are: ## preliminaries in this section, we briefly review the risk formulations of standard supervised learning and pu learning in the binary classification setting. ## methodology in this section, we introduce the proposed conf-mpu learning for ds-ner in the multi-class classification setting with solid theoretical analysis. ## ds-ner classification in this section, we describe the setup for the ds-ner classification. ## experiments in this section, we evaluate the proposed conf-mpu and compare with other baseline methods. ## conclusion in this paper, we present a novel multi-class positive and unlabeled learning method called conf-mpu for the ds-ner task. conf-mpu estimates the empirical classification risks using the confidence estimation of a token being an entity token on the distantly labeled training data to prevent the model from overfitting to the false negatives. we empirically show that conf-mpu can significantly reduce the potential risk estimation bias caused by pu assumption. the extensive experiments illustrate that compared with existing ds-ner methods, conf-mpu is more robust to various types of dictionaries and can handle the incomplete labeling problem effectively. rconf-mpu (f ) restricts the loss on each labeled positive sample to be at least 0, in the following proof, we will derive the error bounds from rconf-mpu (f ) to r(f ), from r(f ) to r(f ), and from r(f ) to r(f ) in order. let us first derive the error bound from rconf-mpu (f ) to r(f ). for ease of notation, let