Sampling Better Negatives for Distantly Supervised Named Entity Recognition (2023)

Abstract: Distantly supervised named entity recognition (DS-NER) has been proposed to exploit the automatically labeled training data instead of human annotations. The distantly annotated datasets are often noisy and contain a considerable number of false negatives. The recent approach uses a weighted sampling approach to select a subset of negative samples for training. However, it requires a good classifier to assign weights to the negative samples. In this paper, we propose a simple and straightforward approach for selecting the top negative samples that have high similarities with all the positive samples for training. Our method achieves consistent performance improvements on four distantly supervised NER datasets. Our analysis also shows that it is critical to differentiate the true negatives from the false negatives.

## introduction named entity recognition (ner) is one of the fundamental tasks in natural language processing, and it aims to extract the mentioned entities in the text. existing supervised approaches @xcite @xcite have achieved great performance on many ner datasets. however, they still heavily rely on the human-annotated training datasets. distantly supervised approaches @xcite @xcite @xcite @xcite @xcite have been proposed to exploit the automatically labeled training data generated from the knowledge bases (kbs) or dictionaries. for such distantly supervised datasets, the annotated entities mostly have correct labels, but the overall annotations are frequently incomplete due to the limited coverage of entities in kbs. we include comparisons of the distantly-annotated and human-annotated datasets in appendix a. self-training has been demonstrated as an effective strategy for addressing the noisy labeled training data @xcite @xcite @xcite . specifically, they iteratively refine the entity labels through teacher-student models. in this way, the number of false positive and false negative samples can be reduced. however, such approaches usually require training multiple models with multiple iterations. another line of work improves the single-stage model by reducing the number of false negative samples during training. in general, the problem of false negatives is more severe than false positives. @xcite proposed to sample a portion of negative samples with a uniform sampling distribution for training. by selecting a subset of all the negative samples, fewer false negative samples are involved in the training process. the sampling strategy can be enhanced by using a weighted sampling distribution @xcite . specifically, the negative samples are assigned with different sampling probabilities based on the predicted label distributions. however, this approach also depends on quality of the classifier to derive the distribution. in this paper, we propose a simple and straightforward approach to sampling the negatives for training. intuitively, the false negatives are positive samples but are unrecognized based on distant supervision, and they should have high similarities with positive samples that have the same gold entity type. for the example in fig. 1 , "michelle obama" is not identified as per in the distantly labeled dataset. the false negative sample "michelle obama" should have high similarity with the positive sample "barack obama" that has the per label. additionally, the false negative sample should not have high similarity with other positive samples of different entity types, such as "boston". therefore, when the negative samples have high similarities with all the positive samples, they are more likely to be true negatives. we select these top negative samples for training, and we denote our approach as top-neg. unlike the previous approach of relying on a classifier for assigning sampling probability, our approach exploits the encoded representations to derive the similarity scores. compared to the baseline methods, our approach demonstrates consistent performance improvement on four distantly supervised ner datasets. our analysis shows that not all the negative samples are required for training, but it is critical to filter the false negatives. ## approach the objective of ner task is to extract all the entities in the text. given a sentence of length n, @xmath0, x 2 , ..., x n }, we denote all possible enumerated spans in x as @xmath1,1 , s 1,2 , ..., s i,j , ..., s n,n }, where i and j are the start and end position of span s i,j , and the span length is limited as 0 ≤ j -@xmath2. for all the enumerated spans, our approach predicts the corresponding entity types from a predefined label space, including the o label (not an entity). ## experiments datasets we evaluate our approach on four distantly supervised ner datasets: conll03 (tjong kim sang and de meulder, 2003), bc5cdr @xcite , wnut16 @xcite , and wikigold @xcite . the distantly supervised datasets are obtained from @xcite and @xcite . we use the distantly supervised data for training and the humanannotated development and test sets for evaluation. the statistics of the datasets are given in appendix a. ## conclusion in this work, we propose an improved approach of sampling the negatives to reduce the number of false negative samples for training. specifically, we differentiate the true negatives from the false negative samples by measuring the similarity of the negatives with the positive samples. the experiment results have demonstrated the effectiveness of our approach. future work may focus on clustering the negative samples to further differentiate the true negatives from the false negatives.