Rethinking Negative Sampling for Handling Missing Entity Annotations (2022)

Abstract: Negative sampling is highly effective in handling missing annotations for named entity recognition (NER). One of our contributions is an analysis on how it makes sense through introducing two insightful concepts: missampling and uncertainty. Empirical studies show low missampling rate and high uncertainty are both essential for achieving promising performances with negative sampling. Based on the sparsity of named entities, we also theoretically derive a lower bound for the probability of zero missampling rate, which is only relevant to sentence length. The other contribution is an adaptive and weighted sampling distribution that further improves negative sampling via our former analysis. Experiments on synthetic datasets and well-annotated datasets (e.g., CoNLL-2003) show that our proposed approach benefits negative sampling in terms of F1 score and loss convergence. Besides, models with improved negative sampling have achieved new state-of-the-art results on real-world datasets (e.g., EC).

## introduction with powerful neural networks and abundant well-labeled corpora, named entity recognition (ner) models have achieved promising performances @xcite @xcite . however, in many scenarios, available training data is lowquality, which means a portion of named entities are absent in annotations. fig. 1 depicts a sentence and its incomplete annotations. fine-grained ner @xcite is a typical case. its training data is mainly obtained through applying weak supervision to unlabeled corpora. past works @xcite find missing annotations impact ner models and refer this to unlabeled entity problem. recently, @xcite find it's the misguidance of unlabeled entities to ner models in train-japan , co-hosts of the world cup in 2002 and ranked 20th in the world by fifa , are favourites to regain their title here . ing that causes their poor performances. to eliminate this adverse impact, they propose a simple yet effective approach based on negative sampling. compared with its counterparts @xcite @xcite , this method is of high flexibility, without relying on external resources, heuristics, etc. while negative sampling has handled missing annotations well, there is no systematic study on how it works, especially what potential factors are involved. from a number of experiments, we find missampling and uncertainty both worth receiving attention. missampling means that some unlabeled entities are mistakenly drawn into the set of training negatives by negative sampling. to quantitively describe this, we define missampling rate, the proportion of unlabeled entities in sampled negatives, for a sentence. uncertainty indicates how hard a sampled negative is for ner models to recognize, and we use entropy to estimate it. empirical studies show low missampling rate and high uncertainty are both indispensable for effectively applying negative sampling. besides, based on the observation that entities are commonly sparse, we provide a lower bound for the probability of zero missampling rate with theoretical proof, which is only related to sentence length. originally, @xcite adopt uniform sampling distribution for negative sampling. inspired by former findings, we introduce a weighted sampling distribution to displace the uniform one, which takes missampling and uncertainty into account. our distribution is purely computed from the predictions of an ner model. this means it coevolves with the model throughout the training process. the adaptive property of our method is appealing since it doesn't rely on manual annotations or additional models to indicate valuable negatives. we have conducted extensive experiments to verify the effectiveness of our weighed sampling distribution. results on synthetic datasets and wellannotated datasets (e.g., ontonotes 5.0) show that weighted sampling distribution improves negative sampling in performances and loss convergence. notably, with improved negative sampling, our ner models have established new state-of-the-art performances on real-world datasets, like ec @xcite . ## analyzing negative sampling we analyze how negative sampling leads ner models that suffer from missing entity annotations to promising results from two angles: missampling and uncertainty. ## improving negative sampling the previous section shows that the effectiveness of negative sampling is dependent on two factors: missampling and uncertainty. as a result, if we had considered both quantities when sampling negatives, we should see larger improvements from final models. in this section, we propose an adaptive and weighted sampling distribution based on these two factors. unfortunately, since missampling rate is defined on top of the unlabeled entities y which is unknown in practice, it is not straightforward to apply missampling for improving negative sampling. therefore, we assume that an oracle model, z i,j,@xmath0,j ), exists, which is likely to predict the ground-truth label for every span x i,j . then we define a score v i,j as the difference between the score z i,j,o and the maximum label score on the span (i, j): v i,@xmath1,j,o -max @xmath2,j,l . intuitively, if v i,j is high, then z i,j,o is high and max @xmath3,j,l is low. in other words, x i,j is likely to be with "o" label and thus the missampling rate should be small. hence sampling such a span as a negative won't hurt ner models. note that max @xmath4,j,l in the right hand acts as normalization, making v i,j comparable among different spans (i, j). we also define an uncertainty score, u i,j , as the entropy of the label distribution for a span: as discussed in sec. 3.2.2, training a ner model with the negatives of higher uncertainty scores, u i,j , brings better performances. based on v i,j and u i,j , we design the following weighted sampling distribution to displace the uniform one when sampling k negatives from s without replacement: to make our approach practical for use, we should specify how to approximate the oracle model, p o (l | x i,j ). in the simulation experiment in sec. 3.2.1, the oracle model is a fixed model via standard negative sampling which is learned on held-out training data. it's natural to use such a fixed model to approximate the oracle model here. however, this will cause a side-effect that our approach is not self-contained due to its dependence on an external model. consequently, we consider an adaptive style: directly using the ner model, p (l | x i,j ; θ), itself as the oracle model whose parameter θ is learned during the training process. under this scheme, t is scheduled as √ c -c, where c is the number of training epochs and 0 ≤ @xmath5. since the ner model p (l | x i,j ; θ) is not accurate in early epochs of training, a more uniform sampling distribution (i.e., higher t ) is safer for sampling negatives. finally, we get a weighted sampling distribution with the ner model, p (l | x i,j ; θ), adaptively approximating the oracle model. our training procedure is the same as that of vanilla negative sampling (see fig. 2 ), except for sampling distribution. ## experiments to evaluate our proposed variant (i.e., negative sampling w/ weighted sampling distribution) , we have conducted extensive experiments on underannotated cases: synthetic datasets and real-world datasets. we also validate its superiority in wellannotated scenarios. ## related work a number of ner models @xcite @xcite @xcite based on end-to-end neural networks and well-labeled data have achieved promising performances. a representative work is bi-lstm crf @xcite . however, in many situations (e.g., distantly supervised ner), these seemingly perfect models severely suffer from unlabeled entity problem, where massive named entities are not annotated in training data. there are some techniques developed by earlier works to mitigate this issue. fuzzy crf and autoner @xcite allow ner models to learn from high-quality phrases that might be potential named entities. mining these phrases demands external resources @xcite , which is not flexible for practical usage. moreover, there is no guarantee that unlabeled entities are fully covered by these phrases. pu learning @xcite adopts a weighted training loss and assigns low weights to false negative instances. this approach is limited by requiring prior information or heuristics. partial crf @xcite is an extension of crf, which marginalizes the loss over all candidates that are compatible with the incomplete annotation. while being theoretically attractive, this approach still needs a portion of well-annotated data to obtain true negatives, which limits its use in real-world applications. for example, in fine-grained ner @xcite , all the training data are produced through weak supervision, and its manual anno-tation is very difficult, so obtaining enough highquality data is not practical. recently, @xcite find that unlabeled entities severely misguide the ner models during training. based on this observation, they introduce a simple yet effective approach using negative sampling. it's much more flexible than other methods, without resorting to external resources, heuristics, etc. however, @xcite haven't well explained why negative sampling works and there are weaknesses in their principle analysis. in this paper, we first show two factors that affect how negative sampling avoids ner models from being impacted by missing annotations. notably, a theoretical guarantee is provided for the zero missampling rate. then, we propose weighted sampling distribution to further improve negative sampling based on our former findings. ## conclusion negative sampling succeeds in handling missing annotations. in particular, the fine-grained ner module of our online text understanding service, texsmart @xcite , adopts this technique because of its massive lowquality training data. in this work, we have made two contributions. on the one hand, we analyze why negative sampling succeeds in handling unlabeled entity problem from two perspectives: missampling and uncertainty. empirical studies show both low missampling rates and high uncertainties are essential for applying negative sampling. based on entity sparsity, we also provide a theoretical lower bound for the probability of zero missampling rate. on the other hand, we propose an adaptive and weighted sampling distribution that takes missampling and uncertainty into account. we have conducted extensive experiments to verify whether this further improves the effectiveness of negative sampling. results on synthetic datasets and well-annotated datasets show that our approach benefits in performances and loss convergence. with improved negative sampling, our ner models also have achieved new state-of-the-art results on real-world datasets (e.g., news).