SANTA : Separate Strategies for Inaccurate and Incomplete Annotation Noise in Distantly-Supervised Named Entity Recognition (2023)

Abstract: Distantly-Supervised Named Entity Recognition effectively alleviates the burden of time-consuming and expensive annotation in the supervised setting. But the context-free matching process and the limited coverage of knowledge bases introduce inaccurate and incomplete annotation noise respectively. Previous studies either considered only incomplete one or indiscriminately handle two types of noise with the same strategy. In this paper, we argue that the different causes of two types of noise bring up the requirement of different strategies in model architecture. Therefore, we propose the SANTA to handle these two types of noise separately with (1) Memory-smoothed Focal Loss and Entity-aware KNN to relieve the entity ambiguity problem caused by inaccurate annotation, and (2) Boundary Mixup to alleviate decision boundary shifting problem caused by incomplete annotation and a noise-tolerant loss to improve the model’s robustness. Benefiting from our separate tailored strategies, we confirm in the experiment that the two types of noise are well mitigated.SANTA also achieves a new state-of-the-art on five public datasets.

## introduction as a fundamental task in nlp, named entity recognition (ner) aims to locate and classify named entities in text, which plays an important role in many tasks such as knowledge graph construction @xcite and relation extraction @xcite . to alleviate the burden of annotation in the supervised setting, distantly-supervised named entity recognition (ds-ner) is widely used in real-world scenarios. it can automatically generate labeled training data by matching entities in existing knowledge bases with snippets in plain figure 1 : a sample generated by distant supervision. "amazon" and "washington" are inaccurate annotations. "410 terry ave n" is the incomplete annotation. due to the sensitivity to the noise, the original supervised methods achieve poor performances in ds-ner. therefore, many works have been proposed to handle the issue. some works attempted to focus on solving incomplete annotation noise in ds-ner, including positive-unlabeled (pu) learning @xcite , negative sampling @xcite , and retrieval augmented inference with contrastive learning @xcite . however, the ignorance of inaccurate annotation noise limits the model to further improve the performance. recently, @xcite jointly trained two teacher-student networks to handle the two types of noise with the same strategy. @xcite adopted a noise-robust learning scheme and self-training for the whole training set to avoid overfitting in noise. however, by handling both types of noise with the same strategy, these methods failed to address the unique characteristics of different type of noise, thereby limiting their ability to effectively handle the noise in ds-ner. as the causes of the two types of noise are different, both of two noise may lead to different problems in ds-ner task. inaccurate annotation noise in training data can lead to serious entity ambiguity problem in the ds-ner. as exampled in figure 1 , when a model is trained on data where "washington" is consistently labeled as "per" (person) due to context-free matching, the model may continue to predict "washington" as a "per" even in contexts where it should be labeled as a "loc" (location), such as when it refers to the city of washington. incomplete annotation noise can lead to the decision boundary shifting problem @xcite in the ds-ner task. this problem occurs when the model is trained on data where some entity spans are not labeled, causing the model to shift its decision boundary, making it more likely to predict an entity span as a non-entity type. therefore, the noise in the spans labeled as entities by distant supervision leads to the ambiguity problem, and the noise in the spans labeled as non-entities leads decision boundary shifting problem. to further improve the performance in ds-ner, we argue that the two types of noise should be handled separately with specialized designs in model architecture. this can help the model to address the specific problems posed by each type of noise and lead to the better overall performance of the model. in this paper, we propose the separate strategies for inaccurate and incomplete annotation noise in ds-ner (santa). unlike previous works in ds-ner, we introduce different strategies to handle the two types of noise respectively. for inaccurate annotation, we propose memory-smoothed focal loss (mfl) and entity-aware knn. these strategies aim to address the entity ambiguity problem posed by inaccurate annotation noise. for incomplete annotation, we propose boundary mixup to handle decision boundary shifting problem caused by incomplete annotation, which generates augmented instances by combining the instances around the boundary and the entity instances. due to further training on the augmented instances, the biased decision boundary can be pushed towards right (fully supervised) side as the augmented instances exist between the biased boundary and the right boundary. meanwhile, we empirically analyze the characteristics of incomplete annotation noise, then adopt a noise-tolerant loss to further improve the model's robustness to this type of noise. experiments show santa achieves state-of-theart on five public ds-ner datasets. further analysis shows the effectiveness of each designed module and separate handling. ## related work to address data scarcity problem, several studies attempted to annotate datasets via distant supervision. using external knowledge bases can easily get training data through string matching, but introduces two issues: inaccurate annotation noise and incomplete annotation noise. to address these issues, various methods have been proposed. ## method as shown in figure 2 , santa separately handles the spans labeled as entities and labeled as non-entities. santa uses the mfl and entityaware knn to address the entity ambiguity problem caused by inaccurate annotation. santa adopts noise-tolerant gce + sr loss and boundary mixup to handle the incomplete annotation. ## experiment compared with extensive baselines, santa achieves significant improved performance in five datasets. we also conduct experiments and provide analyses to justify the effectiveness of santa.