SCL - RAI : Span-based Contrastive Learning with Retrieval Augmented Inference for Unlabeled Entity Problem in NER (2022)

Abstract: Unlabeled Entity Problem (UEP) in Named Entity Recognition (NER) datasets seriously hinders the improvement of NER performance. This paper proposes SCL-RAI to cope with this problem. Firstly, we decrease the distance of span representations with the same label while increasing it for different ones via span-based contrastive learning, which relieves the ambiguity among entities and improves the robustness of the model over unlabeled entities. Then we propose retrieval augmented inference to mitigate the decision boundary shifting problem. Our method significantly outperforms the previous SOTA method by 4.21% and 8.64% F1-score on two real-world datasets.

## introduction as a fundamental task in nlp, named entity recognition aims to locate and classify named entities in the text. due to the large-scale wellannotated datasets, deep-learning based methods @xcite have achieved great success. however, in real-world datasets, such as @xcite with 112 fine-grained named entity tags, a large set of entity classes may cause inevitable missing annotations. moreover, to obtain large ner datasets in practical scenarios, the distant supervision approach @xcite may make this problem even worse, since the entity dictionary cannot cover all entities. previous work @xcite find that this problem seriously hinders the performance of the ner model and name this problem as unlabeled entity problem. as shown in figure 1 , the unlabeled second "nba" may confuse model and introduce unnecessary noise. to cope with this problem, several attempts from different perspectives have been proposed. inspired by positive-unlabeled (pu) learning @xcite , @xcite use a weighted loss to assign low weights to false negative words and build distinct binary classifiers for different entity types. however, they require prior information or heuristics @xcite and the unlabeled entities still misguide the classifiers, bringing ambiguity among neighboring entities @xcite . @xcite ; @xcite introduce the partial crf @xcite to marginalize the instances that are consistent with the incomplete annotation. however, they require additional well-annotated corpus to get ground truth negative instances, which are usually unavailable in practice. recently, @xcite perform down-sampling among non-entity instances within annotation when computing loss function, in order to mitigate the misguidance from possible unlabeled entities. @xcite further propose a weighted and adaptive sampling distribution to introduce direction to real unlabeled entities when down-sampling. however, the inherent randomness of sampling strategy may still keep some unlabeled entities when computing loss then make the decision boundary biased @xcite . as shown in figure 2 , the learned decision boundary for training example containing unlabeled entity instances tends to shift from the expected boundary towards the entity side. the previous works do not consider this problem. to deal with these challenges, this paper proposes the span-based contrastive learning with retrieval augmented inference (scl-rai) to tackle unlabeled entity problem, which mitigates labeled entity instance non-entity instance unlabeled entity instance figure 2: illustration for decision boundary shifting phenomenon. the solid line is the learned boundary from datasets with unlabeled entities, the dashed line represents the expected boundary. the limitations mentioned above, i.e., demanding additional corpus, ambiguity among neighboring entities and decision boundary shifting problem. firstly, scl-rai tries to decrease the distance among span representations with the same labels while increasing it for different ones. benefiting from our well-designed span-based contrastive learning, the ambiguity between entities is mitigated by the increased representation distance, so the model can capture the differences among different entity labels. we show in experiment that this contrastive learning objective also improves the model robustness under unlabeled entities. furthermore, we propose retrieval augmented inference to relieve decision boundary shifting phenomenon. it caches the center point representation for each entity type from the training set. then, it computes a label distribution via cached representation and interpolates it with the distribution from the backbone ner model. experiments on two real-world datasets show that scl-rai significantly outperforms previous sota methods. ## methodology our scl-rai consists of three modules: spanbased ner model, span-based contrastive learning, and retrieval augmented inference. ## conclusion we propose the scl-rai to cope with unlabeled entity problem in ner. benefiting from our welldesigned span-based contrastive learning and retrieval augmented inference, experiments on two real-world datasets show that scl-rai achieves more promising results than sota methods.