A Class-Rebalancing Self-Training Framework for Distantly-Supervised Named Entity Recognition (2023)

Abstract: Distant supervision reduces the reliance on human annotation in the named entity recognition tasks. The class-level imbalanced distant annotation is a realistic and unexplored problem, and the popular method of self-training can not handle class-level imbalanced learning. More importantly, self-training is dominated by the high-performance class in selecting candidates, and deteriorates the low-performance class with the bias of generated pseudo label. To address the class-level imbalance performance, we propose a class-rebalancing self-training framework for improving the distantly-supervised named entity recognition. In candidate selection, a class-wise flexible threshold is designed to fully explore other classes besides the high-performance class. In label generation, injecting the distant label, a hybrid pseudo label is adopted to provide straight semantic information for the low-performance class. Experiments on five flat and two nested datasets show that our model achieves state-of-the-art results. We also conduct extensive research to analyze the effectiveness of the flexible threshold and the hybrid pseudo label.

## introduction the named entity recognition (ner) task recognizes the location and classification of the named entity. to reduce the reliance on the human annotation of the supervised ner, some works turn to distant supervision to generate large-scale labeled data automatically @xcite @xcite . distant supervision is to match the words in sentences with labeled concepts in the collected knowledge bases @xcite . the distantly-labeled data obtained from rule-based matching is accompanied by noisy labels. previous works in distant supervision mainly focus on the unlabeled entity @xcite and mislabeled entity @xcite . the class-level imbalanced distant annotation has been underestimated in the distantly supervised named entity recognition (ds-ner), where the distant label of the entity class varies in quality, as shown in figure 1a . more specifically, the classwise quality of the distant label depends on the coverage of class-related knowledge bases, and it is hard for the knowledge bases to include all the entities of the semantic-rich class comprehensively. the entity class with the high-quality distant annotation induces the high-performance class, and the low-quality distant annotation is related to the low-performance class. while self-training @xcite is an effective method in the ds-ner task @xcite @xcite , they have not been thoroughly evaluated on the class-level imbalanced learning. selftraining uses the prediction of the model itself to train again, and effectively uncovers the unlabeled entity. the following works study the mislabeled entity from two aspects: candidate selection and label generation. for example, scdl @xcite selects consistent and high-confidence data for model training; roster @xcite generates pseudo labels with the prediction of the contextualized augmented data. however, the initial model in self-training is trained on noisy data and is biased toward the high-performance class, then the subsequent training intensifies the bias and deteriorates the low-performance class, as shown in figure 1 . in figure 1b , the selected candidates are dominated by the high-performance class, as the recall is larger than the precision only in the highperformance class. this tendency selection can improve the generalization of the high-performance class, but impair the exploration of other lowperformance classes. actually, a predefined constant threshold struggles to handle the difference in class-wise learning ability @xcite , and limits the model to only focus on the highperformance class. in figure 1c , the generated pseudo label fails to explore the low-performance class during self-training, as the performance degradation occurs in the low-performance class. when the generated pseudo label from the biased model misleads the semantic information of the lowperformance class, the iterative update with the guide of pseudo label expands the negative impacts in the low-performance class @xcite . in this work, we propose a unified self-training framework, called clim, to address the class-level imbalance learning in the ds-ner task. for the dominance from the high-performance class, we calculate the current learning ability for each entity class, and adjust the class-wise threshold to improve the candidate selection. for the degradation in the low-performance class, we leverage the semantic information in the distantly-labeled entities, and generate a hybrid pseudo label to improve the label generation. the above two parts of candidate selection and label generation are mutually beneficial. the generated hybrid pseudo label improves the feature capture for the low-performance class by injecting the distant label. and better feature representation improves the exploration of the low-performance class, as more candidates from the low-performance class are selected through the class-wise threshold. the contributions are as follows: self-training with data augmentation. selftraining @xcite consists of both candidate selection and label generation. specifically, self-training only selects candidate whose largest class probability fall above a predefined threshold; the generated pseudo label comes from the prediction of the model itself. referred to self-training in the semi-supervised learning @xcite , the perturbed inputs with different augmentation is used to decouple the similar predictions on the same input. and also, this data augmentation improve the model robustness and achieves competitive performance @xcite . different from the previous works that focus on the classification task with the external task-relevant unlabeled data, our work extends augmentation-driven self-training to the named entity recognition task with only noisy data. backbone. for the contextual span representation h (s i ) = x start(i) ; x end(i) ; ϕ (s i ) , x start(i) and x end(i) is the embedding of the start and end token in span s i , ϕ (s i ) is the span width embedding with the random initialization. and the output of classifier g θ is the probability distribution over entity classes, which is formulated as f θ (s i ) = g θ (h (s i )) ∈ r c . among them, θ represents the learnable parameters, and c is the number of entity classes. for simplicity, the probability distribution f θ (s i ) is represented as p i . ## clim we advance self-training to tackle the class-level imbalance learning, with more detailed consideration in the candidate selection and label generation. the overview of our framework is illustrated in figure 2 , and the training algorithm is shown in algorithm 1. ## conclusion this work advances the class-rebalancing selftraining in the distantly-supervised named entity recognition. with the class-wise flexible threshold and the fine-grained hybrid pseudo label in self-training, our work tackles the dominance from the high-performance class and the degradation in the low-performance class. on this basis, the experiments show state-of-the-art results on seven benchmarks. and the comprehensive analysis further proves the more consistent performance in class-level learning and the stronger semantics classification ability. our work, especially the advanced designs in self-training, positively impacts robust learning with noisy data. it provides a classrebalancing method to explore the semantic information in distantly-labeled data.