Improving Self-training for Cross-lingual Named Entity Recognition with Contrastive and Prototype Learning (2023)

Abstract: In cross-lingual named entity recognition (NER), self-training is commonly used to bridge the linguistic gap by training on pseudo-labeled target-language data. However, due to sub-optimal performance on target languages, the pseudo labels are often noisy and limit the overall performance. In this work, we aim to improve self-training for cross-lingual NER by combining representation learning and pseudo label refinement in one coherent framework. Our proposed method, namely ContProto mainly comprises two components: (1) contrastive self-training and (2) prototype-based pseudo-labeling. Our contrastive self-training facilitates span classification by separating clusters of different classes, and enhances cross-lingual transferability by producing closely-aligned representations between the source and target language. Meanwhile, prototype-based pseudo-labeling effectively improves the accuracy of pseudo labels during training. We evaluate ContProto on multiple transfer pairs, and experimental results show our method brings substantial improvements over current state-of-the-art methods.

## introduction cross-lingual named entity recognition (ner) @xcite has seen substantial performance improvement since the emergence of large-scale multilingual pretrained language models @xcite . however, there is still a noticeable gap between zero-shot cross-lingual transfer and monolingual ner models trained with target-language labeled data. to further bridge the linguistic gap between the source and target language, self-training is widely adopted to exploit the abundant languagespecific information in unlabeled target-language data @xcite @xcite . in general, self-training (sometimes referred to as teacher-student learning @xcite ) uses a weak tagger (i.e. teacher model) trained on source-language data to assign pseudo labels onto unlabeled target-language data, which is then combined with labeled source-language data to train the final model (i.e. student model). nevertheless, due to sub-optimal performances on target languages, the pseudo-labeled data contains a large number of errors and might limit the performances of ner models trained on them. to optimize self-training for cross-lingual ner, several methods have been proposed to improve the quality of pseudo labels. one line of work focuses on selecting curated pseudo-labeled data for selftraining via reinforcement learning @xcite or an adversarial discriminator @xcite . however, they do not fully utilize all the unlabeled data available. @xcite exploit the full unlabeled dataset and alleviate the noise in pseudo labels by aggregating predictions from multiple teacher models. likewise, @xcite develop multi-round self-training which iteratively re-trains the teacher model to generate more accurate pseudo-labels. despite their effectiveness, both multi-teacher and multi-round self-training impose a large computational overhead. furthermore, the aforementioned methods are mostly data-driven and ignore the explicit modeling of cross-lingual alignment in the representation space. in this work, we take a different approach and propose contproto as a novel self-training framework for cross-lingual ner. unlike existing data selection methods, contproto sufficiently leverages knowledge from all available unlabeled targetlanguage data. compared with multi-teacher or multi-round self-training, our method improves pseudo label quality without training separate mod-els. moreover, we explicitly align the representations of source and target languages to enhance the model's cross-lingual transferability. specifically, contproto comprises two key components, namely contrastive self-training and prototypebased pseudo-labeling. firstly, we introduce a contrastive objective for cross-lingual ner selftraining. whereas typical supervised contrastive learning @xcite treats labeled entities of the same class as positive pairs, we further construct pseudo-positive pairs comprising of a labeled source-language entity and a target-language span predicted as the same entity type by the current model. hence, such contrastive objective not only separates different entity classes for easier classification, but also better aligns representations of the source and target language, achieving enhanced cross-lingual transferability. secondly, we propose a prototype-based pseudo-labeling to refine pseudo labels on-the-fly at each training step. we start with constructing class-specific prototypes based on the representations produced by contrastive selftraining, which can be regarded as cluster centroids of each entity type. then, by ranking the distances between the representation of an unlabeled span and each prototype, we gradually shift its soft pseudo label towards the closest class. as a result, errors in pseudo labels are dynamically corrected during training. it is noteworthy that our contrastive self-training and prototype-based pseudo-labeling are mutually beneficial. on one hand, entity clusters generated by contrastive learning make it easier to determine the closest prototype and update pseudo labels correctly. in turn, the model trained on the refined pseudo labels becomes more accurate when classifying unlabeled spans, and yields more reliable positive pairs for contrastive learning. our contributions are summarized as follows: ## experiments in this section, we verify the effectiveness of con-tproto by conducting experiments on two public ner datasets with six cross-lingual transfer pairs and performing comparisons with various baseline models. ## conclusions in this work, we propose contproto as a novel selftraining framework for cross-lingual ner, which synergistically incorporates representation learning and pseudo label refinement. specifically, our contrastive self-training first generates representations where different classes are separated, while explicitly enforcing the alignment between source and target languages. leveraging the class-specific representation clusters induced by contrastive learning, our prototype-based pseudo-labeling scheme further denoises pseudo labels using prototypes to infer true labels of target language spans. as a result, the model trained with more reliable pseudo labels is more accurate on the target languages. in our method, the contrastive and prototype learning components are mutually beneficial, where the for-mer induces clusters which makes it easier to identify the closest prototype, and the latter helps to construct more accurate sample pairs for contrastive learning. evaluated on multiple cross-lingual transfer pairs, our method brings in substantial improvements over various baseline methods.