Uncertainty-Aware Cross-Lingual Transfer with Pseudo Partial Labels (2022)

Abstract: Large-scale multilingual pre-trained language models have achieved remarkable performance in zero-shot cross-lingual tasks. A recent study has demonstrated the effectiveness of self-learning-based approach on cross-lingual transfer, where only unlabeled data of target languages are required, without any efforts to annotate gold labels for target languages. However, it suffers from noisy training due to the incorrectly pseudo-labeled samples. In this work, we propose an uncertainty-aware Cross-Lingual Transfer framework with Pseudo-Partial-Label (CLTP)1 to maximize the utilization of unlabeled data by reducing the noise introduced in the training phase. To estimate pseudo-partial-label for each unlabeled data, we propose a novel estimation method, considering both prediction confidence and the limitation to the number of similar labels. Extensive experiments are conducted on two cross-lingual tasks, including Named Entity Recognition (NER) and Natural Language Inference (NLI) across 40 languages, which shows our method can outperform the baselines on both high-resource and low-resource languages, such as 6.9 on Kazakh (kk) and 5.2 Marathi (mr) for NER.

## introduction the multilingual pre-trained language models such as mbert @xcite , xlm-r @xcite and mt5 @xcite are able to support zero-shot transfer from a source language to target languages. despite the remarkable performance on direct zero-shot cross-lingual transfer tasks, one would apply semi-supervised learning on target languages to obtain more robust and accurate predictions in a practical scenario. recent studies @xcite validate the effectiveness of self-learning in "ni silaha ya plastiki ya moja kwa moja inayopiga risasi." ## model in this section, we propose a partial-label based self-learning framework to boost cross-lingual transfer performance. the overview of the proposed framework is presented in section 3.2. the technical details for the uncertainty-aware pseudopartial-label estimation and partial label learning are described in sections 3.3 and 3.4, respectively. ## conclusion in this work, we propose an uncertainty-aware pseudo-partial-label framework for cross-lingual transfer. with the auxiliary of pseudo-partial-labels, cltp framework improves the model by reducing the noise introduced in training phase while maximizing unlabeled data utilization. moreover, we propose a novel pseudo-partial-label estimation method that considers both prediction confidence and the limitation to the number of similar classes. the proposed framework is evaluated on two tasks of ner and nli and improves the performance of the pre-trained model by a solid margin (11.1 f1 for ner and 2.4% accuracy score for nli on average). compared to other self-learning based methods, our framework surpasses the baselines on both high-resource and low-resource languages, such as 6.9 on kazakh and 5.2 marathi for ner.