Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation (2021)

Abstract: Recent multilingual pre-trained language models have achieved remarkable zero-shot performance, where the model is only finetuned on one source language and directly evaluated on target languages. In this work, we propose a self-learning framework that further utilizes unlabeled data of target languages, combined with uncertainty estimation in the process to select high-quality silver labels. Three different uncertainties are adapted and analyzed specifically for the cross lingual transfer: Language Heteroscedastic/Homoscedastic Uncertainty (LEU/LOU), Evidential Uncertainty (EVI). We evaluate our framework with uncertainties on two cross-lingual tasks including Named Entity Recognition (NER) and Natural Language Inference (NLI) covering 40 languages in total, which outperforms the baselines significantly by 10 F1 for NER on average and 2.5 accuracy for NLI.

## introduction recent multilingual pre-trained language models such as mbert @xcite , xlm-r @xcite and mt5 @xcite have demonstrated remarkable performance on various direct zero-shot cross-lingual transfer tasks, where the model is finetuned on the source language, and directly evaluated on multiple target languages that are unseen in the task-finetuning stage. while direct zero-shot transfer is a sensible testbed to assess the multilinguality of language models, one would apply supervised or semi-supervised learning on target languages to obtain more robust and accurate predictions in a practical scenario. in this work, we investigate self-learning (also known as "pseudo labels") as one way to apply semi-supervised learning on cross-lingual transfer, where only unlabeled data of target languages are required, without any efforts to annotate gold labels for target languages. as self-learning has been proven effective in certain tasks of computer vision @xcite and natural language processing @xcite @xcite , we propose to formalize an iterative self-learning framework for multilingual tasks using pre-trained models, combined with explicit uncertainty estimation in the process to guide the cross-lingual transfer. our self-learning (sl) framework utilizes any multilingual pre-trained models as the backbone, and iteratively grows the training set by adding predictions of target language data as silver labels. we reckon two important observations from our preliminary study (baselines in §4). first, compared with self-training one target language at a time, jointly training multiple languages together can improve the performance on most languages, especially for certain low-resource languages that can achieve up to 8.6 f1 gain in ner evaluation. therefore, our sl framework features the joint training strategy, maximizing potentials of different languages benefiting each other. second, compared with simply using all unlabeled data as silver labels without considering prediction confidence, estimating uncertainties becomes critical in the transfer process, as higher quality of silver labels should lead to better performance. we hence introduce three different uncertainty estimations in the sl framework. specifically, we adapt uncertainty estimation techniques based on variational inference and evidence learning for our cross-lingual transfer, namely leu, lou and evi ( §3.2). we evaluate our framework and three uncertainties on two multilingual tasks from xtreme @xcite : named entity recognition (ner), and natural language inference (nli). empirical results suggest leu to be the best uncertainty estimation overall, while the others can also perform well on certain languages ( §4.1). our analysis shows further evaluation of different estimations, corroborating the correlation between the uncertainty quality and the final sl performance. characteristics of dif-ferent estimations are also discussed, including the language similarities learned by lou and the current limitation of evi in the sl process ( §5). our contributions in this work can be summarized as follows. (1) we propose the self-learning framework for the cross-lingual transfer and identify the importance of uncertainty estimation under this setting. (2) we adapt three different uncertainty estimations in our framework, and evaluate the framework on both ner and nli tasks covering 40 languages in total, improving the performance of both high-resource and low-resource languages on both tasks by a solid margin (10 f1 for ner and 2.5 accuracy score for nli on average). (3) further analysis is conducted to compare different uncertainties and their characteristics. ## related work we introduce the work of uncertain estimation briefly. as deep learning models are optimized by minimizing the loss without special care on the uncertainty, they are usually poor at quantifying uncertainty and tend to make over-confident predictions, despite producing high accuracies @xcite . estimating the uncertainty of deep learning models has been recently studied in nlp tasks @xcite @xcite . there are two main uncertainty types in bayesian modelling @xcite : epistemic uncertainty that captures the model uncertainty itself, which can be explained away with more data; aleatoric uncertainty that captures the intrinsic data uncertainty regardless of models. aleatoric uncertainty can further be devided into two sub-types: heteroscedastic uncertainty that depends on input data, and homoscedastic uncertainty that remains constant for all data within a task. in this work, we only focus on aleatoric uncertainty, as it is more closely related to our sl process to select confident and high-quality predictions within each iteration. ## approach we keep the same model architecture throughout our experiments: a multilingual pre-trained language model is employed to encode each input sequence, followed by a linear layer to classify on the hidden state of cls token for nli, and of each token for ner, which is the same model setting from xtreme @xcite . cross-entropy (ce) loss is used during training in the baseline. ## conclusion in this work, we propose a self-learning framework combined with explicit uncertainty estimation for cross-lingual transfer. three different uncertainties are adapted, and the entire framework is evaluated on two tasks of ner and nli, surpassing the baseline by a large margin. further analysis shows the evaluation and characteristics of each uncertainty.