Mitigating Language-Level Performance Disparity in m PLM s via Teacher Language Selection and Cross-lingual Self-Distillation (2024)

Abstract: Large-scale multilingual Pretrained Language Models (mPLMs) yield impressive performance on cross-language tasks, yet significant performance disparities exist across different languages within the same mPLM. Previous studies endeavored to narrow these disparities by supervise fine-tuning the mPLMs with multilingual data.However, obtaining labeled multilingual data is time-consuming, and fine-tuning mPLM with limited labeled multilingual data merely encapsulates the knowledge specific to the labeled data.Therefore, we introduce **ALSACE** to leverage the learned knowledge from the well-performing languages to guide under-performing ones within the same mPLM, eliminating the need for additional labeled multilingual data. Experiments show that ALSACE effectively mitigates language-level performance disparity across various mPLMs while showing the competitive performance on different multilingual NLU tasks, ranging from full resource to limited resource settings. The code for our approach is available at https://github.com/pkunlp-icler/ALSACE.

## introduction recently, multilingual pre-trained language models (mplms) have attracted significant attention @xcite . these mplms, such as mbert @xcite and mt5 @xcite , are pre-trained on extensive amounts of corpus across hundreds of different languages, which enables them to handle multiple languages within a single model and effectively perform cross-lingual tasks @xcite @xcite @xcite . however, all mplms share a key limitation. due to discrepancies in the quality and quantity of pretraining corpus available for different languages, * equal contribution. to utilize the different knowledge across different languages within the same mplm and mitigate the need for the labeled data, we introduce teacher language selection and cross-lingual self-distillation (alsace), which leverages the knowledge from the selected teacher languages to reduce the performance disparity among the languages. specifically, alsace mainly consists of in china, it @xmath0. two stages: teacher language selection and cross-lingual self-distillation. for teacher language selection, the motivation is that high-resource languages may not be ideal for probing knowledge to supervise the other languages. for instance, although persian is a relatively low-resource language, it may provide more precise answers for kenya's cultural queries than english due to the closer linguistic proximity @xcite between persian and swahili. different from simply using the knowledge from high-resource languages (e.g., english) to improve the performance of low-resource languages (e.g., swahili), we introduce teacher language selection to choose reliable teacher languages for a specific task to supervise the student languages. specifically, we employ a majority voting strategy to generate pseudolabels derived from the consensus of the mplms' predictions across different languages in the given multilingual corpus. then, we utilize the average confidence score of the different languages on the generated pseudo labels as the indicator to select the teacher languages automatically. as a result, we can select adaptive teachers for different tasks using the unlabeled sentences in the corpus. we further propose cross-lingual self-distillation to leverage the knowledge from each selected teacher language to supervise other languages, reducing the performance disparity. we further propose crosslingual self-distillation to leverage the knowledge from each selected teachers languages to supervise other languages, reducing the performance disparity. it employs a consistency loss that encourages closer alignment between the model output distributions of each reliable teacher language and other languages. in this way, mplms can effectively mitigate the language-level performance disparity without relying on the supervised multilingual data. experiments show alsace consistently mitigates language-level performance disparity in various mplms and show the competitive performance on different multilingual benchmarks, including xnli @xcite , paws-x @xcite and xcopa @xcite . we also conduct knowledge probing experiments on the ge-omlama @xcite as shown in figure 1 , demonstrating that alsace effectively mitigates language-level performance disparity by addressing knowledge disparity. moreover, our experiments show that alsace improves performance not only in low-resource languages but also in highresource languages. this finding illustrates that alsace enables effective knowledge transfer between different languages instead of only transferring knowledge from high-resource to low-resource languages. further analysis shows that alsace can transfer both general knowledge across different languages and language-specific knowledge, i.e., some specific knowledge locally shared by people speaking the specific language, which is only present in the corpus of some specific languages. ## related work knowledge disparity leads to language-level performance disparity in mplms. the mplms have shown strong capabilities in many nlp tasks including natural language generation (nlg) @xcite @xcite @xcite and natural language understanding (nlu) @xcite @xcite @xcite . however, there is a noticeable performance disparity across different languages in the same mplm. several works are proposed to investigate the reason of language-level performance disparity in mplms. @xcite ; @xcite demonstrate that mplms could learn different knowledge from different languages data in the pre-training corpus, but imbalanced corpus might lead to knowledge disparity for different languages. @xcite suggests that the performance disparities across different languages could be attributed to the imbalanced knowledge distribution of these languages acquired during the pre-training phase. @xcite further observe that different languages within a single mplm can retain distinct knowledge that is locally shared by the people speaking the specific language. therefore, we attempt to address language-level performance disparity from the knowledge disparity perspective.