Super donors and super recipients: Studying cross-lingual transfer between high-resource and low-resource languages (2024)

Abstract: Despite the increasing popularity of multilingualism within the NLP community, numerous languages continue to be underrepresented due to the lack of available resources.Our work addresses this gap by introducing experiments on cross-lingual transfer between 158 high-resource (HR) and 31 low-resource (LR) languages.We mainly focus on extremely LR languages, some of which are first presented in research works.Across 158*31 HR–LR language pairs, we investigate how continued pretraining on different HR languages affects the mT5 model’s performance in representing LR languages in the LM setup.Our findings surprisingly reveal that the optimal language pairs with improved performance do not necessarily align with direct linguistic motivations, with subtoken overlap playing a more crucial role. Our investigation indicates that specific languages tend to be almost universally beneficial for pretraining ( super donors ), while others benefit from pretraining with almost any language ( super recipients ). This pattern recurs in various setups and is unrelated to the linguistic similarity of HR-LR pairs.Furthermore, we perform evaluation on two downstream tasks, part-of-speech (POS) tagging and machine translation (MT), showing how HR pretraining affects LR language performance.

## introduction according to the endangered languages project @xcite , more than 3000 languages are at risk of extinction. in recent years, the nlp community has undoubtedly broadened its efforts and presented very ambitious projects @xcite to incorporate more and more languages into practical use. however, even well-known multilingual transformer models (e.g., mbert @xcite , xlm-r (conneau et al., 2020), and mt5 @xcite ) and crosslingual benchmarks (xglue @xcite , xtreme @xcite ) cover only about 100 of the most presented languages. in this work, we aim to tackle this gap in underrepresentation or even the absence of experiments for lr languages, considering constraints in both labeled and unlabeled text data, as well as linguistic knowledge and experimental base. we present the study of cross-lingual transfer in the case of extremely lr languages, examining how continued pretraining on hr languages impacts model performance on lr languages in the masked language modeling (mlm) setup with additional measurements of its effects on downstream performance (see figure 1 for more details). we aim to explore whether it is possible to conduct model pretraining on hr languages and observe improvements compared to zero-shot performance when evaluating on unseen lr languages. additionally, we measure model performance on downstream tasks: pos tagging and mt. in more detail, we first collect a dataset with raw text data (see @xcite for the list of lan-guages and section 3 for criteria of lr and hr languages). next, we exclude some languages due to data quality issues. thereby, we experiment with 158 hr and 31 lr languages, resulting in 4898 hr-lr language pairs for further investigation of cross-lingual transfer. next we assess the zero-shot performance of the mt5 model on the raw data from lr languages. afterward, we perform continued pretraining of the model with the mlm objective on data from each hr language and evaluate performance of fine-tuned models on all lr languages. finally, we analyze the factors, such as data and linguistic features, that lead to successful cross-lingual transfer between hr and lr languages. we also measure the downstream performance of successful hr-lr language pairs in pos tagging and mt tasks for lr languages with available annotated data. we do not use data from lr languages during training and use it for evaluation only. we use the term donor to denote the languages that serve as sources for knowledge transfer through continued pretraining. on the other hand, we use the term recipient to indicate the languages used to evaluate transfer learning efficiency. the main contributions of this work can be summarized as follows: (i) we collect and present the dataset with 189 languages. (ii) we conduct crosslingual transfer experiments between 158 hr and 31 lr languages. (iii) we interpret cross-lingual transfer results across data and linguistic features. (iv) we investigate how cross-lingual transfer impacts the performance of downstream tasks, focusing mainly on the pos tagging and mt tasks. the code is available 1 . ## related work the cross-lingual transfer involves leveraging existing resources available for hr languages to improve methods for lr languages. this approach can be particularly beneficial for lr languages that lack extensive linguistic resources and data for nlp applications. @xcite state that the mbert's performance for lr languages is not on par with that for hr languages, and there is an unequal representation of languages within models. @xcite show that mbert context embeddings capture similarities between languages, but achieving a proper crosslingual representation requires the availability of 1 code: https://github.com/vitaly-protasov/lr_ transfer parallel corpora, which is lacking for most lr languages. @xcite show that the selection of pretraining languages significantly impacts the performance, indicating that there are more effective donors than english. additionally, @xcite show that russian and german can serve as better donors for reliable transfer. @xcite experiment with different number of languages during pretraining and find it promising in terms of impact on performance on unseen languages. there are also different suggested strategies for choosing the proper donor language. @xcite propose using vocabulary overlap to find a better hr donor. @xcite demonstrate that typological motivation in language selection positively impacts the transfer learning scores, as well as the size of the source language. @xcite show that the type of language script used plays an essential role, and transliteration helps to improve the quality of transfer learning. @xcite also show that fine-tuning on linguistically similar languages (defined using wals @xcite improves the performance on several downstream tasks. @xcite investigate cross-lingual transfer using diverse data, revealing that morphology and language modeling performance are strong predictors of its success. @xcite ; @xcite establish that no individual wals feature stands out as the most crucial across various tasks. transfer learning has long emerged as a pivotal technique in machine translation, particularly for lr languages. @xcite introduce an approach that uses hr language data to enhance neural machine translation (nmt) for lr language pairs, achieving notable improvements in their translation quality. further exploration by @xcite reveal that word embeddings are a critical component of transfer learning, and their proper alignment is essential for optimal results. these findings highlight the critical role of transfer learning in addressing challenges associated with the limited availability of linguistic data in nmt. the studies mentioned above focus on wellresourced languages with labeled data, which has resulted in neglecting lr languages that already lack available data. this study aims to address this gap by investigating cross-lingual transfer for several understudied lr languages. we assemble from various existing sources a text corpus. appendix a.3 lists all used languages. ## cross-lingual transfer methodology the main goal of our experiments is to figure out whether the training on hr language donors improves the modeling of lr recipient languages, try to interpret it and to observe possible performance of transfer learning in downstream tasks. ## conclusion in this work, we present extensive experiments on cross-lingual transfer for hr-lr language pairs, leveraging hr languages for continued pretraining of the mt5. we observe that the performance of cross-lingual transfer significantly correlates with morphological features. additionally, a higher degree of overlap between subtokens can contribute to better performance. meanwhile, other characteristics do not significantly correlate with cross-lingual transfer results. we also observe improved downstream evaluation results, showing successful pos tagging and mt tasks performance. finally, some lr languages tend to be super recipients, namely benefiting from all languages, and some hr languages tend to be super donors namely benefiting all languages with no apparent linguistic relation between donor and recipient.