Kardeş- NLU : Transfer to Low-Resource Languages with the Help of a High-Resource Cousin – A Benchmark and Evaluation for T urkic Languages (2024)

Abstract: Cross-lingual transfer (XLT) driven by massively multilingual language models (mmLMs) has been shown largely ineffective for low-resource (LR) target languages with little (or no) representation in mmLM’s pretraining, especially if they are linguistically distant from the high-resource (HR) source language. Much of the recent focus in XLT research has been dedicated to LR language families , i.e., families without any HR languages (e.g., families of African languages or indigenous languages of the Americas). In this work, in contrast, we investigate a configuration that is arguably of practical relevance for more of the world’s languages: XLT to LR languages that do have a close HR relative. To explore the extent to which a HR language can facilitate transfer to its LR relatives, we (1) introduce Kardeş-NLU, an evaluation benchmark with language understanding datasets in five LR Turkic languages: Azerbaijani, Kazakh, Kyrgyz, Uzbek, and Uyghur; and (2) investigate (a) intermediate training and (b) fine-tuning strategies that leverage Turkish in XLT to these target languages. Our experimental results show that both - integrating Turkish in intermediate training and in downstream fine-tuning - yield substantial improvements in XLT to LR Turkic languages. Finally, we benchmark cutting-edge instruction-tuned large language models on Kardeş-NLU, showing that their performance is highly task- and language-dependent.

## introduction transformer-based massively multilingual language models (mmlms), such as mbert @xcite , xlm-r @xcite , and mt5 @xcite , have substantially advanced multilingual nlp. these models have enabled rapid development of language technologies * these authors contributed equally. 1 https://github.com/lksenel/kardes-nlu for a wide range of low-resource (lr) languages by means of cross-lingual transfer (xlt) from highresource (hr) languages, using zero-shot @xcite or few-shot transfer techniques @xcite . mmlms are, however, biased towards hr languages and xlt with mmlms yields especially poor transfer performance for lr target languages that are (i) underrepresented in mmlms' pretraining corpora and (ii) linguistically distant from the source language @xcite . besides these reasons, such poor xlt is also a consequence of the curse of multilinguality @xcite , i.e., a reduced representational quality of supported languages, stemming from mmlms' parameters being shared by many linguistically diverse languages. in recent years, a large body of work focused on improving xlt abilities of mmlms, ranging from models that aim to better align representation subspaces of source and target language with crosslingual supervision @xcite @xcite @xcite to those that improve the mmlms' representational capacity for individual, mostly lr languages @xcite @xcite . at the same time, an incredible amount of effort has also been dedicated to the creation of new multilingual evaluation benchmarks that either encompass sets of linguistically diverse languages @xcite @xcite or focus on lr languages @xcite @xcite @xcite . the vast majority of existing work, however, assumes (i) zero-shot downstream transfer from (ii) english as the source. that is primarily because, on the one hand, for most tasks, training data is only available in english. on the other hand, many of the recent benchmarks cover lr language families, i.e., families without any hr languages (e.g., some african language families or indigenous languages of the americas): this prevents the creation of high-quality silver-standard training data in a (closely) related hr language (e.g., via machine translation (mt)), as no such language exists. contributions. 1) in this work, we contribute to the body of evaluation resources for lr xlt with kardeş-nlu, 2 an evaluation benchmark covering three natural language understanding (nlu) tasks-natural language inference (nli), semantic text similarity (sts), and commonsense reasoning, in particular choice of plausible alternatives (copa)-for five turkic languages-azerbaijani (az), kazakh (kk), kyrgyz (ky), uyghur (ug), and @xcite . we focus on turkic languages because, unlike most concurrent work, we aim to explore a highly underinvestigated xlt research question: to what extent can lr languages that do have a linguistically and genealogically (close) hr relatives profit from those relatives @xcite . 2) we extend a number of established (i) intermediate training and (ii) finetuning approaches (covering both zero-shot and few-shot xlt) for improving lr xlt by incorporating turkish as the hr sibling of the kardeş-nlu languages; and show that the mixture of incorporating turkish in intermediate training and in task-specific fine-tuning results in substantial performance gains. 3) given the praised generalization abilities of large instruction-based language models (llms) @xcite @xcite , we additionally evaluate (zeroshot) two multilingual llms on kardeş-nlu-the open mt0 @xcite and commercial chatgpt-showing that their performance is highly task-and language-dependent and in some cases substantially trails that of xlt with traditionally fine-tuned "small" mmlms. ## kardeş transfer: leveraging turkish we next attempt to improve xlt to lr kardeş-nlu languages by explicitly incorporating turkish as the close hr relative into the process. we try to (1) increase mmlms' capacity for the target languages as well as their alignment with turkish via intermediate lm training and (2) leverage turkish as an additional source language in downstream zero-shot and few-shot transfer. ## results and discussion zero-shot transfer. including turkish as a fine-tuning source language (tr and en,tr) brings consistent gains over transfer from english only, regardless of the intermediate training strategy. the best results are almost always obtained when we fine-tune on both english and turkish (en,tr): we hypothesize that such fine-tuning establishes task-specific representational associations between the two languages and allows the transfer to benefit from both (i) xlm-r's unmatched representational quality for english and (ii) proximity of turkish to the target languages. the effect is then further amplified when intermediate training (balm and bjlm) increases the xlm-r's capacity for turkish and the target language and strengthens the alignments between them. this is confirmed by the fact that intermediate training on the target language alone (tllm) brings downstream gains (compared to base) for nli but not for the other two tasks. looking at individual languages, we observe the least (and smallest) gains for azerbaijani and kazakh, the two most-resourced kardeş-nlu languages, and the most (and largest) gains for the three less-resourced languages: uyghur, uzbek, and kyrgyz (e.g., compared to base transfer from en on nli, bjlm with transfer from en,tr leads to gains of 5.0% for kyrgyz, 5.1% for uzbek, and 17.2% for uyghur). we see the largest gains (by a wide margin) for uyghur, despite the script mismatch between the intermediate training (arabic script) and evaluation (uyghur in cyrillic script). the intermediate bilingual training for uyghur, which improves representations of arabic-script tokens, would thus likely yield even larger gains if the uyghur test instances were in the arabic script. few-shot transfer. kardeş-nlu: a difficult few-shot xlt benchmark. not only does the comparison of zero-shot and few-shot results in instruction-based llms on kardeş-nlu. given the recent popularity of instruction-tuned llms as competent "generalizers" @xcite , we additionally evaluate (zeroshot) two state-of-the-art multilingual llms on kardeş-nlu: foot_7 mt0 @xcite , as the open model tuned on instructions derived from nlp tasks, and chatgpt, as the commercial model tuned from human instructions and feedback. to this end, we slightly modify the instructions and prompts proposed by @xcite : we provide further details in the appendix §a. figure 1 compares the best zero-shot xlt performance (based on xlm-r) for each language from ## conclusion in this work, we contribute to the body of evaluation resources for low-resource (lr) cross-lingual transfer (xlt) by introducing kardeş-nlu, an evaluation benchmark covering three nlu tasks (nli, sts, and copa)-for five turkic languages: azerbaijani, kazakh, kyrgyz, uyghur, and uzbek. kardeş-nlu allows investigation of an understudied xlt approach: leveraging a high-resource (hr) language to improve transfer to linguistically and genealogically related lr languages. we extend existing intermediate training and fine-tuning approaches for improving lr xlt to integrate turkish as the hr "sibling" of the kardeş-nlu languages. through comprehensive experimentation and analysis, we demonstrated that adding turkish in task-specific fine-tuning can provide significant xlt gains for kardeş-nlu languages that are further amplified by incorporating turkish in bilingual intermediate training strategies. what is more, we also find that kardeş-nlu is a difficult benchmark for few-shot xlt, observing that established fewshot transfer methods are not effective. finally, we evaluated two cutting-edge instruction-tuned large language models-mt0 and chatgpt-on kardeş-nlu, showing that their (zero-shot) performance is inferior on lower-resourced kardeş-nlu languages (uyghur, uzbek, kyrgyz) and greatly varies across tasks. this proves that there is still a long way to (truly) multilingual nlp. in our subsequent efforts, we will not only seek to extend kardeş-nlu with additional lr turkic languages, but also explore how to leverage hr siblings in lr xlt for other language families.