Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks (2024)

Abstract: Zero-shot cross-lingual transfer, which implies finetuning of the multilingual pretrained language model on input-output pairs in one language and using it to make task predictions for inputs in other languages, was widely studied for natural language understanding but is understudied for generation. Previous works notice a frequent problem of generation in a wrong language and propose approaches to address it, usually using mT5 as a backbone model. In this work we compare various approaches proposed from the literature in unified settings, also including alternative backbone models, namely mBART and NLLB-200. We first underline the importance of tuning learning rate used for finetuning, which helps to substantially alleviate the problem of generation in the wrong language. Then, we show that with careful learning rate tuning, the simple full finetuning of the model acts as a very strong baseline and alternative approaches bring only marginal improvements. Finally, we find that mBART performs similarly to mT5 of the same size, and NLLB-200 can be competitive in some cases. Our final zero-shot models reach the performance of the approach based on data translation which is usually considered as an upper baseline for zero-shot cross-lingual transfer in generation.

## introduction multilingual pretrained language models (mplms) such as mbert @xcite , mbart @xcite , and mt5 @xcite provide high-quality representations for texts in various languages and serve as a a universal backbone for finetuning on language-specific task data. the latter, however, is not always available for a language of interest, providing motivation for studying zeroshot cross-lingual capabilities of mplms. in this setting, the model is adapted, e.g. finetuned, on input-output pairs in a source language, usually english, and then applied in a zero-shot manner to make predictions for inputs in another target language, seen only at the pretraining stage. while the described setting was broadly studied for natural language understanding tasks @xcite @xcite @xcite , work on zero-shot cross-lingual transfer in generation is more limited @xcite @xcite . previous work highlight two main problems arising in this scenario: producing incoherent or irrelevant answers, and generating text in a wrong language. a series of potential solutions were proposed, such as freezing the parts of the weights during finetuning, utilizing parameterefficient finetuning methods, mixing-in the unsupervised target language data together with the supervised source language data, or using more than one source language. a common strategy is also to perform an intermediate tuning of the model on the language generation task in a self-supervised manner (as opposed to denoising tasks used for pretraining). however, despite listed efforts, the state of zeroshot cross-lingual generation still remains unclear and poses open questions: methods proposed for mitigating generation in the wrong language, were all tested on different tasks and benchmarks, and not compared to methods from other works, making it hard to establish the best performing one. the contribution of this work is conducting a deep empirical study addressing the listed questions. we consider most commonly used multilingual encoder-decoder mplms, namely mt5 and mbart, as well as the translation model nllb-200. we systematically study six adaptation methods, investigate the effect of intermediate tuning, pay attention to adaptation hyperparameters, and compare models and adaptation methods in a unified setting. we consider two tasks: summarization and questions answering (qa). our main findings are as follows: ## related work all works on zero-shot cross-lingual transfer in generation underline (and try to address) the severe problem of generating in a wrong language at the test time. this problem is also referred to under terms catastrophic forgetting (of languages not participating in finetuning, @xcite , source language hallucination @xcite , or accidential translation problem (li and murray, 2023). @xcite propose to overcome generation in a wrong language by using parameter-efficient finetuning instantiated by prompt-tuning @xcite . they also mix-in the unsupervised target language task together with the supervised source language task, and factorize learnable prompts into language and task components. @xcite propose mmt5 (modular mt5), allocating a small amount of languagespecific parameters in the model during pretraining and freezing them during task-specific finetuning. to alleviate generation in a wrong language, they freeze some additional mmt5 parameters during finetuning, e. g. the embedding layer and feed forward layers in transformer decoder. li and murray (2023) argue that learning language-invariant representations during finetuning is harmful for crosslingual generation and propose finetuning on the data from more than one source language to avoid generation in a wrong language, with mt5 as a backbone model. @xcite is the only work which considers the other backbone model than mt5: they perform an intermediate tuning of mbart on an auxiliary unsupervised task on hindi, japanese and english. to avoid generation in a wrong language, they freeze embeddings and the transformer decoder, and mix-in the data from auxiliary pretraining during finetuning. in our work we are interested to compare all previously proposed approaches in a unified setting, to better assess the impact of different factors on the zero-shot cross-lingual transfer in generation. alternative approaches to zero-shot crosslingual transfer include data translation approaches, often referred as translate-train and translate-test paradigms. the former one implies translating the train task data to target languages and finetuning the model on this translated data, and the latter one implies translating test input examples into the source language, generating outputs in the source language and translating them back into the target language. the drawbacks of these approaches in-clude a high computational cost either at training or testing time, lack of high-quality translation models for low-resource languages, and potential inconsistencies between sentences in translation @xcite . despite its computational cost, data translation is a strong approach which is usually considered as an upper baseline for zero-shot approaches. another related field is few-shot crosslingual transfer in generation which assumes access to a small amount of labeled examples in the target language @xcite @xcite . this setting is out of scope of this study, but could be considered in the future work. ## methodology and experimental setup adaptation methods. we investigate the following adaptation methods: in the rest of the text term "full finetuning" refers to the finetuning of all weights on the english task data only, even though two last described methods also finetune all weights. we do not consider mmt5 as it was not publicly released and requires substantial resources for pretraining. we also experiment with intermediate tuning (it) of the model, used in several works and performed before finetuning on the task data. standard encoder-decoder mplms rely on a self-supervised denoising training, where often the input corresponds to corrupted text (eg. with masked tokens or permuted sentences), and the output can follow some very specific structure (eg. a masked span rather than a full sentence, output containing special tokens, etc.). therefore, in their raw form, these mplms are not necessarily well suited to receive well-formed text as an input and generate clean text as an output. it performs finetuning on a language modeling-like task, e.g. predicting the continuation of a paragraph based on its beginning, to compensate for this gap. it was shown to be necessary in @xcite for prompt tuning of mt5 and in @xcite for full or partial finetuning of mbart. we systematically test the necessity of it for all methods and models. ## experiments first, we investigate the effect of the learning rate, intermediate tuning and the adaptation method for two most commonly used models, mt5 and mbart. second, we compare them with other models and consider larger models. finally, we present some qualitative examples and observations from manual inspection of predictions. in general, model predictions reaching highest metric values in our plots, form quite meaningful and reasonable responses to the considered tasks; more details in section 5. effect of learning rate. we begin our study with analysing the effect of lr on the full finetuning on the english task data. with too small or too large lr the model does not learn even the english task because of too short steps or divergence. for the range of lrs when the english task is learned well, we observe that larger lrs lead to the effect reported in other works, when the model overfits to the source english language and generates answers in english when applied for inputs in other languages. however, with the reduced lr, this effect almost completely eliminates and the model mostly generates in the target language. this effect is demonstrated in figure 1 on a subset of languages and in fig. 9 -12 in appendix on all considered languages. figure 1 also shows a comparison of enhancements of full finetuning proposed in the literature, such as mixing-in target languages or freezing the decoder and embeddings. even though these enhancements improve performance and percentage of outputs in the correct language with fixed lr, we find that reducing lr in full finetuning often brings larger improvements. reducing lr for other methods makes them even stronger. we note that performance in english is usually a little higher with larger lr. this may raise a hypothesis that for non-english languages, outputs generated with larger lr in english may be of higher semantic quality than the ones generated in the correct target language with smaller lr. in appendix e we test this hypothesis and demonstrate that this is not the case. ## inspection of predictions we inspected a subset of predictions in french and russian and found that models achieving highest scores in both tasks generate fluent, meaningful and reasonable predictions in a lot of cases, but sometimes have issues with factualness, grammaticality or hallucinations. examples are shown in figure 4 . analyzing effects of lr, we observe that increasing lr leads first to increase in code switching and then to wrong language generation, while reducing lr leads to producing rudiments of pretraining in generation. for example, models sometimes generate extra tokens used in pretraining, such @xmath0, see rows 2 and 4 in figure 4 . in most cases this does not affect meaningfulness of predictions, but in rare cases leads to mt5 producing incomplete sentences, which may look unreasonable in sum-marization, e.g. "<@xmath1." (translated from french). the reason is that in mt5 pretraining @xmath2. the described effect is eliminated by intermediate tuning (row 3 in fig. 4 ). in the same fashion, mbart average lengths are closer to groundtruth average lengths than mt5 in summarization, and the reverse effect takes place in qa. the reason is that in mt5 pretraining, the targets are only fragments masked in the input, which are shorter than targets in mbart pretraining represented by full sequences (they need to be reconstructed from the masked inputs). notably, data translation can produce translationrelated errors, e.g. in rows 5 and 9 models generate a wrong male article "un", probably because this was a dominating article in the translated data. ## conclusion in this work, we conducted a deep systematic study of how to achieve high-performing zero-shot crosslingual transfer in generation. our study highlights the high importance of careful learning rate tuning and the usefilness of the intermediate tuning. we show that with these two ingredients, mt5 and mbart achieve strong results with simple full finetuning, i.e. closely approach the performance of translate-train in summarization and reach it in question answering. the performance gap in summarization is closed by using several source languages for mt5 and freezing decoder and embeddings for mbart. translation-pretrained nllb-200 shows surprisingly good performance in summarization but lags behind in question answering. we urge future works to pay more attention to hyperparameter tuning and to report more rigorously their experimental setup, as well as consider a wider spectrum of models and baselines in the experiments.