Crosslingual Generalization through Multitask Finetuning (2023)

Abstract: Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task genrealization to non-English languages that appear only in the pretraining corpus. Finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of-the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human-written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at https://github.com/bigscience-workshop/xmtf .

## introduction large language models pretrained on vast amounts of text show some capability of solving tasks expressed in natural language, even without explicit training on these tasks @xcite . finetuning on groups of language tasks has been shown to significantly boost this zero-shot task generalization of language models @xcite @xcite . for example, @xcite finetune on tasks like summarization and question answering leading to better performance on unseen tasks like natural language inference. previous work has focused on multitask finetuning in the context of large english language models and tasks. multilingual large language models show the same zero-shot learning capabilities for both monolingual and crosslingual tasks @xcite @xcite . however, zero-shot performance tends to be significantly lower than finetuned performance. thus, task-specific or language-specific transfer learning via finetuning remains the predominant practice @xcite @xcite . this is particularly challenging for low-resource languages or tasks with limited data available, such as writing a fable that teaches a specified moral. in the spirit of multitask finetuning, it would be desirable to improve the zero-shot task generalization of multilingual models to make them usable on tasks from low-resource languages without requiring further finetuning. to address this goal, we focus on crosslingual multitask finetuning. due to the difficulty of collecting supervised task data in low-resource languages, previous work typically aims to transfer capabilities learned from finetuning on english data, which can improve performance on non-english language tasks @xcite @xcite . we investigate whether english-only multitask finetuning also improves performance on non-english held-out tasks using the multilingual bloom @xcite and mt5 @xcite models. we find that after finetuning on the english-only multitask mixture used for t0 @xcite (p3), performance on a diverse set of en es pt fr ar id zh hi code vi ur te ta bn mr sw gu pa ne yo ig ny zu xh sn ts rw lg tn nso rn ml kn or as ln wotum ki st fon ca eu ak bm tw 25 5 1 0.1 0.01 0.001 0.0001 % of corpus xp3 roots mt5 corpus figure 2: language composition of xp3, roots, and the corpus of mt5. all roots and xp3 languages are depicted. the mt5 corpus covers additional languages that are not included in the graph. to investigate whether multilingual task data can further improve performance, we extend p3 to xp3 by adding datasets from 46 different languages that cover tasks previously not present in p3 (such as translation and program synthesis). finetuning on xp3 leads to even better zero-shot task generalization in both english and non-english compared to the p3-trained baseline. models finetuned on xp3 perform best on english prompts, even for non-english samples. hypothesizing that better performance could be attained by training on non-english prompts, we construct a variant of xp3 with machine-translated prompts called xp3mt. we find that finetuning on machine-translated prompts is enough to significantly increase performance on held-out tasks with non-english human-written prompts. however, reducing the number of en-glish prompts in the finetuning also worsens english prompt performance on multilingual tasks. notably, we also find that models finetuned on xp3 generalize to held-out tasks in languages never intentionally seen during pretraining nor finetuning. we conduct a contamination analysis and find that only small amounts of these languages were included in the pretraining corpus. thus, we hypothesize the models learn some language-and task-agnostic capabilities. we publicly release all our datasets and models (urls in appendix §c). ## finetuning data and models to study crosslingual multitask prompted finetuning, we create xp3 by extending the p3 dataset collection with additional non-english tasks. we finetune both bloom and mt5 models on xp3. we refer to appendix §c for public links to released models and datasets. ## results we first examine generalization to new tasks in languages included in finetuning in §4.1. then, in §4.2, we look at language generalization: can models generalize to tasks in languages that (a) they have only seen during pretraining and (b) they have never seen intentionally? in §4.3, we investigate performance on multilingual prompts and finetuning on xp3mt. scaling laws are analyzed in §4.4. finally, §4.5 looks at performance on generative tasks and §4.6 at the effect of language proportions on performance. ## conclusion in this work we investigated crosslingual multitask finetuning. we developed xp3, a corpus consisting of tasks in 46 languages. further, we have extended xp3 to xp3mt with machine-translated prompts. we have finetuned pretrained bloom and mt5 models on the newly created corpora as well as the english-only p3 corpus to produce bloomz and mt0 models. we found that english-only finetuning suffices for a multilingual pretrained large language model to generalize to tasks in other pretrained languages. however, finetuning on multiple languages using xp3 provided even better performance. we have further observed finetuned models to be capable of generalization to new tasks in languages they have never intentionally seen. we investigated multilingual prompting and found performance after finetuning on english prompts only to be poor. however, finetuning on a corpus with machinetranslated prompts (xp3mt) lead to significantly better performance on human-written non-english prompts. comparing models from 560 million up to 176 billion parameters revealed that the performance gap between only pretraining and finetuning widens as parameters increase. lastly, we found multitask finetuning on billions of short targets biases models to produce short answers, which can hurt performance on generative tasks. we proposed a simple workaround by forcing a minimum generation length at inference. to contribute to future progress on improving zero-shot generalization, we release all datasets and models introduced in this work.