Turning E nglish-centric LLM s Into Polyglots: How Much Multilinguality Is Needed? (2024)

Abstract: The vast majority of today’s large language models (LLMs) are English-centric, having been pretrained predominantly on English text. Yet, in order to meet user expectations, models need to be able to respond appropriately in multiple languages once deployed in downstream applications. This requires strong cross-lingual transfer abilities.In this work, we investigate the minimal amount of multilinguality required during finetuning to elicit cross-lingual generalisation in English-centric LLMs. In experiments across four LLMs, we find that multilingual instruction tuning with as few as two to three languages is both necessary and sufficient to elicit effective cross-lingual generalisation, with the limiting factor being the degree to which a target language is seen during pretraining. Evaluations on five different tasks further reveal that multilingual instruction tuning is most beneficial for generative tasks that assume input/output language agreement, such as in chat settings, while being of less importance for highly structured classification-style tasks.

## introduction conversational instruction tuning is a popular method for aligning large pretrained language models (llms) with user expectations such that they can effectively respond to a user's input query and follow natural language instructions @xcite @xcite ). an implicit expectation of conversational chatbots is that the language of a model's response should match that of the user's input query. for instance, unless otherwise specified, a germanlanguage input should result in a german-language output. however, since the vast majority of training ˚work partially conducted during a research internship at textshuttle. 1 our code and data is available at https://github.com/ zurichnlp/multilingual-instruction-tuning . data is in english, many instruction-tuned llms struggle to respond consistently in other languages @xcite @xcite @xcite . despite limited exposure, however, englishcentric llms such as llama 2 seemingly achieve near-perfect input/output (io) language agreement when tuned with relatively few multilingual conversational instructions. figure 1 depicts io language agreement, as measured by openlid @xcite , and compares multilingual tuning on the language-diverse guanaco dataset @xcite with monolingual tuning on an english-only subset of instructions. as can be seen, multilingual tuning elicits strong io language agreement -within the bounds of open-lid's error rates -for non-english languages seen to varying degrees during pretraining and finetuning without degrading performance on english. this observation raises two major questions, which we aim to address in this paper: q1: how much multilinguality is required during finetuning to elicit 0-shot cross-lingual generalisation in english-centric llms? q2: which languages and tasks benefit most from multilingual instruction tuning of english-centric llms? to investigate these questions, we instruction tune four distinct english-centric llms with varying degrees of multilinguality and evaluate performance on a diverse set of up to 19 languages across five different tasks. specifically, we consider high-, medium-, and low-resource target languages with regard to a model's pretraining data distribution and generative tasks, such as single-turn dialogue, sentence simplification, and extractive question answering, as well as more structured tasks aimed at assessing commonsense reasoning and language understanding. our results indicate that multilingual instruction tuning is crucial for eliciting cross-lingual transfer on generative tasks that assume io language agreement while being less important in structured tasks that are commonly used to benchmark llm performance. furthermore, we empirically show that only a small number of finetuning languages is required to promote cross-lingual transfer. this highlights that tuning data for all potential target languages is not necessary to derive capable polyglot chat models from english-centric llms. ## experimental setup to explore the multilingual capabilities of englishcentric llms, we instruction tune a series of mod-els on a fixed-size set of examples, varying the number of languages available. following this, we evaluate the resulting models in multiple target languages on five distinct tasks that are representative of how llms may be used in downstream applications. ## further analysis our experimental results on five distinct tasks demonstrate that multilingual instruction tuning benefits open-ended single-turn dialogue and sentence simplification in non-english languages most strongly. in this section, we focus on single-turn dialogue to investigate the impact of instruction diversity vs. language diversity and model scaling. ## discussion our findings show that multilingual instruction tuning aids cross-lingual transfer in english-centric llms, though its effectiveness on downstream performance varies across tasks and the degree to which a specific target language is seen during pretraining. for high-and medium-resource languages, we observe substantial performance gains on single-turn dialogue and sentence simplification with monolingual prompting, as well as marginal gains on extractive qa with monolingual prompting. meanwhile, for low-resource languages and highly structured tasks that impose a strict constraint on the output space regardless of the input language (e.g., multiple choice qa), multilingual instruction tuning has little impact. this distinction highlights that multilingual instruction tuning is most beneficial for more open-ended generative tasks that assume io language agreement, and crucially, that this benefit is limited to highand medium-resource languages with respect to model's pretraining language distribution. most surprisingly, we find that multilingual instruction tuning with just two to three languages is necessary and sufficient to promote this generalisation in all four of the english-centric llms considered. furthermore, we consistently see that performance tends to plateau rather quickly and adding more instruction tuning languages -including the target language itself -typically provides no significant gains. this underscores that the performance gains are largely the result of cross-lingual transfer and suggests that there may be diminishing returns associated with scaling up multilingual instruction tuning beyond just a handful of languages. we posit that our findings align with the superficial alignment hypothesis put forward by @xcite , which states that a model acquires its knowledge and abilities during pretraining, while instruction tuning simply guides the model towards a desirable 'subdistribution of formats' to use when prompted. we refine this hypothesis by adding that just a small amount of multilingual instruction data also encourages the model to learn a simple mapping between input and output language. this guides the model towards language-specific subdistributions, leading to better performance on tasks where io language agreement is required. ## conclusion we investigated the minimal amount of multilingual instruction tuning required to elicit multilingual capabilities of four distinct english-centric llms. our results show that finetuning with as few as two to three languages is necessary and sufficient to promote effective cross-lingual transfer, allowing models to better exploit the relatively small amounts of non-english data seen during pretraining. experiments on five distinct tasks revealed that this can lead to significant performance improvements for high-and medium-resource languages on open-ended generative tasks that assume input/output language agreement. while the effectiveness of cross-lingual transfer is indeed good news for llm developers, future work could explore methods to reduce the performance gap between english and non-english languages, particularly for smaller models, as well as investigating tasks for which language-specific instruction tuning may be of greater importance, such as tasks involving cultural awareness.