Zero-shot cross-lingual transfer in instruction tuning of large language models (2024)

Abstract: Instruction tuning (IT) is widely used to teach pretrained large language models (LLMs) to follow arbitrary instructions, but is under-studied in multilingual settings. In this work, we conduct a systematic study of zero-shot cross-lingual transfer in IT, when an LLM is instruction-tuned on English-only data and then tested on user prompts in other languages. We advocate for the importance of evaluating various aspects of model responses in multilingual instruction following and investigate the influence of different model configuration choices. We find that cross-lingual transfer does happen successfully in IT even if all stages of model training are English-centric, but only if multiliguality is taken into account in hyperparameter tuning and with large enough IT data. English-trained LLMs are capable of generating correct-language, comprehensive and helpful responses in other languages, but suffer from low factuality and may occasionally have fluency errors.

## introduction instruction tuning (it) helps to align large language models (llms) with users expectations so that llms are capable of understanding user queries and generating helpful, comprehensive and focused responses without few-shot examples. contrary to standard nlp datasets that are focused on particular tasks, it datasets consist of diverse instructions representing various tasks and possible user requests, enabling generalization to new instructions which were unseen during training @xcite . most of the it research has focused on english, leaving multilingual instruction following a rather understudied area. several recent works aim to extend instruction tuning beyond english by creating target language it datasets via automatic translation of english instructions @xcite , distillation of outputs of powerful models such as in this work, we take a close look at zero-shot cross-lingual transfer in instruction tuning, when the llm is tuned solely on english instruction data and then prompted to follow instructions in target languages without any additional target-language adaptation. such an approach has the clear advantages of low cost and easy applicability to various target languages but is often considered just as a simple baseline, without detailed analysis. we aim to deeper understand (rq1) what are the capabilities and limits of the zero-shot approach as well as (rq2) which factors influence the successful cross-lingual knowledge transfer. the most common strategy for evaluating instruction following capabilities consists of scoring the helpfulness of model responses on some publicly available set of diverse instructions, e.g. alpacafarm @xcite , with a powerful model, e.g. gpt-3.5. we argue that such high-level evaluation is insufficient and not infor-mative enough for a multi-facet task of open-ended generation, especially in the multilingual scenario. we advocate for using a more careful evaluation pipeline, including the evaluation of various aspects of model responses (fluency, content, relevance to the task etc.), controlling the distribution and the complexity of the tasks in the evaluation set, and using both automatic metrics and human inspection of predictions. this allows us to characterize the weak and strong sides of multilingual responses generated by the model tuned on englishonly data (rq1) and to better understand the influence of factors such as the base model (multilingual / english-centric, model size), it data size, adaptation strategy and hyperparameters (rq2). our key findings include: • cross-lingual transfer does happen successfully in instruction tuning (it) even if all stages of model training are english-centric, but only if multilinguality is taken into account in it hyperparameter tuning and with large enough it data; ## related work most of the works in multilingual it aim to extend the it dataset with non-english data @xcite @xcite , or decompose non-english instructions by pivoting through english translations @xcite . chen et al. (2024); kew et al. (2023); shaham et al. (2024) advocate for the sufficiency of a "pinch" of multilinguality in it, represented by a small amount of updates on multilingual it data, small amount of multilingual it data mixed with english data, or having only 2-3 languages in the it data. we focus on english-only it, trying to better assess capabilities and limits of such settings. the concurrent work of shaham et al. ( 2024 ) does demonstrate the proof-of-the-concept results on zero-shot cross-lingual transfer in it, but attributes it to the multilinguality of palm-2 pretraining data. we show that cross-lingual transfer in it works well even for english-centric models and conduct a more deep and systematic investigation of this effect. we cover more related works in appendix a. ## our evaluation methodology to better understand the strong and weak sides of multilingual responses generated by the model tuned on english-only data, we devise a multi-facet evaluation strategy which includes evaluation of various aspects of generated responses, controlling task distribution and complexity, and using both model-based and human evaluation. evaluation criteria. we conduct main evaluation using both manual predictions inspection (on a subset of the evaluation set) and gpt-3.5 evaluation (on the full evaluation set). to control qualitative aspects of generated texts, we judge them with 6 criteria: helpfulness (how helpful in general is the response for the user), language correctness (does the language of the response match the language of the task), fluency, factual accuracy, logical coherence and harmlessness. five of these criteria (except language correctness) were introduced in @xcite and in our preliminary study we found that they reflect well the weaknesses of model responses. we also use the same scale from 0 to 2 for each criteria. we also introduce lightweight surface metrics: language correctness (how often the language of the response matches the language of the task), spellcheck correctness (which portion of words in the responses pass spell checking), and relevance to the task (how often responses are relevant to their tasks, evaluated using llama-2-chat-7b). these metrics serve to identify if a model passes a minimal bar on the quality of multilingual answers and help to select hyperparameters and filter out noneffective model configurations without gpt-based evaluation. control of the task distribution. we identify a diverse set of 25 "tasks" present in alpacafarm (e.g. write an email, give advice, rewrite text etc.) and select a subset of 113 instructions from alpacafarm that include a balanced number of instructions per "task". thus obtained set of 113 instructions is used in gpt-3.5-based evaluation, and a stratified subset of 30 instructions is used in human evaluation. controlling the task distribution ensures that none of the tasks dominates the evaluation set, leading to more reliable conclusions, and allows us ## experimental setup we study the effect of various choices such as the base model, the size of the english instruction data, adaptation strategy (full or parameter-efficient finetuning), and adaptation hyperparameters. base llms. in our work we consider (1) llama-2 (touvron et al., 2023) at 7b and 13b sizes, (2) towerbase-7b (alves et al., 2024), built on top of llama-2-7b, further trained on balanced data covering 10 languages. in the former case, the multilingual instruction-following capabilities of the model arise solely from the small amount of occasional multilingual data which is always present in english-centric pretraining corpora crawled from the internet @xcite . the latter case allows us to assess an importance of multilinguality at pretraining. instruction tuning datasets. we perform instruction tuning on two english instruction datasets: dolly @xcite (denoted dolly-en), 15k crowdsourced instructions covering 7 different categories (creative writing, open and close qa, classification, brainstorming, information extraction), and lima @xcite (denoted lima-en), 1k samples, carefully selected from various datasets (eg. stackexchange, wikihow, etc.). in order to assess the importance of instructions multilinguality, we also consider multilingual dolly data (dolly-dt), extended by adding its automatic translations (cf. appendix b for details) into three languages (fr, pt, ru). it strategy. we consider two most popular supervised finetuning techniques: full finetuning (ft) and lora finetuning. evaluation. we evaluate responses in four languages: english, french, portuguese, and russian, and curate translations of the evaluation set into the specified languages. manual inspection of predictions was conducted by the native or fluent speakers employed at our research laboratory. we select llama-2-13b/dolly-en/ft as an anchor model configuration and apply changes to it one-by-one, i.e. changing the base model, it data, or the adaptation method. we train all model configurations with three learning rates (lrs) and choose the best lr based on surface metrics. for more experimental details, see appendix b ## conclusion in this work we demonstrate the possibility of zeroshot cross-lingual transfer of instruction following capability. we devise a multi-facet evaluation methodology, allowing us to pinpoint the main capabilities and limitations of such transfer and to point important future research directions. we highlight the critical role of lr tuning and it data size, which we hope will help in future works on it.