Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions? (2024)

Abstract: The adaption of multilingual pre-trained LLMs into eloquent and helpful assistants is essential to facilitate their use across different language regions. In that spirit, we are the first to conduct an extensive study of the performance of multilingual models instruction-tuned on different language compositions on parallel instruction-tuning benchmarks across a selection of the most spoken Indo-European languages. We systematically examine the effects of language and instruction dataset size on a mid-sized and a large, multilingual LLMs by instruction-tuning them on parallel instruction-tuning datasets. Our results demonstrate that instruction-tuning on parallel instead of monolingual corpora benefits cross-lingual instruction following capabilities by up to 9.9%. Furthermore, we show that the Superficial Alignment Hypothesis does not hold in general, as the investigated multilingual 7B parameter model presents a counter-example requiring large-scale instruction-tuning datasets. Finally, we conduct a human annotation study to understand the alignment between human-based and GPT-4-based evaluation within multilingual chat scenarios.

## introduction llms have a significant impact on the daily work of many, as they are practical to use and assist in solving natural text problems ranging from creative writing to math problems @xcite . one of the primary reasons for their fast adoption as assistants is their facilitated usage by simply instructing the model to conduct a specific task. the training of such an assistant involves multiple stages of model training @xcite . first, an extensive, compute-intensive pre-training over large document corpora is conducted where the model is typically trained to predict the next token in a sequence. while pre-trained llms already exhibit significant capabilities, there is a discrepancy between their pre-training objective and the users' need for an instruction-following model. therefore, in the second step, fine-tuning is performed where the model is trained to follow the users' instructions within a chat environment. the second step is crucial for the model to solve complex, multi-turn user requests. with the availability of strong open-source english-centric models @xcite , many english-centric instruction-tuning datasets emerged @xcite @xcite . while there are adoptions of monolingual english models for other languages @xcite , the choice and training of tokenizers is critical for the fair representation of languages @xcite within pre-trained models, as otherwise, underrepresented languages are not only more resource intensive regarding training and inference, but also the performance for those languages drop. a fundamental problem is the availability of appropriate open-source, multilingual datasets and benchmarks for training and assessing instruction-tuned llms. here, especially the lack of multilingual benchmarks targeting instruction-tuned models represents a major gap, as previous instruction-tuned multilingual models are only evaluated on zeroor few-shot, academic benchmarks targeting pretrained llms @xcite . however, it is essential to evaluate the multilingual instructionfollowing capabilities of the model on instruction benchmarks to realistically assess the helpfulness of a model as a chat assistant. to the best of our knowledge, there is no study investigating multilingual instruction-tuning focusing on multilingual user request performance where the base model has been pre-trained with a large amount of multilingual data where english is represented only by around 42%. we tackle this research gap by translating mt-bench into the par-allel benchmark mt-bench-x and systematically investigate how the language and size of instruction datasets impact the instruction-tuning of pretrained, mid-sized, and large multilingual llms for the germanic and italo-western language family, including english, german, french, italian, and spanish, on this novel benchmark dataset. to answer the research question of whether multilingual models pre-trained with a focus on indo-european languages require instruction-tuning in all target languages to show competitive instructionfollowing capabilities across target languages, we make the following contributions: ## related work this section provides an overview of instructiontuning datasets and aspects important for their utilization. ## multilingual instruction-tuning data to investigate the defined research questions, we require high-quality parallel instruction-tuning datasets of different sizes. while there exist multilingual instruction datasets, the distribution of languages is highly skewed towards english or datasets contain shorter, less complex responses @xcite , as a preliminary study shown in the appendix a reveals. an exception here is bactrian-x @xcite , a largescale, parallel instruction-tuning dataset. therefore, we select bactrian-x (section 3.1) as a largescale multilingual instruction-tuning dataset and translate a smaller english-only instruction-tuning dataset lima @xcite into our target languages (section 3.2). for both datasets, we created different multilingual compositions (section 3.3) foot_3 . ## multilingual benchmarks to evaluate the multilingual instruction-following capabilities of the models, a comprehensive multilingual benchmark for our target languages is indispensable. thus, we translated mt-bench @xcite (section 4.1) into the five target languages, followed by human quality assurance for each language to create the multilingual mt-bench variant mt-bench-x (section 4.2). we employed mt-bench-x to conduct a machine evaluation and a human evaluation (section 4.3) to assess the reliability and limitations of mt-bench-x. ## multilingual performance analysis to answer the question of whether polyglot models require multilingual or monolingual instructions, we conducted several experiments. we describe the experimental setup in section 5.1, followed by the evaluation by gpt-4-as-a-judge of single score ratings and pair-wise evaluation (section 5.2). moreover, we analyze the role of dataset quality by comparing same-sized synthetic with curated datasets in section 5.3 and elaborate the superficial alignment hypothesis for the multilingual setup in section 5.4. additionally, we conducted a qualitative analysis (appendix g). ## human evaluation human evaluation is the gold standard for evaluating the output of generative models, as responses can be complex and highly diverse. following the human evaluation as described in section 4.3 on answers of the 24eu-7b model tuned on bactrian-x, we: i.) inspect the role of positional bias in mt-bench-x (section 6.1), ii.) compare the voting distributions between humans and gpt-4 in section 6.2, and iii.) analyze the correlation of agreement levels between humans and gpt-4 in section 6.3. ## conclusion & future work we see this work as a fundamental step towards supportive multilingual assistants. across a selection of major indo-european languages, we thoroughly analyzed instruction-tuned models using parallel instruction-tuning benchmarks. hereby, the creation of multilingual benchmarks and datasets are crucial to investigate multilingual instructiontuning capabilities. our findings highlight the benefits of instruction-tuning on parallel datasets, showcasing improvements of up to 9.9% in crosslingual instruction-following capabilities compared to monolingual corpora. additionally, our results challenge the generalizability of the superficial alignment hypothesis, showing that extensive instruction-tuning datasets are necessary for midsized multilingual models, but not necessarily for larger models. we extended the findings of @xcite for german and analyzed the similarities and differences between human evaluation and evaluation with gpt-4-as-a-judge. we identify disparities between human evaluations and those generated by gpt-4 in multilingual chat scenarios. by illuminating these challenges, we emphasize the need for future research to address them. additionally, we recognize the need to explore the impact of multilingual multi-turn dataset variants, which we leave as an avenue for future exploration. by addressing these challenges head-on, we can improve the performance of generative assistants in real-world communication contexts, advancing the field of natural language processing for practical applications.