Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca (2024)

Abstract: Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. While such efforts are often carried out in a single language, we empirically analyze cost-efficient strategies for multilingual scenarios. Our study employs the Alpaca dataset and machine translations of it to form multilingual data, which is then used to tune LLMs through either low-rank adaptation or full-parameter training. Under a controlled computation budget, comparisons show that multilingual tuning is on par or better than tuning a model for each language. Furthermore, multilingual tuning with downsampled data can be as powerful and more robust. Our findings serve as a guide for expanding language support through instruction tuning.

## introduction language capacity has attracted much attention in pre-trained language models. some pioneering works focused on a single language @xcite , while later works aim to cover multiple languages @xcite . in the recent blossom of open-source llms, english-centric ones include gpt-2, llama, and pythia @xcite @xcite , and multilingual ones are represented by @xcite . multilingual models seem attractive when considering operational costs, cross-lingual transfer, and low-resource languages @xcite ), yet english-centric models can possess good multilingual transferability @xcite . instruction tuning makes llms follow and respond to inputs @xcite . with multilingual instruction data becoming feasible and available, this paper compares monolingual and multilingual instruction tuning applied to english-centric and multilingual llms to search for the optimal strategy to support multiple languages. unlike prior works on multilingual multi-nlp-task tuning @xcite , we focus on open-ended question answering under language generation. our data setting combines two low-cost practices: self-instruct, which distils data from a powerful llm @xcite and the idea of leveraging machine translation to create multilingual datasets @xcite . we fine-tune several decoder llms with either full-parameter fine-tuning (fft) or low-rank adaptation @xcite with different language combinations. our experiments feature a fixed computation budget to offer practical insights. it is shown that multilingual tuning is preferred to monolingual tuning for each language under lora, but the results are mixed under fft. english-tuned llms are not well-versed in responding in other languages, whereas a downsampled multilingual tuning scheme proposed by us is more robust. finally, we examine our model performance on unseen languages and various llms of roughly the same size. ## related work many large language models appeared recently: the closed-source gpt model family @xcite @xcite ; open-source english-centric models like llama @xcite , openllama @xcite , and pythia @xcite ; open-source multilingual models like mt5 @xcite and bloom @xcite . these models have exhibited different degrees of language versatility. llm pre-training data is usually skewed towards english. one way to improve an llm's coverage of non-english languages is through continued pretraining @xcite . another rich body of literature looks into multilingualism in instruction tuning, which is used to adjust base models to respond to input @xcite @xcite . it trains an llm by providing downstream tasks' input and output in a specific format. early research created a multilingual instruction dataset using machine translation and showed that multilingual tuning gained higher performance than english-only fine-tuning @xcite . they also found that low-cost translated instructions are superior to human-written non-english prompts on multiple language understanding tasks. lately, multiple contemporaneous papers delv-ing into multilingual instruction tuning have been made public on arxiv-some appeared before our work and some after. this reflects the importance and interest in widening llms' language support. @xcite created an instruction dataset with instructions translated from english but responses generated by an llm. when tuned with lora, their monolingual models outperform multilingual ones on language understanding tasks. @xcite created a multilingual counterpart of alpaca using self-instruct. it has also been showcased that translation instructions improve cross-lingual capabilities @xcite @xcite and research explored more crosslingual task data and multilingual tuning @xcite . moreover, researchers have unveiled that fine-tuning on a modest number of languagesapproximately three-seems to effectively instigate cross-lingual transfer in downstream tasks @xcite . ## conclusion this paper presents a study of instruction tuning of large language models in different language contexts. our study in a resource-controlled setting suggests that multilingual tuning offers more benefits compared to monolingual tuning. we find that multilingual tuning on a downsampled dataset achieves better robustness on unseen languages. computation in this work was performed on lumi, karolina, and baskerville. we acknowledge csc-it center for science, finland for awarding this project access to the lumi supercomputer, owned by the eurohpc joint undertaking, hosted by csc (finland) and the lumi consortium through finnish extreme scale call (project luminmt). karolina was supported by the ministry of education, youth and sports of the czech republic through the e-infra cz (id:90254). the baskerville tier 2 hpc was funded by the epsrc and ukri through the world class labs scheme (ep/t022221/1) and the digital research infrastructure programme (ep/w032244/1) and is operated by advanced research computing at the university of birmingham.