Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrations (2024)

Abstract: The language ability of Large Language Models (LLMs) is often unbalanced towards English because of the imbalance in the distribution of the pre-training data. This disparity is demanded in further fine-tuning and affecting the cross-lingual abilities of LLMs. In this paper, we propose to empower Instruction-tuned LLMs (It-LLMs) in languages other than English by building semantic alignment between them. Hence, we propose CrossAlpaca , an It-LLM with cross-lingual Instruction-following and Translation-following demonstrations to improve semantic alignment between languages. We validate our approach on the multilingual Question Answering (QA) benchmarks XQUAD and MLQA and adapted versions of MMLU and BBH.Our models, tested over six different languages, outperform the It-LLMs tuned on monolingual data. The final results show that instruction tuning on non-English data is not enough and that semantic alignment can be further improved by Translation-following demonstrations.

## introduction large language models (llms) achieve comprehensive language abilities through pre-training on large corpora @xcite @xcite . hence, the acquired language abilities follow the corpora features, mostly available in english @xcite . this phenomenon produces an imbalance in both pre-training @xcite and fine-tuning @xcite . consequently, llms underperform in languages different from english @xcite . efforts to increase multilingual abilities propose continuing pre-training with large-scale monolingual data @xcite @xcite . however, learning a language from monolingual data requires considerable data and computational resources. in this paper, we propose crossalpaca, an instruction-tuned llm (it-llm) empowered with a semantic alignment between english and other languages. we analyze how to elicit the non-english abilities of it-llms by focusing on the crucial phase: instruction-tuning over instructionfollowing demonstrations. hence, we study the effect of cross-lingual alignment by proposing translation-following demonstrations to improve the instruction-tuning phase. in our experiments, we use llama-7b @xcite as the llm backbone and consider six target languages selected from among the available data (shown in behind an instruction-tuning phase, we evaluate the performance of our six language-specific crossalpacas using four different benchmarks: two native-multilingual, i.e., xquad @xcite and mlqa @xcite , and two native-monolinguals, mmlu @xcite and bbh @xcite . the results show that crossalpacas instructed with languagespecific instruction and translation data far outperform alpacas instructed only with non-english demonstrations. however, even though crossalpaca reduces the gap, the translation-following models of the original alpaca still perform best. this result shows that llama's learning abilities on english data are superior to those on non-english data. the semantic alignment task also improves the cross-lingual abilities of it-llms. our findings can be summarized as follows: this example shows the it-crossalpaca, fine-tuned on it-alpaca and translation-following demonstrations. ## experiments in order to observe the english and non-english abilities of large language models (llms) and the impact of the instruction-tuning approach in cross-lingual scenarios, we propose crossalpaca. our approach is based on instruction-tuning on language-specific data augmented with a crosslingual semantic alignment. hence, we set several baseline models (section 4.1), which we augmented with our crossalpaca approach (section 4.2). finally, we performed a series of systematic evaluations (section 4.3.1) to observe the impact of the proposed intervention. ## results improving non-english abilities in instructiontuned large language models (it-llms) remains challenging. however, the x-crossalpacas revealed improved results in cross-lingual question answering (qa) benchmarks and maintained logical-mathematical skills. from the results of figure 3 (further detailed in the fine-grained analysis highlighted the importance of cross-lingual alignment data and the critical issues with non-english data. this opens the way for new hypotheses regarding the imbalance of pre-training languages and learning abilities via instruction-tuning. ## conclusion in this paper, we proposed crossalpaca, an approach to empowering the instruction-tuning of llms on non-english data. specifically, we coupled instruction-following (alpaca-style) demonstrations with translation-following demonstrations. our method seeks to instruct the llm to semantic alignment between english and non-english overperforms models instructed on non-english texts. in particular, thanks to our crossalpaca demonstrations, the instructed models achieved significant performance improvements on four question answering benchmarks xquad, mlqa, mmlu, and bbh. in addition, we observe that semantic alignment strengthens with increasing translation-following data; this demonstrates the de-facto abilities of it-llms to learn from instructions. our approach and results contribute to improved research on the potential for producing more powerful llms for non-english languages.