Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models? (2024)

Abstract: Multilingual large language models are designed, claimed, and expected to cater to speakers of varied languages. We hypothesise that the current practices of fine-tuning and evaluating these models may not perfectly align with this objective owing to a heavy reliance on translation, which cannot cover language-specific knowledge but can introduce translation defects. It remains unknown whether the nature of the instruction data has an impact on the model output; conversely, it is questionable whether translated test sets can capture such nuances. Due to the often coupled practices of using translated data in both stages, such imperfections could have been overlooked. This work investigates these issues using controlled native or translated data during the instruction tuning and evaluation stages. We show that native or generation benchmarks reveal a notable difference between native and translated instruction data especially when model performance is high, whereas other types of test sets cannot. The comparison between round-trip and single-pass translations reflects the importance of knowledge from language-native resources. Finally, we demonstrate that regularization is beneficial to bridging this gap on structured but not generative tasks.

## introduction instruction tuning, or supervised fine-tuning, can prepare a large language model (llm) for better task generalization and natural interactions in downstream applications @xcite @xcite . major efforts of building instruction datasets centre on english @xcite @xcite , whereas the multilingual counterparts remain modest in size, variety, and coverage. many multilingual instruction datasets have been seeded from english data and developed using translation as part of the pipeline @xcite @xcite . a notable exception is aya, a year-long project that invited volunteers around the globe to write and edit prompt-response examples in their native language @xcite , making it a language-native dataset. 2 although the aya dataset was contributed by volunteers, it still carries a high social utility cost considering the personnel hours devoted. in contrast, translating existing resources by machine, or even by human, is a more convenient option. nonetheless, translated data carry imperfections @xcite ): 1) it represents the culture and knowledge specific to the original language; 2) the translation process introduces translationese, an unnatural language style @xcite , as well as errors since certain content or tasks can be corrupted, e.g. grammatical error correction. on the other hand, recent research discovered that instruction tuning is "superficial" where an llm mainly learns the response format @xcite , and it cannot enhance knowledge at the current scale @xcite . these insights imply that the shortcomings of translated data might not propagate into an instruction-tuned model. hence, when widening language support, an important question arises: is translated data sufficient for instruction tuning? we regard "sufficiency" as the fact that, when used to fine-tune an llm, translated instructions should lead to output quality similar to that of native data. yet, examining this training data factor cannot be separated from carefully considering the evaluation protocol, because many existing multilingual benchmarks have been created via translation. this translation bias, if present in both training and test sets, could hinder a meaningful con-clusion. we thus put forward our second research question: if translated and native instruction data make a difference, would a translated benchmark capture it? subsequently, we use round-trip translated data to answer: which is the cause of the gap, translation defects or missing language-specific knowledge in instructions? finally, when translated data is hardly avoidable: what techniques can we adopt to bridge the performance gap? this work systematically investigates native and translated data used during instruction tuning and evaluation. we experiment with eight models of varying sizes and data distributions and evaluate these models on nine benchmarks of different natures: translated versus native as well as classification versus generation. empirical results suggest that a prudent choice in multilingual llm evaluation is crucial. foreshadowing the answers to the research questions raised earlier: these insights mean that opposite conclusions can be made when different combinations of instruction and test sets are adopted. based on the findings, we recommend multilingual llms be evaluated on a range of benchmarks to include language-native and generative tasks. ## conclusion and future work this work systematically analysed the effects of native and translated data on both the llm instruction tuning and evaluation ends. the difference in data leads to result gaps on native test sets and generative benchmarks. we showed that knowledge mismatch is more likely to cause performance degradation rather than translation errors. with regularization, translated instruction data can potentially catch up with native data on structured benchmarks but not generative tasks. given our findings, we would like to call for prudent choices in multilingual llm benchmarking. while the current work provides comprehensive empirical results and extrinsic evaluation, future work can consider investigating the knowledge in data intrinsically. more broadly, it is meaningful to coordinate efforts to develop large-scale native test sets that more accurately assess the breadth of languages and cultures llms aim to serve.