A Fair Comparison without Translationese: E nglish vs. Target-language Instructions for Multilingual LLM s (2025)

Abstract: Most large language models are multilingual instruction executors. Prior studies suggested that English instructions are more effective than target-language instructions even for non-English tasks; however, these studies often use datasets and instructions translated from English, which introduce biases known as translationese, hindering an unbiased comparison. To address this issue, we conduct a fair comparison between English and target-language instructions by eliminating translationese effects. Contrary to previous studies, our experiments across several tasks reveal that the advantage of adopting English instructions is not overwhelming. Additionally, we report on the features of generated texts and the instruction-following abilities when using respective instructions.

## introduction in recent years, large language models (llm) have demonstrated outstanding performance across a variety of natural language processing (nlp) tasks. to fully leverage their capabilities, it is crucial to provide these models with appropriate instructions @xcite . specifically, because multilingual llms (mllm) offer better non-english performance, an unavoidable question-should instructions be given in english or the target-language?-has been under discussion in several studies @xcite @xcite . a reasonable consideration of this issue is that the training process for mllms is still dominated by english data, suggesting that english instructions might be more effective, even for non-english tasks. indeed, previous studies have reported the effectiveness of english instructions by comparing the lan-(a) a common experimental setting in previous studies. the target-language instructions and test datasets were translated from english, which introduces the influence of translationese. guages used in instructions for mllms @xcite . however, a flaw exists in these studies: the target-language datasets and instructions were produced by translating from english (figure 1a ). texts produced through translation are prone to information loss, unnatural expressions, and stylistic differences compared to texts written by native speakers-phenomena referred to as "translationese" foot_1 @xcite @xcite . consequently, target-language datasets translated from english may exhibit expressions that resemble english writing style or contain content influenced by the cultural and contextual background of english-speaking countries. additionally, targetlanguage instructions translated from english may contain different information. these factors in-dicate the possibility that english instructions in previous studies were inherently advantaged, making it likely that comparisons between english and target-language instructions were biased. to this end, our study aims to conduct a fair comparison between english and target-language instructions in mllms, by eliminating translationese effects. specifically, we leverage targetlanguage datasets and instructions that are not translated from english to investigate performance differences across a range of tasks (figure 1b ). in particular, for the classification task, we employ multiple classification label sets to explore changes resulting from variations in the label sets. our experimental results reveal that, contrary to previous studies, whether english or target-language instructions tend to perform better depends on the task and labels. additionally, we conduct a detailed analysis comparing the features of generated texts and the instruction-following abilities of mllms when using english versus target-language instructions. this study contributes to a deeper understanding of how to effectively leverage mllms by offering an equitable comparison of instruction languages. the main contributions of this study are as follows: ## related work prompts for instruction-tuned models generally contain both instances and instructions. the study on whether mllms should be provided prompts in english or target-language can be categorized into instance-based and instruction-based approaches. the instance-based approach focuses on translating instances into english. @xcite and @xcite reported the effectiveness of having the llm itself translate instances into english and then process them. conversely, @xcite reported translating instances into english led to a decrease in performance for palm2. in contrast, the instruction-based approach, to which this study belongs, focuses on the language used for the instructions or prompt templates while keeping the instances unchanged. lin et al. (2022), muennighoff et al. (2023) and ahuja et al. (2023) reported the effectiveness of english instructions and prompt templates, even for non-english tasks. however, these studies used multilingual datasets translated from english, such as xnli @xcite , as test data or targetlanguage instructions translated from english, without considering the influence of translationese. on the other hand, @xcite used datasets not based on english but differed from our study by employing prompt templates based on translations and focusing on encoder-only models. ## methodology in this section, we describe the methodology and experimental setup to conduct a fair comparison between english and target-language instructions in mllms, eliminating translationese effects. ## results lexical simplification task the experimental results indicate that target-language instructions tend to outperform english instructions in the ls task. additionally, in japanese, the performance of instructions translated from english significantly decreased because the numerical information contained in the english instructions was lost in translation (appendix a.1). this result indicates that comparisons between english instructions and targetlanguage instructions translated from english, as in previous studies, may not always be fair. in such biased conditions, english instructions are unjustly evaluated as more effective. machine reading comprehension task the experimental results indicate that english instructions tend to outperform target-language instructions in the mrc task. this trend contrasts with the ls task, indicating that whether english or target-language instructions perform better varies depending on the task. review classification task the experimental results indicate that in settings with english classification labels, english instructions tend to outperform target-language instructions. conversely, in settings with target-language labels, target-language instructions tend to outperform english instructions. these findings suggest that in classification tasks, the optimal language depends on the classification labels, and using instructions that align with the labels' language can enhance the performance of mllms. ## conclusion in this study, we conducted a fair comparison between english and target-language instructions for mllms, eliminating the influence of translationese. we revealed that whether english or target-language instructions tend to perform better depends on the task and classification labels. additionally, we demonstrated that mllms exhibited differences in the features of generated texts and their instruction-following abilities when using english and target-language instructions.