Do Multilingual Language Models Think Better in E nglish? (2024)

Abstract: Translate-test is a popular technique to improve the performance of multilingual language models. This approach works by translating the input into English using an external machine translation system before running inference. However, these improvements can be attributed to the use of a separate translation system, which is typically trained on large amounts of parallel data not seen by the language model. In this work, we introduce a new approach called self-translate that leverages the few-shot translation capabilities of multilingual language models. This allows us to analyze the effect of translation in isolation. Experiments over 5 tasks show that self-translate consistently outperforms direct inference, demonstrating that language models are unable to leverage their full multilingual potential when prompted in non-English languages. Our code is available at https://github.com/juletx/self-translate.

## introduction multilingual autoregressive language models like xglm @xcite , bloom @xcite and palm @xcite have shown impressive capabilities on many tasks and languages. however, performance is usually lower for non-english languages, especially for low-resource ones @xcite . a common approach to mitigate this problem is to use translate-test, where the test data is translated into english using an external machine translation (mt) system, and then fed into the model. while primarily explored in the traditional pretrain/finetune paradigm @xcite , early evidence has shown that translate-test can also bring sizeable improvements for few-shot learning with autoregressive language models @xcite . however, translate-test relies on a separate mt system, which is usually trained on large amounts as shown in figure 1 , we find that self-translate works better than solving the task directly in the original language. this demonstrates that multilingual language models are unable to leverage their full potential when prompted in non-english languages. we find this phenomenon to be consistent across tasks, and more prominent for large models and high-resource languages. all in all, our work reveals an important limitation of multilingual language models, and prompts for future work to unleash their full potential without the need for in direct inference (standard) the task is solved by prompting the model in the original language. in self-translate (proposed), we first translate the input into english by prompting the same model, and then solve the task in english. ## experimental settings we next describe our experimental design, and report additional details in appendix a. ## results multilingual language models do transfer capabilities across languages. one possible explanation for the previous finding is that language models acquire capabilities separately for each language, without any effective cross-lingual transfer. however, a closer comparison of llama and xglm refutes this hypothesis. in particular, we observe that llama is much better than xglm in mgsm despite being worse in other tasks. this is because mgsm is an emergent task @xcite , and xglm, being smaller and less capable, obtains near 0 accuracy. in contrast, llama is more capable at solving math word problems, and it is able to leverage this capability even if prompted in other languages. the superior performance of self-translate shows that this cross-lingual transfer is not fully effective, but our results suggest that it does happen to a large extent. self-translate is more effective for high-resource languages and large models. figure 1 shows that the gap between self-translate and direct inference gets larger at scale. similarly, as shown by mt outperforms self-translate, but the gap narrows at scale. as shown by figure 3 (top), nllb performs better than self-translate, meaning that it can still be beneficial to use an external mt system. however, the gap narrows at scale, as the translation capabilities of the largest models approach nllb (figure 3 , bottom). given the recent claims that state-of-the-art multilingual language models are competitive with traditional mt systems (vi- ## related work translate-test is a strong baseline in the traditional pretrain/finetune paradigm @xcite . early evidence shows that it is also effective for prompting autoregressive language models @xcite , as these models have irregular performance depending on the input language @xcite . @xcite propose a systematic way of qualifying the performance disparities of llms under multilingual settings, employing a novel back-translation-based prompting method. recent work has shown that multilingual language models are good translators @xcite @xcite , which our ap-proach exploits to replace the external mt system in translate-test. concurrent to our work, @xcite propose a more complex prompting method that involves translating the input, but they only experiment with proprietary models and do not study the role of translation in isolation. finally, @xcite show that using synthetic parallel data from unsupervised mt can improve the performance of multilingual models, but they focus on pretraining seq2seq models. ## conclusion we have proposed a new method called selftranslate, where we use a multilingual language model to translate the test data into english, and then feed the translated data to the same model to solve the task. self-translate consistently outperfoms the standard direct inference approach, which directly feeds the test data in the original language. our approach does not involve any additional data or training, showing that language models are not able to leverage their full multilingual potential when prompted in non-english languages. in the future, we would like to explore training methods to mitigate this issue without the need of intermediate inference steps. our code and data will be available upon acceptance.