Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models (2025)

Abstract: Large language models (LLMs) have demonstrated multilingual capabilities, yet they are mostly English-centric due to the imbalanced training corpora. While prior works have leveraged this bias to enhance multilingual performance through translation, they have been largely limited to natural language processing (NLP) tasks. In this work, we extend the evaluation to real-world user queries and non-English-centric LLMs, offering a broader examination of multilingual performance. Our key contribution lies in demonstrating that while translation into English can boost the performance of English-centric LLMs on NLP tasks, it is not universally optimal. For culture-related tasks that need deep language understanding, prompting in the native language proves more effective as it better captures the nuances of culture and language. Our experiments expose varied behaviors across LLMs and tasks in the multilingual context, underscoring the need for a more comprehensive approach to multilingual evaluation. Therefore, we call for greater efforts in developing and evaluating LLMs that go beyond English-centric paradigms.

## introduction large language models (llms) frequently demonstrate the capability to understand and generate text across multiple languages, a skill attributed to their training on vast corpora composed of texts from various languages @xcite @xcite @xcite . however, these datasets are often disproportionately dominated by english content @xcite @xcite , resulting in ## translation for real user queries nlp tasks typically focus on specific linguistic aspects, which may not fully encapsulate the breadth and complexity of real-world user queries which cover diverse topics and require nuanced comprehension. moreover, these benchmarks are often constructed by translating from the english data @xcite @xcite @xcite . this approach leads to datasets that are not truly challenging, as they miss the rich culture-specific elements crucial for truly nuanced language understanding for different languages. to assess the impact of translation on real-world queries, we extract user requests from sharegpt foot_2 , a website to share real conversations with chatgpt. ## related work multilingual evaluation. since the release of chatgpt, the evaluation of llms has attracted the attention of the research community @xcite . shi et al., 2022 evaluated llms on mgsm and found that the models demonstrated strong multilingual reasoning capabilities, even for low-resource languages. bang et al., 2023 evaluated chatgpt on 23 datasets covering 8 nlp tasks. they found that chatgpt failed to generalize its capabilities to non-latin scripts. to cover tasks, ahuja et al., 2023 evaluated chat-gpt and gpt-4 on 16 nlp datasets across 70 languages and compared them with state-of-the-art non-autoregressive models. concurrently, lai et al., 2023 evaluated chatgpt on 7 different tasks across 37 diverse languages. however, these evaluations are primarily limited to standard nlp tasks and largely overlook real-world scenarios and cultural knowledge (fung et al., 2024), which are crucial for understanding the practical applicability of llms. multilingual prompting strategies. the translate-test is a popular technique used to refine the performance of multilingual nlp benchmarks (conneau et al., 2018; ponti et al., 2020; artetxe et al., 2023; moghe et al., 2023; qi et al., 2022; huang et al., 2022). in the era of llms, various strategies have been developed to enhance the performance of llms using multilingual datasets. shi et al., 2022 discovered that en-cot outperforms native-cot. huang et al., 2023 introduced cross-lingual-thought prompting to minimize language disparities. in parallel, qin et al., 2023b introduced cross-lingual prompting, and etxaniz et al., 2023 suggested self-translate to elevate their performances. effective in translating prompts into english, these methods excel in nlp tasks but remain uncertain in real-world applications. their success hinges on the english-centric nature of the llms. our study evaluates translation effectiveness across nlp tasks, real user queries, and non-english-centric llms, revealing the limitations of these methods. ## conclusion we have conducted a thorough evaluation of llms in various multilingual tasks. these tasks include traditional nlp benchmarks, real user queries, and culture-related tasks. even though translationbased methods are simple and effective strategies to overcome the limitations inherent in englishcentric llms, they are not optimal for all scenarios, highlighting the necessity of more comprehensive multilingual evaluation. the experiment on non-english-centric llms and culture-related tasks demonstrates that employing prompts in the native language emerges as a more effective approach. this method is particularly adept at capturing the subtleties and intricacies unique to each language. the challenge of the setting is that it requires llms to be proficient in various languages, calling for the prioritization of research and development efforts toward the creation of strong multilingual llms.