When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy (2025)

Abstract: Recent Large Reasoning Models (LRMs) with thinking traces have shown strong performance on English reasoning tasks. However, the extent to which LRMs can think in other languages is less studied. This is as important as answer accuracy for real-world applications since users may find the thinking trace useful for oversight only if expressed in their languages. In this work, we comprehensively evaluate two leading families of LRMs on our established benchmark XReasoning. Surprisingly, even the most advanced models often revert to English or produce fragmented reasoning in other languages, revealing a substantial gap in the capability of thinking in non-English languages. Promoting models to reason in the user’s language via prompt hacking enhances readability and oversight. This could gain user trust, but reduces answer accuracy, exposing an important trade-off. We further demonstrate that targeted post-training, even with just 100 instances, can mitigate this language mismatch, although accuracy is still degraded. Our results reveal the limited multilingual reasoning capabilities of current LRMs and suggest directions for future research. All code and datasets are released at https://github.com/Betswish/mCoT-XReasoning.

## introduction large language models (llms) have demonstrated impressive reasoning capabilities when prompted to explicitly generate thinking traces step-by-step in natural language @xcite . recent studies further show that encouraging models to engage in long thinking traces at inference time, or training for this behavior, significantly enhances their reasoning accuracy @xcite . this approach has led to the development of a new category of models designed to generate more extensive and detailed reasoning processes, referred to as large reasoning models (lrms) @xcite @xcite . however, their capabilities are mostly tested on english tasks @xcite . it is less investigated to what extent lrms can think in a user's native language and how this affects their reasoning accuracy, when it comes to multilingual reasoning tasks. illustrated by figure 1 (a), the matching of the thinking language is as important as the accuracy because it makes the trace more readable and easier for users to verify. even if the answers are correct, the thinking traces in a language users cannot understand may undermine their trust in the model and lead to dissatisfaction with the responses, which becomes especially pronounced as tasks grow more complex in practice. in this work, we comprehensively evaluate six state-of-the-art open-sourced lrms belonging to two families: distilled-r1 (deepseek-ai, 2025), and skywork-or1 @xcite . due to the lack of multilingual reasoning datasets, we introduce a novel benchmark named xreasoning, where we observe that even the latest 32b lrms suffer from the issue of language mismatch in their thinking traces. nevertheless, we reveal that this mismatch can be significantly mitigated with prompt-hacking techniques (c.f., section 4.1), as shown in figure 1 (b ). however, this comes at the cost of reduced model performance, highlighting a trade-off between explainability and accuracy that may limit the practical value and application of these models. finally, we explore whether post-training can mitigate language mismatches in thinking traces, based on our observation that existing lrms are more likely to reason in the correct language when prompted to think in english or chinese, the two most prevalent languages in their training data @xcite . the results demonstrate that post-training, even with few instances, improves lrm's capability of thinking in the user's language but does not completely resolve the trade-off between thinking language matching and accuracy. overall, we make the following contributions: (i) we reveal the important trade-off between language matching and answer accuracy when the lrms face multilingual users. (ii) we propose and opensource a new benchmark, xreasoning, with challenging math and science questions to evaluate multilingual reasoning capabilities of advanced lrms. (iii) we show that language mismatch of thinking traces can be alleviated through prompt hacking or post-training, but both lead to a drop in answer accuracy. such persistence of this trade-off for current models highlights the need for future work toward more user-friendly multilingual lrms. ## related work multilingual reasoning benchmarks recent research has expanded model reasoning evaluation beyond english. @xcite introduced the multilingual grade school math (mgsm) benchmark, comprising 250 math problems in 11 languages. similarly to @xcite , this work illustrates an uneven answer accuracy across languages, typically favoring english with a marginally higher-than-random accuracy for low-resource languages like bengali and swahili. however, with the rapid shift from llms to lrms, the limitations of existing multilingual benchmarks have become evident @xcite . these benchmarks often fail to capture the nuanced accuracy differences across languages; for instance, recent lrms even @xmath0. to fix this limitation, we propose xreasoning, a new benchmark covering more challenging math and science questions to better evaluate these lrms. ## experimental setup benchmarks as mentioned in section 2, existing datasets, like mgsm, are insufficient for evaluating the recent lrms. for our evaluation, we start from three reasoning datasets of challenging english questions, aime2024 @xcite , aime2025 (kaggle, 2025), and gpqa @xcite , and use gpt-4o-mini (openai, 2024) to translate all questions into the other ten languages covered by mgsm, foot_0 following previous work on these languages @xcite @xcite . the resulting parallel challenging questions, combined with mgsm, form the xreasoning benchmark, consisting of 370 questions, each in 11 languages. 3 see appendix b for examples. evaluation as a focus of this work, we evaluate language matching rate by calculating the ratio of instances for which the lrms correctly follow the instruction to think in the specified language. the commonly used langdetect toolkit @xcite @xcite @xcite ) is adopted to predict the language of the thinking trace between special thinking tokens '<@xmath1. ## conclusion in this work, we investigate the overlooked issue of language matching in thinking traces and its trade-off with answer accuracy in multilingual reasoning systems. using our new xreasoning benchmark, we show that even state-of-the-art lrms struggle to generate thinking traces in a user-specified language. simple prompt-hacking reduces language mismatch but at a substantial cost to accuracy, especially on complex tasks. further, targeted post-training on 100-250 examples sharply improves language alignment but still markedly reduces accuracy, highlighting this persistent trade-off. we argue that advanced adaptation strategies, such as reinforcement learning @xcite @xcite , will be critical to resolve this tension and develop multilingual reasoning systems suitable for practical, trustworthy applications.