Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap (2025)

Abstract: Large language models (LLMs) demonstrate exceptional performance on complex reasoning tasks. However, despite their strong reasoning capabilities in high-resource languages (e.g., English and Chinese), a significant performance gap persists in other languages. To investigate this gap in Korean, we introduce HRM8K, a benchmark comprising 8,011 English-Korean parallel bilingual math problems. Through systematic analysis of model behaviors, we identify a key finding: these performance disparities stem primarily from difficulties in comprehending non-English inputs, rather than limitations in reasoning capabilities. Based on these findings, we propose UST(Understand, Solve, and Translate), a method that strategically uses English as an anchor for reasoning and solution generation. By fine-tuning the model on 130k synthetically generated data points, method achieves a 10.91% improvement on the HRM8K benchmark and reduces the multilingual performance gap from 11.6%% to 0.7%%. Additionally, we show that improvements from method generalize effectively to different Korean domains, demonstrating that capabilities acquired from machine-verifiable content can be generalized to other areas. We publicly release the benchmark, training dataset, and models.

## introduction large language models (llms) have made remarkable progress in reasoning tasks, often surpassing expert human performance @xcite . however, this exceptional reasoning capability is primarily observed in high-resource languages, with significant performance gaps in lower-resource languages @xcite . this disparity likely stems from llms' difficulty in transferring their foundational * equal contribution 1 https://huggingface.co/haerae-hub capabilities, including reasoning skills learned in high-resource languages like english or chinese, to lower-resource languages @xcite . to investigate this gap in korean mathematical reasoning, we introduce hrm8k, a bilingual benchmark comprising 8,011 questions in both korean and english. the questions are carefully curated from existing benchmarks @xcite @xcite and korean examinations to create a perfectly parallel evaluation structure. through systematic evaluation on hrm8k, we reveal that the performance gap primarily stems from difficulties in comprehending non-english inputs, rather than limitations in reasoning capabilities. this finding challenges previous studies that suggest using english chainof-thought (cot) reasoning for multilingual questions @xcite , as we show that llms are heavily influenced by the input language itself. based on these insights, we propose ust (understand, solve, and translate), a training method that strategically uses english as an anchor for reasoning and solution generation. our approach builds on recent findings that llms effectively use english as a pivot language for processing multilingual inputs @xcite . by training on 130k synthetically generated instances, ust achieves a 10.91% improvement on the hrm8k benchmark and reduces the multilingual performance gap from 11.6% to 0.7%. furthermore, we demonstrate that these improvements generalize beyond mathematics to different korean domains, suggesting broader applications. in summary, the main contributions of this work are as follows: mathematics benchmarks mathematical reasoning has emerged as a crucial capability for language models @xcite @xcite , leading to the development of numerous benchmarks and datasets @xcite @xcite . traditional datasets such as gsm8k @xcite and math @xcite primarily target grade-school to undergraduate-level problems, while more recent efforts introduce olympiad-level challenges @xcite @xcite @xcite . although these benchmarks prove valuable for evaluating english-language mathematical reasoning, fewer resources exist for non-english or bilingual math problems @xcite @xcite . in the korean context, most benchmarks emphasize language understanding @xcite , general knowledge @xcite , or commonsense reasoning @xcite , with mathematics being largely underrepresented. while the open ko-llm leaderboard @xcite has begun translating some popular english benchmarks into korean, those translated sets are not publicly accessible. meanwhile, kmmlu @xcite ) includes only about 100 math problems, insufficient for broader evaluation. to address this gap, we propose hrm8k, a large-scale bilingual korean-english math benchmark comprising 8,011 problems, covering both competition-level korean questions and parallel translations of existing english benchmarks. multilingual reasoning and language models recent llms have shown remarkable performance in english @xcite @xcite , but many still underperform in multilingual scenarios @xcite . such performance discrepancies are attributed to limited exposure to lower-resource languages during pre-training. consequently, much research has focused on enhancing multilingual reasoning skills, including methods that explicitly use english as a 'pivot' for crosslingual tasks @xcite . for example, plug @xcite aligns internal reasoning in different languages, enabling the model to leverage stronger english reasoning for other languages. despite these developments, few works have thoroughly examined how to best calibrate reasoning between high-and low-resource languages in complex mathematical contexts. some studies investigate altering the fraction of multilingual data in training @xcite or conduct smallerscale experiments on bilingual math tasks @xcite ), yet a clear, large-scale solution remains elusive. against this backdrop, our work introduces ust, a multilingual reasoning method that intentionally routes math problems in lowerresource languages through english-based reasoning. we show that this strategy drastically narrows the performance gap and advances multilingual math capabilities. ## hrm8k in this section, we introduce the composition of the hrm8k benchmark and explain its construction process. we also conduct a contamination check to ensure data quality. detailed information about each dataset and post-processing methods are provided in appendix a. ## multilingual performance gaps a recurring observation in large language models (llms) is that performance can vary significantly depending on the language of the prompt, even if the underlying task remains the same. we confirm this phenomenon on the hrm8k benchmark: as shown in this section further investigates the causes of this gap. we first describe our experimental design (section 4.1), then analyze the results (section 4.2), and finally explore how multi-step prompting might mitigate these issues (section 4.3). ## understand, solve and translate based on the experiments in section 4, we conjecture that for smaller models improving their ability to understand korean questions and allowing it to reason in english can address or bypass the constraints limiting llms in solving korean questions. to validate this hypothesis, we fine-tune llms on a custom dataset designed to guide models through three stages: understanding korean questions, solving them in english, and translating solutions back to korean. in this section, we explain the details of our training dataset (section 5.1), report performance gains (section 5.3), and conduct ablations on the effectiveness of the training (section 5.4). ## (ours) ## tracing the performance gains in this work, we demonstrate that routing through english understanding and reasoning steps enhances model performance on hrm8k. to understand the source of these improvements, we evaluate our ust-trained model on two additional benchmarks: kmmlu @xcite and mgsm @xcite . our model shows consistent improvements across all kmmlu categories, with the largest gains in stem (+4.5) and the smallest in humss (+2.0) 3 . however, when tested on mgsm, the model shows performance drops in swahili (-3.6) and thai (-4.8). these contrasting results suggest that our gains stem not from general improvements in mathematical reasoning, but rather from enhanced korean-specific capabilities and better korean-to-english reasoning alignment. our findings align with our initial goal: addressing the performance gap between english and korean reasoning on identical questions (section 5.3). the effectiveness of our approach is demonstrated in two ways. first, it recovers most of the performance achieved with e2e prompting ( ## conclusion in this paper, we propose ust, a training method that leverages english as an anchor language to enhance reasoning capabilities in korean, and introduce hrm8k, a benchmark of 8,011 english-korean parallel mathematics problems. our analysis reveals that the performance gap in multilingual reasoning primarily stems from difficulties in processing non-english inputs. through extensive experiments, we demonstrate that ust effectively bridges this gap and shows promising generalization to various korean domains beyond mathematics. our approach offers a simple yet effective solution for non-english language users to benefit from advanced reasoning capabilities typically available only in english, suggesting a practical direction for improving multilingual reasoning capabilities in language models.