R o M ath: A Mathematical Reasoning Benchmark in R omanian (2025)

Abstract: Mathematics has long been conveyed through natural language, primarily for human understanding. With the rise of mechanized mathematics and proof assistants, there is a growing need to understand informal mathematical text, yet most existing benchmarks focus solely on English, overlooking other languages. This paper introduces RoMath, a Romanian mathematical reasoning benchmark suite comprising three subsets: Baccalaureate, Competitions and Synthetic, which cover a range of mathematical domains and difficulty levels, aiming to improve non-English language models and promote multilingual AI development. By focusing on Romanian, a low-resource language with unique linguistic features, RoMath addresses the limitations of Anglo-centric models and emphasizes the need for dedicated resources beyond simple automatic translation. We benchmark several open-weight language models, highlighting the importance of creating resources for underrepresented languages. The code and datasets are available for research purposes.

## introduction mathematics has been a central intellectual preoccupation to humans since the beginning of civilization, the first mathematical writings dating back approximately 4000 years @xcite . historically and in the present, mathematics has been mostly written, spoken and taught in natural language, albeit with its own specialized vocabulary, having strict formalism only sparsely introduced between free-text explanations and reasoning. the primary audience of mathematical reasoning is other humans, not computers. the natural language of mathematics contains a mix of formulas, symbols, neologisms, jargon and words with different meanings than their common meaning (e.g., "real" / "imaginary" numbers). mathematics implies rigor and precise reasoning, qualitatively different from general nlp. there is a pressing need to automatically process and understand the existing large amount of mathematical text written in natural language to enable efficient knowledge extraction, facilitate automated theorem proving, and enhance accessibility for both researchers and automated systems. recently, large language models (llms) have shown great promise in handling a multitude of natural language tasks, including tackling mathematical reasoning problems @xcite @xcite . out of the common benchmark suite for evaluating llms, datasets such as gsm8k @xcite and math @xcite remained central in the development of reasoning models @xcite , and continue to be challenging even for the larger, proprietary models @xcite . current mathematics benchmarks and datasets have focused solely on english, mostly disregarding other low-resourced languages. the tacit requirement for using ai tools is fluency in english @xcite . however, mathematical reasoning ability is independent of the underlying language @xcite and anglo-centric models have been shown to exhibit the same biases of the english language, even when prompted in other languages @xcite @xcite . the focus on datasets and models in a language other than english allows the democratization of learning for underrepresented languages and cultures. recently, romanian llm development has started to flourish with initiatives such as @xcite , having finetuned several llms on romanian text. however, for evaluation, the authors used translated versions of popular english datasets and several native romanian benchmarks, but no evaluation was performed on dedicated reasoning tasks in romanian. aside from code generation @xcite , currently, there is no reasoning benchmark for romanian. in this paper, we propose romath foot_1 , foot_2 , a romanian mathematical reasoning benchmarking suite comprised of three datasets, baccalaureate, competitions and synthetic, each with its own particularities. romath aims to provide a comprehensive benchmark suite, having high-school-level problems across multiple domains (linear and abstract algebra, calculus, limits, geometry, probabilities) and across multiple levels of difficulty, ranging from easy calculations, to baccalaureate-level problems, to more difficult, proof-centric, competitionlevel problems. the purpose of romath is to provide a mathematical benchmark for romanian and to stimulate the development of enhanced reasoning capabilities of non-english llms. this work makes the following contributions: pretraining datasets for mathematics. there has been an ongoing interest in representation learning for both mathematical expressions and text @xcite . however, beyond representation learning, with the recent success of llms in a wide range of tasks, there has been increased attention to training and evaluating the mathematical reasoning of llms. for pretraining, the general approach is to filter common crawl web pages and pdfs to obtain high quality math tokens. for instance, datasets such as mathwebpages @xcite , proofpile @xcite , and openwebmath @xcite are used to pretrain high-performing llms specialized in math, such as minerva @xcite and llema @xcite . ## method we describe below the process for collecting baccalaureate and competitions, the two subsets that are collected by crawling publicly available pdfs. the synthetic subset is comprised of programmatically generated problems directly in romanian. ## conclusions and future directions in this paper, we proposed romath, a benchmarking suite consisting of three datasets with mathematical problems written in romanian: baccalaureate, competitions and synthetic. we detailed the construction process and composition for each subset and benchmarked several open-weight llms under different training and evaluation scenarios. we are the first to provide quantitative results for mathematical reasoning in romanian. surprisingly, we found that mathematics problems written in romanian can be properly handled by english-centric models, providing proper solutions in romanian. it is unclear why this occurs, especially since such models are not explicitly trained on romanian math tokens and most models have strong language filters to train only on english. our results suggest that such llms would potentially receive a passing grade (i.e., more than 50%) on the romanian baccalaureate exam, scoring an average of ∼56% across all problems in baccalaureate. an important future direction is reliable automatic annotations with chain-of-thought (cot) traces for multilingual reasoning problems. our results indicate that a significant factor in improving performance in mathematical reasoning is the presence of intermediate reasoning steps in the solutions. performance is not reliably improved by fine-tuning without cot, and the presence of more detailed solutions enables scalable training with reinforcement learning algorithms such as grpo @xcite . currently, only a subset of romath contains intermediate steps for problem solutions, and further structured annotations could significantly increase the data quality.