B angla MATH : A B angla benchmark dataset for testing LLM mathematical reasoning at grades 6, 7, and 8 (2025)

Abstract: Large Language Models (LLMs) have tremendous potential to play a key role in supporting mathematical reasoning, with growing use in education and AI research. However, most existing benchmarks are limited to English, creating a significant gap for low-resource languages. For example, Bangla is spoken by nearly 250 million people who would collectively benefit from LLMs capable of native fluency. To address this, we present BanglaMATH, a dataset of 1.7k Bangla math word problems across topics such as Arithmetic, Algebra, Geometry, and Logical Reasoning, sourced from Bangla elementary school workbooks and annotated with details like grade level and number of reasoning steps. We have designed BanglaMATH to evaluate the mathematical capabilities of both commercial and open-source LLMs in Bangla, and we find that Gemini 2.5 Flash and DeepSeek V3 are the only models to achieve strong performance, with \ge 80% accuracy across three elementary school grades. Furthermore, we assess the robustness and language bias of these top-performing LLMs by augmenting the original problems with distracting information, and translating the problems into English. We show that both LLMs fail to maintain robustness and exhibit significant performance bias in Bangla. Our study underlines current limitations of LLMs in handling arithmetic and mathematical reasoning in low-resource languages, and highlights the need for further research on multilingual and equitable mathematical understanding.

## introduction mathematical reasoning is one of the cornerstones of human intelligence and remains a critical focus in the pursuit of artificial intelligence (ai). as ai continues to evolve, empowering machines with a deep and comprehensive understanding of mathematics not only showcases technological advancement but also represents a key milestone toward developing more generalized and capable ai systems. recently, the emergence of large language models (llms) has significantly reshaped the ai landscape, establishing them as powerful tools for automating complex tasks. llms have demonstrated remarkable proficiency in mathematical problem-solving (romera-paredes et al., @xcite , prompting extensive evaluations of their capabilities across a variety of domains @xcite . notably, llms such as chatgpt @xcite along with @xcite have shown impressive performance in generating and interpreting natural language, while the more recent gpt-4 (openai, @xcite has set new standards in both linguistic and logical tasks. the capacity to understand and solve mathematical problems is an especially desirable trait for llms, with wide-ranging applications in education, science, and industry. however, evaluating their mathematical competence is inherently challenging. while numerous benchmark datasets have been developed to assess mathematical reasoning @xcite , most are limited to the english language, with a few available in chinese @xcite . mathematical reasoning in bangla, despite the widespread use of llms by banglaspeaking communities for educational purposes, remains largely unexplored. to address this gap, we introduce the bangla mathmatical benchmark (banglamath) dataset which consists of 1.7k elementary school level math problems from bangladesh, the first benchmark dataset designed specifically to evaluate mathematical reasoning in bangla. the banglamath dataset comprises 1.7k elementary-level math word problems sourced from authentic bangla workbooks and examination materials. the dataset includes multiple-choice questions, logical puzzles, and descriptive reasoning problems, each annotated with question type metadata to facilitate fine-grained evaluation. in figure 1 , we display an example problem from the dataset, in bangla and english, with the correct (human) answer and and an incorrect one from chatgpt. we posit that the evaluation of llms should mirror that of human learners, which would allow us to convey results in a manner that is more intuitive and accessible. we assess the performance of several widely used large language models (llms)-including both commercial apis and open-source modelson the banglamath dataset. each problem in banglamath is annotated with grade-level information, allowing us to conduct fine-grained evaluations similar to stating "chatgpt scored 72 out of 100 in a sixth-grade math exam." our results show that gemini 2.5 flash and deepseek v3 consistently achieve high performance (@xmath0. we evaluate model performance across varying levels of arithmetic and reasoning complexity, and observe that accuracy significantly decreases as problem complexity increases. we also examine the robustness of these models by injecting distracting information into the math problems. our findings reveal that both of the top-performing models (gemini 2.5 flash and deepseek v3) are easily misled by the presence of such irrelevant information, resulting in incorrect reasoning and answers. to further investigate language bias in llms, we translate the banglamath dataset into english and re-evaluate the top-performing models (gemini 2.5 flash and deepseek v3). interestingly, both models show an improvement in accuracy @xmath1.5% when tested on the english-translated version, highlighting a performance disparity for low-resource languages like bangla. ## banglamath dataset we can frame our work as being driven by the question: do llms have the ability to perform mathematical reasoning in bangla? to address this, we introduce the banglamath dataset, specifically designed to evaluate llms on a diverse set of mathematical problems written in bangla. our focus spans across various categories of mathematics, including arithmetic, algebra, geometry, and logical reasoning, to provide a comprehensive assessment of llms' generalpurpose reasoning and arithmetic capabilities. ## related works math-related datasets are available predominantly in english @xcite , making them unsuitable for evaluating the reasoning abilities of llms in bangla. traditional math word problem (mwp) datasets like addsub @xcite and multiarith @xcite are integrated into broader mwp repositories. other similar datasets include singleeq @xcite , aqua @xcite . gsm8k @xcite and svamp @xcite take advantage of detailed annotations and have prevailed in recent evaluations of llms. similarly, math dataset @xcite , which collects problems from american high school mathematics competitions and categorizes them into seven subjects: pre-algebra, algebra, number theory, counting and probability, geometry, intermediate algebra, and pre-calculus. however, these problems are extremely challenging-even for humans, the accuracy rate is only 40%. given that many llms are still in early stages, using overly difficult problems may have limited utility in evaluating their capabilities. other than english, several chinese and a hindi @xcite math-related datasets exist. agi-eval @xcite and c-eval @xcite target general-purpose, multidisciplinary evaluation for llms and contain subsets specifically designed to assess mathematical abilities. the math problems in these datasets range from middle school to college level and are often quite complex. similarly, math23k @xcite and ape210k @xcite , cmath @xcite and k6 @xcite , comprise elementary school-level math word problems. cmath and k6, are two datasets that are relatively similar to ours and are developed concurrently. both focus on math word problems from elementary school and organize instances by grade level. our work is most direclty inspired by the cmath dataset, which contains 1.7k problems collected from online workbooks and exams, while k6 comprises 600 problems collected from an educational institution. however, neither of these two datasets has been publicly released, which prevents us from conducting an empirical comparison with them. additionally, ape210k, contains an enormous 210k chinese math word problems from elementary school. its test set alone includes as many as 5,000 problems. however, the test sets do not provide annotations specific to llm evaluation. ## conclusion we have introduced banglamath, a novel dataset designed to enable fine-grained evaluation of large language models (llms) on elementarylevel mathematical word problems in bangla. to the best of our knowledge, banglamath is the first bangla mathematical benchmark dataset designed to help evaluate the mathematical reasoning abilities of llms. our results show that banglamath poses a significant challenge even for state-of-the-art llms, with performance declining as the grade level increases. additionally, we observe that as the arithmetic and reasoning complexity of problems increases, the accuracy of all evaluated models decreases. our robustness analysis reveals that top-performing models such as gemini 2.5 flash and deepseek v3 struggle when faced with irrelevant or distracting information, highlighting a vulnerability in realworld problem comprehension. furthermore, we uncover a clear language bias-performance improves when bangla problems are translated into english. this suggests that current llms are less effective at reasoning in low-resource languages like bangla compared to english. we believe banglamath can serve as a valuable benchmark for advancing non-english mathematical reasoning research, and help evaluate the cross-linguistic mathematical reasoning capabilities of llms.