B en N um E val: A Benchmark to Assess LLM s’ Numerical Reasoning Capabilities in B engali (2025)

Abstract: Large Language Models (LLMs) demonstrate exceptional proficiency in general-purpose tasks but struggle with numerical reasoning, particularly in low-resource languages like Bengali. Despite advancements, limited research has explored their numerical reasoning capabilities in these languages. To address this gap, we present BenNumEval (Bengali Numerical Evaluation), a benchmark designed to assess LLMs on numerical reasoning tasks in Bengali. It comprises six diverse tasks and a total of 3.2k samples curated from real-world problem-solving scenarios. Our extensive evaluations reveal that even with advanced prompting techniques such as Cross-Lingual Prompting (XLP) and Cross-Lingual Chain-of-Thought Prompting (XCoT), LLMs fall notably short of human-level performance, particularly when using Bengali Native Prompting (BNaP). These findings underscore the substantial gap between current LLM capabilities and human expertise in numerical reasoning, highlighting the need for more robust and linguistically inclusive AI models to advance Bengali Language Processing and equitable AI development. The source code for the system and evaluation pipeline is publicly available on GitHub.

## introduction numerical reasoning is an essential ability that is vital in numerous everyday situations. given the prevalence of numerical data in textual content, the ability to perform numerical reasoning is essential for interpreting and solving problems encountered in everyday life. this process involves extracting critical details from text and transforming them into mathematical formats. recently, llms have demonstrated considerable potential in addressing general-purpose tasks 1 https://github.com/kawsar-pie/bennumeval within the ai community, showcasing their efficiency across a broad spectrum of applications @xcite . however, despite these advancements, current state-of-the-art ai systems remain fragile and often need help when presented with mathematical reasoning problems that are posed in a slightly altered way @xcite . this constraint makes llms less effective for realworld problem-solving, particularly in handling numerical reasoning tasks requiring multiple reasoning steps @xcite . while these models perform adequately with simple, single-step tasks, they frequently falter when confronted with more complex problems, such as math word problems, which require the models to comprehend the text, determine the appropriate mathematical operations, and arrive at the correct solution @xcite . this highlights the need for further advancements to enhance their robustness in handling such multifaceted challenges. to address these challenges, various benchmarks have been developed for numerical reasoning. for example, mgsm @xcite , a multilingual benchmark that includes ten languages with diverse linguistic structures. @xcite developed numglue and lila benchmark @xcite , which assess how effectively advanced language models can manage numerical reasoning tasks across various problems in english. this work introduces bennumeval (bengali numerical evaluation), a novel benchmark dataset designed for numerical reasoning tasks in the bengali language. currently, there is a lack of dedicated datasets for evaluating state-of-the-art (sota) models on these tasks in bengali, making it difficult to assess their performance accurately. this limitation highlights a significant gap in evaluating the numerical reasoning capabilities of models in low-resource languages. bennumeval seeks to address this challenge by providing a structured benchmark, fostering new research opportunities in bengali language processing (blp), and advancing ai capabilities in underrepresented languages. the main contributions of this work are as follows. ## related work numerical reasoning has grown from small-scale to large-scale datasets, supporting deep learning research. this section chronologically reviews vital datasets and models, showcasing their impact on advancing mathematical reasoning in nlp. ## overview of tasks in bennumeval evaluating llms on numerical reasoning is challenging, particularly when creating a benchmark in bengali due to the lack of resources. to address this, we introduce bennumeval, a benchmark dataset with six tasks focused on numerical reasoning in bengali (see ## dataset distributions the bennumeval dataset is designed to cover a wide range of numerical reasoning tasks. this sec-tion provides an overview of the distribution of the dataset, detailing the composition and structure across different task categories. we analyze the dataset's variety in terms of task type, sample size, and format, highlighting its balanced approach to challenging models in various aspects of numerical reasoning. ## results and discussions the bennumeval dataset is utilized to assess the performance of various llms (see ## conclusion this paper introduced bennumeval, a novel benchmark dataset comprising six diverse tasks designed to assess the numerical reasoning capabilities of large language models (llms) in bengalia resource-constrained language with limited computational linguistic resources. our extensive evaluation highlights a significant performance gap between state-of-the-art llms and human proficiency in numerical reasoning tasks, particularly in bengali. these findings underscore the pressing need for more robust and linguistically inclusive ai models capable of handling mathematical reasoning in low-resource languages. by advancing research in bengali language processing, bennu-meval serves as a critical stepping stone toward developing more equitable and effective ai systems for underrepresented languages.