B - REASO : A Multi-Level Multi-Faceted B engali Evaluation Suite for Foundation Models (2025)

Abstract: The fast growth of large language models (LLMs) necessitates the urgent need for new NLP benchmarks. We provide B-REASO, the first inclusive Bengali assessment suite created to evaluate advanced foundation model knowledge and reasoning skills in a Bengali language setup. The B-REASO includes multiple-choice questions with four different degrees of difficulty: professional, college, high school, and middle school. The questions cover 50 different fields, from science and engineering to the humanities. Alongside B-REASO, there is B-REASO HEAVY, a subset of extremely difficult B-REASO topics that need for sophisticated reasoning skills to answer. We do a thorough assessment of the most sophisticated LLMs on B-REASO, encompassing models with an English focus. Findings show that only Claude-3.5-Sonnet was able to get an average accuracy of more than 65%, indicating that contemporary LLMs still have a long way to go. We hope that B-REASO will support the creation and expansion of foundation models for Bengali users by assisting in the analysis of significant advantages and disadvantages of these models. We open-source our code and data at https://github.com/kraritt/b-reaso.

## introduction natural language processing (nlp) has seen a revolution with the advent of large language models (llms). in the age of llms, assessment plays a critical function as the central force that shapes the direction of artificial intelligence (ai) progress. to assess lms' natural language understanding skills, traditional nlp benchmarks @xcite have been frequently used. however, since bigger models are performing at a level comparable to humans, these standards are no longer as applicable, leaving limited area for further study @xcite @xcite . recent llm evaluation has focused mostly on assessing sophisticated reasoning skills and exploring advanced world knowledge in order to adequately benchmark llms @xcite @xcite . meanwhile, as multilingual and other llms tailored for various regional languages @xcite have become more popular, evaluation benchmarks for languages other than english-have also been introduced @xcite . nevertheless, the english language remains the primary focus of the contemporary benchmarks, which leads to a restricted comprehension of llms' proficiency in other languages rapidly. as per our best knowledge, there is no comprehensive previous work that has been done on llms' understanding specifically in bengali language setup. this study focuses on assessing foundation models' advanced capabilities in the context of bengali-world's seventh foot_0 most commonly spoken languages. we introduce b-reaso (figure 1 ), the first complete bengali evaluation suite designed to fully evaluate llms' advanced knowledge and reasoning skills in a bengali environment, in an effort to close the gap between bengali llm development and assessment. as shown in ## experimental results for ao and cot prompting, the experimental findings are shown in ## conclusion we think that llm evaluations ought to go beyond simple conversational bots and help developers get llms ready for more complicated cases. the development of the difficult evaluation suite b-reaso was primarily driven by this goal. in this