Studying the capabilities of Large Language Models in solving Combinatorics Problems posed in H indi (2025)

Abstract: There are serious attempts at improving the mathematical acumen of LLMs in questions posed in English. In India, where a large fraction of the students study in regional languages, there is a need to assess and improve these state-of-the-art LLMs in their reasoning abilities in regional languages as well. As Hindi is a language predominantly used in India, this study proposes a new dataset on mathematical combinatorics problems consisting of a parallel corpus of problems in English and Hindi collected from NCERT textbooks. We evaluate the “raw” single-shot capabilities of these LLMs in solving problems posed in Hindi. Then we apply a chain-of-thought approach to evaluate the improvement in the abilities of the LLMs at solving combinatorics problems posed in Hindi. Our study reveals that while smaller LLMs like LLaMa3-8B shows a significant drop in performance when questions are posed in Hindi, versus questions posed in English, larger LLMs like GPT4-turbo shows excellent capabilities at solving problems posed in Hindi, almost at par its abilities in English. We make two primary inferences from our study: (1) large models like GPT4 can be readily deployed in schools where Hindi is the primary language of study, especially in rural India; (2) there is a need to improve the multilingual capabilities of smaller models. As these smaller open-source models can be deployed on not so expensive GPUs, it is easier for schools to provide these models to the students, and hence, the latter is an important direction for future research.

## introduction large language models (llms) have revolutionized the technological landscape, with newer applications emerging each day. one of the prime benefactors of this revolution has been the education sector. while initially these models were used as a large knowledge base for facts, the recent models also excel at reasoning tasks like program-ming and mathematics. this has benefited a large class of students who are using these models as a "personalized tutor" to understand their course material. these language models are essentially trained over a large corpus of text across the breadth of the internet-online books, wiki articles, blogs, code repositories-to capture the essence of human knowledge. however, most of the text available on the internet is in english. in a country like india, 68.83%(cen) of the population is rural, who predominantly communicate in hindi and other regional languages. in fact, more than 58%(nue) of the population undergo their school education in the regional languages. even prestigious exams like iit-jee is conducted in thirteen languages, namely english, assamese, bengali, gujarati, hindi, kannada, malayalam, marathi, odia, punjabi, tamil, telugu, and urdu. hence, it raises an important question the regional language speaking population of the country is equally benefited by the llms as the urban, english-speaking population. or, is the emergence of llms increasing the chasm between the indian population that is being educated in english and other regional languages. in this work, we investigate the effectiveness of large language models at solving questions presented to them in hindi, and compare its effectiveness at handling the same problems in english. we use the ncert textbooks (nce) for the english and hindi to collect mathematics questions in the area of combinatorics. we use multiple strategies and prompting techniques to study the gap in the capabilities of the llms at solving mathematical problems posed in these two languages. we conduct this study on three popular models: gpt-3.5 @xcite , gpt-4 @xcite and llama3-8b(lla). the reason for selecting these models were that the chat interface of gpt-3.5 is now available freely, making it the most accessible model for students. gpt-4 is a superior model, but is available against a small monthly fees, and so, is reasonably accessible to students. llama3-8b is a small "open" model that can be run on notvery-expensive gpus; hence, we believe that soon, schools may decide to host such models within their premises for their students. we made the following inferences from our study: this work makes the following contributions: in the future, we intend to broaden the scope of this study to more languages, more models and more prompting strategies. ## overview in this work, we attempt to study the following research questions: to conduct our analysis, we create our own dataset sourcing problems in the area of combinatorics from higher secondary mathematics ncert textbook (nce) in hindi and english languages. the dataset contains total of 100 problems in english sourced from english version of the ncert book and their corresponding parallel counterparts in hindi sourced from hindi version of the ncert book. these problems can be categorised into five subcategories: fundamental principle of counting, permutation with restrictions, permutation without restrictions, combination with restrictions, and combination without restrictions. the distribution of problems in these subcategories are shown in we conducted experiments on three well known large language models: llama3-8b, gpt-3.5 turbo-175b and gpt-4 turbo. we used the api calls for the inference of llama3, and chat version of gpt-3.5 turbo and gpt-4 turbo for our experimentation . we conduct all experiments on nvidia rtx a4000 gpus. as the responses of the llms are sampled from a distribution, we execute each prompt thrice: if any of the answers is correct, we mark the problem as solved successfully. we prompt the llms via two prompting strategies: (1) a plain one-shot prompt, requesting the llm to solve the problem, and (2) a chain-ofthought prompt that asks the llm to infer the subcategory, and then asks it solve the problem given the subcategory. we discuss this in the subsequent section. ## one-shot prompting in this set of experiments, we prompted the large language models to solve the provided problem. the prompt instructions remain the same for english and hindi, and only the problem statement is provided in the chosen language. we show an example of the prompts used in figure 4 . figure 2 (without the hashed bars) shows the performance of the llm models for english versus hindi. there indeed seems to be a chasm between the performance of english versus hindi, especially for the smaller llama3-8b model. all the llms show a decline in accuracy when prompted for hindi problems as compared to the english problems. the overall difference between the accuracy of english and hindi problems ranges from 8 percent to 14 percent across all llms. the smallest variation in the accuracies if for the case of gpt-4 and highest variation is observed in gpt-3.5. ## chain-of-thought prompting in this strategy, we apply the following steps: a sample prompt given to the llms in this stage is given in figure 1 . ## error analysis the performance of this scheme depends on the following factors: 1. the understanding of hindi language by the large language models, i.e. how well llama, gpt-3.5 and gpt-4 understand hindi? (task 1) 2. the accuracy of the classification into subcategories, i.e. does the llm classify the problems into the right subcategories? (task 2) 3. the accuracy of problem solving once the subcategory is provided. (task 3) for task 1, we utilized hindi comprehension problems derived from ncert textbooks (nce) to evaluate the performance of large language models (llms). specifically, we curated a dataset comprising ten passages in hindi, each accompanied by five corresponding questions. these passages and questions were directly provided as prompts to the llms to assess their accuracy on this task. our results indicate that llama3-8b achieved an accuracy of 50%, whereas gpt-3.5 and gpt-4 turbo both attained 76% accuracy. these findings highlight the superior proficiency of gpt-3.5 and gpt-4 turbo in understanding hindi compared to the smaller llama3 model. this also concludes the similar trends observed in task involving combinatorics problems framed in hindi, further corrobo-rating the relative strengths of gpt-based models in processing the hindi language. to further understand the impact of this on combinatorics problems, we ran another set of experiments in task 3 where we manually provided these subcategories within the prompt. the first two bars in the plots 5 and 6 show the solving accuracy corresponding to one-shot and chain-of-thought prompting (for english and hindi, respectively). the third bar shows the accuracy of the end-to-end pipeline for solving the mathematical problems if the subcategory is provided (manually) within the prompt; we refer to this as "manual subcategory". we highlight the inference of the performance of language models on problems posed in english with chain-of-thought prompts and manual subcategory prompts from the second and third bar of the plot 5 after the results obtained in task 2. interestingly, llama3-8b provides a curious case: though its subcategory inference accuracy is low, the inference accuracy of the end-to-end pipeline increases with chain-of-thought prompting. still more strangely, its accuracy drops if we manually provide the right categories for english problems. we are still trying to understand this counter-intuitive behavior from llama3-8b. ## related work in this section, we will discuss about any recent works related to our llms solving mathematical problems in english. to the best of our knowledge, there is currently a lack of research on improving the mathematical capabilities of llms in regional languages. attempts have been made to improve the math- (a) (b) figure 8: (a) gpt-3.5 turbo performance in three strategies for english problems (b) gpt-3.5 turbo performance in three strategies for hindi problems: orange bars: chain-of-thought prompt strategy, green bars: manual subcategory prompt strategy and blue bars: one-shot prompt strategy. red line shows there are no samples/problems for which llm chose subcategory accurately. some works targeting multilingual tasks include xstreet @xcite , which targets to improve the reasoning capabilities including but not limited to mathematics of llms across non-english languages: arabic, spanish, russian, chinese, and japanese. here, they leveraged the reasoning capabilities of llms trained on code or programs, which they claim that are good reasoners from their study as compared to the llms trained on non-code data. conceptmath @xcite , another study that targets to analyse and comprehend the llms in mathematical reasoning tasks in english and chinese. they did this study on elementary and middle school level mathematical data. their study focuses ore on the granules of mathematics like statistic, geometry, etc. instead of studying mathematical work problems as a whole. this study contributed to improvement part using an efficient fine-tuning setting where post their analysis on granular level, they used benchmark datasets like math @xcite and gsm-8k @xcite along with their data, to fine-tune the llm to improve it's performance in that mathematical area. @xcite uses chainof-thought technique with high-quality in-context learning exemplars obtained by multilingual dense retrieval to enhance llm's performance in mathematics. ## supplementary materials we encourage readers to review the prompts used and datasets created for this study. the access to the datasets developed and the prompts used to carry out this study is given in this github link: foot_0 . the supplementary materials accompanying this paper include a folder named datasets which includes three csv files, one for each of the language models evaluated in the study, containing problems in permutations and combinations presented in both english and hindi. there is prompts file having the prompts used to generate the responses from llms. furthermore, these prompts can be utilized to interface with the language models. these resources are provided to ensure transparency, reproducibility, and ease of future research based on our findings. ## conclusions and future work our main focus of study was analysing the performance of llms in solving combinatorics problems in hindi so as to assess them, if they can be readily deployed in the education sector. for our study, we used gpt-3.5, a freely available llm with a chat interface; llama3-8b, a small "open" source model that can be run on an affordable gpu, and gpt-4 turbo, one of the most powerful models available currently. in future research, we plan to significantly expand our dataset to encompass over 100 problems per subcategory, aiming to improve both its comprehensiveness and robustness. this effort will facilitate a deeper exploration of mathematical problem-solving across diverse categories, ensuring more representative benchmarks. furthermore, we will broaden the linguistic scope of our study by incorporating additional indian regional languages, such as bengali, tamil, assamese, and urdu, alongside non-indian languages, including greek and arabic. this expansion will enable a cross-cultural examination of mathematical reasoning and problem formulation in various linguistic contexts. to further enhance the scope and impact of our work, we intend to evaluate the capabilities of emerging state-of-the-art language models on our enriched datasets. by incorporating models with improved architectures and training paradigms, we aim to uncover new insights into their generalization and adaptability. additionally, we plan to use our dataset for fine-tuning smaller, efficient models, such as llama3, with a focus on exploring their potential for targeted improvements in performance, particularly in resource-constrained environments. this dual approach promises to deepen our understanding of model behavior while driving innovation in both large-scale and lightweight language model applications. we believe that such studies would benefit a country like india or others (once the analysis and scope of this work expands to other regions and their regional languages), where there exists a large number of regional languages in which education is imparted, and show the way forward for llms effective currently for all segments of the indian population with the intention of expanding this to other countries.