Beyond the Hype: Identifying and Analyzing Math Word Problem-Solving Challenges for Large Language Models (2024)

Abstract: Despite not being explicitly trained for this purpose, models like Mistral and LLaMA have demonstrated impressive results across numerous tasks, including generating solutions to Mathematical Word Problems (MWPs). A MWP involves translating a textual description into a mathematical model or equation that solving it. However, these models face challenges in accurately interpreting and utilizing the numerical information present in the MWP statements, which can lead to errors in the generated solutions. To better understand the limitations of LLMs, we analyzed the MWP where models failed to accurately solve problems from the SVAMP dataset. By categorizing these MWPs, we identify specific types of problems where the models are most prone to errors, providing insights into the underlying challenges faced by LLMs in problem-solving scenarios and open new modeling opportunities. By understanding the expected errors, researchers can design strategies to adequately model problems more effectively and choose the most suitable LLM for solving them taking into account each model’s strengths and weaknesses.

## introduction llms have expanded the boundaries of understanding and generating natural language @xcite . moreover, recent research has found llms to be capable of producing high-quality source code @xcite . llms excel at producing text sequences, but also show reasoning capabilities that have been previously applied to math word problem (mwp) solving @xcite by transforming the mwp in natural language to the mathematical language. in this context, recent research @xcite has explored llms in the context of education by producing source-code that can be compiled into a solution graph for tutoring and supervisation purposes. this paper aims to investigate the types of mwp statements that llms have difficulties solving by analyzing incorrect samples produced in previous studies. to this end, we select the svamp dataset and three different models: openmath/mistral-7b from nvidia, llama3-8b foot_0 , and codellama 34b foot_1 @xcite , which demonstrated high performance in mwp-solving task. by focusing on problem statements where these models failed, we identify patterns in the sources of errors. the provided analysis foot_2 can direct research towards a better understanding of the reasoning limitations of language models. ## background and related work a mwp model solution can be understood as the result of reducing the initial mwp to a graph of mathematical relationships between quantities. consider the mwp where we have bought a car and paid 12 bills of 400 euros each. if the car costs 12 800 euros, we need to determine how much money is left to pay. a possible problem model would establish the following relationships: the total price of the car equals the sum of the money already paid and the money left to pay. furthermore, the money already paid is calculated by multiplying the value of a single bill by the number of bills paid. automatically solving a math problem articulated in natural language presents a significant challenge, necessitating both comprehension and accurate reasoning. this process requires techniques to extract not only the quantities explicitly stated in the mwp but also those implied by terms such as "twice", "half" or "left". additionally, solving the mwp demands an understanding of the relationships among these quantities, the identification of the target quantity, and the sequence of operations needed to achieve the final result. in essence, solving a problem from natural language is a task primarily concerned with knowledge extraction and the identification of advanced relationships @xcite . recent studies have shown that llms show a problem-solving ability similar to that of children, despite differences in the type of mwp they are able to solve best @xcite . the task of automated mwp solving has been a topic of interest in the literature since the 1960s, inspiring a recent survey @xcite . recent efforts in solving mathematical word problems (mwps) have concentrated on constructing expression (or equation) trees. these methods focus on creating arithmetic expressions by forming equivalent trees. however, due to the exponential growth of the search space as the number of quantities increases, alternatives leveraging reinforcement learning techniques have been explored @xcite . in the last year, prompting techniques have been developed to force reasoning on decoder-only transformers @xcite . moreover, the introduction of new-generation llms like llama2 and mistral has also led to new studies in the field. in this direction, @xcite proposed a method that incorporates mwp solving, quantity value assignment and naming as well as capabilities for establishing relationships, without the need for fine-tuning the underlying llm. ## dataset the svamp mwp dataset @xcite consists of 1000 elementary-level arithmetic word problems, each solvable by expressions requiring no more than two operators. this dataset provides annotated solutions for each mwp. svamp was selected for evaluating and analyzing the performance of llms in solving mwps, being widely recognized as one of the most challenging datasets for arithmetic mwp solving @xcite . ## analysis we focused our analysis on the characteristics of mwps that llms tend to solve incorrectly. these mwps have been selected by studying the generated samples obtained in previous studies @xcite . out of the three studied models we have chosen codellama-34b as the worst model, as it provided the worst accuracy on the svamp dataset. we decided to delve into those problems that the worst performing model (codellama-34b) failed to solve accurately, in order to draw some conclusions on the structure of said mwps. after an initial visual inspection, we found specific patterns in the mwps for which models tend to fail and categorized them to better understand the limitations of the models based on the type of mwp they are trying to solve, into at least one of the following three types: mwp with unfeasible solutions (us), mwp with unnecessary quantities (uq), and mwp involving comparisons (cp). mwps that fall into the unfeasible solution category are mwps that, although can be solved analytically, the solution obtained is not practical or possible in the presented scenario. this happens, for instance, when one or more quantities in the solution take an integer or real value where only natural numbers are physically possible. an example of this type of statement is "a waiter had 12 customers. after some left he still had 14 customers. then he got 10 new customers. how many customers does he have now?". this problem implies that somehow -2 customers left the restaurant. mwps with the unnecessary quantities category contain one or more quantities that are not required to solve the problem. we have observed that most llms have a natural tendency to use all the quantities present in the statement to produce a solution, typically leading to errors in the reasoning. a good example of these mwp is "rebecca wants to split a collection of eggs into groups of 6. rebecca has 18 eggs 72 bananas and 66 marbles. how many groups will be created?". in this case, bananas and marbles amounts are unnecessary in determining how many groups of eggs will be created. however, models will still add it to the number of eggs. what does the other categories have been created by combining the previously identified categories, us∧uq, us∧cp, uq∧cp, and us∧uq∧cp. these categories contain the samples that can be tagged in more than one category. finally, an additional category unidentified has been created, containing the mwps that have not been labelled in any of the previous categories. this type of problems has a simple structure with no irrelevant comparison elements or quantities. mwps that codellama-34b fails to solve correctly have been classified by two independent volunteer annotators. the annotators were asked to tag which of the identified problems were present in each of the selected mwps. according to both annotators' responses we computed cohen's kappa for each of the three separate problems. in all cases, we obtained a κ > 0.6, indicating substantial agreement was achieved by the annotators. finally, since annotator #1 had more experience in the field, it was decided to choose samples from that annotator. the analysis of the classification of the selected samples shows that the mwps can be classified into at least one of the identified categories in over 70% of the mwps which codellama fails to solve. codellama fails to solve 45.8% of problems in the uq category, indicating that these problems present the greatest challenge for the model. 37.8% in the cp category, and 21.8% in the us category. additionally, 14.5% of the problems fall into both the us and uq categories, 10.9% fall into both the us and cp categories, 16% fall into both the uq and cp categories, and 6.9% fall into all three categories: us, uq, and cp. finally, 29.1% of the problems do not fall into any of these specified categories. an intriguing observation arises when examining this category of the unsolved problems. these problems typically have clearer statements and generally require a straightforward operation to reach a solution. despite their apparent simplicity, codellama still faces notable challenges in solving these problems. in summary, a total of 275 mwps (those which codellama failed to solve) were selected for analysis in the top two top-performing models. within the 275 problems incorrectly solved by codellama, we analyzed the errors made by the mistral and llama models. mistral accounted for errors in 173 problems, with 60% of these errors being attributable to the same issues present in the codellama model. similarly, llama had 247 problems with erroneous solutions, and 58% of these errors could be explained by the same mistakes made by codellama. this indicates that we were able to identify the characteristics of mwps of 60% of mistral's errors and 58% of llama's errors, respectively. through this analysis, we can understand a significant portion of the common sources of errors in both mistral and llama. in ## conclusion and future work in this work, we have analyzed the performance of three llms in solving mwps on the svamp dataset and categorized the sources of errors. the categorization of mwp statement error sources reflects specific patterns in which the models fail to correctly solve these problems. identifying these patterns provides valuable insights into the limita-tions of current llms. the provided study shows that, in general, there are three categories of challenging problems for which models tend to generate wrong solutions. moreover, the results show that statistically, models tend to fail significantly more in problems that fall into one of these categories than in any other type of problems. the fact that these categories can be identified, and the shown difference in performance in these categories shows that llms are still weak for certain types of reasoning. the identification of the reasons that cause an incorrect solution in the case of statements that cannot be classified in any of the 3 identified categories is not straightforward. the initial inspection showed that these problems apparently have a clear statement, and there is no reason as to why the models consistently fail to solve the problem. a possible explanation of this issue might be related to the type of relations encoded in the mwps, as suggested by previous research @xcite . the presented analysis is a work in progress which examines the characteristics of mwp statements where the selected llms fail to provide correct solutions. the initial categorical classification offered as part of this work serves as a preliminary step towards modeling math problems based on categories that reflect the likelihood of being correctly solved by different llms. future work will continue analysing the samples for which the topperforming models fail, in order to gather insights into the reasoning gaps and generate strategies to overcome such failures. we also plan to examine and compare the error rates across different categories made by llms with those made by real students.