What Makes Math Word Problems Challenging for LLM s? (2024)

Abstract: This paper investigates the question of what makes math word problems (MWPs) in English challenging for large language models (LLMs). We conduct an in-depth analysis of the key linguistic and mathematical characteristics of MWPs. In addition, we train feature-based classifiers to better understand the impact of each feature on the overall difficulty of MWPs for prominent LLMs and investigate whether this helps predict how well LLMs fare against specific categories of MWPs.

## introduction in recent years, large language models (llms) have not only demonstrated huge potential across a range of core nlp tasks @xcite @xcite , but also exhibited a number of emergent abilities, such as an ability to solve mathematical puzzles @xcite . math word problems (mwps) have been proposed as a challenging testbed for llms, as they test not only the ability of the models to deal with purely mathematical expressions, but also their reasoning and natural language understanding abilities @xcite @xcite . experiments show that even quite powerful llms are still challenged by mwps @xcite . at the same time, most previous work has either focused on evaluation of llms' performance on mwps or on changes in their behavior in response to progressive-hint prompting, prompt paraphrasing or similar approaches @xcite @xcite , while an in-depth analysis of what exactly makes math problems challenging for llms is lacking. we aim to address this knowledge gap. a recent study by @xcite demonstrates a strong connection between reading 1 our code, data, and analysis are publicly available at github.com/kvadityasrivatsa/analyzing-llms-for-mwps skills and math outcomes in students. we hypothesize that llms' ability to solve mwps correctly may similarly rely on: (1) the linguistic complexity of the questions; (2) the conceptual complexity of the tasks (e.g., the number of steps and types of math operations involved); and (3) the amount of real-world knowledge required to solve the tasks. supporting this intuition, our preliminary analysis of the gsm8k dataset @xcite suggests that relatively short questions with a small number of described entities, a few calculation steps and a limited range of operators involved in the solution (e.g., mark is 7 years older than amy, who is 15. how old will mark be in 5 years?) are typically answered correctly by a range of llms. at the same time, long questions requiring real-world knowledge (e.g., how many cents there are in a dollar) and extended natural language understanding (nlu) (e.g., interpretation of a lower price) pose challenges for llms (see figure 1 ). in this paper, we formulate and investigate two research questions: (1) which characteristics of the input math word questions make them complex for an llm? and (2) based on these characteristics, can we predict whether a particular llm will be able to solve specific input mwps correctly? ## methodology data we use the gsm8k dataset @xcite , divided into 7, 473 training and 1, 319 test instances, because of the high quality of humangenerated mwps. this dataset contains a diverse set of problems in english with minimal amount of recurring templates. furthermore, the difficulty level of the problems is tailored for llms, allowing for a wide variation in correctness across models and question types, which is ideal for our featurebased analysis. approach we collect solution attempts from several llms to the questions from the gsm8k training and test sets. next, we train statistical classifiers on a filtered subset of questions to predict if they are consistently solved correctly or incorrectly across multiple runs of the models. our approach is relatively simple but it allows us to investigate which of the features are most indicative of the challenges llms face in solving math problems. ## experiments 3.1 solution generation to collect solution attempts from the llms, we use a simple task-specific prompt (see appendix b) to minimize any bias imposed on the model generation. we query each llm 5 times on each question with varying generation seeds and a temperature of 0.8. a soft-matching strategy is then used to extract the final answer from the solutions. using each llm's attempted solutions, every question is assigned a mean success rate using (# of correct answers) / (# of solution attempts). ## conclusions this work aims to identify what aspects of mwps make them difficult for llms to solve. to this end, we extract key features (spanning linguistic, mathematical, and real-world knowledge & nlubased aspects) to predict whether several llms can reliably solve mwps from gsm8k. we find that questions with a high number and diversity of math operations using infrequent numerical tokens are particularly challenging to solve. in addition, we show that lengthy questions with low readability scores and those requiring real-world knowledge are also seldom solved correctly. our future work will rely on these findings to make informed modifications to questions in order to study the impact on llms' reasoning and mwp-solving abilities. figure 4 provides an example of an informed modification, which leads to improved llm performance.