Disentangling Text and Math in Word Problems: Evidence for the Bidimensional Structure of Large Language Models’ Reasoning (2025)

Abstract: Do LLMs process text and mathematics as a unified skill, or do these components rely on distinct underlying mechanisms? We investigate this question by disentangling the textual interpretation and mathematical solving steps in word problems drawn from Brazil’s largest college entrance exam (ENEM) and GSM8K, a popular grade school-level benchmark. Using the symbolic solver SymPy, we transform word problems into equivalent purely mathematical representations, isolating equation formulation from textual comprehension. Our extended benchmarks enable a structured analysis of LLM performance across these two dimensions. Through empirical evaluations, we find that small-scale LLMs struggle significantly more with text interpretation than with equation solving, with accuracy dropping by a factor of 2 to 7 when solving full word problems compared to their math-only counterparts. Exploratory factor analysis confirms a bidimensional structure in LLM reasoning, where models exhibit distinct proficiencies in textual and mathematical components, underscoring the need for targeted improvements in language comprehension. By analyzing the latent factors associated with each model, our findings provide a framework for researchers and practitioners to make informed choices when selecting models based on computational costs and the nature of their tasks.

## introduction math word problems represent a crucial category of cognitive tasks evaluated in large language models, requiring decomposing complex textual descriptions into smaller components and applying systematic logical reasoning, typically over multiple steps @xcite . an example of a math word problem is the famous baseball bat and ball cost brain teaser @xcite : a baseball bat and ball together cost one dollar and 10 cents. the bat costs one dollar more than the ball. how much does the ball cost?. arriving at the correct answer requires two steps: (1) interpreting the problem expressed in natural language, which involves extracting and relating variables to build the mathematical equation x + (1 + x) = 1.10, and (2) solving the equation using mathematical reasoning, where the linguistic details are no longer relevant. motivated by educational research showing that humans need a combination of cognitive skills to solve math word problems @xcite , we pose two research questions: (1) how frequently do small-scale language models err in equation construction compared to equation solving? (2) do different language models exhibit systematic differences in their ability to construct equations versus solve them? these questions are important for nlp because they help characterize the limitations of language models in real-world analytical problems, which come in textual, not math forms. while correctly answering a math word problem suggests that a model possesses the necessary skills (though guessing remains a possibility), incorrect answers can stem from different sources: an error in interpreting the question and building the underlying mathematical equation, an error in solving the equation, or both. to decompose these sources of errors, we extend existing benchmarks containing word problems to include each problem in their math form, in which the textual interpretation and formulation of the underlying equation has already been conducted. in the baseball ball price problem, once the story-based question has been understood and the variables identified, it can be translated into a purely mathematical problem: solve x + (1 + x) = 1.10. when a human or ai is presented with this version of the problem, it does not have to account for implicit details or linguistic ambiguities but must still demonstrate mathematical reasoning in solving a linear equation. conversely, when faced with the original word problem, both humans and ais can struggle to construct the correct equation. it is common for college-level students to misinterpret the problem and construct the incorrect equation x+1.00 = 1.10, leading to the wrong answer of 0.10 instead of 0.05. we use word problems sampled from two sources containing problems of different difficulty levels: (1) 40 questions extracted from the largest brazilian college admission-level math exam (enem), which tests highly analytical capabilities @xcite , and (2) an equalsized sample from gsm8k @xcite , a popular benchmark containing grade-school level (and hence easier) word problems. to systematically convert word problems into their pure-math equivalents, we leverage python's symbolic solver sympy @xcite . we use a state-of-theart generative language model (gpt-4-turbo-2024-04-09) to generate the sympy source code that represents the underlying mathematical structure of each problem. this symbolic representation serves as the basis for formulating the pure-math version of the question, removing any linguistic interpretation challenges. after constructing extended benchmarks based on enem and gsm8k and evaluating 35 models from different families in different versions and stages of fine-tuning and with zero-shot and two-shot chain-of-thought prompts, our findings are: in addition to making the extended benchmarks publicly available to encourage further research, our work aligns with the growing trend of designing controlled benchmarks that test specific capabilities of language models by systematically varying key features of the problem space. this approach enables more nuanced analyses beyond accuracy @xcite and increases the relevance and informative power of benchmarking efforts (v. @xcite . ## related work in educational assessments, a word problem is a mathematical or logical problem presented as a narrative rather than a direct mathematical equation @xcite . math word problems require test-takers to identify explicit and implicit quantities in the text, model their relationships through mathematical operations, and apply appropriate mathematical techniques to solve them @xcite . these characteristics make word problems a compelling way to evaluate the reasoning and problem-solving capabilities of ai models, attracting the nlp community for some decades @xcite . with the rise in popularity of large language models, benchmarks such as gsm8k @xcite and math @xcite have become popular choices for assessing model performance, as they provide structured problem sets with varying levels of complexity. to elucidate reasons that make language models fail, there is a growing trend in ai evaluation that designs benchmarks that systematically vary key features of problem formulations to enable a more nuanced analysis @xcite . controlled benchmarks have explored different aspects, including the inclusion of distracting contextual information @xcite , variations in numerical representations @xcite , search space size @xcite , com-positional tasks @xcite and asking the llm whether a given solution is correct @xcite . a critical aspect of those workswhich we build upon -is that accuracy as an aggregate metric is not enough to understand reasoning and hence strategies that evaluate step-specific errors are crucial @xcite @xcite . our work introduces a new controlled dimension to llm evaluation: comparing model performance on textual vs. symbolic representations of math word problems. recent work has used llms to generate word problems @xcite @xcite ; in this work we are interested in systematically converting existing word problems into their equivalent mathematical equations using python's symbolic solver, sympy @xcite . this structured transformation allows for a controlled comparison of language model performance across different representations of the same underlying problem. while previous studies have used symbolic solvers to assist language models in solving word problems @xcite , to the best of our knowledge we are the first to leverage symbolic solvers to construct a controlled benchmark that enables direct comparison between textual and symbolic problem representations in llm evaluation. our work is also inspired by studies on the behavior and skills that humans need to solve math word problems @xcite . the reasons that make word problems challenging for humans have been thoroughly investigated @xcite , and it is accepted in the educational community that linguistic and mathematical factors are distinct, unique skills that contribute to student performance @xcite . recent work has examined word problem characteristics impacting llms and found that they struggle more with longer questions and rare numerical tokens @xcite . in our work, we deepen the analysis on how large language models solve word problems by applying exploratory factor analysis @xcite , a dimensionality reduction technique widely used in the social sciences to uncover latent patterns in complex datasets. while at least one study has employed factor analysis to examine llm response patterns-finding that reasoning, comprehension, and language modeling are distinct high-level capabilities in llms @xcite )-our study extends this approach by leveraging a fine-grained, question-level annotation of each problem (whether it is in the math or word form). this approach allows us to quantify latent cognitive dimensions that differentiate model performance across textual and symbolic formats, providing deeper insight into how llms engage with different types of problem representations. ## benchmarks we have built two benchmarks following the same methodology but with a different source of word problems. the first source is gsm8k @xcite foot_0 , which contains grade school-level problems and has been used by many previous work evaluating language models @xcite . gsm8k is a source of prototype word problems, in which mathematical word problems follow a simple, linear syntax and are relatively short @xcite . to explore questions that involve more complex interactions between text interpretation and mathematical reasoning, we used the math section of the brazilian exame nacional do ensino médio (enem) as a source of complex word problems, characterized by presenting potentially redundant information in a syntax that does not merely mirror the mathematical task @xcite . enem is the world's second-largest university entrance exam after china's gaokao, taken annually by millions of brazilian students (silveira and mauá, 2018), and mathematics is consistently the most challenging section for both human test-takers and language models, with scores around 30% @xcite . according to the official documentation (instituto nacional de estudos e pesquisas educacionais anísio teixeira, 2025), the exam assesses geometric reasoning, magnitudes and measurement interpretation, data analysis from graphs and tables, and trend prediction through extrapolation and interpolation. a typical enem word problem is shown below: the subway system in a municipality offers two types of tickets, differentiated by their colors: blue and red. these tick-ets are sold in booklets, each containing nine tickets of the same color and with the same unit price. two booklets of blue tickets and one booklet of red tickets are sold for r$ 32.40. it is known that the price of one blue ticket minus the price of one red ticket is equal to the price of one red ticket plus five cents. what is the price, in reais, of a booklet of red tickets? transforming the word question in its pure math form. starting from the observation that word problems require building a mathematical equation as a well-recognized subproblem @xcite , we leverage the symbolic solver sympy 2 @xcite to generate mathematical unambiguous representations of word problems. this is how the subway ticket problem is modeled and solved using sympy: we then submit a new prompt to gpt-4 instructing it to convert the sympy code into a question in the format "solve... <equation>.," where all known variables are replaced with their actual values, and only the unknowns are retained. for the subway ticket enem word problem, the equivalent math formulation generated from the sympy code above is solve 9*r from 18b + 9r = 32.40 and @xmath0.05. a key advantage of using sympy is its enforcement of a structured and precise format for mathematical expressions, eliminating unnecessary linguistic details and focusing solely on the mathematical formulation. to construct the extended benchmark, we considered 40 enem math questions from exams spanning 2016 to 2024, using gpt-4 to generate their corresponding sympy representations, yielding a total of 80 questions. we selected questions randomly to ensure diversity regarding difficulty and 2 version 1.12. problem type, excluding those requiring images or tables to focus on language-based mathematical reasoning. the questions were translated into english using gpt-4 before the sympy pipeline. to validate correctness, we executed the generated sympy code and verified that it produced the expected answer. we manually reviewed and curated the sympy representations to ensure accuracy and fidelity to the original problems. the same process has been conducted with gsm8k; the full prompts can be found in appendix a, and examples of questions in their word and math forms can be found in appendix b. there is ongoing debate regarding the efficacy of multiple-choice versus open-ended questions for assessing reasoning abilities @xcite . we opted for open-ended responses to eliminate the confounding effect of random guessing in multiplechoice formats. this ensures cleaner signal integrity in our analysis, reducing noise and enabling more precise evaluation of model reasoning. ## accuracy in word and math forms we evaluate 35 models, focusing on small language small language models to small-scale large language models (1.5b, 3b, 7b, 14b), across 9 families (gemini, qwen, falcon, phi, mistral, gemma, llama, numinamath, distilled versions of deepseek-r1) and training strategies (pre-trained vs. fine-tuned). we included multiple versions of the same family and also math-specialized models (e.g., qwen2.5-math-7b) to track model evolution over time. the models were tested with 0-shot and 2-shot settings, using chain-of-thought prompt styles following prior work @xcite . the prompt details and full list of models is provided in appendixes a and c. ## exploratory factor analysis in pyschometrics, determining the dimensionality of an instrument is an important step towards assessing its validity and reliability. it informs whether the instrument measures one or multiple constructs, how its scores should be interpreted, and which models such as item response theory apply @xcite . unidimensional instruments (i.e., those measuring a single construct) tend to have high internal consistency because all items are strongly correlated with the same underlying factor. a popular metric is cronbach's alpha (α) @xcite , which measures the internal consistency of a set of items by calculating the average correlation between all pairs of items and considering how each item contributes to the overall variance of the test scores. for the enem and gsm8k we found α of 0.972 and 0.978, respectively, indicating very high internal consistency and minimal random error. the high consistency we found supports that the size of our benchmarks is enough for our study and is consistent with previous work that found that large benchmarks such as gsm8k, albeit containing thousands of items, can be reduced to a few dozen itens and keep a good degree of discrimination among llms @xcite . however, an instrument exhibiting high internal consistency is not necessarily unidimensional: exceeding those of the random data are considered significant, suggesting they account for meaningful variance (timmerman and @xcite . our experiments revealed two statistically significant factors on both benchmarks. details are provided in factor scores. do specific versions of a family of models possess different problem-solving skills? to answer that question, we resort to factor scores @xcite , which are numerical values representing an observation's position on the latent factors identified through factor analysis. these scores are derived from observed variable scores and factor loadings, providing a way to es-timate the contribution of each factor to a specific case. in explaining factors. recent work has looked at the characteristics that make word problems harder for language models (srivatsa and kochmar, 2024). we can extend this kind of analysis by investigating if there are characteristics that link a question to either f1 or f2. notably, in word forms, additional tokens, terms, operators, and equation depth have a greater impact on the overall difficulty of the question. for example, while questions in their mathematical form are linked to f2 when they contain, on average, 13.67 terms, in word form, just 9.13 terms are sufficient. ## conclusions analytical problems require multiple capabilities, including understanding the problem statement and extracting and solving a mathematical model. to disentangle natural language understanding from pure mathematical reasoning, we used the symbolic solver sympy to introduce extended versions of benchmarks (brazilian enem and gsm8k) that present math word problems in both word-based forms and mathematical components only. through structured problem reformulation and factor analysis of 35 large language models, we found that while pure mathematical problemsolving skills are relatively common, only a subset of small-scale language models can effectively handle word problems requiring both interpretation and mathematical modeling. our work contributes to the growing body of research focused on developing benchmarks that test specific model capabilities by systematically varying key features @xcite , and, as models become more multi-modal and diverse in their reasoning capabilities, factor analysis can play a larger role as part of language model evaluation frameworks.