Memory or Reasoning? Explore How LLM s Compute Mixed Arithmetic Expressions (2025)

Abstract: Large language models (LLMs) can solve complex multi-step math reasoning problems, but little is known about how these computations are implemented internally. Many recent studies have investigated the mechanisms of LLMs on simple arithmetic tasks (e.g., a+b , a\times b ), but how LLMs solve mixed arithmetic tasks still remains unexplored. This gap highlights the limitation of these findings in reflecting real-world scenarios. In this work, we take a step further to explore how LLMs compute mixed arithmetic expressions. We find that LLMs follow a similar workflow to mixed arithmetic calculations: first parsing the complete expression, then using attention heads to aggregate information to the last token position for result generation, without step-by-step reasoning at the token dimension. However, **for some specific expressions, the model generates the final result depends on the generation of intermediate results at the last token position, which is similar to human thinking.** Furthermore, we propose a **C**ausal **E**ffect **D**riven **F**ine-tuning method (CEDF) to adaptively enhance the identified key components used to execute mixed arithmetic calculations to improve LLMs reasoning ability.

## introduction in recent years, large language models (llms) have demonstrated excellent performance in complex reasoning tasks @xcite . through large-scale pre-training, in-context learning, and chain-of-thought(cot) techniques, llms can solve math problems at the high school level and above @xcite . however, their complex reasoning ability is relatively poor without using cot @xcite . when solving arithmetic problems without cot, do llms output corresponding answers based on memory directly, or do they perform operations * corresponding author equivalent to arithmetic rules internally (reasoning)? this reflects whether the models have truly understood arithmetic problems, further determining the extent to which we can trust the outputs of llms. in recent years, with the development of interpretability techniques, the understanding of how large models tackle simple text comprehension tasks and perform basic arithmetic calculations has been steadily increasing @xcite @xcite . however, how llms accomplish more complex mathematical reasoning tasks, which are prevalent in real-world applications, still remains mysterious. in this work, we preliminarily explore mixed arithmetic calculation, a complex multi-step mathematical reasoning problem. people solve the values of sub-expressions of mixed arithmetic expressions step by step in the order of the parenthesis, exponents, multiplication, addition, and subtraction (pemdas) rules, ultimately arriving at the correct answer. do llms execute mixed arithmetic calculations in the same rule? 1 to figure out the answer to this question, we conducted the following explorations: 1) based on causal effect attribution, we identify the key components that the model uses to execute different mixed arithmetic expressions. 2) using patchscopes @xcite and pattern analysis, we explore the composition of key components and the functional roles of each component. through knockout experiments, we verified these perspectives. therefore, we confirmed that the model completes mixed arithmetic tasks as follows: as the activation propagates forward, the model first understands task instruction, then comprehensively parses the expression, and finally aggregates the operand information to the last token for result generation. this forward answer generation process is similar to simple arithmetic tasks without step-bystep reasoning, according to the pemdas rule at the token dimension. however, we find that llms exhibit similarities to human thinking in other aspects (perspectives 1 & 2). in summary, llms complete mixed arithmetic tasks in a manner that is neither purely memorybased nor purely reasoning-based, reflecting both similarities and differences compared to human thought processes. compared to previous work, we emphasize the following new perspectives: ## attention head function analysis 6.1 method we conducted comparative studies and pattern analysis to reveal the functionalities of different types of attention heads to understand how attention heads assist mlps in arithmetic calculation tasks. ## conclusion do llms execute mixed arithmetic calculations in the same rule as human? our work thoroughly investigates how llms complete mixed arithmetic tasks to answer this question. on the one hand, the model follows a workflow different from humans to complete this task, namely task understanding, operand representation, expression parsing, information aggregation, and arithmetic "reasoning". on the other hand, the model exhibits patterns similar to human thinking in many details, such as understanding the correlations between different operator types and relying on intermediate results to generate the final result at the last token. in summary, our answer is that neither memory nor reasoning accurately applies. our work provides new insights into understanding the mathematical reasoning capabilities of llms and offers new inspirations for future research, which will help improve the reliability of llms.