Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics (2025)

Abstract: Assisting LLMs with code generation improved their performanceon mathematical reasoning tasks.However, the evaluation of code-assisted LLMs is generally restricted to execution correctness, lacking a rigorous evaluation of their generated programs.In this work, we bridge this gap by conducting an in-depth analysis of code-assisted LLMs generated programs in response to math reasoning tasks, with a focus on evaluating the soundness of the underlying reasoning processes. For this purpose, we assess the generations of five LLMs, on several math datasets, both manually and automatically, and propose a taxonomy of generated programs based on their logical soundness.Our findings show that the capabilities of models significantly impact the logic implemented to solve the problem. Closed-source LLMs ground their programs in mathematical concepts, whereas open-source models often resort to unsound reasoning, relying on memorized information and exhaustive searches. Furthermore, increasing the difficulty of problems decreases sound generations for all models, revealing a critical shortcoming of LLMs on complex mathematics, contrary to what accuracy metrics suggest.Our work highlights the need for more holistic evaluations of code-assisted LLMs beyond execution accuracy metrics, toward a better understanding of LLMs’ limits in the math domain.

## introduction large language models (llms) have recently achieved outstanding performance on complex reasoning tasks such as mathematical reasoning, powered by scale and multi-step reasoning approaches. particularly, the chain-of-thought (cot) @xcite requires an llm to generate the explicit reasoning steps, before generating the final answer. despite its success, investigating cot reasoning steps revealed critical flows of llms, such as committing calculation errors @xcite and generating false positive chains @xcite , i.e: containing reasoning errors yet generating correct final answers. code-assisted reasoning approaches @xcite @xcite @xcite proposed to solve these problems by instructing llms to generate programmatic reasoning steps instead, e.g: python programs, and delegate their execution to an external interpreter, which ensures precise calculations and faithfulness. such approaches have been found to further improve llms' performance on math tasks. however, performance improvement is predominantly measured by the correctness of the execution outcome @xcite @xcite , rather than the quality of the generated programs and the underlying reasoning process. this is problematic, as the generated programs can rely on exhaustive searches or memorized information to produce correct answers, leading to untrusted and more difficult-to-verify programs. figure 1 shows programs generated by several llms using these hacks when solving math problems. the goal addressed in this work is to evaluate the reasoning processes of code-assisted llms when solving mathematical tasks by analyzing their generated programs. in our evaluation, we focus on assessing the soundness of the logic governing llms' solutions and its impact on end performance. we also assess other aspects of the generated programs, such as api calls, complexity, and the most common errors, for a more comprehensive evaluation. our assessment begins by manually analyzing a subset of the generated programs, produced by gpt4o-mini, gpt4, qwen2.5, llama3, and star-coder2, as they solve math problems from the as-div and math500 datasets (3.4). given the observations from the manual analysis, we design a taxonomy reflecting the different logic types used by evaluated models (3.5). to extend the analysis to the complete set of generated programs, we employ ## evaluation of intermediate programs the primary focus of this evaluation is to assess the soundness of underlying reasoning processes, i.e: the logic implemented in the generated programs of code-assisted llms. we consider a program to be logically sound if it grounds the implementation in a math concept, where a math concept refers to the principle used to solve a specific mathematical problem, e.g: using the euclidean algorithm to find the greatest common divisor of two numbers. sound programs align more with human reasoning and can be easily verified and trusted. in contrast, unsound programs are harder to verify and can't guarantee finding a solution to the given problem. additionally, we examine the characteristics of generated programs in terms of cyclomatic complexity, types of errors, and api calls to provide a more comprehensive evaluation. ## conclusion in this work, we conducted an in-depth analysis of code-assisted llms generated programs in response to math problems. our assessment focuses on evaluating the logical soundness of underlying reasoning processes implemented in the generated programs. we categorized the generated programs into six different categories, which make up our proposed taxonomy of logical soundness. three of which represent sound reasoning that ground the programs in verifiable math concepts. in contrast, the other three exploit memorized information or exhaustive searches to find final answers. we an-